can't run resampling query (#9) · Issues · felyx / felyx_server

can't run resampling query

the following query fails with an exception.....

#!bash

felyx query '{"extraction": {"remapping": {"hrdds": false, "input": {"selection": [{"start_time": "2007-10-04T00:00:01", "site_list": ["ghr057"], "stop_time": "2009-10-05T00:00:00", "data_type": "miniprods", "dataset_list": ["S3A_OL_2_WRR"]}]}, "resolution": 0.01, "linear_zero_weight_distance": 25000}}, "image_format": "png", "operation_name": "image_map", "value_range": [-1.0, 1.0], "field": "CHL_OC4ME"}' /export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85 -v

2015-07-02 22:36:52,639 (PID 6285) felyx (DEBUG): Executing query with output going to /export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85.
2015-07-02 22:36:52,723 (PID 6285) felyx_work.miniprod.tools (DEBUG): Path /export/home/sumba/project/felyx/backend/workspace/jobs already exists.
2015-07-02 22:36:52,725 (PID 6285) felyx_work.miniprod.tools (DEBUG): Path /export/home/sumba/project/felyx/backend/workspace/jobs already exists.
2015-07-02 22:36:52,728 (PID 6285) felyx_work.query (DEBUG): Attempting to process query: {u'extraction': {u'remapping': {u'hrdds': False, u'input': {u'selection': [{u'start_time': u'2007-10-04T00:00:01', u'dataset_list': [u'S3A_OL_2_WRR'], u'site_list': [u'ghr057'], u'data_type': u'miniprods', u'stop_time': u'2009-10-05T00:00:00'}]}, u'resolution': 0.01, u'linear_zero_weight_distance': 25000}}, u'image_format': u'png', u'operation_name': u'image_map', u'value_range': [-1.0, 1.0], u'field': u'CHL_OC4ME'}
2015-07-02 22:36:52,728 (PID 6285) felyx_work.query.handler (DEBUG): Adding default JSON packaging step to query.
2015-07-02 22:36:52,750 (PID 6285) felyx_work.query.selection (DEBUG): Selection returned the following results: {"step_name": "selection", "results": {"S3A_OL_2_WRR": {"ghr057": {"miniprods": ["20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc"], "times": ["2008-01-08T07:06:57"]}}}, "arguments": {"selection": {}}}
2015-07-02 22:36:52,750 (PID 6285) felyx_work.query.handler (INFO): Requesting execution of query task: {'inputs': [None], 'granularity': 1, 'sequence': [{'args': {u'hrdds': False, 'input_data': {u'results': [{u'S3A_OL_2_WRR': {u'ghr057': {u'miniprods': [u'20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc'], u'urls': [u'http://127.0.0.1:1080/felyx/extraction/miniprod/20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc'], u'times': [u'2008-01-08T07:06:57']}}}]}, u'resolution': 0.01, u'linear_zero_weight_distance': 25000}, 'taskID': u'remapping'}, {'args': {'outfile': '/export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85'}, 'taskID': 'json'}]}
2015-07-02 22:36:52,750 (PID 6285) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/45d0b879-c9aa-44f8-94eb-73d5ecd45beb-201507022236.
2015-07-02 22:36:52,750 (PID 6285) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/45d0b879-c9aa-44f8-94eb-73d5ecd45beb-201507022236/logs.
2015-07-02 22:36:52,750 (PID 6285) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/45d0b879-c9aa-44f8-94eb-73d5ecd45beb-201507022236/heap.
2015-07-02 22:36:52,750 (PID 6285) felyx_work.schedulers.local (DEBUG): Job cost: 2
 Total cost: 2
2015-07-02 22:36:52,751 (PID 6285) felyx_work.schedulers (DEBUG): Created task file /export/home/sumba/project/felyx/backend/workspace/jobs/45d0b879-c9aa-44f8-94eb-73d5ecd45beb-201507022236/task.json.
WARNING:root:Unable to load 'contrib' plugins, are there any?
2015-07-02 22:36:53,018 (PID 6285) felyx_work.processors (DEBUG): TaskID:45d0b879-c9aa-44f8-94eb-73d5ecd45beb
2015-07-02 22:36:53,018 (PID 6285) felyx_work.processors (DEBUG): JobID:0
2015-07-02 22:36:53,018 (PID 6285) felyx_work.processors (DEBUG): StepCount:2
2015-07-02 22:36:53,018 (PID 6285) felyx_work.processors (DEBUG): Input:None
2015-07-02 22:36:53,019 (PID 6285) felyx_work.processors.plugins.baseplugin (ERROR): Plugin task remapping raised an exception. The following was provided by the plugin: The input to the remmaping plugin must contain either the results from a selection step or a colocation plugin.
Traceback (most recent call last):
  File "/opt/felyx/env/local/lib/python2.7/site-packages/felyx_work/processors/plugins/baseplugin.py", line 456, in perform
    raise Exception(return_value['error'])
Exception: The input to the remmaping plugin must contain either the results from a selection step or a colocation plugin.
2015-07-02 22:36:53,019 (PID 6285) felyx_work.processors (INFO): X0:False,0,Failed
2015-07-02 22:36:53,019 (PID 6285) felyx_work.processors (ERROR): >0s

note that the selection alone (in above workflow) does work

#!bash

felyx query '{"extraction":  {"selection": [{"start_time": "2007-10-04T00:00:01", "site_list": ["ghr057"], "stop_time": "2009-10-05T00:00:00", "data_type": "miniprods", "dataset_list": ["S3A_OL_2_WRR"]}]}}' /export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85 -v


2015-07-02 22:40:51,326 (PID 6435) felyx (WARNING): A request was made to process a query, with the output going to file /export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85. This file already exists. If you are not running in test mode, this may be an indication of an attack.
2015-07-02 22:40:51,326 (PID 6435) felyx (DEBUG): Executing query with output going to /export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85.
2015-07-02 22:40:51,408 (PID 6435) felyx_work.miniprod.tools (DEBUG): Path /export/home/sumba/project/felyx/backend/workspace/jobs already exists.
2015-07-02 22:40:51,411 (PID 6435) felyx_work.miniprod.tools (DEBUG): Path /export/home/sumba/project/felyx/backend/workspace/jobs already exists.
2015-07-02 22:40:51,413 (PID 6435) felyx_work.query (DEBUG): Attempting to process query: {u'extraction': {u'selection': [{u'start_time': u'2007-10-04T00:00:01', u'dataset_list': [u'S3A_OL_2_WRR'], u'site_list': [u'ghr057'], u'data_type': u'miniprods', u'stop_time': u'2009-10-05T00:00:00'}]}}
2015-07-02 22:40:51,413 (PID 6435) felyx_work.query.handler (DEBUG): Adding default JSON packaging step to query.
2015-07-02 22:40:51,425 (PID 6435) felyx_work.query.selection (DEBUG): Selection returned the following results: {"step_name": "selection", "results": {"S3A_OL_2_WRR": {"ghr057": {"miniprods": ["20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc"], "times": ["2008-01-08T07:06:57"]}}}, "arguments": {"selection": {}}}
2015-07-02 22:40:51,425 (PID 6435) felyx_work.query.handler (INFO): Requesting execution of query task: {'inputs': [None], 'granularity': 1, 'sequence': [{'args': {'input_data': {u'results': [{u'S3A_OL_2_WRR': {u'ghr057': {u'miniprods': [u'20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc'], u'urls': [u'http://127.0.0.1:1080/felyx/extraction/miniprod/20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc'], u'times': [u'2008-01-08T07:06:57']}}}]}, 'outfile': '/export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85'}, 'taskID': 'json'}]}
2015-07-02 22:40:51,425 (PID 6435) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/14d4e9fc-e4bf-4c07-b7ac-fbbaa395a1d0-201507022240.
2015-07-02 22:40:51,425 (PID 6435) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/14d4e9fc-e4bf-4c07-b7ac-fbbaa395a1d0-201507022240/logs.
2015-07-02 22:40:51,425 (PID 6435) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/14d4e9fc-e4bf-4c07-b7ac-fbbaa395a1d0-201507022240/heap.
2015-07-02 22:40:51,426 (PID 6435) felyx_work.schedulers.local (DEBUG): Job cost: 1
 Total cost: 1
2015-07-02 22:40:51,426 (PID 6435) felyx_work.schedulers (DEBUG): Created task file /export/home/sumba/project/felyx/backend/workspace/jobs/14d4e9fc-e4bf-4c07-b7ac-fbbaa395a1d0-201507022240/task.json.
WARNING:root:Unable to load 'contrib' plugins, are there any?
2015-07-02 22:40:51,695 (PID 6435) felyx_work.processors (DEBUG): TaskID:14d4e9fc-e4bf-4c07-b7ac-fbbaa395a1d0
2015-07-02 22:40:51,695 (PID 6435) felyx_work.processors (DEBUG): JobID:0
2015-07-02 22:40:51,695 (PID 6435) felyx_work.processors (DEBUG): StepCount:1
2015-07-02 22:40:51,696 (PID 6435) felyx_work.processors (DEBUG): Input:None
2015-07-02 22:40:51,696 (PID 6435) felyx_work.processors (INFO): .0:True,0,Success
2015-07-02 22:40:51,696 (PID 6435) felyx_work.processors (INFO): >0s
2015-07-02 22:40:51,696 (PID 6435) felyx_work.processors (INFO): Output was: True

the following query fails with an exception.....

```
#!bash

felyx query '{"extraction": {"remapping": {"hrdds": false, "input": {"selection": [{"start_time": "2007-10-04T00:00:01", "site_list": ["ghr057"], "stop_time": "2009-10-05T00:00:00", "data_type": "miniprods", "dataset_list": ["S3A_OL_2_WRR"]}]}, "resolution": 0.01, "linear_zero_weight_distance": 25000}}, "image_format": "png", "operation_name": "image_map", "value_range": [-1.0, 1.0], "field": "CHL_OC4ME"}' /export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85 -v

2015-07-02 22:36:52,639 (PID 6285) felyx (DEBUG): Executing query with output going to /export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85.
2015-07-02 22:36:52,723 (PID 6285) felyx_work.miniprod.tools (DEBUG): Path /export/home/sumba/project/felyx/backend/workspace/jobs already exists.
2015-07-02 22:36:52,725 (PID 6285) felyx_work.miniprod.tools (DEBUG): Path /export/home/sumba/project/felyx/backend/workspace/jobs already exists.
2015-07-02 22:36:52,728 (PID 6285) felyx_work.query (DEBUG): Attempting to process query: {u'extraction': {u'remapping': {u'hrdds': False, u'input': {u'selection': [{u'start_time': u'2007-10-04T00:00:01', u'dataset_list': [u'S3A_OL_2_WRR'], u'site_list': [u'ghr057'], u'data_type': u'miniprods', u'stop_time': u'2009-10-05T00:00:00'}]}, u'resolution': 0.01, u'linear_zero_weight_distance': 25000}}, u'image_format': u'png', u'operation_name': u'image_map', u'value_range': [-1.0, 1.0], u'field': u'CHL_OC4ME'}
2015-07-02 22:36:52,728 (PID 6285) felyx_work.query.handler (DEBUG): Adding default JSON packaging step to query.
2015-07-02 22:36:52,750 (PID 6285) felyx_work.query.selection (DEBUG): Selection returned the following results: {"step_name": "selection", "results": {"S3A_OL_2_WRR": {"ghr057": {"miniprods": ["20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc"], "times": ["2008-01-08T07:06:57"]}}}, "arguments": {"selection": {}}}
2015-07-02 22:36:52,750 (PID 6285) felyx_work.query.handler (INFO): Requesting execution of query task: {'inputs': [None], 'granularity': 1, 'sequence': [{'args': {u'hrdds': False, 'input_data': {u'results': [{u'S3A_OL_2_WRR': {u'ghr057': {u'miniprods': [u'20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc'], u'urls': [u'http://127.0.0.1:1080/felyx/extraction/miniprod/20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc'], u'times': [u'2008-01-08T07:06:57']}}}]}, u'resolution': 0.01, u'linear_zero_weight_distance': 25000}, 'taskID': u'remapping'}, {'args': {'outfile': '/export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85'}, 'taskID': 'json'}]}
2015-07-02 22:36:52,750 (PID 6285) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/45d0b879-c9aa-44f8-94eb-73d5ecd45beb-201507022236.
2015-07-02 22:36:52,750 (PID 6285) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/45d0b879-c9aa-44f8-94eb-73d5ecd45beb-201507022236/logs.
2015-07-02 22:36:52,750 (PID 6285) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/45d0b879-c9aa-44f8-94eb-73d5ecd45beb-201507022236/heap.
2015-07-02 22:36:52,750 (PID 6285) felyx_work.schedulers.local (DEBUG): Job cost: 2
 Total cost: 2
2015-07-02 22:36:52,751 (PID 6285) felyx_work.schedulers (DEBUG): Created task file /export/home/sumba/project/felyx/backend/workspace/jobs/45d0b879-c9aa-44f8-94eb-73d5ecd45beb-201507022236/task.json.
WARNING:root:Unable to load 'contrib' plugins, are there any?
2015-07-02 22:36:53,018 (PID 6285) felyx_work.processors (DEBUG): TaskID:45d0b879-c9aa-44f8-94eb-73d5ecd45beb
2015-07-02 22:36:53,018 (PID 6285) felyx_work.processors (DEBUG): JobID:0
2015-07-02 22:36:53,018 (PID 6285) felyx_work.processors (DEBUG): StepCount:2
2015-07-02 22:36:53,018 (PID 6285) felyx_work.processors (DEBUG): Input:None
2015-07-02 22:36:53,019 (PID 6285) felyx_work.processors.plugins.baseplugin (ERROR): Plugin task remapping raised an exception. The following was provided by the plugin: The input to the remmaping plugin must contain either the results from a selection step or a colocation plugin.
Traceback (most recent call last):
  File "/opt/felyx/env/local/lib/python2.7/site-packages/felyx_work/processors/plugins/baseplugin.py", line 456, in perform
    raise Exception(return_value['error'])
Exception: The input to the remmaping plugin must contain either the results from a selection step or a colocation plugin.
2015-07-02 22:36:53,019 (PID 6285) felyx_work.processors (INFO): X0:False,0,Failed
2015-07-02 22:36:53,019 (PID 6285) felyx_work.processors (ERROR): >0s

```
note that the selection alone (in above workflow) does work

```
#!bash

felyx query '{"extraction":  {"selection": [{"start_time": "2007-10-04T00:00:01", "site_list": ["ghr057"], "stop_time": "2009-10-05T00:00:00", "data_type": "miniprods", "dataset_list": ["S3A_OL_2_WRR"]}]}}' /export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85 -v

2015-07-02 22:40:51,326 (PID 6435) felyx (WARNING): A request was made to process a query, with the output going to file /export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85. This file already exists. If you are not running in test mode, this may be an indication of an attack.
2015-07-02 22:40:51,326 (PID 6435) felyx (DEBUG): Executing query with output going to /export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85.
2015-07-02 22:40:51,408 (PID 6435) felyx_work.miniprod.tools (DEBUG): Path /export/home/sumba/project/felyx/backend/workspace/jobs already exists.
2015-07-02 22:40:51,411 (PID 6435) felyx_work.miniprod.tools (DEBUG): Path /export/home/sumba/project/felyx/backend/workspace/jobs already exists.
2015-07-02 22:40:51,413 (PID 6435) felyx_work.query (DEBUG): Attempting to process query: {u'extraction': {u'selection': [{u'start_time': u'2007-10-04T00:00:01', u'dataset_list': [u'S3A_OL_2_WRR'], u'site_list': [u'ghr057'], u'data_type': u'miniprods', u'stop_time': u'2009-10-05T00:00:00'}]}}
2015-07-02 22:40:51,413 (PID 6435) felyx_work.query.handler (DEBUG): Adding default JSON packaging step to query.
2015-07-02 22:40:51,425 (PID 6435) felyx_work.query.selection (DEBUG): Selection returned the following results: {"step_name": "selection", "results": {"S3A_OL_2_WRR": {"ghr057": {"miniprods": ["20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc"], "times": ["2008-01-08T07:06:57"]}}}, "arguments": {"selection": {}}}
2015-07-02 22:40:51,425 (PID 6435) felyx_work.query.handler (INFO): Requesting execution of query task: {'inputs': [None], 'granularity': 1, 'sequence': [{'args': {'input_data': {u'results': [{u'S3A_OL_2_WRR': {u'ghr057': {u'miniprods': [u'20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc'], u'urls': [u'http://127.0.0.1:1080/felyx/extraction/miniprod/20080108070657_ghr057_S3A_OL_2_WRR_S3A_OL_2_LRR____20080108T065652_20080108T074025_20150521T125214_2613_084_049______LN1_D_NT_001.nc'], u'times': [u'2008-01-08T07:06:57']}}}]}, 'outfile': '/export/home/sumba/project/felyx/backend/workspace/web_requests/66a607a9-083b-4a5f-b0d3-470a1bb19c85'}, 'taskID': 'json'}]}
2015-07-02 22:40:51,425 (PID 6435) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/14d4e9fc-e4bf-4c07-b7ac-fbbaa395a1d0-201507022240.
2015-07-02 22:40:51,425 (PID 6435) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/14d4e9fc-e4bf-4c07-b7ac-fbbaa395a1d0-201507022240/logs.
2015-07-02 22:40:51,425 (PID 6435) felyx_work.miniprod.tools (DEBUG): Creating path /export/home/sumba/project/felyx/backend/workspace/jobs/14d4e9fc-e4bf-4c07-b7ac-fbbaa395a1d0-201507022240/heap.
2015-07-02 22:40:51,426 (PID 6435) felyx_work.schedulers.local (DEBUG): Job cost: 1
 Total cost: 1
2015-07-02 22:40:51,426 (PID 6435) felyx_work.schedulers (DEBUG): Created task file /export/home/sumba/project/felyx/backend/workspace/jobs/14d4e9fc-e4bf-4c07-b7ac-fbbaa395a1d0-201507022240/task.json.
WARNING:root:Unable to load 'contrib' plugins, are there any?
2015-07-02 22:40:51,695 (PID 6435) felyx_work.processors (DEBUG): TaskID:14d4e9fc-e4bf-4c07-b7ac-fbbaa395a1d0
2015-07-02 22:40:51,695 (PID 6435) felyx_work.processors (DEBUG): JobID:0
2015-07-02 22:40:51,695 (PID 6435) felyx_work.processors (DEBUG): StepCount:1
2015-07-02 22:40:51,696 (PID 6435) felyx_work.processors (DEBUG): Input:None
2015-07-02 22:40:51,696 (PID 6435) felyx_work.processors (INFO): .0:True,0,Success
2015-07-02 22:40:51,696 (PID 6435) felyx_work.processors (INFO): >0s
2015-07-02 22:40:51,696 (PID 6435) felyx_work.processors (INFO): Output was: True

```