Merge pull request #713 from crim-ca/job-prefer

CHANGES.rst

@@ -12,14 +12,50 @@ Changes
 
 Changes:
 --------
-- No change.
-
-Fixes:
-------
+- Add support of ``response: raw`` execution request body parameter as alternative to ``response: document``,
+  which allows directly returning the result contents or ``Link`` headers rather then embedding them in a `JSON`
+  response (fixes `#376 <https://github.com/crim-ca/weaver/issues/376>`_).
+- Add support of ``Prefer: return=minimal`` and ``Prefer: return=representation`` header as alternative method
+  to request the ``response: document`` and ``response: raw`` parameters
+  (fixes `#414 <https://github.com/crim-ca/weaver/issues/414>`_).
+  Minor differences exist according to supplied ``transmissionMode`` and the original data/link results.
+  See `Process Execution <file:///home/francis/dev/weaver/docs/build/html/processes.html#proc-op-execute>`_
+  documentation for details.
+- Add support of ``outputs`` execution request body parameter to filter returned outputs from
+  the ``GET /jobs/{jobId}/results`` (async) or returned directly (sync) from ``POST /processes/{processId}/execution``
+  (fixes `#380 <https://github.com/crim-ca/weaver/issues/380>_`).
+- Add support of ``Accept: multipart/*`` and ``Accept: multipart/mixed`` when submitting an execution to obtain
+  the results as multiple parts embedded within the response contents. Parts are represented with their default
+  data/link representation, unless overridden by corresponding ``transmissionMode`` per output ID.
+- Add ``output_links``/``-oL``/``--output-link`` parameter to Python client and CLI to retrieve ``Link`` headers
+  as `Job` results. Due to the multiple ``Link`` headers returned by `Job` results, this cannot be performed
+  automatically without the assumption of which ``rel`` links correspond to actual output IDs to extract.
+- Add ``output_filter``/``--oF``/``--output-filter`` parameter to Python client and CLI to indicate
+  any ``outputs`` to be filtered when submitting the `Process` execution.
+- Update ``Preference-Applied`` header reported by execution responses to
+  include ``return=minimal`` or ``return=representation`` as applicable by the requested ``Prefer`` header.
+- Update documentation with a mapping of *Process Execution Results* according to
+  submitted ``response`` body parameter (*OGC API - Processes v1.0*),
+  the ``Prefer: return`` header (*OGC API - Processes v2.0*), the requested ``Accept`` header,
+  and any relevant ``transmissionMode`` request body overrides per filtered ``outputs``.
+
+Fixes:
+------
+- Fix ``GET /jobs/{jobId}/inputs`` contents to correctly return the submitted ``outputs`` definition
+  for `Process` execution (fixes `#715 <https://github.com/crim-ca/weaver/issues/715>`_).
+- Fix missing ``Link`` header with ``rel: monitor`` relationship in the created `Job` responses
+  (fixes `#596 <https://github.com/crim-ca/weaver/issues/596>`_).
+- Fix missing ``/rec/core/link-header`` definition in ``GET /conformance`` response reporting
+  that ``Link`` headers are returned for corresponding references of a given request
+  (fixes `#378 <https://github.com/crim-ca/weaver/issues/378>`_).
+- Fix ``transmissionMode: value`` that was ignored for ``response: document`` if the output was represented by default
+  as a *complex*  file URL, and ``transmissionMode: reference`` that was ignored if the output was *literal*  data.
+  The ``transmissionMode`` will now return the appropriate inline data or URL as requested.
 - Add missing conformance and requirement references for *OGC API - Processes - Part 2: DRU*
   (fixes `##620 <https://github.com/crim-ca/weaver/issues/620>`_).
 - Add the appropriate HTTP error type to respect ``/conf/dru/deploy/unsupported-content-type``
   (fixes `#624 <https://github.com/crim-ca/weaver/issues/624>`_).
+- Fix S3 bucket storage for result file missing the output ID in the path to match local WPS output storage structure.
 
 .. _changes_5.9.0:
 

docs/_static/custom.css

@@ -3,10 +3,10 @@
     max-width: none;
 }
 
-/* force code-table to align their cells to top (align code line numbers between columns)
+/* force table-code to align their cells to top (align code line numbers between columns)
     note: class attribute must be applied to match this specific type of table
 */
-.code-table tbody tr td {
+.table-code tbody tr td {
     vertical-align: top !important;
 
     /* max-width: min-content; */
@@ -21,11 +21,39 @@
 div[class^="highlight"] {
     max-width: min-content;
 }
-.code-table tbody tr td div {
+.table-code tbody tr td div {
     max-width: none !important;
 }
 */
 
+.table-exec-results thead,
+.table-exec-results tbody {
+    vertical-align: top !important;
+}
+
+.table-exec-results thead {
+    background-color: #CCCCCC;
+}
+
+.table-exec-results thead tr:nth-child(1) > th:nth-child(1),
+.table-exec-results thead tr:nth-child(1) > th:nth-child(2) {
+    border-bottom-color: #777777 !important;
+}
+
+.table-exec-results tr:nth-child(1) > th:nth-child(2),
+.table-exec-results tr:nth-child(1) > th:nth-child(3),
+.table-exec-results tr:nth-child(2) > th:nth-child(2),
+.table-exec-results td:nth-child(2),
+.table-exec-results td:nth-child(4) {
+    border-left-color: #777777 !important;
+    border-left-width: medium !important;
+}
+
+/* avoid unnecessary spacing causing table to be massively longer than needed */
+.table-exec-results ul {
+    margin-bottom: 0 !important;
+}
+
 /* override table width restrictions
    avoids cells trying to fit all their text single line with a slider
    instead, text will wrap according to specified :widths: specifications

docs/_templates/autoapi/index.rst

@@ -8,8 +8,6 @@ This page contains reference documentation of the source code.
 .. toctree::
    :titlesonly:
 
-   {% for page in pages %}
-   {% if page.top_level_object and page.display %}
+   {% for page in pages|selectattr("is_top_level_object") %}
    {{ page.include_path }}
-   {% endif %}
    {% endfor %}

docs/examples/job_execute_outputs_transmission.json

@@ -0,0 +1,8 @@
+{
+  "inputs": {"<...>": "<...>"},
+  "outputs": {
+    "output-default": {},
+    "output-by-value": {"transmissionMode": "value"},
+    "output-by-ref": {"transmissionMode": "reference"}
+  }
+}

docs/examples/job_inputs.json

@@ -0,0 +1,77 @@
+{
+  "inputs": {
+    "calc": "4.26 * ((C / A) ** 3.94)",
+    "band_a": {
+      "href": "https://example.com/wpsoutputs/weaver/users/23/6f197568-38f5-42f4-851c-0c56d446094c/product/T29SPC_20190601T110621_B02_10m.jp2",
+      "type": "image/jp2"
+    },
+    "band_c": {
+      "href": "https://example.com/wpsoutputs/weaver/users/23/977799a0-bf63-4406-a419-6d686c9a8fc9/product/T29SPC_20190601T110621_B04_10m.jp2",
+      "type": "image/jp2"
+    },
+    "name": "output"
+  },
+  "outputs": {
+    "result": {
+      "transmissionMode": "reference"
+    }
+  },
+  "links": [
+    {
+      "title": "Job status.",
+      "hreflang": "en-CA",
+      "href": "https://example.com/weaver/processes/calculate-band/jobs/034151ec-a87e-41ed-8ab4-8afb22b48e96",
+      "type": "application/json",
+      "rel": "status"
+    },
+    {
+      "title": "Job status generic endpoint.",
+      "hreflang": "en-CA",
+      "href": "https://example.com/weaver/jobs/034151ec-a87e-41ed-8ab4-8afb22b48e96",
+      "type": "application/json",
+      "rel": "alternate"
+    },
+    {
+      "title": "New job submission endpoint for the corresponding process.",
+      "hreflang": "en-CA",
+      "href": "https://example.com/weaver/processes/calculate-band/jobs/execution",
+      "type": "application/json",
+      "rel": "http://www.opengis.net/def/rel/ogc/1.0/execute"
+    },
+    {
+      "title": "Submitted job inputs for process execution.",
+      "hreflang": "en-CA",
+      "href": "https://example.com/weaver/processes/calculate-band/jobs/034151ec-a87e-41ed-8ab4-8afb22b48e96/inputs",
+      "type": "application/json",
+      "rel": "inputs"
+    },
+    {
+      "title": "Job outputs of successful process execution (extended outputs with metadata).",
+      "hreflang": "en-CA",
+      "href": "https://example.com/weaver/processes/calculate-band/jobs/034151ec-a87e-41ed-8ab4-8afb22b48e96/outputs",
+      "type": "application/json",
+      "rel": "outputs"
+    },
+    {
+      "title": "Job results of successful process execution (direct output values mapping).",
+      "hreflang": "en-CA",
+      "href": "https://example.com/weaver/processes/calculate-band/jobs/034151ec-a87e-41ed-8ab4-8afb22b48e96/results",
+      "type": "application/json",
+      "rel": "http://www.opengis.net/def/rel/ogc/1.0/results"
+    },
+    {
+      "title": "Job statistics collected following process execution.",
+      "hreflang": "en-CA",
+      "href": "https://example.com/weaver/processes/calculate-band/jobs/034151ec-a87e-41ed-8ab4-8afb22b48e96/statistics",
+      "type": "application/json",
+      "rel": "statistics"
+    },
+    {
+      "title": "List of collected job logs during process execution.",
+      "hreflang": "en-CA",
+      "href": "https://example.com/weaver/processes/calculate-band/jobs/034151ec-a87e-41ed-8ab4-8afb22b48e96/logs",
+      "type": "application/json",
+      "rel": "logs"
+    }
+  ]
+}

docs/examples/job_outputs_listing.json

@@ -0,0 +1,13 @@
+{
+  "outputs": [
+    {
+      "id": "output-file",
+      "href": "https://example.com/wpsoutputs/f93a15be-6e16-11ea-b667-08002752172a/output_netcdf.nc",
+      "type": "application/x-netcdf"
+    },
+    {
+      "id": "output-data",
+      "value": 3.1416
+    }
+  ]
+}

docs/examples/job_outputs_mapping.json

@@ -0,0 +1,9 @@
+{
+  "outputs": {
+    "output-file": {
+      "href": "https://example.com/wpsoutputs/f93a15be-6e16-11ea-b667-08002752172a/output_netcdf.nc",
+      "type": "application/x-netcdf"
+    },
+    "output-data": 3.1416
+  }
+}

docs/examples/job_results_document_minimal.json

@@ -0,0 +1,7 @@
+{
+  "output-file": {
+    "href": "https://example.com/wpsoutputs/f93a15be-6e16-11ea-b667-08002752172a/output_netcdf.nc",
+    "type": "application/x-netcdf"
+  },
+  "output-data": 3.1416
+}

docs/examples/job_results_raw_multi.http

@@ -0,0 +1,15 @@
+HTTP/1.1 200 OK
+Host: weaver.example.com
+Content-Type: multipart/mixed; boundary=43003e2f205a180ace9cd34d98f911ff
+
+--43003e2f205a180ace9cd34d98f911ff
+Content-Type: application/x-netcdf
+Content-ID: <output-file@f93a15be-6e16-11ea-b667-08002752172a>
+Content-Location: https://example.com/wpsoutputs/f93a15be-6e16-11ea-b667-08002752172a/output-file/output_netcdf.nc
+
+--43003e2f205a180ace9cd34d98f911ff
+Content-Type: text/plain
+Content-ID: <output-data@f93a15be-6e16-11ea-b667-08002752172a>
+
+3.1416
+--43003e2f205a180ace9cd34d98f911ff--

docs/examples/job_results_raw_single_data.http

@@ -0,0 +1,6 @@
+HTTP/1.1 200 OK
+Host: weaver.example.com
+Content-ID: <output@f93a15be-6e16-11ea-b667-08002752172a>
+Content-Type: application/x-netcdf
+
+<netcdf data>

docs/examples/job_results_raw_single_ref.http

@@ -0,0 +1,7 @@
+HTTP/1.1 204 No Content
+Host: weaver.example.com
+Content-Length: 0
+Content-Type: application/x-netcdf
+Content-ID: <output@f93a15be-6e16-11ea-b667-08002752172a>
+Content-Location: https://example.com/wpsoutputs/f93a15be-6e16-11ea-b667-08002752172a/output/output_netcdf.nc
+Link: <https://example.com/wpsoutputs/f93a15be-6e16-11ea-b667-08002752172a/output/output_netcdf.nc>; rel="output"; type="application/x-netcdf"

docs/source/appendix.rst

@@ -205,6 +205,10 @@ Glossary
           More recent `Media-Type` naming is employed for the general use of ``Content-Type`` data representation in
           multiple situations and contexts.
 
+        .. seealso::
+            - |iana-link|_
+            - |edam-link|_
+
     OAS
     OpenAPI
         OpenAPI Specification (`OAS`) defines a standard, programming language-agnostic interface description for

docs/source/conf.py

@@ -245,7 +245,7 @@ def doc_redirect_include(file_path):
 
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
-# html_logo = None
+html_logo = "../_static/crim.png"
 
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
@@ -313,7 +313,7 @@ def doc_redirect_include(file_path):
 # Sphinx supports the following languages:
 #   'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
 #   'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
-# html_search_language = 'en'
+html_search_language = "en"
 
 # A dictionary with options for the search language support, empty by default.
 # Now only 'ja' uses this config value

docs/source/configuration.rst

@@ -325,7 +325,7 @@ they are optional and which default value or operation is applied in each situat
     default. Explicit response media-types can be requested in both cases using either an explicit ``Accept`` header
     of the desired media-type, or their corresponding ``f`` query format.
   |
-  | This option is Only applicable when |weaver-wps-restapi-html|_ is enabled. Otherwise, :term:`JSON` responses are
+  | This option is only applicable when |weaver-wps-restapi-html|_ is enabled. Otherwise, :term:`JSON` responses are
     always employed by default.
 
   .. versionadded:: 5.7.0
@@ -695,7 +695,7 @@ and all corresponding functionalities, including `API` endpoints, will be disabl
 
 - | ``weaver.quotation_docker_image = <image-reference>`` [:class:`str`]
   |
-  | Specifies the :term:`Docker` image used as |quote-estimator|_ to evaluate a :term:`Quote`
+  | Specifies the :term:`Docker` image used for :ref:`quote_estimation` to evaluate a :term:`Quote`
     for the eventual :term:`Process` execution.
   |
   | Required if ``weaver.quotation`` is enabled.
@@ -708,7 +708,7 @@ and all corresponding functionalities, including `API` endpoints, will be disabl
 
 - | ``weaver.quotation_docker_username = <username>`` [:class:`str`]
   |
-  | Username to employ for authentication when retrieving the :term:`Docker` image used as |quote-estimator|_.
+  | Username to employ for authentication when retrieving the :term:`Docker` image used as :ref:`quote_estimation`.
   |
   | Only required if the :term:`Docker` image is not accessible publicly or already
     provided through some other means when requested by the :term:`Docker` daemon.
@@ -722,7 +722,7 @@ and all corresponding functionalities, including `API` endpoints, will be disabl
 
 - | ``weaver.quotation_docker_password = <username>`` [:class:`str`]
   |
-  | Password to employ for authentication when retrieving the :term:`Docker` image used as |quote-estimator|_.
+  | Password to employ for authentication when retrieving the :term:`Docker` image used as :ref:`quote_estimation`.
   |
   | Only required if the :term:`Docker` image is not accessible publicly or already
     provided through some other means when requested by the :term:`Docker` daemon.
@@ -739,7 +739,7 @@ and all corresponding functionalities, including `API` endpoints, will be disabl
   |
   | Currency code in `ISO-4217 <https://www.iso.org/iso-4217-currency-codes.html>`_ format used by default.
   |
-  | It is up to the specified |quote-estimator|_ algorithm defined by ``weaver.quotation_docker_image`` and
+  | It is up to the specified :ref:`quote_estimation` algorithm defined by ``weaver.quotation_docker_image`` and
     employed by the various :term:`Process` to ensure that the returned :ref:`quote_estimation` cost makes
     sense according to the specified default currency.
   |
@@ -803,7 +803,7 @@ and all corresponding functionalities, including `API` endpoints, will be disabl
 
 - | ``weaver.quotation_currency_token = <API access token>`` [:class:`str`]
   |
-  | Password to employ for authentication when retrieving the :term:`Docker` image used as |quote-estimator|_.
+  | Password to employ for authentication when retrieving the :term:`Docker` image used as :ref:`quote_estimation`.
   |
   | Only required if the :term:`Docker` image is not accessible publicly or already
     provided through some other means when requested by the :term:`Docker` daemon.