diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index cbc3650d5..c17169819 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -65,7 +65,7 @@ jobs: - name: Install and run OpenSearch 📦 uses: esmarkowski/opensearch-github-action@v1.0.0 with: - version: 2.12.0 + version: 2.18.0 security-disabled: true port: 9209 - name: Install and run MongoDB diff --git a/docs/source/cql.rst b/docs/source/cql.rst index 4ba1c6c14..66a6d6364 100644 --- a/docs/source/cql.rst +++ b/docs/source/cql.rst @@ -3,15 +3,18 @@ CQL support =========== +OGC Common Query Language (`CQL2`_) is a generic language designed to provide enhanced query and subset/filtering to (primarily) feature and record data. + Providers --------- -As of now the available providers supported for CQL filtering are limited to :ref:`Elasticsearch ` and :ref:`PostgreSQL `. - +CQL2 support is implemented in various pygeoapi feature and record providers. See the :ref:`feature ` and :ref:`metadata ` provider sections +for current provider support. + Limitations ----------- -Support of CQL is limited to `Simple CQL filter `_ and thus it allows to query with the +Support of CQL is limited to `Basic CQL2 `_ and thus it allows to query with the following predicates: * comparison predicates @@ -21,20 +24,20 @@ following predicates: Formats ------- -At the moment Elasticsearch supports only the CQL dialect with the JSON encoding `CQL-JSON `_. +Supported providers leverage the CQL2 dialect with the JSON encoding `CQL-JSON `_. -PostgreSQL supports both CQL-JSON and CQL-text dialects, `CQL-JSON `_ and `CQL-TEXT `_ +PostgreSQL supports both `CQL2 JSON `_ and `CQL text `_ dialects. Queries ^^^^^^^ The PostgreSQL provider uses `pygeofilter `_ allowing a range of filter expressions, see examples for: -* `Comparison predicates `_ -* `Spatial predicates `_ -* `Temporal predicates `_ +* `Comparison predicates (`Advanced `_, `Case-insensitive `_) +* `Spatial predicates `_ +* `Temporal predicates `_ -Using Elasticsearch the following type of queries are supported right now: +Using Elasticsearch the following type of queries are supported currently: * ``between`` predicate query * Logical ``and`` query with ``between`` and ``eq`` expression @@ -59,11 +62,11 @@ A ``BETWEEN`` example for a specific property through an HTTP POST request: curl --location --request POST 'http://localhost:5000/collections/nhsl_hazard_threat_all_indicators_s_bc/items?f=json&limit=50&filter-lang=cql-json' \ --header 'Content-Type: application/query-cql-json' \ --data-raw '{ - "between": { - "value": { "property": "properties.MHn_Intensity" }, - "lower": 0.59, - "upper": 0.60 - } + "op": "between", + "args": [ + {"property": "properties.MHn_Intensity"}, + [0.59, 0.60] + ] }' Or @@ -73,11 +76,11 @@ Or curl --location --request POST 'http://localhost:5000/collections/recentearthquakes/items?f=json&limit=10&filter-lang=cql-json' --header 'Content-Type: application/query-cql-json' --data-raw '{ - "between":{ - "value":{"property": "ml"}, - "lower":4, - "upper":4.5 - } + "op": "between", + "args": [ + {"property": "ml"}, + [4, 4.5] + ] }' The same ``BETWEEN`` query using HTTP GET request formatted as CQL text and URL encoded as below: @@ -93,7 +96,11 @@ An ``EQUALS`` example for a specific property: curl --location --request POST 'http://localhost:5000/collections/recentearthquakes/items?f=json&limit=10&filter-lang=cql-json' --header 'Content-Type: application/query-cql-json' --data-raw '{ - "eq":[{"property": "user_entered"},"APBE"] + "op": "=", + "args": [ + {"property": "user_entered"}, + "APBE" + ] }' A ``CROSSES`` example via an HTTP GET request. The CQL text is passed via the ``filter`` parameter. @@ -115,7 +122,6 @@ The same example, but this time providing a geometry in EWKT format: curl "http://localhost:5000/collections/beni/items?filter=DWITHIN(geometry,SRID=3857;POINT(1392921%205145517),100,meters)" - - - Note that the CQL text has been URL encoded. This is required in curl commands but when entering in a browser, plain text can be used e.g. ``CROSSES(foo_geom, LINESTRING(28 -2, 30 -4))``. + +.. _`CQL2`: https://docs.ogc.org/is/21-065r2/21-065r2.html diff --git a/docs/source/development.rst b/docs/source/development.rst index 59941d780..23bd8c6d5 100644 --- a/docs/source/development.rst +++ b/docs/source/development.rst @@ -22,44 +22,6 @@ Tests can be run locally as part of development workflow. They are also run on To run all tests, simply run ``pytest`` in the repository. To run a specific test file, run ``pytest tests/api/test_itemtypes.py``, for example. - -CQL extension lifecycle ------------------------ - -Limitations -^^^^^^^^^^^ - -This workflow is valid only for the `CQL-JSON` format. - -Schema -^^^^^^ - -The Common Query Language (CQL) is the part 3 of the standard OGC API - Features. This extension has its specification available at -`OGC API - Features - Part 3: Filtering and the Common Query Language (CQL) `_ and the schema exists in development at -`cql.json `_. - -Model generation -^^^^^^^^^^^^^^^^ - -pygeoapi uses a class-based Python model interface to translate the schema into Python objects defined by `pydantic `_ models. -The model is generated with the pre-processing of the schema through the utility ``datamodel-codegen``, which is part -of the `datamodel-code-generator `_ package: - - -.. code-block:: bash - - # Generate from local downloaded json schema file - datamodel-codegen --input ~/Download/cql-schema.json --input-file-type jsonschema --output ./pygeoapi/models/cql_update.py --class-name CQLModel - -Note that datamodel-code-generator must be explicitly installed, as it is not a pygeoapi runtime dependency - -How to merge -^^^^^^^^^^^^ - -Once the new pydantic models have been generated then the content of the Python file ``cql_update.py`` can be used to replace the old classes within the ``cql.py`` file. -Update everything above the function ``get_next_node`` and then verify if the tests for the CQL are still passing, for example ``test_post_cql_json_between_query`` -in ``tests/test_elasticsearch__provider.py``. - Working with Spatialite on OSX ------------------------------ diff --git a/pygeoapi/api/itemtypes.py b/pygeoapi/api/itemtypes.py index 0cd50be40..c001a051b 100644 --- a/pygeoapi/api/itemtypes.py +++ b/pygeoapi/api/itemtypes.py @@ -44,7 +44,7 @@ import urllib.parse from pygeofilter.parsers.ecql import parse as parse_ecql_text -from pygeofilter.parsers.cql_json import parse as parse_cql_json +from pygeofilter.parsers.cql2_json import parse as parse_cql2_json from pyproj.exceptions import CRSError from pygeoapi import l10n @@ -54,7 +54,6 @@ from pygeoapi.provider.base import ( ProviderGenericError, ProviderTypeError, SchemaType) -from pygeoapi.models.cql import CQLModel from pygeoapi.util import (CrsTransformSpec, filter_providers_by_type, filter_dict_by_key_value, get_crs_from_uri, get_provider_by_type, get_supported_crs_list, @@ -438,8 +437,10 @@ def get_collection_items( LOGGER.debug('Processing filter-crs parameter') filter_crs_uri = request.params.get('filter-crs', DEFAULT_CRS) + LOGGER.debug('processing filter parameter') cql_text = request.params.get('filter') + if cql_text is not None: try: filter_ = parse_ecql_text(cql_text) @@ -455,13 +456,29 @@ def get_collection_items( return api.get_exception( HTTPStatus.BAD_REQUEST, headers, request.format, 'InvalidParameterValue', msg) + elif request.data: + try: + request_data = request.data.decode() + filter_ = parse_cql2_json(request_data) + filter_ = modify_pygeofilter( + filter_, + filter_crs_uri=filter_crs_uri, + storage_crs_uri=provider_def.get('storage_crs'), + geometry_column_name=provider_def.get('geom_field'), + ) + except Exception: + msg = 'Bad CQL JSON' + LOGGER.error(f'{msg}: {request_data}') + return api.get_exception( + HTTPStatus.BAD_REQUEST, headers, request.format, + 'InvalidParameterValue', msg) else: filter_ = None LOGGER.debug('Processing filter-lang parameter') filter_lang = request.params.get('filter-lang') # Currently only cql-text is handled, but it is optional - if filter_lang not in [None, 'cql-text']: + if filter_lang not in [None, 'cql-json', 'cql-text']: msg = 'Invalid filter language' return api.get_exception( HTTPStatus.BAD_REQUEST, headers, request.format, @@ -637,291 +654,6 @@ def get_collection_items( return headers, HTTPStatus.OK, to_json(content, api.pretty_print) -def post_collection_items( - api: API, request: APIRequest, dataset) -> Tuple[dict, int, str]: - """ - Queries collection or filter an item - - :param request: A request object - :param dataset: dataset name - - :returns: tuple of headers, status code, content - """ - - request_headers = request.headers - - if not request.is_valid(PLUGINS['formatter'].keys()): - return api.get_format_exception(request) - - # Set Content-Language to system locale until provider locale - # has been determined - headers = request.get_response_headers(SYSTEM_LOCALE, **api.api_headers) - - properties = [] - reserved_fieldnames = ['bbox', 'f', 'limit', 'offset', - 'resulttype', 'datetime', 'sortby', - 'properties', 'skipGeometry', 'q', - 'filter-lang', 'filter-crs'] - - collections = filter_dict_by_key_value(api.config['resources'], - 'type', 'collection') - - if dataset not in collections.keys(): - msg = 'Invalid collection' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - - LOGGER.debug('Processing query parameters') - - LOGGER.debug('Processing offset parameter') - try: - offset = int(request.params.get('offset')) - if offset < 0: - msg = 'offset value should be positive or zero' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - except TypeError as err: - LOGGER.warning(err) - offset = 0 - except ValueError: - msg = 'offset value should be an integer' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - - LOGGER.debug('Processing limit parameter') - try: - limit = int(request.params.get('limit')) - # TODO: We should do more validation, against the min and max - # allowed by the server configuration - if limit <= 0: - msg = 'limit value should be strictly positive' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - except TypeError as err: - LOGGER.warning(err) - limit = int(api.config['server']['limit']) - except ValueError: - msg = 'limit value should be an integer' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - - resulttype = request.params.get('resulttype') or 'results' - - LOGGER.debug('Processing bbox parameter') - - bbox = request.params.get('bbox') - - if bbox is None: - bbox = [] - else: - try: - bbox = validate_bbox(bbox) - except ValueError as err: - msg = str(err) - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - - LOGGER.debug('Processing datetime parameter') - datetime_ = request.params.get('datetime') - try: - datetime_ = validate_datetime(collections[dataset]['extents'], - datetime_) - except ValueError as err: - msg = str(err) - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - - LOGGER.debug('processing q parameter') - val = request.params.get('q') - - q = None - if val is not None: - q = val - - LOGGER.debug('Loading provider') - - try: - provider_def = get_provider_by_type( - collections[dataset]['providers'], 'feature') - except ProviderTypeError: - try: - provider_def = get_provider_by_type( - collections[dataset]['providers'], 'record') - except ProviderTypeError: - msg = 'Invalid provider type' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'NoApplicableCode', msg) - - try: - p = load_plugin('provider', provider_def) - except ProviderGenericError as err: - return api.get_exception( - err.http_status_code, headers, request.format, - err.ogc_exception_code, err.message) - - LOGGER.debug('processing property parameters') - for k, v in request.params.items(): - if k not in reserved_fieldnames and k not in p.fields.keys(): - msg = f'unknown query parameter: {k}' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - elif k not in reserved_fieldnames and k in p.fields.keys(): - LOGGER.debug(f'Add property filter {k}={v}') - properties.append((k, v)) - - LOGGER.debug('processing sort parameter') - val = request.params.get('sortby') - - if val is not None: - sortby = [] - sorts = val.split(',') - for s in sorts: - prop = s - order = '+' - if s[0] in ['+', '-']: - order = s[0] - prop = s[1:] - - if prop not in p.fields.keys(): - msg = 'bad sort property' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - - sortby.append({'property': prop, 'order': order}) - else: - sortby = [] - - LOGGER.debug('processing properties parameter') - val = request.params.get('properties') - - if val is not None: - select_properties = val.split(',') - properties_to_check = set(p.properties) | set(p.fields.keys()) - - if (len(list(set(select_properties) - - set(properties_to_check))) > 0): - msg = 'unknown properties specified' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - else: - select_properties = [] - - LOGGER.debug('processing skipGeometry parameter') - val = request.params.get('skipGeometry') - if val is not None: - skip_geometry = str2bool(val) - else: - skip_geometry = False - - LOGGER.debug('Processing filter-crs parameter') - filter_crs = request.params.get('filter-crs', DEFAULT_CRS) - LOGGER.debug('Processing filter-lang parameter') - filter_lang = request.params.get('filter-lang') - if filter_lang != 'cql-json': # @TODO add check from the configuration - msg = 'Invalid filter language' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - - LOGGER.debug('Querying provider') - LOGGER.debug(f'offset: {offset}') - LOGGER.debug(f'limit: {limit}') - LOGGER.debug(f'resulttype: {resulttype}') - LOGGER.debug(f'sortby: {sortby}') - LOGGER.debug(f'bbox: {bbox}') - LOGGER.debug(f'datetime: {datetime_}') - LOGGER.debug(f'properties: {select_properties}') - LOGGER.debug(f'skipGeometry: {skip_geometry}') - LOGGER.debug(f'q: {q}') - LOGGER.debug(f'filter-lang: {filter_lang}') - LOGGER.debug(f'filter-crs: {filter_crs}') - - LOGGER.debug('Processing headers') - - LOGGER.debug('Processing request content-type header') - if (request_headers.get( - 'Content-Type') or request_headers.get( - 'content-type')) != 'application/query-cql-json': - msg = 'Invalid body content-type' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidHeaderValue', msg) - - LOGGER.debug('Processing body') - - if not request.data: - msg = 'missing request data' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'MissingParameterValue', msg) - - filter_ = None - try: - # Parse bytes data, if applicable - data = request.data.decode() - LOGGER.debug(data) - except UnicodeDecodeError: - msg = 'Unicode error in data' - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - - # FIXME: remove testing backend in use once CQL support is normalized - if p.name == 'PostgreSQL': - LOGGER.debug('processing PostgreSQL CQL_JSON data') - try: - filter_ = parse_cql_json(data) - filter_ = modify_pygeofilter( - filter_, - filter_crs_uri=filter_crs, - storage_crs_uri=provider_def.get('storage_crs'), - geometry_column_name=provider_def.get('geom_field') - ) - except Exception: - msg = 'Bad CQL text' - LOGGER.error(f'{msg}: {data}') - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - else: - LOGGER.debug('processing CQL_JSON data') - try: - filter_ = CQLModel.parse_raw(data) - except Exception: - msg = 'Bad CQL text' - LOGGER.error(f'{msg}: {data}') - return api.get_exception( - HTTPStatus.BAD_REQUEST, headers, request.format, - 'InvalidParameterValue', msg) - - try: - content = p.query(offset=offset, limit=limit, - resulttype=resulttype, bbox=bbox, - datetime_=datetime_, properties=properties, - sortby=sortby, - select_properties=select_properties, - skip_geometry=skip_geometry, - q=q, - filterq=filter_) - except ProviderGenericError as err: - return api.get_exception( - err.http_status_code, headers, request.format, - err.ogc_exception_code, err.message) - - return headers, HTTPStatus.OK, to_json(content, api.pretty_print) - - def manage_collection_item( api: API, request: APIRequest, action, dataset, identifier=None) -> Tuple[dict, int, str]: diff --git a/pygeoapi/django_/views.py b/pygeoapi/django_/views.py index fe0b7386b..682ef51ce 100644 --- a/pygeoapi/django_/views.py +++ b/pygeoapi/django_/views.py @@ -180,7 +180,7 @@ def collection_items(request: HttpRequest, collection_id: str) -> HttpResponse: 'create', collection_id, skip_valid_check=True) else: response_ = execute_from_django( - itemtypes_api.post_collection_items, + itemtypes_api.get_collection_items, request, collection_id, skip_valid_check=True,) elif request.method == 'OPTIONS': response_ = execute_from_django(itemtypes_api.manage_collection_item, diff --git a/pygeoapi/flask_app.py b/pygeoapi/flask_app.py index a5d0c2695..744862d4f 100644 --- a/pygeoapi/flask_app.py +++ b/pygeoapi/flask_app.py @@ -281,7 +281,7 @@ def collection_items(collection_id, item_id=None): skip_valid_check=True) else: return execute_from_flask( - itemtypes_api.post_collection_items, request, + itemtypes_api.get_collection_items, request, collection_id, skip_valid_check=True) elif request.method == 'OPTIONS': return execute_from_flask( diff --git a/pygeoapi/models/cql.py b/pygeoapi/models/cql.py deleted file mode 100644 index f803d9288..000000000 --- a/pygeoapi/models/cql.py +++ /dev/null @@ -1,506 +0,0 @@ -# ****************************** -*- -# flake8: noqa -# generated by datamodel-codegen: -# filename: cql-schema.json -# timestamp: 2021-03-13T21:05:20+00:00 -# ================================================================= -# -# Authors: Francesco Bartoli -# -# Copyright (c) 2024 Francesco Bartoli -# -# Permission is hereby granted, free of charge, to any person -# obtaining a copy of this software and associated documentation -# files (the "Software"), to deal in the Software without -# restriction, including without limitation the rights to use, -# copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following -# conditions: -# -# The above copyright notice and this permission notice shall be -# included in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. -# -# ================================================================= - -from datetime import date, datetime -from typing import Any, List, Literal, Optional, Union - -from pydantic import BaseModel, Field - - -class CQLModel(BaseModel): - __root__: 'Union[\n ComparisonPredicate,\n SpatialPredicate,\n TemporalPredicate,\n AndExpression\n ]' - - -class AndExpression(BaseModel): - and_: 'List[ComparisonPredicate]' = Field(..., alias='and') - - -class NotExpression(BaseModel): - not_: 'List[Any]' = Field(..., alias='not') - - -class OrExpression(BaseModel): - or_: 'List[Any]' = Field(..., alias='or') - - -class PropertyRef(BaseModel): - property: 'Optional[str]' = None - - -class ScalarLiteral(BaseModel): - __root__: 'Union[str, float, bool]' - - -class Bbox(BaseModel): - __root__: 'List[float]' - - -class LinestringCoordinate(BaseModel): - __root__: 'List[Any]' - - -class Linestring(BaseModel): - type: Literal['LineString'] - coordinates: 'List[LinestringCoordinate]' = Field(...) - bbox: 'Optional[List[float]]' = Field(None) - - -class MultilineStringCoordinate(BaseModel): - __root__: 'List[Any]' - - -class Multilinestring(BaseModel): - type: Literal['MultiLineString'] - coordinates: 'List[List[MultilineStringCoordinate]]' - bbox: 'Optional[List[float]]' = Field(None) - - -class Multipoint(BaseModel): - type: Literal['MultiPoint'] - coordinates: 'List[List[float]]' - bbox: 'Optional[List[float]]' = Field(None) - - -class MultipolygonCoordinateItem(BaseModel): - __root__: 'List[Any]' - - -class Multipolygon(BaseModel): - type: Literal['MultiPolygon'] - coordinates: 'List[List[List[MultipolygonCoordinateItem]]]' - bbox: 'Optional[List[float]]' = Field(None) - - -class Point(BaseModel): - type: Literal['Point'] - coordinates: 'List[float]' = Field(...) - bbox: 'Optional[List[float]]' = Field(None) - - -class PolygonCoordinatesItem(BaseModel): - __root__: 'List[Any]' - - -class Polygon(BaseModel): - type: Literal['Polygon'] - coordinates: 'List[List[PolygonCoordinatesItem]]' - bbox: 'Optional[List[float]]' = Field(None) - - -class TimeString(BaseModel): - __root__: 'Union[date, datetime]' - - -class EnvelopeLiteral(BaseModel): - bbox: 'Bbox' - - -class GeometryLiteral(BaseModel): - __root__: 'Union[\n Point,\n Linestring,\n Polygon,\n Multipoint,\n Multilinestring,\n Multipolygon\n ]' - - -class TypedTimeString(BaseModel): - datetime: 'TimeString' - - -class PeriodString(BaseModel): - __root__: 'List[Union[TimeString, str]]' = Field(...) - - -class SpatialLiteral(BaseModel): - __root__: 'Union[GeometryLiteral, EnvelopeLiteral]' - - -class TemporalLiteral(BaseModel): - __root__: 'Union[TimeString, PeriodString]' - - -class TypedPeriodString(BaseModel): - datetime: 'PeriodString' - - -class TypedTemporalLiteral(BaseModel): - __root__: 'Union[TypedTimeString, TypedPeriodString]' - - -class ArrayPredicate(BaseModel): - __root__: 'Union[\n AequalsExpression,\n AcontainsExpression,\n AcontainedByExpression,\n AoverlapsExpression,\n ]' - - -class ComparisonPredicate(BaseModel): - __root__: 'Union[\n BinaryComparisonPredicate,\n IsLikePredicate,\n IsBetweenPredicate,\n IsInListPredicate,\n IsNullPredicate,\n ]' - - -class SpatialPredicate(BaseModel): - __root__: 'Union[\n IntersectsExpression,\n EqualsExpression,\n DisjointExpression,\n TouchesExpression,\n WithinExpression,\n OverlapsExpression,\n CrossesExpression,\n ContainsExpression,\n ]' - - -class TemporalPredicate(BaseModel): - __root__: 'Union[\n BeforeExpression,\n AfterExpression,\n MeetsExpression,\n MetbyExpression,\n ToverlapsExpression,\n OverlappedbyExpression,\n BeginsExpression,\n BegunbyExpression,\n DuringExpression,\n TcontainsExpression,\n EndsExpression,\n EndedbyExpression,\n TequalsExpression,\n AnyinteractsExpression,\n ]' - - -class AcontainedByExpression(BaseModel): - acontainedBy: 'ArrayExpression' - - -class AcontainsExpression(BaseModel): - acontains: 'ArrayExpression' - - -class AequalsExpression(BaseModel): - aequals: 'ArrayExpression' - - -class AfterExpression(BaseModel): - after: 'TemporalOperands' - - -class AnyinteractsExpression(BaseModel): - anyinteracts: 'TemporalOperands' - - -class AoverlapsExpression(BaseModel): - aoverlaps: 'ArrayExpression' - - -class BeforeExpression(BaseModel): - before: 'TemporalOperands' - - -class BeginsExpression(BaseModel): - begins: 'TemporalOperands' - - -class BegunbyExpression(BaseModel): - begunby: 'TemporalOperands' - - -class BinaryComparisonPredicate(BaseModel): - __root__: 'Union[\n EqExpression, LtExpression, GtExpression, LteExpression, GteExpression\n ]' - - -class ContainsExpression(BaseModel): - contains: 'SpatialOperands' - - -class CrossesExpression(BaseModel): - crosses: 'SpatialOperands' - - -class DisjointExpression(BaseModel): - disjoint: 'SpatialOperands' - - -class DuringExpression(BaseModel): - during: 'TemporalOperands' - - -class EndedbyExpression(BaseModel): - endedby: 'TemporalOperands' - - -class EndsExpression(BaseModel): - ends: 'TemporalOperands' - - -class EqualsExpression(BaseModel): - equals: 'SpatialOperands' - - -class IntersectsExpression(BaseModel): - intersects: 'SpatialOperands' - - -class Between(BaseModel): - value: 'ValueExpression' - lower: 'ScalarExpression' = Field(None) - upper: 'ScalarExpression' = Field(None) - - -class IsBetweenPredicate(BaseModel): - between: 'Between' - - -class In(BaseModel): - value: 'ValueExpression' - list: 'List[ValueExpression]' - nocase: 'Optional[bool]' = True - - -class IsInListPredicate(BaseModel): - in_: 'In' = Field(..., alias='in') - - -class IsLikePredicate(BaseModel): - like: 'ScalarOperands' - wildcard: 'Optional[str]' = '%' - singleChar: 'Optional[str]' = '.' - escapeChar: 'Optional[str]' = '\\' - nocase: 'Optional[bool]' = True - - -class IsNullPredicate(BaseModel): - isNull: 'ScalarExpression' - - -class MeetsExpression(BaseModel): - meets: 'TemporalOperands' - - -class MetbyExpression(BaseModel): - metby: 'TemporalOperands' - - -class OverlappedbyExpression(BaseModel): - overlappedby: 'TemporalOperands' - - -class OverlapsExpression(BaseModel): - overlaps: 'SpatialOperands' - - -class TcontainsExpression(BaseModel): - tcontains: 'TemporalOperands' - - -class TequalsExpression(BaseModel): - tequals: 'TemporalOperands' - - -class TouchesExpression(BaseModel): - touches: 'SpatialOperands' - - -class ToverlapsExpression(BaseModel): - toverlaps: 'TemporalOperands' - - -class WithinExpression(BaseModel): - within: 'SpatialOperands' - - -class ArrayExpression(BaseModel): - __root__: 'List[Union[PropertyRef, FunctionRef, ArrayLiteral]]' = Field( - ... # , max_items=2, min_items=2 - ) - - -class EqExpression(BaseModel): - eq: 'ScalarOperands' - - -class GtExpression(BaseModel): - gt: 'ScalarOperands' - - -class GteExpression(BaseModel): - gte: 'ScalarOperands' - - -class LtExpression(BaseModel): - lt: 'ScalarOperands' - - -class LteExpression(BaseModel): - lte: 'ScalarOperands' - - -class ScalarExpression(BaseModel): - __root__: 'Union[ScalarLiteral,\n PropertyRef,\n FunctionRef,\n ArithmeticExpression]' - - -class ScalarOperands(BaseModel): - __root__: 'List[ScalarExpression]' = Field(...) - - -class SpatialOperands(BaseModel): - __root__: 'List[GeomExpression]' = Field(...) - - -class TemporalOperands(BaseModel): - __root__: 'List[TemporalExpression]' = Field(...) - # , max_items=2, min_items=2) - - -class ValueExpression(BaseModel): - __root__: 'Union[ScalarExpression, SpatialLiteral, TypedTemporalLiteral]' - - -class ArithmeticExpression(BaseModel): - __root__: 'Union[AddExpression, SubExpression, MulExpression, DivExpression]' - - -class ArrayLiteral(BaseModel): - __root__: 'List[\n Union[\n ScalarLiteral,\n SpatialLiteral,\n TypedTemporalLiteral,\n PropertyRef,\n FunctionRef,\n ArithmeticExpression,\n ArrayLiteral,\n ]\n ]' - - -class FunctionRef(BaseModel): - function: 'Function' - - -class GeomExpression(BaseModel): - __root__: 'Union[SpatialLiteral, PropertyRef, FunctionRef]' - - -class TemporalExpression(BaseModel): - __root__: 'Union[TemporalLiteral, PropertyRef, FunctionRef]' - - -# here -class AddExpression(BaseModel): - add_: 'ArithmeticOperands' = Field(..., alias='+') - - -# here -class DivExpression(BaseModel): - div_: 'ArithmeticOperands' = Field(None, alias='/') - - -class Function(BaseModel): - name: 'str' - arguments: 'Optional[\n List[\n Union[\n ScalarLiteral,\n SpatialLiteral,\n TypedTemporalLiteral,\n PropertyRef,\n FunctionRef,\n ArithmeticExpression,\n ArrayLiteral,\n ]\n ]\n ]' = None - - -# here -class MulExpression(BaseModel): - mul_: 'ArithmeticOperands' = Field(..., alias='*') - - -# here -class SubExpression(BaseModel): - sub_: 'ArithmeticOperands' = Field(..., alias='-') - - -class ArithmeticOperands(BaseModel): - __root__: 'List[\n Union[ArithmeticExpression, PropertyRef, FunctionRef, float]\n ]' = Field(...) - - -CQLModel.update_forward_refs() -AndExpression.update_forward_refs() -ArrayPredicate.update_forward_refs() -ComparisonPredicate.update_forward_refs() -SpatialPredicate.update_forward_refs() -TemporalPredicate.update_forward_refs() -AcontainedByExpression.update_forward_refs() -AcontainsExpression.update_forward_refs() -AequalsExpression.update_forward_refs() -AfterExpression.update_forward_refs() -AnyinteractsExpression.update_forward_refs() -AoverlapsExpression.update_forward_refs() -BeforeExpression.update_forward_refs() -BeginsExpression.update_forward_refs() -BegunbyExpression.update_forward_refs() -BinaryComparisonPredicate.update_forward_refs() -ContainsExpression.update_forward_refs() -CrossesExpression.update_forward_refs() -DisjointExpression.update_forward_refs() -DuringExpression.update_forward_refs() -EndedbyExpression.update_forward_refs() -EndsExpression.update_forward_refs() -EqualsExpression.update_forward_refs() -IntersectsExpression.update_forward_refs() -Between.update_forward_refs() -In.update_forward_refs() -IsBetweenPredicate.update_forward_refs() -IsLikePredicate.update_forward_refs() -IsNullPredicate.update_forward_refs() -ValueExpression.update_forward_refs() -MeetsExpression.update_forward_refs() -MetbyExpression.update_forward_refs() -OverlappedbyExpression.update_forward_refs() -OverlapsExpression.update_forward_refs() -TcontainsExpression.update_forward_refs() -TequalsExpression.update_forward_refs() -TouchesExpression.update_forward_refs() -ToverlapsExpression.update_forward_refs() -WithinExpression.update_forward_refs() -ArrayExpression.update_forward_refs() -EqExpression.update_forward_refs() -GtExpression.update_forward_refs() -GteExpression.update_forward_refs() -LtExpression.update_forward_refs() -LteExpression.update_forward_refs() -ScalarExpression.update_forward_refs() -ScalarOperands.update_forward_refs() -SpatialOperands.update_forward_refs() -TemporalOperands.update_forward_refs() -ArithmeticExpression.update_forward_refs() -ArrayLiteral.update_forward_refs() -ScalarLiteral.update_forward_refs() -PropertyRef.update_forward_refs() -FunctionRef.update_forward_refs() -AddExpression.update_forward_refs() -DivExpression.update_forward_refs() -MulExpression.update_forward_refs() -SubExpression.update_forward_refs() - - -def get_next_node(obj): - logical_op = None - if obj.__repr_name__() == 'AndExpression': - next_node = obj.and_ - logical_op = 'and' - elif obj.__repr_name__() == 'OrExpression': - next_node = obj.or_ - logical_op = 'or' - elif obj.__repr_name__() == 'NotExpression': - next_node = obj.not_ - logical_op = 'not' - elif obj.__repr_name__() == 'ComparisonPredicate': - next_node = obj.__root__ - elif obj.__repr_name__() == 'SpatialPredicate': - next_node = obj.__root__ - elif obj.__repr_name__() == 'TemporalPredicate': - next_node = obj.__root__ - elif obj.__repr_name__() == 'IsBetweenPredicate': - next_node = obj.between - elif obj.__repr_name__() == 'Between': - next_node = obj.value - elif obj.__repr_name__() == 'ValueExpression': - next_node = obj.__root__ or obj.lower or obj.upper - elif obj.__repr_name__() == 'ScalarExpression': - next_node = obj.__root__ - elif obj.__repr_name__() == 'ScalarLiteral': - next_node = obj.__root__ - elif obj.__repr_name__() == 'PropertyRef': - next_node = obj.property - elif obj.__repr_name__() == 'BinaryComparisonPredicate': - next_node = obj.__root__ - elif obj.__repr_name__() == 'EqExpression': - next_node = obj.eq - logical_op = 'eq' - else: - raise ValueError("Object not valid") - - return (logical_op, next_node) diff --git a/pygeoapi/provider/elasticsearch_.py b/pygeoapi/provider/elasticsearch_.py index ef603ce8d..4c0ae20c1 100644 --- a/pygeoapi/provider/elasticsearch_.py +++ b/pygeoapi/provider/elasticsearch_.py @@ -3,7 +3,7 @@ # Authors: Tom Kralidis # Francesco Bartoli # -# Copyright (c) 2023 Tom Kralidis +# Copyright (c) 2024 Tom Kralidis # Copyright (c) 2024 Francesco Bartoli # # Permission is hereby granted, free of charge, to any person @@ -36,13 +36,14 @@ import uuid from elasticsearch import Elasticsearch, exceptions, helpers -from elasticsearch_dsl import Search, Q +from pygeofilter.backends.elasticsearch import to_filter + +from elasticsearch_dsl import Search from pygeoapi.provider.base import (BaseProvider, ProviderConnectionError, ProviderQueryError, ProviderItemNotFoundError) -from pygeoapi.models.cql import CQLModel, get_next_node -from pygeoapi.util import get_envelope, crs_transform +from pygeoapi.util import crs_transform LOGGER = logging.getLogger(__name__) @@ -313,7 +314,7 @@ def query(self, offset=0, limit=10, resulttype='results', try: LOGGER.debug('querying Elasticsearch') if filterq: - LOGGER.debug(f'adding cql object: {filterq.json()}') + LOGGER.debug(f'adding cql object: {filterq}') query = update_query(input_query=query, cql=filterq) LOGGER.debug(json.dumps(query, indent=4)) @@ -585,183 +586,8 @@ def __repr__(self): return f' {self.data}' -class ESQueryBuilder: - def __init__(self): - self._operation = None - self.must_value = {} - self.should_value = {} - self.mustnot_value = {} - self.filter_value = {} - - def must(self, must_value): - self.must_value = must_value - return self - - def should(self, should_value): - self.should_value = should_value - return self - - def must_not(self, mustnot_value): - self.mustnot_value = mustnot_value - return self - - def filter(self, filter_value): - self.filter_value = filter_value - return self - - @property - def operation(self): - return self._operation - - @operation.setter - def operation(self, value): - self._operation = value - - def build(self): - if self.must_value: - must_clause = self.must_value or {} - if self.should_value: - should_clause = self.should_value or {} - if self.mustnot_value: - mustnot_clause = self.mustnot_value or {} - if self.filter_value: - filter_clause = self.filter_value or {} - else: - filter_clause = {} - - # to figure out how to deal with logical operations - # return match_clause & range_clause - clauses = must_clause or should_clause or mustnot_clause - filters = filter_clause - if self.operation == 'and': - res = Q( - 'bool', - must=[clause for clause in clauses], - filter=[filter for filter in filters]) - elif self.operation == 'or': - res = Q( - 'bool', - should=[clause for clause in clauses], - filter=[filter for filter in filters]) - elif self.operation == 'not': - res = Q( - 'bool', - must_not=[clause for clause in clauses], - filter=[filter for filter in filters]) - else: - if filters: - res = Q( - 'bool', - must=[clauses], - filter=[filters]) - else: - res = Q( - 'bool', - must=[clauses]) - - return res - - -def _build_query(q, cql): - - # this would be handled by the AST with the traverse of CQL model - op, node = get_next_node(cql.__root__) - q.operation = op - if isinstance(node, list): - query_list = [] - for elem in node: - op, next_node = get_next_node(elem) - if not getattr(next_node, 'between', 0) == 0: - property = next_node.between.value.__root__.__root__.property - lower = next_node.between.lower.__root__.__root__ - upper = next_node.between.upper.__root__.__root__ - query_list.append(Q( - { - 'range': - { - f'{property}': { - 'gte': lower, 'lte': upper - } - } - } - )) - if not getattr(next_node, '__root__', 0) == 0: - scalars = tuple(next_node.__root__.eq.__root__) - property = scalars[0].__root__.property - value = scalars[1].__root__.__root__ - query_list.append(Q( - {'match': {f'{property}': f'{value}'}} - )) - q.must(query_list) - elif not getattr(node, 'between', 0) == 0: - property = node.between.value.__root__.__root__.property - lower = None - if not getattr(node.between.lower, - '__root__', 0) == 0: - lower = node.between.lower.__root__.__root__ - upper = None - if not getattr(node.between.upper, - '__root__', 0) == 0: - upper = node.between.upper.__root__.__root__ - query = Q( - { - 'range': - { - f'{property}': { - 'gte': lower, 'lte': upper - } - } - } - ) - q.must(query) - elif not getattr(node, '__root__', 0) == 0: - next_op, next_node = get_next_node(node) - if not getattr(next_node, 'eq', 0) == 0: - scalars = tuple(next_node.eq.__root__) - property = scalars[0].__root__.property - value = scalars[1].__root__.__root__ - query = Q( - {'match': {f'{property}': f'{value}'}} - ) - q.must(query) - elif not getattr(node, 'intersects', 0) == 0: - property = node.intersects.__root__[0].__root__.property - if property == 'geometry': - geom_type = node.intersects.__root__[ - 1].__root__.__root__.__root__.type - if geom_type == 'Polygon': - coordinates = node.intersects.__root__[ - 1].__root__.__root__.__root__.coordinates - coords_list = [ - poly_coords.__root__ for poly_coords in coordinates[0] - ] - filter_ = Q( - { - 'geo_shape': { - 'geometry': { - 'shape': { - 'type': 'envelope', - 'coordinates': get_envelope( - coords_list) - }, - 'relation': 'intersects' - } - } - } - ) - query_all = Q( - {'match_all': {}} - ) - q.must(query_all) - q.filter(filter_) - return q.build() - - -def update_query(input_query: Dict, cql: CQLModel): +def update_query(input_query: Dict, cql): s = Search.from_dict(input_query) - query = ESQueryBuilder() - output_query = _build_query(query, cql) - s = s.query(output_query) - + s = s.query(to_filter(cql)) LOGGER.debug(f'Enhanced query: {json.dumps(s.to_dict())}') return s.to_dict() diff --git a/pygeoapi/provider/opensearch_.py b/pygeoapi/provider/opensearch_.py index b6b308e5d..3e84de0e8 100644 --- a/pygeoapi/provider/opensearch_.py +++ b/pygeoapi/provider/opensearch_.py @@ -36,13 +36,14 @@ import uuid from opensearchpy import OpenSearch, helpers -from opensearch_dsl import Search, Q +from opensearch_dsl import Search + +from pygeofilter.backends.opensearch import to_filter from pygeoapi.provider.base import (BaseProvider, ProviderConnectionError, ProviderQueryError, ProviderItemNotFoundError) -from pygeoapi.models.cql import CQLModel, get_next_node -from pygeoapi.util import get_envelope, crs_transform +from pygeoapi.util import crs_transform LOGGER = logging.getLogger(__name__) @@ -294,7 +295,7 @@ def query(self, offset=0, limit=10, resulttype='results', try: LOGGER.debug('querying OpenSearch') if filterq: - LOGGER.debug(f'adding cql object: {filterq.json()}') + LOGGER.debug(f'adding cql object: {filterq}') query = update_query(input_query=query, cql=filterq) LOGGER.debug(json.dumps(query, indent=4)) @@ -560,183 +561,9 @@ def __repr__(self): return f' {self.data}' -class OpenSearchQueryBuilder: - def __init__(self): - self._operation = None - self.must_value = {} - self.should_value = {} - self.mustnot_value = {} - self.filter_value = {} - - def must(self, must_value): - self.must_value = must_value - return self - - def should(self, should_value): - self.should_value = should_value - return self - - def must_not(self, mustnot_value): - self.mustnot_value = mustnot_value - return self - - def filter(self, filter_value): - self.filter_value = filter_value - return self - - @property - def operation(self): - return self._operation - - @operation.setter - def operation(self, value): - self._operation = value - - def build(self): - if self.must_value: - must_clause = self.must_value or {} - if self.should_value: - should_clause = self.should_value or {} - if self.mustnot_value: - mustnot_clause = self.mustnot_value or {} - if self.filter_value: - filter_clause = self.filter_value or {} - else: - filter_clause = {} - - # to figure out how to deal with logical operations - # return match_clause & range_clause - clauses = must_clause or should_clause or mustnot_clause - filters = filter_clause - if self.operation == 'and': - res = Q( - 'bool', - must=[clause for clause in clauses], - filter=[filter for filter in filters]) - elif self.operation == 'or': - res = Q( - 'bool', - should=[clause for clause in clauses], - filter=[filter for filter in filters]) - elif self.operation == 'not': - res = Q( - 'bool', - must_not=[clause for clause in clauses], - filter=[filter for filter in filters]) - else: - if filters: - res = Q( - 'bool', - must=[clauses], - filter=[filters]) - else: - res = Q( - 'bool', - must=[clauses]) - - return res - - -def _build_query(q, cql): - - # this would be handled by the AST with the traverse of CQL model - op, node = get_next_node(cql.__root__) - q.operation = op - if isinstance(node, list): - query_list = [] - for elem in node: - op, next_node = get_next_node(elem) - if not getattr(next_node, 'between', 0) == 0: - property = next_node.between.value.__root__.__root__.property - lower = next_node.between.lower.__root__.__root__ - upper = next_node.between.upper.__root__.__root__ - query_list.append(Q( - { - 'range': - { - f'{property}': { - 'gte': lower, 'lte': upper - } - } - } - )) - if not getattr(next_node, '__root__', 0) == 0: - scalars = tuple(next_node.__root__.eq.__root__) - property = scalars[0].__root__.property - value = scalars[1].__root__.__root__ - query_list.append(Q( - {'match': {f'{property}': f'{value}'}} - )) - q.must(query_list) - elif not getattr(node, 'between', 0) == 0: - property = node.between.value.__root__.__root__.property - lower = None - if not getattr(node.between.lower, - '__root__', 0) == 0: - lower = node.between.lower.__root__.__root__ - upper = None - if not getattr(node.between.upper, - '__root__', 0) == 0: - upper = node.between.upper.__root__.__root__ - query = Q( - { - 'range': - { - f'{property}': { - 'gte': lower, 'lte': upper - } - } - } - ) - q.must(query) - elif not getattr(node, '__root__', 0) == 0: - next_op, next_node = get_next_node(node) - if not getattr(next_node, 'eq', 0) == 0: - scalars = tuple(next_node.eq.__root__) - property = scalars[0].__root__.property - value = scalars[1].__root__.__root__ - query = Q( - {'match': {f'{property}': f'{value}'}} - ) - q.must(query) - elif not getattr(node, 'intersects', 0) == 0: - property = node.intersects.__root__[0].__root__.property - if property == 'geometry': - geom_type = node.intersects.__root__[ - 1].__root__.__root__.__root__.type - if geom_type == 'Polygon': - coordinates = node.intersects.__root__[ - 1].__root__.__root__.__root__.coordinates - coords_list = [ - poly_coords.__root__ for poly_coords in coordinates[0] - ] - filter_ = Q( - { - 'geo_shape': { - 'geometry': { - 'shape': { - 'type': 'envelope', - 'coordinates': get_envelope( - coords_list) - }, - 'relation': 'intersects' - } - } - } - ) - query_all = Q( - {'match_all': {}} - ) - q.must(query_all) - q.filter(filter_) - return q.build() - - -def update_query(input_query: Dict, cql: CQLModel): +def update_query(input_query: Dict, cql): s = Search.from_dict(input_query) - query = OpenSearchQueryBuilder() - output_query = _build_query(query, cql) - s = s.query(output_query) + s = s.query(to_filter(cql)) LOGGER.debug(f'Enhanced query: {json.dumps(s.to_dict())}') return s.to_dict() diff --git a/pygeoapi/starlette_app.py b/pygeoapi/starlette_app.py index 9c44a9b12..984e62302 100644 --- a/pygeoapi/starlette_app.py +++ b/pygeoapi/starlette_app.py @@ -326,7 +326,7 @@ async def collection_items(request: Request, collection_id=None, item_id=None): 'create', collection_id, skip_valid_check=True) else: return await execute_from_starlette( - itemtypes_api.post_collection_items, + itemtypes_api.get_collection_items, request, collection_id, skip_valid_check=True, diff --git a/tests/test_elasticsearch__provider.py b/tests/test_elasticsearch__provider.py index 6fc89bee1..ba1cbb4aa 100644 --- a/tests/test_elasticsearch__provider.py +++ b/tests/test_elasticsearch__provider.py @@ -2,7 +2,7 @@ # # Authors: Tom Kralidis # -# Copyright (c) 2020 Tom Kralidis +# Copyright (c) 2024 Tom Kralidis # Copyright (c) 2024 Francesco Bartoli # # Permission is hereby granted, free of charge, to any person @@ -28,11 +28,13 @@ # # ================================================================= +import json + +from pygeofilter.parsers.cql2_json import parse as parse_cql2_json import pytest from pygeoapi.provider.base import ProviderItemNotFoundError from pygeoapi.provider.elasticsearch_ import ElasticsearchProvider -from pygeoapi.models.cql import CQLModel @pytest.fixture() @@ -72,90 +74,89 @@ def config_cql(): @pytest.fixture() def between(): between_ = { - "between": { - "value": {"property": "properties.pop_max"}, - "lower": 10000, - "upper": 100000 - } + 'op': 'between', + 'args': [ + {'property': 'properties.pop_max'}, + [10000, 100000] + ] } - return CQLModel.parse_obj(between_) + return parse_cql2_json(json.dumps(between_)) @pytest.fixture() def between_upper(): between_ = { - "between": { - "value": {"property": "properties.pop_max"}, - "upper": 100000 - } + 'op': '<', + 'args': [ + {'property': 'properties.pop_max'}, + 100000 + ] } - return CQLModel.parse_obj(between_) + return parse_cql2_json(json.dumps(between_)) @pytest.fixture() def between_lower(): between_ = { - "between": { - "value": {"property": "properties.pop_max"}, - "lower": 10000 - } + 'op': '>', + 'args': [ + {'property': 'properties.pop_max'}, + 10000 + ] } - return CQLModel.parse_obj(between_) + return parse_cql2_json(json.dumps(between_)) @pytest.fixture() def eq(): eq_ = { - "eq": [ - {"property": "properties.featurecla"}, - "Admin-0 capital" + 'op': '=', + 'args': [ + {'property': 'properties.featurecla'}, + 'Admin-0 capital' ] } - return CQLModel.parse_obj(eq_) + return parse_cql2_json(json.dumps(eq_)) @pytest.fixture() def _and(eq, between): and_ = { - "and": [ - { - "between": { - "value": { - "property": "properties.pop_max" - }, - "lower": 100000, - "upper": 1000000 - } - }, - { - "eq": [ - {"property": "properties.featurecla"}, - "Admin-0 capital" + 'op': 'and', + 'args': [{ + 'op': 'between', + 'args': [ + {'property': 'properties.pop_max'}, + [100000, 1000000] + ] + }, { + 'op': '=', + 'args': [ + {'property': 'properties.featurecla'}, + 'Admin-0 capital' ] - } - ] + }] } - return CQLModel.parse_obj(and_) + return parse_cql2_json(json.dumps(and_)) @pytest.fixture() def intersects(): - intersects = {"intersects": [ - {"property": "geometry"}, - { - "type": "Polygon", - "coordinates": [ - [ - [10.497565, 41.520355], - [10.497565, 43.308645], - [15.111823, 43.308645], - [15.111823, 41.520355], - [10.497565, 41.520355] - ] - ] - } - ]} - return CQLModel.parse_obj(intersects) + intersects = { + 'op': 's_intersects', + 'args': [ + {'property': 'geometry'}, { + 'type': 'Polygon', + 'coordinates': [[ + [10.497565, 41.520355], + [10.497565, 43.308645], + [15.111823, 43.308645], + [15.111823, 41.520355], + [10.497565, 41.520355] + ]] + } + ]} + return parse_cql2_json(json.dumps(intersects)) def test_query(config): diff --git a/tests/test_models.py b/tests/test_models.py index 8cbc189d7..7e0070258 100644 --- a/tests/test_models.py +++ b/tests/test_models.py @@ -1,8 +1,10 @@ # ================================================================= # # Authors: Francesco Bartoli +# Tom Kralidis: # # Copyright (c) 2024 Francesco Bartoli +# Copyright (c) 2024 Tom Kralidis # # Permission is hereby granted, free of charge, to any person # obtaining a copy of this software and associated documentation @@ -31,16 +33,6 @@ from polyfactory.pytest_plugin import register_fixture from pygeoapi.models.provider.base import GeospatialDataType -from pygeoapi.models.cql import (CQLModel, ComparisonPredicate, - ScalarExpression, SpatialPredicate, - TemporalPredicate, AndExpression, - Between, EqExpression, ScalarOperands, - IntersectsExpression, SpatialOperands) - - -@register_fixture -class CQLModelFactory(ModelFactory[CQLModel]): - ... @register_fixture @@ -48,56 +40,8 @@ class GeospatialDataTypeFactory(ModelFactory[GeospatialDataType]): ... -@register_fixture -class BetweenModelFactory(ModelFactory[Between]): - ... - - -@register_fixture -class EqExpressionModelFactory(ModelFactory[EqExpression]): - ... - - -@register_fixture -class IntersectsExpressionModelFactory(ModelFactory[IntersectsExpression]): - ... - - -def test_cql_model(cql_model_factory: CQLModelFactory) -> None: - cql_model_instance = cql_model_factory.build() - assert isinstance(cql_model_instance, CQLModel) - assert cql_model_instance.dict() - assert type(cql_model_instance.__root__) in [ - ComparisonPredicate, SpatialPredicate, TemporalPredicate, AndExpression - ] - - def test_provider_base_geospatial_data_type( geospatial_data_type_factory: GeospatialDataTypeFactory) -> None: gdt_instance = geospatial_data_type_factory.build() assert gdt_instance.dict() assert isinstance(gdt_instance, GeospatialDataType) - - -def test_between_model(between_model_factory: BetweenModelFactory) -> None: - between_model_instance = between_model_factory.build() - assert isinstance(between_model_instance, Between) - assert between_model_instance.dict() - assert type(between_model_instance.lower) is ScalarExpression - assert type(between_model_instance.upper) is ScalarExpression - - -def test_eq_expression_model( - eq_expression_model_factory: EqExpressionModelFactory) -> None: - eqexpr_model_instance = eq_expression_model_factory.build() - assert isinstance(eqexpr_model_instance, EqExpression) - assert eqexpr_model_instance.dict() - assert type(eqexpr_model_instance.eq) is ScalarOperands - - -def test_intersects_expression_model( - intersects_expression_model_factory: IntersectsExpressionModelFactory) -> None: # noqa - intersectsexpr_model_instance = intersects_expression_model_factory.build() - assert isinstance(intersectsexpr_model_instance, IntersectsExpression) - assert intersectsexpr_model_instance.dict() - assert type(intersectsexpr_model_instance.intersects) is SpatialOperands diff --git a/tests/test_opensearch__provider.py b/tests/test_opensearch__provider.py index 3ba180169..40117e74b 100644 --- a/tests/test_opensearch__provider.py +++ b/tests/test_opensearch__provider.py @@ -28,11 +28,13 @@ # # ================================================================= +import json + +from pygeofilter.parsers.cql2_json import parse as parse_cql2_json import pytest from pygeoapi.provider.base import ProviderItemNotFoundError from pygeoapi.provider.opensearch_ import OpenSearchProvider -from pygeoapi.models.cql import CQLModel @pytest.fixture() @@ -72,90 +74,89 @@ def config_cql(): @pytest.fixture() def between(): between_ = { - "between": { - "value": {"property": "properties.pop_max"}, - "lower": 10000, - "upper": 100000 - } + 'op': 'between', + 'args': [ + {'property': 'properties.pop_max'}, + [10000, 100000] + ] } - return CQLModel.parse_obj(between_) + return parse_cql2_json(json.dumps(between_)) @pytest.fixture() def between_upper(): between_ = { - "between": { - "value": {"property": "properties.pop_max"}, - "upper": 100000 - } + 'op': '<', + 'args': [ + {'property': 'properties.pop_max'}, + 100000 + ] } - return CQLModel.parse_obj(between_) + return parse_cql2_json(json.dumps(between_)) @pytest.fixture() def between_lower(): between_ = { - "between": { - "value": {"property": "properties.pop_max"}, - "lower": 10000 - } + 'op': '>', + 'args': [ + {'property': 'properties.pop_max'}, + 10000 + ] } - return CQLModel.parse_obj(between_) + return parse_cql2_json(json.dumps(between_)) @pytest.fixture() def eq(): eq_ = { - "eq": [ - {"property": "properties.featurecla"}, - "Admin-0 capital" + 'op': '=', + 'args': [ + {'property': 'properties.featurecla'}, + 'Admin-0 capital' ] } - return CQLModel.parse_obj(eq_) + return parse_cql2_json(json.dumps(eq_)) @pytest.fixture() def _and(eq, between): and_ = { - "and": [ - { - "between": { - "value": { - "property": "properties.pop_max" - }, - "lower": 100000, - "upper": 1000000 - } - }, - { - "eq": [ - {"property": "properties.featurecla"}, - "Admin-0 capital" + 'op': 'and', + 'args': [{ + 'op': 'between', + 'args': [ + {'property': 'properties.pop_max'}, + [100000, 1000000] + ] + }, { + 'op': '=', + 'args': [ + {'property': 'properties.featurecla'}, + 'Admin-0 capital' ] - } - ] + }] } - return CQLModel.parse_obj(and_) + return parse_cql2_json(json.dumps(and_)) @pytest.fixture() def intersects(): - intersects = {"intersects": [ - {"property": "geometry"}, - { - "type": "Polygon", - "coordinates": [ - [ - [10.497565, 41.520355], - [10.497565, 43.308645], - [15.111823, 43.308645], - [15.111823, 41.520355], - [10.497565, 41.520355] - ] - ] - } - ]} - return CQLModel.parse_obj(intersects) + intersects = { + 'op': 's_intersects', + 'args': [ + {'property': 'geometry'}, { + 'type': 'Polygon', + 'coordinates': [[ + [10.497565, 41.520355], + [10.497565, 43.308645], + [15.111823, 43.308645], + [15.111823, 41.520355], + [10.497565, 41.520355] + ]] + } + ]} + return parse_cql2_json(json.dumps(intersects)) def test_query(config): diff --git a/tests/test_postgresql_provider.py b/tests/test_postgresql_provider.py index 3c48263ea..a011f3e64 100644 --- a/tests/test_postgresql_provider.py +++ b/tests/test_postgresql_provider.py @@ -8,7 +8,7 @@ # Bernhard Mallinger # # Copyright (c) 2019 Just van den Broecke -# Copyright (c) 2024 Tom Kralidis +# Copyright (c) 2025 Tom Kralidis # Copyright (c) 2022 John A Stevenson and Colin Blackburn # Copyright (c) 2023 Francesco Bartoli # Copyright (c) 2024 Bernhard Mallinger @@ -50,8 +50,7 @@ from pygeoapi.api import API from pygeoapi.api.itemtypes import ( - get_collection_items, get_collection_item, manage_collection_item, - post_collection_items + get_collection_items, get_collection_item, manage_collection_item ) from pygeoapi.provider.base import ( ProviderConnectionError, @@ -542,7 +541,7 @@ def test_get_collection_items_postgresql_cql_invalid_filter_language(pg_api_): # Act req = mock_api_request({ - 'filter-lang': 'cql-json', # Only cql-text is valid for GET + 'filter-lang': 'cql-jsonfoo', 'filter': cql_query }) rsp_headers, code, response = get_collection_items( @@ -581,16 +580,31 @@ def test_get_collection_items_postgresql_cql_bad_cql(pg_api_, bad_cql): assert error_response['description'] == 'Bad CQL text' -def test_post_collection_items_postgresql_cql(pg_api_): +def test_get_collection_items_postgresql_cql_json(pg_api_): """ Test for PostgreSQL CQL - requires local PostgreSQL with appropriate data. See pygeoapi/provider/postgresql.py for details. """ # Arrange - cql = {"and": [{"between": {"value": {"property": "osm_id"}, - "lower": 80800000, - "upper": 80900000}}, - {"isNull": {"property": "name"}}]} + cql = { + 'op': 'and', + 'args': [{ + 'op': 'between', + 'args': [ + {'property': 'osm_id'}, + [80800000, 80900000] + ] + }, { + # FIXME: the below query is in CQL style, not CQL2 + # needs a fix in pygeofilter + # 'op': 'isNull', + # 'args': [ + # {'property': 'name'} + # ] + 'op': 'isNull', + 'args': {'property': 'name'} + }] + } # werkzeug requests use a value of CONTENT_TYPE 'application/json' # to create Content-Type in the Request object. So here we need to # overwrite the default CONTENT_TYPE with the required one. @@ -601,7 +615,7 @@ def test_post_collection_items_postgresql_cql(pg_api_): req = mock_api_request({ 'filter-lang': 'cql-json' }, data=cql, **headers) - rsp_headers, code, response = post_collection_items( + rsp_headers, code, response = get_collection_items( pg_api_, req, 'hot_osm_waterways') # Assert @@ -611,7 +625,7 @@ def test_post_collection_items_postgresql_cql(pg_api_): assert ids == expected_ids -def test_post_collection_items_postgresql_cql_invalid_filter_language(pg_api_): +def test_get_collection_items_postgresql_cql_json_invalid_filter_language(pg_api_): # noqa """ Test for PostgreSQL CQL - requires local PostgreSQL with appropriate data. See pygeoapi/provider/postgresql.py for details. @@ -627,14 +641,14 @@ def test_post_collection_items_postgresql_cql_invalid_filter_language(pg_api_): req = mock_api_request({ 'filter-lang': 'cql-text' # Only cql-json is valid for POST }, data=cql, **headers) - rsp_headers, code, response = post_collection_items( + rsp_headers, code, response = get_collection_items( pg_api_, req, 'hot_osm_waterways') # Assert assert code == HTTPStatus.BAD_REQUEST error_response = json.loads(response) assert error_response['code'] == 'InvalidParameterValue' - assert error_response['description'] == 'Invalid filter language' + assert error_response['description'] == 'Bad CQL JSON' @pytest.mark.parametrize("bad_cql", [ @@ -643,7 +657,7 @@ def test_post_collection_items_postgresql_cql_invalid_filter_language(pg_api_): # At some point this may return UnexpectedEOF '{"in": {"value": {"property": "id"}, "list": [1, 2}}' ]) -def test_post_collection_items_postgresql_cql_bad_cql(pg_api_, bad_cql): +def test_get_collection_items_postgresql_cql_json_bad_cql(pg_api_, bad_cql): """ Test for PostgreSQL CQL - requires local PostgreSQL with appropriate data. See pygeoapi/provider/postgresql.py for details. @@ -657,14 +671,14 @@ def test_post_collection_items_postgresql_cql_bad_cql(pg_api_, bad_cql): req = mock_api_request({ 'filter-lang': 'cql-json' }, data=bad_cql, **headers) - rsp_headers, code, response = post_collection_items( + rsp_headers, code, response = get_collection_items( pg_api_, req, 'hot_osm_waterways') # Assert assert code == HTTPStatus.BAD_REQUEST error_response = json.loads(response) assert error_response['code'] == 'InvalidParameterValue' - assert error_response['description'] == 'Bad CQL text' + assert error_response['description'] == 'Bad CQL JSON' def test_get_collection_items_postgresql_crs(pg_api_):