Merge branch 'main' into 2779-failwarn-when-there-are-no-selected-str…

…eams
meltano · Jan 15, 2025 · 9101008 · 9101008
2 parents d1d4287 + b4c6ecc
commit 9101008
Show file tree

Hide file tree

Showing 49 changed files with 1,317 additions and 834 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml
@@ -15,7 +15,7 @@ body:
     attributes:
       label: Singer SDK Version
       description: Version of the library you are using
-      placeholder: "0.42.1"
+      placeholder: "0.43.1"
     validations:
       required: true
   - type: checkboxes

diff --git a/.github/workflows/constraints.txt b/.github/workflows/constraints.txt
@@ -1,5 +1,5 @@
 griffe~=1.5
 pip==24.3.1
-poetry==1.8.5
+poetry==2.0.1
 pre-commit==4.0.1
 nox==2024.10.9
diff --git a/.github/workflows/cookiecutter-e2e.yml b/.github/workflows/cookiecutter-e2e.yml
@@ -48,7 +48,7 @@ jobs:
       with:
         python-version: 3.x
 
-    - uses: astral-sh/setup-uv@v4
+    - uses: astral-sh/setup-uv@v5
       with:
         version: ">=0.4.30"
 

diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -65,7 +65,7 @@ jobs:
         name: Packages
         path: dist
     - name: Publish
-      uses: pypa/[email protected].2
+      uses: pypa/[email protected].3
 
   upload-to-release:
     name: Upload files to release

diff --git a/.github/workflows/version_bump.yml b/.github/workflows/version_bump.yml
@@ -51,7 +51,7 @@ jobs:
 
     - name: Bump version
       id: cz-bump
-      uses: commitizen-tools/[email protected].0
+      uses: commitizen-tools/[email protected].1
       with:
         increment: ${{ github.event.inputs.bump != 'auto' && github.event.inputs.bump || '' }}
         prerelease: ${{ github.event.inputs.prerelease != 'none' && github.event.inputs.prerelease || '' }}

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -37,14 +37,14 @@ repos:
   - id: trailing-whitespace
 
 - repo: https://github.com/python-jsonschema/check-jsonschema
-  rev: 0.30.0
+  rev: 0.31.0
   hooks:
   - id: check-dependabot
   - id: check-github-workflows
   - id: check-readthedocs
 
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.8.1
+  rev: v0.9.1
   hooks:
   - id: ruff
     args: [--fix, --exit-non-zero-on-fix, --show-fixes]
@@ -59,6 +59,6 @@ repos:
       )$
 
 - repo: https://github.com/python-poetry/poetry
-  rev: 1.8.0
+  rev: 2.0.1
   hooks:
   - id: poetry-check
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,54 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## v0.43.1 (2024-12-10)
+
+### 🐛 Fixes
+
+- [#2807](https://github.com/meltano/sdk/issues/2807) Allow developers to set `RESTStream.http_method`
+
+## v0.43.0 (2024-12-10)
+
+### ✨ New
+
+- [#2482](https://github.com/meltano/sdk/issues/2482) Allow SQL tap developers to auto-skip certain schemas from discovery
+- [#2784](https://github.com/meltano/sdk/issues/2784) Added a new built-in setting `activate_version` for targets to optionally disable processing of `ACTIVATE_VERSION` messages
+- [#2780](https://github.com/meltano/sdk/issues/2780) Numeric values are now parsed as `decimal.Decimal` in REST and GraphQL stream responses
+- [#2775](https://github.com/meltano/sdk/issues/2775) Log a stream's bookmark (if it's avaiable) when its sync starts
+- [#2703](https://github.com/meltano/sdk/issues/2703) Targets now emit record count from the built-in batch file processor
+- [#2774](https://github.com/meltano/sdk/issues/2774) Accept a `maxLength` limit for VARCHARs
+- [#2769](https://github.com/meltano/sdk/issues/2769) Add `versioning-strategy` to dependabot config of Cookiecutter templates
+- [#2765](https://github.com/meltano/sdk/issues/2765) The last received Singer message is now logged when the target fails
+- [#2762](https://github.com/meltano/sdk/issues/2762) Support other content-types in REST streams
+
+### 🐛 Fixes
+
+- [#2790](https://github.com/meltano/sdk/issues/2790) Ensure the required global folder tap settings are merged into the concrete implementation settings
+- [#2785](https://github.com/meltano/sdk/issues/2785) Use FS-specific `listdir` in folder tap
+- [#2778](https://github.com/meltano/sdk/issues/2778) The path of the offending field is now printed for config validation errors
+- [#2770](https://github.com/meltano/sdk/issues/2770) Respect standard Singer stream metadata `table-key-properties`, `replication-key` and `forced-replication-method`
+- [#2755](https://github.com/meltano/sdk/issues/2755) Safely compare UUID replication keys with state bookmarks -- _**Thanks @nikzavada!**_
+
+### ⚙️ Under the Hood
+
+- [#2805](https://github.com/meltano/sdk/issues/2805) Rename setting `activate_version` to `process_activate_version_messages`
+- [#2788](https://github.com/meltano/sdk/issues/2788) Fail early if input files to `--catalog` or `--state` do not exist
+- [#2781](https://github.com/meltano/sdk/issues/2781) Added a class method to instantiate `SQLToJSONSchema` from the tap configuration
+- [#2566](https://github.com/meltano/sdk/issues/2566) Standardize on JSON Schema Draft 2020-12 to validate stream schemas
+- [#2751](https://github.com/meltano/sdk/issues/2751) Dropped support for Python 3.8
+
+### ⚡ Performance Improvements
+
+- [#2793](https://github.com/meltano/sdk/issues/2793) Improved discovery performance for SQL taps
+
+### 📚 Documentation Improvements
+
+- [#2796](https://github.com/meltano/sdk/issues/2796) Document how to configure nested stream maps values with environment variables in Meltano
+
+### 📦 Packaging changes
+
+- [#2797](https://github.com/meltano/sdk/issues/2797) SQL taps now require SQLAlchemy 2.0+
+
 ## v0.42.1 (2024-11-11)
 
 ### 🐛 Fixes

diff --git a/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/.github/workflows/build.yml b/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/.github/workflows/build.yml
@@ -42,4 +42,4 @@ jobs:
     - name: Publish
       ## TODO: create a trusted publisher on PyPI
       ## https://docs.pypi.org/trusted-publishers/
-      uses: pypa/[email protected].2
+      uses: pypa/[email protected].3
diff --git a/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/.pre-commit-config.yaml b/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/.pre-commit-config.yaml
@@ -24,7 +24,7 @@ repos:
   - id: check-github-workflows
 
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.8.1
+  rev: v0.8.3
   hooks:
   - id: ruff
     args: [--fix, --exit-non-zero-on-fix, --show-fixes]

diff --git a/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/README.md b/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/README.md
@@ -6,11 +6,11 @@ Built with the [Meltano Mapper SDK](https://sdk.meltano.com) for Singer Mappers.
 
 <!--
 
-Developer TODO: Update the below as needed to correctly describe the install procedure. For instance, if you do not have a PyPi repo, or if you want users to directly install from your git repo, you can modify this step as appropriate.
+Developer TODO: Update the below as needed to correctly describe the install procedure. For instance, if you do not have a PyPI repo, or if you want users to directly install from your git repo, you can modify this step as appropriate.
 
 ## Installation
 
-Install from PyPi:
+Install from PyPI:
 
 ```bash
 pipx install {{ cookiecutter.mapper_id }}

diff --git a/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/pyproject.toml b/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/pyproject.toml
@@ -31,12 +31,12 @@ packages = [
 
 [tool.poetry.dependencies]
 python = ">=3.9"
-singer-sdk = { version="~=0.42.1"{{ ', extras = ["faker"]' if cookiecutter.faker_extra }} }
+singer-sdk = { version="~=0.43.1"{{ ', extras = ["faker"]' if cookiecutter.faker_extra }} }
 fs-s3fs = { version = "~=1.1.1", optional = true }
 
 [tool.poetry.group.dev.dependencies]
 pytest = ">=8"
-singer-sdk = { version="~=0.42.1", extras = ["testing"] }
+singer-sdk = { version="~=0.43.1", extras = ["testing"] }
 
 [tool.poetry.extras]
 s3 = ["fs-s3fs"]

diff --git a/cookiecutter/tap-template/{{cookiecutter.tap_id}}/.github/workflows/build.yml b/cookiecutter/tap-template/{{cookiecutter.tap_id}}/.github/workflows/build.yml
@@ -42,4 +42,4 @@ jobs:
     - name: Publish
       ## TODO: create a trusted publisher on PyPI
       ## https://docs.pypi.org/trusted-publishers/
-      uses: pypa/[email protected].2
+      uses: pypa/[email protected].3
diff --git a/cookiecutter/tap-template/{{cookiecutter.tap_id}}/.pre-commit-config.yaml b/cookiecutter/tap-template/{{cookiecutter.tap_id}}/.pre-commit-config.yaml
@@ -24,7 +24,7 @@ repos:
   - id: check-github-workflows
 
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.8.1
+  rev: v0.8.3
   hooks:
   - id: ruff
     args: [--fix, --exit-non-zero-on-fix, --show-fixes]

diff --git a/cookiecutter/tap-template/{{cookiecutter.tap_id}}/README.md b/cookiecutter/tap-template/{{cookiecutter.tap_id}}/README.md
@@ -6,11 +6,11 @@ Built with the [Meltano Tap SDK](https://sdk.meltano.com) for Singer Taps.
 
 <!--
 
-Developer TODO: Update the below as needed to correctly describe the install procedure. For instance, if you do not have a PyPi repo, or if you want users to directly install from your git repo, you can modify this step as appropriate.
+Developer TODO: Update the below as needed to correctly describe the install procedure. For instance, if you do not have a PyPI repo, or if you want users to directly install from your git repo, you can modify this step as appropriate.
 
 ## Installation
 
-Install from PyPi:
+Install from PyPI:
 
 ```bash
 pipx install {{ cookiecutter.tap_id }}

diff --git a/cookiecutter/tap-template/{{cookiecutter.tap_id}}/pyproject.toml b/cookiecutter/tap-template/{{cookiecutter.tap_id}}/pyproject.toml
@@ -30,7 +30,7 @@ packages = [
 
 [tool.poetry.dependencies]
 python = ">=3.9"
-singer-sdk = { version="~=0.42.1", extras = [
+singer-sdk = { version="~=0.43.1", extras = [
     {%- if cookiecutter.auth_method == "JWT" -%}"jwt", {% endif -%}
     {%- if cookiecutter.faker_extra -%}"faker",{%- endif -%}
 ] }
@@ -45,9 +45,9 @@ sqlalchemy = "~=2.0.36"
 [tool.poetry.group.dev.dependencies]
 pytest = ">=8"
 {%- if cookiecutter.auth_method == "JWT" %}
-singer-sdk = { version="~=0.42.1", extras = ["jwt", "testing"] }
+singer-sdk = { version="~=0.43.1", extras = ["jwt", "testing"] }
 {%- else %}
-singer-sdk = { version="~=0.42.1", extras = ["testing"] }
+singer-sdk = { version="~=0.43.1", extras = ["testing"] }
 {%- endif %}
 
 [tool.poetry.extras]

diff --git a/cookiecutter/target-template/{{cookiecutter.target_id}}/.github/workflows/build.yml b/cookiecutter/target-template/{{cookiecutter.target_id}}/.github/workflows/build.yml
@@ -42,4 +42,4 @@ jobs:
     - name: Publish
       ## TODO: create a trusted publisher on PyPI
       ## https://docs.pypi.org/trusted-publishers/
-      uses: pypa/[email protected].2
+      uses: pypa/[email protected].3
diff --git a/cookiecutter/target-template/{{cookiecutter.target_id}}/.pre-commit-config.yaml b/cookiecutter/target-template/{{cookiecutter.target_id}}/.pre-commit-config.yaml
@@ -24,7 +24,7 @@ repos:
   - id: check-github-workflows
 
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.8.1
+  rev: v0.8.3
   hooks:
   - id: ruff
     args: [--fix, --exit-non-zero-on-fix, --show-fixes]

diff --git a/cookiecutter/target-template/{{cookiecutter.target_id}}/README.md b/cookiecutter/target-template/{{cookiecutter.target_id}}/README.md
@@ -6,11 +6,11 @@ Build with the [Meltano Target SDK](https://sdk.meltano.com).
 
 <!--
 
-Developer TODO: Update the below as needed to correctly describe the install procedure. For instance, if you do not have a PyPi repo, or if you want users to directly install from your git repo, you can modify this step as appropriate.
+Developer TODO: Update the below as needed to correctly describe the install procedure. For instance, if you do not have a PyPI repo, or if you want users to directly install from your git repo, you can modify this step as appropriate.
 
 ## Installation
 
-Install from PyPi:
+Install from PyPI:
 
 ```bash
 pipx install {{ cookiecutter.target_id }}

diff --git a/cookiecutter/target-template/{{cookiecutter.target_id}}/pyproject.toml b/cookiecutter/target-template/{{cookiecutter.target_id}}/pyproject.toml
@@ -30,15 +30,15 @@ packages = [
 
 [tool.poetry.dependencies]
 python = ">=3.9"
-singer-sdk = { version="~=0.42.1"{{ ', extras = ["faker"]' if cookiecutter.faker_extra }} }
+singer-sdk = { version="~=0.43.1"{{ ', extras = ["faker"]' if cookiecutter.faker_extra }} }
 fs-s3fs = { version = "~=1.1.1", optional = true }
 {%- if cookiecutter.serialization_method != "SQL" %}
 requests = "~=2.32.3"
 {%- endif %}
 
 [tool.poetry.dev-dependencies]
 pytest = ">=8"
-singer-sdk = { version="~=0.42.1", extras = ["testing"] }
+singer-sdk = { version="~=0.43.1", extras = ["testing"] }
 
 [tool.poetry.extras]
 s3 = ["fs-s3fs"]

diff --git a/docs/conf.py b/docs/conf.py
@@ -29,7 +29,7 @@
 author = "Meltano Core Team and Contributors"
 
 # The full version, including alpha/beta/rc tags
-release = "0.42.1"
+release = "0.43.1"
 
 
 # -- General configuration -------------------------------------------------------------

diff --git a/docs/guides/sql-target.md b/docs/guides/sql-target.md
@@ -50,3 +50,56 @@ class MyConnector(SQLConnector):
         to_sql.register_format_handler("uri", URI)
         return to_sql
 ```
+
+### Use the `x-sql-datatype` JSON Schema extension
+
+You can register new type handlers for the `x-sql-datatype` extension:
+
+```python
+from my_sqlalchemy_dialect import URI
+
+
+class MyConnector(SQLConnector):
+    @functools.cached_property
+    def jsonschema_to_sql(self):
+        to_sql = JSONSchemaToSQL()
+        to_sql.register_sql_datatype_handler("smallint", sa.types.SMALLINT)
+        return to_sql
+```
+
+Then you can annotate the tap' catalog to specify the SQL type:
+
+````{tab} meltano.yml
+```yaml
+# https://docs.meltano.com/concepts/plugins/#schema-extra
+plugins:
+  extractors:
+  - name: tap-example
+    schema:
+      addresses:
+        number:
+          x-sql-datatype: smallint
+```
+````
+
+````{tab} JSON catalog
+```json
+{
+  "streams": [
+    {
+      "stream": "addresses",
+      "tap_stream_id": "addresses",
+      "schema": {
+        "type": "object",
+        "properties": {
+          "number": {
+            "type": "integer",
+            "x-sql-datatype": "smallint"
+          }
+        }
+      }
+    }
+  ]
+}
+```
+````
diff --git a/docs/python_tips.md b/docs/python_tips.md
@@ -39,7 +39,7 @@ creating and managing virtual environments, these two tools automate the process
     `poetry add -D <pippable-dev-only-ref>`.
   - If version conflicts occur, relax the version constraints in `pyproject.toml` for the
     libraries where the conflict is reported, then try again.
-  - Poetry can also publish your libraries to PyPi.
+  - Poetry can also publish your libraries to PyPI.
 - **pipx**: Install pipx once, and then use pipx _instead of_ pip.
   - If you are using poetry for development, then all other pip-installables should be
     executable tools and programs, which is what pipx is designed for.

diff --git a/docs/stream_maps.md b/docs/stream_maps.md
@@ -230,11 +230,12 @@ can be referenced directly by mapping expressions.
 
 The following functions and namespaces are available for use in mapping expressions:
 
-| Function                                     | Description                                                                                                                                                                                                                                                            |
-| :------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [`md5()`](inv:python:py:module:#hashlib)     | Returns an inline MD5 hash of any string, outputting the string representation of the hash's hex digest. This is defined by the SDK internally with native python: [`hashlib.md5(<input>.encode("utf-8")).hexdigest()`](inv:python:py:method:#hashlib.hash.hexdigest). |
-| [`datetime`](inv:python:py:module:#datetime) | This is the datetime module object from the Python standard library. You can access [`datetime.datetime`](inv:python:py:class:#datetime.datetime), [`datetime.timedelta`](inv:python:py:class:#datetime.timedelta), etc.                                               |
-| [`json`](inv:python:py:module:#json)         | This is the json module object from the Python standard library. Primarily used for calling [`json.dumps()`](inv:python:py:function:#json.dumps) and [`json.loads()`](inv:python:py:function:#json.loads).                                                             |
+| Function                                             | Description                                                                                                                                                                                                                                                                  |
+| :--------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [`md5()`](inv:python:py:function:#hashlib.md5)       | Returns an inline MD5 hash of any string, outputting the string representation of the hash's hex digest. This is defined by the SDK internally with native python: [`hashlib.md5(<input>.encode("utf-8")).hexdigest()`](inv:python:py:method:#hashlib.hash.hexdigest).       |
+| [`sha256()`](inv:python:py:function:#hashlib.sha256) | Returns an inline SHA256 hash of any string, outputting the string representation of the hash's hex digest. This is defined by the SDK internally with native python: [`hashlib.sha256(<input>.encode("utf-8")).hexdigest()`](inv:python:py:method:#hashlib.hash.hexdigest). |
+| [`datetime`](inv:python:py:module:#datetime)         | This is the datetime module object from the Python standard library. You can access [`datetime.datetime`](inv:python:py:class:#datetime.datetime), [`datetime.timedelta`](inv:python:py:class:#datetime.timedelta), etc.                                                     |
+| [`json`](inv:python:py:module:#json)                 | This is the json module object from the Python standard library. Primarily used for calling [`json.dumps()`](inv:python:py:function:#json.dumps) and [`json.loads()`](inv:python:py:function:#json.loads).                                                                   |
 
 #### Built-in Variable Names
 

diff --git a/noxfile.py b/noxfile.py
@@ -105,7 +105,7 @@ def benches(session: nox.Session) -> None:
 @nox.session(name="deps", python=main_python_version)
 def dependencies(session: nox.Session) -> None:
     """Check issues with dependencies."""
-    session.install(".[s3,testing]")
+    session.install(".[docs,faker,jwt,parquet,s3,ssh,testing]")
     session.install("deptry")
     session.run("deptry", "singer_sdk", *session.posargs)