V24.11.1 (#1543)

* add new tests * add unit tests * ran black * Update schematic/models/validate_attribute.py Co-authored-by: BryanFauble <[email protected]> * added tests * Update README.md * Update README.md * add unit tests * run black * Update README.md * temp commit * remove old tests * [FDS-2386] Synapse entity tracking and code concurrency updates (#1505) * [FDS-2386] Synapse entity tracking and code concurrency updates * ran black * Update CODEOWNERS * updated data model type rules to include error param * fix validate type attribute to use msg level param * added error handling * run black * create Node class * sat up Node class so that nodes with no displayName fields cause an error on creation * ran black * ran mypy * added new configs for CLI tests * added new manifests for testing CLI commands * automate manual CLI tests * ran black * Update CODEOWNERS * Update scan_repo.yml * Update .github/CODEOWNERS * Update .github/workflows/scan_repo.yml * Attach additional telemetry data to OTEL traces (#1519) * Attach additional telemetry data to OTEL traces * feat: added tracing for cross manifest validation and file name validation (#1509) * add tracing for GX validation * temp commit * Updating contribution doc to expect squash and merge (#1534) * [FDS-2491] Integration tests for Schematic API Test plan (#1512) Integration tests for Schematic API Test plan * [FDS-2500] Add Integration Tests for: Manifest Validation (#1516) * Add Integration Tests for: Manifest Validation * [FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530) Also install `typing-extensions` in the build * manual test files now being saved in manifests folder * manual test files now being saved in manifests folder * remove lines to delete json files that were under git control * ran black * add try finally blocks to remove created files * ran black * add lines to remove created json files * Update file annotation store process to require filename be present in order to annotate file * add lines to remove created json files * Revert "Update file annotation store process to require filename be present in order to annotate file" This reverts commit f57c718. * Don't attempt to annotate the table * add code in finally blocks to reset config to default values, when tests change them * complete submit manifest command test * ran black * add test for bug case * update test for table tidyness * remove unused import * remove etag column if already present when building temp file view * catch all exceptions to switch to sequential mode * update test for updated data * Revert "update test for updated data" This reverts commit 255e3c0. * Revert "catch all exceptions to switch to sequential mode" This reverts commit 68b0b24. * catch ValueErrors as well * Updates for integration test failures (#1537) * Updates for integration test failures, Config file reset and scope changes * add todos for removing config resets * [FDS-2525] Authenticated export of telemetry data (#1527) * Authenticated export of telemetry data, updating to HTTP otel library * temp reduce tests * restore tests * uncomment tests * redid how files are deleted, manual tests values are set * ran black * [SCHEMATIC-157] Make some dependencies required to avoid `schematic CLI` commands from potentially erroring when doing a pip install (#1540) * Make otel flash non-optional * Add dependencies as non-optional * Update pyproject.toml --------- Co-authored-by: BryanFauble <[email protected]> Co-authored-by: Jenny V Medina <[email protected]> Co-authored-by: Thomas Yu <[email protected]> Co-authored-by: Lingling <[email protected]> Co-authored-by: GiaJordan <[email protected]>
Sage-Bionetworks · Nov 14, 2024 · 2f40625 · 2f40625
1 parent d68f624
commit 2f40625
Show file tree

Hide file tree

Showing 67 changed files with 9,925 additions and 2,822 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -1,2 +1,2 @@
 # Global owners of whole repo
-* @andrewelamb @GiaJordan @linglp
+* @Sage-Bionetworks/schematic-developers
diff --git a/.github/workflows/scan_repo.yml b/.github/workflows/scan_repo.yml
@@ -17,7 +17,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Run Trivy vulnerability scanner in repo mode
-        uses: aquasecurity/trivy-action@master
+        uses: aquasecurity/trivy-action@0.28.0
         with:
           # the scan targets the file system.
           scan-type: 'fs'

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -134,7 +134,16 @@ jobs:
           SERVICE_ACCOUNT_CREDS: ${{ secrets.SERVICE_ACCOUNT_CREDS }}
         run: >
           poetry run pytest --durations=0 --cov-append --cov-report=term --cov-report=html:htmlcov --cov-report=xml:coverage.xml --cov=schematic/
-          -m "not (rule_benchmark)" --reruns 4 -n 8 --ignore=tests/unit
+          -m "not (rule_benchmark or single_process_execution)" --reruns 4 -n 8 --ignore=tests/unit
+
+      - name: Run integration tests single process
+        if: ${{ contains(fromJSON('["3.10"]'), matrix.python-version) }}
+        env:
+          SYNAPSE_ACCESS_TOKEN: ${{ secrets.SYNAPSE_ACCESS_TOKEN }}
+          SERVICE_ACCOUNT_CREDS: ${{ secrets.SERVICE_ACCOUNT_CREDS }}
+        run: >
+          poetry run pytest --durations=0 --cov-append --cov-report=term --cov-report=html:htmlcov --cov-report=xml:coverage.xml --cov=schematic/
+          -m "single_process_execution" --reruns 4 --ignore=tests/unit
 
 
       - name: Upload pytest test results

diff --git a/.gitignore b/.gitignore
@@ -164,7 +164,7 @@ clean.sh
 # Intermediate files
 data/json_schema_logs/json_schema_log.json
 great_expectations/checkpoints/manifest_checkpoint.yml
-great_expectations/expectations/Manifest_test_suite.json
+great_expectations/expectations/Manifest_test_suite*.json
 
 tests/data/example.MockComponent.schema.json
 tests/data/mock_manifests/Invalid_Test_Manifest_censored.csv
@@ -177,6 +177,7 @@ tests/data/schema.gpickle
 # Created during testting
 Example*
 manifests/*
+https:*
 
 # schematic config file
 config.yml
diff --git a/.readthedocs.yml b/.readthedocs.yml
@@ -7,14 +7,15 @@ version: 2
 
 # Set the version of Python and other tools you might need
 build:
-  os: ubuntu-20.04
+  os: ubuntu-22.04
   tools:
     python: "3.9"
   jobs:
     post_install:
       - pip install poetry==1.3.0
       - poetry config virtualenvs.create false
-      - poetry install --with doc
+      - poetry install --all-extras
+      - pip install typing-extensions==4.12.2
     #Poetry will install my dependencies into the virtualenv created by readthedocs if I set virtualenvs.create=false
     # You can also specify other tool versions:
     # nodejs: "16"

diff --git a/CONTRIBUTION.md b/CONTRIBUTION.md
@@ -49,7 +49,7 @@ For new features, bugs, enhancements:
 #### 4. Pull Request and Review
 * Create a PR from `develop-<feature/fix-name>` into the develop branch of the upstream repo
 * Request a code review on the PR
-* Once code is approved merge in the develop branch. We suggest creating a merge commit for a cleaner commit history on the `develop` branch.
+* Once code is approved merge in the develop branch. The **"Squash and merge"** strategy should be used for a cleaner commit history on the `develop` branch. The description of the squash commit should include enough information to understand the context of the changes that were made.
 * Once the actions pass on the main branch, delete the `develop-<feature/fix-name>` branch
 
 ### Updating readthedocs documentation

diff --git a/README.md b/README.md
diff --git a/env.example b/env.example
@@ -5,9 +5,37 @@ SERVER_DOMAIN=localhost
 USE_LISTEN_PORT=81
 SERVICE_ACCOUNT_CREDS='Provide service account creds'
 
-# Integration testing variables (Optional)
+## Opentelemetry configuration variables (Optional)
+## Only otlp is supported
 # TRACING_EXPORT_FORMAT=otlp
 # LOGGING_EXPORT_FORMAT=otlp
-# TRACING_SERVICE_NAME=unique-name-testing
-# LOGGING_SERVICE_NAME=unique-name-testing
-# LOGGING_INSTANCE_NAME=unique-name-testing
+# TRACING_SERVICE_NAME=schematic-api
+# LOGGING_SERVICE_NAME=schematic-api
+## Other examples: dev, staging, prod
+# DEPLOYMENT_ENVIRONMENT=local
+# OTEL_EXPORTER_OTLP_ENDPOINT=https://..../telemetry
+## Opentelemetry authentication
+# TELEMETRY_EXPORTER_CLIENT_ID=...
+# TELEMETRY_EXPORTER_CLIENT_SECRET-...
+# TELEMETRY_EXPORTER_CLIENT_TOKEN_ENDPOINT=...
+# TELEMETRY_EXPORTER_CLIENT_AUDIENCE=...
+## Alternative Opentelemetry authentication: Sets a static Authorization header to use for all requests. Used when developing locally
+# OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer ey...
+
+
+# Used during integration test run to determine if files will be output for manual
+# inspection. These tests cannot fully finish all validation via code. All of these
+# tests will be marked by pytest "manual_verification_required"
+# More information: https://sagebionetworks.jira.com/wiki/spaces/SCHEM/pages/3055779846/Schematic+API+test+plan 
+MANUAL_TEST_VERIFICATION=false
+
+# Used to determine if a local flask instance is created during integration testing. If
+# this is true schematic tests will use a schematic API server running outside of the
+# context of the integration test. The url used is defined below.
+USE_DEPLOYED_SCHEMATIC_API_SERVER=false
+
+# The URL used to execute integration tests for schematic API. Defaults to localhost.
+# dev: https://schematic-dev.api.sagebionetworks.org
+# staging: https://schematic-staging.api.sagebionetworks.org
+# prod: https://schematic.api.sagebionetworks.org
+SCHEMATIC_API_SERVER_URL=http://localhost:3001
diff --git a/main.py b/main.py
@@ -1,7 +1,4 @@
 import os
-import connexion
-from schematic import CONFIG
-from flask_cors import CORS
 from schematic_api.api import app