Code

dev to main BLOCKED by issue #252#245

Code

Open

smohiudd wants to merge 135 commits intomainNASA-IMPACT/veda-data-airflow:mainfrom devNASA-IMPACT/veda-data-airflow:devCopy head branch name to clipboard

+2,241-808

Commits

Commits on Jul 19, 2024

• WIP: dynamic ingest mapping for continuous tracing

  ividito

  committed
  c1d67da

  Copy full SHA for c1d67da

Commits on Jul 31, 2024

• feat: use taskflow for concurrent task runs

  ividito

  committed
  55744c6

  Copy full SHA for 55744c6

• clean up debug code

  ividito

  committed
  ee98cf7

  Copy full SHA for ee98cf7

Commits on Aug 1, 2024

• Bugfix vector ingest

  ividito

  committed
  6e151a9

  Copy full SHA for 6e151a9

Commits on Aug 20, 2024

• merge dev

  smohiudd

  committed
  82c724f

  Copy full SHA for 82c724f

• add generic vector pipeline

  smohiudd

  committed
  54b885d

  Copy full SHA for 54b885d

• remove process generic vector pipeline

  smohiudd

  committed
  a7b51e1

  Copy full SHA for a7b51e1

Commits on Aug 27, 2024

• Update dags/veda_data_pipeline/veda_generic_vector_pipeline.py

  ividito

  authored
  318d39a

  Copy full SHA for 318d39a

Commits on Sep 9, 2024

• Restructure dataset get_files output, add retries to some tasks to avoid SIGKILL errors

  ividito

  committed
  0ab867b

  Copy full SHA for 0ab867b

Commits on Oct 4, 2024

• Changes to make vector ingest work in shared VPC environment

  ividito

  committed
  f1411fe

  Copy full SHA for f1411fe

Commits on Oct 16, 2024

• added the statistics calculations in pipeline

  

  vishal

  committed
  b6b4b5e

  Copy full SHA for b6b4b5e

• generating JSON file for the stats and uploading it to S3

  

  vishal

  committed
  42e88d7

  Copy full SHA for 42e88d7

Commits on Oct 17, 2024

• using temp file for JSON

  

  vishal

  committed
  a5dcb51

  Copy full SHA for a5dcb51

Commits on Oct 18, 2024

• feat:port event-driven vector automation to terraform

  ividito

  committed
  4db9b2c

  Copy full SHA for 4db9b2c

• fix: feature flag for vector automation lambda

  ividito

  committed
  ebb9993

  Copy full SHA for ebb9993

• fix: adjust log group count

  ividito

  committed
  4eca40e

  Copy full SHA for 4eca40e

• testing JSON update for multiple datasets

  

  vishal

  committed
  f98add0

  Copy full SHA for f98add0

• fix tf vector automation vars

  smohiudd

  committed
  0005892

  Copy full SHA for 0005892

• update vars in event bridge tf

  smohiudd

  committed
  38bff94

  Copy full SHA for 38bff94

• Merge pull request #239 from NASA-IMPACT/fix/vector-automation

  Show description for 90aed8f
  smohiudd

  authored
  90aed8f

  Copy full SHA for 90aed8f

• add new line

  smohiudd

  committed
  021f4ff

  Copy full SHA for 021f4ff

• Merge pull request #240 from NASA-IMPACT/fix/vector-automation

  Show description for 8ec0cd4
  smohiudd

  authored
  8ec0cd4

  Copy full SHA for 8ec0cd4

• tested the creation of JSON file for different datasets

  

  vishal

  committed
  5678be5

  Copy full SHA for 5678be5

• fix tag filter

  smohiudd

  committed
  abba102

  Copy full SHA for abba102

• Merge pull request #241 from NASA-IMPACT/fix/vector-automation

  Show description for 006e989
  smohiudd

  authored
  006e989

  Copy full SHA for 006e989

• update vector subnet ref

  smohiudd

  committed
  f181ca7

  Copy full SHA for f181ca7

• Merge pull request #242 from NASA-IMPACT/fix/vector-automation

  Show description for 39401a0
  smohiudd

  authored
  39401a0

  Copy full SHA for 39401a0

• fix vector subnet ids

  smohiudd

  committed
  857e2fa

  Copy full SHA for 857e2fa

• Merge pull request #243 from NASA-IMPACT/fix/vector-automation

  Show description for 9b43b89
  smohiudd

  authored
  9b43b89

  Copy full SHA for 9b43b89

Commits on Oct 21, 2024

• conditional vector subnets

  smohiudd

  committed
  e3870e7

  Copy full SHA for e3870e7

• feat: disable default api gateway endpoint for workflows api

  botanical

  committed
  266be89

  Copy full SHA for 266be89

• Merge pull request #246 from NASA-IMPACT/fix/vector-ingest-subnets

  Show description for 65f0a8c
  smohiudd

  authored
  65f0a8c

  Copy full SHA for 65f0a8c

• feat: add variable to configure disabling default apigw endpoint

  botanical

  committed
  d4a57e3

  Copy full SHA for d4a57e3

Commits on Oct 22, 2024

• Merge pull request #197 from NASA-IMPACT/fix/asset-handling

  Show description for b9a9323
  smohiudd

  authored
  b9a9323

  Copy full SHA for b9a9323

Commits on Oct 23, 2024

• Update infrastructure/terraform.tfvars.tmpl

  botanical

  authored
  d77d946

  Copy full SHA for d77d946

• Update infrastructure/terraform.tfvars.tmpl

  botanical

  authored
  7af892a

  Copy full SHA for 7af892a

Commits on Oct 24, 2024

• Using pythonoperators instead of ECS operator

  amarouane-ABDELHAK

  committed
  6d3e29a

  Copy full SHA for 6d3e29a

• Merge pull request #247 from NASA-IMPACT/jt/issue-452-disable-default-endpoint

  Show description for 61a797d
  botanical

  authored
  61a797d

  Copy full SHA for 61a797d

• Switching to pythonOperator

  amarouane-ABDELHAK

  committed
  686c4c8

  Copy full SHA for 686c4c8

Commits on Oct 25, 2024

• Merge pull request #249 from NASA-IMPACT/feature/use-pythonoperators

  Show description for c190564
  amarouane-ABDELHAK

  authored
  c190564

  Copy full SHA for c190564

• test: temporarily use test buckets

  anayeaye

  committed
  305dd43

  Copy full SHA for 305dd43

Commits on Oct 28, 2024

• feat:port event-driven vector automation to terraform

  

  ividito

  authored and

  vishal

  committed
  188f96e

  Copy full SHA for 188f96e

• fix: feature flag for vector automation lambda

  

  ividito

  authored and

  vishal

  committed
  a2e3bad

  Copy full SHA for a2e3bad

• fix: adjust log group count

  

  ividito

  authored and

  vishal

  committed
  12ba68c

  Copy full SHA for 12ba68c

• fix tf vector automation vars

  

  smohiudd

  authored and

  vishal

  committed
  f532a6f

  Copy full SHA for f532a6f

• update vars in event bridge tf

  

  smohiudd

  authored and

  vishal

  committed
  89d5251

  Copy full SHA for 89d5251

• add new line

  

  smohiudd

  authored and

  vishal

  committed
  9c636e0

  Copy full SHA for 9c636e0

• fix tag filter

  

  smohiudd

  authored and

  vishal

  committed
  47b0707

  Copy full SHA for 47b0707

• update vector subnet ref

  

  smohiudd

  authored and

  vishal

  committed
  8169ab6

  Copy full SHA for 8169ab6

• fix vector subnet ids

  

  smohiudd

  authored and

  vishal

  committed
  66cfb79

  Copy full SHA for 66cfb79

• conditional vector subnets

  

  smohiudd

  authored and

  vishal

  committed
  c449cb9

  Copy full SHA for c449cb9

• WIP: dynamic ingest mapping for continuous tracing

  

  ividito

  authored and

  vishal

  committed
  0df795e

  Copy full SHA for 0df795e

• feat: use taskflow for concurrent task runs

  

  ividito

  authored and

  vishal

  committed
  bfce3d5

  Copy full SHA for bfce3d5

• clean up debug code

  

  ividito

  authored and

  vishal

  committed
  31a3e84

  Copy full SHA for 31a3e84

• Bugfix vector ingest

  

  ividito

  authored and

  vishal

  committed
  a811fe2

  Copy full SHA for a811fe2

• add generic vector pipeline

  

  smohiudd

  authored and

  vishal

  committed
  3c034ae

  Copy full SHA for 3c034ae

• remove process generic vector pipeline

  

  smohiudd

  authored and

  vishal

  committed
  4a5dc02

  Copy full SHA for 4a5dc02

• Update dags/veda_data_pipeline/veda_generic_vector_pipeline.py

  

  ividito

  authored and

  vishal

  committed
  de8b69d

  Copy full SHA for de8b69d

• Restructure dataset get_files output, add retries to some tasks to avoid SIGKILL errors

  

  ividito

  authored and

  vishal

  committed
  d28432d

  Copy full SHA for d28432d

• Changes to make vector ingest work in shared VPC environment

  

  ividito

  authored and

  vishal

  committed
  26feccf

  Copy full SHA for 26feccf

• feat: disable default api gateway endpoint for workflows api

  

  botanical

  authored and

  vishal

  committed
  18470d0

  Copy full SHA for 18470d0

• feat: add variable to configure disabling default apigw endpoint

  

  botanical

  authored and

  vishal

  committed
  919e04b

  Copy full SHA for 919e04b

• Update infrastructure/terraform.tfvars.tmpl

  

  botanical

  authored and

  vishal

  committed
  81f94de

  Copy full SHA for 81f94de

• Update infrastructure/terraform.tfvars.tmpl

  

  botanical

  authored and

  vishal

  committed
  7485696

  Copy full SHA for 7485696

• Using pythonoperators instead of ECS operator

  

  amarouane-ABDELHAK

  authored and

  vishal

  committed
  75fef32

  Copy full SHA for 75fef32

• Switching to pythonOperator

  

  amarouane-ABDELHAK

  authored and

  vishal

  committed
  6f080a7

  Copy full SHA for 6f080a7

• tested the creation of JSON file for different datasets

  

  vishal

  committed
  dafb213

  Copy full SHA for dafb213

• optimizing the automation

  

  vishal

  committed
  884c296

  Copy full SHA for 884c296

Commits on Oct 29, 2024

• Merge branch 'dev' into feature/optimize_automated_cog_transformation

  

  vishal

  committed
  05cd27d

  Copy full SHA for 05cd27d

• testing the optimized approach for automation

  

  vishal

  committed
  8d64aaa

  Copy full SHA for 8d64aaa

• successfully tested the automation for gridded population dataset

  

  vishal

  committed
  3ada005

  Copy full SHA for 3ada005

• Add S3 event bridge

  amarouane-ABDELHAK

  committed
  a619098

  Copy full SHA for a619098

• Add S3 event bridge

  amarouane-ABDELHAK

  committed
  4c9753f

  Copy full SHA for 4c9753f

• Add S3 event bridge

  amarouane-ABDELHAK

  committed
  59e1938

  Copy full SHA for 59e1938

Commits on Oct 30, 2024

• Add env deployment example

  amarouane-ABDELHAK

  committed
  685e7db

  Copy full SHA for 685e7db

• Merge pull request #251 from NASA-IMPACT/feat/use-sm2a-in-eventbrige-lambda

  Show description for dd0ea9e
  amarouane-ABDELHAK

  authored
  dd0ea9e

  Copy full SHA for dd0ea9e

Commits on Nov 4, 2024

• updated readme for the folder and resolved the failure bug

  

  vishal

  committed
  a64099a

  Copy full SHA for a64099a

Commits on Nov 6, 2024

• only execute deploy action on push to dev branch

  anayeaye

  committed
  8a3c584

  Copy full SHA for 8a3c584

• Merge pull request #254 from NASA-IMPACT/ci/remove-staging-action

  Show description for 4a277d3
  anayeaye

  authored
  4a277d3

  Copy full SHA for 4a277d3

• ci: do not automatically deploy mwaa to any environment

  anayeaye

  committed
  db73561

  Copy full SHA for db73561

Commits on Nov 8, 2024

• Merge pull request #255 from NASA-IMPACT/ci/remove-all-mwaa-deploy-actions

  Show description for c450be2
  anayeaye

  authored
  c450be2

  Copy full SHA for c450be2

Commits on Nov 13, 2024

• Adding condition on s3 event bridge

  amarouane-ABDELHAK

  committed
  b258d41

  Copy full SHA for b258d41

Commits on Nov 14, 2024

• Adding condition on s3 event bridge

  amarouane-ABDELHAK

  committed
  eb5e32c

  Copy full SHA for eb5e32c

• Fix some missing variables

  amarouane-ABDELHAK

  committed
  4909f43

  Copy full SHA for 4909f43

Commits on Nov 15, 2024

• Remove .env file

  amarouane-ABDELHAK

  committed
  87a08e2

  Copy full SHA for 87a08e2

• Remove .env file

  amarouane-ABDELHAK

  committed
  cf19bd3

  Copy full SHA for cf19bd3

• Remove the airflow config to be created by CICD

  amarouane-ABDELHAK

  committed
  1335a92

  Copy full SHA for 1335a92

Commits on Nov 21, 2024

• print all transfer exceptions

  anayeaye

  committed
  dbcd631

  Copy full SHA for dbcd631

Commits on Nov 22, 2024

• Merge pull request #257 from NASA-IMPACT/feat/use-sm2a-in-eventbrige-lambda

  Show description for 32b2acb
  amarouane-ABDELHAK

  authored
  32b2acb

  Copy full SHA for 32b2acb

• Merge branch 'dev' into fix/reveal-transfer-exception

  anayeaye

  committed
  7872364

  Copy full SHA for 7872364

• print client err on failed transfer

  anayeaye

  committed
  4374f34

  Copy full SHA for 4374f34

• raise transfer exception

  anayeaye

  committed
  091bc80

  Copy full SHA for 091bc80

• update secret variable type to string and add assume role arns to airflow_custom_variables"

  anayeaye

  committed
  4f14c96

  Copy full SHA for 4f14c96

• Merge pull request #260 from NASA-IMPACT/fix/reveal-transfer-exception

  Show description for ef7be91
  anayeaye

  authored
  ef7be91

  Copy full SHA for ef7be91

Commits on Nov 25, 2024

• fix(dockerfile) remove pypgstac install

  anayeaye

  committed
  fe09938

  Copy full SHA for fe09938

• fix(dag-requirements) remove pypgstac install

  anayeaye

  committed
  56738e9

  Copy full SHA for 56738e9

• Merge pull request #262 from NASA-IMPACT/fix/remove-pypgstac

  Show description for 66193a8
  anayeaye

  authored
  66193a8

  Copy full SHA for 66193a8

• let ingest api handle datetime validation and just pass through inputs in collection generation dag

  anayeaye

  committed
  4fbd229

  Copy full SHA for 4fbd229

Commits on Nov 26, 2024

• Merge pull request #263 from NASA-IMPACT/fix/collection-datetime-format

  Show description for d9799cc
  anayeaye

  authored
  d9799cc

  Copy full SHA for d9799cc

• temp print statement

  smohiudd

  committed
  926be29

  Copy full SHA for 926be29

• Merge pull request #264 from NASA-IMPACT/fix/collection-print

  Show description for 7e05d2f
  smohiudd

  authored
  7e05d2f

  Copy full SHA for 7e05d2f

• remove comma

  smohiudd

  committed
  2974701

  Copy full SHA for 2974701

• Merge pull request #265 from NASA-IMPACT/fix/collection-print

  Show description for 5d8757e
  smohiudd

  authored
  5d8757e

  Copy full SHA for 5d8757e

Commits on Dec 3, 2024

• Strip thumbnail assets from payload before discovery

  ividito

  committed
  64683af

  Copy full SHA for 64683af

• rename var

  ividito

  committed
  9d7c611

  Copy full SHA for 9d7c611

• Update dags/veda_data_pipeline/veda_dataset_pipeline.py

  Show description for 3a48975
  

  ividito

  and

  anayeaye

  committed
  3a48975

  Copy full SHA for 3a48975

• Fix callable

  ividito

  committed
  0e5cc7c

  Copy full SHA for 0e5cc7c

Commits on Dec 6, 2024

• Refactor Dataset Pipeline

  amarouane-ABDELHAK

  committed
  53416d0

  Copy full SHA for 53416d0

• Refactor Dataset Pipeline

  amarouane-ABDELHAK

  committed
  5a41003

  Copy full SHA for 5a41003

• Return status of the ingest

  amarouane-ABDELHAK

  committed
  cfb2bf2

  Copy full SHA for cfb2bf2

Commits on Dec 11, 2024

• Update dags/veda_data_pipeline/groups/discover_group.py

  Show description for 4e0416b
  

  amarouane-ABDELHAK

  and

  botanical

  authored
  4e0416b

  Copy full SHA for 4e0416b

• Adding aut ingest

  amarouane-ABDELHAK

  committed
  534e571

  Copy full SHA for 534e571

• Fix the bug of updating a pointer

  amarouane-ABDELHAK

  committed
  000a38c

  Copy full SHA for 000a38c

• Merge branch 'refactor-dataset-pipeline' of https://github.com/NASA-IMPACT/veda-data-airflow into refactor-dataset-pipeline

  amarouane-ABDELHAK

  committed
  64bea83

  Copy full SHA for 64bea83

• Fix updating the copy of config instead of the pointer

  amarouane-ABDELHAK

  committed
  74573b7

  Copy full SHA for 74573b7

• Merge pull request #268 from NASA-IMPACT/refactor-dataset-pipeline

  Show description for e6ca7d4
  amarouane-ABDELHAK

  authored
  e6ca7d4

  Copy full SHA for e6ca7d4

• Fixing automation DAG

  amarouane-ABDELHAK

  committed
  da43490

  Copy full SHA for da43490

• Fixing automation DAG

  amarouane-ABDELHAK

  committed
  ac3d475

  Copy full SHA for ac3d475

• Fixing automation DAG

  amarouane-ABDELHAK

  committed
  b584575

  Copy full SHA for b584575

• Add slack notification libraries

  amarouane-ABDELHAK

  committed
  487f225

  Copy full SHA for 487f225

• Add slack notification libraries

  amarouane-ABDELHAK

  committed
  82f1109

  Copy full SHA for 82f1109

• Merge pull request #236 from NASA-IMPACT/feature/optimize_automated_cog_transformation

  Show description for 4fdf934
  amarouane-ABDELHAK

  authored
  4fdf934

  Copy full SHA for 4fdf934

Commits on Dec 12, 2024

• Update SM2A workers python version to 3.11

  amarouane-ABDELHAK

  committed
  603a713

  Copy full SHA for 603a713

• Update the CMD for the workers

  amarouane-ABDELHAK

  committed
  45175e0

  Copy full SHA for 45175e0

Commits on Dec 16, 2024

• Merge pull request #271 from NASA-IMPACT/fix/update-workers-python-version

  Show description for b5b02be
  amarouane-ABDELHAK

  authored
  b5b02be

  Copy full SHA for b5b02be

Commits on Dec 20, 2024

• Add placeholder transfer to dataset DAG (no-op for now, just testing control flow)

  Show description for 9a8d9d2
  ividito

  committed
  9a8d9d2

  Copy full SHA for 9a8d9d2

• More makefile adjustments, adjust control flow

  ividito

  committed
  95fe45a

  Copy full SHA for 95fe45a

• Reorder transfer <> discovery, change transfer task input

  ividito

  committed
  caad8ba

  Copy full SHA for caad8ba

• Break out promotion to new DAG

  ividito

  committed
  4a5f166

  Copy full SHA for 4a5f166

• Remove transfer flag

  ividito

  committed
  829fa24

  Copy full SHA for 829fa24

• Light refactors to shared tasks, fixes to align promotion to dataset refactors

  ividito

  committed
  dee4542

  Copy full SHA for dee4542

• set transfer dry run default back to false

  ividito

  committed
  46e66e6

  Copy full SHA for 46e66e6

• Fix makefile, fix ti KeyError

  ividito

  committed
  da0650c

  Copy full SHA for da0650c

• regex updated

  siddharth0248

  committed
  2f39ead

  Copy full SHA for 2f39ead

• Merge pull request #274 from NASA-IMPACT/regex/sid

  Show description for c22f94c
  siddharth0248

  authored
  c22f94c

  Copy full SHA for c22f94c