Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(transfer-dag): add needed assume role and stop discarding exceptions #260

Merged
merged 5 commits into from
Nov 22, 2024

Conversation

anayeaye
Copy link
Contributor

What

  • Raise exception at end of transfer loop if any exceptions or client errors occurred during loop
  • Correct format of variable stored in aws secrets manager (string required)
  • Add assume role arns to airflow_custom_variables stored in secrets manager for DAGs

Why

  • Transfer DAG successfully completed without reporting client errors
  • Assume role arns were needed for successful boto copy in transfer dag

How tested

  • Deployed to smce-staging and executed transfer DAG
  • Confirmed that assume role arns were added to airflow custom variables in secrets manager

@anayeaye anayeaye requested review from amarouane-ABDELHAK, smohiudd and slesaad and removed request for amarouane-ABDELHAK, smohiudd and slesaad November 22, 2024 19:14
@anayeaye anayeaye merged commit ef7be91 into dev Nov 22, 2024
3 checks passed
@anayeaye anayeaye deleted the fix/reveal-transfer-exception branch November 22, 2024 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants