Updates to deploy PDS Nucleus in MCP Test and Prod environments #98

ramesh-maddegoda · 2024-05-10T00:31:06Z

🗒️ Summary

This pull request updates Terraform scripts, IAM policy files, Lambda functions and documentation to deploy PDS Nucleus in MCP Test and Prod environments.

♻️ Related Issues

#89

…mentation to deploy PDS Nucleus in MCP Test and Prod environments. Refer to the issue: #89

Refer to the issue: #89

…. Force delete ECR repositories. Update provider version. Refer to the issue: #89

terraform/terraform-modules/ecs-ecr/ecs_ecr.tf

Refer to the issue: #89

jordanpadams · 2024-05-29T16:07:52Z

@ramesh-maddegoda note: the secrets workflow is failing so the secrets baseline may need to be regenerated

nutjob4life · 2024-05-29T16:54:00Z

@ramesh-maddegoda FYI, to create a new secrets baseline:

$ detect-secrets scan --disable-plugin AbsolutePathDetectorExperimental \
    --exclude-files '\.secrets..*' \
    --exclude-files '\.git.*' \
    --exclude-files '\.pre-commit-config\.yaml' \
    --exclude-files '\.mypy_cache' \
    --exclude-files '\.pytest_cache' \
    --exclude-files '\.tox' \
    --exclude-files '\.venv' \
    --exclude-files 'venv' \
    --exclude-files 'dist' \
    --exclude-files 'build' \
    --exclude-files '.*\.egg-info' \
    > .secrets.baseline

nutjob4life

Whoa … this was an enormous pull request! +2628, -1547 😅

Let me know if I should clarify any remarks/questions. Thanks in advance!

.secrets.baseline

nutjob4life · 2024-05-29T18:27:27Z

terraform/README.md

+password = ChangeMe!
+```
+
+18. Use the PDS Data Upload Manager (DUM) tool to upload files to pds_nucleus_staging_bucket.


Whew! I opened these up in a Markdown viewer. Thanks for a really complete set of instructions 😌

terraform/terraform-modules/ecs-ecr/docker/config-init/Dockerfile

nutjob4life · 2024-05-29T18:57:27Z

terraform/terraform-modules/ecs-ecr/docker/s3-to-efs-copy/Dockerfile

+ENV LD_LIBRARY_PATH="/cfitsio-4.4.0/lib"
+
+COPY entrypoint.sh /usr/local/bin/entrypoint.sh
+ENTRYPOINT ["bash", "/usr/local/bin/entrypoint.sh"]


terraform/terraform-modules/ecs-ecr/variables.tf

nutjob4life · 2024-05-29T20:53:39Z

terraform/terraform-modules/ecs-ecr/variables.tf

+  sensitive   = true
+}
+
+#variable "pds_nucleus_config_init_ecr_image_path" {


nutjob4life · 2024-05-29T20:53:47Z

terraform/terraform-modules/ecs-ecr/variables.tf

+}
+
+
+#variable "pds_nucleus_s3_to_efs_copy_ecr_image_path" {


terraform/terraform-modules/ecs-ecr/variables.tf

terraform/terraform-modules/mwaa-env/requirements.txt

ramesh-maddegoda · 2024-05-30T18:13:45Z

Whoa … this was an enormous pull request! +2628, -1547 😅

Let me know if I should clarify any remarks/questions. Thanks in advance!

Thank a lot for your review!

…WS resources based on provided lists of Node specific configs. UPDATE to incorporate review comments and code cleanup

…n.com/mwaa/latest/userguide/mwaa-create-role.html#mwaa-create-role-aocmk

…S/nucleus into mcp-deployment-terraform

…ther code cleanup.

…mpty s3 buckets during terraform destroy UPDATE code with new policy documents

…ithub.io/registry/user/harvest_job_configuration.html

…rets and code improvements

…ber of layers

…ion_boundary_for_iam_roles

…t manager

ramesh-maddegoda · 2024-06-20T16:56:07Z

@ramesh-maddegoda FYI, to create a new secrets baseline:

$ detect-secrets scan --disable-plugin AbsolutePathDetectorExperimental \
    --exclude-files '\.secrets..*' \
    --exclude-files '\.git.*' \
    --exclude-files '\.pre-commit-config\.yaml' \
    --exclude-files '\.mypy_cache' \
    --exclude-files '\.pytest_cache' \
    --exclude-files '\.tox' \
    --exclude-files '\.venv' \
    --exclude-files 'venv' \
    --exclude-files 'dist' \
    --exclude-files 'build' \
    --exclude-files '.*\.egg-info' \
    > .secrets.baseline

@jordanpadams and @nutjob4life , I created a new secret baseline.

UPDATE Terraform scripts, IAM policy files, Lambda functions and docu…

8a28dbd

…mentation to deploy PDS Nucleus in MCP Test and Prod environments. Refer to the issue: #89

ramesh-maddegoda requested review from tloubrieu-jpl, nutjob4life, sjoshi-jpl and jordanpadams as code owners May 10, 2024 00:31

ramesh-maddegoda mentioned this pull request May 10, 2024

Deploy Nucleus in Test env #89

Closed

3 tasks

ramesh-maddegoda added 3 commits May 9, 2024 17:50

UPDATE README with permission_boundary_for_iam_roles details.

214bee7

Refer to the issue: #89

UPDATE the path related to the Terraform directory

36c0d8a

Refer to the issue: #89

UPDATE code to externalize availability zones and permission boundary…

32073f0

…. Force delete ECR repositories. Update provider version. Refer to the issue: #89

viviant100 reviewed May 11, 2024

View reviewed changes

terraform/terraform-modules/ecs-ecr/ecs_ecr.tf Outdated Show resolved Hide resolved

ramesh-maddegoda added 5 commits May 14, 2024 01:52

UPDATE code to automatically update IAM policies based on AWS account ID

da0ce64

Refer to the issue: #89

UPDATE code to automatically update IAM policies based on AWS account ID

58acc8d

Refer to the issue: #89

UPDATE code to automatically update IAM policies based on AWS account ID

ecf0903

Refer to the issue: #89

UPDATE code to automatically update IAM policies based on AWS account ID

41d8729

Refer to the issue: #89

UPDATE code to automatically update IAM policies based on AWS account ID

eeb5edc

Refer to the issue: #89

nutjob4life requested changes May 29, 2024

View reviewed changes

ramesh-maddegoda and others added 11 commits May 30, 2024 13:15

UPDATE code to support multiple PDS Nodes by creating Node specific A…

6566302

…WS resources based on provided lists of Node specific configs. UPDATE to incorporate review comments and code cleanup

ADD More permissions to MWAA execution role

6c651d1

ADD policy files as placeholders for terraform initial run

f9d23cf

Update secrets.baseline

bf5a2cc

Updated secrets.baseline

d878490

Delete .github/workflows/build-docs.yml

cc62def

Update build_docs.yml

2d0da69

UPDATE MWAA execution role policy based on the https://docs.aws.amazo…

e7ddd77

…n.com/mwaa/latest/userguide/mwaa-create-role.html#mwaa-create-role-aocmk

Merge branch 'mcp-deployment-terraform' of https://github.com/NASA-PD…

d8b98b6

…S/nucleus into mcp-deployment-terraform

UPDATE IAM policies after resolving the MWAA log creation issue and o…

e500724

…ther code cleanup.

REMOVE force delete option from S3 buckets to avoid deleting s3 non-e…

65902b5

…mpty s3 buckets during terraform destroy UPDATE code with new policy documents

ramesh-maddegoda added 10 commits June 11, 2024 18:23

UPDATE NODE_NAME environment variable name in lambda functions

986d131

ADD placeholder shell script to deploy ECR images

3af2f05

UPDATE to use PDS Node name format as mentioned in https://nasa-pds.g…

1d389d6

…ithub.io/registry/user/harvest_job_configuration.html

ADD node specific default DAGs

6a0041e

UPDATE to use PDS Node name format as mentioned in https://nasa-pds.g…

2a3e056

…ithub.io/registry/user/harvest_job_configuration.html

UPDATE to retrieve Opensearch credentials from AWS Secret Manager sec…

63fe890

…rets and code improvements

UPDATE Docker RUN command in config-init Dockerfile to reduce the num…

0fd39d3

…ber of layers

UPDATE secret baseline

f00b889

UPDATE variable name from permission_boundary_for_iam_role to permiss…

f7e16fa

…ion_boundary_for_iam_roles

UPDATE instructions to use Opensearch credentials stored in AWS secre…

1f3fb52

…t manager

ramesh-maddegoda mentioned this pull request Jun 20, 2024

Read harvest auth credentials from AWS secret manager #100

Closed

ramesh-maddegoda and others added 2 commits June 20, 2024 10:07

UPDATE requirements file to use the compatible release operator

2ce4718

Answering "yes" to each is-secret question

fb6e67d

jordanpadams approved these changes Jun 20, 2024

View reviewed changes

jordanpadams merged commit 304a2c9 into main Jun 20, 2024
1 check passed

jordanpadams deleted the mcp-deployment-terraform branch June 20, 2024 18:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates to deploy PDS Nucleus in MCP Test and Prod environments #98

Updates to deploy PDS Nucleus in MCP Test and Prod environments #98

ramesh-maddegoda commented May 10, 2024

jordanpadams commented May 29, 2024

nutjob4life commented May 29, 2024

nutjob4life left a comment

nutjob4life May 29, 2024

nutjob4life May 29, 2024

nutjob4life May 29, 2024

nutjob4life May 29, 2024

ramesh-maddegoda Jun 20, 2024

ramesh-maddegoda commented May 30, 2024

ramesh-maddegoda commented Jun 20, 2024

		}


		#variable "pds_nucleus_s3_to_efs_copy_ecr_image_path" {

Updates to deploy PDS Nucleus in MCP Test and Prod environments #98

Updates to deploy PDS Nucleus in MCP Test and Prod environments #98

Conversation

ramesh-maddegoda commented May 10, 2024

🗒️ Summary

♻️ Related Issues

jordanpadams commented May 29, 2024

nutjob4life commented May 29, 2024

nutjob4life left a comment

Choose a reason for hiding this comment

nutjob4life May 29, 2024

Choose a reason for hiding this comment

nutjob4life May 29, 2024

Choose a reason for hiding this comment

nutjob4life May 29, 2024

Choose a reason for hiding this comment

nutjob4life May 29, 2024

Choose a reason for hiding this comment

ramesh-maddegoda Jun 20, 2024

Choose a reason for hiding this comment

ramesh-maddegoda commented May 30, 2024

ramesh-maddegoda commented Jun 20, 2024