[CMDCT-3750] Add a seed mechanism (#14965)

Co-authored-by: benmartin-coforma <[email protected]>

.github/build_vars.sh

@@ -5,7 +5,6 @@ var_list=(
   'AWS_DEFAULT_REGION'
   'CODE_CLIMATE_ID'
   'STAGE_PREFIX'
-  'SEED_DATABASE'
 )
 
 set_value() {

.github/workflows/deploy.yml

@@ -33,7 +33,6 @@ jobs:
           AWS_DEFAULT_REGION: ${{ secrets[env.BRANCH_SPECIFIC_VARNAME_AWS_DEFAULT_REGION] || secrets.AWS_DEFAULT_REGION }}
           CODE_CLIMATE_ID: ${{ secrets.CODE_CLIMATE_ID }}
           STAGE_PREFIX: ${{ secrets.STAGE_PREFIX }}
-          SEED_DATABASE: ${{ secrets[env.BRANCH_SPECIFIC_VARNAME_SEED_DATABASE] }}
       - name: Configure AWS credentials for GitHub Actions OIDC
         uses: aws-actions/configure-aws-credentials@v4
         with:

README.md

@@ -5,11 +5,12 @@
 [![Test Coverage](https://api.codeclimate.com/v1/badges/6dd9a7765a10a72867f2/test_coverage)](https://codeclimate.com/repos/64497017eab34100ce938fe6/test_coverage)
 
 ### Integration Environment Deploy Status:
-| Branch  | Build Status |
-| ------------- | ------------- |
-| master  | ![deploy](https://github.com/Enterprise-CMCS/macpro-mdct-seds/actions/workflows/deploy.yml/badge.svg)  |
-| val  | ![deploy](https://github.com/Enterprise-CMCS/macpro-mdct-seds/actions/workflows/deploy.yml/badge.svg?branch=val)  |
-| production  | ![deploy](https://github.com/Enterprise-CMCS/macpro-mdct-seds/actions/workflows/deploy.yml/badge.svg?branch=production)  |
+
+| Branch     | Build Status                                                                                                            |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------- |
+| master     | ![deploy](https://github.com/Enterprise-CMCS/macpro-mdct-seds/actions/workflows/deploy.yml/badge.svg)                   |
+| val        | ![deploy](https://github.com/Enterprise-CMCS/macpro-mdct-seds/actions/workflows/deploy.yml/badge.svg?branch=val)        |
+| production | ![deploy](https://github.com/Enterprise-CMCS/macpro-mdct-seds/actions/workflows/deploy.yml/badge.svg?branch=production) |
 
 SEDS is the CMCS MDCT application for collecting state data related to Medicaid and CHIP quarterly enrollment data on a quarterly basis. The collected data assists CMCS in monitoring, managing, and better understanding Medicaid and CHIP programs.
 
@@ -40,19 +41,21 @@ On each PR, a linter and prettier check runs. These checks must pass for a PR to
 - Run Eslint using `yarn lint`
 - Run Prettier using `npx prettier --write .`
 
-
 ## Local Dev
 
 ### Running MDCT Workspace Setup
-Team members are encouraged to setup all MDCT Products using the script located in the [MDCT Tools Repository](https://github.com/Enterprise-CMCS/macpro-mdct-tools). Please refer to the README for instructions running the MDCT Workspace Setup. After Running workspace setup team members can continue in this README for instructions on running the application locally. 
+
+Team members are encouraged to setup all MDCT Products using the script located in the [MDCT Tools Repository](https://github.com/Enterprise-CMCS/macpro-mdct-tools). Please refer to the README for instructions running the MDCT Workspace Setup. After Running workspace setup team members can continue in this README for instructions on running the application locally.
 
 #### For developers that have run workspace setup running the applicaiton locally please run the following
-1) cd ~/Projects/macpro-mdct-seds/
-2) `./run local --update-env` or `./run local` 
+
+1. cd ~/Projects/macpro-mdct-seds/
+2. `./run local --update-env` or `./run local`
 
    note: the `./run local --update-env` command will reach out to 1Password to bring in secret values and populate .env files that are git ignored. If you use the `./run local` command you will need to have either previously run with the `--update-env flag` or provide your own .env files.
 
 #### For developers that cannot run the workspace setup script or wish to only run SEDS see steps below.
+
 If you do not set don't have yarn, nvm, or java installed, see [Requirements](#requirements)
 
 Ensure you either have a 1Password account and have 1Password CLI installed. Alternatively, reach out to the team for an example of .env files.
@@ -134,20 +137,20 @@ export function listUsers() {
 
 ### Adding New Forms (quarterly)
 
-1. If necessary, create a new form template for the year in [src/database/initial_data_load/](src/database/initial_data_load/)
+1. If necessary, create a new form template for the year in [services/database/data/initial_data_load/](services/database/data/initial_data_load/)
    1. Example: `form_questions_2022.json`
-2. Add the new form to seed > form-questions > sources in [services/data-deployment/serverless.yml](./services/data-deployment/serverless.yml)
+2. Add the new form to seed > form-questions > sources in [services/database/handlers/seed](./services/database/handlers/seed/)
 
    Example:
 
    ```form-questions:
     table: ${self:custom.stage}-form-questions
     sources:
     [
-    ../../src/database/initial_data_load/form_questions_2022.json,
-    ../../src/database/initial_data_load/form_questions_2021.json,
-    ../../src/database/initial_data_load/form_questions_2020.json,
-    ../../src/database/initial_data_load/form_questions_2019.json,
+    data/initial_data_load/form_questions_2022.json,
+    data/initial_data_load/form_questions_2021.json,
+    data/initial_data_load/form_questions_2020.json,
+    data/initial_data_load/form_questions_2019.json,
     ]
    ```
 
@@ -182,7 +185,7 @@ found in [src/dms](src/dms).
 Validate json files against schema to ensure accuracy before each commit.
 
 - Schema Location: [src/database/schema/](src/database/schema/)
-- Initial Data Location: [src/database/initial_data_load](src/database/initial_data_load)
+- Initial Data Location: [services/database/data/initial_data_load](services/database/data/initial_data_load)
 
 1. Install AJV globally in your environment
    1. `npm install -g ajv-cli`
@@ -248,18 +251,19 @@ Pull requests are being accepted.
 
 ## Slack Webhooks:
 
-This repository uses 3 webhooks to publish to  3 different channels all in CMS Slack.
+This repository uses 3 webhooks to publish to 3 different channels all in CMS Slack.
 
 - SLACK_WEBHOOK: This pubishes to the `macpro-mdct-seds-alerts` channel. Alerts published there are for deploy or test failures to the `master`, `val`, or `production` branches.
 
 - INTEGRATIONS_SLACK_WEBHOOK: This is used to publish new pull requests to the `mdct-integrations-channel`
 
 - PROD_RELEASE_SLACK_WEBHOOK: This is used to publish to the `mdct-prod-releases` channel upon successful release of Seds to production.
 
-    - Webhooks are created by CMS tickets, populated into GitHub Secrets
+  - Webhooks are created by CMS tickets, populated into GitHub Secrets
 
 ## GitHub Actions Secret Management:
-- Secrets are added to GitHub secrets by GitHub Admins 
+
+- Secrets are added to GitHub secrets by GitHub Admins
 - Development secrets are maintained in a 1Password vault
 
 ## License
@@ -278,5 +282,4 @@ work worldwide through the CC0 1.0 Universal public domain dedication.
 
 ### Contributors
 
-This project made possible by the [Serverless Stack](https://serverless-stack.com/) and its authors/contributors. The extremely detailed tutorial, code examples, and serverless pattern is where this project started. 
-
+This project made possible by the [Serverless Stack](https://serverless-stack.com/) and its authors/contributors. The extremely detailed tutorial, code examples, and serverless pattern is where this project started.

services/database/README.md

@@ -10,7 +10,7 @@ year and the old files moved there for historical retention as needed.
 
 SEDS data is seeded to a developer's local dynamodb via the serverless.yml file in services > database.
 
-SEDS data is seeded to the AWS environments via the serverless.yml file in services > data-deployment.
+SEDS data is seeded to the AWS environments via the serverless.yml file in services > database > handlers > seed.
 
 When seeding data, it is important to remember that any records in a table with a matching partition key will be OVERWRITTEN as part of the update process.
 Any records without a matching partition key will be inserted into the table.
@@ -22,12 +22,12 @@ After the deployment has successfully occurred to both VAL and PROD, it is impor
 If the files are not removed from that yml file, they will be deployed again with the next merge and the records which the files relate to will be overwritten.
 
 1. Prepare necessary json files for deployment
-2. Add the files to services > data-deployment > serverless.yml to deploy them to AWS environments
+2. Add the files to services > database > handlers > seed to deploy them to AWS environments
 3. Add the files to services > database > serverless.yml to deploy them locally
 4. Merge to `master`
 5. Merge from `master` to `val`
 6. Merge from `val` to `production`
-7. Create a new branch and remove the loaded files from services > data-deployment > serverless.yml to stop the new data from being overwritten in AWS environments
+7. Create a new branch and remove the loaded files from services > database > handlers > seed to stop the new data from being overwritten in AWS environments
 8. Merge to `master`
 9. Merge from `master` to `val`
 10. Merge from `val` to `production`
@@ -46,7 +46,7 @@ The best practice is:
 1. Take the form_questions_XXXX.json file for the most recent year and copy it
 2. Rename it to the new year
 3. Find & Replace the most recent year with the new year
-4. Add the new json file to the serverless.yml files in services > database and services > data-deployment
+4. Add the new json file to the serverless.yml files in services > database and the handler in services > database > handlers > seed
 5. Remove the load of the most recent year's form_questions_XXXX.json file (optional)
 6. Archive the most recent year's form_questions_XXXX.json file (optional)