Checkpoint: Arkime Config details uploaded to Parameter Store

[chelma]

Description

• Checkpoint commit. Still need to update the in-Container behavior to use the config stored in S3, remove the old config behavior, and update destroy-cluster to remove the config bucket.
• This update is a quick PR to cover the behavior to upload the Arkime Config's details to Parameter Store during create-cluster

Tasks

• create-cluster/destroy-cluster implement Config RFC #83

Testing

• Added unit tests
• Ran the create-cluster command against my account twice to demonstrate the code's ability to create/upload the config on the first pass and skip that logic on the second pass once it saw it was already uploaded.

(.venv) chelma@3c22fba4e266 aws-aio % ./manage_arkime.py create-cluster --name MyCluster3 --preconfirm-usage
2023-07-19 15:09:32 - Debug-level logs save to file: /Users/chelma/workspace/Arkime/aws-aio/manage_arkime/manage_arkime.log
2023-07-19 15:09:32 - Using AWS Credential Profile: default
2023-07-19 15:09:32 - Using AWS Region: default from AWS Config settings
2023-07-19 15:09:34 - Usage report:
Arkime Metadata:
    Session Retention [days]: 30
    User History Retention [days]: 120
Capture Nodes:
    Max Count: 2
    Desired Count: 1
    Min Count: 1
    Type: m5.xlarge
OpenSearch Domain:
    Master Node Count: 3
    Master Node Type: m6g.large.search
    Data Node Count: 2
    Data Node Type: r6g.large.search
    Data Node Volume Size [GB]: 1024
S3:
    PCAP Retention [days]: 30

2023-07-19 15:09:34 - Ensuring Arkime Config dir exists for cluster: MyCluster3
2023-07-19 15:09:34 - Arkime Config dir exists at: /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3
2023-07-19 15:09:34 - Copying default Arkime Config to dir: /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3
2023-07-19 15:09:34 - Cluster config directory not empty; skipping copy
2023-07-19 15:09:34 - Determining the status of S3 bucket: arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster3
2023-07-19 15:09:35 - S3 Bucket arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster3 already exists; no work needed
2023-07-19 15:09:35 - Uploading Arkime config for Capture Nodes...
2023-07-19 15:09:35 - Turning Capture configuration at /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3/capture into tarball at /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3/capture.t
gz
2023-07-19 15:09:35 - Uploading config tarball to S3 bucket: arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster3
2023-07-19 15:09:36 - Uploading Arkime config for Viewer Nodes...
2023-07-19 15:09:37 - Turning Viewer configuration at /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3/viewer into tarball at /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3/viewer.tgz
2023-07-19 15:09:37 - Uploading config tarball to S3 bucket: arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster3
2023-07-19 15:09:39 - Executing command: deploy MyCluster3-CaptureBucket MyCluster3-CaptureNodes MyCluster3-CaptureVPC MyCluster3-OSDomain MyCluster3-ViewerNodes
2023-07-19 15:09:39 - NOTE: This operation can take a while.  You can 'tail -f' the logfile to track the status.
2023-07-19 15:12:12 - Deployment succeeded

(.venv) chelma@3c22fba4e266 aws-aio % ./manage_arkime.py create-cluster --name MyCluster3 --preconfirm-usage
2023-07-19 14:52:24 - Debug-level logs save to file: /Users/chelma/workspace/Arkime/aws-aio/manage_arkime/manage_arkime.log
2023-07-19 14:52:24 - Using AWS Credential Profile: default
2023-07-19 14:52:24 - Using AWS Region: default from AWS Config settings
2023-07-19 14:52:26 - Usage report:
Arkime Metadata:
    Session Retention [days]: 30
    User History Retention [days]: 120
Capture Nodes:
    Max Count: 2
    Desired Count: 1
    Min Count: 1
    Type: m5.xlarge
OpenSearch Domain:
    Master Node Count: 3
    Master Node Type: m6g.large.search
    Data Node Count: 2
    Data Node Type: r6g.large.search
    Data Node Volume Size [GB]: 1024
S3:
    PCAP Retention [days]: 30

2023-07-19 14:52:26 - Ensuring Arkime Config dir exists for cluster: MyCluster3                                                                                                                           2023-07-19 14:52:26 - Arkime Config dir exists at: /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3
2023-07-19 14:52:26 - Copying default Arkime Config to dir: /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3
2023-07-19 14:52:26 - Cluster config directory not empty; skipping copy
2023-07-19 14:52:27 - Determining the status of S3 bucket: arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster3
2023-07-19 14:52:27 - S3 Bucket arkimeconfig-968674222892-us-east-2-mycluster3 already exists; no work needed
2023-07-19 14:52:27 - Uploading Arkime config for Capture Nodes...
2023-07-19 14:52:28 - Config has been uploaded previously; skipping
2023-07-19 14:52:28 - Uploading Arkime config for Viewer Nodes...
2023-07-19 14:52:28 - Config has been uploaded previously; skipping
2023-07-19 14:52:29 - Executing command: deploy MyCluster3-CaptureBucket MyCluster3-CaptureNodes MyCluster3-CaptureVPC MyCluster3-OSDomain MyCluster3-ViewerNodes
2023-07-19 14:52:29 - NOTE: This operation can take a while.  You can 'tail -f' the logfile to track the status.
2023-07-19 14:55:01 - Deployment succeeded

License

I confirm that this contribution is made under an Apache 2.0 license and that I have the authority necessary to make this contribution on behalf of its copyright owner.

[awick]

LGTM