OpenSearch Domain capacity now configurable #58

chelma · 2023-06-01T16:57:13Z

Description

Made the OpenSearch Domain used for SPI data configurable via the command-line.
By default, the user gets a minimal Capture Node ASG and OpenSearch Domain. If they specify non-default values, they get a cluster that should be capable of servicing their indicated traffic load. If they re-deploy after setting non-default values, the previously configured values are re-used unless they provide new values.
The user can change the individual capacity options and the code will gracefully pull the other, unspecified values if they exist.

Issues

Size OpenSearch Domain based on Expected traffic #56

Testing

Added Unit Tests
Tested the behavior manually using the CLI. Confirmed it correctly followed the constraints we imposed with our capacity plan for a few scenarios as a supplement to the extensive unit testing. Pasted below is the CLI output and cluster capacity configuration after deployment.

(.venv) chelma@3c22fba4e266 aws-aio % ./manage_arkime.py create-cluster --name MyCluster3 --expected-traffic 0.01 --spi-days 30 --replicas 1
2023-06-01 16:55:46 - Debug-level logs save to file: /Users/chelma/workspace/Arkime/aws-aio/manage_arkime.log
2023-06-01 16:55:46 - Using AWS Credential Profile: default
2023-06-01 16:55:46 - Using AWS Region: default from AWS Config settings
2023-06-01 16:55:47 - Executing command: deploy MyCluster3-CaptureBucket MyCluster3-CaptureNodes MyCluster3-CaptureVPC MyCluster3-OSDomain MyCluster3-ViewerNodes
2023-06-01 16:55:47 - NOTE: This operation can take a while.  You can 'tail -f' the logfile to track the status.
2023-06-01 17:15:28 - Deployment succeeded

-----------------------------------------------------------

{"busArn":"arn:aws:events:us-east-2:968674222892:event-bus/MyCluster3CaptureNodesClusterBus224EB36F","busName":"MyCluster3CaptureNodesClusterBus224EB36F","clusterName":"MyCluster3","vpceServiceId":"vpce-svc-0d6b05027bae05312","capacityPlan":{"captureNodes":{"instanceType":"m5.xlarge","desiredCount":1,"maxCount":2,"minCount":1},"captureVpc":{"numAzs":2},"ecsResources":{"cpu":3584,"memory":15360},"osDomain":{"dataNodes":{"count":2,"instanceType":"t3.small.search","volumeSize":100},"masterNodes":{"count":3,"instanceType":"m5.large.search"}}},"userConfig":{"expectedTraffic":0.01,"spiDays":30,"replicas":1}}

(.venv) chelma@3c22fba4e266 aws-aio % ./manage_arkime.py create-cluster --name MyCluster3 --expected-traffic 0.1
2023-06-01 17:16:35 - Debug-level logs save to file: /Users/chelma/workspace/Arkime/aws-aio/manage_arkime.log
2023-06-01 17:16:35 - Using AWS Credential Profile: default
2023-06-01 17:16:35 - Using AWS Region: default from AWS Config settings
2023-06-01 17:16:36 - Executing command: deploy MyCluster3-CaptureBucket MyCluster3-CaptureNodes MyCluster3-CaptureVPC MyCluster3-OSDomain MyCluster3-ViewerNodes
2023-06-01 17:16:36 - NOTE: This operation can take a while.  You can 'tail -f' the logfile to track the status.
2023-06-01 17:45:43 - Deployment succeeded
-----------------------------------------------------------

{"busArn":"arn:aws:events:us-east-2:968674222892:event-bus/MyCluster3CaptureNodesClusterBus224EB36F","busName":"MyCluster3CaptureNodesClusterBus224EB36F","clusterName":"MyCluster3","vpceServiceId":"vpce-svc-0d6b05027bae05312","capacityPlan":{"captureNodes":{"instanceType":"m5.xlarge","desiredCount":1,"maxCount":2,"minCount":1},"captureVpc":{"numAzs":2},"ecsResources":{"cpu":3584,"memory":15360},"osDomain":{"dataNodes":{"count":2,"instanceType":"r6g.large.search","volumeSize":1024},"masterNodes":{"count":3,"instanceType":"m6g.large.search"}}},"userConfig":{"expectedTraffic":0.1,"spiDays":30,"replicas":1}}

(.venv) chelma@3c22fba4e266 aws-aio % ./manage_arkime.py create-cluster --name MyCluster3
2023-06-01 17:46:55 - Debug-level logs save to file: /Users/chelma/workspace/Arkime/aws-aio/manage_arkime.log
2023-06-01 17:46:55 - Using AWS Credential Profile: default
2023-06-01 17:46:55 - Using AWS Region: default from AWS Config settings
2023-06-01 17:46:56 - Executing command: deploy MyCluster3-CaptureBucket MyCluster3-CaptureNodes MyCluster3-CaptureVPC MyCluster3-OSDomain MyCluster3-ViewerNodes
2023-06-01 17:46:56 - NOTE: This operation can take a while.  You can 'tail -f' the logfile to track the status.
2023-06-01 17:47:24 - Deployment succeeded

-----------------------------------------------------------

{"busArn":"arn:aws:events:us-east-2:968674222892:event-bus/MyCluster3CaptureNodesClusterBus224EB36F","busName":"MyCluster3CaptureNodesClusterBus224EB36F","clusterName":"MyCluster3","vpceServiceId":"vpce-svc-0d6b05027bae05312","capacityPlan":{"captureNodes":{"instanceType":"m5.xlarge","desiredCount":1,"maxCount":2,"minCount":1},"captureVpc":{"numAzs":2},"ecsResources":{"cpu":3584,"memory":15360},"osDomain":{"dataNodes":{"count":2,"instanceType":"r6g.large.search","volumeSize":1024},"masterNodes":{"count":3,"instanceType":"m6g.large.search"}}},"userConfig":{"expectedTraffic":0.1,"spiDays":30,"replicas":1}}

License

I confirm that this contribution is made under an Apache 2.0 license and that I have the authority necessary to make this contribution on behalf of its copyright owner.

Signed-off-by: Chris Helma <[email protected]>

awick · 2023-06-01T17:23:51Z

README.md


 ```
-./manage_arkime.py create-cluster --name MyCluster --expected-traffic 10
+./manage_arkime.py create-cluster --name MyCluster --expected-traffic 1 --spi-days 30 --replicas 2


I'm afraid of copy paste, lets make this replicas 1

awick · 2023-06-01T17:26:41Z

manage_arkime.py

@@ -58,15 +58,27 @@ def destroy_demo_traffic(ctx):
 @click.option(
    "--expected-traffic", 
    help=("The amount of traffic, in gigabits-per-second, you expect your Arkime Cluster to receive."


to be more correct should be something like "The average amount of traffic, ..."

Will change

awick · 2023-06-01T17:35:39Z

manage_arkime/commands/create_cluster.py

-def cmd_create_cluster(profile: str, region: str, name: str, expected_traffic: float):
+class MustProvideAllParams(Exception):
+    def __init__(self):
+        super().__init__("If you specify one of the optional capacity parameters, you must specify all of them.")


why? is this because create vs update which we will have later?

If keeping this, 3 changes

remove the stack trace, we shouldn't show stack traces when its a command line arg issue, it confuses people

the error message should include ALL the ones required OR the ones missing. (Guessing ALL is easier)

the "default" print out of help is confusing, since there aren't defaults i guess, or only defaults when none are set?

It has to do with how we're storing the application state in SSM for the cluster and its capacity. Definitely possible for this constraint to be loosened but it's just more code that needs to be written.

The short version is that I want our capacity plan to be stable between deployments even if the underlying code for the capacity plan changes until the user specifically indicates they want to update the capacity (users should not be "surprised" by an infrastructure change). This means we need to store the calculated capacity plan rather than just the inputs to the capacity plan (expected traffic, SPI days, replicas). In this scenario, we don't actually need to store those inputs as long as the user provide all three of them when they want a change. This is the approach I took for the first pass.

If we store the inputs in SSM as well, it will enable us to accept a single one of the parameters and generate a capacity plan using the existing values of the other ones, but it makes the code more complex.

Let's talk it through.

@awick and I talked it through and agreed the path forward is to store the inputs as well and accept the additional complexity. Will update.

Signed-off-by: Chris Helma <[email protected]>

awick

LGTM - Still deleting and redeploying everything.

At some point we should put delete protection on OS and S3

chelma · 2023-06-02T13:45:13Z

At some point we should put delete protection on OS and S3

Good idea; made a new issue for that: #59

OpenSearch Domain capacity now configurable

296736d

Signed-off-by: Chris Helma <[email protected]>

chelma added the Capture Configurability Work to make capture configurable and handle changes in configuration label Jun 1, 2023

chelma requested review from awick and 31453 June 1, 2023 16:57

awick requested changes Jun 1, 2023

View reviewed changes

Made capacity options individually changable

0a15045

Signed-off-by: Chris Helma <[email protected]>

awick approved these changes Jun 2, 2023

View reviewed changes

chelma merged commit 77a8e5d into main Jun 2, 2023

chelma deleted the capture-cap-3 branch June 2, 2023 13:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenSearch Domain capacity now configurable #58

OpenSearch Domain capacity now configurable #58

chelma commented Jun 1, 2023 •

edited

Loading

awick Jun 1, 2023

chelma Jun 1, 2023

awick Jun 1, 2023

chelma Jun 1, 2023

awick Jun 1, 2023

chelma Jun 1, 2023

chelma Jun 1, 2023

awick left a comment

chelma commented Jun 2, 2023

OpenSearch Domain capacity now configurable #58

OpenSearch Domain capacity now configurable #58

Conversation

chelma commented Jun 1, 2023 • edited Loading

Description

Issues

Testing

License

awick Jun 1, 2023

Choose a reason for hiding this comment

chelma Jun 1, 2023

Choose a reason for hiding this comment

awick Jun 1, 2023

Choose a reason for hiding this comment

chelma Jun 1, 2023

Choose a reason for hiding this comment

awick Jun 1, 2023

Choose a reason for hiding this comment

chelma Jun 1, 2023

Choose a reason for hiding this comment

chelma Jun 1, 2023

Choose a reason for hiding this comment

awick left a comment

Choose a reason for hiding this comment

chelma commented Jun 2, 2023

chelma commented Jun 1, 2023 •

edited

Loading