CLOUPD-193335: Deletion Protection Serverless Private Endpoints

[josvazg]

All Submissions:

• Have you signed our CLA?
• Put closes #XXXX in your comment to auto-close the issue that your PR fixes (if there is one).
• Update docs/release-notes/release-notes.md if your changes should be included in the release notes for the next release.

[github-actions]

https://app.codecov.io/github/mongodb/mongodb-atlas-kubernetes/commit/1a5c121a51a1d1709006d4c4ea7670b4d360458a

[josvazg]

Still a draft as it should go after bugfix #1082

[josvazg]

debugging

[josvazg]

Even after a rebase the issues persist. Seems main is not affected.
Debugging the database user integration test failure first.

[josvazg]

Debugging some panics in CI tests.

[josvazg]

I think there are more usages of .Spec.DeploymentSpec. leading to flaky CI failures. And there seems to be a persistent helm upgrade issue. Still do not understand how my changes are related to that, but will keep on debugging and sending fixes in this branch.

[josvazg]

@igor-karpukhin gave me the right hint here.
Somehow my code must be saving the converted deployment to Kubernetes, changing the original spec. We do the conversion here:
https://github.com/mongodb/mongodb-atlas-kubernetes/blob/CLOUDP-193335/deletion-protection-spe/pkg/controller/atlasdeployment/atlasdeployment_controller.go#L165

if deployment.IsLegacyDeployment() {
		if err := ConvertLegacyDeployment(&deployment.Spec); err != nil {
			result = workflow.Terminate(workflow.Internal, err.Error())
			log.Errorw("failed to convert legacy deployment", "error", err)
			return result.ReconcileResult(), nil
		}
		deployment.Spec.DeploymentSpec = nil
	}

But the key point is that conversion must never be seen in Kubernetes. It is ONLY used in memory for convenience so we always handle only serverless deployments and advanced. aAs old deployments get converted to their advanced equivalent.

My code somewhere must be saving the converted spec to Kubernetes, so the rest of the test code and even Helm fail as they assume a default configuration which is missing, as it was converted to advanced. This conversion is wrong, it should not have happened.

[josvazg]

Seems all issues, including the helm error and the panics were due to ApplyLastConfigApplied adding the annotation and updating the converted deployment in Kubernetes, instead of the original deployment without conversion. This update was not expected and all tests break accordingly.

The fix is to actually make sure ApplyLastConfigApplied is called with the original (non converted) deployment object.

[josvazg]

2 errors remain that might not be flaky:

Summarizing 2 Failures:
  [FAIL] AtlasDeployment Create the advanced deployment with enabled autoscaling [It] Should Succeed [int, AtlasDeployment, deployment-non-backups]
  /home/runner/work/mongodb-atlas-kubernetes/mongodb-atlas-kubernetes/test/int/deployment_test.go:209
  [FAIL] AtlasDeployment Create the advanced deployment & change the InstanceSize [It] Should Succeed [int, AtlasDeployment, deployment-non-backups]
  /home/runner/work/mongodb-atlas-kubernetes/mongodb-atlas-kubernetes/test/int/deployment_test.go:868

Debugging those now.

[josvazg]

2 errors remain that might not be flaky:

Summarizing 2 Failures:
  [FAIL] AtlasDeployment Create the advanced deployment with enabled autoscaling [It] Should Succeed [int, AtlasDeployment, deployment-non-backups]
  /home/runner/work/mongodb-atlas-kubernetes/mongodb-atlas-kubernetes/test/int/deployment_test.go:209
  [FAIL] AtlasDeployment Create the advanced deployment & change the InstanceSize [It] Should Succeed [int, AtlasDeployment, deployment-non-backups]
  /home/runner/work/mongodb-atlas-kubernetes/mongodb-atlas-kubernetes/test/int/deployment_test.go:868

Debugging those now.

The problem was due to handleAutoscaling making changes in the Kubernetes spec for advanced deployment here:
https://github.com/mongodb/mongodb-atlas-kubernetes/blob/main/pkg/controller/atlasdeployment/advanced_deployment.go#L192

func handleAutoscaling(ctx *workflow.Context, desiredDeployment *mdbv1.AdvancedDeploymentSpec, currentDeployment *mongodbatlas.AdvancedCluster) error {
	...
	syncInstanceSize := func(s *mdbv1.Specs, as *mdbv1.AdvancedAutoScalingSpec) error {
		if s != nil {
			...

			if isInstanceSizeTheSame(currentDeployment, size) {
				size = ""
			}

			s.InstanceSize = size

Those changes were being saved to the Kubernetes state when updating the deployment after (re)setting the previous config here:
https://github.com/mongodb/mongodb-atlas-kubernetes/blob/main/pkg/controller/customresource/protection.go#L50

func ApplyLastConfigApplied(ctx context.Context, resource mdbv1.AtlasCustomResource, k8sClient client.Client) error {
...
	return k8sClient.Update(ctx, resource, &client.UpdateOptions{})
}

This was only happening for advanced cluster, because I was NOT making a copy of the Kubernetes deployment structure unless the conversion was actually needed.

The fix was to simply always make the converted deployment variable a separate copy. That way, the original deployment will be unchanged, and no changes in the reconcile loop will be reflected back into the Kubernetes etcd status.

A separate matter is whether or not those ephemeral changes to the deployment struct make actual sense, but that is out of the scope of this PR.

[josvazg]

Current persistent failures are in encryption-at-rest a well know flaky test that, AFAIK, is totally unrelated to the changes here:

[FAIL] Encryption at REST test Encryption at rest for AWS, GCP, Azure [It] Test[encryption-at-rest-aws]: Can add Encryption at Rest to AWS project [encryption-at-rest, encryption-at-rest-aws]

Will be debugging it now.

[josvazg]

Current persistent failures are in encryption-at-rest a well know flaky test that, AFAIK, is totally unrelated to the changes here:

[FAIL] Encryption at REST test Encryption at rest for AWS, GCP, Azure [It] Test[encryption-at-rest-aws]: Can add Encryption at Rest to AWS project [encryption-at-rest, encryption-at-rest-aws]

Will be debugging it now.

Created a follow up issue for that.

Now the test passed, we are ready for review.

[igor-karpukhin]

@josvazg this is done because of the way we operate with autoscaling. For example, if you set instanceSize to be M10 and enabled autoscaling from M10 to M40, later, after cluster is autoscaled to M30, if you try to modify the AtlasDeployment resource, the instanceSize is going to be M10, and we send this M10 to Atlas. Which is wrong. That's why if instance size is equal to the one in Atlas and autoscaling is enabled, we remove this field before sending PATCH request.

[josvazg]

@josvazg this is done because of the way we operate with autoscaling. For example, if you set instanceSize to be M10 and enabled autoscaling from M10 to M40, later, after cluster is autoscaled to M30, if you try to modify the AtlasDeployment resource, the instanceSize is going to be M10, and we send this M10 to Atlas. Which is wrong. That's why if instance size is equal to the one in Atlas and autoscaling is enabled, we remove this field before sending PATCH request.

Thanks for the explanation. Then it means that ephemeral changes to the k8s struct is like an intermediate representation to ease syncing with Atlas. Makes sense.

[josvazg]

Rebased

[helderjs]

I noticed something strange with your rebase. Changes already in the main branch are showing as new in your PR, I pointed out 2 but there are more, even entire files.

[josvazg]

I noticed something strange with your rebase. Changes already in the main branch are showing as new in your PR, I pointed out 2 but there are more, even entire files.

Thanks for the heads up, checking...