Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

not waiting for Application resources to be created when using ApplicationSet causes a race condition #85

Closed
dedri-github opened this issue Jan 2, 2025 · 3 comments · May be fixed by #88
Closed

not waiting for Application resources to be created when using ApplicationSet causes a race condition #85

dedri-github opened this issue Jan 2, 2025 · 3 comments · May be fixed by #88

Comments

@dedri-github
Copy link

Hello,
When having a repository with ApplicationSets, it seems like argocd-diff-preview doesn't wait for Applications to be generated by the ApplicationSet controller after applying the yaml resources. Instead it continues immediately to retrieve the Application resources from the cluster, and there's a race condition between those stages:

  1. If at least one Application was created quickly enough between those steps and it's retrieved, the loop is allowed to wait for OutOfSync and more and more Applications are detected on each iteration.
  2. No Application was created fast enough between those two steps. In this case the loop is terminated and argocd-diff-preview says:
[2025-01-02T13:23:52Z INFO  argocd_diff_preview::extract] 🌚 Got all resources from 0 applications for dudu-base-branch

Here's how it looks in the logs. 0 Applications are detected on the base branch and 2 are initially detected on the target branch. On future iterations of the loop it detects more and more that are created by the ApplicationSet controller from the target branch:

[2025-01-02T13:23:51Z INFO  argocd_diff_preview::argocd] 🦑 Argo CD installed successfully
[2025-01-02T13:23:51Z INFO  argocd_diff_preview::extract] 🌚 Getting resources from base-branch
[2025-01-02T13:23:51Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl apply -f apps_base_branch.yaml
[2025-01-02T13:23:52Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl get applications -A -oyaml
[2025-01-02T13:23:52Z INFO  argocd_diff_preview::extract] 🌚 Got all resources from 0 applications for dudu-base-branch
[2025-01-02T13:23:52Z INFO  argocd_diff_preview::extract] 💾 Resources stored in: './output/base/<app_name>'
[2025-01-02T13:23:52Z INFO  argocd_diff_preview::extract] 🧼 Removing applications
[2025-01-02T13:23:52Z DEBUG argocd_diff_preview::extract] 🗑 Deleting ApplicationSets
[2025-01-02T13:23:52Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl delete applicationsets.argoproj.io --all -A
[2025-01-02T13:23:56Z DEBUG argocd_diff_preview::extract] 🗑 Deleted ApplicationSets
[2025-01-02T13:23:56Z DEBUG argocd_diff_preview::extract] 🗑 Deleting Applications
[2025-01-02T13:23:56Z DEBUG argocd_diff_preview::utils] Spawning command: kubectl delete applications.argoproj.io --all -A
[2025-01-02T13:24:01Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl get applications -A --no-headers
[2025-01-02T13:24:01Z INFO  argocd_diff_preview::extract] 🧼 Removed applications successfully
[2025-01-02T13:24:06Z INFO  argocd_diff_preview::extract] 🌚 Getting resources from target-branch
[2025-01-02T13:24:06Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl apply -f apps_target_branch.yaml
[2025-01-02T13:24:07Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl get applications -A -oyaml
[2025-01-02T13:24:07Z INFO  argocd_diff_preview::extract] ⏳ Waiting for 2 out of 2 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 180 seconds...
[2025-01-02T13:24:12Z INFO  argocd_diff_preview::extract] ⏳ Waiting for 78 out of 78 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 175 seconds...
[2025-01-02T13:24:17Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl get applications -A -oyaml

When running argo-diff-preview again with no change in the sources, no Application is detected at all, which shows the race condition more clearly:

[2025-01-02T13:28:28Z INFO  argocd_diff_preview::argocd] 🦑 Argo CD installed successfully
[2025-01-02T13:28:28Z INFO  argocd_diff_preview::extract] 🌚 Getting resources from base-branch
[2025-01-02T13:28:28Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl apply -f apps_base_branch.yaml
[2025-01-02T13:28:28Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl get applications -A -oyaml
[2025-01-02T13:28:28Z INFO  argocd_diff_preview::extract] 🌚 Got all resources from 0 applications for dudu-base-branch
[2025-01-02T13:28:28Z INFO  argocd_diff_preview::extract] 💾 Resources stored in: './output/base/<app_name>'
[2025-01-02T13:28:28Z INFO  argocd_diff_preview::extract] 🧼 Removing applications
[2025-01-02T13:28:28Z DEBUG argocd_diff_preview::extract] 🗑 Deleting ApplicationSets
[2025-01-02T13:28:28Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl delete applicationsets.argoproj.io --all -A
[2025-01-02T13:28:32Z DEBUG argocd_diff_preview::extract] 🗑 Deleted ApplicationSets
[2025-01-02T13:28:32Z DEBUG argocd_diff_preview::extract] 🗑 Deleting Applications
[2025-01-02T13:28:32Z DEBUG argocd_diff_preview::utils] Spawning command: kubectl delete applications.argoproj.io --all -A
[2025-01-02T13:28:37Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl get applications -A --no-headers
[2025-01-02T13:28:37Z INFO  argocd_diff_preview::extract] 🧼 Removed applications successfully
[2025-01-02T13:28:42Z INFO  argocd_diff_preview::extract] 🌚 Getting resources from target-branch
[2025-01-02T13:28:42Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl apply -f apps_target_branch.yaml
[2025-01-02T13:28:43Z DEBUG argocd_diff_preview::utils] Running shell command: kubectl get applications -A -oyaml
[2025-01-02T13:28:43Z INFO  argocd_diff_preview::extract] 🌚 Got all resources from 0 applications for dudu-target-branch
[2025-01-02T13:28:43Z INFO  argocd_diff_preview::extract] 💾 Resources stored in: './output/target/<app_name>'
@dedri-github dedri-github changed the title not waiting for Application resources to be created when using ApplicationSet not waiting for Application resources to be created when using ApplicationSet causes a race condition Jan 2, 2025
@dedri-github dedri-github mentioned this issue Jan 5, 2025
Open
@dedri-github
Copy link
Author

@dag-andersen , I opened PR #88 that resolves the issue for our setup.
I see that you already implemented a logic that uses argocd appset generate instead, so this PR reverts that.
It is up to you whether to use this PR or stay with your implementation.

Thanks

@dag-andersen
Copy link
Owner

Hi @dedri-github,
Thank you for showing interest in the project and highlighting this problem! 🚀 I didn’t encounter this issue during my testing, but I was aware there was a chance it could happen. This should be fixed as part of v0.0.29. Let me know if you experience any further issues.

@dedri-github
Copy link
Author

The issue seems to be fixed, thanks @dag-andersen!

@dag-andersen dag-andersen mentioned this issue Jan 7, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Logic that waits for ApplicationSet processing before continuing dedri-github/argocd-diff-preview
2 participants
@dedri-github @dag-andersen