Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Make init-job run as hook only on install #450

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jonasbadstuebner
Copy link

There are multiple issues stating the init-job is not run as expected

This is because there is a deadlock with the CRDB-StatfulSet requiring the init-job to run, which is only ran by Helm when the StatefulSet is considered ready

The optional use of the --wait of helm install is causing differing observations

This PR attempts to fix the problem by using the Job as plain Job instead of hook when the Chart is initially installed

There are multiple issues stating the init-job is not run as expected

This is because there is a deadlock with the CRDB-StatfulSet requiring the init-job to run, which is only ran by Helm when the StatefulSet is considered ready

The optional use of the --wait of `helm install` is causing differing observations

This PR attempts to fix the problem by using the Job as plain Job instead of hook when the Chart is initially installed
@jonasbadstuebner
Copy link
Author

⚠ Not tested yet ⚠
The rendering looks good - I am just not sure if this behaves as I think it does.
If someone else wants to test it, feel free.

Expected behavior:
On install (--wait given or not does not matter):

1. The CRDB-StatefulSet and the init-Job are deployed almost at the same time
2. The init-job initializes the DB - once it is ready for initialization - and finishes
3. The `helm install` command succeeds
4. The Job is eventually auto-deleted by Kubernetes due to the `ttlSecondsAfterFinished` being 0
   (this unblocks the `helm upgrade`)

On Upgrade:

(0. The init-Job is removed, if present from a previous `helm upgrade` - should not be
    there anymore from the `helm install` step)
1. The CRDB-StatefulSet is upgraded
2. The init-Job is run after the StatefulSet is upgraded and ready
3. The `helm upgrade` command succeeds

@jonasbadstuebner
Copy link
Author

The open issues related to the init-job deadlock:
For sure:
#69
#287
#402

Probably:
#345 (same error message)

The closed issues related to the init-job deadlock:
#195
#234
#389 (I'd argue this is the same topic, even though the person put the blame on terraform)
#400
#410

This just shows that the topic is highly confusing, as people expect a simple helm install to work. I really hope my solution works.
Otherwise, I would love to see a big note in the README at least, to not use --wait on helm install.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant