Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When no more jobs are idle on a particular group, the decision engine stops renewing the credentials #446

Open
StevenCTimm opened this issue Dec 7, 2022 · 0 comments
Assignees
Labels
bug Something isn't working operations Issue filed by Operations team

Comments

@StevenCTimm
Copy link
Contributor

The following issue has been known for quite some time, but now with scitokens that last just a couple hours it has become critical.

When no jobs are idle for a particular group in the decision engine, then it no longer advertises any glideclient classads
to the factory. Since glideclient classads are the means by which credentials are passed to the factory, it means
that the factory no longer has updated credentials to contact the remote resource, and therefore loses contact with any
glideins that may be still active.

This is also the case when the frontend is reconfigured but there are still glideins from the previous config that are active.. credentials don't get renewed for those either.

This is now a serious operational issue, we need to do something.

This was observed in decision engine 1.7.4 but it affects all versions because the behavior is in glideinwms/glide_frontend_element.py and the particular advertising logic of that routine hasn't changed since the DE was written.

@StevenCTimm StevenCTimm added bug Something isn't working operations Issue filed by Operations team labels Dec 7, 2022
@goodenou goodenou self-assigned this Dec 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working operations Issue filed by Operations team
Projects
None yet
Development

No branches or pull requests

2 participants