Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reconfigurations of decision engine cause short lived google credentials not to be renewed #370

Open
StevenCTimm opened this issue Sep 29, 2021 · 0 comments
Labels
operations Issue filed by Operations team

Comments

@StevenCTimm
Copy link
Contributor

I observed the following:

  1. I reconfigured the decision engine with a number of glideins to google cloud active at the time.
  2. as a result the GceAuthFile string changed to a different hash, but the already-submitted glideins
    did not change the GceAuthFile
  3. I observed that the old cached credential was no longer getting renewed, thus after a while causing
    the google cloud glideins to hold.. although apparently only at completion. i.e. the credential that
    the GceGahp uses to poll the cloud is OK, but the individual GceAuthFile used to submit and clean up afterwards is not.

This is something that in theory can happen with any credential but is much more likely to be visible
with Google credentials because they expire after 1/2 hour. We have seen NERSC jobs be caught out
with an old ssh key pair before too.

@StevenCTimm StevenCTimm added the operations Issue filed by Operations team label Oct 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
operations Issue filed by Operations team
Projects
None yet
Development

No branches or pull requests

1 participant