[remote runtime] poll runtime info to wait until alive instead of using long timeout #4334
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rbren
reviewed
Oct 11, 2024
rbren
reviewed
Oct 11, 2024
openhands/runtime/remote/runtime.py
Outdated
| logger.info( | ||
| f'Waiting for runtime pod to be active. Current status: {pod_status}' | ||
| ) | ||
| if pod_status == 'Running': |
There was a problem hiding this comment.
I think there's a chance it could be Running but not Ready--in fact I'm pretty sure that's the case
We should maybe put Ready as a new pod_status, and then we wouldn't need to check /alive (since Ready waits for the /alive)
There was a problem hiding this comment.
Yep, this is exactly the reason why i keep the /alive call! I think having a Ready status (use the same probe we had in the cluster) will be the most ideal.
There was a problem hiding this comment.
runtime API side now returns Ready pod status when readiness probe gives true - this PR should be ready to merge!
Co-authored-by: Robert Brennan <[email protected]>
1 task
Co-authored-by: Xingyao Wang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
End-user friendly description of the problem this fixes or functionality that this introduces
Improve the reliability of Remote Runtime.
Give a summary of what the PR does, explaining any non-trivial design decisions
This PR use
/runtime/<runtime_id>endpoint to poll runtime pod info, and will start action execution once the pod is in 'Running' state. It will throw an error properly if the pod fails and/or is in a "Not Found" state after a fixed amount of retries.Need to wait until #4325 is merged
Link of any specific issues this addresses