Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include more annotations on Helm releases e.g. Sample metadata #18

Open
craigwalton-dsit opened this issue Dec 19, 2024 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@craigwalton-dsit
Copy link
Collaborator

Some users frame their evals as a single Task but with many Samples. We currently only include the task name in the annotations (which end up as annotations on pods etc.). This can make it hard when you see a failed Pod to understand what it relates to.

This is because this is what Inspect provides us with. But it also provides the Sample metadata.

2 ideas:

  1. Suggest that Inspect adds a sample id to the inspect SandboxEnvironment interface (4 methods). This is quite a faff as its a change in the core of inspect which lots of environment providers depend on. And sample id might not be useful to some people.
  2. Read an optional HELM_ANNOTATIONS value from the Sample.metadata dict (which Inspect gives us already). This would itself be a dict and we'd put it all into the release annotations so you'd see any values you set there alongside the inspectTaskName.
Screenshot 2024-12-16 at 18 01 32

We could log this additional metadata when we have a Helm install error for example.

You could also use this for injecting more useful info in, such as the eval set name or other identifiers.

Note: We'd have to sort out the fact that the metadata parameter is typed as dict[str, str] whereas I think it ought to be dict[str, Any] as per Sample.metadata. If it really can't be dict[str, Any] then we'd have to define a convention like HELM_ANNOTATION_*.

@craigwalton-dsit craigwalton-dsit added the enhancement New feature or request label Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant