[Metrics] Add original job definition name to pipeline metrics tags

[Emile-Filteau]

When tracking pipeline metrics, we currently can only filter down by pipeline name, executor name and job id. There is currently no way of filtering them by job name.

I would even argue that we should maybe remove the jobId from the tags, because it has a very high cardinality.

I'm not sure i'm happy with adding this field right on the QueueItem. I wonder if we should have some sort of tags dictionary instead containing both the executor name and the job definition name?

Another avenue would be to leverage the job labels, but i feel like this might be mixing up some concepts.

[isra17]

Another avenue would be to leverage the job labels, but i feel like this might be mixing up some concepts.

I think it could make sense no? Kubernetes does it for a few fields. You can namespace label key such as saturn/job-definition-name=some-job-name

[Emile-Filteau]

My main concern is that we already have all the job definition labels on it.

You would suggest adding a special key to that labels dict that we know we can recover later?

[isra17]

My main concern is that we already have all the job definition labels on it.

You would suggest adding a special key to that labels dict that we know we can recover later?

Could just simply add the label to the template here: https://github.com/Flared/saturn/blob/main/src/saturn_engine/worker_manager/config/declarative_job_definition.py#L34 ?

[Emile-Filteau]

Updated the PR, i changed the approach so that we now simply add the job definition name as a default label on the QueueItem. Since we already collect all the labels in saturn.pipeline.message we will now also have the job name.

I also added the labels to saturn.pipeline.usage which was only adding the pipeline and executor as tags.

[isra17]

Nice, one comment, but lgtm otherwise!