Using kfp v2.0+ not able to add pod labels in kubeflow pipeline workflow

[yifengwang66]

Discussed in #11014

^{Originally posted by emilyyujieli July 17, 2024}
Environment
KFP version: 2.2.0
kubeflow verison: 1.8
KFP SDK version: 2.7.0

Background
We are trying to add istio sidecar in kubeflow pipeline pod then we can use service mesh to do authentication due to project requirement.

Issue
If we use kfp v1 and add parameter kubernetes.add_pod_label(task=task,label_key="sidecar.istio.io/inject",label_value="true") in the code, then use KFP SDK v1 to compile the script. label can be injected in related workflow and run successfully.
However if we add the same parammeter in kfp v2 script and use kfp SDK v2 to compile, l can see the label in compiled file showed as below

platform_spec:
  platforms:
    kubernetes:
      deploymentSpec:
        executors:
          exec-load:
            podMetadata:
              labels:
                sidecar.istio.io/inject: 'true'

However when l try to upload the compiled file and execute it via kubeflow pipeline, the label doesn't be injected in related workflow.

Question
If this parameter kubernetes.add_pod_label still available in KFP v2? How can l add label in kubeflow pipeline workflow yaml if using kfp v2 since not able to find any information on kubeflow official doucument.

Regards & thanks!