Keda Istio and CA Bundle issues

[1it]

I'm currently working on Keda implementation in our org.
We do use Istio with mTLS enforced as security requirement, so it's not an option to disable Istio sidecar injection.
I've managed to apply a workaround with Keda-only ports exclusion:

podAnnotations:
  # -- Pod annotations for KEDA operator
  keda:
    traffic.sidecar.istio.io/excludeInboundPorts: "9666"
    traffic.sidecar.istio.io/excludeOutboundPorts: "9443,6443"
  # -- Pod annotations for KEDA Metrics Adapter
  metricsAdapter:
    traffic.sidecar.istio.io/excludeInboundPorts: "6443"
    traffic.sidecar.istio.io/excludeOutboundPorts: "9666,9443"
  # -- Pod annotations for KEDA Admission webhooks
  webhooks:
    traffic.sidecar.istio.io/excludeInboundPorts: "9443"
    traffic.sidecar.istio.io/excludeOutboundPorts: "9666,6443"

That allow Keda to communicate internally with it's own mTLS which is fine for the security perspective (connection remain encrypted) at the same time Keda components can properly communicate with other mesh services (like Prometheus).

Another thing that is remaining is communication with Kube API. As I understand, Keda Metrics Adapter is registred as APIService
, so Kube API is trying to communicate with it using it's own TLS certificate which is not recognised by Keda (by default). As a result I see these errors in Metrics Adapter logs:

E0626 15:32:14.332513       1 authentication.go:73] "Unable to authenticate the request" err="verifying certificate SN=111689421568838424618468378762933967135546290727, SKID=F8:61:82:02:BA:27:32:EE:0F:CA:F9:E8:F0:D1:3D:93:BE:E5:24:CD, AKID=76:45:00:B0:D2:7B:DF:04:FA:92:B3:EC:04:4F:4B:01:1A:C1:27:D3 failed: x509: certificate signed by unknown authority"

Also the APIService is getting 401 as Keda rejecting the request.

After spending some time I've applied a workaround to inject K8s CA to kedaorg-certs, that helped:

K8S_CA=$(kubectl -n keda get cm kube-root-ca.crt -o jsonpath='{.data.ca\.crt}')
KEDA_CA=$(kubectl -n keda get secret kedaorg-certs -o jsonpath='{.data.ca\.crt}' | base64 -d)
MERGED_CA=$(echo -e "${KEDA_CA}\n${K8S_CA}" | base64)

kubectl -n keda patch secret kedaorg-certs \
   	-p "{\"data\":{\"ca.crt\":\"${MERGED_CA}\"}}" --type=merge

Later on, I've made practically the same through initContainer.
NOTE: We use cert-manager in our cluster, so I ditched the one in Keda helm-chart.
That's kind of OK, but I'm wondering is there any other way to deal with CA certificate injection? Can I just provide a directory containing all the CA files?

[nickmhankins]

We are running into this same issue. There is an option to add your own CA files to the operator's trust store, however it doesn't seem to work for us.

We're adding it with kustomize:

- target:
    group: apps
    version: v1
    kind: Deployment
    name: keda-operator
    namespace: keda
  patch: |-
    - op: add
      path: /spec/template/spec/containers/0/volumeMounts/1
      value:
        name: ca
        mountPath: /custom/ca
    - op: add
      path: /spec/template/spec/volumes/1
      value: 
        name: ca
        secret:
          secretName: ca

According to the source here, it's supposed to log the new certificates that it's adding but I don't see that in any of the logs.

[1it]

We are running into this same issue. There is an option to add your own CA files to the operator's trust store, however it doesn't seem to work for us.

We're adding it with kustomize:

- target:
    group: apps
    version: v1
    kind: Deployment
    name: keda-operator
    namespace: keda
  patch: |-
    - op: add
      path: /spec/template/spec/containers/0/volumeMounts/1
      value:
        name: ca
        mountPath: /custom/ca
    - op: add
      path: /spec/template/spec/volumes/1
      value: 
        name: ca
        secret:
          secretName: ca

According to the source here, it's supposed to log the new certificates that it's adding but I don't see that in any of the logs.

As I understand, that one is meant for client connections only (for scalers). So it's not going to work if you want to use Keda's components internal communication.

[nickmhankins]

Well....that is disappointing