svclb pod not returning SSL Certificates.

[murphye]

I am using k3d v4.2.0, but have narrowed down to this being a Klipper svclb issue. I am using the Istio proxy service, and port 80 is working fine. However when I enable SSL/TLS for routing to 443, I cannot connect properly because the SSL certificate is not being returned to the client.

I am starting my k3d cluster with this command:

k3d cluster create --registry-create --k3s-server-arg '--no-deploy=traefik' -p "9080:80@loadbalancer" -p "9443:43@loadbalancer" istio-workshop

If I connect to the istio-ingressgateway directly, it's fine. If I connect to svclb-istio-ingressgateway that is where the problem begins.

Connecting to svclb-istio-ingressgateway with openssl. No certificate returned. Error.

k port-forward svclb-istio-ingressgateway-xnxb4 7443:43 -n istio-system

openssl s_client -cipher ALL -servername istioinaction.io -connect localhost:7443
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 414 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Connecting to istio-ingressgateway with openssl. Certificate returned. Correct.

k port-forward istio-ingressgateway-5686db779c-z2hk7 7443:43 -n istio-system

openssl s_client -cipher ALL -servername istioinaction.io -connect localhost:7443
CONNECTED(00000003)
depth=0 CN = istioinaction.io
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = istioinaction.io
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = istioinaction.io
verify return:1
---
Certificate chain
 0 s:CN = istioinaction.io
   i:CN = istio-workshop-ca
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIQW8bMG/ndnqBqqT3ItMqukjANBgkqhkiG9w0BAQsFADAc
MRowGAYDVQQDExFpc3Rpby13b3Jrc2hvcC1jYTAeFw0yMTAyMjUxNzE2MzJaFw0z
MTAyMjMxNzE2MzJaMBsxGTAXBgNVBAMTEGlzdGlvaW5hY3Rpb24uaW8wggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCazmNTsvJa/yNcncyeN/V3HSCYU5p3
/vi38KdWiZXkFnaAXhdQtaKD/cOqzVPAWRm7hFVUnNYIgBXeYgsubUwjd9/pot12
u343pFYD+8BSZd0/dRUjLHi4R4wE2+GgX4u0uKgGupl4p7FMpIp0l0bknpIFxYVi
/RP3jnIli09YzHTdhtsY+b4iyl6XKhqOeKO0WqRnKLr6Z2PV/1U2xe+McB1Z9ELC
2bF9/d/wj/+hUrheS3EMMZxPgv4H/cXv5v8u5nskneWr1QSVxt6tXc1fAb5oVR/M
vrKZnxIMMez3AmB0gcJyGoLMBN6JUlmXLSKnCiN0KIMiiMrgjv6n5ej3AgMBAAGj
gY8wgYwwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEF
BQcDAjAdBgNVHQ4EFgQUNqSZ1h5/Pzfvgb0yz/GNm8sP/8QwHwYDVR0jBBgwFoAU
JSpD5fHAjP/YLycK4mABIsY/y/YwGwYDVR0RBBQwEoIQaXN0aW9pbmFjdGlvbi5p
bzANBgkqhkiG9w0BAQsFAAOCAQEAh58Osb17EpCc2+qbToMiE4uaFiWISPMva+MV
WGPRgwk26lKN8TA2rgxB65qTtxfZTtmoB55OWuKAIvzWrcNnPw4GzIIi8dhX7k9a
NVZlKBVqNXrk284uXXrqycXKFZyTcwVE0IALS4ckIrDREl5L+N/EoGsAukFWKxny
Oh2Qua/qUi8XFylN3Um919kQq2TCzZe2KtEA02I0WC2y6b+rNwEZgyOC9AxN3d7S
4+fU3bUAofEx27DC4aXj52GliTrQvMEeY2wT9k8Oxjs/t5kT7/uz7zxxPZ9A6OYJ
DFd/vIrk2FHlrznfkRYYKCxLhNnsdHY+J9paO/VF8GOhPbSpIQ==
-----END CERTIFICATE-----
subject=CN = istioinaction.io

issuer=CN = istio-workshop-ca

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1343 bytes and written 494 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---

Logs from svclb-istio-ingressgateway.

k logs svclb-istio-ingressgateway-xnxb4 -c lb-port-443 -n istio-system
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s 10.43.152.110/32 -p TCP --dport 443 -j DNAT --to 10.43.152.110:443
+ iptables -t nat -I POSTROUTING -d 10.43.152.110/32 -p TCP -j MASQUERADE
+ '[' '!' -e /pause ]
+ mkfifo /pause

svclb-istio-ingressgateway pod spec.

k get pod svclb-istio-ingressgateway-xnxb4 -o yaml -n istio-system       
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2021-03-09T20:53:37Z"
  generateName: svclb-istio-ingressgateway-
  labels:
    app: svclb-istio-ingressgateway
    controller-revision-hash: 64c454b8cb
    pod-template-generation: "1"
    svccontroller.k3s.cattle.io/svcname: istio-ingressgateway
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName: {}
        f:labels:
          .: {}
          f:app: {}
          f:controller-revision-hash: {}
          f:pod-template-generation: {}
          f:svccontroller.k3s.cattle.io/svcname: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"6629db22-fc1a-4261-9c90-fff35a96c0ad"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:affinity:
          .: {}
          f:nodeAffinity:
            .: {}
            f:requiredDuringSchedulingIgnoredDuringExecution:
              .: {}
              f:nodeSelectorTerms: {}
        f:containers:
          k:{"name":"lb-port-80"}:
            .: {}
            f:env:
              .: {}
              k:{"name":"DEST_IP"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"DEST_PORT"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"DEST_PROTO"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"SRC_PORT"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:ports:
              .: {}
              k:{"containerPort":80,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:hostPort: {}
                f:name: {}
                f:protocol: {}
            f:resources: {}
            f:securityContext:
              .: {}
              f:capabilities:
                .: {}
                f:add: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
          k:{"name":"lb-port-443"}:
            .: {}
            f:env:
              .: {}
              k:{"name":"DEST_IP"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"DEST_PORT"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"DEST_PROTO"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"SRC_PORT"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:ports:
              .: {}
              k:{"containerPort":443,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:hostPort: {}
                f:name: {}
                f:protocol: {}
            f:resources: {}
            f:securityContext:
              .: {}
              f:capabilities:
                .: {}
                f:add: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
          k:{"name":"lb-port-15012"}:
            .: {}
            f:env:
              .: {}
              k:{"name":"DEST_IP"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"DEST_PORT"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"DEST_PROTO"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"SRC_PORT"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:ports:
              .: {}
              k:{"containerPort":15012,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:hostPort: {}
                f:name: {}
                f:protocol: {}
            f:resources: {}
            f:securityContext:
              .: {}
              f:capabilities:
                .: {}
                f:add: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
          k:{"name":"lb-port-15021"}:
            .: {}
            f:env:
              .: {}
              k:{"name":"DEST_IP"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"DEST_PORT"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"DEST_PROTO"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"SRC_PORT"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:ports:
              .: {}
              k:{"containerPort":15021,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:hostPort: {}
                f:name: {}
                f:protocol: {}
            f:resources: {}
            f:securityContext:
              .: {}
              f:capabilities:
                .: {}
                f:add: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
          k:{"name":"lb-port-15443"}:
            .: {}
            f:env:
              .: {}
              k:{"name":"DEST_IP"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"DEST_PORT"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"DEST_PROTO"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"SRC_PORT"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:ports:
              .: {}
              k:{"containerPort":15443,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:hostPort: {}
                f:name: {}
                f:protocol: {}
            f:resources: {}
            f:securityContext:
              .: {}
              f:capabilities:
                .: {}
                f:add: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
        f:dnsPolicy: {}
        f:enableServiceLinks: {}
        f:restartPolicy: {}
        f:schedulerName: {}
        f:securityContext: {}
        f:terminationGracePeriodSeconds: {}
        f:tolerations: {}
      f:status:
        f:conditions:
          k:{"type":"ContainersReady"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Initialized"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Ready"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
        f:containerStatuses: {}
        f:hostIP: {}
        f:phase: {}
        f:podIP: {}
        f:podIPs:
          .: {}
          k:{"ip":"10.42.0.12"}:
            .: {}
            f:ip: {}
        f:startTime: {}
    manager: k3s
    operation: Update
    time: "2021-03-09T20:53:51Z"
  name: svclb-istio-ingressgateway-xnxb4
  namespace: istio-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: svclb-istio-ingressgateway
    uid: 6629db22-fc1a-4261-9c90-fff35a96c0ad
  resourceVersion: "1221"
  uid: bdc816f5-17b8-417d-9a91-6afd73789356
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - k3d-istio-workshop-server-0
  containers:
  - env:
    - name: SRC_PORT
      value: "15021"
    - name: DEST_PROTO
      value: TCP
    - name: DEST_PORT
      value: "15021"
    - name: DEST_IP
      value: 10.43.152.110
    image: rancher/klipper-lb:v0.1.2
    imagePullPolicy: IfNotPresent
    name: lb-port-15021
    ports:
    - containerPort: 15021
      hostPort: 15021
      name: lb-port-15021
      protocol: TCP
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-kbcwx
      readOnly: true
  - env:
    - name: SRC_PORT
      value: "80"
    - name: DEST_PROTO
      value: TCP
    - name: DEST_PORT
      value: "80"
    - name: DEST_IP
      value: 10.43.152.110
    image: rancher/klipper-lb:v0.1.2
    imagePullPolicy: IfNotPresent
    name: lb-port-80
    ports:
    - containerPort: 80
      hostPort: 80
      name: lb-port-80
      protocol: TCP
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-kbcwx
      readOnly: true
  - env:
    - name: SRC_PORT
      value: "443"
    - name: DEST_PROTO
      value: TCP
    - name: DEST_PORT
      value: "443"
    - name: DEST_IP
      value: 10.43.152.110
    image: rancher/klipper-lb:v0.1.2
    imagePullPolicy: IfNotPresent
    name: lb-port-443
    ports:
    - containerPort: 443
      hostPort: 443
      name: lb-port-443
      protocol: TCP
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-kbcwx
      readOnly: true
  - env:
    - name: SRC_PORT
      value: "15012"
    - name: DEST_PROTO
      value: TCP
    - name: DEST_PORT
      value: "15012"
    - name: DEST_IP
      value: 10.43.152.110
    image: rancher/klipper-lb:v0.1.2
    imagePullPolicy: IfNotPresent
    name: lb-port-15012
    ports:
    - containerPort: 15012
      hostPort: 15012
      name: lb-port-15012
      protocol: TCP
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-kbcwx
      readOnly: true
  - env:
    - name: SRC_PORT
      value: "15443"
    - name: DEST_PROTO
      value: TCP
    - name: DEST_PORT
      value: "15443"
    - name: DEST_IP
      value: 10.43.152.110
    image: rancher/klipper-lb:v0.1.2
    imagePullPolicy: IfNotPresent
    name: lb-port-15443
    ports:
    - containerPort: 15443
      hostPort: 15443
      name: lb-port-15443
      protocol: TCP
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-kbcwx
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: k3d-istio-workshop-server-0
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
    operator: Exists
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Exists
  - key: CriticalAddonsOnly
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/pid-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists
  volumes:
  - name: default-token-kbcwx
    secret:
      defaultMode: 420
      secretName: default-token-kbcwx
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-03-09T20:53:37Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2021-03-09T20:53:51Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2021-03-09T20:53:51Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2021-03-09T20:53:37Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://61cf11f9ae1667a5f4fd3c4055cd42b6d5904e2fde1f03bc228946334816336c
    image: docker.io/rancher/klipper-lb:v0.1.2
    imageID: docker.io/rancher/klipper-lb@sha256:2fb97818f5d64096d635bc72501a6cb2c8b88d5d16bc031cf71b5b6460925e4a
    lastState: {}
    name: lb-port-15012
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-03-09T20:53:50Z"
  - containerID: containerd://19b255038f99ec223de724a4693f2d04b2400099991997e5bd0828e42486d224
    image: docker.io/rancher/klipper-lb:v0.1.2
    imageID: docker.io/rancher/klipper-lb@sha256:2fb97818f5d64096d635bc72501a6cb2c8b88d5d16bc031cf71b5b6460925e4a
    lastState: {}
    name: lb-port-15021
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-03-09T20:53:50Z"
  - containerID: containerd://66bbb06af9b587a5ad3295396ebe170967171c21d9e8673040603a44b2a40753
    image: docker.io/rancher/klipper-lb:v0.1.2
    imageID: docker.io/rancher/klipper-lb@sha256:2fb97818f5d64096d635bc72501a6cb2c8b88d5d16bc031cf71b5b6460925e4a
    lastState: {}
    name: lb-port-15443
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-03-09T20:53:50Z"
  - containerID: containerd://f403495ccf69bb2e401ee88f1f924df9423659a645d979b5556d5760d4cafe74
    image: docker.io/rancher/klipper-lb:v0.1.2
    imageID: docker.io/rancher/klipper-lb@sha256:2fb97818f5d64096d635bc72501a6cb2c8b88d5d16bc031cf71b5b6460925e4a
    lastState: {}
    name: lb-port-443
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-03-09T20:53:50Z"
  - containerID: containerd://8fa8f3b38ae4a461d79e5e8fd4174452f6a9464930c8893964873309f3658aa2
    image: docker.io/rancher/klipper-lb:v0.1.2
    imageID: docker.io/rancher/klipper-lb@sha256:2fb97818f5d64096d635bc72501a6cb2c8b88d5d16bc031cf71b5b6460925e4a
    lastState: {}
    name: lb-port-80
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-03-09T20:53:50Z"
  hostIP: 172.26.0.2
  phase: Running
  podIP: 10.42.0.12
  podIPs:
  - ip: 10.42.0.12
  qosClass: BestEffort
  startTime: "2021-03-09T20:53:37Z"

istio-ingressgateway pod spec.

 k get pod istio-ingressgateway-5686db779c-z2hk7 -o yaml -n istio-system       
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/path: /stats/prometheus
    prometheus.io/port: "15020"
    prometheus.io/scrape: "true"
    sidecar.istio.io/inject: "false"
  creationTimestamp: "2021-03-09T20:53:37Z"
  generateName: istio-ingressgateway-5686db779c-
  labels:
    app: istio-ingressgateway
    chart: gateways
    heritage: Tiller
    install.operator.istio.io/owning-resource: unknown
    istio: ingressgateway
    istio.io/rev: 1-8-3
    operator.istio.io/component: IngressGateways
    pod-template-hash: 5686db779c
    release: istio
    service.istio.io/canonical-name: istio-ingressgateway
    service.istio.io/canonical-revision: 1-8-3
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:prometheus.io/path: {}
          f:prometheus.io/port: {}
          f:prometheus.io/scrape: {}
          f:sidecar.istio.io/inject: {}
        f:generateName: {}
        f:labels:
          .: {}
          f:app: {}
          f:chart: {}
          f:heritage: {}
          f:install.operator.istio.io/owning-resource: {}
          f:istio: {}
          f:istio.io/rev: {}
          f:operator.istio.io/component: {}
          f:pod-template-hash: {}
          f:release: {}
          f:service.istio.io/canonical-name: {}
          f:service.istio.io/canonical-revision: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"c7f93765-ead6-427e-86b9-be304827145c"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:affinity:
          .: {}
          f:nodeAffinity:
            .: {}
            f:preferredDuringSchedulingIgnoredDuringExecution: {}
            f:requiredDuringSchedulingIgnoredDuringExecution:
              .: {}
              f:nodeSelectorTerms: {}
        f:containers:
          k:{"name":"istio-proxy"}:
            .: {}
            f:args: {}
            f:env:
              .: {}
              k:{"name":"CA_ADDR"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"CANONICAL_REVISION"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef:
                    .: {}
                    f:apiVersion: {}
                    f:fieldPath: {}
              k:{"name":"CANONICAL_SERVICE"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef:
                    .: {}
                    f:apiVersion: {}
                    f:fieldPath: {}
              k:{"name":"HOST_IP"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef:
                    .: {}
                    f:apiVersion: {}
                    f:fieldPath: {}
              k:{"name":"INSTANCE_IP"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef:
                    .: {}
                    f:apiVersion: {}
                    f:fieldPath: {}
              k:{"name":"ISTIO_META_CLUSTER_ID"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"ISTIO_META_OWNER"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"ISTIO_META_ROUTER_MODE"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"ISTIO_META_WORKLOAD_NAME"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"JWT_POLICY"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"NODE_NAME"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef:
                    .: {}
                    f:apiVersion: {}
                    f:fieldPath: {}
              k:{"name":"PILOT_CERT_PROVIDER"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"POD_NAME"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef:
                    .: {}
                    f:apiVersion: {}
                    f:fieldPath: {}
              k:{"name":"POD_NAMESPACE"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef:
                    .: {}
                    f:apiVersion: {}
                    f:fieldPath: {}
              k:{"name":"SERVICE_ACCOUNT"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef:
                    .: {}
                    f:apiVersion: {}
                    f:fieldPath: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:lifecycle:
              .: {}
              f:preStop:
                .: {}
                f:exec:
                  .: {}
                  f:command: {}
            f:name: {}
            f:ports:
              .: {}
              k:{"containerPort":8080,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:protocol: {}
              k:{"containerPort":8443,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:protocol: {}
              k:{"containerPort":15012,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:protocol: {}
              k:{"containerPort":15021,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:protocol: {}
              k:{"containerPort":15090,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:name: {}
                f:protocol: {}
              k:{"containerPort":15443,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:protocol: {}
            f:readinessProbe:
              .: {}
              f:failureThreshold: {}
              f:httpGet:
                .: {}
                f:path: {}
                f:port: {}
                f:scheme: {}
              f:initialDelaySeconds: {}
              f:periodSeconds: {}
              f:successThreshold: {}
              f:timeoutSeconds: {}
            f:resources:
              .: {}
              f:limits:
                .: {}
                f:cpu: {}
                f:memory: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:memory: {}
            f:securityContext:
              .: {}
              f:allowPrivilegeEscalation: {}
              f:capabilities:
                .: {}
                f:drop: {}
              f:privileged: {}
              f:readOnlyRootFilesystem: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
            f:volumeMounts:
              .: {}
              k:{"mountPath":"/etc/istio/config"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/etc/istio/ingressgateway-ca-certs"}:
                .: {}
                f:mountPath: {}
                f:name: {}
                f:readOnly: {}
              k:{"mountPath":"/etc/istio/ingressgateway-certs"}:
                .: {}
                f:mountPath: {}
                f:name: {}
                f:readOnly: {}
              k:{"mountPath":"/etc/istio/pod"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/etc/istio/proxy"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/var/lib/istio/data"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/var/run/ingress_gateway"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/var/run/secrets/istio"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/var/run/secrets/tokens"}:
                .: {}
                f:mountPath: {}
                f:name: {}
                f:readOnly: {}
        f:dnsPolicy: {}
        f:enableServiceLinks: {}
        f:restartPolicy: {}
        f:schedulerName: {}
        f:securityContext:
          .: {}
          f:fsGroup: {}
          f:runAsGroup: {}
          f:runAsNonRoot: {}
          f:runAsUser: {}
        f:serviceAccount: {}
        f:serviceAccountName: {}
        f:terminationGracePeriodSeconds: {}
        f:volumes:
          .: {}
          k:{"name":"config-volume"}:
            .: {}
            f:configMap:
              .: {}
              f:defaultMode: {}
              f:name: {}
              f:optional: {}
            f:name: {}
          k:{"name":"gatewaysdsudspath"}:
            .: {}
            f:emptyDir: {}
            f:name: {}
          k:{"name":"ingressgateway-ca-certs"}:
            .: {}
            f:name: {}
            f:secret:
              .: {}
              f:defaultMode: {}
              f:optional: {}
              f:secretName: {}
          k:{"name":"ingressgateway-certs"}:
            .: {}
            f:name: {}
            f:secret:
              .: {}
              f:defaultMode: {}
              f:optional: {}
              f:secretName: {}
          k:{"name":"istio-data"}:
            .: {}
            f:emptyDir: {}
            f:name: {}
          k:{"name":"istio-envoy"}:
            .: {}
            f:emptyDir: {}
            f:name: {}
          k:{"name":"istio-token"}:
            .: {}
            f:name: {}
            f:projected:
              .: {}
              f:defaultMode: {}
              f:sources: {}
          k:{"name":"istiod-ca-cert"}:
            .: {}
            f:configMap:
              .: {}
              f:defaultMode: {}
              f:name: {}
            f:name: {}
          k:{"name":"podinfo"}:
            .: {}
            f:downwardAPI:
              .: {}
              f:defaultMode: {}
              f:items: {}
            f:name: {}
      f:status:
        f:conditions:
          k:{"type":"ContainersReady"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Initialized"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Ready"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
        f:containerStatuses: {}
        f:hostIP: {}
        f:phase: {}
        f:podIP: {}
        f:podIPs:
          .: {}
          k:{"ip":"10.42.0.11"}:
            .: {}
            f:ip: {}
        f:startTime: {}
    manager: k3s
    operation: Update
    time: "2021-03-09T20:53:39Z"
  name: istio-ingressgateway-5686db779c-z2hk7
  namespace: istio-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: istio-ingressgateway-5686db779c
    uid: c7f93765-ead6-427e-86b9-be304827145c
  resourceVersion: "1186"
  uid: a5638e42-ab1e-4e4e-9a5b-7afc57165b74
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - amd64
        weight: 2
      - preference:
          matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - ppc64le
        weight: 2
      - preference:
          matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - s390x
        weight: 2
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - amd64
            - ppc64le
            - s390x
  containers:
  - args:
    - proxy
    - router
    - --domain
    - $(POD_NAMESPACE).svc.cluster.local
    - --proxyLogLevel=warning
    - --proxyComponentLogLevel=misc:error
    - --log_output_level=default:info
    - --serviceCluster
    - istio-ingressgateway
    env:
    - name: JWT_POLICY
      value: third-party-jwt
    - name: PILOT_CERT_PROVIDER
      value: istiod
    - name: CA_ADDR
      value: istiod-1-8-3.istio-system.svc:15012
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: HOST_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: SERVICE_ACCOUNT
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.serviceAccountName
    - name: CANONICAL_SERVICE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels['service.istio.io/canonical-name']
    - name: CANONICAL_REVISION
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels['service.istio.io/canonical-revision']
    - name: ISTIO_META_WORKLOAD_NAME
      value: istio-ingressgateway
    - name: ISTIO_META_OWNER
      value: kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway
    - name: ISTIO_META_ROUTER_MODE
      value: standard
    - name: ISTIO_META_CLUSTER_ID
      value: Kubernetes
    image: docker.io/istio/proxyv2:1.8.3
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - sh
          - -c
          - sleep 5
    name: istio-proxy
    ports:
    - containerPort: 15021
      protocol: TCP
    - containerPort: 8080
      protocol: TCP
    - containerPort: 8443
      protocol: TCP
    - containerPort: 15012
      protocol: TCP
    - containerPort: 15443
      protocol: TCP
    - containerPort: 15090
      name: http-envoy-prom
      protocol: TCP
    readinessProbe:
      failureThreshold: 30
      httpGet:
        path: /healthz/ready
        port: 15021
        scheme: HTTP
      initialDelaySeconds: 1
      periodSeconds: 2
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 128Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/istio/config
      name: config-volume
    - mountPath: /var/run/secrets/istio
      name: istiod-ca-cert
    - mountPath: /var/run/secrets/tokens
      name: istio-token
      readOnly: true
    - mountPath: /var/run/ingress_gateway
      name: gatewaysdsudspath
    - mountPath: /var/lib/istio/data
      name: istio-data
    - mountPath: /etc/istio/pod
      name: podinfo
    - mountPath: /etc/istio/ingressgateway-certs
      name: ingressgateway-certs
      readOnly: true
    - mountPath: /etc/istio/ingressgateway-ca-certs
      name: ingressgateway-ca-certs
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: istio-ingressgateway-service-account-token-ht8zm
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: k3d-istio-workshop-server-0
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1337
    runAsGroup: 1337
    runAsNonRoot: true
    runAsUser: 1337
  serviceAccount: istio-ingressgateway-service-account
  serviceAccountName: istio-ingressgateway-service-account
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - configMap:
      defaultMode: 420
      name: istio-ca-root-cert
    name: istiod-ca-cert
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels
        path: labels
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations
        path: annotations
    name: podinfo
  - emptyDir: {}
    name: istio-envoy
  - emptyDir: {}
    name: gatewaysdsudspath
  - emptyDir: {}
    name: istio-data
  - name: istio-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: istio-ca
          expirationSeconds: 43200
          path: istio-token
  - configMap:
      defaultMode: 420
      name: istio-1-8-3
      optional: true
    name: config-volume
  - name: ingressgateway-certs
    secret:
      defaultMode: 420
      optional: true
      secretName: istio-ingressgateway-certs
  - name: ingressgateway-ca-certs
    secret:
      defaultMode: 420
      optional: true
      secretName: istio-ingressgateway-ca-certs
  - name: istio-ingressgateway-service-account-token-ht8zm
    secret:
      defaultMode: 420
      secretName: istio-ingressgateway-service-account-token-ht8zm
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-03-09T20:53:37Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2021-03-09T20:53:39Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2021-03-09T20:53:39Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2021-03-09T20:53:37Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://3f8d6e55d111efcdd31f113e73cbd07ee4f8ffd8ba26481460546b22533c960c
    image: docker.io/istio/proxyv2:1.8.3
    imageID: docker.io/istio/proxyv2@sha256:5cfde7ffd5b921cf805f4cf18013d3f1b825f41fe1bd1d977d805c45ca955d5a
    lastState: {}
    name: istio-proxy
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-03-09T20:53:37Z"
  hostIP: 172.26.0.2
  phase: Running
  podIP: 10.42.0.11
  podIPs:
  - ip: 10.42.0.11
  qosClass: Burstable
  startTime: "2021-03-09T20:53:37Z"

[murphye]

If all the svclb is supposed to do is IP Tables routing, why this is happening is beyond me...

[murphye]

More info for you:

k get svc -n istio-system
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                      AGE
istiod                 ClusterIP      10.43.25.101    <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP                                        93m
istiod-1-8-3           ClusterIP      10.43.233.100   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP                                        92m
istio-ingressgateway   LoadBalancer   10.43.152.110   172.26.0.2    15021:30331/TCP,80:30864/TCP,443:31938/TCP,15012:30935/TCP,15443:30695/TCP   91m

k get svc -n istio-system istio-ingressgateway -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"istio-ingressgateway","install.operator.istio.io/owning-resource":"istio-ingress-gw-install","install.operator.istio.io/owning-resource-namespace":"istio-system","istio":"ingressgateway","istio.io/rev":"1-8-3","operator.istio.io/component":"IngressGateways","operator.istio.io/managed":"Reconcile","operator.istio.io/version":"1.8.3","release":"istio"},"name":"istio-ingressgateway","namespace":"istio-system"},"spec":{"ports":[{"name":"status-port","port":15021,"protocol":"TCP","targetPort":15021},{"name":"http2","port":80,"protocol":"TCP","targetPort":8080},{"name":"https","port":443,"protocol":"TCP","targetPort":8443},{"name":"tcp-istiod","port":15012,"protocol":"TCP","targetPort":15012},{"name":"tls","port":15443,"protocol":"TCP","targetPort":15443}],"selector":{"app":"istio-ingressgateway","istio":"ingressgateway"},"type":"LoadBalancer"}}
  creationTimestamp: "2021-03-09T20:53:37Z"
  labels:
    app: istio-ingressgateway
    install.operator.istio.io/owning-resource: istio-ingress-gw-install
    install.operator.istio.io/owning-resource-namespace: istio-system
    istio: ingressgateway
    istio.io/rev: 1-8-3
    operator.istio.io/component: IngressGateways
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.8.3
    release: istio
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app: {}
          f:install.operator.istio.io/owning-resource: {}
          f:install.operator.istio.io/owning-resource-namespace: {}
          f:istio: {}
          f:istio.io/rev: {}
          f:operator.istio.io/component: {}
          f:operator.istio.io/managed: {}
          f:operator.istio.io/version: {}
          f:release: {}
      f:spec:
        f:externalTrafficPolicy: {}
        f:ports:
          .: {}
          k:{"port":80,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":443,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":15012,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":15021,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":15443,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
          f:istio: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: istioctl
    operation: Update
    time: "2021-03-09T20:53:37Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:loadBalancer:
          f:ingress: {}
    manager: k3s
    operation: Update
    time: "2021-03-09T20:53:51Z"
  name: istio-ingressgateway
  namespace: istio-system
  resourceVersion: "1223"
  uid: e9ad5ede-1316-4c08-af24-1d8f488bac54
spec:
  clusterIP: 10.43.152.110
  clusterIPs:
  - 10.43.152.110
  externalTrafficPolicy: Cluster
  ports:
  - name: status-port
    nodePort: 30331
    port: 15021
    protocol: TCP
    targetPort: 15021
  - name: http2
    nodePort: 30864
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 31938
    port: 443
    protocol: TCP
    targetPort: 8443
  - name: tcp-istiod
    nodePort: 30935
    port: 15012
    protocol: TCP
    targetPort: 15012
  - name: tls
    nodePort: 30695
    port: 15443
    protocol: TCP
    targetPort: 15443
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 172.26.0.2

[comphilip]

@murphye

k port-forward istio-ingressgateway-5686db779c-z2hk7 7443:43 -n istio-system

You port-forward port's 43 port to localhost 7443 port and certificate works well. While in pod yaml there is no 43 container port declared:

    - containerPort: 15021
      protocol: TCP
    - containerPort: 8080
      protocol: TCP
    - containerPort: 8443
      protocol: TCP
    - containerPort: 15012
      protocol: TCP
    - containerPort: 15443
      protocol: TCP
    - containerPort: 15090
      name: http-envoy-prom
      protocol: TCP

istio-ingressgateway service declared 443 to pod's 8443, so your pod ports and make it sync with those of service.

[juniorz]

@murphye , did you figure why this happened? I am facing the same issue, standard Istio install with minimal profile via IstioOperator.

[murphye]

@juniorz No. I have not tried this in a long time. You may want to try MetalLB.