Karmada execution-controller failed to resolve webhook address

[NickYadance]

Events:

Type     Reason                  Age                From                  Message
  ----     ------                  ----               ----                  -------
  Normal   ApplyPolicySucceed      14s                resource-detector     Apply cluster policy(global-clusterpropagationpolicy) succeed
  Warning  SyncFailed              8s (x10 over 13s)  execution-controller  Failed to create resource(default/sample-ud) in member cluster(recommend): Internal error occurred: failed calling webhook "muniteddeployment.kb.io": failed to call webhook: Post "https://kruise-webhook-service.kruise-system.svc:443/mutate-apps-kruise-io-v1alpha1-uniteddeployment?timeout=10s": dial tcp: lookup kruise-webhook-service.kruise-system.svc on 10.153.64.2:53: no such host
  Normal   SyncWorkSucceed         3s (x12 over 14s)  binding-controller    Sync work of resourceBinding(default/sample-ud-uniteddeployment) successful.
  Normal   AggregateStatusSucceed  3s (x12 over 14s)  binding-controller    Update resourceBinding(default/sample-ud-uniteddeployment) with AggregatedStatus successfully.
  Normal   ScheduleBindingSucceed  3s (x11 over 14s)  karmada-scheduler     Binding has been scheduled
  Warning  SyncFailed              3s (x2 over 12s)   execution-controller  Failed to create resource(default/sample-ud) in member cluster(recommend): Internal error occurred: failed calling webhook "muniteddeployment.kb.io": failed to call webhook: Post "https://kruise-webhook-service.kruise-system.svc:443/mutate-apps-kruise-io-v1alpha1-uniteddeployment?timeout=10s": dial tcp: lookup kruise-webhook-service.kruise-system.svc on 10.153.64.1:53: no such host

The webhook config is updated according to this tutorial : install-gatekeeper-components-on-host-cluster

---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  labels:
    clusterpropagationpolicy.karmada.io/name: global-clusterpropagationpolicy
  name: kruise-mutating-webhook-configuration
webhooks:
  - admissionReviewVersions:
      - v1
      - v1beta1
    clientConfig:
      url: https://kruise-webhook-service.kruise-system.svc:443/mutate-apps-kruise-io-v1alpha1-uniteddeployment
# Replced with url
#      service:
#        name: kruise-webhook-service
#        namespace: kruise-system
#        path: /mutate-pod
#        port: 443

And the karmada-api-server can call webhook without issue, but not karmada-controller-manager.

E0227 03:07:01.074536       1 objectwatcher.go:87] Failed to create resource(kind=UnitedDeployment, default/sample-ud) in cluster recommend, err is Internal error occurred: failed calling webhook "muniteddeployment.kb.io": failed to call webhook: Post "https://kruise-webhook-service.kruise-system.svc:443/mutate-apps-kruise-io-v1alpha1-uniteddeployment?timeout=10s": dial tcp: lookup kruise-webhook-service.kruise-system.svc on 10.153.64.2:53: no such host 
I0227 03:07:01.074725       1 recorder.go:103] "events: Failed to create resource(default/sample-ud) in member cluster(recommend): Internal error occurred: failed calling webhook \"muniteddeployment.kb.io\": failed to call webhook: Post \"https://kruise-webhook-service.kruise-system.svc:443/mutate-apps-kruise-io-v1alpha1-uniteddeployment?timeout=10s\": dial tcp: lookup kruise-webhook-service.kruise-system.svc on 10.153.64.2:53: no such host" type="Warning" object={Kind:UnitedDeployment Namespace:default Name:sample-ud UID:43740f32-47ca-4eb7-a4f7-fa42751440f1 APIVersion:apps.kruise.io/v1alpha1 ResourceVersion: FieldPath:} reason="SyncFailed"

Dns lookup failed in karmada-controller-manager

/ # nslookup kruise-webhook-service.kruise-system.svc
Server:         10.96.0.10
Address:        10.96.0.10:53


** server can't find kruise-webhook-service.kruise-system.svc: NXDOMAIN

Dns lookup succeed in a debug pod in the same namespace

/ # nslookup kruise-webhook-service.kruise-system.svc
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kruise-webhook-service.kruise-system.svc.cluster.local
Address: 10.96.175.244

[whitewindmills]

First, you should a new svc kruise-webhook-service under the karmada-system namespace and refer to the kruise-webhook-service.kruise-system.svc. For example:

apiVersion: v1
kind: Service
metadata:
  name: kruise-webhook-service
  namespace: karmada-system
spec:
  externalName: kruise-webhook-service.kruise-system.svc.cluster.local
  internalTrafficPolicy: Cluster
  ports:
    - port: 443
      protocol: TCP
      targetPort: 443
  sessionAffinity: None
  type: ExternalName

Second, you should update the MutatingWebhookConfiguration's url to https://kruise-webhook-service.karmada-system.svc:443/mutate-apps-kruise-io-v1alpha1-uniteddeployment

[NickYadance]

Thx for the reply.

1. I have the correct svc as below

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"kruise-webhook-service","namespace":"kruise-system"},"spec":{"ports":[{"port":443,"targetPort":9876}],"selector":{"control-plane":"controller-manager"}}}
  creationTimestamp: "2023-02-24T02:57:40Z"
  labels:
    clusterpropagationpolicy.karmada.io/name: global-clusterpropagationpolicy
  name: kruise-webhook-service
  namespace: kruise-system
  resourceVersion: "1926"
  uid: f1450a86-2fc5-40e5-89d3-35078ef63e52
spec:
  clusterIP: 10.98.147.49
  clusterIPs:
  - 10.98.147.49
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 443
    protocol: TCP
    targetPort: 9876
  selector:
    control-plane: controller-manager
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

2. I have updated the webhook confguration just as how u did.

The dns is a little different between karmada-controller-manager and my debug pod. The karmada controller seems to have the incorrect dns resolver address and that causes the issue.

# karmada controller
/ # nslookup kruise-webhook-service.kruise-system.svc
Server:         10.96.0.10
Address:        10.96.0.10:53


** server can't find kruise-webhook-service.kruise-system.svc: NXDOMAIN

# debug pod
/ # nslookup kruise-webhook-service.kruise-system.svc
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kruise-webhook-service.kruise-system.svc.cluster.local
Address: 10.96.175.244

[whitewindmills]

Thx for the reply.

1. I have the correct svc as below

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"kruise-webhook-service","namespace":"kruise-system"},"spec":{"ports":[{"port":443,"targetPort":9876}],"selector":{"control-plane":"controller-manager"}}}
  creationTimestamp: "2023-02-24T02:57:40Z"
  labels:
    clusterpropagationpolicy.karmada.io/name: global-clusterpropagationpolicy
  name: kruise-webhook-service
  namespace: kruise-system
  resourceVersion: "1926"
  uid: f1450a86-2fc5-40e5-89d3-35078ef63e52
spec:
  clusterIP: 10.98.147.49
  clusterIPs:
  - 10.98.147.49
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 443
    protocol: TCP
    targetPort: 9876
  selector:
    control-plane: controller-manager
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

2. I have updated the webhook confguration just as how u did.

The dns is a little different between karmada-controller-manager and my debug pod. The karmada controller seems to have the incorrect dns resolver address and that causes the issue.

# karmada controller
/ # nslookup kruise-webhook-service.kruise-system.svc
Server:         10.96.0.10
Address:        10.96.0.10:53


** server can't find kruise-webhook-service.kruise-system.svc: NXDOMAIN

# debug pod
/ # nslookup kruise-webhook-service.kruise-system.svc
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kruise-webhook-service.kruise-system.svc.cluster.local
Address: 10.96.175.244

You'd better check your karmada-controller-manager's DNS config(/etc/resolv.conf).

[NickYadance]

The dns logs of controller-manager.

#:~/api-server/karmada$ kubectl logs --namespace=kube-system -l k8s-app=kube-dns | grep -v "etcd"   
[INFO] 10.244.1.38:35242 - 17883 "AAAA IN kruise-webhook-service.kruise-system.svc. udp 58 false 512" NOERROR qr,aa,rd,ra 58 0.000924903s
[INFO] 10.244.1.38:35242 - 17377 "A IN kruise-webhook-service.kruise-system.svc. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 58 0.004017635s

[NickYadance]

After searching out for a while, found out that it's caused by kube-api-server being unable to resolve in-cluster dns.

But there is potential issue with Karmada components dns related to alpine, and i have made a simple PR here.