RabbitMQ Messaging Topology Kubernetes Operator: User creation fails on DNS Lookup

[rowi1de]

Name and Version

bitnami/rabbitmq-cluster-operator:3.8.1

What architecture are you using?

None

What steps will reproduce the bug?

Chart.yaml

apiVersion: v2
name: rabbitmq-cluster-operator
description: RabbitMQ Cluster Operator

type: application
version: 3.8.1

dependencies:
  - name: rabbitmq-cluster-operator
    repository: https://charts.bitnami.com/bitnami
    version: 3.8.1

rabbitmq-cluster-operator:
  clusterOperator:
    replicaCount: 1
    resources:
      requests:
        cpu: 200m
        memory: 1Gi
      limits:
        cpu: 500m
        memory: 1Gi
  msgTopologyOperator:
    hostNetwork: true

cluster.yaml

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: cluster
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "-5"
spec:
  replicas: 1
  rabbitmq:
    additionalPlugins:
      - rabbitmq_shovel
      - rabbitmq_shovel_management
  resources:
    requests:
      cpu: 125m
      memory: 1Gi
    limits:
      cpu: 250m
      memory: 1Gi

User.yaml

apiVersion: rabbitmq.com/v1beta1
kind: User
metadata:
  labels:
    argocd.argoproj.io/instance: rabbitmq-cluster
  name: user-from-code
  namespace: rabbitmq
spec:
  importCredentialsSecret:
    name: funnel-rabbitmq
  rabbitmqClusterReference:
    name: cluster
    namespace: rabbitmq
  tags:
    - management

Are you using any custom parameters or values?

See values.yaml above.

What is the expected behavior?

The User Creation works

What do you see instead?

Put "http://cluster.rabbitmq.svc:15672/api/users/user-from-code\": dial tcp: lookup cluster.rabbitmq.svc on 172.24.216.2:53: no such host"

I'm really confused about the used cluster name: cluster.rabbitmq.svc

We also have a service which is deployed as part of the "Cluster"

apiVersion: v1
kind: Service
metadata:
  annotations:
    helm.sh/hook: 'post-install,post-upgrade'
    helm.sh/hook-weight: '-5'
  creationTimestamp: '2023-09-25T12:40:37Z'
  labels:
    app.kubernetes.io/component: rabbitmq
    app.kubernetes.io/name: cluster
    app.kubernetes.io/part-of: rabbitmq
    argocd.argoproj.io/instance: rabbitmq-cluster
  name: cluster
  namespace: rabbitmq
  ownerReferences:
    - apiVersion: rabbitmq.com/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: RabbitmqCluster
      name: cluster
      uid: 3a6919d8-625f-4154-a632-13da5a51e967
  resourceVersion: '80299219'
  uid: d850540e-0ad9-463d-85db-cae2fe5287d8
spec:
  clusterIP: 10.xxx.xxx.xxx
  clusterIPs:
    -10.xxx.xxx.xxx
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - appProtocol: amqp
      name: amqp
      port: 5672
      protocol: TCP
      targetPort: 5672
    - appProtocol: http
      name: management
      port: 15672
      protocol: TCP
      targetPort: 15672
    - appProtocol: prometheus.io/metrics
      name: prometheus
      port: 15692
      protocol: TCP
      targetPort: 15692
  selector:
    app.kubernetes.io/name: cluster
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Additional information

2023-09-28T08:59:56.000Z DEV/rabbitmq/rabbitmq-cluster-operator rabbitmq-cluster-operator-rabbitmq-messag {"level":"error","ts":"2023-09-28T08:59:54Z","msg":"Reconciler error","controller":"user","controllerGroup":"rabbitmq.com","controllerKind":"User","User":{"name":"funnel-from-code","namespace":"rabbitmq"},"namespace":"rabbitmq","name":"funnel-from-code","reconcileID":"e950c368-a213-4a38-9079-77ef2969e965","error":"Put \"http://cluster.rabbitmq.svc:15672/api/users/funnel-from-code\": dial tcp: lookup cluster.rabbitmq.svc on 172.24.216.2:53: no such host","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/bitnami/blacksmith-sandox/rmq-messaging-topology-operator-1.12.0/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/bitnami/blacksmith-sandox/rmq-messaging-topology-operator-1.12.0/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/bitnami/blacksmith-sandox/rmq-messaging-topology-operator-1.12.0/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226"}

bitnami/charts#19593

[Zerpet]

The problem seems to be with the usage of a host network. The Topology Operator uses kubernetes short names i.e. <svc-name>.<namespace>.svc. In your case, you seem to be able to resolve the hostname when the internal domain name is appended to the shortname.

You can use the env variable in the Topology Operator MESSAGING_DOMAIN_NAME, and set the value to the internal kubernetes domain name cluster.local. That will force the Topology Operator to use the FQDN, instead of the short name.

https://rabbitmq.com/kubernetes/operator/install-topology-operator.html#non-default-k8s-domain

Edit: Improving response with additions from @rowi1de

as I've set hostNetwork:true I also needed to override the default DNS policy to use dnsPolicy: ClusterFirstWithHostNet
I'm using https://artifacthub.io/packages/helm/bitnami/rabbitmq-cluster-operator

rabbitmq-cluster-operator:
  clusterOperator:
    replicaCount: 1
    resources:
      requests:
        cpu: 200m
        memory: 1Gi
      limits:
        cpu: 500m
        memory: 1Gi
  msgTopologyOperator:
    hostNetwork: true
    dnsPolicy: ClusterFirstWithHostNet
    extraEnvVars:
      # https://rabbitmq.com/kubernetes/operator/install-topology-operator.html#non-default-k8s-domain
      # force FQDN instead of <svc-name>.<namespace>.svc because of hostNetwork
      - name: MESSAGING_DOMAIN_NAME
        value: cluster.local

Edit: potentially required override with hostNetwork: true

RabbitMQ may fail to resolve its own hostname when host network is used. In that case, an override must be used to correct the DNS resolution in rabbitmq container:

---
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbit-hostnetwork
spec:
  image: rabbitmq:3.11-management
  resources:
    requests:
      cpu: "1"
      memory: "1Gi"
    limits:
      cpu: "1"
      memory: "1Gi"
  override:
    statefulSet:
      spec:
        template:
          spec:
            hostNetwork: true
            dnsPolicy: ClusterFirstWithHostNet
            containers:
            - name: rabbitmq

[rowi1de]

Thanks for the hint, this part helped with one addition:

as I've set hostNetwork:true I also needed to override the default DNS policy to use dnsPolicy: ClusterFirstWithHostNet
I'm using https://artifacthub.io/packages/helm/bitnami/rabbitmq-cluster-operator

rabbitmq-cluster-operator:
  clusterOperator:
    replicaCount: 1
    resources:
      requests:
        cpu: 200m
        memory: 1Gi
      limits:
        cpu: 500m
        memory: 1Gi
  msgTopologyOperator:
    hostNetwork: true
    dnsPolicy: ClusterFirstWithHostNet
    extraEnvVars:
      # https://rabbitmq.com/kubernetes/operator/install-topology-operator.html#non-default-k8s-domain
      # force FQDN instead of <svc-name>.<namespace>.svc because of hostNetwork
      - name: MESSAGING_DOMAIN_NAME
        value: cluster.local

[Zerpet]

Thank you for circling back 🚀 I'll update my response with this for future users that find this discussion.