Cilium/Multus doesn't create an additional network interface

[seilerre]

Environmental Info:
RKE2 Version:

rke2 version v1.31.4+rke2r1 (5142bee)
go version go1.22.9 X:boringcrypto

Node(s) CPU architecture, OS, and Version:

Linux master-01 6.8.0-51-generic #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec 5 13:09:44 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

k get node
NAME STATUS ROLES AGE VERSION
master-01 Ready control-plane,etcd,master 58m v1.31.4+rke2r1
worker-01 Ready 57m v1.31.4+rke2r1
worker-02 Ready 57m v1.31.4+rke2r1

Describe the bug:

Multus is not called, an example Pod is created without a new network interface.

Steps To Reproduce:

I used lablabs.rke2 with this config:

rke2_version: "v1.31.4+rke2r1"
    rke2_airgap_mode: false
    rke2_artifact_path: /tmp
    rke2_cni:
      - multus
      - cilium

Then I did the following to have cni.exclusive = false: https://docs.rke2.io/networking/multus_sriov#using-multus-with-cilium

The cluster then looks like this

k get pod -n kube-system
NAME                                                    READY   STATUS      RESTARTS      AGE
cilium-operator-f878658d7-cn7dz                         1/1     Running     0             64m
cilium-operator-f878658d7-knrtn                         1/1     Running     0             64m
cilium-qspkm                                            1/1     Running     0             64m
cilium-r6k2r                                            1/1     Running     0             63m
cilium-t2sz8                                            1/1     Running     0             63m
cloud-controller-manager-master-01                      1/1     Running     0             64m
etcd-master-01                                          1/1     Running     0             64m
helm-install-rke2-cilium-kp2sz                          0/1     Completed   0             55m
helm-install-rke2-coredns-ljnxw                         0/1     Completed   0             64m
helm-install-rke2-ingress-nginx-f9vxn                   0/1     Completed   1             64m
helm-install-rke2-metrics-server-qjd5q                  0/1     Completed   1             64m
helm-install-rke2-multus-mrddl                          0/1     Completed   0             54m
helm-install-rke2-snapshot-controller-6xtdb             0/1     Completed   2             64m
helm-install-rke2-snapshot-controller-crd-sxswv         0/1     Completed   1             64m
helm-install-rke2-snapshot-validation-webhook-jlwnz     0/1     Completed   1             64m
kube-apiserver-master-01                                1/1     Running     0             64m
kube-controller-manager-master-01                       1/1     Running     0             64m
kube-proxy-master-01                                    1/1     Running     0             64m
kube-proxy-worker-01                                    1/1     Running     0             62m
kube-proxy-worker-02                                    1/1     Running     0             62m
kube-scheduler-master-01                                1/1     Running     0             64m
rke2-coredns-rke2-coredns-55bdf87668-lnjmh              1/1     Running     0             64m
rke2-coredns-rke2-coredns-55bdf87668-vxrhw              1/1     Running     0             62m
rke2-coredns-rke2-coredns-autoscaler-65c8c6bd64-rz57k   1/1     Running     0             64m
rke2-ingress-nginx-controller-gm4vx                     1/1     Running     0             59m
rke2-ingress-nginx-controller-qq7hb                     1/1     Running     0             59m
rke2-ingress-nginx-controller-s6gc8                     1/1     Running     0             59m
rke2-metrics-server-7c85d458bd-67qcz                    1/1     Running     3 (60m ago)   62m
rke2-multus-dhcp-299bt                                  1/1     Running     0             54m
rke2-multus-dhcp-kl4w4                                  1/1     Running     0             54m
rke2-multus-dhcp-v9jms                                  1/1     Running     0             54m
rke2-multus-frlt8                                       1/1     Running     0             63m
rke2-multus-qslpp                                       1/1     Running     0             63m
rke2-multus-rp2gz                                       1/1     Running     0             64m
rke2-snapshot-controller-65bc6fbd57-8l7dc               1/1     Running     3 (60m ago)   62m
rke2-snapshot-validation-webhook-859c7896df-mzbvh       1/1     Running     3 (60m ago)   62m

Then I created the following NetworkAttachmentDefinition. I also tried others with type dhcp, ...

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: multus-iot
  namespace: test
spec:
  config: |-
    {
      "cniVersion": "0.3.1",
      "type": "ipvlan",
      "master": "eth1",
      "ipam": {
        "type": "static",
        "routes": [
          { "dst": "192.168.0.0/16", "gw": "192.168.20.1" }
        ]
      }
    }

Creating the following Pods works, however no interface is added:

apiVersion: v1
kind: Pod
metadata:
  name: sample-pod
  namespace: test
  annotations:
    k8s.v1.cni.cncf.io/networks: |
      [{
        "name": "multus-iot",
        "namespace": "test",
        "mac": "c6:5e:a4:8e:7a:59",
        "ips": ["192.168.20.202/24"]
      }]
spec:
  containers:
  - name: sample-pod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine

Expected behavior:

Multus creates a second network interface in addition to the one Cilium is providing.

Actual behavior:

The created Pod looks like this:

k describe pod sample-pod
Name:             sample-pod
Namespace:        test
Priority:         0
Service Account:  default
Node:             master-01/172.19.10.22
Start Time:       Thu, 02 Jan 2025 18:18:10 +0100
Labels:           <none>
Annotations:      env-injector-webhook-status: injected
                  k8s.v1.cni.cncf.io/networks:
                    [{
                      "name": "multus-iot",
                      "namespace": "test",
                      "mac": "c6:5e:a4:8e:7a:59",
                      "ips": ["192.168.20.202/24"]
                    }]
Status:           Running
IP:               10.42.0.246
IPs:
  IP:  10.42.0.246
Containers:
  sample-pod:
    Container ID:  containerd://3ff6d79b9765939046aa35b513c5ec18c9140b23eda9851e22a5ddbef708fd92
    Image:         alpine
    Image ID:      docker.io/library/alpine@sha256:21dc6063fd678b478f57c0e13f47560d0ea4eeba26dfc947b2a4f81f686b9f45
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/ash
      -c
      trap : TERM INT; sleep infinity & wait
    State:          Running
      Started:      Thu, 02 Jan 2025 18:18:12 +0100
    Ready:          True
    Restart Count:  0
    Environment:
      redacted proxy settings
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j9ms5 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  kube-api-access-j9ms5:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  8m46s  default-scheduler  Successfully assigned test/sample-pod to master-01
  Normal  Pulling    8m46s  kubelet            Pulling image "alpine"
  Normal  Pulled     8m45s  kubelet            Successfully pulled image "alpine" in 979ms (979ms including waiting). Image size: 3655264 bytes.
  Normal  Created    8m45s  kubelet            Created container sample-pod
  Normal  Started    8m45s  kubelet            Started container sample-pod

As you can see the events not even mention Multus

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
116: eth0@if117: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 8a:08:7b:3e:79:e6 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.246/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::8808:7bff:fe3e:79e6/64 scope link
       valid_lft forever preferred_lft forever

Additional context / logs:

I first played around with K3s, where I installed Cilium manually, first with a pretty vanilla installation, then adjusting the cni paths, as I suspected that this might be a problem cilium install --version 1.16.5 --set=ipam.operator.clusterPoolIPv4PodCIDRList="10.42.0.0/16" --set=cni.exclusive=false --set=cni.binPath='/var/lib/rancher/k3s/data/cni/' --set=cni.confPath='/var/lib/rancher/k3s/agent/etc/cni/net.d'. As I couldn't get it to work with K3s I switched to RKE2, as multus and cilium is part of the standard installation and I hoped I had more luck with this

[brandond]

I suspect there is an issue with either your NetworkAttachmentDefinition or the networks annotation. I'm no Multus expert but I can't say I've ever seen JSON in the annotation value like that - normally its something like
k8s.v1.cni.cncf.io/networks: NetworkAttachmentDefitionName@InterfaceName - so for you that would be something like k8s.v1.cni.cncf.io/networks: multus-iot@eth1