Support VPN only functionality on DigitalOcean managed k8s

[spigell]

Greetings! I use kilo only as a vpn server to inCluster resources, i.e. I have only one pod with kilo as deployment. On bare metal k3s cluster it works fine. But on DO managed k8s there are some troubles:

1. nodes doesn't have a wireguard module. Trying to use boringtun.
2. Even with boringtun interface do not in UP state and no config applied to it.

There is my deployment (also tried as DaemonSet WITH/OR HostNet+privileged):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/managed-by: pulumi
    app.kubernetes.io/name: kilo
    app.kubernetes.io/part-of: kilo
  name: kilo
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kilo
      app.kubernetes.io/part-of: kilo
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kilo
        app.kubernetes.io/part-of: kilo
    spec:
      containers:
      - args:
        - kilo0
        - --foreground
        - --verbosity=debug
        - --disable-drop-privileges=true
        image: leonnicolas/boringtun
        name: boringtun
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /var/run/wireguard
          name: wireguard
          readOnly: false
      - args:
        - --kubeconfig=/etc/kubernetes/kubeconfig
        - --hostname=$(NODE_NAME)
        - --cni=false
        - --log-level=all
        - --port=51821
        - --create-interface=false
        - --topology-label=test.io/region
        - --interface=kilo0
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        image: squat/kilo:latest
        name: kilo
        ports:
        - containerPort: 1107
          name: metrics
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - SYS_MODULE
          privileged: false
        volumeMounts:
        - mountPath: /etc/kubernetes
          name: kubeconfig
        - mountPath: /scripts/
          name: scripts
          readOnly: true
        - mountPath: /lib/modules
          name: lib-modules
          readOnly: true
        - mountPath: /run/xtables.lock
          name: xtables-lock
          readOnly: false
        - mountPath: /var/run/wireguard
          name: wireguard
          readOnly: false
        - mountPath: /var/lib/kilo
          name: kilo-dir
      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: false
      initContainers:
      - args:
        - /scripts/init.sh
        command:
        - /bin/sh
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: squat/kilo
        name: prepare-configs
        volumeMounts:
        - mountPath: /etc/kubernetes
          name: kubeconfig
        - mountPath: /scripts/
          name: scripts
          readOnly: true
        - mountPath: /var/lib/kilo
          name: kilo-dir
        - mountPath: secrets
          name: key
      serviceAccountName: kilo
      volumes:
      - name: kilo-dir
      - name: key
        secret:
          secretName: kilo-private-key
      - hostPath:
          path: /lib/modules
        name: lib-modules
      - hostPath:
          path: /var/run/wireguard
        name: wireguard
      - name: kubeconfig
      - configMap:
          name: kilo-scripts
        name: scripts
      - hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
        name: xtables-lock

Starting logs:

boringtun   Dec 08 00:19:33.796  INFO boringtun: BoringTun started successfully
boringtun     at src/main.rs:186
boringtun 
kilo {"caller":"mesh.go:143","component":"kilo","level":"debug","msg":"using 10.244.1.77/32 as the private IP address","ts":"2021-12-08T00:19:34.835604164Z"}
kilo {"caller":"mesh.go:154","component":"kilo","level":"debug","msg":"using 10.244.1.77/32 as the public IP address","ts":"2021-12-08T00:19:34.835705287Z"}
kilo {"caller":"main.go:274","msg":"Starting Kilo network mesh 'ee480dece4ceab3fd68b1f4a09e4e67da25003a6'.","ts":"2021-12-08T00:19:34.838064595Z"}
kilo {"caller":"mesh.go:277","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-12-08T00:19:35.045470243Z"}
kilo {"caller":"mesh.go:279","component":"kilo","event":"add","level":"debug","msg":"processing local node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"k8s-simple-uwlih","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-12-08T00:19:35.045566238Z"}
kilo {"caller":"mesh.go:396","component":"kilo","level":"debug","msg":"local node differs from backend","ts":"2021-12-08T00:19:35.046097141Z"}
kilo {"caller":"mesh.go:402","component":"kilo","level":"debug","msg":"successfully reconciled local node against backend","ts":"2021-12-08T00:19:35.056785065Z"}
kilo {"caller":"mesh.go:277","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-12-08T00:19:35.05690891Z"}
kilo {"caller":"mesh.go:288","component":"kilo","event":"add","in-mesh":false,"level":"debug","msg":"received non ready node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"k8s-simple-uwlik","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-12-08T00:19:35.057043385Z"}
kilo {"caller":"mesh.go:306","component":"kilo","event":"add","level":"info","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"k8s-simple-uwlik","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-12-08T00:19:35.057125143Z"}
kilo {"caller":"mesh.go:277","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-12-08T00:19:35.058249307Z"}
kilo {"caller":"mesh.go:279","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"10.244.1.77","Port":51821},"Key":"VHZxU1o0NEZlYlJEVWE3d1BTblVrbVk0ek40aTZXZDFReXBySndLMktuUT0=","NoInternalIP":false,"InternalIP":{"IP":"10.244.1.77","Mask":"/////w=="},"LastSeen":1638922775,"Leader":false,"Location":"","Name":"k8s-simple-uwlih","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-12-08T00:19:35.05835424Z"}

Inside the pod:

bash-5.0# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: kilo0: <POINTOPOINT,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 500
    link/none 
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether d2:3f:6f:24:9e:0e brd ff:ff:ff:ff:ff:ff link-netnsid 0

bash-5.0# wg
interface: kilo0
  listening port: 42557

There is no errors, pod works fine. If i will setup a interface correctly via ip and wg set commands it starts working. The DO managed k8s uses cillium as cni and I aware that there is no support fot it. Is there any change to make it working only as vpn gateway with support of CR Peer?