[RFE] Support for an explicit and reasonable way for custom whereabouts.kubeconfig path (e.g. k3s)

[dannypv05261]

Is your feature request related to a problem? Please describe.

When whereabouts.kubeconfig is not located in the default CNI folder /etc/cni/net.d/whereabouts.d in host (k3s e.g. /var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig). There is no explicit and reasonable way to configure kubeconfig value in whereabouts.conf due to mixing up the container internal path and the host file path as a variable.

Below is the snapshot of code in install-cni.sh.
With changing to customize path with env variable CNI_CONF_DIR, I can make kubeconfig value ={custom_path}/whereabouts.d/whereabouts.kubeconfig in whereabouts.kubeconfig (WHEREABOUTS_KUBECONFIG_LITERAL), but no whereabouts.kubeconfig and whereabouts.conf will be in host eventually since changing on CNI_CONF_DIR also affect internal container path

CNI_CONF_DIR=${CNI_CONF_DIR:-"/host/etc/cni/net.d"}
WHEREABOUTS_RECONCILER_CRON=${WHEREABOUTS_RECONCILER_CRON:-30 4 * * *}

# Make a whereabouts.d directory (for our kubeconfig)

mkdir -p $CNI_CONF_DIR/whereabouts.d
WHEREABOUTS_KUBECONFIG=$CNI_CONF_DIR/whereabouts.d/whereabouts.kubeconfig
WHEREABOUTS_FLATFILE=$CNI_CONF_DIR/whereabouts.d/whereabouts.conf # Yuki~ Nikhil's note: imo we should remove "flatfile" from whereabouts vocabulary and call this "WHEREABOUTS_CONF_FILE" instead. Flatfile may be the format but it's confusing naming.
WHEREABOUTS_KUBECONFIG_LITERAL=$(echo "$WHEREABOUTS_KUBECONFIG" | sed -e s'|/host||')

.....

  touch $WHEREABOUTS_FLATFILE
  chmod ${KUBECONFIG_MODE:-600} $WHEREABOUTS_FLATFILE
  cat > $WHEREABOUTS_FLATFILE <<EOF
{
  "datastore": "kubernetes",
  "kubernetes": {
    "kubeconfig": "${WHEREABOUTS_KUBECONFIG_LITERAL}"
  },
  "reconciler_cron_expression": "${WHEREABOUTS_RECONCILER_CRON}"
}

The workaround I used below is to make container internal CNI folder path same as host folder path by charging the volumeMount path cni-net-dir and adding env variable CNI_CONF_DIR with same path as below, but it is not a good practice for users to change volumeMount path. The good practice should have the fixed internal path /host/etc/cni/net.d, users map their volume to the cni-net-dir and define custom settings in env variable.

      env:
        - name: CNI_CONF_DIR
          value: "/var/lib/rancher/k3s/agent/etc/cni/net.d"

        volumeMounts:
        - name: cni-net-dir
          # mountPath: /host/etc/cni/net.d
          mountPath: /var/lib/rancher/k3s/agent/etc/cni/net.d
        - name: cron-scheduler-configmap
          mountPath: /cron-schedule

    volumes:
        - name: cnibin
          hostPath:
            #path: /opt/cni/bin
            path: /var/lib/rancher/k3s/data/current/bin
        - name: cni-net-dir
          hostPath:
            #path: /etc/cni/net.d
            path: /var/lib/rancher/k3s/agent/etc/cni/net.d

Another workaround is to create a link file from /etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig to the custom path. But it seems not a good idea to apply additional changes in host.

Describe the solution you'd like

In install-cni.sh, change variable from WHEREABOUTS_KUBECONFIG_LITERAL to the existing env variable WHEREABOUTS_KUBECONFIG_FILE_HOST

  touch $WHEREABOUTS_FLATFILE
  chmod ${KUBECONFIG_MODE:-600} $WHEREABOUTS_FLATFILE
  cat > $WHEREABOUTS_FLATFILE <<EOF
{
  "datastore": "kubernetes",
  "kubernetes": {
    "kubeconfig": "${WHEREABOUTS_KUBECONFIG_FILE_HOST}"
  },
  "reconciler_cron_expression": "${WHEREABOUTS_RECONCILER_CRON}"
}
EOF

I submitted a request for that in case it is useful. #466

In this way, if I need to use the custom path, the only changes in daemonset-install.yaml are in volumes and env. It aligns with the general practice of the deployment file. It will not affect people using the default path as well.

      env:
        - name: WHEREABOUTS_KUBECONFIG_FILE_HOST
          value: "/var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"

      volumes:
        - name: cnibin
          hostPath:
            #path: /opt/cni/bin
            path: /var/lib/rancher/k3s/data/current/bin
        - name: cni-net-dir
          hostPath:
            #path: /etc/cni/net.d
            path: /var/lib/rancher/k3s/agent/etc/cni/net.d

Describe alternatives you've considered
Maybe mark the workaround in doc / README.md if this PR doesn't accept.

Additional context

[manuelbuil]

I recently updated the K3s docs and now there is a guide on how to deploy Multus + Whereabouts setting the correct path. Please have a look: https://docs.k3s.io/networking/multus-ipams. It'd be great if you could give it a try!

[koendelaat]

@manuelbuil

I recently updated the K3s docs and now there is a guide on how to deploy Multus + Whereabouts setting the correct path. Please have a look: https://docs.k3s.io/networking/multus-ipams. It'd be great if you could give it a try!

I've tested your guide on how to deploy Multus + Whereabouts. However this doesn't work for me out of the box.
Modifications I needed to make it work:

1. Create symlink for /etc/cni/net.d/whereabouts.d/ to /var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/

I think this makes sense, given this line in config.go:

confdirs := []string{"/etc/kubernetes/cni/net.d/whereabouts.d/whereabouts.conf", "/etc/cni/net.d/whereabouts.d/whereabouts.conf", "/host/etc/cni/net.d/whereabouts.d/whereabouts.conf"}

Without this modification, I get the following error: error adding container to network <network-name> : config file not found

And to my understanding, this is relative to the node/host, not from any pod.

[dannypv05261]

@manuelbuil Sorry for missing your message and thanks for the effort to k3s community. I just run your helm in a separate k3s cluster, since I don't want to touch my existing environment that works well with workaround. Unfortunately, the result as @koendelaat mentioned.

Back to my initial intention, I am not asking for a workable solution on K3S, but I am raising a request to enhance the configurable CNI path. Correct me if I am wrong, the path looks configurable with 'volumes' section in daemonset-install.yaml and the env variable CNI_CONF_DIR in install-cni.sh but it seems not the case now. It looks misleading and ispent me some time on figuring out the issue. If I am just overthinking and the project actually has no plan to support variable CNI_CONF_DIR, it is fine to remain unchanged.

[koendelaat]

I'm looking for a solution for K3s, so created the issue there: k3s-io/k3s#10545