Can AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG be generalized for multi-homed pods?

[constant-mihai]

What happened:

I would like to have multi-homed pods without having to use multus. I come from a telecom background and this requirement comes from needing to implement data routers which sit in multiple subnets.

I know that I can use AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG to create a secondary network interface for the pod.
I would like to understand better if it was ever considered to generalize this concept to support more than just a secondary network interface.

By looking at the code, this seems possible.

func MyENIConfig(ctx context.Context, k8sClient client.Client) (*v1alpha1.ENIConfigSpec, error) {

Could be extended to retrieve a list of v1alpha1.ENIConfigSpec. After which we iterate through them and allocate an ENI for each element in the list in tryAllocateENI.

routed-eni-cni-plugin would need to create veth pairs for the extra ENIs attached on the node. Not every pod will need to be in the networks configured in custom networking. We would need to pass in configuration to the routed-eni-cni-plugin for it to only assign IPs (and veth pairs) to the pods which need it. It’s still unclear what would be the best way to do this. Kubernetes annotations?

Lastly, I see that the CNI plugin does not instantiate a kubernetes client, but instead sends RPC requests to amazon-k8s-agent.

service CNIBackend {
  rpc AddNetwork (AddNetworkRequest) returns (AddNetworkReply) {}
  rpc DelNetwork (DelNetworkRequest) returns (DelNetworkReply) {}
}

This API would probably need to be extended.

I would love to hear if you've ever considered this or if you see any major road blocks in developing such functionality.

Thanks!

Environment:
Not relevant to the question.

[achevuru]

AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG is not meant to create secondary interfaces to pods. It is just a mode that will allow you to place your pods in a different subnet other than the primary subnet of the node (i.e.,) your pods will be assigned IPs from a subnet that belongs to say a secondary CIDR of your VPC.

A single CNI plugin ideally shouldn't be configuring multiple interfaces for a pod and it is best that it is handled by chained plugins (that are paired with their own IPAM plugins). If you don't want to rely on Multus, you can just chain the CNI plugins in the config file.

Also, why would a CNI plugin need to instantiate a K8S client? I don't think I understood the Q. At it's core, these are just general Linux container networking plugins and they have no specific dependency on K8S and the rpc interface you pointed out is between the CNI and it's IPAM plugin(aws-k8-agent).

[constant-mihai]

AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG is not meant to create secondary interfaces to pods. It is just a mode that will allow you to place your pods in a different subnet other than the primary subnet of the node (i.e.,) your pods will be assigned IPs from a subnet that belongs to say a secondary CIDR of your VPC.

I see! Yes, this is a different requirement than mine.

A single CNI plugin ideally shouldn't be configuring multiple interfaces for a pod and it is best that it is handled by chained plugins (that are paired with their own IPAM plugins). If you don't want to rely on Multus, you can just chain the CNI plugins in the config file.

So, let's say that I would want a second, third (aso) ENI attached to the node and that I would want to configure these interfaces the same way routed-eni-cni-plugin configures them. How would I achieve that? Can I chain routed-eni-cni-plugin a second time?

Also, why would a CNI plugin need to instantiate a K8S client? I don't think I understood the Q. At it's core, these are just general Linux container networking plugins and they have no specific dependency on K8S and the rpc interface you pointed out is between the CNI and it's IPAM plugin(aws-k8-agent).

It doesn't. I was trying (maybe poorly) to list the extensions I see required to support configuring multiple interfaces. The RPC interface possibly being one of them. AFAIU, the CNI should not configure multiple interfaces, so I'm not sure if this point is worth debating further.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[constant-mihai]

Closing this since it seems that either I'm way off mark here or extending the AWS-CNI the way I proposed is not desired.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.