Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatibility between ENABLE_SUBNET_DISCOVERY and ENABLE_PREFIX_DELEGATION? #2959

Closed
lorengordon opened this issue Jun 17, 2024 · 8 comments
Closed

Comments

@lorengordon
Copy link

lorengordon commented Jun 17, 2024

What happened:

Setting ENABLE_PREFIX_DELEGATION=true and ENABLE_SUBNET_DISCOVERY=true does not appear to allocate IPs from secondary subnets tagged with kubernetes.io/role/cni=1. I'm pretty new to K8S and EKS, so rather likely I'm doing something wrong. But if I change ENABLE_PREFIX_DELEGATION to false, then the IPs are allocated from the secondary subnets.

To get prefix delegation working with the secondary subnets, I can set things up using AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone and an ENIConfig custom resource for each availability zone. But I was hoping to simplify things by just tagging the subnets with kubernetes.io/role/cni=1 and removing the custom network config.

Environment:

  • Kubernetes version (use kubectl version): v1.30.0-eks-036c24b
  • CNI Version: v1.18.2-eksbuild.1
  • OS (e.g: cat /etc/os-release): Amazon Linux 2023.4.20240528
  • Kernel (e.g. uname -a): Linux ip-10-200-2-107 6.1.91-99.172.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 21 16:55:08 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
@orsenthil
Copy link
Member

Setting ENABLE_PREFIX_DELEGATION=true and ENABLE_SUBNET_DISCOVERY=true does not appear to allocate IPs from secondary subnets tagged with kubernetes.io/role/cni=1.

There shouldn't be any dependency between these two features. What you are likely running into is this open issue that we have with Enhanced Subnet Discovery - #2908 ; which is being fixed right now - #2954

@lorengordon
Copy link
Author

lorengordon commented Jun 18, 2024

@orsenthil I certainly don't know enough to say otherwise, but reading #2908, the writeup doesn't sound like it's related. My primary subnet is not exhausted, it does have IP space, a /28 is getting assigned, and the pods are created and running. It's just that the /28 is getting assigned from the wrong subnet, one that is not tagged.

  • When ENABLE_PREFIX_DELEGATION=true, it appears to ignore subnets tagged with kubernetes.io/role/cni=1
  • When ENABLE_PREFIX_DELEGATION=false, it assigns IPs from the tagged subnets

Anyway, hopefully we're saying the same thing, and #2954 does indeed resolve my issue also!

@jchen6585
Copy link
Contributor

jchen6585 commented Jun 18, 2024

@lorengordon When ENABLE_PREFIX_DELEGATION=true, is a secondary ENI being created? The tagged subnets will only be taken into account when a secondary ENI is being created. A secondary ENI is only created if existing ENI's have breached the IP capacity.

@lorengordon
Copy link
Author

lorengordon commented Jun 18, 2024

Oh, interesting. No, a secondary ENI is not being created. It's just assigning prefixes to the primary ENI for the node. Maybe this is intended behavior at the moment. The confusing part to me about the behavior in that case would be that ENABLE_SUBNET_DISCOVERY is enabled, but it is assigning IPs out of a subnet that is not tagged, even though there are subnets that are tagged.

However, if I disable ENABLE_PREFIX_DELEGATION, then the IPs are assigned from the tagged subnets, as expected. So, I guess I'd expect ENABLE_PREFIX_DELEGATION should interact with ENABLE_SUBNET_DISCOVERY, and consider exclusively the tagged subnets when deciding whether to create an ENI and assign prefixes?

@jchen6585
Copy link
Contributor

jchen6585 commented Jun 18, 2024

The subnets considered are the subnet the node is created in and subnets tagged with kubernetes.io/role/cni. from https://github.com/aws/amazon-vpc-cni-k8s?tab=readme-ov-file#enable_subnet_discovery-v1180. It is expected to assign from the default subnet until the primary ENI is exhausted. Once the primary ENI is exhausted, the secondary ENI will be created in the tagged subnet assuming 1/ The tagged subnet has the most # of free IPs 2/ The tagged subnet contains a contiguous /28 CIDR (if prefix delegation is enabled).

It sounds like you are trying to force some pods onto a certain subnet, in which Custom Networking would fit that use case better.

@lorengordon
Copy link
Author

lorengordon commented Jun 18, 2024

Hmm. Ok, possibly, maybe I'm doing this all wrong, and I need to use the secondary subnets for the node group primary interface?

It sounds like you are trying to force some pods onto a certain subnet, in which Custom Networking would fit that use case better.

To the degree that some is a subset of all, yeah....

Yes, I am trying to manage CIDR allocations that are routed across a larger corporate environment, while also allowing EKS to deploy 1000s of pods and minimizing the impact on the corporate CIDR. So, yeah, the primary VPC CIDR is only /24 (routed), and the secondary VPC CIDR is /16 (not routed). Before ENABLE_SUBNET_DISCOVERY was released, I was using Custom Networking, with the nodes' primary interfaces deployed in the primary VPC CIDR and pods deployed in the secondary VPC CIDR. Using ENABLE_PREFIX_DELEGATION and Custom Networking with CRDs for the ENIConfig, it seemed to work. Maybe it wasn't working though, and I misunderstood what was happening; I still feel very lost when it comes to understanding what K8S is doing. Maybe I was doing that wrong also! I guess I don't know what happens if the node group primary interfaces aren't routed...

Edit: Tried deploying the node group to the secondary subnets, but that didn't work without routing. I can get it working using a Private NAT Gateway. Not sure that's worth it. Sounding like back to Custom Networking with ENIConfig CRDs :(

@lorengordon
Copy link
Author

I'll go ahead and close this. The combination of ENABLE_PREFIX_DELEGATION and ENABLE_SUBNET_DISCOVERY simply do not work as I'd hoped, but they seem to be working "as intended". I can make things work the way I'd like using Custom Networking, so I'll keep that approach instead.

Copy link

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants