GEP: Gateway Routability

[dprotaso]

Discussed in #1247

^{Originally posted by evankanderson June 30, 2022}
Currently, the GatewayAddress field supports only external (LoadBalanced) Gateway definitions. I'd like to suggest adding a ClusterLocal or ClusterIP AddressType with Extended support (like the existing IPAddress and Hostname support).

One of the early feature requests for Knative was the ability to deploy an application using Knative's HTTP routing support, but make it only available within the cluster. I want to be able to specify both the "internal" (service.namespace.svc) and "external" (service.namespace.example.com) Gateways using the same GatewayClass on the cluster, but ensure that the "internal" service is only reachable within the cluster. This would greatly simplify deployment for users over the instructions we have today.

[robscott]

Although this was on pace to be released as experimental in v0.8.0 (thanks to lots of great work by @dprotaso), the initial API review with @thockin and @khenidak raised some important questions that I don't think we have sufficient time to resolve before the v0.8.0 release.

I think the most important points are:

1. "Public" and "Private" are hard to define
2. Those become especially hard to define when combined with the incoming Kubernetes multi-network initiative

Until we can answer these questions, we've decided to pull this GEP back to provisional with these specific questions marked as "unresolved". Unfortunately that means this will miss the v0.8.0 release.

[Rycieos]

I think any abstraction we try to make here will inherently be too limiting. If we make some field that accepts either Public or Private, inherently someone will need some different behavior.

For myself, I have needed to set these fields on the Service.Spec:

- type
- loadBalancerClass
- loadBalancerIP
- ipFamilyPolicy
- loadBalancerSourceRanges
- sessionAffinity

While a Gateway now has the Infrastructure.annotations field, not all of these settings are supported as annotations, and those that are are implementation and cloud provider specific.

My current workaround is to let the Gateway create a Service, and then clone that service with the same selector, but set my own values for the other fields. Then I use that Service instead of the one the Gateway creates.

What has been proposed before in the linked discussion is a a ref in the Gateway to a Service, with is used either as a template for a Service that is created, or that is taken ownership of.

The second option, ownership, would solve another problem (which may be better suited in a separate issue) which is how can we make a Gateway use an already existing Load Balancer.

That said, the problem with a reference is dependency ordering.

My proposal is to add a new field, Gateway.Spec.ServiceSpec, which would mirror the Service.Spec, and would instruct the Gateway how to program the Service it creates.

Related: #912

[mikemorris]

The second option, ownership, would solve another problem (which may be better suited in a separate issue) which is how can we make a Gateway use an already existing Load Balancer.

This has a dedicated open discussion over in #1687

[tculp]

I agree with @Rycieos. In particular, I would like to be able to set loadBalancerSourceRanges on created services. Right now the lack of that capability is preventing me from switching Istio deployments to use the Gateway API

[dprotaso]

@Rycieos @tculp I believe this PR where the parameterRef in the Gateway will give you the knobs you want

#2924

[howardjohn]

IMO a first class field is still highly desirable. parameterRef is a vendor-specific escape hatch; we shouldn't settle to make every vendor implement core functionality in different ways

[Rycieos]

I agree with @howardjohn. The strength of Gateway API is in how much of the API is standardized, meaning documentation is correct for all providers, and a user can switch providers with ease. I think any of the use cases mentioned in this thread will be common enough that having them as per provider configuration will be a detriment to the project as a whole.

[tculp]

In addition, fields like loadBalancerSourceRanges started out as annotations but were promoted to spec-level fields because they were generic and useful enough to be specifically included. I think a case could be made for all spec-level fields to be supported.

Imagine if Deployments didn't support a top-level pod-spec field in the template, that would be crazy.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tculp]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[dprotaso]

@robscott you mentioned cluster IP gateways in the meeting today - I'd view that as a subset of this GEP. Really it's what I'm looking for.

Can you elaborate what you're thinking?

[NFRBEZ]

I agree with @tculp

I have the case over GKE where i would like to restrict access to the gateway to some defined IP ranges, which work by using loadBalancingSourceRanges.
Without ability to do it from gateway definition, i have to go on my service and to update the ranges by hand, which is a non sense

[mikemorris]

@tculp @NFRBEZ I think restricting access to a Gateway based on IP ranges is more aligned with the scope of #1141 than this issue, which is focused more on assigning an in-cluster or private network address to a Gateway instead of an external public address.

My proposal is to add a new field, Gateway.Spec.ServiceSpec, which would mirror the Service.Spec, and would instruct the Gateway how to program the Service it creates.

@Rycieos The difficulty with this is that Gateway is not defined as a resource which creates Services in the same way Deployments create Pods. Rather, an underlying Service is a (relatively common) implementation detail of some Gateway API implementations, and consolidating the user-facing configuration onto the Gateway resource should likely be the goal, while exposing implementation details which aren't part of the spec definition is likely an anti-pattern we'd prefer to avoid.

[Rycieos]

@Rycieos The difficulty with this is that Gateway is not defined as a resource which creates Services in the same way Deployments create Pods. Rather, an underlying Service is a (relatively common) implementation detail of some Gateway API implementations, and consolidating the user-facing configuration onto the Gateway resource should likely be the goal, while exposing implementation details which aren't part of the spec definition is likely an anti-pattern we'd prefer to avoid.

That is a very good point. I still think that load balancer configuration is a common use case that needs to be supported, but having a way for implementations to not support or ignore that config would be needed.