diff --git a/geps/gep-1762.md b/geps/gep-1762.md new file mode 100644 index 0000000000..1a890cdb4b --- /dev/null +++ b/geps/gep-1762.md @@ -0,0 +1,248 @@ +# GEP-1762: In Cluster Gateway Deployments + +* Status: Provisional + +## Overview + +Gateway API provides a common abstraction over different implementations, whether they are implemented by cloud load balancers, in-cluster deployments, or other mechanisms. However, many in-cluster implementations have solved some of the same problems in different ways. + +Related discussions: + +* [Support cluster-local Gateways](https://github.com/kubernetes-sigs/gateway-api/discussions/1247) +* [Scaling Gateway Resources](https://github.com/kubernetes-sigs/gateway-api/discussions/1355) +* [Manual deployments](https://github.com/kubernetes-sigs/gateway-api/issues/1687) +* [Merging Gateways](https://github.com/kubernetes-sigs/gateway-api/pull/1863/) +* [Per-Gateway Infrastructure](https://github.com/kubernetes-sigs/gateway-api/pull/1757) + +## Goals + +* Provide prescriptive guidance for how in-cluster implementations should behave. +* Provide requirements for how in-cluster implementations should behave. + +Note that some changes will be suggestions, while others will be requirements. + +## Non-Goals + +* Provide guidance to how out-of-cluster implementations should behave. Rather, this document aims to bring consistency between these types. + +## Terminology + +This document uses a few terms throughout. To ensure consistency, they are defined below: + +* In-cluster deployment: refers to an implementation that actuates a `Gateway` by running a data plane in the cluster. + This is *often*, but not necessarily, by deploying a `Deployment`/`DaemonSet` and `Service`. +* Automated deployment: refers to an implementation that automatically deploys the data plane based on a `Gateway`. + That is, the user simply creates a `Gateway` resource and the rest is handled behind the scenes by the implementation. + +## Design + +This GEP both introduces new API fields, and standardizes how implementations should behave when implementing the existing API. + +### Automated Deployments + +A simple `Gateway`, as is configured below is assumed to be an automated deployment: + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: my-gateway +spec: + gatewayClassName: example + listeners: + - name: default + port: 80 + protocol: HTTP +``` + +With this configuration, an implementation: + +* MUST mark the Gateway as `Programmed` and provide an address in `Status.Addresses` where the Gateway can be reached on each configured port. +* MUST label all generated resources (Service, Deployment, etc) with `gateway.networking.k8s.io/gateway-name: my-gateway` (where `my-gateway` is the name of the Gateway resource). +* MUST provision generated resources in the same namespace as the Gateway if they are namespace scoped resources. + * Cluster scoped resources are not recommended. +* SHOULD name all generated resources `my-gateway-example` (`-`). + This is not simply `NAME` to reduce the chance of conflicts with existing resources. + Where required, this can also serve as the prefix for the object. + +### Customizations + +With any in-cluster deployment, customization requirements will arise. + +Some common requirements would be: + +* `Service.spec.type`, to control whether a service is a `ClusterIP` or `LoadBalancer`. +* IP in the Service to assign to it. +* Arbitrary labels and annotations on generated resources. +* Any other arbitrary fields; the list is unbounded. Some examples would be: + * CPU and memory requests + * Service `externalTrafficPolicy` + * Affinity rules. + +This GEP currently only aims to solve a subset of these concerns. Additional concerns may be addressed in future revisions or other GEPs. + +#### Gateway Type + +This is handled by [GEP-1651](https://github.com/kubernetes-sigs/gateway-api/pull/1653), so won't be described here. + +#### Gateway IP + +This section just clarifies and existing part of the spec, how to handle `.spec.addresses` for in-cluster implementations. +Like all other Gateway types, this should impact the address the `Gateway` is reachable at. + +For implementations using a `Service`, this means the `clusterIP` or `loadBalancerIP` (depending on the `Service` type). + +For example: + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: my-gateway +spec: + addresses: + - type: IPAddress + value: 1.1.1.1 + gatewayClassName: example + listeners: + - name: default + port: 80 + protocol: HTTP +``` + +This would generate a `Service` with `clusterIP` or `loadBalancerIP`, depending on the Service type. + + +#### Labels and Annotations + +Labels and annotations for generated resources are specified in `infrastructure`: + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: my-gateway +spec: + infrastructure: + labels: + foo: bar + annotations: + name: my-annotation +``` + +These are both `map[string]string` types, just like in `ObjectMeta`. + +Any labels or annotations here are added to all generated resources. +Note this may mean an annotation intended for a `Service` may end up on a `Deployment` (for example). +This is typically not a concern; however, if an implementation is aware of specific meanings of certain labels or annotations, they MAY +exclude these from irrelevant resources. + +This is intended to clearly identify resources associated with a specific application, environment, or Gateway. +Additionally, it can be used support integration with the kitchen-sync of Kubernetes extensions which rely on labels and annotations. + +Validation will be added to prevent any usage with `gateway.networking.k8s.io/` prefix, to avoid conflicts with `gateway.networking.k8s.io/gateway-name` or other future additions. + +#### Arbitrary Customization + +GEP-1867 introduces a new `infrastructure` field, which allows customization of some common configurations (version, size, etc) +and allows a per-Gateway generic `parametersRef`. +This can be utilized for the remainder of customizations. + +### Resource Attachment + +Resources Generated in response to the `Gateway` will have two attributes: + +* A `gateway.networking.k8s.io/gateway-name: ` label. +* A name `-`. This format is not strictly required for implementations, but strongly recommended for consistency in attachment. + +The generated resources MUST be in the same namespaces as the `Gateway`. + +Implementations MAY set `ownerReferences` to the `Gateway` in most cases, as well, but this is not required +as some implementations may have different cleanup mechanisms. + +The `gateway.networking.k8s.io/gateway-name` label and standardize resource naming format can be relied on to attach resources to. +While "Policy attachment" in Gateway API would use attachment to the actual `Gateway` resource itself, +many existing resources attach only to resources like `Deployment` or `Service`. + +An example using these: +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: gateway +spec: + gatewayClassName: example + listeners: + - name: default + hostname: "example.com" + port: 80 + protocol: HTTP +--- +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: gateway +spec: + # Match the generated Deployment by reference + # Note: Do not use `kind: Gateway`. + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: gateway-example + minReplicas: 2 + maxReplicas: 5 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 50 +--- +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: gateway +spec: + minAvailable: 1 + selector: + # Match the generated Deployment by label + matchLabels: + gateway.networking.k8s.io/metadata.name: gateway +``` + +Note: there is [discussion](https://github.com/kubernetes-sigs/gateway-api/discussions/1355) around a way to attach a HPA to a Gateway directly. + +## API + +This GEP extends the `infrastructure` API introduced in [GEP-1867](https://gateway-api.sigs.k8s.io/geps/gep-1867). + +```go +type GatewayInfrastructure struct { + // Labels that should be applied to any resources created in response to this Gateway. + // + // For implementations creating other Kubernetes objects, this should be the `metadata.labels` field on resources. + // For other implementations, this refers to any relevant (implementation specific) "labels" concepts. + // + // An implementation may chose to add additional implementation-specific labels as they see fit. + // + // Support: Extended + // +kubebuilder:validation:MaxItems=8 + Labels map[string]string `json:"labels,omitempty"` + // Annotations that should be applied to any resources created in response to this Gateway. + // + // For implementations creating other Kubernetes objects, this should be the `metadata.annotations` field on resources. + // For other implementations, this refers to any relevant (implementation specific) "annotations" concepts. + // + // An implementation may chose to add additional implementation-specific annotations as they see fit. + // + // Support: Extended + // +kubebuilder:validation:MaxItems=8 + Annotations map[string]string `json:"annotations,omitempty"` + ... +} +``` + +## Future Work + +* Allow various policies, [such as HPA](https://github.com/kubernetes-sigs/gateway-api/discussions/1355), to attach directly to `Gateway` rather than just `Deployment`.