Skip to content

Commit

Permalink
update egci crd
Browse files Browse the repository at this point in the history 
Signed-off-by: bzsuni <[email protected]>
  • Loading branch information
@bzsuni
bzsuni committed Aug 2, 2023
1 parent 3324215 commit 67aaef6
Show file tree
Hide file tree
Showing 14 changed files with 727 additions and 572 deletions.
38 changes: 19 additions & 19 deletions charts/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,25 +23,25 @@ helm install egressgateway egressgateway/egressgateway --namespace kube-system

### Feature parameters

| Name | Description | Value |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| `feature.enableIPv4` | Enable IPv4 | `true` |
| `feature.enableIPv6` | Enable IPv6 | `false` |
| `feature.datapathMode` | iptables mode, [`iptables`, `ebpf`] | `iptables` |
| `feature.tunnelIpv4Subnet` | Tunnel IPv4 subnet | `172.31.0.0/16` |
| `feature.tunnelIpv6Subnet` | Tunnel IPv6 subnet | `fd11::/112` |
| `feature.tunnelDetectMethod` | Tunnel base on which interface [`defaultRouteInterface`, `interface=eth0`] | `defaultRouteInterface` |
| `feature.iptables.backendMode` | Iptables mode can be specified as `nft` or `legacy`, with `auto` meaning automatic detection. The default value is `auto`. | `auto` |
| `feature.vxlan.name` | The name of VXLAN device | `egress.vxlan` |
| `feature.vxlan.port` | VXLAN port | `7789` |
| `feature.vxlan.id` | VXLAN ID | `100` |
| `feature.vxlan.disableChecksumOffload` | Disable checksum offload | `true` |
| `feature.egressIgnoreCIDR.autoDetect.podCIDR` | cni cluster used | `calico` |
| `feature.egressIgnoreCIDR.autoDetect.clusterIP` | if ignore service ip | `true` |
| `feature.egressIgnoreCIDR.autoDetect.nodeIP` | if ignore node ip | `true` |
| `feature.egressIgnoreCIDR.custom` | CIDRs provided manually | `[]` |
| `feature.maxNumberEndpointPerSlice` | max number of endpoints per slice | `100` |
| `feature.announcedInterfacesToExclude` | The list of network interface excluded for announcing Egress IP. | `["^cali.*","br-*"]` |
| Name | Description | Value |
| -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| `feature.enableIPv4` | Enable IPv4 | `true` |
| `feature.enableIPv6` | Enable IPv6 | `true` |
| `feature.datapathMode` | iptables mode, [`iptables`, `ebpf`] | `iptables` |
| `feature.tunnelIpv4Subnet` | Tunnel IPv4 subnet | `172.31.0.0/16` |
| `feature.tunnelIpv6Subnet` | Tunnel IPv6 subnet | `fd11::/112` |
| `feature.tunnelDetectMethod` | Tunnel base on which interface [`defaultRouteInterface`, `interface=eth0`] | `defaultRouteInterface` |
| `feature.iptables.backendMode` | Iptables mode can be specified as `nft` or `legacy`, with `auto` meaning automatic detection. The default value is `auto`. | `auto` |
| `feature.vxlan.name` | The name of VXLAN device | `egress.vxlan` |
| `feature.vxlan.port` | VXLAN port | `7789` |
| `feature.vxlan.id` | VXLAN ID | `100` |
| `feature.vxlan.disableChecksumOffload` | Disable checksum offload | `true` |
| `feature.clusterCIDR.autoDetect.podCidrMode` | cni cluster used, it can be specified as `k8s`, `calico` or `""` | `k8s` |
| `feature.clusterCIDR.autoDetect.clusterIP` | if ignore service ip | `true` |
| `feature.clusterCIDR.autoDetect.nodeIP` | if ignore node ip | `true` |
| `feature.clusterCIDR.extraCidr` | CIDRs provided manually | `[]` |
| `feature.maxNumberEndpointPerSlice` | max number of endpoints per slice | `100` |
| `feature.announcedInterfacesToExclude` | The list of network interface excluded for announcing Egress IP. | `["^cali.*","br-*"]` |

### Egressgateway agent parameters

Expand Down
90 changes: 56 additions & 34 deletions charts/crds/egressgateway.spidernet.io_egressclusterinfos.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,44 +36,66 @@ spec:
metadata:
type: object
spec:
type: object
status:
properties:
egressIgnoreCIDR:
autoDetect:
properties:
clusterIP:
properties:
ipv4:
items:
type: string
type: array
ipv6:
items:
type: string
type: array
type: object
default: true
type: boolean
nodeIP:
properties:
ipv4:
items:
type: string
type: array
ipv6:
items:
type: string
type: array
type: object
podCIDR:
properties:
ipv4:
items:
type: string
type: array
ipv6:
items:
type: string
type: array
type: object
default: true
type: boolean
podCidrMode:
default: k8s
type: string
type: object
extraCidr:
items:
type: string
type: array
type: object
status:
properties:
clusterIP:
properties:
ipv4:
items:
type: string
type: array
ipv6:
items:
type: string
type: array
type: object
extraCidr:
items:
type: string
type: array
nodeIP:
additionalProperties:
properties:
ipv4:
items:
type: string
type: array
ipv6:
items:
type: string
type: array
type: object
type: object
podCIDR:
additionalProperties:
properties:
ipv4:
items:
type: string
type: array
ipv6:
items:
type: string
type: array
type: object
type: object
type: object
required:
Expand Down
10 changes: 10 additions & 0 deletions charts/templates/egressClusterInfo.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressClusterInfo
metadata:
name: default
spec:
autoDetect:
clusterIP: {{ .Values.feature.clusterCIDR.autoDetect.clusterIP }}
podCidrMode: {{ .Values.feature.clusterCIDR.autoDetect.podCidrMode }}
nodeIP: {{ .Values.feature.clusterCIDR.autoDetect.nodeIP }}
extraCidr: {{ .Values.feature.clusterCIDR.extraCidr }}
14 changes: 7 additions & 7 deletions charts/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,16 +46,16 @@ feature:
id: 100
## @param feature.vxlan.disableChecksumOffload Disable checksum offload
disableChecksumOffload: true
egressIgnoreCIDR:
clusterCIDR:
autoDetect:
## @param feature.egressIgnoreCIDR.autoDetect.podCIDR cni cluster used
podCIDR: "calico"
## @param feature.egressIgnoreCIDR.autoDetect.clusterIP if ignore service ip
## @param feature.clusterCIDR.autoDetect.podCidrMode cni cluster used, it can be specified as `k8s`, `calico` or `""`
podCidrMode: "k8s"
## @param feature.clusterCIDR.autoDetect.clusterIP if ignore service ip
clusterIP: true
## @param feature.egressIgnoreCIDR.autoDetect.nodeIP if ignore node ip
## @param feature.clusterCIDR.autoDetect.nodeIP if ignore node ip
nodeIP: true
## @param feature.egressIgnoreCIDR.custom CIDRs provided manually
custom: []
## @param feature.clusterCIDR.extraCidr CIDRs provided manually
extraCidr: []
## @param feature.maxNumberEndpointPerSlice max number of endpoints per slice
maxNumberEndpointPerSlice: 100
## @param feature.announcedInterfacesToExclude The list of network interface excluded for announcing Egress IP.
Expand Down
80 changes: 44 additions & 36 deletions docs/proposal/04-auto-detect-egress-ignore-cidr/README_zh-CN.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,63 +2,71 @@

## 动机

为了简化 Egress 策略的配置，引入 Egress Ignore CIDR 功能，允许以手动和自动的方式获取集群的 CIDR。当 EgressGatewayPolicy 的 `destSubnet` 字段为空时，数据面将会自动匹配 EgressClusterStatus CR 中的 CIDR 之外的流量，并将其转发到 Egress 网关。
为了简化 Egress 策略的配置，引入 Egress Ignore CIDR 功能，允许以手动和自动的方式获取集群的 CIDR。当 EgressGatewayPolicy 的 `destSubnet` 字段为空时，数据面将会自动匹配 EgressClusterInfo CR 中的 CIDR 之外的流量，并将其转发到 Egress 网关。

## 目标

* 优化 EgressGatewayPolicy 使用体验

## 设计

### 配置文件

修改配置文件，增加如下配置：

```yaml
feature:
egressIgnoreCIDR:
autoDetect:
podCIDR: "" # 1
clusterIP: true # 2
nodeIP: true # 3
custom:
- "10.6.1.0/24"
```
1. 支持设置为支持 kube-ovn, calico, k8s 等；
2. 支持设置为 Service CIDR 自动检测；
3. 支持设置为 Node IP 自动检测，当新加一个节点时，自动将节点的的所有 IP 更新到 EgressClusterStatus。
### EgressClusterStatus CRD
### EgressClusterInfo CRD

集群级 CRD。

```yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressClusterStatus
kind: EgressClusterInfo
metadata:
name: "default" # 1
name: default # 1
spec:
autoDetect:
clusterIP: true # 2
nodeIP: true # 3
podCidrMode: k8s # 4
extraCidr: # 5
- 10.10.10.1
status:
egressIgnoreCIDR:
nodeIP:
clusterIP: # 6
ipv4:
- 172.41.0.0/16
ipv6:
- fd41::/108
extraCidr: # 7
- 10.10.10.1
nodeIP: # 8
egressgateway-control-plane:
ipv4:
- 172.18.0.3
ipv6:
- fc00:f853:ccd:e793::3
egressgateway-worker:
ipv4:
- "10.6.0.1"
- 172.18.0.2
ipv6:
- "fd00::1"
clusterIP:
- fc00:f853:ccd:e793::2
egressgateway-worker2:
ipv4:
- "10.6.0.1"
- 172.18.0.4
ipv6:
- "fd00::1"
podCIDR:
- fc00:f853:ccd:e793::4
podCIDR: # 9
k8s:
ipv4:
- "10.6.0.0/24"
- 172.40.0.0/16
ipv6:
- "fd00::1/122"
- fd40::/48
```
1. 名称为 `default`，由系统维护只能创建一个;
2. 根据 `egressIgnoreCIDR.autoDetect` 配置检测出的集群 CIDR 或 IP。
2. `clusterIP`，如果设置为 `true`，`Service CIDR` 会自动检测
3. `nodeIP`，如果设置为 `true`，会自动检测 `nodeIP` 相关变化，并动态更新到 `EgressClusterInfo` 的 `status.nodeIP` 中
4. `podCidrMode`，目前支持 `k8s`、 `calico`、 `""`，表示 自动检测对应的 podCidr，默认为 `k8s`，如果为 `""` 表示不检测
5. `extraCidr`，可手动填写要忽略掉的 `IP` 集合
6. `status.clusterIP`，如果 `spec.autoDetect.clusterIP` 为 `true`，则自动检测集群 `Service CIDR`，并更新到此处
7. `status.extraCidr`，对应 `spec.extraCidr`
8. `status.nodeIP`，如果 `spec.autoDetect.nodeIP` 为 `true`，则自动检测集群 `nodeIP`，并更新到此处
9. `status.podCIDR`，对应 `spec.autoDetect.podCidrMode`，进行相关 `podCidr` 的更新

### 数据面策略

Expand All @@ -77,9 +85,9 @@ iptables -A EGRESSGATEWAY-MARK-REQUEST -t mangle -m conntrack --ctdir ORIGINAL \

#### Controller

新增一个控制循环，根据 `egressIgnoreCIDR.autoDetect` 配置来 Watch 集群的相关资源，更新自动检测的 CIDR 到 EgressClusterStatus CR 的 `status.egressIgnoreCIDR` 中。
新增一个控制循环，根据 `spec.autoDetect` 配置来 Watch 集群的相关资源，更新自动检测的 CIDR 到 EgressClusterInfo CR 的 `status` 中。

#### Agent

* 在 Policy 的控制循环中，处理 EgressClusterStatus 更新到名为 `egress-ingore-cidr` 的 ipset 中；
* 在 Policy 的控制循环中，处理 EgressClusterInfo 更新到名为 `egress-ingore-cidr` 的 ipset 中；
* 对于 `destSubnet` 字段为空时的 EgressGatewayPolicy 策略，使用 `egress-ingore-cidr` 的 ipset 匹配流量。
61 changes: 40 additions & 21 deletions docs/reference/EgressClusterInfo.en.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,33 +4,52 @@ The EgressClusterInfo CRD introduces the Egress Ignore CIDR feature to simplify
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressClusterInfo
metadata:
name: "default" # 1
spec: {}
name: default # 1
spec:
autoDetect:
clusterIP: true # 2
nodeIP: true # 3
podCidrMode: k8s # 4
extraCidr: # 5
- 10.10.10.1
status:
egressIgnoreCIDR: # 2
clusterIP: # 3
clusterIP: # 6
ipv4:
- 172.41.0.0/16
ipv6:
- fd41::/108
extraCidr: # 7
- 10.10.10.1
nodeIP: # 8
egressgateway-control-plane:
ipv4:
- "172.41.0.0/16"
- 172.18.0.3
ipv6:
- "fd41::/108"
nodeIP:
- fc00:f853:ccd:e793::3
egressgateway-worker:
ipv4:
- "172.18.0.3"
- "172.18.0.4"
- "172.18.0.2"
- 172.18.0.2
ipv6:
- "fc00:f853:ccd:e793::3"
- "fc00:f853:ccd:e793::4"
- "fc00:f853:ccd:e793::2"
podCIDR:
- fc00:f853:ccd:e793::2
egressgateway-worker2:
ipv4:
- "172.40.0.0/16"
- 172.18.0.4
ipv6:
- "fd40::/48"
- fc00:f853:ccd:e793::4
podCIDR: # 9
k8s:
ipv4:
- 172.40.0.0/16
ipv6:
- fd40::/48
```
1. The name defaults to `default`, maintained by the system, only one can be created, and it cannot be modified.
2. `egressIgnoreCIDR` defines the CIDR that EgressGateway should ignore.
3. `clusterIP` is the default service-cluster-ip-range for the cluster. Whether it is enabled is specified by the EgressGateway configuration file's default `egressIgnoreCIDR.autoDetect.clusterIP`.
4. `nodeIP` is the collection of IP addresses for the cluster nodes (only taking the IP from the Node yaml `status.address`, in the case of multiple network cards, other network card IPs are treated as external IPs). Whether it is enabled is specified by the EgressGateway configuration file's default `egressIgnoreCIDR.autoDetect.nodeIP`.
5. `podCIDR` is the CIDR used by the cluster's CNI. It is specified by the egressgateway configuration file's default `egressIgnoreCIDR.autoDetect.podCIDR`.
1. The name is `default`, only one can be created by the system maintenance;
2. `clusterIP`, if set to `true`, `Service CIDR` will be detected automatically
3. `nodeIP`, if it is set to `true`, it will automatically detect changes related to `nodeIP` and dynamically update it to `status.nodeIP` of `EgressClusterInfo`
4. `podCidrMode`, currently supports `k8s`, `calico`, `""`, which means automatic detection of the corresponding podCidr, the default is `k8s`, if it is `""`, it means no detection
5. `extraCidr`, you can manually fill in the `IP` set to be ignored
6. `status.clusterIP`, if `spec.autoDetect.clusterIP` is `true`, then automatically detect the cluster `Service CIDR`, and update here
7. `status.extraCidr`, corresponding to `spec.extraCidr`
8. `status.nodeIP`, if `spec.autoDetect.nodeIP` is `true`, then automatically detect cluster `nodeIP`, and update here
9. `status.podCIDR`, corresponding to `spec.autoDetect.podCidrMode`, update related `podCidr`
Loading

0 comments on commit 67aaef6

Please sign in to comment.