diff --git a/docs/administrator/security/certificate-configuration.md b/docs/administrator/security/certificate-configuration.md new file mode 100644 index 00000000..5e8188a6 --- /dev/null +++ b/docs/administrator/security/certificate-configuration.md @@ -0,0 +1,334 @@ +--- +title: Certificate Configuration +--- + +## Certificate Configuration + +Karmada components use digital certificates to verify each other's identities, preventing malicious third parties from impersonating and stealing information or attacking the system. During mutual authentication with digital certificates between two components, the following files are involved: + +- Business Certificate (Client/Server): A certificate used to prove one's own identity. +- Private Key: The private key corresponding to the public key included in the business certificate. +- CA Root Certificate: The root certificate issued by the Certificate Authority (CA) that signed the business certificate. + +Both the business certificate and the CA root certificate have an expiration date. Therefore, when a business certificate or a CA root certificate is about to expire or has already expired, new certificates need to be replaced to ensure that the components can operate normally. + +### Prerequisites + +You need to install the following tools: + +- `openssl` ([Installation Guide](https://github.com/openssl/openssl#download)) +- `cfssl` ([Installation Guide](github.com/cloudflare/cfssl/cmd)) +- `cfssljson` ([Installation Guide](github.com/cloudflare/cfssl/cmd)) + + + +> NOTE: The following examples are based on the installation method described in [Installation from Source](https://karmada.io/docs/installation/fromsource/#install). When using other installation methods, appropriate adaptation is required. + +### Karmada Certificates + +Karmada certificate storage path: Default to `${HOME}/.karmada`. + +The current certificates used by Karmada include: + +```shell +$ tree +. +├── apiserver.crt +├── apiserver.key +├── ca-config.json +├── ca.crt +├── ca.key +├── etcd-ca-config.json +├── etcd-ca.crt +├── etcd-ca.key +├── etcd-client.crt +├── etcd-client.key +├── etcd-server.crt +├── etcd-server.key +├── front-proxy-ca-config.json +├── front-proxy-ca.crt +├── front-proxy-ca.key +├── front-proxy-client.crt +├── front-proxy-client.key +├── karmada.crt +└── karmada.key +``` + +#### **Karmada Certificate Overview** + +Certificates can be categorized into three sets based on the issuing CA: + +- Issued by ca + + The certificates `apiserver` and `karmada` are both issued by the CA certificate `ca`. The key attributes of these certificates are as follows: + + | Certificates | common name(CN) | organization(og) | hosts | + | ------------ | ----------------- | ---------------- | ------------------------------------------------------------ | + | apiserver | karmada-apiserver | / | "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" "${api server's ip from kubecomfig by context karmada host}" | + | karmada | system:admin | system:masters | kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" "${interpreter_webhook_example_service_external_ip_address}" | + + The `${api server's ip from kubecomfig by context karmada host}` can be obtained by executing `./get_server_ip.sh ${HOST_CLUSTER_NAME}`. + + ```shell + #!/usr/bin/env bash + # get_server_ip.sh + context_name=$1 + cluster_name=$(kubectl config view --template='{{ range $_, $value := .contexts }}{{if eq $value.name '"\"${context_name}\""'}}{{$value.context.cluster}}{{end}}{{end}}') + apiserver_url=$(kubectl config view --template='{{range $_, $value := .clusters }}{{if eq $value.name '"\"${cluster_name}\""'}}{{$value.cluster.server}}{{end}}{{end}}') + echo "${apiserver_url}" | awk -F/ '{print $3}' | sed 's/:.*//' + ``` + + > PS: The `karmada api server config` in the `kubeconfig` file is composed of the `karmada` certificate. + +- Issued by etcd-ca + + The certificates `etcd-server` and `etcd-client` are both issued by the CA certificate `etcd-ca`. The key attributes of these certificates are as follows: + + | certificates | common name(CN) | organization(og) | hosts | + | ------------ | --------------- | ---------------- | ------------------------------------------------------------ | + | etcd-server | etcd-server | / | kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" | + | etcd-client | etcd-client | / | "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" | + +- Issued by front-proxy-ca + + The certificate `front-proxy-client` is issued by the CA certificate `front-proxy-ca`. The key attributes of these certificates are as follows: + + | certificates | common name(CN) | organization(og) | hosts | + | ------------------ | ------------------ | ---------------- | ------------------------------------------------------------ | + | front-proxy-client | front-proxy-client | / | kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" | + +#### How Karmada Components Use Certificates + +Karmada components use certificates by mounting secrets. The current secrets used by Karmada for storing certificates are: + +- [karmada-cert-secret](https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/karmada-cert-secret.yaml) +- [kubeconfig](https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/secret.yaml) + +- [karmada-webhook-cert-secret](https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/karmada-webhook-cert-secret.yaml) + +You can find the corresponding certificates for each secret from the [cert-secret-generation](https://github.com/karmada-io/karmada/blob/19d1146c3510942809f48d399fc2079ce3a79a66/hack/deploy-karmada.sh#L102-L124). + +Karmada components acquire the necessary certificates by mounting secrets. The detailed usage of certificates by each component is as follows: + +- **etcd** mounts the secret `karmada-cert-secret` and uses the certificates for + + ```yaml + - --cert-file=/etc/karmada/pki/etcd-server.crt + - --key-file=/etc/karmada/pki/etcd-server.key + - --trusted-ca-file=/etc/karmada/pki/etcd-ca.crt + ``` + +- **karmada-apiserver** mounts the secret `karmada-cert-secret` and uses the certificates for: + + ```yaml + - --client-ca-file=/etc/karmada/pki/ca.crt + - --etcd-cafile=/etc/karmada/pki/etcd-ca.crt + - --etcd-certfile=/etc/karmada/pki/etcd-client.crt + - --etcd-keyfile=/etc/karmada/pki/etcd-client.key + - --kubelet-client-certificate=/etc/karmada/pki/karmada.crt + - --kubelet-client-key=/etc/karmada/pki/karmada.key + - --service-account-key-file=/etc/karmada/pki/karmada.key + - --service-account-signing-key-file=/etc/karmada/pki/karmada.key + - --proxy-client-cert-file=/etc/karmada/pki/front-proxy-client.crt + - --proxy-client-key-file=/etc/karmada/pki/front-proxy-client.key + - --requestheader-client-ca-file=/etc/karmada/pki/front-proxy-ca.crt + - --tls-cert-file=/etc/karmada/pki/apiserver.crt + - --tls-private-key-file=/etc/karmada/pki/apiserver.key + ``` + +- **karmada-controller-manager** mounts the secret `kubeconfig` and uses the certificates for + + ```yaml + - --kubeconfig=/etc/kubeconfig + ``` + +- **karmada-aggregated-apiserver** mounts the secrets karmada-cert-secret and kubeconfig and uses the certificates for: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --authentication-kubeconfig=/etc/kubeconfig + - --authorization-kubeconfig=/etc/kubeconfig + - --etcd-cafile=/etc/karmada/pki/etcd-ca.crt + - --etcd-certfile=/etc/karmada/pki/etcd-client.crt + - --etcd-keyfile=/etc/karmada/pki/etcd-client.key + - --tls-cert-file=/etc/karmada/pki/karmada.crt + - --tls-private-key-file=/etc/karmada/pki/karmada.key + ``` + +- **karmada-scheduler-estimator** mounts the secret `karmada-cert-secret` and uses the certificates for: + + ```yaml + - --grpc-auth-cert-file=/etc/karmada/pki/karmada.crt + - --grpc-auth-key-file=/etc/karmada/pki/karmada.key + - --grpc-client-ca-file=/etc/karmada/pki/ca.crt + ``` + +- **karmada-scheduler** mounts the secrets `karmada-cert-secret` and `kubeconfig` and uses the certificates for: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --scheduler-estimator-ca-file=/etc/karmada/pki/ca.crt + - --scheduler-estimator-cert-file=/etc/karmada/pki/karmada.crt + - --scheduler-estimator-key-file=/etc/karmada/pki/karmada.key + ``` + +- **karmada-descheduler** mounts the secrets `karmada-cert-secret` and `kubeconfig` and uses the certificates for: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --scheduler-estimator-ca-file=/etc/karmada/pki/ca.crt + - --scheduler-estimator-cert-file=/etc/karmada/pki/karmada.crt + - --scheduler-estimator-key-file=/etc/karmada/pki/karmada.key + ``` + +- **karmada-metrics-adapter** mounts the secrets `karmada-cert-secret` and `kubeconfig` and uses the certificates for: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --authentication-kubeconfig=/etc/kubeconfig + - --authorization-kubeconfig=/etc/kubeconfig + - --client-ca-file=/etc/karmada/pki/ca.crt + - --tls-cert-file=/etc/karmada/pki/karmada.crt + - --tls-private-key-file=/etc/karmada/pki/karmada.key + ``` + +- **karmada-search** mounts the secrets `karmada-cert-secret` and `kubeconfig` and uses the certificates for: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --authentication-kubeconfig=/etc/kubeconfig + - --authorization-kubeconfig=/etc/kubeconfig + - --etcd-cafile=/etc/karmada/pki/etcd-ca.crt + - --etcd-certfile=/etc/karmada/pki/etcd-client.crt + - --etcd-keyfile=/etc/karmada/pki/etcd-client.key + - --tls-cert-file=/etc/karmada/pki/karmada.crt + - --tls-private-key-file=/etc/karmada/pki/karmada.key + ``` + +- **karmada-webhook** mounts the secrets `webhook-cert` and `kubeconfig` and uses the certificates for: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --cert-dir=/var/serving-cert + ``` + +#### Checking Certificate Expiry Dates + +You can use the command `openssl x509 -noout -dates -in path/to/cert` to extract the before and after dates of the certificate. + +```bash +$ openssl x509 -noout -dates -in path/to/cert +notBefore=Aug 24 16:00:00 2024 GMT +notAfter=Aug 29 16:00:00 2024 GMT +``` + +The output indicates that this certificate becomes valid at 16:00 on August 24, 2024, and expires at 16:00 on August 29, 2024. + + +### **Certificate Replacement** + +When a business certificate expires or is about to expire, you can manually replace it with a new certificate to extend its validity. + +#### generate a new certificate + +1. **Determine the issuing CA of the expired certificate.** Refer to the **Karmada Certificate Overview** for the relationship between business certificates and their issuing CAs. + +2. **Obtain the Common Name (CN) and other details from the expired certificate.** To ensure consistency before and after the replacement, it is recommended to directly inspect the expired business certificate to obtain the `CN` and other details. + + ```shell + $ openssl x509 -noout -text -in path/to/cert + Certificate: + Data: + ... ... + Subject: O = system:masters, CN = system:admin + ... ... + X509v3 Subject Alternative Name: + DNS:kubernetes.default.svc, DNS:*.etcd.karmada-system.svc.cluster.local, DNS:*.karmada-system.svc.cluster.local, DNS:*.karmada-system.svc, DNS:localhost, IP Address:127.0.0.1 + ... ... + ``` + + The output of the command indicates that the expired certificate has the following details: `CN=system:admin`, `O=system:masters`, and `hosts=kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1"`. + +3. **Use the CA and certificate information to issue a new certificate.** + + ```shell + #!/usr/bin/env bash + # signCert.sh + dest_dir=$1 + ca=$2 + ca_dir=$3 + cert_name=$4 + cn=${5:-$4} + og=$6 + hosts="" + SEP="" + shift 6 + while [[ -n "${1:-}" ]]; do + hosts+="${SEP}\"$1\"" + SEP="," + shift 1 + done + ${sudo} /usr/bin/env bash -e < NOTE: 以下案例均基于从[Installation from Source](https://karmada.io/docs/installation/fromsource/#install)的安装方式进行讲述,采用其他安装方式时需进行相应的适配 + +### Karmada 证书 + +Karmada 证书存储路径: 默认为${HOME}/.karmada + +当前 karmada 所使用的证书共有: + +```shell +$ tree +. +├── apiserver.crt +├── apiserver.key +├── ca-config.json +├── ca.crt +├── ca.key +├── etcd-ca-config.json +├── etcd-ca.crt +├── etcd-ca.key +├── etcd-client.crt +├── etcd-client.key +├── etcd-server.crt +├── etcd-server.key +├── front-proxy-ca-config.json +├── front-proxy-ca.crt +├── front-proxy-ca.key +├── front-proxy-client.crt +├── front-proxy-client.key +├── karmada.crt +└── karmada.key +``` +#### Karmada 证书简介 + +依据证书所签发的 CA可以分为三套证书: + +- 由ca签发 + + 证书 apiserver和karmada均由CA ca证书签发,证书的关键属性如下: + + | 证书 | common name(CN) | organization(og) | hosts | + | --------- | ----------------- | ---------------- | ------------------------------------------------------------ | + | apiserver | karmada-apiserver | / | "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" "${api server's ip from kubecomfig by context karmada host}" | + | karmada | system:admin | system:masters | kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" "${interpreter_webhook_example_service_external_ip_address}" | + + 其中${api server's ip from kubecomfig by context karmada host}可以通过执行 + + ```shell + #!/usr/bin/env bash + context_name=$1 + cluster_name=$(kubectl config view --template='{{ range $_, $value := .contexts }}{{if eq $value.name '"\"${context_name}\""'}}{{$value.context.cluster}}{{end}}{{end}}') + apiserver_url=$(kubectl config view --template='{{range $_, $value := .clusters }}{{if eq $value.name '"\"${cluster_name}\""'}}{{$value.cluster.server}}{{end}}{{end}}') + echo "${apiserver_url}" | awk -F/ '{print $3}' | sed 's/:.*//' + ``` + + 得到。 +PS: kubeconfig file中的karmada api server config所使用的证书是karmada证书 + +- 由etcd-ca签发 + + 证书etcd-client和etcd-server均由CA etcd-ca证书签发,证书的关键属性如下: + + | 证书 | common name(CN) | organization(og) | hosts | + | ----------- | --------------- | ---------------- | ------------------------------------------------------------ | + | etcd-server | etcd-server | / | kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" | + | etcd-client | etcd-client | / | "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" | + +- 由front-proxy-ca签发 + + 证书 front-proxy-client由CA front-proxy-ca证书签发,证书的关键属性如下: + + | 证书 | common name(CN) | organization(og) | hosts | + | ------------------ | ------------------ | ---------------- | ------------------------------------------------------------ | + | front-proxy-client | front-proxy-client | / | kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" | + +#### Karmada 组件如何使用证书 +karmada通过secret来store证书。当前Karmada用于store证书的secret有: + +- [karmada-cert-secret](https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/karmada-cert-secret.yaml) +- [kubeconfig](https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/secret.yaml) + +- [karmada-webhook-cert-secret](https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/karmada-webhook-cert-secret.yaml) + +可从[cert-secret-generation](https://github.com/karmada-io/karmada/blob/19d1146c3510942809f48d399fc2079ce3a79a66/hack/deploy-karmada.sh#L102-L124) 查找各secret所对应的证书。 + +Karmada 组件通过挂载 secret来获取所需证书,各组件的证书使用详情如下: + +- etcd + + 挂载的secret: karmada-cert-secret + + 证书使用场景: + + ```yaml + - --cert-file=/etc/karmada/pki/etcd-server.crt + - --key-file=/etc/karmada/pki/etcd-server.key + - --trusted-ca-file=/etc/karmada/pki/etcd-ca.crt + ``` + +- karmada apiserver + + 挂载的secret: karmada-cert-secret + + 证书使用场景: + + ```yaml + - --client-ca-file=/etc/karmada/pki/ca.crt + - --etcd-cafile=/etc/karmada/pki/etcd-ca.crt + - --etcd-certfile=/etc/karmada/pki/etcd-client.crt + - --etcd-keyfile=/etc/karmada/pki/etcd-client.key + - --kubelet-client-certificate=/etc/karmada/pki/karmada.crt + - --kubelet-client-key=/etc/karmada/pki/karmada.key + - --service-account-key-file=/etc/karmada/pki/karmada.key + - --service-account-signing-key-file=/etc/karmada/pki/karmada.key + - --proxy-client-cert-file=/etc/karmada/pki/front-proxy-client.crt + - --proxy-client-key-file=/etc/karmada/pki/front-proxy-client.key + - --requestheader-client-ca-file=/etc/karmada/pki/front-proxy-ca.crt + - --tls-cert-file=/etc/karmada/pki/apiserver.crt + - --tls-private-key-file=/etc/karmada/pki/apiserver.key + ``` + +- karmada controller manager + + 挂载的secret: kubeconfig + + 证书的使用场景: + + ```yaml + - --kubeconfig=/etc/kubeconfig + ``` + +- karmada aggregated apiserver + + 挂载的secret: karmada-cert-secret 和 kubeconfig + + 证书的使用场景: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --authentication-kubeconfig=/etc/kubeconfig + - --authorization-kubeconfig=/etc/kubeconfig + - --etcd-cafile=/etc/karmada/pki/etcd-ca.crt + - --etcd-certfile=/etc/karmada/pki/etcd-client.crt + - --etcd-keyfile=/etc/karmada/pki/etcd-client.key + - --tls-cert-file=/etc/karmada/pki/karmada.crt + - --tls-private-key-file=/etc/karmada/pki/karmada.key + ``` + +- karmada-scheduler-estimator + + 挂载的secret: karmada-cert-secret + + 证书的使用场景: + + ```yaml + - --grpc-auth-cert-file=/etc/karmada/pki/karmada.crt + - --grpc-auth-key-file=/etc/karmada/pki/karmada.key + - --grpc-client-ca-file=/etc/karmada/pki/ca.crt + ``` + +- karmada-scheduler + + 挂载的secret: karmada-cert-secret 和 kubeconfig + + 证书的使用场景: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --scheduler-estimator-ca-file=/etc/karmada/pki/ca.crt + - --scheduler-estimator-cert-file=/etc/karmada/pki/karmada.crt + - --scheduler-estimator-key-file=/etc/karmada/pki/karmada.key + ``` + +- karmada-descheduler + 挂载的secret: karmada-cert-secret 和 kubeconfig + + 证书的使用场景: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --scheduler-estimator-ca-file=/etc/karmada/pki/ca.crt + - --scheduler-estimator-cert-file=/etc/karmada/pki/karmada.crt + - --scheduler-estimator-key-file=/etc/karmada/pki/karmada.key + ``` + +- karmada-metrics-adapter + + 挂载的secret:karmada-cert-secret 和 kubeconfig + + 证书的使用场景: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --authentication-kubeconfig=/etc/kubeconfig + - --authorization-kubeconfig=/etc/kubeconfig + - --client-ca-file=/etc/karmada/pki/ca.crt + - --tls-cert-file=/etc/karmada/pki/karmada.crt + - --tls-private-key-file=/etc/karmada/pki/karmada.key + ``` + +- karmada-search + + 挂载的secret:karmada-cert-secret 和 kubeconfig + + 证书的使用场景: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --authentication-kubeconfig=/etc/kubeconfig + - --authorization-kubeconfig=/etc/kubeconfig + - --etcd-cafile=/etc/karmada/pki/etcd-ca.crt + - --etcd-certfile=/etc/karmada/pki/etcd-client.crt + - --etcd-keyfile=/etc/karmada/pki/etcd-client.key + - --tls-cert-file=/etc/karmada/pki/karmada.crt + - --tls-private-key-file=/etc/karmada/pki/karmada.key + ``` + +- karmada-webhook + + 挂载的secret:webhook-cert 和 kubeconfig + + 证书的使用场景: + + ```yaml + - --kubeconfig=/etc/kubeconfig + - --cert-dir=/var/serving-cert + ``` + +#### 查看证书有效期 + +通过命令`openssl x509 -noout -dates -in path/to/cert`可以解析出证书的before和after dates. + +```bash +$ openssl x509 -noout -dates -in path/to/cert +notBefore=Aug 24 16:00:00 2024 GMT +notAfter=Aug 29 16:00:00 2024 GMT +``` + +以上输出结果表明,此证书在2024年8月24号16点生效,到2024年8月29号16点失效。 + + +### 业务证书的更换 + +当业务证书过期或即将过期时,可以手动替换新证书来获得更久的使用期限。 + +#### 生成新的证书 + +1. 确定过期证书的签发CA + + [Karmada 证书简介](#Karmada 证书简介)介绍了业务证书和CA的对应关系,据此可以确定过期证书的签发CA + +2. 获取过期证书的CN等信息 + + 业务证书的CN、host等信息可能会与环境有关,比如apiserver证书的hosts值。因此为了保证证书替换前后的功能完全一致,建议直接查看已过期的业务证书来获取证书的CN等信息。 + + ```shell + $ openssl x509 -noout -text -in path/to/cert + Certificate: + Data: + ... ... + Subject: O = system:masters, CN = system:admin + ... ... + X509v3 Subject Alternative Name: + DNS:kubernetes.default.svc, DNS:*.etcd.karmada-system.svc.cluster.local, DNS:*.karmada-system.svc.cluster.local, DNS:*.karmada-system.svc, DNS:localhost, IP Address:127.0.0.1 + ... ... + ``` + + 过期证书的CN=system:admin, O = system:masters, hosts=kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" + +3. 利用CA和证书信息签发新的证书 + + ```bash + #!/usr/bin/env bash + # signCert.sh + dest_dir=$1 + ca=$2 + ca_dir=$3 + cert_name=$4 + cn=${5:-$4} + og=$6 + hosts="" + SEP="" + shift 6 + while [[ -n "${1:-}" ]]; do + hosts+="${SEP}\"$1\"" + SEP="," + shift 1 + done + ${sudo} /usr/bin/env bash -e <