How to add private CA

[Vertiwell]

Hi, is it possible to add a private CA chain cert for trust?
I added mine by ConfigMap to /etc/ssl/certs/, however still getting:
http: TLS handshake error from 10.42.8.197:45314: remote error: tls: bad certificate

[canterberry]

Thanks for asking!

In your scenario, what is the client and what is the server? This will help advise on where/how to wire up the CA trust anchors.

i.e: is the error happening when trying to connect to the registry or from the registry? The more detail you can provide about the client and server certificate setup you have now, the better! Without, of course, sharing any sensitive key/certificate content.

[Vertiwell]

Hi, thanks for responding!
The error is when I try to connect to the registry with:
echo ${DPASS} | docker login https://${SUBDOMAIN} --username ${DUSER} --password-stdin

The certificate comes from Hashicorp Vault CA, via a Cert-Manager ClusterIssuer, terminated by a Traefik IngressRoute on a MetalLB load balancer.

My OS is Ubuntu.

This script installs the helm chart:

#!/bin/bash
### Deploying Docker Registry on Kubernetes for Debian/Ubuntu based OS
## Baseline Guide: https://faun.pub/install-a-private-docker-container-registry-in-kubernetes-7fb25820fc61
# Type of Deployment: Helm
#
## Upgrade: helm repo update && helm upgrade registry twuni/docker-registry --install -n kube-system -f /root/helm/registry/custom-values.yaml --version 2.1.0
#
### Minimum Requirements ###
## Kubernetes Cluster
## Cert-Manager
## Storage
#
## The following base packages are required:
# Helm, Package Manager
# curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
# chmod 700 get_helm.sh
# ./get_helm.sh
#
# Docker.io is required to login to the local private repository
apt-get install docker.io -y
# Htpasswd is used to create and update the flat-files used to store usernames and password
apt-get install apache2-utils -y
### Installation ###
#
## Bring in the variables from the main configuration file
# DOMAIN=example.com
# ISSUER=ca-issuer
# STORAGECLASS=longhorn
source /root/repo/deployment_config
# Set a directory to store files generated by this script (used for upgrades later)
mkdir -p /root/helm/registry
## Set Variables:
# Provide a domain
SUBDOMAIN=registry.${DOMAIN}
# Chart Version: helm repo update && helm search repo twuni/docker-registry --versions
CHARTVERSION=2.1.0
# Set directory as a variable
FILEDIR=/root/helm/registry
# Namespace for this application
NAMESPACE=registry
# Create Namespace
cat <<EOF >${FILEDIR}/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ${NAMESPACE}
EOF
# Apply namespace
kubectl apply -f ${FILEDIR}/namespace.yaml
#
# Create a username for the registry
DUSER=$(pwgen -csn 24 1) && \
# Create a password for the registry
DPASS=$(pwgen -csn 24 1) && \
# Create secret to store credentials
cat <<EOF >${FILEDIR}/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: docker-registry-secret
  namespace: ${NAMESPACE}
type: Opaque
stringData:
  docker-username: "${DUSER}"
  docker-password: "${DPASS}"
EOF
# Apply secret
kubectl apply -f ${FILEDIR}/secret.yaml
# Create a password file to be loaded into Kubernetes
htpasswd -Bbc ${FILEDIR}/registry.password ${DUSER} ${DPASS} >/dev/null 2>&1
# Add this password file to the K3S cluster using a secret
kubectl create secret -n ${NAMESPACE} generic docker-registry-htpasswd --from-file ${FILEDIR}/registry.password >/dev/null 2>&1
# Create the TLS Certificate (standard template against name.whatever.com your domain is)
cat <<EOF >${FILEDIR}/tls-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tls-cert
  namespace: ${NAMESPACE}
spec:
  commonName: ${SUBDOMAIN}
  secretName: tls-cert
  dnsNames:
    - ${SUBDOMAIN}
    - registry
    - registry.registry.svc
    - registry.registry.svc.cluster.local
  issuerRef:
    name: ${ISSUER}
    kind: ClusterIssuer
EOF
# Apply the certificate
kubectl apply -f ${FILEDIR}/tls-cert.yaml
# Add the Helm Chart
helm repo add twuni https://helm.twun.io && helm repo update
# Create custom values file
# Helm Chart Values: https://github.com/twuni/docker-registry.helm/blob/main/values.yaml
cat <<EOF >${FILEDIR}/custom-values.yaml
replicaCount: 3
persistence:
  accessMode: 'ReadWriteOnce'
  enabled: true
  size: 5Gi
  storageClass: '${LOCAL}'
tlsSecretName: tls-cert
secrets:
  haSharedSecret: ""
  htpasswd: "docker-registry-htpasswd"
extraVolumeMounts:
  - name: ca-store
    mountPath: /etc/ssl/certs/custom-ca.crt
    subPath: custom-ca.crt
    readOnly: true
extraVolumes:
  - name: ca-store
    configMap:
      name: trust-operator
EOF
# Install the Helm Chart with custom values:
helm upgrade registry twuni/docker-registry --install -n ${NAMESPACE} -f ${FILEDIR}/custom-values.yaml --create-namespace --version ${CHARTVERSION}
# Clean up
# rm files containing secrets or passwords
# Wipe everything
# kubectl delete ns registry; rm -r /root/helm/registry

[Vertiwell]

This script exposes the registry service:

#!/bin/bash
### Docker Registry Access
### Minimum Requirements ###
## Requires docker-registry script
## Requires a Load Balancer (MetalLB) - See script
## Requires an Ingress Controller (Traefik) - See script
#
### Installation Steps ###
## Bring in the variables from the main configuration file
source /root/repo/deployment_config
## Set Variables:
# Set a subdomain to use
SUBDOMAIN=registry.${DOMAIN}
# Set directory as a variable
FILEDIR=/root/helm/registry
# Namespace
NAMESPACE=registry
#
# Create the Certificate (standard template against name.whatever.com your domain is)
cat <<EOF >${FILEDIR}/registry-dashboard-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: registry-dashboard-cert
  namespace: ${NAMESPACE}
spec:
  commonName: ${SUBDOMAIN}
  secretName: registry-dashboard-cert
  dnsNames:
    - ${SUBDOMAIN}
  issuerRef:
    name: ${ISSUER}
    kind: ClusterIssuer
EOF
# Create the IngressRoute to direct traffic to your application
cat <<EOF >${FILEDIR}/registry-dashboard-ingress.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: registry-dashboard-ingress
  namespace: ${NAMESPACE}
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(\`${SUBDOMAIN}\`) && PathPrefix(\`/\`)
    kind: Rule
    services:
    - name: registry-docker-registry
      port: 5000
  tls:
      secretName: registry-dashboard-cert
EOF
# Deploy Certificate and Ingress to cluster
kubectl apply $(ls ${FILEDIR}/registry-*.yaml | awk ' { print " -f " $1 } ') && sleep 5 && \
# Test
printf ""$(kubectl -n kube-system get svc traefik -o json | jq -r .status.loadBalancer.ingress[].ip)" "${SUBDOMAIN}"\n" >> /etc/hosts && \
# Get username and password from file
export DUSER=$(kubectl get secret -n registry docker-registry-secret -o jsonpath="{.data.docker-username}" | base64 --decode) && \
export DPASS=$(kubectl get secret -n registry docker-registry-secret -o jsonpath="{.data.docker-password}" | base64 --decode) && \
# Create Secret
kubectl create secret docker-registry secret --docker-server=${SUBDOMAIN} --docker-username=${DUSER} --docker-password=${DPASS} --namespace=${NAMESPACE} >/dev/null 2>&1
# Restart docker
systemctl daemon-reload && systemctl restart docker && sleep 5 && \
### Login to your private registry to test it works
echo ${DPASS} | docker login https://${SUBDOMAIN} --username ${DUSER} --password-stdin && \
# Cleanup
rm /root/.docker/config.json
# Wipe everything: kubectl delete IngressRoute docker-registry-service-dashboard-ingress -n registry; kubectl delete certificate docker-registry-service-dashboard-cert -n registry; kubectl delete secret docker-registry-service-dashboard-cert -n registry

[canterberry]

Thanks! That's super helpful. So, it sounds like what you're looking for is how to tell your Docker client which CA to trust when connecting to your registry, which happens to be using a private CA.

I didn't know how to do that, but thinking of it that way helped me find this similar issue: containerd/containerd#3737. I also came across docker/for-mac#4100 (comment) for the format of the Docker configuration file (typically ~/.docker/config.json) where docker login ... ultimately writes (unless the credential store is something else, like Docker Desktop for Mac).

Ultimately, you need to tell the Docker CLI that it needs to use your desired CA when trying to connect to your registry.

Based on those two issues, your Docker config might look something like this:

{
  "auths": {
    "your-private-registry-domain": {
      "auth": "base64(username:password)",
      "tls": {
        "caFile": "/path/to/your-private-ca-certificate.pem"
      }
    }
  }
}

I haven't personally tried that, but based on the links cited above, that's my guess on how to get the Docker CLI to use your private CA certificate for that registry. If that doesn't work, I hope it at least sets you on a path toward the answer, and when you find one, please share!

Also, if anyone else has run into this issue and knows the answer, please chime in. I'll keep this issue open until we know how to make it work.

[Vertiwell]

I will test it out cheers, I do not require this however if I install the docker registry itself:

#!/bin/bash
### Deploying Docker Registry on Kubernetes for Debian/Ubuntu based OS
## Baseline Guide: https://www.nearform.com/blog/how-to-run-a-public-docker-registry-in-kubernetes/
## Baseline Guide: https://faun.pub/install-a-private-docker-container-registry-in-kubernetes-7fb25820fc61
# Type of Deployment: Script
#
### Minimum Requirements ###
## Three Worker Node Cluster
## Cert-Manager - See Script
#
## The following base packages are required:
# Remove docker.sock if already in use
rm -rf /var/run/docker.sock
# Docker.io is required to login to the local private repository
apt-get install docker.io -y
# Htpasswd is used to create and update the flat-files used to store usernames and password
apt-get install apache2-utils -y
#
### Installation Steps ###
## Bring in the variables from the main configuration file
source /root/repo/deployment_config
# Set a directory to store files generated by this script (used for upgrades later)
mkdir -p /root/scripts/registry
## Set Variables:
# Counter for loops
n=0
# Set directory as a variable
FILEDIR=/root/scripts/registry
# Namespace
NAMESPACE=registry
# Create Namespace
cat <<EOF >${FILEDIR}/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ${NAMESPACE}
EOF
# Apply namespace
kubectl apply -f ${FILEDIR}/namespace.yaml
# Create a username for the registry
DUSER=$(pwgen -csn 24 1) && \
# Create a password for the registry
DPASS=$(pwgen -csn 24 1) && \
# Create secret to store credentials
cat <<EOF >${FILEDIR}/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: docker-registry-secret
  namespace: ${NAMESPACE}
type: Opaque
stringData:
  docker-username: "${DUSER}"
  docker-password: "${DPASS}"
EOF
# Apply secret
kubectl apply -f ${FILEDIR}/secret.yaml
# Create a password file to be loaded into Kubernetes
htpasswd -Bbc ${FILEDIR}/registry.password ${DUSER} ${DPASS} >/dev/null 2>&1
# Add this password file to the K3S cluster using a secret
kubectl create secret -n ${NAMESPACE} generic docker-registry-htpasswd --from-file ${FILEDIR}/registry.password >/dev/null 2>&1
# Create Service
cat <<EOF >${FILEDIR}/docker-registry-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: docker-registry-service
  namespace: ${NAMESPACE}
spec:
  selector:
    app: docker-registry
  ports:
    - protocol: TCP
      port: 5000
EOF
# Apply Service
kubectl apply -f ${FILEDIR}/docker-registry-service.yaml && \
# Create a persistant volume
cat <<EOF >${FILEDIR}/docker-registry-storage.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: registry
  namespace: ${NAMESPACE}
spec:
  storageClassName: ${LOCAL}
  resources:
    requests:
      storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
EOF
# Apply Storage
kubectl apply -f ${FILEDIR}/docker-registry-storage.yaml
# Create Deployment
cat <<EOF >${FILEDIR}/docker-registry-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-registry
  namespace: ${NAMESPACE}
  labels:
    app: docker-registry
spec:
  replicas: 3
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
      - name: docker-registry
        image: registry:2.7.0
        ports:
        - containerPort: 5000
        volumeMounts:
        - name: storage
          mountPath: /var/lib/registry
        - name: htpasswd
          mountPath: ~/docker-registry/auth
        - name: ca-store
          mountPath: /etc/ssl/certs/custom-ca.crt
          subPath: custom-ca.crt
          readOnly: true
        env:
        - name: REGISTRY_AUTH
          value: htpasswd
        - name: REGISTRY_AUTH_HTPASSWD_REALM
          value: Docker Registry
        - name: REGISTRY_AUTH_HTPASSWD_PATH
          value: ~/docker-registry/auth/registry.password
        - name: REGISTRY_STORAGE_DELETE_ENABLED
          value: "true"
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: registry
      - name: htpasswd
        secret:
         secretName: docker-registry-htpasswd
      - name: ca-store
        configMap:
          name: trust-operator
EOF
# Apply Deployment
kubectl apply -f ${FILEDIR}/docker-registry-deployment.yaml
# Wait until Registry is running before moving on
ROLLOUT_STATUS_CMD="kubectl rollout status -w --timeout=300s deployment/docker-registry -n "${NAMESPACE}""
until $ROLLOUT_STATUS_CMD || [ $n -eq 300 ]; do
   $ROLLOUT_STATUS_CMD
   n=$((n + 1))
   sleep 5
done
# Create weekly clean up of images tasks
printf "0 9 * * TUE    root    docker rm -vf \$(docker ps -aq)\n" >> /etc/crontab >/dev/null 2>&1
printf "0 9 * * TUE    root    docker rmi -f \$(docker images -aq)\n" >> /etc/crontab >/dev/null 2>&1
printf "0 9 * * TUE    root    docker volume prune -f\n\n" >> /etc/crontab >/dev/null 2>&1
# Cleanup, don't forget to save the creds from temp-pass-registry and delete it.
rm ${FILEDIR}/secret.yaml ${FILEDIR}/registry.password /root/.docker/config.json >/dev/null 2>&1
# To get the username and pasword:
# kubectl get secret -n registry docker-registry-secret -o jsonpath="{.data.docker-username}" | base64 --decode; kubectl get secret -n registry docker-registry-secret -o jsonpath="{.data.docker-password}" | base64 --decode
# Wipe Everything
# kubectl delete ns registry; rm -r /root/scripts/registry

[Vertiwell]

Doing it this way lets me in no issue:

root@engineering:/home/user# echo ${DPASS} | docker login https://${SUBDOMAIN} --username ${DUSER} --password-stdin
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
root@engineering:/home/user# cat /root/.docker/config.json
{
        "auths": {
                "registry.example.com": {
                        "auth": "SDSzejBvWlVVOGZheONPNmJ0RWn1FSOjdXS12Y3akY3cGrrVndOMzhXSEcwcng5Q=="
                }
        }

[Vertiwell]

What is the purpose of haSharedSecret?

[Vertiwell]

Ok I understand where I went wrong, apologies, I terminated the SSL connection at the Traefik IngressRoute, which then I guess opens a non-SSL connection to docker, which fails.
Changing this to either Server Transport, to establish another SSL connection to docker, or using IngressRouteTCP with tls passthrough fixes this and login succeeds.
I also had to add the htpasswd to the values file for the helm chart, originally I was using the name of the secret.
Sorted...