Cert-Manager SSL Problem

[ThomasKassiotis]

Hello my cert-manager can't produce the ssl certificates.

kubectl get service nginx-ingress-ingress-nginx-controller

NAME                                     TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
nginx-ingress-ingress-nginx-controller   LoadBalancer   10.43.133.130   xxx.xx.xxx.xx   80:31417/TCP,443:31350/TCP   46h

I have access to Management Portal and all the subdomains (grafana etc.) but I have to call the URL with the custom port else i get 404 error.

• https://domain:31350/managementportal (working)
• https://domain/managementportal (404)

kubectl get certificates

NAME                       READY   SECRET                     AGE
radar-base-tls             False   radar-base-tls             64m
radar-base-tls-s3          False   radar-base-tls-s3          64m
radar-base-tls-dashboard   False   radar-base-tls-dashboard   64m

The logs from cert-manager

kubectl logs cert-manager-756bb56c5-bjp5v -n cert-manager

.... (same error for all domains-subdomains)
"msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="graylog.lethe.ics.forth.gr" "resource_kind"="Challenge" "resource_name"="radar-base-tls-4kvfp-3586578948-1398832694" "resource_namespace"="graylog" "resource_version"="v1" "type"="HTTP-01"
"msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="prometheus.lethe.ics.forth.gr" "resource_kind"="Challenge" "resource_name"="radar-base-tls-prometheus-8c25n-909119021-224032767" "resource_namespace"="monitoring" "resource_version"="v1" "type"="HTTP-01"

I think that cert-manager is not looking at the custom port where ingress-controller is listening so he gets a 404.
Any ideas ? Thanks!

[ThomasKassiotis]

Troubleshooting Issuing ACME Certificates

1. kubectl get certificate

NAME                       READY   SECRET                     AGE
radar-base-tls-s3          False   radar-base-tls-s3          2m44s
radar-base-tls             False   radar-base-tls             2m44s
radar-base-tls-dashboard   False   radar-base-tls-dashboard   2m44s

2. kubectl describe certificate (attached 3 files with the logs)
3. kubectl describe certificaterequest (certificate_request.txt)
4. kubectl get clusterissuer

NAME               READY   AGE
letsencrypt-prod   True    153m

5. kubectl describe clusterissuer letsencrypt-prod (clusterissuer.txt)
6. kubectl get orders

radar-base-tls-dashboard-96ggw-2554460961   pending   14m
radar-base-tls-s3-lpdwd-4001823791          pending   14m
radar-base-tls-mcvfd-1432596847             pending   14m

7. kubectl describe orders (orders.txt)
8. kubectl get challenges
9. kubectl describe challenges (challenges.txt)
10. kubectl describe ingress (ingress.txt)

Also when i run kubectl edit svc nginx-ingress-ingress-nginx-controller

my load balancer is empty.
radar-base-tls-s3.txt
radar-base-tls.txt
radar-base-tls-dashboard.txt
certificate_request.txt
challenges.txt
clusterissuer.txt

[keyvaann]

Hi Thomas, as you can see the application't have been properly connected to the ingress connector and when trying to access graylog.lethe.ics.forth.gr for example it responds with 404 error. The issues lies somewhere in Ingress controller and the load balancer and not with the cert-manager because first you need to be able to access the applications via HTTP and then cert-manager will be able to generate certificate for that domain.

I suggest you check the logs for nginx-ingress and look for load balancer docs on Kubernetes or MetalLB. If you're using K3S, @blootsvoets or @afolarin might know more information about load balancer since they have worked with that most recently.

[ThomasKassiotis]

With the help of @iDmple i started k3s without traefik and nginx-ingress controller is now correctly set up. My cert-manager produce the ssl certificate for main-domain, also for the grafana.domain and graylog.domain. I still have issues for s3 and dashboards subdomains. The cert-manager logs now :

"error"="wrong status code '403', expected '200'" "dnsName"="s3.my-domain" 
"error"="wrong status code '403', expected '200'" "dnsName"="dashboard.my-domain" 

[keyvaann]

Please check out ingress configuration for apps with issues and make sure they are defined similarly for the other apps that are working correctly.

[ThomasKassiotis]

Hello I managed to proceed with the ssl ! I have to disable traefik to correctly set up the ingress nginx controller !