Nginx-ingress-controller error on K3S #206
Answered
by
yatharthranjan
Hsankesara
asked this question in
Q&A
-
|
Hi all, I am trying to install an isolated version of RADAR-Kubernetes on of our server using k3s. I am using latest k3s version I am getting an error in ingress-controller and can't seem to find a solution for that. I think it has something to do with version of K3s (and Kubernetes) but I am not sure. The error log is: ubuntu@radar-reserved-2:~/RADAR-Kubernetes$ helmfile sync --concurrency 1
Adding repo elastic https://helm.elastic.co
"elastic" has been added to your repositories
Adding repo jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
Adding repo prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories
Adding repo bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories
Adding repo kongz https://charts.kong-z.com
"kongz" has been added to your repositories
Adding repo fluent https://fluent.github.io/helm-charts
"fluent" has been added to your repositories
Adding repo ingress-nginx https://kubernetes.github.io/ingress-nginx
"ingress-nginx" has been added to your repositories
Adding repo radar https://radar-base.github.io/radar-helm-charts
"radar" has been added to your repositories
Listing releases matching ^kafka-manager$
Affected releases are:
cert-manager (radar/cert-manager) UPDATED
elasticsearch (elastic/elasticsearch) UPDATED
fluent-bit (fluent/fluent-bit) UPDATED
graylog (kongz/graylog) UPDATED
kube-prometheus-stack (radar/kube-prometheus-stack) UPDATED
mongodb (bitnami/mongodb) UPDATED
nginx-ingress (ingress-nginx/ingress-nginx) UPDATED
Upgrading release=mongodb, chart=bitnami/mongodb
Release "mongodb" does not exist. Installing it now.
Upgrading release=elasticsearch, chart=elastic/elasticsearch
Release "elasticsearch" has been upgraded. Happy Helming!
NAME: elasticsearch
LAST DEPLOYED: Fri Oct 7 09:57:35 2022
NAMESPACE: graylog
STATUS: deployed
REVISION: 4
NOTES:
1. Watch all cluster members come up.
$ kubectl get pods --namespace=graylog -l app=elasticsearch-master -w
2. Test cluster health using Helm test.
$ helm test elasticsearch
Listing releases matching ^elasticsearch$
elasticsearch graylog 4 2022-10-07 09:57:35.70657211 +0000 UTC deployed elasticsearch-7.12.0 7.12.0
Upgrading release=graylog, chart=kongz/graylog
Release "graylog" has been upgraded. Happy Helming!
NAME: graylog
LAST DEPLOYED: Fri Oct 7 09:57:37 2022
NAMESPACE: graylog
STATUS: deployed
REVISION: 4
TEST SUITE: None
NOTES:
To connect to your Graylog server:
1. Get the application URL by running these commands:
http://graylog.example.com
2. The Graylog root users
echo "User: admin"
echo "Password: $(kubectl get secret --namespace graylog graylog -o "jsonpath={.data['graylog-password-secret']}" | base64 --decode)"
To send logs to graylog:
NOTE: If `graylog.input` is empty, you cannot send logs from other services. Please make sure the value is not empty.
See https://github.com/KongZ/charts/tree/main/charts/graylog#input for detail
1. TCP
export POD_NAME=$(kubectl get pods --namespace graylog -l "app.kubernetes.io/name=graylog,app.kubernetes.io/instance=graylog" -o jsonpath="{.items[0].metadata.name}")
Run the command
kubectl port-forward $POD_NAME 12222:12222
Then send logs to 127.0.0.1:12222
Listing releases matching ^graylog$
graylog graylog 4 2022-10-07 09:57:37.599586163 +0000 UTC deployed graylog-1.9.2 4.2.3
Upgrading release=fluent-bit, chart=fluent/fluent-bit
Release "fluent-bit" has been upgraded. Happy Helming!
NAME: fluent-bit
LAST DEPLOYED: Fri Oct 7 09:57:40 2022
NAMESPACE: graylog
STATUS: deployed
REVISION: 4
NOTES:
Get Fluent Bit build information by running these commands:
export POD_NAME=$(kubectl get pods --namespace graylog -l "app.kubernetes.io/name=fluent-bit,app.kubernetes.io/instance=fluent-bit" -o jsonpath="{.items[0].metadata.name}")
echo "curl http://127.0.0.1:2020 for Fluent Bit build information"
kubectl --namespace graylog port-forward $POD_NAME 2020:2020
Listing releases matching ^fluent-bit$
fluent-bit graylog 4 2022-10-07 09:57:40.136865894 +0000 UTC deployed fluent-bit-0.15.4 1.7.3
hook[presync] logs | customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io configured
hook[presync] logs | customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io configured
hook[presync] logs | customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io configured
hook[presync] logs | customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io configured
hook[presync] logs | customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io configured
hook[presync] logs | customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io configured
hook[presync] logs |
Upgrading release=cert-manager, chart=radar/cert-manager
Upgrading release=kube-prometheus-stack, chart=radar/kube-prometheus-stack
Release "kube-prometheus-stack" has been upgraded. Happy Helming!
NAME: kube-prometheus-stack
LAST DEPLOYED: Fri Oct 7 09:57:53 2022
NAMESPACE: monitoring
STATUS: deployed
REVISION: 4
Listing releases matching ^kube-prometheus-stack$
kube-prometheus-stack monitoring 4 2022-10-07 09:57:53.223516221 +0000 UTC deployed kube-prometheus-stack-0.2.0 1.0
Upgrading release=nginx-ingress, chart=ingress-nginx/ingress-nginx
UPDATED RELEASES:
NAME CHART VERSION
elasticsearch elastic/elasticsearch 7.12.0
graylog kongz/graylog 1.9.2
fluent-bit fluent/fluent-bit 0.15.4
kube-prometheus-stack radar/kube-prometheus-stack 0.2.0
FAILED RELEASES:
NAME
mongodb
cert-manager
nginx-ingress
in helmfile.d/00-init.yaml: 3 errors:
err 0: failed processing release mongodb: command "/usr/local/bin/helm" exited with non-zero status:
PATH:
/usr/local/bin/helm
ARGS:
0: helm (4 bytes)
1: --kube-context (14 bytes)
2: default (7 bytes)
3: upgrade (7 bytes)
4: --install (9 bytes)
5: --reset-values (14 bytes)
6: mongodb (7 bytes)
7: bitnami/mongodb (15 bytes)
8: --version (9 bytes)
9: 10.11.1 (7 bytes)
10: --create-namespace (18 bytes)
11: --kube-context (14 bytes)
12: default (7 bytes)
13: --namespace (11 bytes)
14: graylog (7 bytes)
15: --values (8 bytes)
16: /tmp/helmfile1578358514/graylog-mongodb-values-6ff8f4dc64 (57 bytes)
17: --values (8 bytes)
18: /tmp/helmfile3290546836/graylog-mongodb-values-ddbb4f555 (56 bytes)
19: --history-max (13 bytes)
20: 10 (2 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
Error: chart "mongodb" matching 10.11.1 not found in bitnami index. (try 'helm repo update'): no chart version found for mongodb-10.11.1
COMBINED OUTPUT:
Release "mongodb" does not exist. Installing it now.
Error: chart "mongodb" matching 10.11.1 not found in bitnami index. (try 'helm repo update'): no chart version found for mongodb-10.11.1
err 1: failed processing release cert-manager: command "/usr/local/bin/helm" exited with non-zero status:
PATH:
/usr/local/bin/helm
ARGS:
0: helm (4 bytes)
1: --kube-context (14 bytes)
2: default (7 bytes)
3: upgrade (7 bytes)
4: --install (9 bytes)
5: --reset-values (14 bytes)
6: cert-manager (12 bytes)
7: radar/cert-manager (18 bytes)
8: --version (9 bytes)
9: 0.1.2 (5 bytes)
10: --wait (6 bytes)
11: --create-namespace (18 bytes)
12: --kube-context (14 bytes)
13: default (7 bytes)
14: --namespace (11 bytes)
15: cert-manager (12 bytes)
16: --set (5 bytes)
17: server_name=example.com (23 bytes)
18: --set (5 bytes)
19: [email protected] (45 bytes)
20: --history-max (13 bytes)
21: 10 (2 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
Error: UPGRADE FAILED: cannot patch "letsencrypt-prod" with kind ClusterIssuer: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority
COMBINED OUTPUT:
Error: UPGRADE FAILED: cannot patch "letsencrypt-prod" with kind ClusterIssuer: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority
err 2: failed processing release nginx-ingress: command "/usr/local/bin/helm" exited with non-zero status:
PATH:
/usr/local/bin/helm
ARGS:
0: helm (4 bytes)
1: --kube-context (14 bytes)
2: default (7 bytes)
3: upgrade (7 bytes)
4: --install (9 bytes)
5: --reset-values (14 bytes)
6: nginx-ingress (13 bytes)
7: ingress-nginx/ingress-nginx (27 bytes)
8: --version (9 bytes)
9: 3.25.0 (6 bytes)
10: --create-namespace (18 bytes)
11: --kube-context (14 bytes)
12: default (7 bytes)
13: --values (8 bytes)
14: /tmp/helmfile2735164784/nginx-ingress-values-587f84b74c (55 bytes)
15: --values (8 bytes)
16: /tmp/helmfile1555449119/nginx-ingress-values-5789879f8d (55 bytes)
17: --history-max (13 bytes)
18: 10 (2 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
Error: UPGRADE FAILED: post-upgrade hooks failed: timed out waiting for the condition
COMBINED OUTPUT:
Error: UPGRADE FAILED: post-upgrade hooks failed: timed out waiting for the conditionAfter looking at the nginx-ingress pod: ubuntu@radar-reserved-2:~/RADAR-Kubernetes$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-ingress-controller-default-backend-94f5f55c9-scpvk 1/1 Running 0 20h
nginx-ingress-controller-85dd697cf4-hwnt2 1/1 Running 0 20h
nginx-ingress-ingress-nginx-controller-7fc7c8b8f8-hvbcp 0/1 CrashLoopBackOff 7 (2m53s ago) 13mubuntu@radar-reserved-2:~/RADAR-Kubernetes$ kubectl logs nginx-ingress-ingress-nginx-controller-7fc7c8b8f8-hvbcp
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v0.44.0
Build: f802554ccfadf828f7eb6d3f9a9333686706d613
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.6
I1007 10:20:23.398628 7 flags.go:208] "Watching for Ingress" class="nginx"
-------------------------------------------------------------------------------
W1007 10:20:23.398686 7 flags.go:213] Ingresses with an empty class will also be processed by this Ingress controller
W1007 10:20:23.399122 7 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1007 10:20:23.399463 7 main.go:241] "Creating API client" host="https://10.43.0.1:443"
I1007 10:20:23.421538 7 main.go:285] "Running in Kubernetes cluster" major="1" minor="24" git="v1.24.6+k3s1" state="clean" commit="a8e0c66d1a90a2bbc4ffa975129ca35756cc7c14" platform="linux/amd64"
I1007 10:20:23.648998 7 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I1007 10:20:23.650746 7 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
W1007 10:20:23.652130 7 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I1007 10:20:23.667388 7 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I1007 10:20:23.679090 7 nginx.go:254] "Starting NGINX Ingress controller"
I1007 10:20:23.747933 7 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"default", Name:"nginx-ingress-ingress-nginx-controller", UID:"8f5d5a8c-66ac-44d3-9c55-20e4cd7b8e68", APIVersion:"v1", ResourceVersion:"39401", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap default/nginx-ingress-ingress-nginx-controller
E1007 10:20:24.786318 7 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E1007 10:20:26.009899 7 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E1007 10:20:28.577442 7 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E1007 10:20:33.247780 7 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E1007 10:20:40.444343 7 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E1007 10:21:03.992740 7 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
I1007 10:21:20.188420 7 main.go:187] "Received SIGTERM, shutting down"
I1007 10:21:20.188445 7 nginx.go:372] "Shutting down controller queues"
E1007 10:21:20.188624 7 store.go:178] timed out waiting for caches to sync
I1007 10:21:20.188668 7 nginx.go:296] "Starting NGINX process"
I1007 10:21:20.188794 7 leaderelection.go:243] attempting to acquire leader lease default/ingress-controller-leader-nginx...
I1007 10:21:20.189091 7 queue.go:78] "queue has been shutdown, failed to enqueue" key="&ObjectMeta{Name:initial-sync,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}"
I1007 10:21:20.189113 7 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I1007 10:21:20.199716 7 status.go:84] "New leader elected" identity="nginx-ingress-ingress-nginx-controller-7fc7c8b8f8-r9jcw"
I1007 10:21:20.207591 7 status.go:131] "removing value from ingress status" address=[]
I1007 10:21:20.207632 7 nginx.go:380] "Stopping admission controller"
I1007 10:21:20.207658 7 nginx.go:388] "Stopping NGINX process"
E1007 10:21:20.207685 7 nginx.go:319] "Error listening for TLS connections" err="http: Server closed"
2022/10/07 10:21:20 [notice] 45#45: signal process started
I1007 10:21:21.213641 7 nginx.go:401] "NGINX process has stopped"
I1007 10:21:21.213701 7 main.go:195] "Handled quit, awaiting Pod deletion"
I1007 10:21:31.213911 7 main.go:198] "Exiting" code=0 |
Beta Was this translation helpful? Give feedback.
Answered by
yatharthranjan
Oct 7, 2022
Replies: 2 comments 2 replies
-
|
Hi, some points -
|
Beta Was this translation helpful? Give feedback.
2 replies
-
Yeah, I am trying that now. Will let you know if that works.
|
Beta Was this translation helpful? Give feedback.
-
|
I have used |
Beta Was this translation helpful? Give feedback.
Answer selected by
Hsankesara
-
|
Seems like the ingress issue is with K8S versions 1.21 + (bitnami/charts#7264) on old nginx-ingress charts. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi, some points -
did you try with the changes in #199 ?
Might be related to this issue #174.
Have you tried downgrading to k8s v1.21 ?
And lastly there are other errors as well (like it is not able to find the mongodb chart from bitnami which is expected hence changes in PR #199 are required)