Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bad address in etcd pods, etcd unavailable or misconfigured in all stolon pods in Kubernetes 1.6 #1

Open
uhhhh2 opened this issue May 15, 2017 · 0 comments

Comments

@uhhhh2
Copy link
Owner

uhhhh2 commented May 15, 2017

I am trying to get this working on Kubernetes 1.6.

What I've changed in the code to get this far: a840802

Kubectl commands used to set up a namespace just for this Helm chart, and get Tiller started in this namespace with sufficient RBAC permissions to run:

kubectl create namespace postgres
kubectl create serviceaccount --namespace postgres tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=postgres:tiller
helm init --service-account=tiller --tiller-namespace=postgres

Helm command to install the chart to the namespace:

helm install ./stolon/ --tiller-namespace postgres --name postgres --namespace postgres --values ../inputs.yaml

Console output after creation:

NAME:   postgres
LAST DEPLOYED: Mon May 15 13:01:45 2017
NAMESPACE: postgres
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/Deployment
NAME                      DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
postgres-stolon-sentinel  3        3        3           0          1s
postgres-stolon-proxy     3        3        3           0          1s

==> v1beta1/StatefulSet
NAME                    DESIRED  CURRENT  AGE
postgres-stolon-keeper  3        0        1s
postgres-etcd           3        0        0s

==> v1/Job
NAME             DESIRED  SUCCESSFUL  AGE
postgres-stolon  1        0           0s

==> v1/Secret
NAME             TYPE    DATA  AGE
postgres-stolon  Opaque  1     1s

==> v1/Service
NAME                    CLUSTER-IP   EXTERNAL-IP  PORT(S)            AGE
postgres-stolon-keeper  10.98.80.22  <none>       5432/TCP           1s
postgres-etcd           None         <none>       2380/TCP,2379/TCP  1s
postgres-stolon-proxy   10.99.252.6  <none>       5432/TCP           1s


NOTES:
Helm cluster in installed and initialized.


To get superuser password run
    PGPASSWORD=$(kubectl get secret --namespace postgres postgres-stolon -o jsonpath="{.data.password}" | base64 --decode; echo)

etcd pod log output, regardless of the --values file used (my own or the provided one):

2017-05-15T17:48:17.560288503Z sh: res-etcd-0: bad number
2017-05-15T17:48:17.561498701Z ping: bad address 'postgres-etcd-0.postgres-etcd'
2017-05-15T17:48:17.562595258Z Waiting for postgres-etcd-0.postgres-etcd to come up
2017-05-15T17:48:18.561915786Z Waiting for postgres-etcd-0.postgres-etcd to come up
2017-05-15T17:48:18.562745479Z ping: bad address 'postgres-etcd-0.postgres-etcd'

More lines that are almost identical to those above (but with different timestamps) follow every few seconds.

stolon sentinel pod output (again, same regardless of --values file used):

2017-05-15T17:48:16.940728833Z [I] 2017-05-15T17:48:16Z sentinel.go:1408: sentinel uid uid=4ac0dc25
2017-05-15T17:48:16.941350884Z [I] 2017-05-15T17:48:16Z sentinel.go:84: Trying to acquire sentinels leadership
2017-05-15T17:48:16.943163321Z [E] 2017-05-15T17:48:16Z sentinel.go:1282: error retrieving cluster data error=client: etcd cluster is unavailable or misconfigured
2017-05-15T17:48:16.947794612Z [E] 2017-05-15T17:48:16Z sentinel.go:104: election loop error error=client: etcd cluster is unavailable or misconfigured
2017-05-15T17:48:21.943965961Z [E] 2017-05-15T17:48:21Z sentinel.go:1282: error retrieving cluster data error=client: etcd cluster is unavailable or misconfigured
2017-05-15T17:48:26.94516699Z [E] 2017-05-15T17:48:26Z sentinel.go:1282: error retrieving cluster data error=client: etcd cluster is unavailable or misconfigured

More lines that are almost identical to those above (but with different timestamps) follow every few seconds.

stolon keeper pod output (again, same regardless of --values file used):

2017-05-15T17:48:17.929769426Z [W] 2017-05-15T17:48:17Z keeper.go:152: password file permissions are too open. This file should only be readable to the user executing stolon! Continuing... file=/etc/secrets/stolon/password mode=01000000777
2017-05-15T17:48:17.92980571Z [I] 2017-05-15T17:48:17Z keeper.go:1412: exclusive lock on data dir taken
2017-05-15T17:48:17.929905302Z [I] 2017-05-15T17:48:17Z keeper.go:391: keeper uid uid=keeper0
2017-05-15T17:48:17.931740677Z [E] 2017-05-15T17:48:17Z keeper.go:537: error retrieving cluster data error=client: etcd cluster is unavailable or misconfigured
2017-05-15T17:48:17.931752786Z [I] 2017-05-15T17:48:17Z postgresql.go:209: stopping database

More "error retrieving cluster data error=client: etcd cluster is unavailable or misconfigured" lines appear in this log every few seconds.

stolon proxy pod output (again, same regardless of --values file used):

2017-05-15T17:48:15.270493093Z [I] 2017-05-15T17:48:15Z proxy.go:291: proxy uid uid=c2dd8b64
2017-05-15T17:48:15.273236878Z [E] 2017-05-15T17:48:15Z proxy.go:175: cannot get cluster data error=client: etcd cluster is unavailable or misconfigured

More lines similar to the second (but with different timestamps) appear every few seconds.

I tried using etcd version 3.0.17 but that didn't change anything.

How can this be fixed?

@uhhhh2 uhhhh2 changed the title Bad address in etcd pods, etcd unavailable or misconfigured in all stolon pods in Kubernetes 1,6 Bad address in etcd pods, etcd unavailable or misconfigured in all stolon pods in Kubernetes 1.6 May 15, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant