TLS communication between Thanos Receiver and remote_write machine

[amrap030]

Hello together,

I am running a kubernetes cluster with thanos, prometheus and grafana which should serve as a central monitoring solution. Other remote machines should send metrics to the receive endpoint of thanos via prometheus remote_write with the grafana agent or another prometheus server in agent mode. When I expose the receive endpoint as a load balancer in kubernetes and use plain http communication between the thanos receive endpoint and the machines that do remote_write everything works fine.

Now I would like to have this communication via TLS and I would like to use my traefik reverse proxy as an ingress with TLS termination, that forwards the pushed metrics internally to the receive endpoint. Unfortunately, when I use the HTTPS address of the reverse proxy, I don't get any metrics anymore and I only can see msg="http: TLS handshake error from 10.1.19.0:29425: remote error: tls: bad certificate" in the traefik logs.

Do you have any idea how to configure this correctly?

[lc-guy]

I've spent a maddeningly long time debugging this myself. Make sure that your client's public certificate is a full-chain certificate, as grafana-agent will NOT attach an accompanying ca_cert entry to the client request. ca_cert only seems to be used to verify the server's certificate.
Also, make sure all your .crt and .key files used in grafana-agent have a newline at the end...