net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[mapshen]

Hi @Wing924 ,

We use logstash-exporter to monitor our logstash instances. It run as containers in host mode using the image you created wing924/logstash-exporter:v0.0.1.

Recently, we noticed the following errors every 30 seconds coming out of one logstash-exporter instance

time="2021-01-08T19:06:48Z" level=warning msg="can't scrape logstash /_node/stats" error="Get http://hostname:9600/_node/stats: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
time="2021-01-08T19:07:18Z" level=warning msg="can't scrape logstash /_node/stats" error="Get http://hostname:9600/_node/stats: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
time="2021-01-08T19:07:48Z" level=warning msg="can't scrape logstash /_node/stats" error="Get http://hostname:9600/_node/stats: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
time="2021-01-08T19:08:18Z" level=warning msg="can't scrape logstash /_node/stats" error="Get http://hostname:9600/_node/stats: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
time="2021-01-08T19:08:48Z" level=warning msg="can't scrape logstash /_node/stats" error="Get http://hostname:9600/_node/stats: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"

Our prometheus instance is configured to scrape the the exporter every 15 seconds, so it seems out of every 2 scrapes, 1 times out immediately. Tried to talk to http://hostname:9600/_node/stats directly inside the exporter container with wget, they all succeeded and timeouts were found. Wondering if you may have any idea why could cause this?

Here are the args we pass to logstash-exporter:

        "Args": [
            "--web.listen-address=:9649",
            "--logstash.scrape-uri=http://hostname:9600"
        ],

which means we use the default --logstash.timeout=5s.

[mapshen]

@Wing924 Got a chance to look at this?