Error: Implementation error => PollClient_timeout on Harvest

[s-kuchi]

Some of my pollers are stopping with PollClient_timeout error. Has anyone encountered this before? Can they tell me how to fix this?

/# bin/harvest version
harvest version 23.05.0-1 (commit 6f74c7a) (build date 2023-05-03T08:08:46-0400) linux/amd64
checking GitHub for latest... you have the latest ‚úì

An excerpt for one poller is below

/{"level":"warn","Poller":"atl_dataservices","error":"0: ","collector":"Ems","object":"Ems","caller":"./poller.go:698","time":"2023-07-07T13:52:04-04:00","message":"init collector-object"}
{"level":"info","Poller":"atl_dataservices","collector":"Zapi:Qtree","path":"conf/zapi/cdot/9.8.0/qtree.yaml","v":"9.3.0","caller":"collector/helpers.go:139","time":"2023-07-07T13:52:04-04:00","message":"best-fit template"}
{"level":"warn","Poller":"atl_dataservices","error":"implementation error => PollClient_timeout","collector":"Zapi","object":"Qtree","caller":"./poller.go:698","time":"2023-07-07T13:52:04-04:00","message":"init collector-object"}
{"level":"info","Poller":"atl_dataservices","collector":"ZapiPerf:Qtree","path":"conf/zapiperf/cdot/9.8.0/qtree.yaml","v":"9.3.0","caller":"collector/helpers.go:139","time":"2023-07-07T13:52:04-04:00","message":"best-fit template"}
{"level":"info","Poller":"atl_dataservices","collector":"ZapiPerf:Qtree","path":"conf/zapiperf/cdot/9.8.0/qtree.yaml","v":"9.3.0","caller":"collector/helpers.go:139","time":"2023-07-07T13:52:04-04:00","message":"best-fit template"}
{"level":"warn","Poller":"atl_dataservices","error":"implementation error => PollClient_timeout","collector":"ZapiPerf","object":"Qtree","caller":"./poller.go:698","time":"2023-07-07T13:52:04-04:00","message":"init collector-object"}
{"level":"info","Poller":"atl_dataservices","collector":"ZapiPerf:NFSv4Node","path":"conf/zapiperf/cdot/9.8.0/nfsv4_node.yaml","v":"9.3.0","caller":"collector/helpers.go:139","time":"2023-07-07T13:52:04-04:00","message":"best-fit template"}
{"level":"info","Poller":"atl_dataservices","collector":"ZapiPerf:NFSv4Node","path":"conf/zapiperf/cdot/9.8.0/nfsv4_node.yaml","v":"9.3.0","caller":"collector/helpers.go:139","time":"2023-07-07T13:52:04-04:00","message":"best-fit template"}
{"level":"warn","Poller":"atl_dataservices","error":"implementation error => PollClient_timeout","collector":"ZapiPerf","object":"NFSv4Node","caller":"./poller.go:698","time":"2023-07-07T13:52:04-04:00","message":"init collector-object"}
{"level":"warn","Poller":"atl_dataservices","caller":"./poller.go:331","time":"2023-07-07T13:52:04-04:00","message":"no collectors initialized, stopping"}
{"level":"info","Poller":"atl_dataservices","caller":"./poller.go:556","time":"2023-07-07T13:52:04-04:00","message":"cleaning up and stopping [pid=2977]"}

[cgrinds]

hi @s-kuchi did you by chance modify any of Harvest's templates in the conf directory? I think the only way this error can happen is if you add client_Timeout to one of the template schedules.

Maybe you added client_timeout: under schedule instead of at the top level?

Here is an example of where in the template file client_timeout should go.
https://github.com/NetApp/harvest/blob/main/conf/rest/9.12.0/qtree.yaml#L5

Are you creating custom templates or modifying Harvest's out-of-the-box ones?
Generally it's better to extend an existing like this
https://netapp.github.io/harvest/23.05/configure-templates/#extend-an-existing-object-template

[s-kuchi]

hi @cgrinds . I used client_timeout: variable in default.yaml in the conf directory since this link talks about it
https://github.com/NetApp/harvest/wiki/Troubleshooting-Harvest

There are a few parameters that you can change to avoid this from happening. You can do this by editing the subtemplate of the resource affected. E.g. you can add the parameters in conf/zapiperf/cdot/9.8.0/volume.yaml or conf/zapi/cdot/9.8.0/volume.yaml. If the errors happen for most of the resources, you can add them in the main template of the collector (conf/zapi/default.yaml or conf/zapiperf/default.yaml) to apply them on all objects.

Adding it to every yaml file in every cdot version seem to be a painful challenge otherwise.

[cgrinds]

hi @s-kuchi can you share the default.yaml files you changed?

[s-kuchi]

@cgrinds : I added the client_timeout line in both zapi/default.yaml and zapiperf/default.yaml

[root@pdgllvsanmgt01 zapi]# more default.yaml

collector: Zapi

schedule:

• data: 180s
• client_timeout: 180s

[root@pdgllvsanmgt01 zapiperf]# more default.yaml

collector: ZapiPerf

schedule:

• counter: 1200s
• instance: 600s
• data: 60s
• client_timeout: 120s

[cgrinds]

Thanks!

Try something like this

collector:          Zapi

# Order here matters!
schedule:
  - data: 180s

client_timeout: 60s

The difference is client_timeout should be at the root level instead of a child of schedule. When client_timeout is a child of schedule it has a different meaning.

[s-kuchi]

Hi @cgrinds : That change worked for most pollers, except one, for which i get the following error. Is this a different unrelated error? If yes, how do I resolve this?

{"level":"info","Poller":"wdc_netapp_video_fas","pid":85533,"caller":"./poller.go:240","time":"2023-07-10T11:49:34-04:00","message":"started as daemon"}
{"level":"warn","Poller":"wdc_netapp_video_fas","error":"connection error => connection error => Post "https://10.14.28.175:443/servlets/netapp.servlets.admin.XMLrequest_filer\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","collector":"Zapi","object":"SecurityAuditDestination","caller":"./poller.go:689","time":"2023-07-10T12:01:14-04:00","message":"abort collector"}
{"level":"warn","Poller":"wdc_netapp_video_fas","caller":"./poller.go:331","time":"2023-07-10T12:01:14-04:00","message":"no collectors initialized, stopping"}
{"level":"info","Poller":"wdc_netapp_video_fas","caller":"./poller.go:556","time":"2023-07-10T12:01:14-04:00","message":"cleaning up and stopping [pid=85533]"}

[cgrinds]

HI @s-kuchi looks like that timeout is happening the first time the poller tries to talk to the cluster. Is it possible that cluster is not reachable from the machine running Harvest or maybe the ip is wrong?

[s-kuchi]

@cgrinds : There was a typo in the IP address and i fixed it. Thank you.