Performance metrics not being collected

[gaur-piyush]

Hi Team,

We are using Harvest to collect ONTAP metrics. However, the performance metrics aren't being collected. I have already enabled the collector in the configuration.

Below is the configuration yaml.

harvest.yml: |
     Exporters:
        prometheus1:
          exporter: Prometheus
          port_range: 12900-14000
          add_meta_tags: false
     Defaults:
       use_insecure_tls: true
       perfer_zapi: true
     Pollers:
       harvest:
         datacenter: {{ .Values.datacenter }}
         addr: "{{ `{{ .pvt_ip }}` }}"
         username: "{{ `{{ .username }}` }}"
         password: "{{ `{{ .password }}` }}"
         collectors:
         - Rest
         - Ems
         exporters:
         - prometheus1

Example of missing metrics

https://netapp.github.io/harvest/23.11/ontap-metrics/#node_volume_avg_latency

Please let me know if we need to add additional configuration for it.

Regards,
Piyush

[rahulguptajss]

@gaur-piyush You should also enable RestPerf collector which is meant for performance metrics.
https://netapp.github.io/harvest/24.11.1/configure-rest/#restperf-collector

[gaur-piyush]

@gaur-piyush Could you please use curl to access the endpoint 12990/metrics in your Kubernetes environment for this pod and verify that the node_volume_avg_latency metric is present?

curl -s http://POD:12990/metrics | grep node_volume_avg_latency

Also, could you upload the output of the following command to the shared location?

curl -s http://POD:12990/metrics

Replace POD as applicable to your environment. You may want to ssh to POD and use wget if port is not exposed.

Both the logs have been shared.

[rahulguptajss]

Thank you. It appears that Harvest is correctly collecting these metrics:

# curl -s localhost:9090/metrics | grep node_volume_avg_latency
node_volume_avg_latency{datacenter="xxxx",cluster="xxxx",node="xxxx"} 28.291848253197113
node_volume_avg_latency{datacenter="xxx",cluster="xxxx",node="xxxx"} 18.33630217400105

Could you verify that Prometheus is able to scrape the Harvest pod correctly? You can do this by navigating to the Prometheus UI and checking under Status -> Targets.

[gaur-piyush]

Thank you. It appears that Harvest is correctly collecting these metrics:

# curl -s localhost:9090/metrics | grep node_volume_avg_latency
node_volume_avg_latency{datacenter="xxxx",cluster="xxxx",node="xxxx"} 28.291848253197113
node_volume_avg_latency{datacenter="xxx",cluster="xxxx",node="xxxx"} 18.33630217400105

Could you verify that Prometheus is able to scrape the Harvest pod correctly? You can do this by navigating to the Prometheus UI and checking under Status -> Targets.

We're using open-telemetry to scrape the metrics from Harvest in prometheus format. Below is the config for it.

config:
    exporters:
        debug:
            verbosity: normal
        googlecloud:
            project: ""
            metric:
                prefix: custom.googleapis.com
            sending_queue:
                enabled: true
                queue_size: 20000
    extensions:
        health_check: 
            endpoint: "0.0.0.0:13133"
    processors:
        batch:
            send_batch_size: 8192
            send_batch_max_size: 10000
            timeout: 10s
        memory_limiter:
            check_interval: 5s
            limit_percentage: 80
            spike_limit_percentage: 30
        resourcedetection:
            detectors: [gcp]
            timeout: 10s
    receivers:
        prometheus:
            config:
                scrape_configs:
                - job_name: 'ontap-metrics-harvest'
                  scrape_interval: 300s
                  static_configs:
                  - targets: ['harvest-poller-harvest-poller-gcnv-harvest.ontap-monitoring.svc.cluster.local:12990'] 
    service:
        extensions:
            - health_check
        pipelines:
            metrics:
                receivers: [prometheus]
                processors: [resourcedetection]
                exporters: [googlecloud, debug]

[rahulguptajss]

If you are utilizing the default Harvest schedule for metric collection, it's recommended to set the scrape_interval to 1 minute (1m). This is because performance metrics are refreshed every minute by default.

In the exporters section of your OpenTelemetry configuration, a prometheus exporter appears to be missing. This is necessary if you intend to forward the collected data to Prometheus. You can find more details on configuring a Prometheus exporter in OpenTelemetry here.

It's worth noting that Harvest metrics are already formatted in OpenMetrics, which means you have the option to scrape them directly using Prometheus, bypassing the need for OpenTelemetry in case.

[gaur-piyush]

If you are utilizing the default Harvest schedule for metric collection, it's recommended to set the scrape_interval to 1 minute (1m). This is because performance metrics are refreshed every minute by default.

In the exporters section of your OpenTelemetry configuration, a prometheus exporter appears to be missing. This is necessary if you intend to forward the collected data to Prometheus. You can find more details on configuring a Prometheus exporter in OpenTelemetry here.

It's worth noting that Harvest metrics are already formatted in OpenMetrics, which means you have the option to scrape them directly using Prometheus, bypassing the need for OpenTelemetry in case.

We're sending the metrics directly to Google Cloud Monitoring. I can try with changing the scrape interval to 1 minute, but this will also impact the other metrics and overload Harvest.

[gaur-piyush]

I had a discussion with @rahulguptajss and it was confirmed that Harvest is scraping the metrics as intended. However, Open Telemetry collector might be having an issue while sending the metrics to GCP. We can close this discussion as we know where to look to fix the issue.