Unrealistic latency shown in Grafana dasboard

[debbrata-netapp]

This latency of 150s+ is seen on the Graphana Dashboard. This is the time of TO/GB.
There is no impact to User/Application.
This latency seems to be bogus. Are we taking internal operations into consideration for some latency visible to customers?
Customer would take this latency as their front-end protocol latency, which is actually not.
How is the latency getting calculated here?
If is not excluding system workloads, then that would explain why we see this bogus value.

Even on perfarchives the Delay Center dashboard does a similar calculation. However, the intent of the dashboard was to show where there is high latency within ONTAP internals and what’s causing it in order to find workloads that may be impacting other workloads.
But this is NetApp internal only view?
https://pas.activeiq.netapp.com/grafana/d/delay-center-view/delay-center-view?orgId=3&from=1710370801000&to=1710385199999&var-customer_name=Rakuten%20Group%20Inc.&var-cluster_name_uuid=nbackup400%20(c84aa22d-3a39-11e7-88e5-00a098b30c30)](https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpas.activeiq.netapp.com%2Fgrafana%2Fd%2Fdelay-center-view%2Fdelay-center-view%3ForgId%3D3%26from%3D1710370801000%26to%3D1710385199999%26var-customer_name%3DRakuten%2520Group%2520Inc.%26var-cluster_name_uuid%3Dnbackup400%2520(c84aa22d-3a39-11e7-88e5-00a098b30c30)&data=05%7C02%7CDebbrata.Mandal%40netapp.com%7C307e99a96b0945cc4f0f08dc6661b44a%7C4b0911a0929b4715944bc03745165b3a%7C0%7C0%7C638497817073404836%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=jquscFZ%2B5xQtxTp%2Fzl1lMDTL4YtDogDcMe4AMLy0jRg%3D&reserved=0)
Can someone look into this and explain the dashboard calculation and it's purpose.

[debbrata-netapp]

This seems to be a working theory for this horrible/unrealistic latency shown from Delay Center View. 
[https://pas.activeiq.netapp.com/grafana/d/delay-center-view/delay-center-view?orgId=3&from=1710370801000&to=1710385199999&var-customer_name=Rakuten%20Group%20Inc.&var-cluster_name_uuid=nbackup400%20(c84aa22d-3a39-11e7-88e5-00a098b30c30)](https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpas.activeiq.netapp.com%2Fgrafana%2Fd%2Fdelay-center-view%2Fdelay-center-view%3ForgId%3D3%26from%3D1710370801000%26to%3D1710385199999%26var-customer_name%3DRakuten%2520Group%2520Inc.%26var-cluster_name_uuid%3Dnbackup400%2520(c84aa22d-3a39-11e7-88e5-00a098b30c30)&data=05%7C02%7CDebbrata.Mandal%40netapp.com%7C307e99a96b0945cc4f0f08dc6661b44a%7C4b0911a0929b4715944bc03745165b3a%7C0%7C0%7C638497817073404836%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=jquscFZ%2B5xQtxTp%2Fzl1lMDTL4YtDogDcMe4AMLy0jRg%3D&reserved=0)

https://pas-jupyter.corp.netapp.com/user-redirect/notebooks/SharedWorkspaces/Users/gstabler/ASUP%202024040322070129%20-%20High%20Latency%20after%20TO%20GB.ipynb](https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpas-jupyter.corp.netapp.com%2Fuser-redirect%2Fnotebooks%2FSharedWorkspaces%2FUsers%2Fgstabler%2FASUP%25202024040322070129%2520-%2520High%2520Latency%2520after%2520TO%2520GB.ipynb&data=05%7C02%7CDebbrata.Mandal%40netapp.com%7C307e99a96b0945cc4f0f08dc6661b44a%7C4b0911a0929b4715944bc03745165b3a%7C0%7C0%7C638497817073413814%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=jyMJnqm3i8ZgjhnqvPA1CB1GFDigt7NrZR%2BEEuLSVJc%3D&reserved=0)
 
|| iops||
![image](https://github.com/NetApp/harvest/assets/154426744/28ded16d-7cbe-4877-8c36-cf98f815f4ff)
|| total_latency ||
![image](https://github.com/NetApp/harvest/assets/154426744/8b5f1401-01cc-4da8-b270-d1ea18232314)

It’s easy to locate the total_latency from Perfecto, we just can’t find the iops. 

Report workload_queue_dblade for ASUP - 2024040322070129

URL: https://perfecto.corp.netapp.com/v1/report/2024040322070129/workload_queue_dblade?selected_counters=cpu_dblade_residence_time+%28us%29
Dr. Perf Command: /u/bnate/projects/dr_perf/testing/dr_perf.py --asup=2024040322070129 --asup_env=prd --component=ontap --debug=4 --query=counter --object=workload_queue_dblade --counter='cpu_dblade_residence_time' --format=table
 
Cluster: nbackup400 (c84aa22d-3a39-11e7-88e5-00a098b30c30)
Node: nbackup401 (c1539887-3a32-11e7-b656-f50a90ba934e)
Model: FAS8200
Release: 9.11.1P8
Time Range: 2024-03-13 23:00:01.000+00:00 - 2024-03-14 03:14:26.002+00:00
 
start                       end                         instance            cpu_dblade_residence_time (us)
2024-03-14 00:02:33 +00:00  2024-03-14 00:16:26 +00:00  _Deleted-Workloads                      1008991433 <<
 
`[](Report` workload_queue_nblade for ASUP - 2024040322070129
 
URL: https://perfecto.corp.netapp.com/v1/report/2024040322070129/workload_queue_nblade?selected_counters=iops+other+%28%2Fs%29%2Ciops+read+%28%2Fs%29%2Ciops+write+%28%2Fs%29
Dr. Perf Command: /u/bnate/projects/dr_perf/testing/dr_perf.py --asup=2024040322070129 --asup_env=prd --component=ontap --debug=4 --query=counter --object=workload_queue_nblade --counter='iops.other','iops.read','iops.write' --format=table
 
Cluster: nbackup400 (c84aa22d-3a39-11e7-88e5-00a098b30c30)
Node: nbackup401 (c1539887-3a32-11e7-b656-f50a90ba934e)
Model: FAS8200
Release: 9.11.1P8
Time Range: 2024-03-13 23:00:01.000+00:00 - 2024-03-14 03:14:26.002+00:00
 
start                       end                         instance  iops other (/s)  iops read (/s)  iops write (/s)
2024-03-14 00:02:33 +00:00  2024-03-14 00:16:26 +00:00  _DEDUP               0.00   
 
It’s actually not zero if we check on the raw data. 
 
$ dr_perf.py --asup=2024040322070129 --query=counter_raw --object=workload_queue_nblade --counter=iops.other --format=table --start=2024-03-14_00:00:00+00:00 --end=2024-03-14_00:18:00+00:00 --instance=_DEDUP
    Cluster: nbackup400 (c84aa22d-3a39-11e7-88e5-00a098b30c30)
    Node: nbackup401 (c1539887-3a32-11e7-b656-f50a90ba934e)
    Time Range: 2024-03-14 00:00:00.000+00:00 - 00:18:00.000+00:00
 
time interval                         workload_queue_nblade  workload_queue_nblade
                                      instance                                iops
                                                                             other
                                                                     (/s)
------------------------------------  ---------------------  ---------------------
2024-03-14 00:00:33 - 00:01:33+00:00  _DEDUP                                     2
2024-03-14 00:01:33 - 00:02:33+00:00  _DEDUP                                     2
2024-03-14 00:02:33 - 00:16:26+00:00  _DEDUP                                     6 <<<< 6 – 2 = 4, this explains the iops. 4 / 833s (00:16:26 – 00:02:33) ~= 0.004802/s
2024-03-14 00:16:26 - 00:17:26+00:00  _DEDUP                                     6
[andyl@vmwlatsrv03-prd ~]$ dr_perf.py --asup=2024040322070129 --query=counter --object=workload_queue_nblade --counter=iops.other --format=table --start=2024-03-14_00:00:00+00:00 --end=2024-03-14_00:18:00+00:00 --instance=_DEDUP
    Cluster: nbackup400 (c84aa22d-3a39-11e7-88e5-00a098b30c30)
    Node: nbackup401 (c1539887-3a32-11e7-b656-f50a90ba934e)
    Time Range: 2024-03-14 00:00:00.000+00:00 - 00:18:00.000+00:00
 
time interval                         workload_queue_nblade  workload_queue_nblade
                                      instance                                iops
                                                                             other
                                                                     (/s)
------------------------------------  ---------------------  ---------------------
2024-03-14 00:00:33 - 00:01:33+00:00  _DEDUP                                  0.00
2024-03-14 00:01:33 - 00:02:33+00:00  _DEDUP                                  0.00
2024-03-14 00:02:33 - 00:16:26+00:00  _DEDUP                                  0.00 <<<< 0.004802/s is too close to Zero, it rounds up to 0
2024-03-14 00:16:26 - 00:17:26+00:00  _DEDUP                                  0.00)

1008991433 seems like a bogus number, but this is NOT the key here. 
•	Why would we want to take these internal operations into consideration for some latency visible to customers? 
o	Customer would take this latency as their front-end protocol latency, which is actually not
•	A bigger concern is, why would Harvest/Grafana shows the same bogus number? Can we tune it in the same way?

[rahulguptajss]

@debbrata-netapp Could you share Harvest version?

[debbrata-netapp]

@rahulguptajss
Thanks for looking into this.
I'll get you that.

[naotoNTAP]

its 7.5.17

[rahulguptajss]

@naotoNTAP I was asking for the Harvest version. By the way, we have resolved latency spike issues in version 24.05. Could you please upgrade to this release and verify if the issue persists?

Here are the references to the issues that have been addressed:

• Issue #2544
• Issue #2769
• Issue #2605