Issue with new metrics that use topk

[cyroxxer]

Hello, i hope me question is at the right place and i'am able to explain the issue/question.

With last two versions 24.08.0 and 24.05.0 we see an issue with some of the metrics that use aggr data. For this example i will try to explain the behavior with the default Aggregate dashboard and the "Top 25 Aggregates by Physical Space Used" graph. With versions before 24.05.0 we were able to get graphs until the day we startet with Harvest/Grafana (Prom retention set to 1 Year). With the latest dashboards we see some graphs don't go behind a specific date in June. In June we renamed some Aggregates ...

As soon i edit the Query from the "Top 25 Aggregates by Physical Space Used" and remove the TOPK section the graph is able to display the data that we assume to see
aggr_space_physical_used{datacenter="$Datacenter",cluster="$Cluster",node="$Node",aggr="$Aggregate"}
and
topk($TopResources, avg_over_time(aggr_space_physical_used{datacenter="$Datacenter",cluster="$Cluster",node="$Node",aggr="$Aggregate"}[3h] @ end()))

Can somebody explain what is happening? Is there an error within the queries or do we need to modify something within our dataset in prometheus?

Thanks for helping

[rahulguptajss]

@cyroxxer Could you please share the output of the following Prometheus query? You can also email the output to us at [email protected].

sum(sum_over_time(aggr_space_physical_used[1y])) by (datacenter, cluster, instance)

[cyroxxer]

You are right with your observation. But i ask myself is this a bug or is it working as designed? I would assume that it could handle aggr renames because this is a standard process wich you can do in Ontap.

Thanks for the link to the old JSON files, i will import them as a workaround. But would be good to know if this behavior is as design or will be changed/fixed?

[rahulguptajss]

@cyroxxer

We have added topk feature to this dashboard, similar to other dashboards, to ensure that not all aggregates are displayed in each panel. This prevents the panels from becoming cluttered with data, making them more useful. To implement topk and maintain dashboard performance, we perform a lookup for the last 3 hours to collect the top k items and plot them over a selected time range in the Grafana panel.

A 3-hour window was chosen as a compromise to performance. If we were to perform a lookup over a period of 1 year to select the top k items, the dashboard's performance would be terrible. Consequently, the dropdowns in the dashboard are populated with the latest data only, so you will not see older aggregate names in the dropdown panels for filtering records. Populating these filter dropdowns with both old and new data would again impact performance and could cause confusion. We've found that most customers only want to see active/recent data which is why most of the dashboard dropdowns filter on latest records.

Given our current design, there isn't an optimal solution for handling topk with older names of aggregates. While all the data is available in Prometheus, the current topk implementation may not display older name aggregate records in the dashboard.

Is renaming objects a frequent operation for your use case? We are open to any suggestions you may have to tackle this issue.

[cyroxxer]

Hi thanks for all the information. We rename some objects due to tech refresh of our clusters or if we have to reorganize our storage. This could be the case maybe each 2-4 years.

I think the dashboards don't provide the information that are requested. For example if i would like to see how the Available Space of one or more Aggr developed over time (in this case 1 Year) the actual dashboard don't work as soon you have renamed objects.

I can't provide a solution to this issue but i think it should be considert in next releases?

[rahulguptajss]

@cyroxxer Sure, we will add this to our backlog.

Are you okay with having dashboards without the Top-K feature for the aggregate dashboard and others? Please note that data loading might be slower in this case.

[cyroxxer]

Hi, sure i would be okay with that.