How to change all nodenames to lower case?

[Kosiare]

Hi everyone!

I've installed via Helm kube-prometheus-stack from prometheus-community repository. Everything works fine but I got a problem with node/host names.

I'm using AKS with one node pool and 8 nodes. An example node hostname is: aks-nodepool-0000A. When I use command kubectl get nodes I can see that the node name is a little bit different because it's written in lowercase, for example aks-nodepool-0000a.

So that makes problem with metrics because some metrics are using lowercase node name like kube_node_status_capacity

and other metrics are with uppercase last letter, for example this one: node_uname_info.

This makes really a mess in my Grafana dashboards because I have to use two variables (one with lowercase and one with uppercase name).I tried to use case insensitive expression kube_node_status_capacity{node=~"(?i:($node))"},resource="cpu"} but it does not work in any query.

Is there a solution to change all nodename node values to lowercase? AFAIK it's not possible to change it in AKS (I would have to recreate whole cluster what is obviously a no-no for me because it's productive cluster).

[brancz]

Just to make sure I understand correctly, the kube_node_status_capacity exposes what you expect, but node_uname_info does not?

[brancz]

hmm it doesn't appear that the node-exporter does anything to the string https://github.com/prometheus/node_exporter/blob/8b466360a35581e0301bd22918be7011cf4203c3/collector/uname_linux.go#L35

could you run uname -n on one of these machines and check what that returns?

From what I can tell the exporters are exporting exactly the data that is handed to them, and the kernel/kubernetes may be doing weird things here. Possibly AKS is doing this under the hood as node naming in kubernetes is technically free form, if this is the case I would open a support ticket with Azure.

[Kosiare]

Yeah, I think you're right because uname -n gave me output with last letter in uppercase. Is there an easy way to convert all values to lowercase or the last thing I can do is open a support ticket in Azure?

[brancz]

I don't believe that's possible today, the closest discussion appears to be prometheus/prometheus#1548

[Kosiare]

Okay, so I'll open ticket to Azure. I'll let you know if they know any solution.

[Kosiare]

So, basically they cannot do anything and told me to use tags in Azure Portal (which is funny because it won't work for sure - how Prometheus would scrape something from their website...).

[SuperQ]

This can now be fixed with the Prometheus lowercase relabel action.

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config