Cannot access Promethus service (via Grafana or directly within the cluster)

[techvd]

I have the default kubernetes installation on Docker Desktop for Windows. This is currently running Kubernetes version 1.22.5

I followed the Quick Start and setup kube-prometheus (https://github.com/prometheus-operator/kube-prometheus#quickstart).
Everything is setup and all the services/deployments are up and running. But Grafana cannot access The prometheus-k8s service.

I ran some tests from a temporary debian pod running inside the cluster. This pod can access the grafana service fine but not the prometheus-k8s service. Here is the test output:

root@debian-dev:/# curl grafana.monitoring:3000
<a href="/login">Found</a>.

root@debian-dev:/# curl prometheus-k8s.monitoring:9090
curl: (7) Failed to connect to prometheus-k8s.monitoring port 9090: Connection refused

Below is the kubectl describe output from a windows shell for both these services. Both services are using ClusterIP (the default).

I haven't changed or customized anything at all so pretty much everything is default. One other test I have done is setup an nginx service manually with 2 pods in the deployment and that service access works just fine from the debian pod so there is something going on with the prometheus-k8s service setup. Any help is greatly appreciated.

>kubectl describe svc grafana -n monitoring
Name: grafana
Namespace: monitoring
Labels: app.kubernetes.io/component=grafana
app.kubernetes.io/name=grafana
app.kubernetes.io/part-of=kube-prometheus
app.kubernetes.io/version=8.3.6
Annotations:
Selector: app.kubernetes.io/component=grafana,app.kubernetes.io/name=grafana,app.kubernetes.io/part-of=kube-prometheus
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.102.163.20
IPs: 10.102.163.20
Port: http 3000/TCP
TargetPort: http/TCP
Endpoints: 10.1.1.1:3000
Session Affinity: None
Events:

>kubectl describe svc prometheus-k8s -n monitoring
Name: prometheus-k8s
Namespace: monitoring
Labels: app.kubernetes.io/component=prometheus
app.kubernetes.io/instance=k8s
app.kubernetes.io/name=prometheus
app.kubernetes.io/part-of=kube-prometheus
app.kubernetes.io/version=2.33.3
Annotations:
Selector: app.kubernetes.io/component=prometheus,app.kubernetes.io/instance=k8s,app.kubernetes.io/name=prometheus,app.kubernetes.io/part-of=kube-prometheus
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.96.63.216
IPs: 10.96.63.216
Port: web 9090/TCP
TargetPort: web/TCP
Endpoints: 10.1.1.15:9090,10.1.1.17:9090
Port: reloader-web 8080/TCP
TargetPort: reloader-web/TCP
Endpoints: 10.1.1.15:8080,10.1.1.17:8080
Session Affinity: ClientIP
Events:

[douglascamata]

@techvd I investigated and found the root cause of the problem, together with two solutions. Please see #1648 (comment).