Thanos does not query.

[SungYil]

The Thanos server does not query the client.

thanos server value.yaml

image:
  tag: 0.24.0
objstoreConfig: |-
  type: s3
  config:
    bucket: s3
    region: ap-northeast-2
    endpoint: s3.ap-northeast-2.amazonaws.com
    access_key: $(AWS_ACCESS_KEY)
    secret_key: $(AWS_SECRET_KEY)
    insecure: false
    signature_version2: false
querier:
  stores:
    - thanos.otherdomain.com:80
    - thanos-discovery.otherdomain.com:80
    - thanos-discovery.otherdomain.com80
    - thanos-discovery.otherdomain.com:80
    - thanos-discovery.otherdomain.com:80
    - thanos-discovery.otherdomain.com:80
    - thanos-discovery.noquerydomain.com:80
  service:
    type: ClusterIP
    ports:
      http: 9090
    nodePorts:
      http: "31254"
    cplusterIP: ""
bucketweb:
  enabled: false
compactor:
  enabled: false
storegateway:
  enabled: true
  extraEnvVars:
    - name: AWS_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          key: AWS_ACCESS_KEY
          name: aws-secret
    - name: AWS_SECRET_KEY
      valueFrom:
        secretKeyRef:
          key: AWS_SECRET_KEY
          name: aws-secret
ruler:
  enabled: false

client value.yaml

prometheus:
  thanosService:
    enabled: true
  extraSecret:
    name: bucket
    data:
      objstore.yml: |
        type: s3
        config:
          bucket: s3
          region: ap-northeast-2
          endpoint: s3.ap-northeast-2.amazonaws.com
          access_key: $(AWS_ACCESS_KEY)
          secret_key: $(AWS_SECRET_KEY)
          insecure: false
          signature_version2: false
  prometheusSpec:
    image:
      tag: v2.47.0
    containers:
      - name: thanos-sidecar
        env:
          - name: AWS_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                key: AWS_ACCESS_KEY
                name: secret
          - name: AWS_SECRET_KEY
            valueFrom:
              secretKeyRef:
                key: AWS_SECRET_KEY
                name: secret
    externalLabels:
      cluster: "client"
    disableCompaction: true
    thanos:
      objectStorageConfig:
        existingSecret:
          name: bucket-config
          key: objstore.yml
grafana:
  enabled: false
  adminPassword: password
  defaultDashboardsEnabled: false
  sidecar:
    datasources:
      enabled: true
      defaultDatasourceEnabled: false
defaultRules:
  create: false
alertmanager:
  enabled: false
prometheusOperator:
  thanosImage:
    tag: v0.24.0
kubeApiServer:
  serviceMonitor:
    metricRelabelings:
      # Drop excessively noisy apiserver buckets.
      - sourceLabels:
          - __name__
        action: drop
        regex: 'etcd_.*'
      - sourceLabels:
          - __name__
        action: drop
        regex: 'apiserver_.*'

server error message

• level=warn ts=2024-02-21T08:25:56.209385483Z caller=endpointset.go:500 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from thanos-discovery.noquerydomain.com:80: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=thanos-discovery.noquerydomain.com:80

It seems that Thanos queries are not reaching the client.
Other client metric information is retrieved normally.
All settings are the same.
I would like to know why I can't get metrics only for servers with the noquerydomain.com domain.
Is there a limit to the number of Thanos queries?