Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MongoDB connectivity issue #19236

Open
mkrum001 opened this issue Dec 10, 2024 · 1 comment
Open

MongoDB connectivity issue #19236

mkrum001 opened this issue Dec 10, 2024 · 1 comment

Comments

@mkrum001
Copy link

Hello,
I have issues with connecting to MongoDB Atlas replicas. I have followed this doccumentation I have 3 MongoDB instances ( : ) from the replica set, configured in my Kubernetes service. They were taken from the "dig" command output, port is the default 27017.

My current config looks like:

apiVersion: v1
kind: Service
metadata:
  name: datadog-mongodb
  labels:
    tags.datadoghq.com/env: 'dev'
    tags.datadoghq.com/service: 'mongodb-atlas'
  annotations:
    ad.datadoghq.com/service.checks: |
      {
        "mongo": {
          "init_config": {},
          "instances": [
            {
              "hosts": ["somecluster-shard-00-00.XXX.mongodb.net"],
              "username": "ENC[k8s_secret@monitoring/datadog-secrets/user]",
              "password": "ENC[k8s_secret@monitoring/datadog-secrets/password]",
              "options": {
                "authSource": "admin"
              },
              "dbm": true,
              "cluster_name": "dev",
              "additional_metrics": ["metrics.commands", "tcmalloc", "top", "collection"],
              "collections_indexes_stats": true,
              "database_autodiscovery": {
                "enabled": true
              }
            },
            {
              "hosts": ["somecluster-shard-00-01.XXX.mongodb.net"],
              "username": "ENC[k8s_secret@monitoring/datadog-secrets/user]",
              "password": "ENC[k8s_secret@monitoring/datadog-secrets/password]",
              "options": {
                "authSource": "admin"
              },
              "dbm": true,
              "cluster_name": "dev",
              "additional_metrics": ["metrics.commands", "tcmalloc", "top", "collection"],
              "collections_indexes_stats": true,
              "database_autodiscovery": {
                "enabled": true
              }
            },
            {
              "hosts": ["somecluster-shard-00-02.XXX.mongodb.net"],
              "username": "ENC[k8s_secret@monitoring/datadog-secrets/user]",
              "password": "ENC[k8s_secret@monitoring/datadog-secrets/password]",
              "options": {
                "authSource": "admin"
              },
              "dbm": true,
              "cluster_name": "dev",
              "additional_metrics": ["metrics.commands", "tcmalloc", "top", "collection"],
              "collections_indexes_stats": true,
              "database_autodiscovery": {
                "enabled": true
              }
            }
          ]
        }
      }
spec:
  ports:
    - port: 27017
      protocol: TCP
      targetPort: 27017
      name: mongodb
---
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    site: {{ .Values.agent.spec.global.site }}
    credentials:
      apiSecret:
        secretName: {{ .Values.agent.spec.global.secret_apikey }}
        keyName: api-key
      appSecret:
        secretName: {{ .Values.agent.spec.global.secret_appkey }}
        keyName: app-key
  features:
    liveContainerCollection:
      enabled: true
    logCollection:
      enabled: true
      containerCollectAll: true
    clusterChecks:
      enabled: true
  override:
    nodeAgent:
      image:
        name: {{ .Values.agent.image }}
      tolerations:
        - operator: Exists
      containers:
        agent:
          env:
            - name: DD_SECRET_BACKEND_COMMAND
              value: "/readsecret_multiple_providers.sh"```

And this is the error from the "agent status" command:

root@datadog-agent-bsml4:/# agent status | grep mongo
    mongo (8.1.0)
      Instance ID: mongo:20835cded2c966e5 [ERROR]
      Configuration Source: kube_services:kube_service://monitoring/datadog-mongodb
      Error: dev-shard-00-02.XXX.mongodb.net:27017: connection closed (configured timeouts: socketTimeoutMS: 30000.0ms, connectTimeoutMS: 30000.0ms), Timeout: 29.999983082991093s, Topology Description: <TopologyDescription id: 6757043df9e935be6e8089b6, topology_type: Single, servers: [<ServerDescription ('dev-shard-00-02.XXX.mongodb.net', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('dev-shard-00-02.XXX.mongodb.net:27017: connection closed (configured timeouts: socketTimeoutMS: 30000.0ms, connectTimeoutMS: 30000.0ms)')>]>
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/mongo/mongo.py", line 283, in check
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/mongo/mongo.py", line 270, in check
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/mongo/mongo.py", line 290, in _refresh_metadata
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/mongo/api.py", line 85, in server_info
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/mongo_client.py", line 1928, in server_info
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/_csot.py", line 107, in csot_wrapper
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/database.py", line 893, in command
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/mongo_client.py", line 1375, in _conn_for_reads
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/mongo_client.py", line 1322, in _select_server
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/topology.py", line 368, in select_server
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/topology.py", line 346, in _select_server
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/topology.py", line 253, in select_servers
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/topology.py", line 303, in _select_servers_loop
      pymongo.errors.ServerSelectionTimeoutError: dev-shard-00-02.XXX.mongodb.net:27017: connection closed (configured timeouts: socketTimeoutMS: 30000.0ms, connectTimeoutMS: 30000.0ms), Timeout: 29.999983082991093s, Topology Description: <TopologyDescription id: 6757043df9e935be6e8089b6, topology_type: Single, servers: [<ServerDescription ('dev-shard-00-02.XXX.mongodb.net', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('dev-shard-00-02.XXX.mongodb.net:27017: connection closed (configured timeouts: socketTimeoutMS: 30000.0ms, connectTimeoutMS: 30000.0ms)')>]>

Auth info of the user that I'm using:

Atlas dev-shard-0 [primary] admin> db.runCommand({ connectionStatus: 1 }).authInfo
{
  authenticatedUsers: [ { user: 'datadog', db: 'admin' } ],
  authenticatedUserRoles: [
    { role: 'datadog', db: 'admin' },
    { role: 'read', db: 'config' },
    { role: 'read', db: 'local' },
    { role: 'read', db: 'mydb' },
    { role: 'read', db: 'clusterMonitor' },
    { role: 'read', db: 'admin' }
  ]
}

In the Mongo database network config I have 0.0.0.0/0 so it is not restricting me from connecting.
Telnet from datadog pod is able to connect on 27017 for any of the three hosts.

I tried to troubleshoot with the following commands:

mongosh "mongodb+srv://user:[email protected]/admin

and successfuly connecting to the cluster address, but as far as I understand it is not correct to use Cluster host. I even tried, but the agent can't resolve shards hosts when using cluster host address.

Then with this one for connecting to the shard host:

mongosh "mongodb://user:[email protected]"

I've got:

MongoServerSelectionError: read ECONNRESET. It looks like this is a MongoDB Atlas cluster. Please ensure that your Network Access List allows connections from your IP.
@mkrum001
Copy link
Author

I was able to make the connection by adding the tls option in service configuration:

"options": {
  "authSource": "admin",
  "tls": "true"
},

Looks like it is required for mongo atlas connectiivity. May be it is a good idea to be added to the "Configure with Kubernetes service annotations" documentation. It is part of the other examples, but not from that one.
The issue can be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant