Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Runtime panic with rabbitmq input #16430

Open
thexeos opened this issue Jan 24, 2025 · 2 comments
Open

Runtime panic with rabbitmq input #16430

thexeos opened this issue Jan 24, 2025 · 2 comments
Labels
bug unexpected problem or unintended behavior

Comments

@thexeos
Copy link

thexeos commented Jan 24, 2025

Relevant telegraf.conf

[[inputs.rabbitmq]]
  url = "http://127.0.0.1:15672"
  tags = {name = "<hostname>"}
  username = "<username>"
  password = "<password>"

Logs from Telegraf

2025-01-24T00:26:50Z D! [inputs.rabbitmq] Requesting "http://127.0.0.1:15672/api/nodes"...
2025-01-24T00:26:50Z D! [inputs.rabbitmq] Requesting "http://127.0.0.1:15672/api/queues"...
2025-01-24T00:26:50Z D! [inputs.rabbitmq] Requesting "http://127.0.0.1:15672/api/exchanges"...
2025-01-24T00:26:50Z D! [inputs.rabbitmq] Requesting "http://127.0.0.1:15672/api/federation-links"...
2025-01-24T00:26:50Z D! [inputs.rabbitmq] Requesting "http://127.0.0.1:15672/api/overview"...
panic: <memcall> could not acquire lock on 0x7f5f3ac73000, limit reached? [Err: cannot allocate memory]

goroutine 1998 [running]:
github.com/awnumar/memguard/core.Panic(...)
        /go/pkg/mod/github.com/awnumar/[email protected]/core/exit.go:86
github.com/awnumar/memguard/core.NewBuffer(0x20)
        /go/pkg/mod/github.com/awnumar/[email protected]/core/buffer.go:73 +0x531
github.com/awnumar/memguard/core.(*Coffer).View(0xc0026f4ea0)
        /go/pkg/mod/github.com/awnumar/[email protected]/core/coffer.go:83 +0x57
github.com/awnumar/memguard/core.Open(0xc0025a5c80)
        /go/pkg/mod/github.com/awnumar/[email protected]/core/enclave.go:115 +0x3f
github.com/awnumar/memguard.(*Enclave).Open(0xc002980008?)
        /go/pkg/mod/github.com/awnumar/[email protected]/enclave.go:43 +0x1a
github.com/influxdata/telegraf/config.(*protectedSecretContainer).Buffer(0xc0018b7be0?)
        /go/src/github.com/influxdata/telegraf/config/secret_protected.go:117 +0x25
github.com/influxdata/telegraf/config.(*Secret).Get(0xc002734060)
        /go/src/github.com/influxdata/telegraf/config/secret.go:193 +0x64
github.com/influxdata/telegraf/plugins/inputs/rabbitmq.(*RabbitMQ).requestEndpoint(0xc002734008, {0x9ae702d, 0xd})
        /go/src/github.com/influxdata/telegraf/plugins/inputs/rabbitmq/rabbitmq.go:338 +0x25e
github.com/influxdata/telegraf/plugins/inputs/rabbitmq.(*RabbitMQ).requestJSON(0x0?, {0x9ae702d, 0xd}, {0x82fb520, 0xc00292a530})
        /go/src/github.com/influxdata/telegraf/plugins/inputs/rabbitmq/rabbitmq.go:363 +0x3f
github.com/influxdata/telegraf/plugins/inputs/rabbitmq.gatherOverview(0xc002734008, {0xaa12de0, 0xc00285c2a0})
        /go/src/github.com/influxdata/telegraf/plugins/inputs/rabbitmq/rabbitmq.go:387 +0x97
github.com/influxdata/telegraf/plugins/inputs/rabbitmq.(*RabbitMQ).Gather.func1(0xc0028937a8?)
        /go/src/github.com/influxdata/telegraf/plugins/inputs/rabbitmq/rabbitmq.go:306 +0x55
created by github.com/influxdata/telegraf/plugins/inputs/rabbitmq.(*RabbitMQ).Gather in goroutine 1997
        /go/src/github.com/influxdata/telegraf/plugins/inputs/rabbitmq/rabbitmq.go:304 +0x1bc
panic: runtime error: index out of range [0] with length 0

goroutine 2002 [running]:
github.com/awnumar/memguard/core.(*Coffer).View(0xc0026f4ea0)
        /go/pkg/mod/github.com/awnumar/[email protected]/core/coffer.go:92 +0x1d8
github.com/awnumar/memguard/core.Open(0xc0025a5c80)
        /go/pkg/mod/github.com/awnumar/[email protected]/core/enclave.go:115 +0x3f
github.com/awnumar/memguard.(*Enclave).Open(0xc000217108?)
        /go/pkg/mod/github.com/awnumar/[email protected]/enclave.go:43 +0x1a
github.com/influxdata/telegraf/config.(*protectedSecretContainer).Buffer(0xc002882bb0?)
        /go/src/github.com/influxdata/telegraf/config/secret_protected.go:117 +0x25
github.com/influxdata/telegraf/config.(*Secret).Get(0xc002734060)
        /go/src/github.com/influxdata/telegraf/config/secret.go:193 +0x64
github.com/influxdata/telegraf/plugins/inputs/rabbitmq.(*RabbitMQ).requestEndpoint(0xc002734008, {0x9acb9dd, 0xa})
        /go/src/github.com/influxdata/telegraf/plugins/inputs/rabbitmq/rabbitmq.go:338 +0x25e
github.com/influxdata/telegraf/plugins/inputs/rabbitmq.(*RabbitMQ).requestJSON(0xc002897650?, {0x9acb9dd, 0xa}, {0x82fb5a0, 0xc002a8f4d0})
        /go/src/github.com/influxdata/telegraf/plugins/inputs/rabbitmq/rabbitmq.go:363 +0x3f
github.com/influxdata/telegraf/plugins/inputs/rabbitmq.gatherNodes(0xc002734008, {0xaa12de0, 0xc00285c2a0})
        /go/src/github.com/influxdata/telegraf/plugins/inputs/rabbitmq/rabbitmq.go:435 +0x8c
github.com/influxdata/telegraf/plugins/inputs/rabbitmq.(*RabbitMQ).Gather.func1(0xc0035a2900?)
        /go/src/github.com/influxdata/telegraf/plugins/inputs/rabbitmq/rabbitmq.go:306 +0x55
created by github.com/influxdata/telegraf/plugins/inputs/rabbitmq.(*RabbitMQ).Gather in goroutine 1997
        /go/src/github.com/influxdata/telegraf/plugins/inputs/rabbitmq/rabbitmq.go:304 +0x1bc

System info

Telegraf 1.33.1, Ubuntu 14.04.6

Docker

No response

Steps to reproduce

This started happening after upgrading to v1.33. I do not have the previous version number, but I believe it was at least v1.29 if not more recent.

Expected behavior

No crash

Actual behavior

Crash after 38 cycles of collection/writing have been completed. The crash occurs during 39th cycle after startup.

Additional info

No response
@thexeos thexeos added the bug unexpected problem or unintended behavior label Jan 24, 2025
@srebhan
Copy link
Member

srebhan commented Jan 24, 2025

@thexeos you should have seen a warning about the required amount of lockable memory when starting Telegraf. This memory is required to handle credentials/secrets in a protect fashion. So you either need to increase the limit for the amount of lockable memory or you use unprotected secrets by using the --unprotected command line option.

@thexeos
Copy link
Author

thexeos commented Jan 25, 2025

@srebhan thank you for the suggestion, unfortunately there was no warning like that.

This is all that was printed at the time:

2025-01-24T00:20:24Z I! Loading config: /etc/telegraf/telegraf.conf
2025-01-24T00:20:24Z W! DeprecationWarning: Option "precision" of plugin "outputs.influxdb" deprecated since version 1.0.0 and will be removed in 1.35.0: option is ignored
2025-01-24T00:20:24Z I! Starting Telegraf 1.33.1 brought to you by InfluxData the makers of InfluxDB
2025-01-24T00:20:24Z I! Available plugins: 236 inputs, 9 aggregators, 33 processors, 26 parsers, 63 outputs, 6 secret-stores
2025-01-24T00:20:24Z I! Loaded inputs: cpu disk diskio haproxy mem net rabbitmq redis sensors swap
2025-01-24T00:20:24Z I! Loaded aggregators:
2025-01-24T00:20:24Z I! Loaded processors:
2025-01-24T00:20:24Z I! Loaded secretstores:
2025-01-24T00:20:24Z I! Loaded outputs: influxdb
2025-01-24T00:20:24Z I! Tags enabled: host=osaka
2025-01-24T00:20:24Z W! Deprecated outputs: 0 and 1 options
2025-01-24T00:20:24Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"osaka", Flush Interval:10s
2025-01-24T00:20:24Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2025-01-24T00:20:24Z D! [agent] Initializing plugins
2025-01-24T00:20:24Z W! DeprecationWarning: Value "false" for option "ignore_protocol_stats" of plugin "inputs.net" deprecated since version 1.27.3 and will be removed in 1.36.0: use the 'inputs.nstat' plugin instead for protocol stats
2025-01-24T00:20:24Z D! [agent] Connecting outputs
2025-01-24T00:20:24Z D! [agent] Attempting connection to [outputs.influxdb]
2025-01-24T00:20:24Z D! [agent] Successfully connected to outputs.influxdb
2025-01-24T00:20:24Z D! [agent] Starting service inputs
2025-01-24T00:20:34Z D! [outputs.influxdb] Wrote batch of 89 metrics in 44.562106ms
2025-01-24T00:20:34Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2025-01-24T00:20:34Z D! [inputs.rabbitmq] Requesting "http://127.0.0.1:15672/api/exchanges"...
2025-01-24T00:20:34Z D! [inputs.rabbitmq] Requesting "http://127.0.0.1:15672/api/overview"...
2025-01-24T00:20:34Z D! [inputs.rabbitmq] Requesting "http://127.0.0.1:15672/api/federation-links"...
2025-01-24T00:20:34Z D! [inputs.rabbitmq] Requesting "http://127.0.0.1:15672/api/nodes"...
2025-01-24T00:20:34Z D! [inputs.rabbitmq] Requesting "http://127.0.0.1:15672/api/queues"...
2025-01-24T00:20:34Z D! [inputs.rabbitmq] HTTP status code: 200 OK
2025-01-24T00:20:34Z D! [inputs.rabbitmq] HTTP status code: 404 Not Found
2025-01-24T00:20:34Z D! [inputs.rabbitmq] HTTP status code: 200 OK
2025-01-24T00:20:34Z E! [inputs.rabbitmq] Error in plugin: getting "/api/federation-links" failed: 404 Not Found
2025-01-24T00:20:34Z D! [inputs.rabbitmq] HTTP status code: 200 OK
2025-01-24T00:20:34Z D! [inputs.rabbitmq] Requesting "http://127.0.0.1:15672/api/nodes/bunny@osaka/memory"...
2025-01-24T00:20:34Z D! [inputs.rabbitmq] HTTP status code: 200 OK
2025-01-24T00:20:35Z D! [inputs.rabbitmq] HTTP status code: 200 OK
2025-01-24T00:20:35Z D! [inputs.rabbitmq] HTTP status code: 200 OK

However, after rebooting the issue seems to have fixed itself. But since the cause of the panic crash is still somewhere in the code waiting for some condition to trigger it, I will keep the issue open for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@thexeos @srebhan