Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grafana-agent should monitor all enabled collectors #178

Open
err404r opened this issue Sep 6, 2024 · 1 comment
Open

Grafana-agent should monitor all enabled collectors #178

err404r opened this issue Sep 6, 2024 · 1 comment

Comments

@err404r
Copy link

err404r commented Sep 6, 2024

Bug Description

When one of the collectors defined in grafana-agent goes down or hangs, it would be nice to have an alert, for it. Some collectors are generating such rules on it's own like hardware observer charm. Others are not. Also this apply to collectors provide by grafana-agent itself. For example node_exporters send node_scrape_collector_success metrics which is 1 when the last attempt was successful and 0 if it is not. So this is an obvious rule, however some in depth research is needed here

To Reproduce

  1. Deploy grafana-agent, for now it will install snap without --classic option.
  2. Check node_scrape_collector_success metrics
  3. This applied to other exporters, but I wasn't able to identify a good metric for them. Some kind of grafan-agent scrape error counter could be used for that. But I wasn't able to find one

Environment

Relevant log output

-

Additional context

No response

@sed-i
Copy link
Contributor

sed-i commented Sep 27, 2024

We should find the analogue of node_scrape_collector_success to all the "enabled by default" collectors.
We should also document somewhere that we need to add those self checks when enabling "disabled by default" collectors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants