Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: don't double export aggregate efficiency metrics #3230

Merged
merged 1 commit into from
Oct 24, 2024
Merged

Conversation

cgrinds
Copy link
Collaborator

@cgrinds cgrinds commented Oct 24, 2024

This was discovered by log messages from Prometheus like this:

msg="Error on ingesting samples with different value but same timestamp"

after enabling debug logging in Prometheus more information is printed

Oct 24 04:24:07 harvest-ubuntu-03 prometheus[32020]: ts=2024-10-24T08:24:07.702Z caller=scrape.go:1859 level=debug component="scrape manager" scrape_pool=prometheus target=
http://localhost:13002/metrics
msg="Duplicate sample for timestamp" series="aggr_total_physical_used{aggr="umeng_aff300_aggr2",cluster="umeng-aff300-01-02",datacenter="nane",node="umeng-aff300-01"}"

This happened when you specified collectors in this order:

collectors:
  - Rest
  - Zapi

The Zapi collector exports aggregate efficiency objects via the AggregateEfficiency template while the REST collector exports them via the Aggregate template. That means the normal object/collector dedup logic won't handle these metrics and they are double exported.

This fix removes the double export and Prometheus ingest error.

@cgrinds cgrinds merged commit 7f1dae4 into main Oct 24, 2024
6 checks passed
@cgrinds cgrinds deleted the cbg-aggr branch October 24, 2024 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants