Adding an "instance" tag for the prometheus scraper

[candlerb]

I've been playing with the prometheus scraper, and it's pretty awesome:

[sources.node_exporter]
type = "prometheus_scrape"
endpoints = [
  "http://host1:9100/metrics",
  "http://host2:9100/metrics",
]
scrape_interval_secs = 60

# Print scraped metrics to stdout
[sinks.print]
type = "console"
inputs = ["node_exporter"]
encoding.codec = "json"

Scraping works instantly. The problem is that I can't see how to distinguish the metrics originating from different hosts. e.g.

$ vector --config vector-scrape.toml  | grep node_memory_Active_bytes
{"name":"node_memory_Active_bytes","timestamp":"2021-09-23T17:14:30.775368953Z","kind":"absolute","gauge":{"value":1648369664.0}}
{"name":"node_memory_Active_bytes","timestamp":"2021-09-23T17:14:30.817781032Z","kind":"absolute","gauge":{"value":305164288.0}}

Which host does each metric belong to??

With a "real" prometheus server, it would add a label instance="...", which defaults to the __address__ which was scraped.

What's the standard way in Vector of distinguish the same metric from multiple sources: a tag? A namespace? Does that association already exist for prometheus scrapes, but is hidden in the "console" sink output?

I can see that internal metrics have tags.host_key and tags.pid_key, but I don't see anything equivalent for prometheus scraping.

I'm only just getting to grips with the Vector data model, so I apologise if I've missed something obvious.

[jszwedko]

Hey @candlerb ! You are absolutely right, we should be adding an instance tag like Prometheus's scraper does. I'll convert this discussion to an issue to track adding that.

[candlerb]

Moved to #9322