Offline deduplication with two replica labels

[LukaszWasko]

I'm looking for working config for Thanos Compact to deduplicate data with two replica labels.
I don't know how to deal with these labels - one generated on prometheus nodes, secodn on thanos receivers (with hashring).

My config:
Prometheus Agent nodes scraping the same targets:

#prom-agentX1
global:
  external_labels:
    cluster: prod-ipl-1
    replica: 0
#prom-agentX2
global:
  external_labels:
    cluster: prod-ipl-1
    replica: 1

#thanos-receiver1
THANOS_RECEIVE_OPTS=--objstore.config-file /etc/thanos/s3.yaml \
  --tsdb.path=/var/lib/thanos/receive \
  --grpc-address=0.0.0.0:10907 \
  --http-address=0.0.0.0:10904 \
  --remote-write.address=0.0.0.0:19291 \
  --label='receive_replica="0"' \
  --receive.replication-factor=1 \
  --receive.local-endpoint=thanos-receiver1:10907 \
  --receive.hashrings-file=/etc/thanos/hashring.json

#thanos-receiver2
THANOS_RECEIVE_OPTS=--objstore.config-file /etc/thanos/s3.yaml \
  --tsdb.path=/var/lib/thanos/receive \
  --grpc-address=0.0.0.0:10907 \
  --http-address=0.0.0.0:10904 \
  --remote-write.address=0.0.0.0:19291 \
  --label='receive_replica="1"' \
  --receive.replication-factor=1 \
  --receive.local-endpoint=thanos-receiver2:10907 \
  --receive.hashrings-file=/etc/thanos/hashring.json

#thanos-receiver3
THANOS_RECEIVE_OPTS=--objstore.config-file /etc/thanos/s3.yaml \
  --tsdb.path=/var/lib/thanos/receive \
  --grpc-address=0.0.0.0:10907 \
  --http-address=0.0.0.0:10904 \
  --remote-write.address=0.0.0.0:19291 \
  --label='receive_replica="2"' \
  --receive.replication-factor=1 \
  --receive.local-endpoint=thanos-receiver3:10907 \
  --receive.hashrings-file=/etc/thanos/hashring.json

and compactor:

THANOS_COMPACT_OPTS='--objstore.config-file /etc/thanos/s3.yaml \
  --data-dir=/var/lib/thanos/compact \
  --http-address=0.0.0.0:10904 \
  --wait \
  --wait-interval 30s \
  --consistency-delay 0s \
  --deduplication.replica-label="replica" \
  --deduplication.func=penalty \
  --compact.enable-vertical-compaction \
  --log.format=json'

So the compactor knows to deduplicate prometheus replica label, but what about receiver label receive_replica?
This is the result:

and to visualize the connections:

EDIT 1:

#/etc/thanos/hashring.json:
[
    {
        "endpoints": [
            "thanos-receiver1:10907",
            "thanos-receiver2:10907",
            "thanos-receiver3:10907"
        ]
    }
]

[LukaszWasko]

No response yet. So maybe I should start a question - which "--deduplication.func" on Compactor should I use if two prometheus agents (label replica: [0, 1]) scrapes the same target and then metrics goes to receivers with hashring (another label receive_replica: [0,1]) to proper offline deduplication work?
I suppose it should be only one metic eg: replica=0, receive_replica=0. The rest should be removed.

[yeya24]

When you run your thanos compactor, actually you can specify multiple deduplication labels by repeating the duplication label flag

[LukaszWasko]

When you run your thanos compactor, actually you can specify multiple deduplication labels by repeating the duplication label flag

Sure I did so in many of my test. But I asking about --deduplication.func:

1. one-to-one (..) This is very common when using Receivers with replication greater than 1 as receiver replication copies samples exactly_
2. penalty deduplication is when the same data is duplicated logically, i.e. the same application is scraped from two different Prometheis. (..) This is a common case when Prometheus HA replicas are used.
   My prometheus scrapes the same target and I'm using 3 x receivers with hashring and "replication greater than 1" (--receive.replication-factor 2)

[yeya24]

If you have multiple prometheus replicas then just choose penalty. It can handle receiver replicas case as well, but using penalty algorithm, not one to one exact match.
The default one to one cannot handle two prometheus replicas cases.