New to Thanos. Questions about PVC of thanos components and S3 storage

[kappa-wingman]

Hello, I am new to Thanos.
I am using kube-prometheus (Jsonnet) to deploy the monitoring tools.

Now I added Thanos (compact, store and query only) to the jsonnet template.
I have some questions:

1. Do thanos-query need a PVC?
2. What is the use of the PVC for thanos-compact and thanos-store?
3. When is the data stored in S3 storage?
   Just now I reduce the Prometheus retention period from 5d to 1d, then I lost the old metrics.
   How to copy existing data in Prometheus PVC to S3 immediately?

Thanks in advance.

[yeya24]

Do thanos-query need a PVC?

No.

What is the use of the PVC for thanos-compact and thanos-store?

Thanos compact uses PVC to cache block metadata and compact/downsample blocks.

Thanos store uses PVC to cache block metadata and index headers.

When is the data stored in S3 storage?

Thanos sidecar uploads to S3 whenever a new block is created.

Just now I reduce the Prometheus retention period from 5d to 1d, then I lost the old metrics.
How to copy existing data in Prometheus PVC to S3 immediately?

So if you made the retention change already, then I think Prometheus already deleted the data older than 1d.
To upload all blocks you can enable thanos sidecar to copy compacted blocks to S3 by enabling --shipper.upload-compacted.

[kappa-wingman]

I face another problem. How to enable '--shipper.upload-compacted'?

I am using the jsonnet (kube-prometheus and kube-thanos).
Since it is an one-off options, I tried to edit the STS and add it manually to the STS (in side sidecar section, in the args) but the option is gone after a while.

I also checked this issue: prometheus-operator/kube-prometheus#1066