Performance multiple store (bucket)

[thibautmery]

Hi,

I have few question before creating "huge" infrastructure. (more thans 1000 servers / many differnt exporters / metrics push / ...).

I want to deploy a stack with prometheus / thanos receiver / thanos store / thanos compactor and thanos query.

We need to split metrics to different bucket and have different long term retetion time. I don't know what are the limit and if at a certain number of bucket, thanos become more and more slowler ?

If we have for exemple 30 different buckets. Is it working without any problem ? Thanos query is able to query all the store and be fast as if we have only one bucket ? What are the "recommandation" about using multiple bucket and store ?

I don't see other feedback about this kind of question.

Thank you very much

[GiedriusS]

With this scale, you should think about the strategy of partitioning. What I mean by that is that all your metrics in different buckets should ideally be divided into many distinct labels. Thanos Query uses those labels to "know" where to send a query. If all your queries will need to be sent to many Thanos Stores then it won't be that fast. Another thing to consider is using hashmod to automatically scale your compactors according to different external labels. Finally, typically the write path isn't the slowest thing, the most important thing is to have a nice caching layer and to ensure that your compactors are always working & caught up with the compaction work stream. I hope this helps.

[thibautmery]

Thank you for your answer, I understand what you mean.

So there is a second one question comes with this answer. If caching is one of the biggest part, is it interesting to duplicate the query frontend instead of creating a big one ?

I guess we can split by needs (infrastructure / developper / ... ) ?

Or one good thanos query frontend with a redis is a good option ?

Thank you :)