Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tidb logs are flooded with "unable to find a store with given labels" #1371

Open
mittalrishabh opened this issue Jun 21, 2024 · 1 comment
Open

Comments

@mittalrishabh
Copy link

Issue
We see a LOT of such logs in our clusters. 2.5M logs in 1hr across our 180 tidb pods.

[2024/06/13 22:39:50.475 +00:00] [WARN] [region_request.go:661] ["unable to find a store with given labels"] [region=2599577092] [stale-read=true] [labels="[{"key":"zone","value":"us-east-1a"}]"]

What we expect
It is spamming the logs. We would like to make it debug only if it is not a critical message.

It seems like it happens when a request is tried on all replicas and is failed. We are seeing 4 request/sec/tidb (2.5M/180/3600) failing on all replicas. I think it is an expected behavior in distributed systems.

@crazycs520
Copy link
Contributor

This log has been removed in the master branch since replica selector v2 was introduced by https://github.com/tikv/client-go/pull/1265/files#diff-0c36d07fbd1fc71c850ff57d323fe12bdf7f2c053ef71b7fec5bb44358f1070dL1003

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants