Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add testcase for doubling cluster with smaller instances while keeping 90% utilization #9597

Open
Tracked by #9305
cezarmoise opened this issue Dec 20, 2024 · 7 comments
Assignees
Labels
area/elastic cloud Issues related to the elastic cloud project area/tablets P3 Medium Priority

Comments

@cezarmoise
Copy link
Contributor

Create 3 i4i.8xlarge with rf=3.
Reach 90% utilization
Add i4i.large under load
Verify space utilization
Reach 90% utilization
Add i4i.large under load
Verify space utilization
Reach 90% utilization
Add i4i.large under load
Verify space utilization

@cezarmoise cezarmoise added area/elastic cloud Issues related to the elastic cloud project area/tablets P3 Medium Priority labels Dec 20, 2024
@cezarmoise cezarmoise self-assigned this Dec 20, 2024
@cezarmoise
Copy link
Contributor Author

cezarmoise commented Dec 20, 2024

Results

Small instance is filled to 100%, test fails

Setup

Initial cluster: 3 x i4i.8xlarge
Write to aprox 90%
Mixed workload at about 30%
Add 1 instance of i4i.large
Wait for tablets to be balanced
Wait 30 minutes
Write enough data to fill a i4i.large to 90%
Repeat 3 times

Argus

Load

Image

Disk Usage

Image
Image

With fix, no error

Setup

Initial cluster: 3 x i4i.8xlarge
Write to aprox 80%
Mixed workload at about 60%
Add 1 instance of i4i.large
Wait for tablets to be balanced
Wait 30 minutes
Write enough data to fill a i4i.large to 90%
Repeat 3 times

Argus

Load

Image

Disk Usage

Image

@cezarmoise
Copy link
Contributor Author

Added first result
@pehala @swasik

After tables balanced, the small instance is at 91%, then when writting data it gets to 100 and out of space error

Image

@cezarmoise
Copy link
Contributor Author

Oppened scylladb/scylladb#22028

@cezarmoise
Copy link
Contributor Author

Replicated again, but it looks like scylladb/scylladb@b74565e is not in enterprise yet. Will re-run with master instead of enterprise

Image
https://argus.scylladb.com/tests/scylla-cluster-tests/b227928c-e2a2-4044-8301-1e06684bf61f

@cezarmoise
Copy link
Contributor Author

Using master:latest it looks ok

Image
https://argus.scylladb.com/tests/scylla-cluster-tests/8598cd23-e820-4b22-9103-b51f455a7ad2
Will update when it finishes

@cezarmoise
Copy link
Contributor Author

Updated results comment.
With the fix, no more out of space errors.
However, I think the fix also improved how much the initial space was used, so with same stress command the initial 8XL cluster is only filled to 80%. @pehala should I run again with updated params to fill to 90%?

@pehala
Copy link
Contributor

pehala commented Jan 10, 2025

Thanks, yes please rerun it but it is a good sign it improved this much

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/elastic cloud Issues related to the elastic cloud project area/tablets P3 Medium Priority
Projects
None yet
Development

No branches or pull requests

2 participants