Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix worker&fuse options & worker tiredstore #3344

Merged
merged 7 commits into from
Jul 24, 2023
Merged

fix worker&fuse options & worker tiredstore #3344

merged 7 commits into from
Jul 24, 2023

Conversation

zwwhdls
Copy link
Member

@zwwhdls zwwhdls commented Jul 17, 2023 •
edited
Loading

Ⅰ. Describe what this PR does

  1. add options in JuiceFSRuntime.spec.fuse and get mount options of fuse pod from it instead of dataset.mount.option
  2. get worker cache from tiredstore
  3. cache-size&cache-dir in worker.options is deprecated, send event in runtime
  4. fix cache capacity in dataset when cache-size is set in worker.options.

Ⅱ. Does this pull request fix one issue?

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

@fluid-e2e-bot
Copy link

fluid-e2e-bot bot commented Jul 17, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@zwwhdls
Copy link
Member Author

zwwhdls commented Jul 17, 2023

/test fluid-e2e

@zwwhdls
fix staticcheck
746189d 
Signed-off-by: zwwhdls <[email protected]>
@zwwhdls
Copy link
Member Author

zwwhdls commented Jul 17, 2023

/retest

@zwwhdls
Copy link
Member Author

zwwhdls commented Jul 17, 2023

/test fluid-e2e

@zwwhdls zwwhdls marked this pull request as ready for review July 19, 2023 09:29
@zwwhdls
Copy link
Member Author

zwwhdls commented Jul 19, 2023

/test fluid-e2e

@codecov
Copy link

codecov bot commented Jul 19, 2023 •
edited
Loading

Codecov Report

Merging #3344 (c2ad341) into master (e7448cb) will decrease coverage by 0.02%.
The diff coverage is 73.19%.

❗ Current head c2ad341 differs from pull request most recent head 2a18519. Consider uploading reports for the commit 2a18519 to get more accurate results

@@            Coverage Diff             @@
##           master    #3344      +/-   ##
==========================================
- Coverage   65.43%   65.41%   -0.02%     
==========================================
  Files         402      402              
  Lines       23244    23286      +42     
==========================================
+ Hits        15209    15232      +23     
- Misses       6254     6267      +13     
- Partials     1781     1787       +6
Impacted Files Coverage Δ
api/v1alpha1/juicefsruntime_types.go 33.33% <ø> (ø)
pkg/ddc/juicefs/transform.go 60.73% <62.74%> (-1.13%) ⬇️
pkg/ddc/juicefs/cache.go 78.40% <69.23%> (+1.86%) ⬆️
pkg/ddc/juicefs/transform_fuse.go 77.40% <90.62%> (-0.89%) ⬇️
pkg/ddc/juicefs/engine.go 82.14% <100.00%> (+0.66%) ⬆️

cheyang
cheyang reviewed Jul 19, 2023
View reviewed changes
pkg/ddc/juicefs/cache.go Outdated Show resolved Hide resolved
cheyang
cheyang reviewed Jul 19, 2023
View reviewed changes
pkg/ddc/juicefs/cache.go Outdated Show resolved Hide resolved
cheyang
cheyang reviewed Jul 19, 2023
View reviewed changes
pkg/ddc/juicefs/cache.go Outdated Show resolved Hide resolved
@zwwhdls
fix worker metric port
2856f2d 
Signed-off-by: zwwhdls <[email protected]>
@zwwhdls
fix static check
7fa569b 
Signed-off-by: zwwhdls <[email protected]>
TrafalgarZZZ
TrafalgarZZZ reviewed Jul 20, 2023
View reviewed changes
pkg/ddc/juicefs/cache.go
// cacheSize is in MiB
cachesize = cacheSizeMB * 1024 * 1024
}
if cachesize != 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the states.cacheCapacity may not be accurate when some of the juicefs worker is not Running. Can we use ready worker replicas as its multiplier?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

pkg/ddc/juicefs/cache.go Show resolved Hide resolved
@sonarcloud
Copy link

sonarcloud bot commented Jul 21, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 10 Code Smells

No Coverage information No Coverage information
8.7% 8.7% Duplication

@TrafalgarZZZ
Copy link
Member

/test fluid-e2e

cheyang
cheyang approved these changes Jul 24, 2023
View reviewed changes
Copy link
Collaborator

@cheyang cheyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@fluid-e2e-bot fluid-e2e-bot bot added the lgtm label Jul 24, 2023
@fluid-e2e-bot
Copy link

fluid-e2e-bot bot commented Jul 24, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheyang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fluid-e2e-bot fluid-e2e-bot bot merged commit 9f2926f into fluid-cloudnative:master Jul 24, 2023
8 checks passed
cheyang pushed a commit to cheyang/fluid that referenced this pull request Aug 21, 2023
@zwwhdls @cheyang
fix worker&fuse options & worker tiredstore (fluid-cloudnative#3344)
cea313d 
* fix worker&fuse options & worker tiredstore

Signed-off-by: zwwhdls <[email protected]>

* fix staticcheck

Signed-off-by: zwwhdls <[email protected]>

* fix dataset capacity & send deprecated event

Signed-off-by: zwwhdls <[email protected]>

* fix typo & add unit test

Signed-off-by: zwwhdls <[email protected]>

* fix worker metric port

Signed-off-by: zwwhdls <[email protected]>

* fix static check

Signed-off-by: zwwhdls <[email protected]>

* fix ready worker num in cache capacity

Signed-off-by: zwwhdls <[email protected]>

---------

Signed-off-by: zwwhdls <[email protected]>
Signed-off-by: cheyang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xliuqq xliuqq xliuqq left review comments

@TrafalgarZZZ TrafalgarZZZ TrafalgarZZZ left review comments

@cheyang cheyang cheyang approved these changes

Assignees

@cheyang cheyang

Labels
approved lgtm
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zwwhdls @TrafalgarZZZ @cheyang @xliuqq