Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buffer refactor #132

Merged
merged 20 commits into from
Feb 2, 2024
Merged

Buffer refactor #132

merged 20 commits into from
Feb 2, 2024

Conversation

approxit
Copy link
Collaborator

What I've done:

  • Refactored whole buffer and scoring buffer from golem.managers.proposals.plugins
  • Introduced composable buffers - simple buffer / background fill / expiration
  • Introduced Waiter to take the heavy lifting from simple buffer
  • Introduced SingleUseSemaphore to take the heavy lifting from bacground fill buffer
  • Moved asyncio related stuff from golem.utils to single name space golem.utils.asyncio, as stuff started to be mixed with other utils
  • Tests for coverage
  • Reformatted the code

Notable remarks:

  • Low level asyncio is tricky, but quite fun to play around :D
  • SimpleBuffer covers quite fully functionality from ErrorReportingQueue, I would remove the second one in favor of the first, but team have different opinions about this, so it will be left as is
  • PR will be changed from draft after ray-on-golem testing session

@approxit
Copy link
Collaborator Author

In addition to main PR descriptions, here are more stuff done:

  • Fixed some log lines to use % instead of f-strings.
  • Use cancel_and_await/cancel_and_await_many when applicable.
  • Renamed Buffer plugin to BufferPlugin from managers level
  • Fixed checks, which require some lib upgrades, and also fix version of python
  • Added python 3.12 in CI
  • Fix python 3.12 checks complaints

@approxit approxit marked this pull request as ready for review January 24, 2024 16:44
@approxit
Copy link
Collaborator Author

And another update:

  • Demand and Proposal resources got new method to get their expiration dates
  • RefreshingDemandManager uses deamnd's expiration date instead of misused property
  • Buffer got remixed interface to support race conditions in buffer composition.

Copy link
Collaborator

@shadeofblue shadeofblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whew... that's a lot of code ... I don't feel I have finished the review and probably it would be good to touch base on a call anyway but here are some comments that I have had...

golem/managers/proposal/plugins/buffer.py Outdated Show resolved Hide resolved
golem/managers/proposal/plugins/buffer.py Outdated Show resolved Hide resolved
golem/managers/proposal/plugins/scoring/scoring_buffer.py Outdated Show resolved Hide resolved
golem/utils/asyncio/buffer.py Outdated Show resolved Hide resolved
golem/utils/asyncio/buffer.py Outdated Show resolved Hide resolved
golem/managers/proposal/plugins/buffer.py Outdated Show resolved Hide resolved
golem/managers/proposal/plugins/scoring/scoring_buffer.py Outdated Show resolved Hide resolved
golem/managers/proposal/plugins/scoring/scoring_buffer.py Outdated Show resolved Hide resolved
golem/utils/asyncio/buffer.py Outdated Show resolved Hide resolved
golem/utils/typing.py Show resolved Hide resolved
Copy link
Collaborator

@shadeofblue shadeofblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please see my comments... otherwise, lgtm afaict

"Target items is now `%s` which is not below min size `%d`, requesting fill not"
" needed",
items_count,
"Proposals count %d is not below minimum size %d, skipping fill",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why "not below" instead of "above"?

Suggested change
"Proposals count %d is not below minimum size %d, skipping fill",
"Proposals count %d is above minimum size %d, skipping fill",

golem/managers/proposal/plugins/buffer.py Show resolved Hide resolved
golem/managers/proposal/plugins/buffer.py Show resolved Hide resolved
@approxit approxit merged commit feb21c7 into main Feb 2, 2024
11 checks passed
@approxit approxit deleted the approxit/buffer-refactor branch February 2, 2024 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants