Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[branch-2.10][improve][broker] Do not retain the data in the system topic #22031

Merged
merged 10 commits into from
Feb 19, 2024

Conversation

liangyepianzhou
Copy link
Contributor

Motivation

For some use case, the users need to store all the messages even though these message are acked by all subscription. So they set the retention policy of the namespace to infinite retention (setting both time and size limits to -1). But the data in the system topic does not need for infinite retention.

Modifications

For system topics, do not retain messages that have already been acknowledged.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

…opic (apache#22022)

### Motivation

For some use case, the users need to store all the messages even though these message are acked by all subscription.
So they set the retention policy of the namespace to infinite retention (setting both time and size limits to `-1`).  But the data in the system topic does not need for infinite retention.

### Modifications

For system topics, do not retain messages that have already been acknowledged.
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Feb 6, 2024
liangyepianzhou and others added 6 commits February 6, 2024 17:21
zookeeper 3.9.1 add a check to print a log.
```java
                if (addr.getPort() == 0) {
                    LOG.info("bound to port {}", this.ss.getLocalAddress());
                }
```
@liangyepianzhou
Copy link
Contributor Author

@yaalsn I have no idea for the failed tests. Could you please take a look? Thanks!

java.lang.IllegalStateException: Check failed: Docker environment should have more than 2GB free disk space

Copy link
Member

@dao-jun dao-jun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Technoboy- Technoboy- closed this Feb 19, 2024
@Technoboy- Technoboy- reopened this Feb 19, 2024
@liangyepianzhou
Copy link
Contributor Author

liangyepianzhou commented Feb 19, 2024

@lhotari The Python - Build 3.9 client / build-wheel and CPP build on Windows / Windows x64 tests failed after #22044 is cherry-picked into branch-2.10. Do you mind taking a look?

@BewareMyPower
Copy link
Contributor

We can ignore these two failed workflows because we no longer maintain the C++/Python code in main repo. I see it does not block merging this cherry-pick. If you have concern about the failed workflows, I think you can remove them from the CI.

@liangyepianzhou liangyepianzhou merged commit c1d8630 into apache:branch-2.10 Feb 19, 2024
35 of 39 checks passed
@liangyepianzhou liangyepianzhou deleted the retention-2.10 branch February 19, 2024 13:04
nodece pushed a commit to nodece/pulsar that referenced this pull request Feb 23, 2024
…opic (apache#22031)

### Motivation

For some use case, the users need to store all the messages even though these message are acked by all subscription. So they set the retention policy of the namespace to infinite retention (setting both time and size limits to `-1`).  But the data in the system topic does not need for infinite retention.

### Modifications

For system topics, do not retain messages that have already been acknowledged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc-not-needed Your PR changes do not impact docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants