Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(clustering): introduce incremental sync for clustering #13157

Merged
merged 163 commits into from
Oct 16, 2024
Merged

Conversation

dndx
Copy link
Member

@dndx dndx commented Jun 4, 2024

KAG-4865
KAG-2986
KAG-2987
KAG-3502
KAG-3258
KAG-5283

NOTE:

  • cluster_rpc now is on by default
  • cluster_incremental_sync now is off by default, we will continue to fix tests after this PR then turn it on.

Flaky (perhaps need rerun):

  • spec/02-integration/09-hybrid_mode/03-pki_spec.lua
  • spec/02-integration/02-cmd/14-vault_spec.lua
  • spec/03-plugins/09-key-auth/04-hybrid_mode_spec.lua

@dndx dndx force-pushed the feat/incremental branch 3 times, most recently from 88e177a to e64c085 Compare June 4, 2024 17:13
@dndx dndx force-pushed the feat/incremental branch 2 times, most recently from 856ca04 to 50528fd Compare August 7, 2024 03:27
@dndx dndx changed the title [WIP] incremental sync PoC Incremental sync PoC Sep 11, 2024
@chronolaw chronolaw force-pushed the feat/incremental branch 2 times, most recently from a558f38 to b11e659 Compare September 24, 2024 08:44
kong/db/strategies/off/init.lua Outdated Show resolved Hide resolved
@chronolaw chronolaw changed the title Incremental sync PoC feat(clustring): introduce incremental sync for clustering Sep 26, 2024
@chronolaw chronolaw changed the title feat(clustring): introduce incremental sync for clustering feat(clusteriing): introduce incremental sync for clustering Sep 26, 2024
@chronolaw chronolaw marked this pull request as ready for review September 30, 2024 08:17
chobits and others added 24 commits October 16, 2024 21:28
* fix(tests): enable inc sync for 01-sync_spec.lua

* fix(incremental): dont delete old entity from LMDB if we clear it already

* dont modify crud type even though we dont delete old entity from lmdb

* localize ns_delta.wipe

* localize ns_delta.wipe - part 2

* fix that cp will not init clustering

* refactor old_entity logic

* cp should always support full sync protocol

* Revert "cp should always support full sync protocol"

This reverts commit 6803958.

* Revert "fix that cp will not init clustering"

This reverts commit b84f261.

* Revert "fix(tests): enable inc sync for 01-sync_spec.lua"

This reverts commit 54d0f54.

---------

Co-authored-by: chronolaw <[email protected]>
@dndx dndx merged commit 398e180 into master Oct 16, 2024
30 checks passed
@dndx dndx deleted the feat/incremental branch October 16, 2024 14:15
@team-gateway-bot
Copy link
Collaborator

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-13157-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-13157-to-master-to-upstream
git checkout -b cherry-pick-13157-to-master-to-upstream
ancref=$(git merge-base 9ae36382db4b9485f8889b8cd80d66a0885c86e7 7c027596914938766d7c93d74406f894dc19aa96)
git cherry-pick -x $ancref..7c027596914938766d7c93d74406f894dc19aa96

@github-actions github-actions bot added the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Oct 16, 2024
@kikito kikito removed the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants