refactor(clustering/rpc): tunings and fixes for rubustness #13771
+177
−90
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
core/db
cherry-pick kong-ee
chronolaw
force-pushed
the
refactor/tuning_inc_sync
branch
from
eada306
to
522dbf3
Compare
chobits
reviewed
Oct 22, 2024
chronolaw
force-pushed
the
refactor/tuning_inc_sync
branch
from
427835a
to
a566ec6
Compare
chronolaw
changed the title
chronolaw
force-pushed
the
refactor/tuning_inc_sync
branch
from
150fa2d
to
c625a60
Compare
chobits
force-pushed
the
refactor/tuning_inc_sync
branch
from
9281822
to
b7963e4
Compare
chronolaw
force-pushed
the
refactor/tuning_inc_sync
branch
from
b7963e4
to
a7ba490
Compare
team-gateway-bot
added
the
author/community
chobits
reviewed
Oct 30, 2024
|
|
||
| -- If the item belongs to a specific workspace, | ||
| -- use it directly without using the default one. | ||
| local ws_id = item.ws_id or workspace_id(schema, options) |
There was a problem hiding this comment.
TODO: need to add a test to check the ws_id value if the schema is non-workspaceable. https://konghq.atlassian.net/browse/KAG-5706
This comment is not a block.
chobits
reviewed
Oct 30, 2024
kong/db/declarative/import.lua
Outdated
Comment on lines
318
to
322
| for _, wid in ipairs {ws_id, GLOBAL_WORKSPACE_TAG} do | ||
| local key = unique_field_key(entity_name, wid, "cache_key", cache_key) | ||
|
|
||
| -- store item_key or nil into lmdb | ||
| t:set(key, idx_value) |
There was a problem hiding this comment.
not a blocker, just a comment for other reviewers.
TODO: we only want to save the entity with key containing specific ws_id, not a GLOBAL workspace id like*, which does not consume too much memory space for lmdb. https://konghq.atlassian.net/browse/KAG-5704
chobits
reviewed
Oct 30, 2024
kong/db/declarative/import.lua
Outdated
Comment on lines
390
to
395
| -- * <entity_name>|*|*|<pk_string> => serialized item | ||
| -- * <entity_name>|<ws_id>|*|<pk_string> => actual item key | ||
| -- | ||
| -- * <entity_name>|*|<unique_field_name>|sha256(field_value) => actual item key | ||
| -- * <entity_name>|<ws_id>|<unique_field_name>|sha256(field_value) => actual item key | ||
| -- | ||
| -- * <entity_name>|*|<foreign_field_name>|<foreign_key>|<pk_string> => actual item key | ||
| -- * <entity_name>|<ws_id>|<foreign_field_name>|<foreign_key>|<pk_string> => actual item key |
There was a problem hiding this comment.
not a blocker, -> https://github.com/Kong/kong/pull/13771/files#r1821977274
chobits
approved these changes
Oct 30, 2024
chronolaw
removed
the
author/community
team-gateway-bot
added
the
author/community
chronolaw
commented
Oct 30, 2024
| @@ -359,7 +359,8 @@ local sync_emitter = { | |||
|
|
|||
| emit_entity = function(self, entity_name, entity_data) | |||
| self.out_n = self.out_n + 1 | |||
| self.out[self.out_n] = { type = entity_name , row = entity_data, version = self.sync_version, } | |||
| self.out[self.out_n] = { type = entity_name , row = entity_data, version = self.sync_version, | |||
| ws_id = kong.default_workspace, } | |||
There was a problem hiding this comment.
Here ws_id is a fallback, if row(entity) has the field ws_id, do_sync() will use it but not delta.ws_id.
| if not res then | ||
| return nil, err | ||
| end | ||
| end | ||
|
|
||
| local res, err = insert_entity_for_txn(t, delta_type, delta_row, nil) | ||
| local res, err = insert_entity_for_txn(t, delta_type, delta_row, opts) |
There was a problem hiding this comment.
row.ws_id ? delta.ws_id?
There was a problem hiding this comment.
In hooks.lua, delta item will always has the field ws_id, so opts.workspace is a fallback.
local deltas = {
{
type = name,
pk = pk,
ws_id = ws_id,
entity = is_delete and ngx_null or entity,
},
}
11-declarative_lmdb_spec.lua fix spec/01-unit/01-db/10-declarative_spec.lua
This reverts commit 522dbf3.
* fix(clustering/sync): set ws_id in do_sync() * comments
* fix(clustering/sync): delta.version may be null * isempty * default_ws_changed
* rename db id to pk * change id to pk in Lua * change pk to json * schema:extract_pk_values
Co-authored-by: Datong Sun <[email protected]>
* change db filed name * rename in rpc.sync * rename in export
…#13815) * fix(incremental sync): fix cache_key handling for select_by_cache_key * fix some comments * fix: remove unnecessary comment
* refactor(dbless): clean logic of selec_by_field * check schema_field
chronolaw
force-pushed
the
refactor/tuning_inc_sync
branch
from
f35c3d3
to
63c17e2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
KAG-5617
Related tickets:
KAG-5602
Checklist
changelog/unreleased/kongorskip-changeloglabel added on PR if changelog is unnecessary. README.mdIssue reference
Fix #[issue number]