feat(index): full sync operations #11178

adrien2p · 2025-01-27T14:46:23Z

Closes: FRMW-2892, FRMW-2893

What
Wired up the building block that we merged previously in order to manage data synchronization. The flow is as follow

On application start
- Build schema object representation from configuration
- Check configuration changes
  - if new entities configured
    - Data synchronizer initialize orchestrator and start sync
      - for each entity
        
        acquire lock
        
        mark existing data as staled
        
        sync all data by batch
        
        marked them not staled anymore
        
        acknowledge each processed batch and renew lock
        
        update metadata with last synced cursor for entity X
        
        release lock
    - remove all remaining staled data
  - if any entities removed from last configuration
    - remove the index data and relations

vercel · 2025-01-27T14:46:24Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
api-reference	🔄 Building (Inspect)	Visit Preview	💬 Add feedback	Feb 5, 2025 1:14pm
medusa-dashboard	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 5, 2025 1:14pm

5 Skipped Deployments

Name	Status	Preview	Updated (UTC)
api-reference-v2	⬜️ Ignored (Inspect)	Visit Preview	Feb 5, 2025 1:14pm
docs-ui	⬜️ Ignored (Inspect)	Visit Preview	Feb 5, 2025 1:14pm
docs-v2	⬜️ Ignored (Inspect)	Visit Preview	Feb 5, 2025 1:14pm
medusa-docs	⬜️ Ignored (Inspect)	Visit Preview	Feb 5, 2025 1:14pm
resources-docs	⬜️ Ignored (Inspect)	Visit Preview	Feb 5, 2025 1:14pm

changeset-bot · 2025-01-27T14:46:28Z

🦋 Changeset detected

Latest commit: 04fad73

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 65 packages

Name	Type
@medusajs/modules-sdk	Patch
@medusajs/index	Patch
@medusajs/utils	Patch
integration-tests-http	Patch
@medusajs/cli	Patch
@medusajs/medusa	Patch
@medusajs/test-utils	Patch
@medusajs/api-key	Patch
@medusajs/auth	Patch
@medusajs/cache-inmemory	Patch
@medusajs/cache-redis	Patch
@medusajs/cart	Patch
@medusajs/currency	Patch
@medusajs/customer	Patch
@medusajs/event-bus-local	Patch
@medusajs/event-bus-redis	Patch
@medusajs/file	Patch
@medusajs/fulfillment	Patch
@medusajs/inventory	Patch
@medusajs/link-modules	Patch
@medusajs/locking	Patch
@medusajs/notification	Patch
@medusajs/order	Patch
@medusajs/payment	Patch
@medusajs/pricing	Patch
@medusajs/product	Patch
@medusajs/promotion	Patch
@medusajs/region	Patch
@medusajs/sales-channel	Patch
@medusajs/stock-location	Patch
@medusajs/store	Patch
@medusajs/tax	Patch
@medusajs/user	Patch
@medusajs/workflow-engine-inmemory	Patch
@medusajs/workflow-engine-redis	Patch
@medusajs/auth-emailpass	Patch
@medusajs/auth-github	Patch
@medusajs/auth-google	Patch
@medusajs/file-local	Patch
@medusajs/file-s3	Patch
@medusajs/fulfillment-manual	Patch
@medusajs/locking-postgres	Patch
@medusajs/locking-redis	Patch
@medusajs/notification-local	Patch
@medusajs/notification-sendgrid	Patch
@medusajs/payment-stripe	Patch
@medusajs/core-flows	Patch
@medusajs/framework	Patch
@medusajs/js-sdk	Patch
@medusajs/orchestration	Patch
@medusajs/types	Patch
@medusajs/workflows-sdk	Patch
@medusajs/medusa-oas-cli	Patch
@medusajs/oas-github-ci	Patch
@medusajs/telemetry	Patch
@medusajs/admin-bundler	Patch
@medusajs/admin-sdk	Patch
@medusajs/admin-shared	Patch
@medusajs/admin-vite-plugin	Patch
@medusajs/dashboard	Patch
@medusajs/icons	Patch
@medusajs/toolbox	Patch
@medusajs/ui-preset	Patch
create-medusa-app	Patch
medusa-dev-cli	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

…to feat/index-sync-wired

carlos-r-l-rodrigues · 2025-01-31T17:00:37Z

@olivermrbl this is ready to review, and it would be good to test a snapshot of it somewhere. Ideally in an existing Database with data.

olivermrbl · 2025-02-03T08:46:57Z

Will do!

olivermrbl · 2025-02-03T17:11:12Z

/snapshot-this

github-actions · 2025-02-03T17:18:19Z

🚀 A snapshot release has been made for this PR

Test the snapshots by updating your package.json with the newly published versions:

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add [email protected]

yarn add @medusajs/[email protected]

yarn add [email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

Latest commit: 178a7f4

olivermrbl

Nice! Ran a few tests against a local DB. Initial sync, adding, updating, and deleting resources from the index all seem to work as expected. Anything else I should try?

carlos-r-l-rodrigues · 2025-02-05T11:58:47Z

Nice! Ran a few tests against a local DB. Initial sync, adding, updating, and deleting resources from the index all seem to work as expected. Anything else I should try?

It seems that you covered all the features. That's it.
Thanks Oli

I'm merging it as there is a bug fix on the internal medusa service to update/delete entries.

feat(index): Wired sync operations

c54faf7

github-actions bot added type: feature team: medusa labels Jan 27, 2025

vercel bot deployed to Preview – medusa-dashboard January 27, 2025 14:47 View deployment

move config checker to sync utils

2a02711

vercel bot deployed to Preview – medusa-dashboard January 27, 2025 14:56 View deployment

cleanup + wired up configuration and sync entity

0e37fb6

vercel bot deployed to Preview – medusa-dashboard January 27, 2025 16:54 View deployment

adrien2p added 3 commits January 27, 2025 17:54

cleanup + wired up configuration and sync entity

384b743

update deps

edc6544

update deps

fee224c

vercel bot deployed to Preview – medusa-dashboard January 27, 2025 16:57 View deployment

more cleanup

1cf85a8

vercel bot deployed to Preview – medusa-dashboard January 28, 2025 09:16 View deployment

fix configuration tests

ca9fbcc

vercel bot deployed to Preview – medusa-dashboard January 28, 2025 10:02 View deployment

add index sync

a1d6d79

vercel bot deployed to Preview – medusa-dashboard January 28, 2025 13:55 View deployment

update last cursor

41eb48c

vercel bot deployed to Preview – medusa-dashboard January 28, 2025 15:17 View deployment

read last cursor

19660ff

vercel bot deployed to Preview – medusa-dashboard January 28, 2025 15:21 View deployment

adrien2p added 2 commits January 28, 2025 16:22

update orchestrator tests

dfe139e

Merge branch 'feat/index-sync-wired' of github.com:medusajs/medusa in…

bc8167c

…to feat/index-sync-wired

vercel bot deployed to Preview – medusa-dashboard January 28, 2025 15:23 View deployment

Add full sync integration tests + fixes

29d82a7

vercel bot deployed to Preview – medusa-dashboard January 28, 2025 17:12 View deployment

handle staled data as part of the sync

e3a80f5

vercel bot deployed to Preview – medusa-dashboard January 31, 2025 16:54 View deployment

wait sync

ac86bd8

vercel bot deployed to Preview – medusa-dashboard January 31, 2025 17:44 View deployment

changeset

1c1e09c

vercel bot deployed to Preview – medusa-dashboard January 31, 2025 17:56 View deployment

Merge branch 'develop' into feat/index-sync-wired

2217d0a

vercel bot deployed to Preview – medusa-dashboard February 3, 2025 11:33 View deployment

query builder fix

f0a107e

vercel bot deployed to Preview – medusa-dashboard February 3, 2025 14:29 View deployment

joinFilter

424f1a0

vercel bot deployed to Preview – medusa-dashboard February 3, 2025 16:15 View deployment

carlos-r-l-rodrigues added 2 commits February 3, 2025 13:44

sync test

68e7bc4

rm log

784f502

vercel bot deployed to Preview – medusa-dashboard February 3, 2025 16:46 View deployment

default amount Float

763b8cc

vercel bot deployed to Preview – medusa-dashboard February 3, 2025 18:09 View deployment

Merge branch 'develop' into feat/index-sync-wired

8cb59ed

vercel bot deployed to Preview – medusa-dashboard February 4, 2025 07:53 View deployment

olivermrbl approved these changes Feb 4, 2025

View reviewed changes

Merge branch 'develop' into feat/index-sync-wired

04fad73

carlos-r-l-rodrigues added the automerge label Feb 5, 2025

vercel bot deployed to Preview – medusa-dashboard February 5, 2025 13:14 View deployment

kodiakhq bot merged commit a33aebd into develop Feb 5, 2025
23 checks passed

kodiakhq bot deleted the feat/index-sync-wired branch February 5, 2025 16:49

github-actions bot mentioned this pull request Feb 5, 2025

Version Packages #11210

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(index): full sync operations #11178

feat(index): full sync operations #11178

adrien2p commented Jan 27, 2025 •

edited by carlos-r-l-rodrigues

Loading

vercel bot commented Jan 27, 2025 •

edited

Loading

changeset-bot bot commented Jan 27, 2025 •

edited

Loading

carlos-r-l-rodrigues commented Jan 31, 2025

olivermrbl commented Feb 3, 2025

olivermrbl commented Feb 3, 2025

github-actions bot commented Feb 3, 2025

olivermrbl left a comment

carlos-r-l-rodrigues commented Feb 5, 2025 •

edited

Loading

feat(index): full sync operations #11178

feat(index): full sync operations #11178

Conversation

adrien2p commented Jan 27, 2025 • edited by carlos-r-l-rodrigues Loading

vercel bot commented Jan 27, 2025 • edited Loading

changeset-bot bot commented Jan 27, 2025 • edited Loading

🦋 Changeset detected

carlos-r-l-rodrigues commented Jan 31, 2025

olivermrbl commented Feb 3, 2025

olivermrbl commented Feb 3, 2025

github-actions bot commented Feb 3, 2025

🚀 A snapshot release has been made for this PR

olivermrbl left a comment

Choose a reason for hiding this comment

carlos-r-l-rodrigues commented Feb 5, 2025 • edited Loading

adrien2p commented Jan 27, 2025 •

edited by carlos-r-l-rodrigues

Loading

vercel bot commented Jan 27, 2025 •

edited

Loading

changeset-bot bot commented Jan 27, 2025 •

edited

Loading

carlos-r-l-rodrigues commented Feb 5, 2025 •

edited

Loading