Refactor search sync to try to reduce memory usage and hash ids

[alfredgrip]

Title speaks for itself.

Idea is to divide data in batches and send it one at a time, this way the garbage collector should be able to drop objects since they go out of scope.

Current batch size is 1000, which I think should be fine.

Also hashes ids which should be a (temporary) fix to #661

[danieladugyan]

There's a lot of code in sync.ts and searchTypes.ts , to be honest it's a bit hard for me to grasp it all😅

[alfredgrip]

There's a lot of code in sync.ts and searchTypes.ts , to be honest it's a bit hard for me to grasp it all😅

Yeah searchTypes.ts is a lot...
Essentially, for every index in Meili there is a type for:

1. What attributes is stored in Meili for that index
2. Which attributes can a user perform a search after
3. Which attributes are returned

Of course, all attributes that can be searched on, or are returned, must be stored in Meilisearch. It not as simple as doing a union of 2. and 3. to get 1. however, since some attributes (likestartDatetime for events) are used purely for sorting and ranking purposes internally by Meili.

Then there are objects like const memberMeilisearchConstants: MemberConstantsMeilisearch = {... which wraps things related to an index in a single object. Here we can specify custom ranking and sorting rules for Meili, such as giving newer members a higher ranking, and tweak which typo tolerance is allowed.

I know that the file is full of types, but it is to prevent us developers from accidentally trying to e.g. do custom ranking rules on an attribute that isn't even stored in Meilisearch.

As for sync.ts, it's basically just dump the data and attributes defined in searchTypes.ts to Meilisearch, but do so in batches. When all the data is dumped, tweak the rankings based on the values and types also defined in searchTypes.ts.

[danieladugyan]

There's a lot of code in sync.ts and searchTypes.ts , to be honest it's a bit hard for me to grasp it all😅

Yeah searchTypes.ts is a lot... Essentially, for every index in Meili there is a type for:

1. What attributes is stored in Meili for that index
2. Which attributes can a user perform a search after
3. Which attributes are returned

Of course, all attributes that can be searched on, or are returned, must be stored in Meilisearch. It not as simple as doing a union of 2. and 3. to get 1. however, since some attributes (likestartDatetime for events) are used purely for sorting and ranking purposes internally by Meili.

Then there are objects like const memberMeilisearchConstants: MemberConstantsMeilisearch = {... which wraps things related to an index in a single object. Here we can specify custom ranking and sorting rules for Meili, such as giving newer members a higher ranking, and tweak which typo tolerance is allowed.

I know that the file is full of types, but it is to prevent us developers from accidentally trying to e.g. do custom ranking rules on an attribute that isn't even stored in Meilisearch.

As for sync.ts, it's basically just dump the data and attributes defined in searchTypes.ts to Meilisearch, but do so in batches. When all the data is dumped, tweak the rankings based on the values and types also defined in searchTypes.ts.

Thanks! I added that as a comment to the top of the file since it helped a lot.