Large list performance

[tiagocorreiaitcenter]

Hello, im trying to implement a FlatList with a large amount of data, between 5-20k of items, and I can't stop thinking if there's no better way of handling it other than waiting for the entire data to load since its a long process? some logic like, .skip and .limit methods, since FlatLists let us know we are reaching the end of the list for example, not sure if you have a better idea of how to implement such scenarios but would like to know how you work with it, thanks

[tiagocorreiaitcenter]

Also sorting plays a big role on this list, its a contacts list, im not sure how we sould implement the sorting since it should based on the database data

[radex]

See: #217

🍉 has no built-in support for this case. The easiest option for you is to add your own column and query based on a range of values in that column to implement chunking/limiting of the list. The better option is to start making steps towards implementing proper DB-level sorting and limiting in 🍉 itself. Look for "limit" and "sorting" in issues to find discussions about this

[tiagocorreiaitcenter]

@radex Thanks will pay attention to it, I was thinking about adding a number to know the order and use stuff like greater than or something like that, but I Have a small issue, I can change a single contact name, and the stored order would be useless sadly, will try to see if I can come around with an idea

[vishgoel007]

@tiagocorreiaitcenter I was facing the same performance issue and i wasted so many days in improving the performance of flat list. Finally i end up using recyclerlistview instead of flat list and it solved my issue.

Hope it helps.

[tiagocorreiaitcenter]

@vishgoel007 The main issue it's actually fetch the entire data from database its a large chunk of data

[Kenneth-KT]

Take a look at this PR #622
It implements SQL ORDER BY, LIMIT and OFFSET by Q.sortBy(sortColumn, sortOrder), Q.take(count) and Q.skip(count) query operators.

[awgv]

@radex Hello. Considering that I know nothing about internal workings of WatermelonDB or LokiJS, and that I was thinking about the problem a bit too much, I thought it would be best to ask for the second opinion here; there’re many ways to store tree structures in relational databases (like files organized in folders, so unlike the OP, not flat) that provide different—but often comparable—performance of CRUD operations under various circumstances—do you think these different approaches would cause the same effects if applied to WatermelonDB with the LokiJS adapter?

I don’t have an immediate problem, was just wondering if I should be investing time in a more appropriate database structure instead of using something simple like the adjacency list for my tree, which might or might not bite me in the future.

[radex]

@awgv No idea, I didn't have a use case for tree structures. Worth experimenting with -- and please do share your findings!

[radex]

Closing this as there is now a possibility for sorting and limiting queries to support this.