RSQLite concurrency issues — solution included - R-hub blog

[utterances-bot]

RSQLite concurrency issues — solution included - R-hub blog

https://blog.r-hub.io/2021/03/13/rsqlite-parallel/

[barnzilla]

Thank you for sharing this post, Gábor. I ran into a concurrency issue on a R project and this has been very helpful.

One caveat: I don't find this solution to scale very well. I am distributing ~2,500 write transactions across 8 nodes (using the parallel package) and if I don't set busy_timeout to 100 * 1000, the database is locked error will be thrown. But, more importantly, it takes much longer to complete these transactions by a factor of ~5x when compared to an approach where each node writes to its own .sqlite file (followed by an aggregation of these files into a single file afterwards).

So, while I'm very thankful for the code you have shared, I think, at the moment, an approach where transactions are written to separate files and then aggregated downstream makes more sense, even if less elegant.