Data replication to scale reads

[Brayden]

Because Durable Objects are limited to 1,000 RPS, having a built-in feature to scale reads via data replication could serve as a strong offering for scaling usage.

Things to consider would be where the logical components live, and which instance(s) get to decide on data handoff as they ingest incoming data. How do we declare a main source? Is it a primary write, distributing each read to the replicas perhaps by way of web sockets? A lot of questions, some with clear answers others needing to mull over in more depth.

Placing this discussion here as a placeholder to come back to as I chew through the thoughts.

[gedw99]

Interesting . Here is how I think about it ..

I run SQLite in origin ( 3 data Center’s in multi master mode ) and keep the version in Cloudflare as the read only.

mutations go to origin, and the WAL pushes them to all the read only SQLite in CF.

you can take it further and WAL off the CF SQLite to keep the browser service worker SQLite up to data as a read only copy of what’s in CF?

Workers also run in both the cf workers and Web workers , using the SQLite locally .

Migrations are an event that is sent up , just like data is sent up from Orion.

[Brayden]

When you say you run SQLite in origin, where is this hosted and with what service (e.g. is it Litestream, Durable Objects, something else altogether)? Curious how you're thinking about listening to WAL changes so they can be pushed to the CF SQLite read replicas.

StarbaseDB instances are based off of Durable Objects in Cloudflare by default so currently escaping that ecosystem and having custom WAL events doesn't seem feasible unless there's something I'm unaware of that could unlock this.

Would love to hear more of your thoughts if you don't mind elaborating a bit on how the implementation would look in your mind with some more detail @gedw99!