-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
## tl;dr Bun has been bothering me for a while, so I've been working on a migration that will get rid of our ORM altogether and just use boring SQL queries for everything. [`sqlc`](https://sqlc.dev/) is a very slick tool to generate code based on plain SQL queries using placeholders for arguments. It's not perfect...I had to do some gymnastics to make a few of the query types work. But the fact that there is no runtime other than the standard SQL driver and some generated code outweighs its limitations IMO. There's no fancy ORM library to worry about mangling your queries, and the learning curve is basically just "how well do you know SQL". ## What's wrong with Bun? - No support for serializable transactions - The SQL driver is not as well maintained as PGX - High learning curve to build complex queries, even if you know SQL well - Relations system is not very powerful and ends up doing N+1 queries a lot of the time. - Configuring the database with struct tags is errorprone, and there aren't great checks to make sure the struct tags actually match the schema. ## Things that suck right now with sqlc - I can't find a good way to have dynamic ORDER BY expressions. So I literally have separate queries for ASC and DESC versions. It's not the end of the world, but it's very frustrating. There's an [issue to fix it](sqlc-dev/sqlc#2061), and some hacky workarounds using CASE statements, but it's not great. - Making the filters play nice with `json_populate_recordset` is a bit of a pain. Switching to the `pgx` driver helped, since I think there was a bug in Bun's pgdriver. ## Migration plan We use Bun in a lot of places and for a lot of things today. - It powers the `authz` database and all the migrations there - It powers the migrations for the `message` database (but not the queries) - It powers the `mls` database and all the queries in the `mlsstore`. The priority right now is to remove it from the `mlsstore`. We will still use it for migrations (`sqlc` can read Bun migrations just fine). This involves replacing the bun `pgdriver` with `pgx` (done in this PR) and replacing all the Bun ORM queries with `sqlc` queries. I have most of the queries written, but I'll split up the actual migration over several PRs. This can be done incrementally, but once the process is complete we can delete the Bun models. We aren't using any of the fancy `sqlc` cloud features and have no plans to. ## What knucklehead brought Bun into our codebase? Ummmm. 😬. That was me.
- Loading branch information
Showing
19 changed files
with
859 additions
and
85 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| #!/bin/bash | ||
| set -e | ||
|
|
||
| docker run --rm -v $(pwd):/src -w /src sqlc/sqlc generate |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| SET statement_timeout = 0; | ||
|
|
||
| --bun:split | ||
|
|
||
| DROP TYPE IF EXISTS inbox_filter; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| SET statement_timeout = 0; | ||
|
|
||
| --bun:split | ||
|
|
||
| CREATE TYPE inbox_filter AS ( | ||
| inbox_id TEXT, | ||
| sequence_id BIGINT | ||
| ); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| -- name: GetAllInboxLogs :many | ||
| SELECT * FROM inbox_log | ||
| WHERE inbox_id = $1 | ||
| ORDER BY sequence_id ASC FOR UPDATE; | ||
|
|
||
| -- name: GetInboxLogFiltered :many | ||
| SELECT a.* FROM inbox_log AS a | ||
| JOIN ( | ||
| SELECT * FROM json_populate_recordset(null::inbox_filter, sqlc.arg(filters)) as b(inbox_id, sequence_id) | ||
| ) as b on b.inbox_id = a.inbox_id AND a.sequence_id > b.sequence_id | ||
| ORDER BY a.sequence_id ASC; | ||
|
|
||
| -- name: InsertInboxLog :one | ||
| INSERT INTO inbox_log (inbox_id, server_timestamp_ns, identity_update_proto) | ||
| VALUES ($1, $2, $3) | ||
| RETURNING sequence_id; | ||
|
|
||
| -- name: CreateInstallation :exec | ||
| INSERT INTO installations (id, wallet_address, created_at, updated_at, credential_identity, key_package, expiration) | ||
| VALUES ($1, $2, $3, $3, $4, $5, $6); | ||
|
|
||
| -- name: UpdateKeyPackage :execrows | ||
| UPDATE installations | ||
| SET key_package = @key_package, updated_at = @updated_at, expiration = @expiration | ||
| WHERE id = @id; | ||
|
|
||
| -- name: FetchKeyPackages :many | ||
| SELECT id, key_package FROM installations | ||
| WHERE ID IN (@ids); | ||
|
|
||
| -- name: GetIdentityUpdates :many | ||
| SELECT * FROM installations | ||
| WHERE wallet_address IN (@wallet_addresses) | ||
| AND (created_at > @start_time OR revoked_at > @start_time) | ||
| ORDER BY created_at ASC; | ||
|
|
||
| -- name: RevokeInstallation :exec | ||
| UPDATE installations | ||
| SET revoked_at = @revoked_at | ||
| WHERE id = @installation_id | ||
| AND revoked_at IS NULL; | ||
|
|
||
| -- name: InsertGroupMessage :one | ||
| INSERT INTO group_messages (group_id, data, group_id_data_hash) | ||
| VALUES ($1, $2, $3) | ||
| RETURNING *; | ||
|
|
||
| -- name: InsertWelcomeMessage :one | ||
| INSERT INTO welcome_messages (installation_key, data, installation_key_data_hash, hpke_public_key) | ||
| VALUES ($1, $2, $3, $4) | ||
| RETURNING *; | ||
|
|
||
| -- name: QueryGroupMessagesAsc :many | ||
| SELECT * FROM group_messages | ||
| WHERE group_id = @group_id | ||
| ORDER BY id ASC | ||
| LIMIT @numrows; | ||
|
|
||
| -- name: QueryGroupMessagesDesc :many | ||
| SELECT * FROM group_messages | ||
| WHERE group_id = @group_id | ||
| ORDER BY id DESC | ||
| LIMIT @numrows; | ||
|
|
||
| -- name: QueryGroupMessagesWithCursorAsc :many | ||
| SELECT * FROM group_messages | ||
| WHERE group_id = $1 | ||
| AND id > $2 | ||
| ORDER BY id ASC | ||
| LIMIT $3; | ||
|
|
||
| -- name: QueryGroupMessagesWithCursorDesc :many | ||
| SELECT * FROM group_messages | ||
| WHERE group_id = $1 | ||
| AND id < $2 | ||
| ORDER BY id DESC | ||
| LIMIT $3; | ||
|
|
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.