Announcement: Spellbook is ready to migrate to DuneSQL

[jeff-dude]

Dear Wizards,

On April 20th, we announced the migration to DuneSQL. We are investing heavily in DuneSQL to make the fastest, most powerful and convenient crypto data querying experience on the planet.

As part of this process, Spellbook will be migrated to run natively on DuneSQL to make the overall process when building spells more intuitive and straightforward.

We will be freezing contributions to the following spells to facilitate their migration to DuneSQL Spellbook. The freeze of these spells will take place next week from June 22nd to June 28th.

nft.trades
dex.trades
labels

During the freeze, these spells on SparkSQL Spellbook will be moved into maintenance mode (no new pull requests) and after they have been migrated, contributions will need to be made in DuneSQL.

For the remaining spells, we will start migrating them over the next few weeks, freezing individual spells as we migrate them. We will share updates on upcoming spells on both github and discord.

To reiterate, we will take the lead on migrating spells to DuneSQL and encourage you all to help.

Don’t hesitate to reach out to us via [email protected] for any questions or concerns.

The data must flow!

Dune Team

Some additional details:

• During the coming week, until June 20th, we will continue to merge PRs as usual for the above mentioned spells.
  • Please make sure that whatever work you want merged is sent before Tuesday June 20th, after that, PRs will no longer be reviewed to be merged on Spark
• During the week between June 22nd and June 28th we will work to migrate the above spells to be DuneSQL native, we will neither merge nor review PRs.
  • Please use comments below if you have an urgent PR. This means a PR that fixes a critical bug.
• After June 28th:
  • We will continue migrating smaller spells, one sector at a time. This will mean small sectors of Spellbook will be frozen for 1-2 days at a time. We will announce freeze dates progressively under this announcement
  • If editing an existing spell, which is not yet migrated, please do help migrate while making other intended changes
  • If adding a new spell, it must be in DuneSQL
  • We will begin reviewing and merging PRs to DuneSQL spells, any PR that targets a migrated spell must be translated to DuneSQL before merging.

[jeff-dude]

please ask questions in the comments here -- it's the easiest way to reach dune teammates & find answers from other contributors.

[0xRobin]

@jeff-dude
List of PRs we should merge/close today as they are impacted by the freeze.
#3576
#3575
#3574
#3567
#3561
#3553
#3542
#3534
#3530
#3478

[jeff-dude]

agreed -- looking through now to cleanup / merge as much as possible prior to the freeze

[anon-r-7]

Thanks for the clear comms. Any update on when DuneSQL will be supported by the dbt run initial model(s) CI (https://github.com/duneanalytics/spellbook/actions/runs/5338887164/workflow?pr=3583#L45)?

[jeff-dude]

that step will become available at some point during the freeze window. feel free to track merged PRs into main to get an idea of changes and how you can prepare to write your spell(s) in dunesql, but more info will be available as we move out of the freeze.

[jeff-dude]

spellbook freeze update 🪄

spellbook will be extending the freeze a few extra days until wednesday, july 5th. there has been some great progress with the migration, but we will take a little extra time to ensure the three main sectors (dex/nft/labels) and their full lineage have been migrated and tested to ensure data quality parity with spark sql spells. for reference, we're looking at 400+ spells included here.

sneak-peek into progress

for anyone who didn't notice, this particular PR contains some of the backend config to help spellbook run on the new dunesql engine. for the foreseeable future, we will continue to run spark engine as well, which is why the gh action attached to PRs which run CI tests now have two jobs -- one for spark, one for dunesql. how we configure the spells will help dictate which job to run in the CI actions -- more to come on this in contribution guide.

next steps

as we come out of the freeze, we plan to have the below items ready:

• contribution guide: how do i continue to build on spellbook in the new dunesql setup?
  • updates to the readme
  • updates to the dune docs
• a high % of spells translated to dunesql, with a focus on dex/nft/labels
  • the rest will continue to run on spark engine until migrated -- but available to query on dunesql via the dune app!

[jeff-dude]

spellbook freeze update 🪄

it's time. spellbook will move past the freeze stage and be open for PR submissions on the dunesql engine 🙌

the readme has been modified with more details on the new processes around contributions to spellbook. please use that as a guide & open any questions in this discussion, so we can help answer any recurring questions that we may have overlooked and continue to enhance the process and contribution guide.

progress on dune-led migrations

in order to maintain a clean git history & keep PRs easier to read, we will release PRs for the three main sectors (dex, nft, labels) in subsets. our first few will be by sector & protocol (i.e. uniswap for dex lineage) and we'll see how that process looks for us to track and keep visible to the community.

note: we have translated the spells within full lineage of these three sectors in a private forked repo, so please don't accidentally duplicate the work -- we got these ones covered, be sure to ask here before starting work if you're confused on which models still need migrated.

here is an example of an nft marketplace we will migrate over soon. these examples are great reference to follow as you look to help us migrate!

next steps

• update the dune docs to reflect new contribution steps -- the readme is set, now we need to push to dune docs
• many PRs from dune team and resident wizards with more and more spells in the dex/nft/labels lineage
• ...and hopefully some PRs from the community, as we re-adjust to life on spellbook built on top of DuneSQL

again, please do ask us here for any help. let's all look to help each other. the quicker we fully migrate, the quicker we can remove a few of the extra steps for the legacy system.

thanks for the patience during the freeze and happy spell casting 🪄

[jeff-dude]

looking for more examples?

this large PR from @0xRobin accomplishes a few things:

• migrating many more nft marketplaces
• showcasing examples of how to leverage dunesql data types, dunesql functions, etc.

taking advantage of these unique data types (i.e. varbinary or uint256) and the corresponding functions to use on those types (varbinary functions found here) will be beneficial for all, including improved performance 📈