Skip to content

Commit

Permalink
T 13697 nft.trades/mints/burns/transactions/fees rework #1222 (#1242)
Browse files Browse the repository at this point in the history 
Brief comments on the purpose of your changes:

This (very large) PR is the first PR that implements the new NFT spell design on Dune V2.
Changes affect:
- Opensea trades/transactions/mints/burns/fees on Ethereum and Solana
- MagicEden trades/transactions/mints on Solana

New columns were added: blockchain, project, version, block_time, token_id, collection, amount_usd, token_standard, trade_type, number_of_items, trade_category, evt_type, seller, buyer, amount_original, amount_raw, currency_symbol, currency_contract, nft_contract_address, project_contract_address, aggregator_name, aggregator_address, tx_hash, block_number, platform_fee_amount_raw, platform_fee_amount, platform_fee_amount_usd, platform_fee_percentage, royalty_fee_amount_raw, royalty_fee_amount, royalty_fee_amount_usd, royalty_fee_percentage, royalty_fee_receive_address, royalty_fee_currency_symbol, block_number, tx_from, tx_to, unique_trade_id

Made sure tests work w/ the new changes, run time is acceptable, fields match between Solana and Ethereum spells (although some fields are missing in Solana spells). 
I also implemented a temporary fix the bytea2numeric issue on Dune v2 that affects OpenSea ethereum spells: Token IDs larger than 64 bits now have the following mention: 'Token ID is larger than 64 bits and can not be displayed' until we find a better solution

*For Dune Engine V2*
I've checked that:

* [x] I tested the query on dune.com after compiling the model with dbt compile (compiled queries are written to the target directory)
* [x] I used "refs" to reference other models in this repo and "sources" to reference raw or decoded tables 
* [x] if adding a new model, I added a test
* [x] the filename is unique and ends with .sql
* [x] each sql file is a select statement and has only one view, table or function defined  
* [x] column names are `lowercase_snake_cased`

When you are ready for a review, tag duneanalytics/data-experience. We will re-open your forked pull request as an internal pull request. Then your spells will run in dbt and the logs will be avaiable in Github Actions DBT Slim CI. This job will only run the models and tests changed by your PR compared to the production project.
  • Loading branch information
@soispoke
soispoke authored Jul 7, 2022
1 parent b711488 commit 8097112
Show file tree
Hide file tree
Showing 48 changed files with 2,212 additions and 503 deletions.
6 changes: 3 additions & 3 deletions spellbook/dbt_project.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]


target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
- "target"
Expand All @@ -31,7 +30,9 @@ models:
+materialized: view
nft:
+schema: nft
+materialized: view
ethereum:
+schema: nft_ethereum
+materialized: view
uniswap:
+schema: uniswap
+materialized: view
Expand Down Expand Up @@ -82,7 +83,6 @@ models:
+schema: transfers_ethereum
+materialized: view


seeds:
spellbook:
balances:
Expand Down
123 changes: 97 additions & 26 deletions spellbook/macros/alter_table_properties.sql
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,58 +82,122 @@ ALTER VIEW balances_ethereum.erc1155_latest SET TBLPROPERTIES('dune.public'='tru
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set magiceden_trades %}
ALTER TABLE magiceden.trades SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["solana"]',
{% set opensea_transactions %}
ALTER TABLE opensea.transactions SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum","solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='project',
'dune.data_explorer.abstraction.name'='magiceden',
'dune.data_explorer.abstraction.name'='opensea',
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set nft_trades %}
ALTER TABLE nft.trades SET TBLPROPERTIES('dune.public'='true',
{% set opensea_trades %}
ALTER VIEW opensea.trades SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum","solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='sector',
'dune.data_explorer.abstraction.name'='nft',
'dune.data_explorer.abstraction.type'='project',
'dune.data_explorer.abstraction.name'='opensea',
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set opensea_active_traders_day %}
ALTER VIEW opensea.active_traders_day SET TBLPROPERTIES('dune.public'='true',
{% set opensea_mints %}
ALTER VIEW opensea.mints SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum","solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='project',
'dune.data_explorer.abstraction.name'='opensea',
'dune.data_explorer.contributors'='["rchen8"]');
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set opensea_trades %}
ALTER TABLE opensea.trades SET TBLPROPERTIES('dune.public'='true',
{% set opensea_burns %}
ALTER VIEW opensea.burns SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum","solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='project',
'dune.data_explorer.abstraction.name'='opensea',
'dune.data_explorer.contributors'='["rchen8","soispoke"]');
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set opensea_txns_day %}
ALTER VIEW opensea.txns_day SET TBLPROPERTIES('dune.public'='true',
{% set opensea_fees %}
ALTER VIEW opensea.fees SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum","solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='project',
'dune.data_explorer.abstraction.name'='opensea',
'dune.data_explorer.contributors'='["rchen8"]');
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set opensea_volume_day %}
ALTER VIEW opensea.volume_day SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum","solana"]',

{% set magiceden_transactions %}
ALTER TABLE magiceden.transactions SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='project',
'dune.data_explorer.abstraction.name'='opensea',
'dune.data_explorer.contributors'='["rchen8"]');
'dune.data_explorer.abstraction.name'='magiceden',
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set magiceden_trades %}
ALTER VIEW magiceden.trades SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='project',
'dune.data_explorer.abstraction.name'='magiceden',
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set magiceden_mints %}
ALTER VIEW magiceden.mints SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='project',
'dune.data_explorer.abstraction.name'='magiceden',
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set nft_transactions %}
ALTER TABLE nft.transactions SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum","solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='sector',
'dune.data_explorer.abstraction.name'='nft',
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set nft_trades %}
ALTER TABLE nft.trades SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum","solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='sector',
'dune.data_explorer.abstraction.name'='nft',
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set nft_mints %}
ALTER TABLE nft.mints SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum","solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='sector',
'dune.data_explorer.abstraction.name'='nft',
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set nft_burns %}
ALTER TABLE nft.burns SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum","solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='sector',
'dune.data_explorer.abstraction.name'='nft',
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set nft_fees %}
ALTER TABLE nft.fees SET TBLPROPERTIES('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum","solana"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='sector',
'dune.data_explorer.abstraction.name'='nft',
'dune.data_explorer.contributors'='["soispoke"]');
{% endset %}

{% set tokens_ethereum_erc20 %}
Expand Down Expand Up @@ -226,12 +290,19 @@ ALTER VIEW ens.view_renewals SET TBLPROPERTIES('dune.public'='true',
{% do run_query(balances_ethereum_erc1155_day) %}
{% do run_query(balances_ethereum_erc1155_hour) %}
{% do run_query(balances_ethereum_erc1155_latest) %}
{% do run_query(opensea_transactions) %}
{% do run_query(opensea_trades) %}
{% do run_query(opensea_mints) %}
{% do run_query(opensea_burns) %}
{% do run_query(opensea_fees) %}
{% do run_query(magiceden_transactions) %}
{% do run_query(magiceden_trades) %}
{% do run_query(magiceden_mints) %}
{% do run_query(nft_transactions) %}
{% do run_query(nft_trades) %}
{% do run_query(opensea_active_traders_day) %}
{% do run_query(opensea_trades) %}
{% do run_query(opensea_txns_day) %}
{% do run_query(opensea_volume_day) %}
{% do run_query(nft_mints) %}
{% do run_query(nft_burns) %}
{% do run_query(nft_fees) %}
{% do run_query(tokens_ethereum_erc20) %}
{% do run_query(transfers_ethereum_erc20) %}
{% do run_query(tokens_ethereum_nft) %}
Expand Down
38 changes: 29 additions & 9 deletions spellbook/macros/optimize_tables.sql
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,33 +24,53 @@ OPTIMIZE transfers_ethereum.erc1155_agg_hour;
OPTIMIZE transfers_ethereum.erc1155_agg_day;
{% endset %}

{% set opensea_trades %}
OPTIMIZE opensea.trades;
{% set opensea_transactions %}
OPTIMIZE opensea.transactions;
{% endset %}

{% set magiceden_trades %}
OPTIMIZE magiceden.trades;
{% set magiceden_transactions %}
OPTIMIZE magiceden.transactions;
{% endset %}

{% set uniswap_trades %}
OPTIMIZE uniswap.trades;
{% set nft_transactions %}
OPTIMIZE nft.transactions;
{% endset %}

{% set nft_trades %}
OPTIMIZE nft.trades;
{% endset %}

{% set nft_mints %}
OPTIMIZE nft.mints;
{% endset %}

{% set nft_burns %}
OPTIMIZE nft.burns;
{% endset %}

{% set nft_fees %}
OPTIMIZE nft.fees;
{% endset %}

{% set uniswap_trades %}
OPTIMIZE uniswap.trades;
{% endset %}

{% do run_query(transfers_ethereum_erc20_agg_hour) %}
{% do run_query(transfers_ethereum_erc20_agg_day) %}
{% do run_query(transfers_ethereum_erc721_agg_hour) %}
{% do run_query(transfers_ethereum_erc721_agg_day) %}
{% do run_query(transfers_ethereum_erc1155_agg_hour) %}
{% do run_query(transfers_ethereum_erc1155_agg_day) %}
{% do run_query(opensea_trades) %}
{% do run_query(magiceden_trades) %}
{% do run_query(uniswap_trades) %}
{% do run_query(opensea_transactions) %}
{% do run_query(magiceden_transactions) %}
{% do run_query(nft_transactions) %}
{% do run_query(nft_trades) %}
{% do run_query(nft_mints) %}
{% do run_query(nft_burns) %}
{% do run_query(nft_fees) %}
{% do run_query(uniswap_trades) %}


{% do log("Tables Optimized", info=True) %}
{%- else -%}
Expand Down
32 changes: 32 additions & 0 deletions spellbook/models/magiceden/magiceden_mints.sql
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{{ config(
alias ='mints'
)
}}

SELECT blockchain,
project,
version,
block_time,
NULL::string as token_id,
NULL::string as collection,
amount_usd,
token_standard,
NULL::string as trade_type,
NULL::string as number_of_items,
NULL::string as trade_category,
evt_type,
NULL::string as seller,
NULL::string as buyer,
amount_original,
amount_raw,
currency_symbol,
currency_contract,
NULL::string as nft_contract_address,
project_contract_address,
NULL::string as aggregator_name,
NULL::string as aggregator_address,
tx_hash,
NULL::string as tx_from,
NULL::string as tx_to,
unique_trade_id
FROM {{ ref('magiceden_solana_mints') }}
Loading

0 comments on commit 8097112

Please sign in to comment.