diff --git a/docs/developers/developer-reference/storage-mappers.md b/docs/developers/developer-reference/storage-mappers.md index 11970baee..0d708ded4 100644 --- a/docs/developers/developer-reference/storage-mappers.md +++ b/docs/developers/developer-reference/storage-mappers.md @@ -1113,3 +1113,55 @@ fn list_per_user_pair(&self, first_addr: &ManagedAddress, second_addr: &ManagedA ``` Using the correct mapper for your situation can greatly decrease gas costs and complexity, so always remember to carefully evaluate your use-case. + +[comment]: # (mx-context-auto) + +## Accessing a value at an address + +Because of the way the storage mappers are structured, it is very easy to access a "remote" value, meaning a value stored +under a key at a different address than the current one. + +If a developer wanted, for example, to iterate over another contract's `SetMapper`, instead of retrieving the values through +a call to an endpoint and then iterating, one could simply create a new `SetMapper` with a specific `address` parameter +(the address of the contract where the remote storage is located) and the `exact key` used by the storage they wish to access. +Afterwards, it acts like a "local" storage mapper, so the `iter` function can be called easily to accomplish the task. + +:::important important +This feature only works `intra-shard`. + +Also note that a remote value found under a key at an address can only be `read`, not modified. +::: + +Let's take this example: +```rust title=contract_to_be_called/lib.rs +#[storage_mapper("my_remote_mapper")] +fn my_set_mapper(&self) -> SetMapper +``` + +This is a simple `SetMapper` registered under the address of `contract_to_be_called`, and the value stored will be registered under the key provided, `my_remote_mapper`. If we wanted to iterate over the values of `my_set_mapper` intra-shard, we could write: + +```rust title=caller_contract/lib.rs +// the address of the contract containing the storage (contract_to_be_called) +#[storage_mapper("contract_address")] +fn contract_address(&self) -> SingleValueMapper; + +#[endpoint] +fn my_endpoint(&self) -> u32 { + let mut sum = 0u32; + let address = self.contract_address().get(); + + // by creating the mapper with the address of the sc and exact storage key + // we get access to the value stored under that key + let mapper: SetMapper = + SetMapper::new_from_address(address, StorageKey::from("my_remote_mapper")); + for number in mapper.iter() { + sum += number + } + + sum +} +``` + +Calling `my_endpoint` will return the sum of the values stored in `contract_to_be_called`'s SetMapper. + +By specifying the expected type, storage key and address, the value can be read and used inside our logic. \ No newline at end of file diff --git a/docs/sdk-and-tools/google-bigquery.md b/docs/sdk-and-tools/google-bigquery.md new file mode 100644 index 000000000..ebbc1d3de --- /dev/null +++ b/docs/sdk-and-tools/google-bigquery.md @@ -0,0 +1,141 @@ +--- +id: google-bigquery +title: Google BigQuery +--- + +[comment]: # "mx-abstract" + +This page succinctly describes how to use Google BigQuery to analyze data from the MultiversX blockchain. + +[comment]: # "mx-context-auto" + +## Overview + +[**BigQuery**](https://cloud.google.com/bigquery/docs/introduction) is Google's fully managed, serverless data warehouse that enables analysis of extremely large datasets using [SQL queries](https://cloud.google.com/bigquery/docs/introduction-sql) and / or visual tools (such as [Google Looker Studio](https://lookerstudio.google.com)); it also has built-in [machine learning capabilities](https://cloud.google.com/bigquery/docs/bqml-introduction). + +[**MultiversX Blockchain data**](https://console.cloud.google.com/marketplace/product/bigquery-public-data/blockchain-analytics-multiversx-mainnet-eu) is published to Google BigQuery, and available (for free) through the [**Google Cloud Marketplace**](https://console.cloud.google.com/marketplace/product/bigquery-public-data/blockchain-analytics-multiversx-mainnet-eu). The dataset, namely [**`bigquery-public-data.crypto_multiversx_mainnet_eu`**](https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=crypto_multiversx_mainnet_eu&page=dataset), is one of many crypto datasets that are available within [**Google Cloud Public Datasets**](https://cloud.google.com/bigquery/public-data). One can query these datasets for free: up to 1TB / month of free processing, every month. + +The MultiversX BigQuery dataset closely resembles the set of indices of the [**MultiversX Elasticsearch instance**](/sdk-and-tools/elastic-search#elasticsearch-indices). Their schema and data are **approximately equivalent**, the data [being mirrored from the Elasticsearch instance to BigQuery](https://github.com/multiversx/multiversx-etl) at regular intervals (most tables are updated _hourly_, and a few are updated every _4 hours_). + +:::note +As of February 2024, the MultiversX BigQuery dataset **is not updated in real-time** (see above). For real-time data, [use the public APIs](/sdk-and-tools/rest-api). +::: + +:::note +If you experience any issue with the published dataset, please [let us know](https://github.com/multiversx/multiversx-etl/issues). +::: + +## Query from BigQuery Studio + +[**Google BigQuery Studio**](https://cloud.google.com/bigquery/docs/query-overview#bigquery-studio) is a unified workspace for Google Cloud's data analytics suite which incorporates, among others, an SQL editor (optionally [assisted by AI](https://cloud.google.com/bigquery/docs/write-sql-duet-ai)) and Python notebooks. It is a great way to explore the MultiversX dataset, and to run queries. Below, we'll explore a few example queries. + +:::tip +Make sure to explore the dataset, the tables and their schema before running queries. Both the schema and a data preview are available in BigQuery Studio. +::: + +#### How many transactions were processed on MultiversX, in the last couple of days? + +```sql +SELECT + DATE(`timestamp`) `day`, + COUNT(*) `transactions` +FROM `bigquery-public-data.crypto_multiversx_mainnet_eu.transactions` +WHERE DATE(`timestamp`) >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY) +GROUP BY `day` +ORDER BY `day` DESC +``` + +#### Which were the top used Smart Contracts, in the last couple of days? + +```sql +SELECT + DATE(`timestamp`) `day`, + `receiver` `contract`, + COUNT(DISTINCT `sender`) `num_users`, +FROM `bigquery-public-data.crypto_multiversx_mainnet_eu.transactions` +WHERE `isScCall` = true +GROUP BY `day`, `contract` +HAVING `day` >= DATE_SUB(CURRENT_DATE(), INTERVAL 3 DAY) AND `num_users` > 1000 +ORDER BY `day` DESC, `num_users` DESC +``` + +#### What ESDT tokens have the most holders? + +```sql +SELECT + `token`, + `type`, + COUNT(_id) `num_holders` +FROM `bigquery-public-data.crypto_multiversx_mainnet_eu.accountsesdt` +WHERE `type` = 'FungibleESDT' OR `type` = 'MetaESDT' +GROUP BY `token`, `type` +HAVING `num_holders` > 5000 +ORDER BY `num_holders` DESC +``` + +#### What are the transactions with the largest transferred EGLD amounts, in the last couple of days? + +```sql +SELECT + `day`, + `hash`, + `sender`, + `receiver`, + `amount` +FROM ( + SELECT + DATE(`timestamp`) `day`, + `_id` `hash`, + `sender`, + `receiver`, + PARSE_BIGNUMERIC(`value`) `amount`, + ROW_NUMBER() OVER (PARTITION BY DATE(`timestamp`) + ORDER BY PARSE_BIGNUMERIC(`value`) DESC) AS `row_num` + FROM + `bigquery-public-data.crypto_multiversx_mainnet_eu.transactions` + WHERE + `status` = 'success' + AND DATE(`timestamp`) >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY) ) +WHERE `row_num` = 1 +ORDER BY `day` DESC +LIMIT 7; +``` + +#### What is the (global) network hitrate, per day, in the last month? + +```sql +SELECT + DATE(`timestamp`) `day`, + -- 14400 is the number of rounds per day, and 3 + 1 = 4 is the number of shards + ROUND(COUNT(*) / (14400 * 4), 4) `hit_rate` +FROM `bigquery-public-data.crypto_multiversx_mainnet_eu.blocks` +WHERE + DATE(`timestamp`) >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) + AND DATE(`timestamp`) < CURRENT_DATE() +GROUP BY `day` +ORDER BY `day` DESC +``` + +:::note +Even if BigQuery includes a generous free tier, it is important to be mindful of the costs associated with running queries. For more information, see [BigQuery pricing](https://cloud.google.com/bigquery/pricing). + +If you believe that specific optimizations can be applied to the dataset (to improve query performance), please [let us know](https://github.com/multiversx/multiversx-etl/issues). +::: + +## Analyze using Looker Studio + +[**Google Looker Studio**](https://lookerstudio.google.com) is a powerful tool for analyzing data and creating (shareable) reports. Out of the box, it connects to BigQuery (and many other data sources), thus it's a great way to explore the MultiversX dataset. + +Example of report created in Looker Studio (leveraging the MultiversX dataset in BigQuery): + +![img](/sdk-and-tools/looker_studio_1.png) + +:::tip +In the BigQuery Studio, you can save the results of a given query as your own BigQuery tables, then immediately import them in Looker Studio, to create visualizations and reports. +::: + +## Programmatic access + +One can also query datasets programmatically, using the [BigQuery client libraries](https://cloud.google.com/bigquery/docs/reference/libraries). + +See how to [query a public dataset with the BigQuery client libraries](https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries). diff --git a/sidebars.js b/sidebars.js index 781e69fdc..fe8a4eb3d 100644 --- a/sidebars.js +++ b/sidebars.js @@ -248,6 +248,7 @@ const sidebars = { ], }, "sdk-and-tools/notifier", + "sdk-and-tools/google-bigquery", "sdk-and-tools/devcontainers", { type: "category", diff --git a/static/sdk-and-tools/looker_studio_1.png b/static/sdk-and-tools/looker_studio_1.png new file mode 100644 index 000000000..b15c77da6 Binary files /dev/null and b/static/sdk-and-tools/looker_studio_1.png differ