Skip to content

Commit

Permalink
metrics table dumper, late aerich import
Browse files Browse the repository at this point in the history
  • Loading branch information
droserasprout committed Oct 13, 2024
1 parent 6848a15 commit a78a9d1
Show file tree
Hide file tree
Showing 8 changed files with 108 additions and 170 deletions.
6 changes: 5 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ FRONTEND_PATH=../interface
help: ## Show this help (default)
@grep -Fh "##" $(MAKEFILE_LIST) | grep -Fv grep -F | sed -e 's/\\$$//' | sed -e 's/##//'

install:
pdm sync --clean

##
##-- CI
##
Expand Down Expand Up @@ -52,6 +55,7 @@ docs: ## Build docs
python scripts/docs.py check-links --source docs
python scripts/docs.py dump-references
python scripts/docs.py dump-demos
python scripts/docs.py dump-metrics
python scripts/docs.py dump-jsonschema
python scripts/docs.py merge-changelog
python scripts/docs.py markdownlint
Expand All @@ -65,7 +69,7 @@ docs_watch: ## Build docs and watch for changes

docs_publish: ## Tag and push `docs-next` ref
git tag -d docs-next && git tag docs-next && git push --force origin docs-next

##

fixme: todo
Expand Down
42 changes: 3 additions & 39 deletions docs/5.advanced/3.monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,45 +78,9 @@ prometheus: {}

The following metrics are exposed under `dipdup` namespace:

| metric name | description | type |
| ---------------------------------------------- | --------------------------------------------------- | --------- |
| `dipdup_indexes_total` | Number of indexes in operation by status | counter |
| `dipdup_index_total_sync_duration_seconds` | Duration of the last index synchronization | histogram |
| `dipdup_index_total_realtime_duration_seconds` | Duration of the last index realtime synchronization | histogram |
| `dipdup_index_levels_to_sync_total` | Number of levels to reach synced state | histogram |
| `dipdup_index_levels_to_realtime_total` | Number of levels to reach realtime state | histogram |
| `dipdup_datasource_head_updated_timestamp` | Timestamp of the last head update | histogram |
| `dipdup_datasource_rollbacks_total` | Number of rollbacks | counter |
| `dipdup_http_errors_total` | Number of HTTP errors | counter |
| `dipdup_http_errors_in_row` | Number of consecutive failed requests | histogram |
| `dipdup_levels_indexed_total` | Total number of levels indexed | gauge |
| `dipdup_levels_nonempty_total` | Total number of nonempty levels indexed | counter |
| `dipdup_levels_total` | Total number of levels | gauge |
| `dipdup_objects_indexed_total` | Total number of objects indexed | counter |
| `dipdup_index_handlers_matched_total` | Index total hits | gauge |
| `dipdup_index_time_in_matcher_seconds` | Time spent in matcher | histogram |
| `dipdup_index_time_in_callbacks_seconds` | Time spent in callbacks | histogram |
| `dipdup_datasource_time_in_requests_seconds` | Time spent in datasource requests | histogram |
| `dipdup_datasource_requests_total` | Total number of datasource requests | counter |
| `dipdup_started_at_timestamp` | Timestamp of the DipDup start | gauge |
| `dipdup_synchronized_at_timestamp` | Timestamp of the last synchronization | gauge |
| `dipdup_realtime_at_timestamp` | Timestamp of the last realtime update | gauge |
| `dipdup_metrics_updated_at_timestamp` | Timestamp of the last metrics update | gauge |
| `dipdup_levels_speed` | Levels per second | gauge |
| `dipdup_levels_speed_average` | Average levels per second | gauge |
| `dipdup_levels_nonempty_speed` | Nonempty levels per second | gauge |
| `dipdup_objects_speed` | Objects per second | gauge |
| `dipdup_time_passed_seconds` | Time passed since the start | gauge |
| `dipdup_time_left_seconds` | Time left estimated until the end | gauge |
| `dipdup_progress` | Progress in percents | gauge |

Also, DipDup exposes the following metrics for compatibility with [Subsquid Cloud](https://app.subsquid.io/):

| metric name | description | type |
| ------------------------------------------ | --------------------------------------------------------- | --------- |
| `sqd_processor_last_block` | Level of the last processed block from Subsquid Network | gauge |
| `sqd_processor_chain_height` | Current chain height as reported by Subsquid Network | gauge |
| `sqd_processor_archive_http_errors_in_row` | Number of consecutive failed requests to Subsquid Network | histogram |
{{ #include 5.advanced/_metrics_table.md }}

Also, DipDup exposes metrics for compatibility with [Subsquid Cloud](https://app.subsquid.io/) under `sqd` namespace.

## Logging

Expand Down
35 changes: 35 additions & 0 deletions docs/5.advanced/_metrics_table.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
<!-- markdownlint-disable first-line-h1 -->
| name | description | type |
|-|-|-|
| dipdup_datasource_head_updated_timestamp | Timestamp of the last head update | Gauge |
| dipdup_datasource_requests | Total number of datasource requests | Counter |
| dipdup_datasource_rollbacks | Number of rollbacks | Counter |
| dipdup_datasource_time_in_requests_seconds | Time spent in datasource requests | Histogram |
| dipdup_http_errors | Number of http errors | Counter |
| dipdup_http_errors_in_row | Number of consecutive failed requests | Gauge |
| dipdup_index_handlers_matched | Index total hits | Counter |
| dipdup_index_levels_to_realtime_total | Number of levels to reach realtime state | Histogram |
| dipdup_index_levels_to_sync_total | Number of levels to reach synced state | Histogram |
| dipdup_index_time_in_callbacks_seconds | Time spent in callbacks | Histogram |
| dipdup_index_time_in_matcher_seconds | Time spent in matcher | Histogram |
| dipdup_index_total_realtime_duration_seconds | Duration of the last index realtime syncronization | Histogram |
| dipdup_index_total_sync_duration_seconds | Duration of the last index syncronization | Histogram |
| dipdup_indexes_total | Number of indexes in operation by status | Gauge |
| dipdup_levels_indexed_total | Total number of levels indexed | Gauge |
| dipdup_levels_nonempty | Total number of nonempty levels indexed | Counter |
| dipdup_levels_nonempty_speed | Nonempty levels per second | Gauge |
| dipdup_levels_speed | Levels per second | Gauge |
| dipdup_levels_speed_average | Average levels per second | Gauge |
| dipdup_levels_total | Total number of levels | Gauge |
| dipdup_metrics_updated_at_timestamp | Timestamp of the last metrics update | Gauge |
| dipdup_objects_indexed | Total number of objects indexed | Counter |
| dipdup_objects_speed | Objects per second | Gauge |
| dipdup_progress | Progress in percents | Gauge |
| dipdup_realtime_at_timestamp | Timestamp of the last realtime update | Gauge |
| dipdup_started_at_timestamp | Timestamp of the DipDup start | Gauge |
| dipdup_synchronized_at_timestamp | Timestamp of the last synchronization | Gauge |
| dipdup_time_left_seconds | Time left estimated until the end | Gauge |
| dipdup_time_passed_seconds | Time passed since the start | Gauge |
| sqd_processor_archive_http_errors_in_row | Number of consecutive failed requests to Subsquid Network | Gauge |
| sqd_processor_chain_height | Current chain height as reported by Subsquid Network | Gauge |
| sqd_processor_last_block | Level of the last processed block from Subsquid Network | Gauge |
93 changes: 0 additions & 93 deletions docs/7.references/1.cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -365,44 +365,6 @@ Discord: <a class="reference external" href="https://discord.gg/aG
</section>


### downgrade

<p>Downgrade to specified version.</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>downgrade<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
</pre></div>
</div>
<p class="rubric">Options</p>
<dl class="std option">
<dt class="sig sig-object std" id="cmdoption-dipdup-schema-downgrade-v">
<span id="cmdoption-dipdup-schema-downgrade-version"></span><span class="sig-name descname"><span class="pre">-v</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--version</span></span><span class="sig-prename descclassname"> <span class="pre">&lt;version&gt;</span></span></dt>
<dd><p>Specified version, default to last.</p>
<dl class="field-list simple">
<dt class="field-odd" style="color: var(--txt-primary);">Default<span class="colon">:</span></dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">-1</span></code></p>
</dd>
</dl>
</dd></dl>

<dl class="std option">
<dt class="sig sig-object std" id="cmdoption-dipdup-schema-downgrade-d">
<span id="cmdoption-dipdup-schema-downgrade-delete"></span><span class="sig-name descname"><span class="pre">-d</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--delete</span></span><span class="sig-prename descclassname"></span></dt>
<dd><p>Delete version files at the same time.</p>
<dl class="field-list simple">
<dt class="field-odd" style="color: var(--txt-primary);">Default<span class="colon">:</span></dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">False</span></code></p>
</dd>
</dl>
</dd></dl>

<dl class="std option">
<dt class="sig sig-object std" id="cmdoption-dipdup-schema-downgrade-yes">
<span class="sig-name descname"><span class="pre">--yes</span></span><span class="sig-prename descclassname"></span></dt>
<dd><p>Confirm the action without prompting.</p>
</dd></dl>

</section>


### export

<p>Print SQL schema including scripts from <cite>sql/on_reindex</cite>.</p>
Expand All @@ -413,24 +375,6 @@ Discord: <a class="reference external" href="https://discord.gg/aG
</section>


### heads

<p>Show current available heads in migrate location.</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>heads<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
</pre></div>
</div>
</section>


### history

<p>List all migrate items.</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
</pre></div>
</div>
</section>


### init

<p>Prepare database schema for running DipDup.</p>
Expand All @@ -441,43 +385,6 @@ Discord: <a class="reference external" href="https://discord.gg/aG
</section>


### migrate

<p>Generate migrate changes file.</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>migrate<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
</pre></div>
</div>
<p class="rubric">Options</p>
<dl class="std option">
<dt class="sig sig-object std" id="cmdoption-dipdup-schema-migrate-name">
<span class="sig-name descname"><span class="pre">--name</span></span><span class="sig-prename descclassname"> <span class="pre">&lt;name&gt;</span></span></dt>
<dd><p>Migrate name.</p>
<dl class="field-list simple">
<dt class="field-odd" style="color: var(--txt-primary);">Default<span class="colon">:</span></dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">'update'</span></code></p>
</dd>
</dl>
</dd></dl>

</section>


### upgrade

<p>Upgrade to specified version.</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>upgrade<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
</pre></div>
</div>
<p class="rubric">Options</p>
<dl class="std option">
<dt class="sig sig-object std" id="cmdoption-dipdup-schema-upgrade-i">
<span id="cmdoption-dipdup-schema-upgrade-in-transaction"></span><span class="sig-name descname"><span class="pre">-i</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--in-transaction</span></span><span class="sig-prename descclassname"> <span class="pre">&lt;in_transaction&gt;</span></span></dt>
<dd><p>Make migrations in transaction or not. Can be helpful for large migrations or creating concurrent indexes.</p>
</dd></dl>

</section>


### wipe

<p>Drop all database tables, functions and views.</p>
Expand Down
36 changes: 18 additions & 18 deletions docs/8.examples/_demos_table.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
<!-- markdownlint-disable first-line-h1 -->
| name | network | description | source |
|-|-|-|-|
| demo_blank | | Empty config for a fresh start | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_blank) |
| demo_evm_events | EVM | ERC-20 token transfers (from event logs) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_evm_events) |
| demo_evm_transactions | EVM | ERC-20 token transfers (from transactions) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_evm_transactions) |
| demo_evm_uniswap | EVM | Uniswap V3 pools, positions, etc. (advanced, uses TimescaleDB) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_evm_uniswap) |
| demo_starknet_events | Starknet | ERC-20 token transfers (from events) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_starknet_events) |
| demo_tezos_auction | Tezos | NFT marketplace (TzColors) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_auction) |
| demo_tezos_dao | Tezos | DAO registry (Homebase DAO) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_dao) |
| demo_tezos_dex | Tezos | DEX balances and liquidity (Quipuswap) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_dex) |
| demo_tezos_domains | Tezos | Domain name service (Tezos Domains) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_domains) |
| demo_tezos_etherlink | Tezos | Etherlink smart rollup transactions | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_etherlink) |
| demo_tezos_events | Tezos | Processing contract events | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_events) |
| demo_tezos_factories | Tezos | Example of spawning indexes in runtime | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_factories) |
| demo_tezos_head | Tezos | Processing head block metadata (realtime only) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_head) |
| demo_tezos_nft_marketplace | Tezos | NFT marketplace (hic at nunc) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_nft_marketplace) |
| demo_tezos_raw | Tezos | Process raw operations without filtering and typed payloads | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_raw) |
| demo_tezos_token | Tezos | FA1.2 token contract operations | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_token) |
| demo_tezos_token_balances | Tezos | FA1.2 token balances | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_token_balances) |
| demo_tezos_token_transfers | Tezos | FA1.2 token transfers | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_token_transfers) |
| demo_blank | | Empty config for a fresh start | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_blank) |
| demo_evm_events | EVM | ERC-20 token transfers (from event logs) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_evm_events) |
| demo_evm_transactions | EVM | ERC-20 token transfers (from transactions) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_evm_transactions) |
| demo_evm_uniswap | EVM | Uniswap V3 pools, positions, etc. (advanced, uses TimescaleDB) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_evm_uniswap) |
| demo_starknet_events | Starknet | ERC-20 token transfers (from events) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_starknet_events) |
| demo_tezos_auction | Tezos | NFT marketplace (TzColors) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_auction) |
| demo_tezos_dao | Tezos | DAO registry (Homebase DAO) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_dao) |
| demo_tezos_dex | Tezos | DEX balances and liquidity (Quipuswap) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_dex) |
| demo_tezos_domains | Tezos | Domain name service (Tezos Domains) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_domains) |
| demo_tezos_etherlink | Tezos | Etherlink smart rollup transactions | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_etherlink) |
| demo_tezos_events | Tezos | Processing contract events | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_events) |
| demo_tezos_factories | Tezos | Example of spawning indexes in runtime | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_factories) |
| demo_tezos_head | Tezos | Processing head block metadata (realtime only) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_head) |
| demo_tezos_nft_marketplace | Tezos | NFT marketplace (hic at nunc) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_nft_marketplace) |
| demo_tezos_raw | Tezos | Process raw operations without filtering and typed payloads | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_raw) |
| demo_tezos_token | Tezos | FA1.2 token contract operations | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_token) |
| demo_tezos_token_balances | Tezos | FA1.2 token balances | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_token_balances) |
| demo_tezos_token_transfers | Tezos | FA1.2 token transfers | [link](https://github.com/dipdup-io/dipdup/tree/8.1.0/src/demo_tezos_token_transfers) |
Loading

0 comments on commit a78a9d1

Please sign in to comment.