diff --git a/CHANGELOG.md b/CHANGELOG.md index c3a8f871f..8d1e5be1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,11 +4,12 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog], and this project adheres to [Semantic Versioning]. -## [Unreleased] +## [7.1.0] - 2023-10-27 ### Added -- cli: Added `--unsafe` and `--compose` flags to `config env` command. +- cli: Added `--unsafe`, `--compose`, `--internal` flags to `config env` command. +- cli: Added missing short equivalents for options in some commands. - cli: Relative paths to be initialized now can be passed to the `init` command as arguments. - tezos.tzkt.token_balances: Added new index. - evm.node: Requesting logs for multiple blocks at once for EVM nodes to improve performance @@ -1224,7 +1225,9 @@ This release contains no changes except for the version number. [semantic versioning]: https://semver.org/spec/v2.0.0.html -[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.0.1...HEAD +[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.1.0...HEAD +[7.1.0]: https://github.com/dipdup-io/dipdup/compare/7.0.2...7.1.0 +[7.0.2]: https://github.com/dipdup-io/dipdup/compare/7.0.1...7.0.2 [7.0.1]: https://github.com/dipdup-io/dipdup/compare/7.0.0...7.0.1 [7.0.0]: https://github.com/dipdup-io/dipdup/compare/7.0.0rc5...7.0.0 [7.0.0rc5]: https://github.com/dipdup-io/dipdup/compare/7.0.0rc4...7.0.0rc5 diff --git a/docs/13.troubleshooting.md b/docs/13.troubleshooting.md index 7545aff63..95a667fa7 100644 --- a/docs/13.troubleshooting.md +++ b/docs/13.troubleshooting.md @@ -78,7 +78,7 @@ Exceptions that occurred during callback execution are reraised as `CallbackErro ```shell [Terminal] # PDM -pdm fmt +pdm format pdm lint # Manually diff --git a/docs/5.advanced/2.feature-flags.md b/docs/5.advanced/2.feature-flags.md index 820b074b3..587c107fa 100644 --- a/docs/5.advanced/2.feature-flags.md +++ b/docs/5.advanced/2.feature-flags.md @@ -35,6 +35,7 @@ Please note that they are not currently a part of the public API and can be chan | env variable | module path | description | | -------------------- | ------------------------ | -------------------------------------------------------- | | `DIPDUP_CI` | `dipdup.env.CI` | Running in GitHub Actions | +| `DIPDUP_DEBUG` | `dipdup.env.DEBUG` | Enable debug logging and additional checks | | `DIPDUP_DOCKER` | `dipdup.env.DOCKER` | Running in Docker | | `DIPDUP_NEXT` | `dipdup.env.NEXT` | Enable experimental features that require schema changes | | `DIPDUP_REPLAY_PATH` | `dipdup.env.REPLAY_PATH` | Path to datasource replay files; used in tests | diff --git a/docs/5.advanced/3.sql.md b/docs/5.advanced/3.sql.md index 1a9b1170c..8707afa7f 100644 --- a/docs/5.advanced/3.sql.md +++ b/docs/5.advanced/3.sql.md @@ -53,9 +53,13 @@ await ctx.execute_sql('my_hook/my_script.sql') ## Managing schema -When using PostgreSQL as database engine you can use `dipdup_approve` and `dipdup_wipe` functions to manage schema state from SQL console if needed: +When using PostgreSQL or Timescale as database engine you can use `dipdup_approve` and `dipdup_wipe` functions to manage schema state from SQL console if needed: ```sql SELECT dipdup_approve('public'); + +-- WARNING: This action is irreversible! All indexed data will be lost! SELECT dipdup_wipe('public'); ``` + +Please note that `dipdup_wipe` function doesn't support preserving immune tables. diff --git a/docs/9.release-notes/1.v7.1.md b/docs/9.release-notes/1.v7.1.md new file mode 100644 index 000000000..8f6214d65 --- /dev/null +++ b/docs/9.release-notes/1.v7.1.md @@ -0,0 +1,157 @@ +--- +title: 7.1.0 +description: DipDup 7.1 release notes +--- + +# Release Notes: 7.1 + +Welcome, developers! + +After maybe-too-long awaited 7.0 stable release we decided to establish shorter and more predictable release cycle. So, expect new DipDup versions to arrive more frequently, but maybe be less groundbreaking. Today we're releasing DipDup 7.1 with new index kind, CLI options, and lots of small improvements. Let's dive in! + +## Indexing Tezos token balances + +Starting with 7.1 a new index kind `tezos.tzkt.token_balances` is available. It allows indexing token balances of Tezos contracts compatible with [FA1.2](https://gitlab.com/tzip/tzip/-/blob/master/proposals/tzip-7/README.md) or [FA2](https://gitlab.com/tzip/tzip/-/blob/master/proposals/tzip-12/tzip-12.md) standards. This index is similar to existing `tezos.tzkt.token_transfers`, but provides only the current balances skipping historical data. In cases when you don't need the history of changes, this index is much more efficient. + +```yaml [dipdup.yaml] +spec_version: 2.0 +package: demo_token_balances + +contracts: + tzbtc_mainnet: + kind: tezos + address: KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn + typename: tzbtc + +datasources: + tzkt: + kind: tezos.tzkt + url: https://api.tzkt.io + +indexes: + tzbtc_holders_mainnet: + kind: tezos.tzkt.token_balances + datasource: tzkt + handlers: + - callback: on_balance_update + contract: tzbtc_mainnet +``` + +## SQL helpers to manage database schema + +Two new SQL functions were added to the schema: `dipdup_approve` and `dipdup_wipe`. These helpers are equivalent to `schema approve` and `schema wipe` CLI commands, but can be used directly in SQL queries. + +```sql +SELECT dipdup_approve('public'); + +-- WARNING: This action is irreversible! All indexed data will be lost! +SELECT dipdup_wipe('public'); +``` + +Please note that `dipdup_wipe` function doesn't support preserving immune tables. + +DipDup has lots of tools to work with SQL like parameterized scripts, queries, and more. Visit the [Advanced SQL](../5.advanced/3.sql.md) page to learn more. + +## More `config env` command options + +Managing environment variables could be a pain, still it's an important part of a modern 12-factor application. Try to keep your project config completely environment-agnostic to make deploy easier and ensure secrets are safe. This release brings a few new options to the `config env` command to make it easier to work with environment variables. + +Compose stack definitions in `deploy` package directory include environment variables from `.env` files (generated with `config env` command without arguments). New `--compose, -c` flag also allows exporting config variables in Compose format. Copy necessary strings from the output to the `environment` section of the service definition and you're good. + +```shell [Terminal] +$ dipdup config env --compose +services: + dipdup: + environment: + - API_KEY=${API_KEY} + - API_URL=${API_URL:-https://api.example.com} +``` + +`--internal, -i` flag allows including internal variables (with `DIPDUP_` prefix) in the command output. All of them are optional and rarely needed, but handy for debugging. See the [Feature Flags](../5.advanced/2.feature-flags.md) page for the full list of internal variables. + +```shell [Terminal] +$ dipdup config env --internal +DIPDUP_CI= +DIPDUP_DEBUG= +DIPDUP_DOCKER= +DIPDUP_NEXT= +DIPDUP_REPLAY_PATH= +DIPDUP_TEST= +``` + +You could notice a new variable here, `DIPDUP_DEBUG`. Setting it to true will increase logging verbosity (equal to `logging: debug` in config) and enable some additional checks. Make sure that this flag is set if you're gathering logs for a bug report. + +Finally, `--unsafe` flag allows exporting environment variables from the current shell session. Missing short option is intentional to prevent accidental usage. Missing example output here is also intentional, because it's, you know, unsafe to share your secrets. Don't do that. + +## Improved base template + +One of DipDup killer features is that we scaffold everything: package metadata, types and callback stubs, Compose definitions etc. We constantly improve the base template to make it more useful. You can run `dipdup new` once, answer a few questions get a fully working project. But you also use `init` command later to refresh files from the base template in your project. + +```shell [Terminal] +# First make sure that repo is clean +git status --porcelane + +# Refresh only `pyproject.toml` file +dipdup init --force pyproject.toml + +# Refresh everything +dipdup init --force --base + +# Check what has changed +git diff +``` + +This feature is a huge time saver when you maintain multiple projects and want them to be consistent. Or to modify the `replay.yaml` file and apply changes to the whole project instead of generating a new one or finding and replacing all occurrences manually. + +In this release we have updated linter rules to be more strict. Scripts to run CI routines were also updated. Run `pdm run -l` to see the full list of available scripts. + +```shell +$ pdm run -l +╭────────┬───────────┬──────────────────────────────────╮ +│ Name │ Type │ Description │ +├────────┼───────────┼──────────────────────────────────┤ +│ all │ composite │ Run all checks │ +│ format │ composite │ Format code with isort and black │ +│ image │ cmd │ Build Docker image │ +│ lint │ composite │ Check code with ruff and mypy │ +╰────────┴───────────┴──────────────────────────────────╯ +``` + +Inspect `pyproject.toml` to see how these scripts are defined or add your own to automate common tasks. + +Here's a basic example of a GitHub Actions workflow to run all checks on every push: + +```yaml [.github/workflows/test.yml] +name: Test DipDup project +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true +on: + push: + branches: + - '**' + +jobs: + test: + name: Test + runs-on: ubuntu-latest + steps: + - name: Install PDM + run: pipx install pdm + + - name: Set up Python + uses: actions/setup-python@main + with: + python-version: '3.11' + cache: 'pip' + + - name: Install project dependencies + run: pdm install + + - name: Run all checks + run: pdm all +``` + +That's all for today. Thanks for reading! + +{{ #include 9.release-notes/_footer.md }} diff --git a/docs/9.release-notes/2.v6.2.md b/docs/9.release-notes/2.v6.2.md deleted file mode 100644 index a373c69c0..000000000 --- a/docs/9.release-notes/2.v6.2.md +++ /dev/null @@ -1,90 +0,0 @@ ---- -title: 6.2.0 -description: DipDup 6.2 release notes ---- - -## What's New - -### New interactive installer - -Starting from this release, DipDup comes with an interactive installer to help you install necessary dependencies. - -Run the command below in the terminal: - -```shell -curl -Lsf https://dipdup.io/install.py | python3 -``` - -Follow the instructions to complete the installation. - -Now you have `dipdup` command available systemwide! Run it without arguments to see available commands. - -You can use `dipdup install/update/uninstall` commands to manage the local installation. - -### Project scaffolding - -`dipdup new` command is now available to create a new project from a template. Run it and follow the questions; a new project will be created in the current directory. You can also use a replay file instead; see `dipdup new --help` for details. - -
Scaffolder screenshot
- -### Kathmandu contract events - -Kathmandu Tezos protocol upgrade has introduced [contract events](https://tezos.gitlab.io/alpha/event.html), a new way to interact with smart contracts. This index allows indexing events using strictly typed payloads. From the developer's perspective, it's similar to the `big_map` index with a few differences. - -An example below is artificial since no known contracts in mainnet are currently using events. - -```yaml [dipdup.yaml] -{{ #include ../src/demo_events/dipdup.yaml:23:32 }} -``` - -Unlike big maps, contracts may introduce new event tags and payloads at any time, so the index must be updated accordingly. - -```python -{{ #include ../src/demo_events/handlers/on_move_event.py:7: }} -``` - -Each contract can have a fallback handler called for all unknown events so you can process untyped data. - -```python -{{ #include ../src/demo_events/handlers/on_other_event.py:6: }} -``` - -## Changes since 5.1.3 - -### Added - -- cli: `new` command to create a new project interactively. -- cli: `install/update/uninstall` commands to manage local DipDup installation. -- index: New index kind `event` to process contract events. -- install: New interactive installer based on pipx (`install.py` or `dipdup-install`). - -### Fixed - -- cli: Fixed commands that don't require a valid config yet crash with `ConfigurationError`. -- codegen: Fail on demand when `datamodel-codegen` is not available. -- codegen: Fixed Jinja2 template caching. -- config: Allow `sentry.dsn` field to be empty. -- config: Fixed greedy environment variable regex. -- hooks: Raise a `FeatureAvailabilityHook` instead of a warning when trying to execute hooks on SQLite. - -### Improved - -- cli: Detect `src/` layout when guessing package path. -- codegen: Improved cross-platform compatibility. -- config: `sentry.user_id` option to set user ID for Sentry (affects release adoption data). -- sentry: Detect environment when not set in config (docker/gha/tests/local) -- sentry: Expose more tags under the `dipdup` namespace. - -### Performance - -- cli: Up to 5x faster startup for some commands. - -### Security - -- sentry: Prevent Sentry from leaking hostname if `server_name` is not set. -- sentry: Notify about using Sentry when DSN is set or crash reporting is enabled. - -### Other - -- ci: A significantly faster execution of GitHub Actions. -- docs: Updated "Contributing Guide" page. diff --git a/docs/9.release-notes/1.v7.0.md b/docs/9.release-notes/2.v7.0.md similarity index 99% rename from docs/9.release-notes/1.v7.0.md rename to docs/9.release-notes/2.v7.0.md index 740794a1a..5375d14b1 100644 --- a/docs/9.release-notes/1.v7.0.md +++ b/docs/9.release-notes/2.v7.0.md @@ -147,3 +147,5 @@ If you have any questions or issues with migration, please open a ticket on GitH | | `event` | `tezos.tzkt.events` | | | `head` | `tezos.tzkt.head` | | | `token_transfer` | `tezos.tzkt.token_transfers` | + +{{ #include 9.release-notes/_footer.md }} diff --git a/docs/9.release-notes/3.v6.0.md b/docs/9.release-notes/3.v6.0.md deleted file mode 100644 index 35b6fdb38..000000000 --- a/docs/9.release-notes/3.v6.0.md +++ /dev/null @@ -1,133 +0,0 @@ ---- -title: 6.0.0 -description: DipDup 6.0 release notes ---- - -# 6.0.0 - -## ⚠ Breaking Changes - -- Project models in `models.py` must be subclassed from `dipdup.models.Model` instead of `tortoise.Model`. -- The deprecated `on_rollback` event hook has been removed in favor of `on_index_rollback`. -- HTTP datasources no longer use caching. `cache show` and `cache clear` commands have been removed. `http.cache` config flag has been removed. -- `--logging-config` option has been removed. Use the `logging` config section or set up logging manually. -- Feature flag options of `run` command have been removed. Use the `advanced` config section instead. - -## Migration from 5.x - -- Replace `tortoise.Model` import with `dipdup.models.Model` in your `models.py` module. -- Remove the `on_rollback` event hook if it still persists in your project. Ensure that `on_index_rollback.py` contains `ctx.rollback` call, or remove it and call `dipdup init`. -- If you have used `buffer_size` config option, remove it to use database-level rollbacks. -- Run `schema approve` command with every schema you want to use with DipDup 6.0. - -## What's New - -### Seamless database-level rollbacks - -The era of handling chain reorgs manually is finally over! Now when DipDup receives a reorg message from TzKT it just rewinds a database to the previous state reverting changes in backtracked blocks level by level. To make this possible, DipDup catches all database modifications and saves diffs in a separate table, `dipdup_model_update` (you don't need to access it directly). - -```python -# INSERT saved with no data, just drop this row on reorg. -trader = Trader(name='Alice', balance=100, active=True) -await trader.save() - -# UPDATE saved with data old values to set them on reorg. -# Diff: {'balance': 100} -trader.balance = 200 -await trader.save() - -# DELETE saved with full copy of data. On reorg this row will be recreated with the same PK. -# Diff: {'name': 'Alice', 'balance': 200, 'active': True} -await trader.delete() -``` - -Bulk class methods like `bulk_insert` and `bulk_update` are supported too. However, for raw queries, DipDup uses prefetching (additional SELECT) to save original values. So, `ReallyHugeTable.filter().delete()` will create efficiently a full copy of the table in `dipdup_model_update`. Most likely you will never need to perform such queries in handlers, but keep that detail in mind. - -Since the Ithacanet protocol, only two last blocks may be backtracked. We do not need to store older diffs, they are removed automatically. If you need to keep more levels or disable this feature, adjust `rollback_depth` config option. - -```yaml [dipdup.yaml] -advanced: - rollback_depth: 2 # 0 to disable -``` - -`on_index_rollback` event hook now looks like this: - -```python -from dipdup.context import HookContext -from dipdup.index import Index - - -async def on_index_rollback( - ctx: HookContext, - index: Index, # type: ignore[type-arg] - from_level: int, - to_level: int, -) -> None: - await ctx.execute_sql('on_index_rollback') - await ctx.rollback( - index=index.name, - from_level=from_level, - to_level=to_level, - ) -``` - -TzKT `buffer_size` option remains available, but it's not required to handle chain reorgs anymore. - -### Crash dumps and automatic reporting - -Now when DipDup catches unhandled exceptions, a crash dump will be saved to the temporary directory. - -``` -dipdup.exceptions.CallbackError: An error occured during callback execution -________________________________________________________________________________ - -`demo_token.hooks.on_restart` callback execution failed: - - Exception: - -Eliminate the reason of failure and restart DipDup. -________________________________________________________________________________ - -Crashdump saved to `/tmp/dipdup/crashdumps/veb7kz07.json` -``` - -This JSON file is the same data Sentry collects on crashes if integration is enabled. It includes a stack trace, local variables of each frame, and other information useful when investigating a crash. Attach this file when sending bug reports to GitHub Issues. - -When preparing a crash dump Sentry can detect sensitive information like database passwords in the crash dump and remove it from the report. So it's generally safe to share the crash dump with the developers. Now you can also send these crash reports automatically to the Baking Bad team. - -Your privacy matters; **crash reporting is disabled by default**. Simulate a crash with a random exception and inspect a crash dump before enabling this option to ensure that report doesn't contain secrets. Then add the following lines to your config: - -```yaml [dipdup.yaml] -advanced: - crash_reporting: True -``` - -## Changes since 5.2.5 - -### Added - -- cli: Added `config export --full` flag to resolve templates before printing config. -- config: Added `advanced.crash_reporting` flag to enable reporting crashes to Baking Bad. -- config: Added `advanced.rollback_depth` field, a number of levels to keep in a database for rollback. -- context: Added `rollback` method to perform database rollback. -- database: Added an internal `ModelUpdate` model to store the latest database changes. -- dipdup: Save Sentry crashdump in `/tmp/dipdup/crashdumps/XXXXXXX.json` on a crash. - -### Fixed - -- config: Do not perform env variable substitution in commented-out lines. -- prometheus: Fixed updating `dipdup_index_handlers_matched_total` metric. - -### Changed - -- codegen: `on_index_rollback` hook calls `ctx.rollback` by default. -- database: Project models must be subclassed from `dipdup.models.Model` - -### Removed - -- cli: `--logging-config` option is removed. -- cli: All `run` command flags are removed. Use the `advanced` section of the config. -- cli: `cache show` and `cache clear` commands are removed. -- config: `http.cache` flag is removed. -- hooks: Removed deprecated `on_rollback` hook. -- index: Do not try to avoid single-level rollbacks by comparing operation hashes. diff --git a/docs/9.release-notes/3.v6.5.md b/docs/9.release-notes/3.v6.5.md new file mode 100644 index 000000000..6e71008ac --- /dev/null +++ b/docs/9.release-notes/3.v6.5.md @@ -0,0 +1,10 @@ +--- +title: 6.5.0 +description: DipDup 6.5 release notes +--- + +DipDup 6.5 is the current old-stable release powering dozens of APIs in production. It will be **supported until March 2024**. We want to give teams enough time for migration or to wait a bit while the current branch is being ironed out. We will continue to release bugfixes and security updates until that date. You can find the old documentation [here](https://docs.dipdup.io/), but some pages may be outdated. If you've found such a page or other issue, please drop us a message. + +- [Documentation](https://docs.dipdup.io) +- [Changelog](https://docs.dipdup.io/CHANGELOG) +- [7.0 Migration Guide](docs/release-notes/v7.0#migration-guide) diff --git a/docs/9.release-notes/4.history.md b/docs/9.release-notes/4.history.md index 3e3b4b6b1..c68cc7de0 100644 --- a/docs/9.release-notes/4.history.md +++ b/docs/9.release-notes/4.history.md @@ -1,8 +1,231 @@ --- title: History -description: DipDup release notes +description: Release notes for legacy DipDup releases --- +# Legacy Releases + +This page contains release notes for legacy DipDup releases. + +## 6.2.0 + +## What's New + +### New interactive installer + +Starting from this release, DipDup comes with an interactive installer to help you install necessary dependencies. + +Run the command below in the terminal: + +```shell +curl -Lsf https://dipdup.io/install.py | python3 +``` + +Follow the instructions to complete the installation. + +Now you have `dipdup` command available systemwide! Run it without arguments to see available commands. + +You can use `dipdup install/update/uninstall` commands to manage the local installation. + +### Project scaffolding + +`dipdup new` command is now available to create a new project from a template. Run it and follow the questions; a new project will be created in the current directory. You can also use a replay file instead; see `dipdup new --help` for details. + +
Scaffolder screenshot
+ +### Kathmandu contract events + +Kathmandu Tezos protocol upgrade has introduced [contract events](https://tezos.gitlab.io/alpha/event.html), a new way to interact with smart contracts. This index allows indexing events using strictly typed payloads. From the developer's perspective, it's similar to the `big_map` index with a few differences. + +An example below is artificial since no known contracts in mainnet are currently using events. + +```yaml [dipdup.yaml] +{{ #include ../src/demo_events/dipdup.yaml:23:32 }} +``` + +Unlike big maps, contracts may introduce new event tags and payloads at any time, so the index must be updated accordingly. + +```python +{{ #include ../src/demo_events/handlers/on_move_event.py:7: }} +``` + +Each contract can have a fallback handler called for all unknown events so you can process untyped data. + +```python +{{ #include ../src/demo_events/handlers/on_other_event.py:6: }} +``` + +## Changes since 5.1.3 + +### Added + +- cli: `new` command to create a new project interactively. +- cli: `install/update/uninstall` commands to manage local DipDup installation. +- index: New index kind `event` to process contract events. +- install: New interactive installer based on pipx (`install.py` or `dipdup-install`). + +### Fixed + +- cli: Fixed commands that don't require a valid config yet crash with `ConfigurationError`. +- codegen: Fail on demand when `datamodel-codegen` is not available. +- codegen: Fixed Jinja2 template caching. +- config: Allow `sentry.dsn` field to be empty. +- config: Fixed greedy environment variable regex. +- hooks: Raise a `FeatureAvailabilityHook` instead of a warning when trying to execute hooks on SQLite. + +### Improved + +- cli: Detect `src/` layout when guessing package path. +- codegen: Improved cross-platform compatibility. +- config: `sentry.user_id` option to set user ID for Sentry (affects release adoption data). +- sentry: Detect environment when not set in config (docker/gha/tests/local) +- sentry: Expose more tags under the `dipdup` namespace. + +### Performance + +- cli: Up to 5x faster startup for some commands. + +### Security + +- sentry: Prevent Sentry from leaking hostname if `server_name` is not set. +- sentry: Notify about using Sentry when DSN is set or crash reporting is enabled. + +### Other + +- ci: A significantly faster execution of GitHub Actions. +- docs: Updated "Contributing Guide" page. + + + +# 6.0.0 + +## ⚠ Breaking Changes + +- Project models in `models.py` must be subclassed from `dipdup.models.Model` instead of `tortoise.Model`. +- The deprecated `on_rollback` event hook has been removed in favor of `on_index_rollback`. +- HTTP datasources no longer use caching. `cache show` and `cache clear` commands have been removed. `http.cache` config flag has been removed. +- `--logging-config` option has been removed. Use the `logging` config section or set up logging manually. +- Feature flag options of `run` command have been removed. Use the `advanced` config section instead. + +## Migration from 5.x + +- Replace `tortoise.Model` import with `dipdup.models.Model` in your `models.py` module. +- Remove the `on_rollback` event hook if it still persists in your project. Ensure that `on_index_rollback.py` contains `ctx.rollback` call, or remove it and call `dipdup init`. +- If you have used `buffer_size` config option, remove it to use database-level rollbacks. +- Run `schema approve` command with every schema you want to use with DipDup 6.0. + +## What's New + +### Seamless database-level rollbacks + +The era of handling chain reorgs manually is finally over! Now when DipDup receives a reorg message from TzKT it just rewinds a database to the previous state reverting changes in backtracked blocks level by level. To make this possible, DipDup catches all database modifications and saves diffs in a separate table, `dipdup_model_update` (you don't need to access it directly). + +```python +# INSERT saved with no data, just drop this row on reorg. +trader = Trader(name='Alice', balance=100, active=True) +await trader.save() + +# UPDATE saved with data old values to set them on reorg. +# Diff: {'balance': 100} +trader.balance = 200 +await trader.save() + +# DELETE saved with full copy of data. On reorg this row will be recreated with the same PK. +# Diff: {'name': 'Alice', 'balance': 200, 'active': True} +await trader.delete() +``` + +Bulk class methods like `bulk_insert` and `bulk_update` are supported too. However, for raw queries, DipDup uses prefetching (additional SELECT) to save original values. So, `ReallyHugeTable.filter().delete()` will create efficiently a full copy of the table in `dipdup_model_update`. Most likely you will never need to perform such queries in handlers, but keep that detail in mind. + +Since the Ithacanet protocol, only two last blocks may be backtracked. We do not need to store older diffs, they are removed automatically. If you need to keep more levels or disable this feature, adjust `rollback_depth` config option. + +```yaml [dipdup.yaml] +advanced: + rollback_depth: 2 # 0 to disable +``` + +`on_index_rollback` event hook now looks like this: + +```python +from dipdup.context import HookContext +from dipdup.index import Index + + +async def on_index_rollback( + ctx: HookContext, + index: Index, # type: ignore[type-arg] + from_level: int, + to_level: int, +) -> None: + await ctx.execute_sql('on_index_rollback') + await ctx.rollback( + index=index.name, + from_level=from_level, + to_level=to_level, + ) +``` + +TzKT `buffer_size` option remains available, but it's not required to handle chain reorgs anymore. + +### Crash dumps and automatic reporting + +Now when DipDup catches unhandled exceptions, a crash dump will be saved to the temporary directory. + +``` +dipdup.exceptions.CallbackError: An error occured during callback execution +________________________________________________________________________________ + +`demo_token.hooks.on_restart` callback execution failed: + + Exception: + +Eliminate the reason of failure and restart DipDup. +________________________________________________________________________________ + +Crashdump saved to `/tmp/dipdup/crashdumps/veb7kz07.json` +``` + +This JSON file is the same data Sentry collects on crashes if integration is enabled. It includes a stack trace, local variables of each frame, and other information useful when investigating a crash. Attach this file when sending bug reports to GitHub Issues. + +When preparing a crash dump Sentry can detect sensitive information like database passwords in the crash dump and remove it from the report. So it's generally safe to share the crash dump with the developers. Now you can also send these crash reports automatically to the Baking Bad team. + +Your privacy matters; **crash reporting is disabled by default**. Simulate a crash with a random exception and inspect a crash dump before enabling this option to ensure that report doesn't contain secrets. Then add the following lines to your config: + +```yaml [dipdup.yaml] +advanced: + crash_reporting: True +``` + +## Changes since 5.2.5 + +### Added + +- cli: Added `config export --full` flag to resolve templates before printing config. +- config: Added `advanced.crash_reporting` flag to enable reporting crashes to Baking Bad. +- config: Added `advanced.rollback_depth` field, a number of levels to keep in a database for rollback. +- context: Added `rollback` method to perform database rollback. +- database: Added an internal `ModelUpdate` model to store the latest database changes. +- dipdup: Save Sentry crashdump in `/tmp/dipdup/crashdumps/XXXXXXX.json` on a crash. + +### Fixed + +- config: Do not perform env variable substitution in commented-out lines. +- prometheus: Fixed updating `dipdup_index_handlers_matched_total` metric. + +### Changed + +- codegen: `on_index_rollback` hook calls `ctx.rollback` by default. +- database: Project models must be subclassed from `dipdup.models.Model` + +### Removed + +- cli: `--logging-config` option is removed. +- cli: All `run` command flags are removed. Use the `advanced` section of the config. +- cli: `cache show` and `cache clear` commands are removed. +- config: `http.cache` flag is removed. +- hooks: Removed deprecated `on_rollback` hook. +- index: Do not try to avoid single-level rollbacks by comparing operation hashes. + ## 5.1.0 ### Migration from 5.0 (optional) diff --git a/docs/9.release-notes/_footer.md b/docs/9.release-notes/_footer.md new file mode 100644 index 000000000..202b67b36 --- /dev/null +++ b/docs/9.release-notes/_footer.md @@ -0,0 +1,9 @@ + +

+___ + +__See you soon! 👋__ + +DipDup is a free open-source software published under the MIT License. We are the small independent team and we rely on your support! Join our socials to discuss this release, ask any questions, or become a sponsor to support DipDup development. + +[Twitter](https://twitter.com/dipdup_io) | [Discord](https://discord.gg/aG8XKuwsQd) | [GitHub](https://github.com/dipdup-io/dipdup) | [❤ Become a Sponsor](https://github.com/sponsors/dipdup-io) diff --git a/pdm.lock b/pdm.lock index 4195bbb51..572382348 100644 --- a/pdm.lock +++ b/pdm.lock @@ -236,7 +236,7 @@ files = [ [[package]] name = "black" -version = "23.10.0" +version = "23.10.1" requires_python = ">=3.8" summary = "The uncompromising code formatter." dependencies = [ @@ -247,12 +247,12 @@ dependencies = [ "platformdirs>=2", ] files = [ - {file = "black-23.10.0-cp311-cp311-macosx_10_16_arm64.whl", hash = "sha256:30b78ac9b54cf87bcb9910ee3d499d2bc893afd52495066c49d9ee6b21eee06e"}, - {file = "black-23.10.0-cp311-cp311-macosx_10_16_x86_64.whl", hash = "sha256:0e232f24a337fed7a82c1185ae46c56c4a6167fb0fe37411b43e876892c76699"}, - {file = "black-23.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:31946ec6f9c54ed7ba431c38bc81d758970dd734b96b8e8c2b17a367d7908171"}, - {file = "black-23.10.0-cp311-cp311-win_amd64.whl", hash = "sha256:c870bee76ad5f7a5ea7bd01dc646028d05568d33b0b09b7ecfc8ec0da3f3f39c"}, - {file = "black-23.10.0-py3-none-any.whl", hash = "sha256:e223b731a0e025f8ef427dd79d8cd69c167da807f5710add30cdf131f13dd62e"}, - {file = "black-23.10.0.tar.gz", hash = "sha256:31b9f87b277a68d0e99d2905edae08807c007973eaa609da5f0c62def6b7c0bd"}, + {file = "black-23.10.1-cp311-cp311-macosx_10_16_arm64.whl", hash = "sha256:b15b75fc53a2fbcac8a87d3e20f69874d161beef13954747e053bca7a1ce53a0"}, + {file = "black-23.10.1-cp311-cp311-macosx_10_16_x86_64.whl", hash = "sha256:e293e4c2f4a992b980032bbd62df07c1bcff82d6964d6c9496f2cd726e246ace"}, + {file = "black-23.10.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7d56124b7a61d092cb52cce34182a5280e160e6aff3137172a68c2c2c4b76bcb"}, + {file = "black-23.10.1-cp311-cp311-win_amd64.whl", hash = "sha256:3f157a8945a7b2d424da3335f7ace89c14a3b0625e6593d21139c2d8214d55ce"}, + {file = "black-23.10.1-py3-none-any.whl", hash = "sha256:d431e6739f727bb2e0495df64a6c7a5310758e87505f5f8cde9ff6c0f2d7e4fe"}, + {file = "black-23.10.1.tar.gz", hash = "sha256:1f8ce316753428ff68749c65a5f7844631aa18c8679dfd3ca9dc1a289979c258"}, ] [[package]] @@ -875,7 +875,7 @@ files = [ [[package]] name = "mypy" -version = "1.6.0" +version = "1.6.1" requires_python = ">=3.8" summary = "Optional static typing for Python" dependencies = [ @@ -883,13 +883,13 @@ dependencies = [ "typing-extensions>=4.1.0", ] files = [ - {file = "mypy-1.6.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:3c8835a07b8442da900db47ccfda76c92c69c3a575872a5b764332c4bacb5a0a"}, - {file = "mypy-1.6.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:24f3de8b9e7021cd794ad9dfbf2e9fe3f069ff5e28cb57af6f873ffec1cb0425"}, - {file = "mypy-1.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:856bad61ebc7d21dbc019b719e98303dc6256cec6dcc9ebb0b214b81d6901bd8"}, - {file = "mypy-1.6.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:89513ddfda06b5c8ebd64f026d20a61ef264e89125dc82633f3c34eeb50e7d60"}, - {file = "mypy-1.6.0-cp311-cp311-win_amd64.whl", hash = "sha256:9f8464ed410ada641c29f5de3e6716cbdd4f460b31cf755b2af52f2d5ea79ead"}, - {file = "mypy-1.6.0-py3-none-any.whl", hash = "sha256:9e1589ca150a51d9d00bb839bfeca2f7a04f32cd62fad87a847bc0818e15d7dc"}, - {file = "mypy-1.6.0.tar.gz", hash = "sha256:4f3d27537abde1be6d5f2c96c29a454da333a2a271ae7d5bc7110e6d4b7beb3f"}, + {file = "mypy-1.6.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:81af8adaa5e3099469e7623436881eff6b3b06db5ef75e6f5b6d4871263547e5"}, + {file = "mypy-1.6.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:8c223fa57cb154c7eab5156856c231c3f5eace1e0bed9b32a24696b7ba3c3245"}, + {file = "mypy-1.6.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a8032e00ce71c3ceb93eeba63963b864bf635a18f6c0c12da6c13c450eedb183"}, + {file = "mypy-1.6.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:4c46b51de523817a0045b150ed11b56f9fff55f12b9edd0f3ed35b15a2809de0"}, + {file = "mypy-1.6.1-cp311-cp311-win_amd64.whl", hash = "sha256:19f905bcfd9e167159b3d63ecd8cb5e696151c3e59a1742e79bc3bcb540c42c7"}, + {file = "mypy-1.6.1-py3-none-any.whl", hash = "sha256:4cbe68ef919c28ea561165206a2dcb68591c50f3bcf777932323bc208d949cf1"}, + {file = "mypy-1.6.1.tar.gz", hash = "sha256:4d01c00d09a0be62a4ca3f933e315455bde83f37f892ba4b08ce92f3cf44bcc1"}, ] [[package]] @@ -953,21 +953,21 @@ files = [ [[package]] name = "orjson" -version = "3.9.9" +version = "3.9.10" requires_python = ">=3.8" summary = "Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy" files = [ - {file = "orjson-3.9.9-cp311-cp311-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:31d676bc236f6e919d100fb85d0a99812cff1ebffaa58106eaaec9399693e227"}, - {file = "orjson-3.9.9-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:678ffb5c0a6b1518b149cc328c610615d70d9297e351e12c01d0beed5d65360f"}, - {file = "orjson-3.9.9-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a71b0cc21f2c324747bc77c35161e0438e3b5e72db6d3b515310457aba743f7f"}, - {file = "orjson-3.9.9-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:ae72621f216d1d990468291b1ec153e1b46e0ed188a86d54e0941f3dabd09ee8"}, - {file = "orjson-3.9.9-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:512e5a41af008e76451f5a344941d61f48dddcf7d7ddd3073deb555de64596a6"}, - {file = "orjson-3.9.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0f89dc338a12f4357f5bf1b098d3dea6072fb0b643fd35fec556f4941b31ae27"}, - {file = "orjson-3.9.9-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:957a45fb201c61b78bcf655a16afbe8a36c2c27f18a998bd6b5d8a35e358d4ad"}, - {file = "orjson-3.9.9-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:d1c01cf4b8e00c7e98a0a7cf606a30a26c32adf2560be2d7d5d6766d6f474b31"}, - {file = "orjson-3.9.9-cp311-none-win32.whl", hash = "sha256:397a185e5dd7f8ebe88a063fe13e34d61d394ebb8c70a443cee7661b9c89bda7"}, - {file = "orjson-3.9.9-cp311-none-win_amd64.whl", hash = "sha256:24301f2d99d670ded4fb5e2f87643bc7428a54ba49176e38deb2887e42fe82fb"}, - {file = "orjson-3.9.9.tar.gz", hash = "sha256:02e693843c2959befdd82d1ebae8b05ed12d1cb821605d5f9fe9f98ca5c9fd2b"}, + {file = "orjson-3.9.10-cp311-cp311-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:cff7570d492bcf4b64cc862a6e2fb77edd5e5748ad715f487628f102815165e9"}, + {file = "orjson-3.9.10-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ed8bc367f725dfc5cabeed1ae079d00369900231fbb5a5280cf0736c30e2adf7"}, + {file = "orjson-3.9.10-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:c812312847867b6335cfb264772f2a7e85b3b502d3a6b0586aa35e1858528ab1"}, + {file = "orjson-3.9.10-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9edd2856611e5050004f4722922b7b1cd6268da34102667bd49d2a2b18bafb81"}, + {file = "orjson-3.9.10-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:674eb520f02422546c40401f4efaf8207b5e29e420c17051cddf6c02783ff5ca"}, + {file = "orjson-3.9.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1d0dc4310da8b5f6415949bd5ef937e60aeb0eb6b16f95041b5e43e6200821fb"}, + {file = "orjson-3.9.10-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:e99c625b8c95d7741fe057585176b1b8783d46ed4b8932cf98ee145c4facf499"}, + {file = "orjson-3.9.10-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:ec6f18f96b47299c11203edfbdc34e1b69085070d9a3d1f302810cc23ad36bf3"}, + {file = "orjson-3.9.10-cp311-none-win32.whl", hash = "sha256:ce0a29c28dfb8eccd0f16219360530bc3cfdf6bf70ca384dacd36e6c650ef8e8"}, + {file = "orjson-3.9.10-cp311-none-win_amd64.whl", hash = "sha256:cf80b550092cc480a0cbd0750e8189247ff45457e5a023305f7ef1bcec811616"}, + {file = "orjson-3.9.10.tar.gz", hash = "sha256:9ebbdbd6a046c304b1845e96fbcc5559cd296b4dfd3ad2509e33c4d9ce07d6a1"}, ] [[package]] @@ -1239,7 +1239,7 @@ files = [ [[package]] name = "pytest" -version = "7.4.2" +version = "7.4.3" requires_python = ">=3.7" summary = "pytest: simple powerful testing with Python" dependencies = [ @@ -1249,8 +1249,8 @@ dependencies = [ "pluggy<2.0,>=0.12", ] files = [ - {file = "pytest-7.4.2-py3-none-any.whl", hash = "sha256:1d881c6124e08ff0a1bb75ba3ec0bfd8b5354a01c194ddd5a0a870a48d99b002"}, - {file = "pytest-7.4.2.tar.gz", hash = "sha256:a766259cfab564a2ad52cb1aae1b881a75c3eb7e34ca3779697c23ed47c47069"}, + {file = "pytest-7.4.3-py3-none-any.whl", hash = "sha256:0d009c083ea859a71b76adf7c1d502e4bc170b80a8ef002da5806527b9591fac"}, + {file = "pytest-7.4.3.tar.gz", hash = "sha256:d989d136982de4e3b29dabcc838ad581c64e8ed52c11fbe86ddebd9da0818cd5"}, ] [[package]] @@ -1429,15 +1429,15 @@ files = [ [[package]] name = "ruamel-yaml" -version = "0.17.35" +version = "0.18.2" requires_python = ">=3" summary = "ruamel.yaml is a YAML parser/emitter that supports roundtrip preservation of comments, seq/map flow style, and map key order" dependencies = [ "ruamel-yaml-clib>=0.2.7; platform_python_implementation == \"CPython\" and python_version < \"3.13\"", ] files = [ - {file = "ruamel.yaml-0.17.35-py3-none-any.whl", hash = "sha256:b105e3e6fc15b41fdb201ba1b95162ae566a4ef792b9f884c46b4ccc5513a87a"}, - {file = "ruamel.yaml-0.17.35.tar.gz", hash = "sha256:801046a9caacb1b43acc118969b49b96b65e8847f29029563b29ac61d02db61b"}, + {file = "ruamel.yaml-0.18.2-py3-none-any.whl", hash = "sha256:92076ac8a83dbf44ca661dbed3c935229c8cbc2f10b05959dd3bd5292d8353d3"}, + {file = "ruamel.yaml-0.18.2.tar.gz", hash = "sha256:9bce33f7a814cea4c29a9c62fe872d2363d6220b767891d956eacea8fa5e6fe8"}, ] [[package]] @@ -1457,27 +1457,27 @@ files = [ [[package]] name = "ruff" -version = "0.1.0" +version = "0.1.3" requires_python = ">=3.7" summary = "An extremely fast Python linter, written in Rust." files = [ - {file = "ruff-0.1.0-py3-none-macosx_10_7_x86_64.whl", hash = "sha256:87114e254dee35e069e1b922d85d4b21a5b61aec759849f393e1dbb308a00439"}, - {file = "ruff-0.1.0-py3-none-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl", hash = "sha256:764f36d2982cc4a703e69fb73a280b7c539fd74b50c9ee531a4e3fe88152f521"}, - {file = "ruff-0.1.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:65f4b7fb539e5cf0f71e9bd74f8ddab74cabdd673c6fb7f17a4dcfd29f126255"}, - {file = "ruff-0.1.0-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:299fff467a0f163baa282266b310589b21400de0a42d8f68553422fa6bf7ee01"}, - {file = "ruff-0.1.0-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0d412678bf205787263bb702c984012a4f97e460944c072fd7cfa2bd084857c4"}, - {file = "ruff-0.1.0-py3-none-manylinux_2_17_ppc64.manylinux2014_ppc64.whl", hash = "sha256:a5391b49b1669b540924640587d8d24128e45be17d1a916b1801d6645e831581"}, - {file = "ruff-0.1.0-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:ee8cd57f454cdd77bbcf1e11ff4e0046fb6547cac1922cc6e3583ce4b9c326d1"}, - {file = "ruff-0.1.0-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:fa7aeed7bc23861a2b38319b636737bf11cfa55d2109620b49cf995663d3e888"}, - {file = "ruff-0.1.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b04cd4298b43b16824d9a37800e4c145ba75c29c43ce0d74cad1d66d7ae0a4c5"}, - {file = "ruff-0.1.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:7186ccf54707801d91e6314a016d1c7895e21d2e4cd614500d55870ed983aa9f"}, - {file = "ruff-0.1.0-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:d88adfd93849bc62449518228581d132e2023e30ebd2da097f73059900d8dce3"}, - {file = "ruff-0.1.0-py3-none-musllinux_1_2_i686.whl", hash = "sha256:ad2ccdb3bad5a61013c76a9c1240fdfadf2c7103a2aeebd7bcbbed61f363138f"}, - {file = "ruff-0.1.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:b77f6cfa72c6eb19b5cac967cc49762ae14d036db033f7d97a72912770fd8e1c"}, - {file = "ruff-0.1.0-py3-none-win32.whl", hash = "sha256:480bd704e8af1afe3fd444cc52e3c900b936e6ca0baf4fb0281124330b6ceba2"}, - {file = "ruff-0.1.0-py3-none-win_amd64.whl", hash = "sha256:a76ba81860f7ee1f2d5651983f87beb835def94425022dc5f0803108f1b8bfa2"}, - {file = "ruff-0.1.0-py3-none-win_arm64.whl", hash = "sha256:45abdbdab22509a2c6052ecf7050b3f5c7d6b7898dc07e82869401b531d46da4"}, - {file = "ruff-0.1.0.tar.gz", hash = "sha256:ad6b13824714b19c5f8225871cf532afb994470eecb74631cd3500fe817e6b3f"}, + {file = "ruff-0.1.3-py3-none-macosx_10_7_x86_64.whl", hash = "sha256:b46d43d51f7061652eeadb426a9e3caa1e0002470229ab2fc19de8a7b0766901"}, + {file = "ruff-0.1.3-py3-none-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl", hash = "sha256:b8afeb9abd26b4029c72adc9921b8363374f4e7edb78385ffaa80278313a15f9"}, + {file = "ruff-0.1.3-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ca3cf365bf32e9ba7e6db3f48a4d3e2c446cd19ebee04f05338bc3910114528b"}, + {file = "ruff-0.1.3-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4874c165f96c14a00590dcc727a04dca0cfd110334c24b039458c06cf78a672e"}, + {file = "ruff-0.1.3-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:eec2dd31eed114e48ea42dbffc443e9b7221976554a504767ceaee3dd38edeb8"}, + {file = "ruff-0.1.3-py3-none-manylinux_2_17_ppc64.manylinux2014_ppc64.whl", hash = "sha256:dc3ec4edb3b73f21b4aa51337e16674c752f1d76a4a543af56d7d04e97769613"}, + {file = "ruff-0.1.3-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:2e3de9ed2e39160800281848ff4670e1698037ca039bda7b9274f849258d26ce"}, + {file = "ruff-0.1.3-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:1c595193881922cc0556a90f3af99b1c5681f0c552e7a2a189956141d8666fe8"}, + {file = "ruff-0.1.3-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0f75e670d529aa2288cd00fc0e9b9287603d95e1536d7a7e0cafe00f75e0dd9d"}, + {file = "ruff-0.1.3-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:76dd49f6cd945d82d9d4a9a6622c54a994689d8d7b22fa1322983389b4892e20"}, + {file = "ruff-0.1.3-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:918b454bc4f8874a616f0d725590277c42949431ceb303950e87fef7a7d94cb3"}, + {file = "ruff-0.1.3-py3-none-musllinux_1_2_i686.whl", hash = "sha256:d8859605e729cd5e53aa38275568dbbdb4fe882d2ea2714c5453b678dca83784"}, + {file = "ruff-0.1.3-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:0b6c55f5ef8d9dd05b230bb6ab80bc4381ecb60ae56db0330f660ea240cb0d4a"}, + {file = "ruff-0.1.3-py3-none-win32.whl", hash = "sha256:3e7afcbdcfbe3399c34e0f6370c30f6e529193c731b885316c5a09c9e4317eef"}, + {file = "ruff-0.1.3-py3-none-win_amd64.whl", hash = "sha256:7a18df6638cec4a5bd75350639b2bb2a2366e01222825562c7346674bdceb7ea"}, + {file = "ruff-0.1.3-py3-none-win_arm64.whl", hash = "sha256:12fd53696c83a194a2db7f9a46337ce06445fb9aa7d25ea6f293cf75b21aca9f"}, + {file = "ruff-0.1.3.tar.gz", hash = "sha256:3ba6145369a151401d5db79f0a47d50e470384d0d89d0d6f7fab0b589ad07c34"}, ] [[package]] @@ -1819,7 +1819,7 @@ files = [ [[package]] name = "web3" -version = "6.11.0" +version = "6.11.1" requires_python = ">=3.7.2" summary = "web3.py" dependencies = [ @@ -1829,7 +1829,7 @@ dependencies = [ "eth-hash[pycryptodome]>=0.5.1", "eth-typing>=3.0.0", "eth-utils>=2.1.0", - "hexbytes>=0.1.0", + "hexbytes<0.4.0,>=0.1.0", "jsonschema>=4.0.0", "lru-dict>=1.1.6", "protobuf>=4.21.6", @@ -1840,8 +1840,8 @@ dependencies = [ "websockets>=10.0.0", ] files = [ - {file = "web3-6.11.0-py3-none-any.whl", hash = "sha256:44e79da6a4765eacf137f2f388e37aa0c1e24a93bdfb462cffe9441d1be3d509"}, - {file = "web3-6.11.0.tar.gz", hash = "sha256:050dea52ae73d787272e7ecba7249f096595938c90cce1a384c20375c6b0f720"}, + {file = "web3-6.11.1-py3-none-any.whl", hash = "sha256:0d39f58cbb0c652b45e711f01e01ec655117b47ba4eefd1f9550c52d205afa8c"}, + {file = "web3-6.11.1.tar.gz", hash = "sha256:d301d7120922d5b9e5c9535ef9780012ea25ea4011c2b177490ba7d3ef886b92"}, ] [[package]] diff --git a/pyproject.toml b/pyproject.toml index 28adf8991..a078580bc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "dipdup" description = "Modular framework for creating selective indexers and featureful backends for dapps" -version = "7.0.2" +version = "7.1.0" license = { text = "MIT" } authors = [ { name = "Lev Gorodetskii", email = "dipdup@drsr.io" }, @@ -109,14 +109,14 @@ _mypy = "mypy src tests scripts" [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint", "test"] +composite = ["format", "lint", "test"] [tool.pdm.scripts.demos] help = "Recreate demo projects from templates" shell = """ python scripts/update_demos.py python scripts/init_demos.py - pdm run fmt + pdm run format pdm run lint """ @@ -140,7 +140,7 @@ cmd = "python scripts/dump_references.py" help = "Find FIXME and TODO comments" cmd = "grep -r -e 'FIXME: ' -e 'TODO: ' -e 'type: ignore' -n src/dipdup --color" -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with black" composite = ["_black"] diff --git a/requirements.dev.txt b/requirements.dev.txt index 0d8d8c184..4b4ef03e3 100644 --- a/requirements.dev.txt +++ b/requirements.dev.txt @@ -16,7 +16,7 @@ asyncpg==0.28.0 attrs==23.1.0 babel==2.12.1 bitarray==2.8.1 -black==23.10.0 +black==23.10.1 certifi==2023.7.22 chardet==5.2.0 charset-normalizer==3.2.0 @@ -54,12 +54,12 @@ lru-dict==1.2.0 MarkupSafe==2.1.3 msgpack==1.0.5 multidict==6.0.4 -mypy==1.6.0 +mypy==1.6.1 mypy-extensions==1.0.0 numpy==1.25.2 openapi-schema-validator==0.4.4 openapi-spec-validator==0.5.7 -orjson==3.9.9 +orjson==3.9.10 packaging==23.1 parsimonious==0.9.0 pathable==0.4.3 @@ -79,7 +79,7 @@ pypika-tortoise==0.1.6 pyrsistent==0.19.3 pysignalr==0.2.0 PySnooper==1.2.0 -pytest==7.4.2 +pytest==7.4.3 pytest-aiohttp==1.0.5 pytest-asyncio==0.21.1 pytest-cov==4.1.0 @@ -92,9 +92,9 @@ regex==2023.8.8 requests==2.31.0 rfc3339-validator==0.1.4 rlp==3.0.0 -ruamel-yaml==0.17.35 +ruamel-yaml==0.18.2 ruamel-yaml-clib==0.2.7 -ruff==0.1.0 +ruff==0.1.3 sentry-sdk==1.32.0 setuptools==68.2.2 six==1.16.0 @@ -121,7 +121,7 @@ typing-extensions==4.7.1 tzlocal==5.0.1 urllib3==2.0.6 watchdog==3.0.0 -web3==6.11.0 +web3==6.11.1 websocket-client==1.6.1 websockets==10.4 yarl==1.9.2 diff --git a/requirements.txt b/requirements.txt index 86c34bdc8..fae56e54c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,7 +14,7 @@ asyncclick==8.1.3.4 asyncpg==0.28.0 attrs==23.1.0 bitarray==2.8.1 -black==23.10.0 +black==23.10.1 certifi==2023.7.22 chardet==5.2.0 charset-normalizer==3.2.0 @@ -50,7 +50,7 @@ mypy-extensions==1.0.0 numpy==1.25.2 openapi-schema-validator==0.4.4 openapi-spec-validator==0.5.7 -orjson==3.9.9 +orjson==3.9.10 packaging==23.1 parsimonious==0.9.0 pathable==0.4.3 @@ -75,7 +75,7 @@ regex==2023.8.8 requests==2.31.0 rfc3339-validator==0.1.4 rlp==3.0.0 -ruamel-yaml==0.17.35 +ruamel-yaml==0.18.2 ruamel-yaml-clib==0.2.7 sentry-sdk==1.32.0 setuptools==68.2.2 @@ -90,6 +90,6 @@ tortoise-orm==0.19.3 typing-extensions==4.7.1 tzlocal==5.0.1 urllib3==2.0.6 -web3==6.11.0 +web3==6.11.1 websockets==10.4 yarl==1.9.2 diff --git a/schema.json b/schema.json index 09e4894e1..8e55a3f25 100644 --- a/schema.json +++ b/schema.json @@ -129,13 +129,6 @@ } ] }, - { - "allOf": [ - { - "$ref": "#/$defs/SubsquidOperationsIndexConfig" - } - ] - }, { "allOf": [ { @@ -178,6 +171,13 @@ } ] }, + { + "allOf": [ + { + "$ref": "#/$defs/TzktTokenBalancesIndexConfig" + } + ] + }, { "allOf": [ { @@ -199,13 +199,6 @@ } ] }, - { - "allOf": [ - { - "$ref": "#/$defs/SubsquidOperationsIndexConfig" - } - ] - }, { "allOf": [ { @@ -247,6 +240,13 @@ "$ref": "#/$defs/TzktTokenTransfersIndexConfig" } ] + }, + { + "allOf": [ + { + "$ref": "#/$defs/TzktTokenBalancesIndexConfig" + } + ] } ] } @@ -287,11 +287,19 @@ "default": null }, "sentry": { - "allOf": [ + "anyOf": [ + { + "allOf": [ + { + "$ref": "#/$defs/SentryConfig" + } + ] + }, { - "$ref": "#/$defs/SentryConfig" + "type": "null" } - ] + ], + "default": null }, "prometheus": { "anyOf": [ @@ -1116,124 +1124,9 @@ ], "default": null }, - "first_level": { - "type": "integer", - "default": 0 - }, - "last_level": { - "type": "integer", - "default": 0 - } - }, - "required": [ - "kind", - "datasource", - "handlers" - ] - }, - "SubsquidOperationsHandlerConfig": { - "type": "object", - "title": "SubsquidOperationsHandlerConfig", - "properties": { - "callback": { - "type": "string" - }, - "contract": { - "anyOf": [ - { - "type": "string" - }, - { - "allOf": [ - { - "$ref": "#/$defs/EvmContractConfig" - } - ] - } - ] - }, - "method": { - "type": "string" - }, - "events": { - "type": "array", - "items": {} - } - }, - "required": [ - "callback", - "contract", - "method", - "events" - ] - }, - "SubsquidOperationsIndexConfig": { - "type": "object", - "title": "SubsquidOperationsIndexConfig", - "properties": { - "kind": { - "enum": [ - "evm.subsquid.operations" - ] - }, - "datasource": { - "anyOf": [ - { - "type": "string" - }, - { - "allOf": [ - { - "$ref": "#/$defs/SubsquidDatasourceConfig" - } - ] - } - ] - }, - "handlers": { - "type": "array", - "items": { - "allOf": [ - { - "$ref": "#/$defs/SubsquidOperationsHandlerConfig" - } - ] - } - }, - "contracts": { - "type": "array", - "items": { - "allOf": [ - { - "$ref": "#/$defs/EvmContractConfig" - } - ] - } - }, - "abi": { - "anyOf": [ - { - "allOf": [ - { - "$ref": "#/$defs/AbiDatasourceConfig" - } - ] - }, - { - "type": "array", - "items": { - "allOf": [ - { - "$ref": "#/$defs/AbiDatasourceConfig" - } - ] - } - }, - { - "type": "null" - } - ], - "default": null + "node_only": { + "type": "boolean", + "default": false }, "first_level": { "type": "integer", @@ -1247,8 +1140,7 @@ "required": [ "kind", "datasource", - "handlers", - "contracts" + "handlers" ] }, "TzktBigMapsHandlerConfig": { @@ -1936,6 +1828,94 @@ "datasource" ] }, + "TzktTokenBalancesHandlerConfig": { + "type": "object", + "title": "TzktTokenBalancesHandlerConfig", + "properties": { + "callback": { + "type": "string" + }, + "contract": { + "anyOf": [ + { + "type": "string" + }, + { + "allOf": [ + { + "$ref": "#/$defs/TezosContractConfig" + } + ] + }, + { + "type": "null" + } + ], + "default": null + }, + "token_id": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null + } + }, + "required": [ + "callback" + ] + }, + "TzktTokenBalancesIndexConfig": { + "type": "object", + "title": "TzktTokenBalancesIndexConfig", + "properties": { + "kind": { + "enum": [ + "tezos.tzkt.token_balances" + ] + }, + "datasource": { + "anyOf": [ + { + "type": "string" + }, + { + "allOf": [ + { + "$ref": "#/$defs/TzktDatasourceConfig" + } + ] + } + ] + }, + "handlers": { + "type": "array", + "items": { + "allOf": [ + { + "$ref": "#/$defs/TzktTokenBalancesHandlerConfig" + } + ] + } + }, + "first_level": { + "type": "integer", + "default": 0 + }, + "last_level": { + "type": "integer", + "default": 0 + } + }, + "required": [ + "kind", + "datasource" + ] + }, "IndexTemplateConfig": { "type": "object", "title": "IndexTemplateConfig", @@ -2110,8 +2090,7 @@ "title": "SentryConfig", "properties": { "dsn": { - "type": "string", - "default": "" + "type": "string" }, "environment": { "anyOf": [ @@ -2161,7 +2140,10 @@ "type": "boolean", "default": false } - } + }, + "required": [ + "dsn" + ] }, "PrometheusConfig": { "type": "object", @@ -2274,11 +2256,19 @@ "default": false }, "api": { - "allOf": [ + "anyOf": [ { - "$ref": "#/$defs/ApiConfig" + "allOf": [ + { + "$ref": "#/$defs/ApiConfig" + } + ] + }, + { + "type": "null" } - ] + ], + "default": null }, "metrics": { "allOf": [ diff --git a/scripts/dump_schema.py b/scripts/dump_schema.py index 55cc32f69..34bf80230 100644 --- a/scripts/dump_schema.py +++ b/scripts/dump_schema.py @@ -1,4 +1,5 @@ #!/usr/bin/env python3 +# NOTE: Run `pdm add -G dev -e ../dc_schema` first. from pathlib import Path import orjson diff --git a/src/demo_auction/README.md b/src/demo_auction/README.md index 86d55b6cb..cb7b8232b 100644 --- a/src/demo_auction/README.md +++ b/src/demo_auction/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_auction/configs/dipdup.compose.yaml b/src/demo_auction/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_auction/configs/dipdup.compose.yaml +++ b/src/demo_auction/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_auction/configs/dipdup.sqlite.yaml b/src/demo_auction/configs/dipdup.sqlite.yaml index dffc3cd8a..2ca8f1e15 100644 --- a/src/demo_auction/configs/dipdup.sqlite.yaml +++ b/src/demo_auction/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_auction.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_auction.sqlite} \ No newline at end of file diff --git a/src/demo_auction/configs/dipdup.swarm.yaml b/src/demo_auction/configs/dipdup.swarm.yaml index e895e7f29..f4afa3749 100644 --- a/src/demo_auction/configs/dipdup.swarm.yaml +++ b/src/demo_auction/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_auction/configs/replay.yaml b/src/demo_auction/configs/replay.yaml index 707bdae87..e5c47cc56 100644 --- a/src/demo_auction/configs/replay.yaml +++ b/src/demo_auction/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_auction/deploy/.env.default b/src/demo_auction/deploy/.env.default index 989816481..554e48901 100644 --- a/src/demo_auction/deploy/.env.default +++ b/src/demo_auction/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_auction/deploy/sqlite.env.default b/src/demo_auction/deploy/sqlite.env.default index 7ea0caab1..f81f25f98 100644 --- a/src/demo_auction/deploy/sqlite.env.default +++ b/src/demo_auction/deploy/sqlite.env.default @@ -1,6 +1,5 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_auction.sqlite TZKT_URL=https://api.tzkt.io diff --git a/src/demo_auction/deploy/swarm.env.default b/src/demo_auction/deploy/swarm.env.default index 0d95462fb..412095482 100644 --- a/src/demo_auction/deploy/swarm.env.default +++ b/src/demo_auction/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_auction_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_auction_db POSTGRES_PASSWORD= diff --git a/src/demo_auction/pyproject.toml b/src/demo_auction/pyproject.toml index e903c5409..47ee321fe 100644 --- a/src/demo_auction/pyproject.toml +++ b/src/demo_auction/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_auction" version = "0.0.1" description = "TzColors NFT marketplace" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_auction ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_big_maps/README.md b/src/demo_big_maps/README.md index 2b4bad68f..939e14408 100644 --- a/src/demo_big_maps/README.md +++ b/src/demo_big_maps/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_big_maps/configs/dipdup.compose.yaml b/src/demo_big_maps/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_big_maps/configs/dipdup.compose.yaml +++ b/src/demo_big_maps/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_big_maps/configs/dipdup.sqlite.yaml b/src/demo_big_maps/configs/dipdup.sqlite.yaml index 955c641b3..8df1ed877 100644 --- a/src/demo_big_maps/configs/dipdup.sqlite.yaml +++ b/src/demo_big_maps/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_big_maps.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_big_maps.sqlite} \ No newline at end of file diff --git a/src/demo_big_maps/configs/dipdup.swarm.yaml b/src/demo_big_maps/configs/dipdup.swarm.yaml index 52b2653bf..bb771298e 100644 --- a/src/demo_big_maps/configs/dipdup.swarm.yaml +++ b/src/demo_big_maps/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_big_maps/configs/replay.yaml b/src/demo_big_maps/configs/replay.yaml index a9204ce65..b2839c674 100644 --- a/src/demo_big_maps/configs/replay.yaml +++ b/src/demo_big_maps/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_big_maps/deploy/.env.default b/src/demo_big_maps/deploy/.env.default index 989816481..554e48901 100644 --- a/src/demo_big_maps/deploy/.env.default +++ b/src/demo_big_maps/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_big_maps/deploy/sqlite.env.default b/src/demo_big_maps/deploy/sqlite.env.default index 566cf9c38..f4c0a9169 100644 --- a/src/demo_big_maps/deploy/sqlite.env.default +++ b/src/demo_big_maps/deploy/sqlite.env.default @@ -1,6 +1,5 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_big_maps.sqlite TZKT_URL=https://api.tzkt.io diff --git a/src/demo_big_maps/deploy/swarm.env.default b/src/demo_big_maps/deploy/swarm.env.default index 00831fed0..172ef4454 100644 --- a/src/demo_big_maps/deploy/swarm.env.default +++ b/src/demo_big_maps/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_big_maps_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_big_maps_db POSTGRES_PASSWORD= diff --git a/src/demo_big_maps/pyproject.toml b/src/demo_big_maps/pyproject.toml index 0b8be9e98..b37085ef1 100644 --- a/src/demo_big_maps/pyproject.toml +++ b/src/demo_big_maps/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_big_maps" version = "0.0.1" description = "Indexing specific big maps" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_big_maps ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_blank/README.md b/src/demo_blank/README.md index f55b34bd5..89e82dd7a 100644 --- a/src/demo_blank/README.md +++ b/src/demo_blank/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_blank/configs/dipdup.compose.yaml b/src/demo_blank/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_blank/configs/dipdup.compose.yaml +++ b/src/demo_blank/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_blank/configs/dipdup.sqlite.yaml b/src/demo_blank/configs/dipdup.sqlite.yaml index 1a6dcab9f..06d6421ae 100644 --- a/src/demo_blank/configs/dipdup.sqlite.yaml +++ b/src/demo_blank/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_blank.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_blank.sqlite} \ No newline at end of file diff --git a/src/demo_blank/configs/dipdup.swarm.yaml b/src/demo_blank/configs/dipdup.swarm.yaml index 56c09abc1..802b9ca5c 100644 --- a/src/demo_blank/configs/dipdup.swarm.yaml +++ b/src/demo_blank/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_blank/configs/replay.yaml b/src/demo_blank/configs/replay.yaml index 4828891cf..9af4c22ad 100644 --- a/src/demo_blank/configs/replay.yaml +++ b/src/demo_blank/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_blank/deploy/.env.default b/src/demo_blank/deploy/.env.default index 00b262cb5..dfeb321f6 100644 --- a/src/demo_blank/deploy/.env.default +++ b/src/demo_blank/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_blank/deploy/sqlite.env.default b/src/demo_blank/deploy/sqlite.env.default index 222603810..4330bbe4a 100644 --- a/src/demo_blank/deploy/sqlite.env.default +++ b/src/demo_blank/deploy/sqlite.env.default @@ -1,5 +1,4 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_blank.sqlite diff --git a/src/demo_blank/deploy/swarm.env.default b/src/demo_blank/deploy/swarm.env.default index c9831dad8..2d795bedd 100644 --- a/src/demo_blank/deploy/swarm.env.default +++ b/src/demo_blank/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_blank_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_blank_db POSTGRES_PASSWORD= diff --git a/src/demo_blank/pyproject.toml b/src/demo_blank/pyproject.toml index d7b9a5f2a..b0ff29b20 100644 --- a/src/demo_blank/pyproject.toml +++ b/src/demo_blank/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_blank" version = "0.0.1" description = "Empty config for a fresh start" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_blank ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_dao/README.md b/src/demo_dao/README.md index 065d774d9..fcfba6d46 100644 --- a/src/demo_dao/README.md +++ b/src/demo_dao/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_dao/configs/dipdup.compose.yaml b/src/demo_dao/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_dao/configs/dipdup.compose.yaml +++ b/src/demo_dao/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_dao/configs/dipdup.sqlite.yaml b/src/demo_dao/configs/dipdup.sqlite.yaml index 775804992..a2c27535e 100644 --- a/src/demo_dao/configs/dipdup.sqlite.yaml +++ b/src/demo_dao/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_dao.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_dao.sqlite} \ No newline at end of file diff --git a/src/demo_dao/configs/dipdup.swarm.yaml b/src/demo_dao/configs/dipdup.swarm.yaml index 27b7e3790..25ebe92c0 100644 --- a/src/demo_dao/configs/dipdup.swarm.yaml +++ b/src/demo_dao/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_dao/configs/replay.yaml b/src/demo_dao/configs/replay.yaml index f789cd121..daf81b2ec 100644 --- a/src/demo_dao/configs/replay.yaml +++ b/src/demo_dao/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_dao/deploy/.env.default b/src/demo_dao/deploy/.env.default index 989816481..554e48901 100644 --- a/src/demo_dao/deploy/.env.default +++ b/src/demo_dao/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_dao/deploy/sqlite.env.default b/src/demo_dao/deploy/sqlite.env.default index 6a1ce24e6..4fb909b89 100644 --- a/src/demo_dao/deploy/sqlite.env.default +++ b/src/demo_dao/deploy/sqlite.env.default @@ -1,6 +1,5 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_dao.sqlite TZKT_URL=https://api.tzkt.io diff --git a/src/demo_dao/deploy/swarm.env.default b/src/demo_dao/deploy/swarm.env.default index 11ffccb53..e964a7f16 100644 --- a/src/demo_dao/deploy/swarm.env.default +++ b/src/demo_dao/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_dao_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_dao_db POSTGRES_PASSWORD= diff --git a/src/demo_dao/pyproject.toml b/src/demo_dao/pyproject.toml index ed8e09859..470f72a99 100644 --- a/src/demo_dao/pyproject.toml +++ b/src/demo_dao/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_dao" version = "0.0.1" description = "Homebase DAO registry" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_dao ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_dex/README.md b/src/demo_dex/README.md index 033c5a45d..44ae6a1fc 100644 --- a/src/demo_dex/README.md +++ b/src/demo_dex/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_dex/configs/dipdup.compose.yaml b/src/demo_dex/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_dex/configs/dipdup.compose.yaml +++ b/src/demo_dex/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_dex/configs/dipdup.sqlite.yaml b/src/demo_dex/configs/dipdup.sqlite.yaml index ac4118239..f49ec3183 100644 --- a/src/demo_dex/configs/dipdup.sqlite.yaml +++ b/src/demo_dex/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_dex.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_dex.sqlite} \ No newline at end of file diff --git a/src/demo_dex/configs/dipdup.swarm.yaml b/src/demo_dex/configs/dipdup.swarm.yaml index ececfb9e7..69d5e6c90 100644 --- a/src/demo_dex/configs/dipdup.swarm.yaml +++ b/src/demo_dex/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_dex/configs/replay.yaml b/src/demo_dex/configs/replay.yaml index adfe558a2..95a149018 100644 --- a/src/demo_dex/configs/replay.yaml +++ b/src/demo_dex/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_dex/deploy/.env.default b/src/demo_dex/deploy/.env.default index 00b262cb5..dfeb321f6 100644 --- a/src/demo_dex/deploy/.env.default +++ b/src/demo_dex/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_dex/deploy/sqlite.env.default b/src/demo_dex/deploy/sqlite.env.default index 9f446ad29..d4d8e31d4 100644 --- a/src/demo_dex/deploy/sqlite.env.default +++ b/src/demo_dex/deploy/sqlite.env.default @@ -1,5 +1,4 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_dex.sqlite diff --git a/src/demo_dex/deploy/swarm.env.default b/src/demo_dex/deploy/swarm.env.default index 26fdfb5c5..224b8971e 100644 --- a/src/demo_dex/deploy/swarm.env.default +++ b/src/demo_dex/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_dex_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_dex_db POSTGRES_PASSWORD= diff --git a/src/demo_dex/pyproject.toml b/src/demo_dex/pyproject.toml index 7a78a63a4..67d81e4d0 100644 --- a/src/demo_dex/pyproject.toml +++ b/src/demo_dex/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_dex" version = "0.0.1" description = "Quipuswap DEX balances and liquidity" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_dex ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_domains/README.md b/src/demo_domains/README.md index 40c5a1f80..e8f2411cc 100644 --- a/src/demo_domains/README.md +++ b/src/demo_domains/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_domains/configs/dipdup.compose.yaml b/src/demo_domains/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_domains/configs/dipdup.compose.yaml +++ b/src/demo_domains/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_domains/configs/dipdup.sqlite.yaml b/src/demo_domains/configs/dipdup.sqlite.yaml index 069e54831..3ab868788 100644 --- a/src/demo_domains/configs/dipdup.sqlite.yaml +++ b/src/demo_domains/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_domains.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_domains.sqlite} \ No newline at end of file diff --git a/src/demo_domains/configs/dipdup.swarm.yaml b/src/demo_domains/configs/dipdup.swarm.yaml index 4fc25843b..751b96ce3 100644 --- a/src/demo_domains/configs/dipdup.swarm.yaml +++ b/src/demo_domains/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_domains/configs/replay.yaml b/src/demo_domains/configs/replay.yaml index d32580d2b..e7c9c4131 100644 --- a/src/demo_domains/configs/replay.yaml +++ b/src/demo_domains/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_domains/deploy/.env.default b/src/demo_domains/deploy/.env.default index 989816481..554e48901 100644 --- a/src/demo_domains/deploy/.env.default +++ b/src/demo_domains/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_domains/deploy/sqlite.env.default b/src/demo_domains/deploy/sqlite.env.default index 0e6e61706..3da773352 100644 --- a/src/demo_domains/deploy/sqlite.env.default +++ b/src/demo_domains/deploy/sqlite.env.default @@ -1,6 +1,5 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_domains.sqlite TZKT_URL=https://api.tzkt.io diff --git a/src/demo_domains/deploy/swarm.env.default b/src/demo_domains/deploy/swarm.env.default index 1cb588da6..6f61958ea 100644 --- a/src/demo_domains/deploy/swarm.env.default +++ b/src/demo_domains/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_domains_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_domains_db POSTGRES_PASSWORD= diff --git a/src/demo_domains/pyproject.toml b/src/demo_domains/pyproject.toml index 83d7de114..d40901cbc 100644 --- a/src/demo_domains/pyproject.toml +++ b/src/demo_domains/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_domains" version = "0.0.1" description = "Tezos Domains name service" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_domains ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_events/README.md b/src/demo_events/README.md index e5fb8f9a2..bd7b32daa 100644 --- a/src/demo_events/README.md +++ b/src/demo_events/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_events/configs/dipdup.compose.yaml b/src/demo_events/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_events/configs/dipdup.compose.yaml +++ b/src/demo_events/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_events/configs/dipdup.sqlite.yaml b/src/demo_events/configs/dipdup.sqlite.yaml index e4353f485..992f0b0f5 100644 --- a/src/demo_events/configs/dipdup.sqlite.yaml +++ b/src/demo_events/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_events.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_events.sqlite} \ No newline at end of file diff --git a/src/demo_events/configs/dipdup.swarm.yaml b/src/demo_events/configs/dipdup.swarm.yaml index 089563185..616fa288f 100644 --- a/src/demo_events/configs/dipdup.swarm.yaml +++ b/src/demo_events/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_events/configs/replay.yaml b/src/demo_events/configs/replay.yaml index 78145a1b7..3a679cbea 100644 --- a/src/demo_events/configs/replay.yaml +++ b/src/demo_events/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_events/deploy/.env.default b/src/demo_events/deploy/.env.default index 00b262cb5..dfeb321f6 100644 --- a/src/demo_events/deploy/.env.default +++ b/src/demo_events/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_events/deploy/sqlite.env.default b/src/demo_events/deploy/sqlite.env.default index 23a1a28fe..e4aa178d4 100644 --- a/src/demo_events/deploy/sqlite.env.default +++ b/src/demo_events/deploy/sqlite.env.default @@ -1,5 +1,4 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_events.sqlite diff --git a/src/demo_events/deploy/swarm.env.default b/src/demo_events/deploy/swarm.env.default index 36868d09d..79f560cc0 100644 --- a/src/demo_events/deploy/swarm.env.default +++ b/src/demo_events/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_events_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_events_db POSTGRES_PASSWORD= diff --git a/src/demo_events/pyproject.toml b/src/demo_events/pyproject.toml index a0d28f383..230f87535 100644 --- a/src/demo_events/pyproject.toml +++ b/src/demo_events/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_events" version = "0.0.1" description = "Processing contract events" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_events ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_evm_events/README.md b/src/demo_evm_events/README.md index acae27b00..4dff84c1c 100644 --- a/src/demo_evm_events/README.md +++ b/src/demo_evm_events/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_evm_events/configs/dipdup.compose.yaml b/src/demo_evm_events/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_evm_events/configs/dipdup.compose.yaml +++ b/src/demo_evm_events/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_evm_events/configs/dipdup.sqlite.yaml b/src/demo_evm_events/configs/dipdup.sqlite.yaml index 7e83193e0..38c42eec4 100644 --- a/src/demo_evm_events/configs/dipdup.sqlite.yaml +++ b/src/demo_evm_events/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_evm_events.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_evm_events.sqlite} \ No newline at end of file diff --git a/src/demo_evm_events/configs/dipdup.swarm.yaml b/src/demo_evm_events/configs/dipdup.swarm.yaml index d68de8233..327c83edd 100644 --- a/src/demo_evm_events/configs/dipdup.swarm.yaml +++ b/src/demo_evm_events/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_evm_events/configs/replay.yaml b/src/demo_evm_events/configs/replay.yaml index 7325023af..4c01efcb8 100644 --- a/src/demo_evm_events/configs/replay.yaml +++ b/src/demo_evm_events/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_evm_events/deploy/.env.default b/src/demo_evm_events/deploy/.env.default index 12b48e540..b27f07c1d 100644 --- a/src/demo_evm_events/deploy/.env.default +++ b/src/demo_evm_events/deploy/.env.default @@ -5,7 +5,6 @@ ALCHEMY_KEY='' ARCHIVE_URL=https://v2.archive.subsquid.io/network/ethereum-mainnet HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_evm_events/deploy/sqlite.env.default b/src/demo_evm_events/deploy/sqlite.env.default index 1f8629991..50686d7be 100644 --- a/src/demo_evm_events/deploy/sqlite.env.default +++ b/src/demo_evm_events/deploy/sqlite.env.default @@ -3,5 +3,4 @@ # ALCHEMY_KEY='' ARCHIVE_URL=https://v2.archive.subsquid.io/network/ethereum-mainnet -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_evm_events.sqlite diff --git a/src/demo_evm_events/deploy/swarm.env.default b/src/demo_evm_events/deploy/swarm.env.default index e9d0ccac5..d498d334a 100644 --- a/src/demo_evm_events/deploy/swarm.env.default +++ b/src/demo_evm_events/deploy/swarm.env.default @@ -5,7 +5,6 @@ ALCHEMY_KEY='' ARCHIVE_URL=https://v2.archive.subsquid.io/network/ethereum-mainnet HASURA_HOST=demo_evm_events_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_evm_events_db POSTGRES_PASSWORD= diff --git a/src/demo_evm_events/pyproject.toml b/src/demo_evm_events/pyproject.toml index 4a660e146..6ccd40d2e 100644 --- a/src/demo_evm_events/pyproject.toml +++ b/src/demo_evm_events/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_evm_events" version = "0.0.1" description = "A very basic indexer for USDt transfers" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_evm_events ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_factories/README.md b/src/demo_factories/README.md index 0dd3b9e0f..33a4358ce 100644 --- a/src/demo_factories/README.md +++ b/src/demo_factories/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_factories/configs/dipdup.compose.yaml b/src/demo_factories/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_factories/configs/dipdup.compose.yaml +++ b/src/demo_factories/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_factories/configs/dipdup.sqlite.yaml b/src/demo_factories/configs/dipdup.sqlite.yaml index 6ebf861d8..93d5aca86 100644 --- a/src/demo_factories/configs/dipdup.sqlite.yaml +++ b/src/demo_factories/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_factories.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_factories.sqlite} \ No newline at end of file diff --git a/src/demo_factories/configs/dipdup.swarm.yaml b/src/demo_factories/configs/dipdup.swarm.yaml index f45839e47..4e9655c1e 100644 --- a/src/demo_factories/configs/dipdup.swarm.yaml +++ b/src/demo_factories/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_factories/configs/replay.yaml b/src/demo_factories/configs/replay.yaml index 2faec184f..4e66cc6d4 100644 --- a/src/demo_factories/configs/replay.yaml +++ b/src/demo_factories/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_factories/deploy/.env.default b/src/demo_factories/deploy/.env.default index 989816481..554e48901 100644 --- a/src/demo_factories/deploy/.env.default +++ b/src/demo_factories/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_factories/deploy/sqlite.env.default b/src/demo_factories/deploy/sqlite.env.default index f04dc153a..08014397b 100644 --- a/src/demo_factories/deploy/sqlite.env.default +++ b/src/demo_factories/deploy/sqlite.env.default @@ -1,6 +1,5 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_factories.sqlite TZKT_URL=https://api.tzkt.io diff --git a/src/demo_factories/deploy/swarm.env.default b/src/demo_factories/deploy/swarm.env.default index aadefb214..26e8d9e07 100644 --- a/src/demo_factories/deploy/swarm.env.default +++ b/src/demo_factories/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_factories_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_factories_db POSTGRES_PASSWORD= diff --git a/src/demo_factories/pyproject.toml b/src/demo_factories/pyproject.toml index c87acd14f..ed99001be 100644 --- a/src/demo_factories/pyproject.toml +++ b/src/demo_factories/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_factories" version = "0.0.1" description = "A very basic indexer for USDt transfers" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_factories ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_head/README.md b/src/demo_head/README.md index 6d8a620ac..39909b33e 100644 --- a/src/demo_head/README.md +++ b/src/demo_head/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_head/configs/dipdup.compose.yaml b/src/demo_head/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_head/configs/dipdup.compose.yaml +++ b/src/demo_head/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_head/configs/dipdup.sqlite.yaml b/src/demo_head/configs/dipdup.sqlite.yaml index 3c75eccf9..ae8548359 100644 --- a/src/demo_head/configs/dipdup.sqlite.yaml +++ b/src/demo_head/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_head.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_head.sqlite} \ No newline at end of file diff --git a/src/demo_head/configs/dipdup.swarm.yaml b/src/demo_head/configs/dipdup.swarm.yaml index 42bb5466d..9deaaea41 100644 --- a/src/demo_head/configs/dipdup.swarm.yaml +++ b/src/demo_head/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_head/configs/replay.yaml b/src/demo_head/configs/replay.yaml index fcae53e38..7a003a1ca 100644 --- a/src/demo_head/configs/replay.yaml +++ b/src/demo_head/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_head/deploy/.env.default b/src/demo_head/deploy/.env.default index 989816481..554e48901 100644 --- a/src/demo_head/deploy/.env.default +++ b/src/demo_head/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_head/deploy/sqlite.env.default b/src/demo_head/deploy/sqlite.env.default index fd957a0b9..35be1f000 100644 --- a/src/demo_head/deploy/sqlite.env.default +++ b/src/demo_head/deploy/sqlite.env.default @@ -1,6 +1,5 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_head.sqlite TZKT_URL=https://api.tzkt.io diff --git a/src/demo_head/deploy/swarm.env.default b/src/demo_head/deploy/swarm.env.default index d02631171..b1ea994bb 100644 --- a/src/demo_head/deploy/swarm.env.default +++ b/src/demo_head/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_head_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_head_db POSTGRES_PASSWORD= diff --git a/src/demo_head/pyproject.toml b/src/demo_head/pyproject.toml index f6a880e16..445d940cc 100644 --- a/src/demo_head/pyproject.toml +++ b/src/demo_head/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_head" version = "0.0.1" description = "Processing head block metadata" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_head ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_nft_marketplace/README.md b/src/demo_nft_marketplace/README.md index df905ded7..91fdd872d 100644 --- a/src/demo_nft_marketplace/README.md +++ b/src/demo_nft_marketplace/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_nft_marketplace/configs/dipdup.compose.yaml b/src/demo_nft_marketplace/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_nft_marketplace/configs/dipdup.compose.yaml +++ b/src/demo_nft_marketplace/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_nft_marketplace/configs/dipdup.sqlite.yaml b/src/demo_nft_marketplace/configs/dipdup.sqlite.yaml index 36b39541f..6af001af8 100644 --- a/src/demo_nft_marketplace/configs/dipdup.sqlite.yaml +++ b/src/demo_nft_marketplace/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_nft_marketplace.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_nft_marketplace.sqlite} \ No newline at end of file diff --git a/src/demo_nft_marketplace/configs/dipdup.swarm.yaml b/src/demo_nft_marketplace/configs/dipdup.swarm.yaml index 8bf3289de..93d0cec83 100644 --- a/src/demo_nft_marketplace/configs/dipdup.swarm.yaml +++ b/src/demo_nft_marketplace/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_nft_marketplace/configs/replay.yaml b/src/demo_nft_marketplace/configs/replay.yaml index 969e2d6d9..1360f7087 100644 --- a/src/demo_nft_marketplace/configs/replay.yaml +++ b/src/demo_nft_marketplace/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_nft_marketplace/deploy/.env.default b/src/demo_nft_marketplace/deploy/.env.default index e9c7004f5..1c9bb1137 100644 --- a/src/demo_nft_marketplace/deploy/.env.default +++ b/src/demo_nft_marketplace/deploy/.env.default @@ -5,7 +5,6 @@ HASURA_HOST=hasura HASURA_SECRET= HEN_MINTER=KT1Hkg5qeNhfwpKW4fXvq7HGZB9z2EnmCCA9 HEN_OBJKTS=KT1RJ6PbjHpwc3M5rw5s2Nbmefwbuwbdxton -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_nft_marketplace/deploy/sqlite.env.default b/src/demo_nft_marketplace/deploy/sqlite.env.default index d9e1d577c..c37d1bdcb 100644 --- a/src/demo_nft_marketplace/deploy/sqlite.env.default +++ b/src/demo_nft_marketplace/deploy/sqlite.env.default @@ -3,6 +3,5 @@ # HEN_MINTER=KT1Hkg5qeNhfwpKW4fXvq7HGZB9z2EnmCCA9 HEN_OBJKTS=KT1RJ6PbjHpwc3M5rw5s2Nbmefwbuwbdxton -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_nft_marketplace.sqlite TZKT_URL=https://api.tzkt.io diff --git a/src/demo_nft_marketplace/deploy/swarm.env.default b/src/demo_nft_marketplace/deploy/swarm.env.default index acaa37194..36933c139 100644 --- a/src/demo_nft_marketplace/deploy/swarm.env.default +++ b/src/demo_nft_marketplace/deploy/swarm.env.default @@ -5,7 +5,6 @@ HASURA_HOST=demo_nft_marketplace_hasura HASURA_SECRET= HEN_MINTER=KT1Hkg5qeNhfwpKW4fXvq7HGZB9z2EnmCCA9 HEN_OBJKTS=KT1RJ6PbjHpwc3M5rw5s2Nbmefwbuwbdxton -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_nft_marketplace_db POSTGRES_PASSWORD= diff --git a/src/demo_nft_marketplace/pyproject.toml b/src/demo_nft_marketplace/pyproject.toml index e1456fa61..ba2b3fbb6 100644 --- a/src/demo_nft_marketplace/pyproject.toml +++ b/src/demo_nft_marketplace/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_nft_marketplace" version = "0.0.1" description = "hic at nunc NFT marketplace" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_nft_marketplace ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_raw/README.md b/src/demo_raw/README.md index 8560d8b20..75ad18113 100644 --- a/src/demo_raw/README.md +++ b/src/demo_raw/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_raw/configs/dipdup.compose.yaml b/src/demo_raw/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_raw/configs/dipdup.compose.yaml +++ b/src/demo_raw/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_raw/configs/dipdup.sqlite.yaml b/src/demo_raw/configs/dipdup.sqlite.yaml index 899523eff..346081864 100644 --- a/src/demo_raw/configs/dipdup.sqlite.yaml +++ b/src/demo_raw/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_raw.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_raw.sqlite} \ No newline at end of file diff --git a/src/demo_raw/configs/dipdup.swarm.yaml b/src/demo_raw/configs/dipdup.swarm.yaml index 8d6704e98..ffc66d0f0 100644 --- a/src/demo_raw/configs/dipdup.swarm.yaml +++ b/src/demo_raw/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_raw/configs/replay.yaml b/src/demo_raw/configs/replay.yaml index b31aff5ef..a82884ced 100644 --- a/src/demo_raw/configs/replay.yaml +++ b/src/demo_raw/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_raw/deploy/.env.default b/src/demo_raw/deploy/.env.default index 989816481..554e48901 100644 --- a/src/demo_raw/deploy/.env.default +++ b/src/demo_raw/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_raw/deploy/sqlite.env.default b/src/demo_raw/deploy/sqlite.env.default index 4c171f2c7..608747b27 100644 --- a/src/demo_raw/deploy/sqlite.env.default +++ b/src/demo_raw/deploy/sqlite.env.default @@ -1,6 +1,5 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_raw.sqlite TZKT_URL=https://api.tzkt.io diff --git a/src/demo_raw/deploy/swarm.env.default b/src/demo_raw/deploy/swarm.env.default index 1a777eade..cd67fdb81 100644 --- a/src/demo_raw/deploy/swarm.env.default +++ b/src/demo_raw/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_raw_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_raw_db POSTGRES_PASSWORD= diff --git a/src/demo_raw/pyproject.toml b/src/demo_raw/pyproject.toml index 0a0e4da86..443364ead 100644 --- a/src/demo_raw/pyproject.toml +++ b/src/demo_raw/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_raw" version = "0.0.1" description = "Process raw operations without filtering and strict typing" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_raw ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_token/README.md b/src/demo_token/README.md index 7af1ae7c6..63fc8d44c 100644 --- a/src/demo_token/README.md +++ b/src/demo_token/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_token/configs/dipdup.compose.yaml b/src/demo_token/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_token/configs/dipdup.compose.yaml +++ b/src/demo_token/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_token/configs/dipdup.sqlite.yaml b/src/demo_token/configs/dipdup.sqlite.yaml index a8f86d97b..e5d29cb18 100644 --- a/src/demo_token/configs/dipdup.sqlite.yaml +++ b/src/demo_token/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_token.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_token.sqlite} \ No newline at end of file diff --git a/src/demo_token/configs/dipdup.swarm.yaml b/src/demo_token/configs/dipdup.swarm.yaml index 1add573b4..e555a0d5a 100644 --- a/src/demo_token/configs/dipdup.swarm.yaml +++ b/src/demo_token/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_token/configs/replay.yaml b/src/demo_token/configs/replay.yaml index e924d5490..0f74900a2 100644 --- a/src/demo_token/configs/replay.yaml +++ b/src/demo_token/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_token/deploy/.env.default b/src/demo_token/deploy/.env.default index 00b262cb5..dfeb321f6 100644 --- a/src/demo_token/deploy/.env.default +++ b/src/demo_token/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_token/deploy/sqlite.env.default b/src/demo_token/deploy/sqlite.env.default index bca89fb79..576242e46 100644 --- a/src/demo_token/deploy/sqlite.env.default +++ b/src/demo_token/deploy/sqlite.env.default @@ -1,5 +1,4 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_token.sqlite diff --git a/src/demo_token/deploy/swarm.env.default b/src/demo_token/deploy/swarm.env.default index 6d95f8883..344b3648b 100644 --- a/src/demo_token/deploy/swarm.env.default +++ b/src/demo_token/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_token_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_token_db POSTGRES_PASSWORD= diff --git a/src/demo_token/pyproject.toml b/src/demo_token/pyproject.toml index 8e991f514..3d22be73b 100644 --- a/src/demo_token/pyproject.toml +++ b/src/demo_token/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_token" version = "0.0.1" description = "TzBTC FA1.2 token operations" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_token ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_token_balances/README.md b/src/demo_token_balances/README.md index 2caca0555..00d61eda6 100644 --- a/src/demo_token_balances/README.md +++ b/src/demo_token_balances/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_token_balances/configs/dipdup.compose.yaml b/src/demo_token_balances/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_token_balances/configs/dipdup.compose.yaml +++ b/src/demo_token_balances/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_token_balances/configs/dipdup.sqlite.yaml b/src/demo_token_balances/configs/dipdup.sqlite.yaml index ec7006d3d..6f551b10d 100644 --- a/src/demo_token_balances/configs/dipdup.sqlite.yaml +++ b/src/demo_token_balances/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_token_balances.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_token_balances.sqlite} \ No newline at end of file diff --git a/src/demo_token_balances/configs/dipdup.swarm.yaml b/src/demo_token_balances/configs/dipdup.swarm.yaml index 6c7c79b82..6cc84a8e4 100644 --- a/src/demo_token_balances/configs/dipdup.swarm.yaml +++ b/src/demo_token_balances/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_token_balances/configs/replay.yaml b/src/demo_token_balances/configs/replay.yaml index ef2c6ef31..df97e7115 100644 --- a/src/demo_token_balances/configs/replay.yaml +++ b/src/demo_token_balances/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_token_balances/deploy/.env.default b/src/demo_token_balances/deploy/.env.default index 00b262cb5..dfeb321f6 100644 --- a/src/demo_token_balances/deploy/.env.default +++ b/src/demo_token_balances/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_token_balances/deploy/sqlite.env.default b/src/demo_token_balances/deploy/sqlite.env.default index 4cacf6bc5..60ba0208e 100644 --- a/src/demo_token_balances/deploy/sqlite.env.default +++ b/src/demo_token_balances/deploy/sqlite.env.default @@ -1,5 +1,4 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_token_balances.sqlite diff --git a/src/demo_token_balances/deploy/swarm.env.default b/src/demo_token_balances/deploy/swarm.env.default index c4811e380..7e5ab144a 100644 --- a/src/demo_token_balances/deploy/swarm.env.default +++ b/src/demo_token_balances/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_token_balances_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_token_balances_db POSTGRES_PASSWORD= diff --git a/src/demo_token_balances/handlers/on_balance_update.py b/src/demo_token_balances/handlers/on_balance_update.py index ec6716d4d..cc4110e61 100644 --- a/src/demo_token_balances/handlers/on_balance_update.py +++ b/src/demo_token_balances/handlers/on_balance_update.py @@ -1,5 +1,4 @@ from decimal import Decimal - from demo_token_balances import models as models from dipdup.context import HandlerContext from dipdup.models.tezos_tzkt import TzktTokenBalanceData diff --git a/src/demo_token_balances/models/__init__.py b/src/demo_token_balances/models/__init__.py index 833a8deea..143eef701 100644 --- a/src/demo_token_balances/models/__init__.py +++ b/src/demo_token_balances/models/__init__.py @@ -1,4 +1,5 @@ from dipdup import fields + from dipdup.models import Model diff --git a/src/demo_token_balances/pyproject.toml b/src/demo_token_balances/pyproject.toml index 32ad2b1ff..145dc214d 100644 --- a/src/demo_token_balances/pyproject.toml +++ b/src/demo_token_balances/pyproject.toml @@ -1,15 +1,16 @@ +# Generated by DipDup 7.1.0+editable [project] name = "demo_token_balances" version = "0.0.1" description = "TzBTC FA1.2 token balances" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -25,10 +26,22 @@ _isort = "isort ." _black = "black ." _ruff = "ruff check --fix ." _mypy = "mypy --no-incremental --exclude demo_token_balances ." -all = { composite = ["fmt", "lint"] } -fmt = { composite = ["_isort", "_black"] } -lint = { composite = ["_ruff", "_mypy"] } -image = "docker buildx build . --load --progress plain -f deploy/Dockerfile -t demo_token_balances:latest" + +[tool.pdm.scripts.all] +help = "Run all checks" +composite = ["format", "lint"] + +[tool.pdm.scripts.format] +help = "Format code with isort and black" +composite = ["_isort", "_black"] + +[tool.pdm.scripts.image] +help = "Build Docker image" +cmd = "docker buildx build . --load --progress plain -f deploy/Dockerfile -t demo_token_balances:latest" + +[tool.pdm.scripts.lint] +help = "Check code with ruff and mypy" +composite = ["_ruff", "_mypy"] [tool.isort] line_length = 120 @@ -42,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_token_transfers/README.md b/src/demo_token_transfers/README.md index 6bf627168..5145c7a0e 100644 --- a/src/demo_token_transfers/README.md +++ b/src/demo_token_transfers/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_token_transfers/configs/dipdup.compose.yaml b/src/demo_token_transfers/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_token_transfers/configs/dipdup.compose.yaml +++ b/src/demo_token_transfers/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_token_transfers/configs/dipdup.sqlite.yaml b/src/demo_token_transfers/configs/dipdup.sqlite.yaml index 924dd08b3..a15ecd6cf 100644 --- a/src/demo_token_transfers/configs/dipdup.sqlite.yaml +++ b/src/demo_token_transfers/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_token_transfers.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_token_transfers.sqlite} \ No newline at end of file diff --git a/src/demo_token_transfers/configs/dipdup.swarm.yaml b/src/demo_token_transfers/configs/dipdup.swarm.yaml index f8787c9c4..908008e98 100644 --- a/src/demo_token_transfers/configs/dipdup.swarm.yaml +++ b/src/demo_token_transfers/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_token_transfers/configs/replay.yaml b/src/demo_token_transfers/configs/replay.yaml index 7531eb545..a9199f63c 100644 --- a/src/demo_token_transfers/configs/replay.yaml +++ b/src/demo_token_transfers/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_token_transfers/deploy/.env.default b/src/demo_token_transfers/deploy/.env.default index 00b262cb5..dfeb321f6 100644 --- a/src/demo_token_transfers/deploy/.env.default +++ b/src/demo_token_transfers/deploy/.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_token_transfers/deploy/sqlite.env.default b/src/demo_token_transfers/deploy/sqlite.env.default index 4f0cfe833..f779a7d66 100644 --- a/src/demo_token_transfers/deploy/sqlite.env.default +++ b/src/demo_token_transfers/deploy/sqlite.env.default @@ -1,5 +1,4 @@ # This env file was generated automatically by DipDup. Do not edit it! # Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option. # -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_token_transfers.sqlite diff --git a/src/demo_token_transfers/deploy/swarm.env.default b/src/demo_token_transfers/deploy/swarm.env.default index beb57d979..2f17e0028 100644 --- a/src/demo_token_transfers/deploy/swarm.env.default +++ b/src/demo_token_transfers/deploy/swarm.env.default @@ -3,7 +3,6 @@ # HASURA_HOST=demo_token_transfers_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_token_transfers_db POSTGRES_PASSWORD= diff --git a/src/demo_token_transfers/pyproject.toml b/src/demo_token_transfers/pyproject.toml index 9ee68568a..801deb846 100644 --- a/src/demo_token_transfers/pyproject.toml +++ b/src/demo_token_transfers/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_token_transfers" version = "0.0.1" description = "TzBTC FA1.2 token transfers" license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_token_transfers ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_uniswap/README.md b/src/demo_uniswap/README.md index 7c23cf6b9..e16afbec7 100644 --- a/src/demo_uniswap/README.md +++ b/src/demo_uniswap/README.md @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. \ No newline at end of file +# Show all available scripts +pdm run --list +``` \ No newline at end of file diff --git a/src/demo_uniswap/configs/dipdup.compose.yaml b/src/demo_uniswap/configs/dipdup.compose.yaml index 98824f36f..ff2b5e1a1 100644 --- a/src/demo_uniswap/configs/dipdup.compose.yaml +++ b/src/demo_uniswap/configs/dipdup.compose.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_uniswap/configs/dipdup.sqlite.yaml b/src/demo_uniswap/configs/dipdup.sqlite.yaml index df747aea5..899fe07ad 100644 --- a/src/demo_uniswap/configs/dipdup.sqlite.yaml +++ b/src/demo_uniswap/configs/dipdup.sqlite.yaml @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/demo_uniswap.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/demo_uniswap.sqlite} \ No newline at end of file diff --git a/src/demo_uniswap/configs/dipdup.swarm.yaml b/src/demo_uniswap/configs/dipdup.swarm.yaml index 30dae3900..f973972a2 100644 --- a/src/demo_uniswap/configs/dipdup.swarm.yaml +++ b/src/demo_uniswap/configs/dipdup.swarm.yaml @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/demo_uniswap/configs/replay.yaml b/src/demo_uniswap/configs/replay.yaml index 5366508de..f7f873e0b 100644 --- a/src/demo_uniswap/configs/replay.yaml +++ b/src/demo_uniswap/configs/replay.yaml @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: dipdup_version: 7 diff --git a/src/demo_uniswap/deploy/.env.default b/src/demo_uniswap/deploy/.env.default index 12b48e540..b27f07c1d 100644 --- a/src/demo_uniswap/deploy/.env.default +++ b/src/demo_uniswap/deploy/.env.default @@ -5,7 +5,6 @@ ALCHEMY_KEY='' ARCHIVE_URL=https://v2.archive.subsquid.io/network/ethereum-mainnet HASURA_HOST=hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=db POSTGRES_PASSWORD= diff --git a/src/demo_uniswap/deploy/sqlite.env.default b/src/demo_uniswap/deploy/sqlite.env.default index b7bc101fe..8045bd5dc 100644 --- a/src/demo_uniswap/deploy/sqlite.env.default +++ b/src/demo_uniswap/deploy/sqlite.env.default @@ -3,5 +3,4 @@ # ALCHEMY_KEY='' ARCHIVE_URL=https://v2.archive.subsquid.io/network/ethereum-mainnet -LOGLEVEL=INFO SQLITE_PATH=/tmp/demo_uniswap.sqlite diff --git a/src/demo_uniswap/deploy/swarm.env.default b/src/demo_uniswap/deploy/swarm.env.default index 54f198fdb..8a61a0f8c 100644 --- a/src/demo_uniswap/deploy/swarm.env.default +++ b/src/demo_uniswap/deploy/swarm.env.default @@ -5,7 +5,6 @@ ALCHEMY_KEY='' ARCHIVE_URL=https://v2.archive.subsquid.io/network/ethereum-mainnet HASURA_HOST=demo_uniswap_hasura HASURA_SECRET= -LOGLEVEL=INFO POSTGRES_DB=dipdup POSTGRES_HOST=demo_uniswap_db POSTGRES_PASSWORD= diff --git a/src/demo_uniswap/models/__init__.py b/src/demo_uniswap/models/__init__.py index 04e1288de..f8ad477b8 100644 --- a/src/demo_uniswap/models/__init__.py +++ b/src/demo_uniswap/models/__init__.py @@ -46,7 +46,7 @@ class Token(CachedModel): # volume in token units volume = fields.DecimalField(decimal_places=18, max_digits=96, default=0) # volume in derived USD - volume_usd = fields.DecimalField(decimal_places=18, max_digits=96, default=0) + volume_usd = fields.DecimalField(decimal_places=18, max_digits=96, default=0, index=True) # volume in USD even on pools with less reliable USD values untracked_volume_usd = fields.DecimalField(decimal_places=18, max_digits=96, default=0) # fees in USD @@ -336,7 +336,7 @@ class Swap(Model): # pointer to transaction transaction_hash = fields.TextField() # timestamp of transaction - timestamp = fields.BigIntField() + timestamp = fields.DatetimeField(index=True) # pool swap occured within pool: fields.ForeignKeyRelation[Pool] = fields.ForeignKeyField('models.Pool', related_name='swaps') # allow indexing by tokens diff --git a/src/demo_uniswap/pyproject.toml b/src/demo_uniswap/pyproject.toml index 0edbcdadb..357a11c16 100644 --- a/src/demo_uniswap/pyproject.toml +++ b/src/demo_uniswap/pyproject.toml @@ -1,16 +1,16 @@ -# Generated by DipDup 7.0.2+editable +# Generated by DipDup 7.1.0+editable [project] name = "demo_uniswap" version = "0.0.1" description = "Uniswap V3 pools, positions, swaps, ticks, etc." license = { text = "MIT" } authors = [ - { name = "John Doe", email = "john_doe@example.com" } + { name = "John Doe", email = "john_doe@example.com" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>=7,<8" + "dipdup>=7,<8", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude demo_uniswap ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = 120 target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/demo_uniswap/sql/on_reindex/01_create_mv_token_price.sql b/src/demo_uniswap/sql/on_reindex/01_create_mv_token_price.sql deleted file mode 100644 index e6dbb0c26..000000000 --- a/src/demo_uniswap/sql/on_reindex/01_create_mv_token_price.sql +++ /dev/null @@ -1,32 +0,0 @@ -CREATE MATERIALIZED VIEW - token_price -AS - -( - SELECT - to_timestamp(timestamp) as ts, - token0_id as token_id, - abs(amount_usd/amount0) as price, - amount_usd as volume - FROM swap - WHERE - amount_usd!=0 - AND - amount0!=0 -) UNION ( - SELECT - to_timestamp(timestamp) as ts, - token1_id as token_id, - abs(amount_usd/amount1) as price, - amount_usd as volume - FROM swap - WHERE - amount_usd!=0 - AND - amount1!=0 -) - -WITH DATA; - -CREATE INDEX token_price_ts ON token_price(ts); -CREATE INDEX token_price_token_id ON token_price(token_id); \ No newline at end of file diff --git a/src/demo_uniswap/sql/on_reindex/10_create_mv_quotes_1m.sql b/src/demo_uniswap/sql/on_reindex/10_create_mv_quotes_1m.sql deleted file mode 100644 index 1cdea4377..000000000 --- a/src/demo_uniswap/sql/on_reindex/10_create_mv_quotes_1m.sql +++ /dev/null @@ -1,25 +0,0 @@ -CREATE MATERIALIZED VIEW - quotes_1m -AS - -SELECT - time_bucket('1 minute'::INTERVAL, ts) AS bucket, - token_id, - candlestick_agg( - ts, - price, - volume - ) as candlestick -FROM token_price - -GROUP BY - bucket, - token_id -ORDER BY - bucket, - token_id - -WITH DATA; - -CREATE INDEX quotes_1m_bucket ON quotes_1m(bucket); -CREATE INDEX quotes_1m_token_id ON quotes_1m(token_id); \ No newline at end of file diff --git a/src/demo_uniswap/sql/on_reindex/11_create_mv_quotes_1d.sql b/src/demo_uniswap/sql/on_reindex/11_create_mv_quotes_1d.sql deleted file mode 100644 index e75ae7bcd..000000000 --- a/src/demo_uniswap/sql/on_reindex/11_create_mv_quotes_1d.sql +++ /dev/null @@ -1,25 +0,0 @@ -CREATE MATERIALIZED VIEW - quotes_1d -AS - -SELECT - time_bucket('1 day'::INTERVAL, ts) AS bucket, - token_id, - candlestick_agg( - ts, - price, - volume - ) as candlestick -FROM token_price - -GROUP BY - bucket, - token_id -ORDER BY - bucket, - token_id - -WITH DATA; - -CREATE INDEX quotes_1d_bucket ON quotes_1d(bucket); -CREATE INDEX quotes_1d_token_id ON quotes_1d(token_id); \ No newline at end of file diff --git a/src/demo_uniswap/sql/on_reindex/20_create_ca_quotes_1m.sql b/src/demo_uniswap/sql/on_reindex/20_create_ca_quotes_1m.sql index e339e07f5..088474fa3 100644 --- a/src/demo_uniswap/sql/on_reindex/20_create_ca_quotes_1m.sql +++ b/src/demo_uniswap/sql/on_reindex/20_create_ca_quotes_1m.sql @@ -1,25 +1,36 @@ --- CREATE MATERIALIZED VIEW --- candlestick_1m --- WITH (timescaledb.continuous) AS --- --- SELECT --- time_bucket('1 minute'::INTERVAL, to_timestamp(timestamp)) AS bucket, --- token0_id as token_id, --- candlestick_agg( --- to_timestamp(timestamp), --- abs(amount_usd/amount0), --- amount_usd --- ) as candlestick --- FROM swap --- WHERE --- amount_usd!=0 --- AND --- amount0!=0 --- --- GROUP BY --- bucket, --- token0_id --- ORDER BY --- bucket, --- token0_id --- WITH NO DATA; \ No newline at end of file +CREATE MATERIALIZED VIEW + candlestick_1m +WITH (timescaledb.continuous) AS + +SELECT + time_bucket('1 minute'::INTERVAL, timestamp) AS bucket, + token0_id as token_id, + candlestick_agg( + timestamp, + abs(amount_usd/amount0), + amount_usd + ) as candlestick +FROM swap + WHERE + amount_usd!=0 + AND + amount0!=0 + +GROUP BY + bucket, + token0_id +ORDER BY + bucket, + token0_id +WITH NO DATA; + +CREATE INDEX candlestick_1m_bucket ON candlestick_1m(bucket); +CREATE INDEX candlestick_1m_token_id ON candlestick_1m(token_id); + +SELECT add_continuous_aggregate_policy( + 'candlestick_1m', + start_offset => INTERVAL '1 hour', + end_offset => INTERVAL '0 minutes', + schedule_interval => INTERVAL '1 minute', + initial_start := '2018-07-01' +); \ No newline at end of file diff --git a/src/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1d.sql b/src/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1d.sql deleted file mode 100644 index 08605fa75..000000000 --- a/src/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1d.sql +++ /dev/null @@ -1,25 +0,0 @@ --- CREATE MATERIALIZED VIEW --- candlestick_1d --- WITH (timescaledb.continuous) AS --- --- SELECT --- time_bucket('1 day'::INTERVAL, to_timestamp(timestamp)) AS bucket, --- token0_id as token_id, --- candlestick_agg( --- to_timestamp(timestamp), --- abs(amount_usd/amount0), --- amount_usd --- ) as candlestick --- FROM swap --- WHERE --- amount_usd!=0 --- AND --- amount0!=0 --- --- GROUP BY --- bucket, --- token0_id --- ORDER BY --- bucket, --- token0_id --- WITH NO DATA; \ No newline at end of file diff --git a/src/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1h.sql b/src/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1h.sql new file mode 100644 index 000000000..e8001bdc3 --- /dev/null +++ b/src/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1h.sql @@ -0,0 +1,36 @@ +CREATE MATERIALIZED VIEW + candlestick_1h +WITH (timescaledb.continuous) AS + +SELECT + time_bucket('1 hour'::INTERVAL, timestamp) AS bucket, + token0_id as token_id, + candlestick_agg( + timestamp, + abs(amount_usd/amount0), + amount_usd + ) as candlestick +FROM swap + WHERE + amount_usd!=0 + AND + amount0!=0 + +GROUP BY + bucket, + token0_id +ORDER BY + bucket, + token0_id +WITH NO DATA; + +CREATE INDEX candlestick_1h_bucket ON candlestick_1h(bucket); +CREATE INDEX candlestick_1h_token_id ON candlestick_1h(token_id); + +SELECT add_continuous_aggregate_policy( + 'candlestick_1h', + start_offset => INTERVAL '2 hour', + end_offset => INTERVAL '0 minutes', + schedule_interval => INTERVAL '1 hour', + initial_start := '2018-07-01' +); \ No newline at end of file diff --git a/src/demo_uniswap/sql/on_reindex/22_create_ca_quotes_1d.sql b/src/demo_uniswap/sql/on_reindex/22_create_ca_quotes_1d.sql new file mode 100644 index 000000000..3938bb1d5 --- /dev/null +++ b/src/demo_uniswap/sql/on_reindex/22_create_ca_quotes_1d.sql @@ -0,0 +1,36 @@ +CREATE MATERIALIZED VIEW + candlestick_1d +WITH (timescaledb.continuous) AS + +SELECT + time_bucket('1 day'::INTERVAL, timestamp) AS bucket, + token0_id as token_id, + candlestick_agg( + timestamp, + abs(amount_usd/amount0), + amount_usd + ) as candlestick +FROM swap + WHERE + amount_usd!=0 + AND + amount0!=0 + +GROUP BY + bucket, + token0_id +ORDER BY + bucket, + token0_id +WITH NO DATA; + +CREATE INDEX candlestick_1d_bucket ON candlestick_1d(bucket); +CREATE INDEX candlestick_1d_token_id ON candlestick_1d(token_id); + +SELECT add_continuous_aggregate_policy( + 'candlestick_1d', + start_offset => INTERVAL '2 days', + end_offset => INTERVAL '0 minutes', + schedule_interval => INTERVAL '1 hour', + initial_start := '2018-07-01' +); \ No newline at end of file diff --git a/src/dipdup/cli.py b/src/dipdup/cli.py index 2fbb676a4..78c130b2f 100644 --- a/src/dipdup/cli.py +++ b/src/dipdup/cli.py @@ -137,7 +137,9 @@ async def wrapper(ctx: click.Context, *args: Any, **kwargs: Any) -> None: async def _check_version() -> None: - if 'rc' in __version__ or 'b' in __version__: + if '+editable' in __version__: + return + if '-rc' in __version__: _logger.warning( 'You are running a pre-release version of DipDup. Please, report any issues to the GitHub repository.' ) @@ -338,7 +340,7 @@ async def config(ctx: click.Context) -> None: @config.command(name='export') @click.option('--unsafe', is_flag=True, help='Resolve environment variables or use default values from the config.') -@click.option('--full', is_flag=True, help='Resolve index templates.') +@click.option('--full', '-f', is_flag=True, help='Resolve index templates.') @click.pass_context @_cli_wrapper async def config_export(ctx: click.Context, unsafe: bool, full: bool) -> None: @@ -361,26 +363,35 @@ async def config_export(ctx: click.Context, unsafe: bool, full: bool) -> None: @config.command(name='env') @click.option('--output', '-o', type=str, default=None, help='Output to file instead of stdout.') @click.option('--unsafe', is_flag=True, help='Resolve environment variables or use default values from the config.') -@click.option('--compose', is_flag=True, help='Output in docker-compose format.') +@click.option('--compose', '-c', is_flag=True, help='Output in docker-compose format.') +@click.option('--internal', '-i', is_flag=True, help='Include internal variables.') @click.pass_context @_cli_wrapper -async def config_env(ctx: click.Context, output: str | None, unsafe: bool, compose: bool) -> None: +async def config_env( + ctx: click.Context, + output: str | None, + unsafe: bool, + compose: bool, + internal: bool, +) -> None: """Dump environment variables used in DipDup config. If variable is not set, default value will be used. """ from dipdup.yaml import DipDupYAMLConfig - config, environment = DipDupYAMLConfig.load( + _, environment = DipDupYAMLConfig.load( paths=ctx.obj.config._paths, environment=unsafe, ) + if internal: + environment.update(env.dump()) if compose: - content = '\nservices:\n dipdup:\n environment:\n' + content = 'services:\n dipdup:\n environment:\n' _tab = ' ' * 6 for k, v in sorted(environment.items()): line = f'{_tab}- {k}=' + '${' + k - if v: + if v is not None: line += ':-' + v + '}' else: line += '}' @@ -402,7 +413,7 @@ async def hasura(ctx: click.Context) -> None: @hasura.command(name='configure') -@click.option('--force', is_flag=True, help='Proceed even if Hasura is already configured.') +@click.option('--force', '-f', is_flag=True, help='Proceed even if Hasura is already configured.') @click.pass_context @_cli_wrapper async def hasura_configure(ctx: click.Context, force: bool) -> None: @@ -477,8 +488,8 @@ async def schema_approve(ctx: click.Context) -> None: @schema.command(name='wipe') -@click.option('--immune', is_flag=True, help='Drop immune tables too.') -@click.option('--force', is_flag=True, help='Skip confirmation prompt.') +@click.option('--immune', '-i', is_flag=True, help='Drop immune tables too.') +@click.option('--force', '-f', is_flag=True, help='Skip confirmation prompt.') @click.pass_context @_cli_wrapper async def schema_wipe(ctx: click.Context, immune: bool, force: bool) -> None: @@ -731,6 +742,17 @@ async def self_update( dipdup.install.install(quiet, force, None, None, None) +@self.command(name='env', hidden=True) +@click.pass_context +@_cli_wrapper +async def self_env(ctx: click.Context) -> None: + import dipdup.install + + env = dipdup.install.DipDupEnvironment() + env.refresh() + env.print() + + @cli.group() @click.pass_context @_cli_wrapper diff --git a/src/dipdup/codegen/__init__.py b/src/dipdup/codegen/__init__.py index 933124a3b..38d305cfa 100644 --- a/src/dipdup/codegen/__init__.py +++ b/src/dipdup/codegen/__init__.py @@ -54,7 +54,7 @@ async def init( self._package.create() replay = self._package.replay - if base: + if base or self._include: if not replay: raise FrameworkException('`--base` option passed but `configs/replay.yaml` file is missing') _logger.info('Recreating base template with replay.yaml') diff --git a/src/dipdup/config/__init__.py b/src/dipdup/config/__init__.py index f045c5199..096fff3e0 100644 --- a/src/dipdup/config/__init__.py +++ b/src/dipdup/config/__init__.py @@ -831,23 +831,23 @@ def _validate(self) -> None: ) self.advanced.rollback_depth = rollback_depth - # NOTE: Bigmap indexes with `skip_history` require early realtime + if self.advanced.early_realtime: + return + + # NOTE: Indexes that process only the current state imply early realtime. from dipdup.config.tezos_tzkt_big_maps import TzktBigMapsIndexConfig + from dipdup.config.tezos_tzkt_token_balances import TzktTokenBalancesIndexConfig for name, index_config in self.indexes.items(): - if isinstance(index_config, TzktBigMapsIndexConfig) and index_config.skip_history != SkipHistory.never: - _logger.warning('`%s` index is configured to skip history; enabling early realtime', name) + is_big_maps = ( + isinstance(index_config, TzktBigMapsIndexConfig) and index_config.skip_history != SkipHistory.never + ) + is_token_balances = isinstance(index_config, TzktTokenBalancesIndexConfig) + if is_big_maps or is_token_balances: + _logger.info('`%s` index is configured to skip history; implying `early_realtime` flag', name) self.advanced.early_realtime = True break - # NOTE: Optional checks - for flag in ( - 'crash_reporting', - 'metadata_interface', - ): - if getattr(self.advanced, flag, None): - _logger.warning('`%s flag was removed and has no effect', flag) - def _resolve_template(self, template_config: IndexTemplateConfig) -> None: _logger.debug('Resolving index config `%s` from template `%s`', template_config.name, template_config.template) diff --git a/src/dipdup/env.py b/src/dipdup/env.py index 90731e2ef..4ca63f3db 100644 --- a/src/dipdup/env.py +++ b/src/dipdup/env.py @@ -73,6 +73,17 @@ def set_test() -> None: TEST: bool +def dump() -> dict[str, str]: + return { + 'DIPDUP_CI': get('DIPDUP_CI') or '', + 'DIPDUP_DEBUG': get('DIPDUP_DEBUG') or '', + 'DIPDUP_DOCKER': get('DIPDUP_DOCKER') or '', + 'DIPDUP_NEXT': get('DIPDUP_NEXT') or '', + 'DIPDUP_REPLAY_PATH': get('DIPDUP_REPLAY_PATH') or '', + 'DIPDUP_TEST': get('DIPDUP_TEST') or '', + } + + def read() -> None: global CI, DEBUG, DOCKER, NEXT, REPLAY_PATH, TEST CI = get_bool('DIPDUP_CI') diff --git a/src/dipdup/install.py b/src/dipdup/install.py index 8a5086eda..00eb1bb34 100755 --- a/src/dipdup/install.py +++ b/src/dipdup/install.py @@ -18,7 +18,6 @@ GITHUB = 'https://github.com/dipdup-io/dipdup.git' WHICH_CMDS = ( - 'bash', 'python3.11', 'pipx', 'dipdup', @@ -27,6 +26,12 @@ 'pyvenv', 'pyenv', ) +ENV_VARS = ( + 'SHELL', + 'VIRTUAL_ENV', + 'PATH', + 'PYTHONPATH', +) WELCOME_ASCII = """\0 ____ _ ____ @@ -68,7 +73,8 @@ def done(msg: str) -> NoReturn: # NOTE: DipDup has `tabulate` dep, don't use this one elsewhere -def _tab(text: str, indent: int = 20) -> str: +# NOTE: Weird default is to match indentation in `EPILOG`. +def _tab(text: str, indent: int = 23) -> str: return text + ' ' * (indent - len(text)) @@ -91,14 +97,20 @@ def print(self) -> None: print(WELCOME_ASCII) print(EPILOG) print() + print(_tab('OS:') + f'{self._os} ({self._arch})') print(_tab('Python:') + sys.version) - print(_tab('PATH:') + os.environ['PATH']) - print(_tab('PYTHONPATH:') + os.environ.get('PYTHONPATH', '')) print() + + for var in ENV_VARS: + if var in os.environ: + print(_tab(var + ':') + os.environ[var]) + print() + for command, path in self._commands.items(): print(_tab(f'{command}:') + (path or '')) - print(_tab('pipx packages:') + ', '.join(self._pipx_packages) + '\n') + print(_tab('pipx packages:') + ', '.join(self._pipx_packages)) + print() def refresh_pipx(self) -> None: """Get installed pipx packages""" diff --git a/src/dipdup/project.py b/src/dipdup/project.py index 8698b1efc..6725a571a 100644 --- a/src/dipdup/project.py +++ b/src/dipdup/project.py @@ -297,8 +297,9 @@ def _render_templates( for path in project_paths: template_path = path.relative_to(Path(__file__).parent) + relative_path = str(Path(*template_path.parts[2:]))[:-3] - if include and not any(str(path).startswith(i) for i in include): + if include and not any(relative_path.startswith(i) for i in include): continue output_base = get_package_path(answers['package']) if exists else Path(answers['package']) diff --git a/src/dipdup/projects/base/README.md.j2 b/src/dipdup/projects/base/README.md.j2 index 6656b7d9a..abfb57901 100644 --- a/src/dipdup/projects/base/README.md.j2 +++ b/src/dipdup/projects/base/README.md.j2 @@ -8,7 +8,7 @@ This project is based on [DipDup](https://dipdup.io), a framework for building f You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup: -```bash +```shell curl -Lsf https://dipdup.io/install.py | python3 ``` @@ -18,44 +18,46 @@ See the [Installation](https://dipdup.io/docs/installation) page for all options Run the indexer in-memory: -```bash +```shell dipdup run ``` Store data in SQLite database: -```bash +```shell dipdup -c . -c configs/dipdup.sqlite.yml run ``` -Or spawn a docker-compose stack: +Or spawn a Compose stack: -```bash -cp deploy/.env.default deploy/.env -# Edit .env before running -docker-compose -f deploy/compose.yaml up +```shell +cd deploy +cp .env.default .env +# Edit .env file before running +docker-compose up ``` ## Development setup We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment: -```bash +```shell pdm install $(pdm venv activate) ``` -Some tools are included to help you keep the code quality high: black, ruff and mypy. +Some tools are included to help you keep the code quality high: black, ruff and mypy. Use scripts from the `pyproject.toml` to run checks manually or in CI: -```bash +```shell # Format code -pdm fmt +pdm format # Lint code pdm lint # Build Docker image pdm image -``` -Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project. +# Show all available scripts +pdm run --list +``` diff --git a/src/dipdup/projects/base/configs/dipdup.compose.yaml.j2 b/src/dipdup/projects/base/configs/dipdup.compose.yaml.j2 index 98824f36f..ff2b5e1a1 100644 --- a/src/dipdup/projects/base/configs/dipdup.compose.yaml.j2 +++ b/src/dipdup/projects/base/configs/dipdup.compose.yaml.j2 @@ -17,6 +17,4 @@ sentry: environment: ${SENTRY_ENVIRONMENT:-""} prometheus: - host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + host: 0.0.0.0 \ No newline at end of file diff --git a/src/dipdup/projects/base/configs/dipdup.sqlite.yaml.j2 b/src/dipdup/projects/base/configs/dipdup.sqlite.yaml.j2 index 9058bbba9..c47caf14b 100644 --- a/src/dipdup/projects/base/configs/dipdup.sqlite.yaml.j2 +++ b/src/dipdup/projects/base/configs/dipdup.sqlite.yaml.j2 @@ -1,5 +1,3 @@ database: kind: sqlite - path: ${SQLITE_PATH:-/tmp/{{ project.package }}.sqlite} - -logging: ${LOGLEVEL:-INFO} \ No newline at end of file + path: ${SQLITE_PATH:-/tmp/{{ project.package }}.sqlite} \ No newline at end of file diff --git a/src/dipdup/projects/base/configs/dipdup.swarm.yaml.j2 b/src/dipdup/projects/base/configs/dipdup.swarm.yaml.j2 index 60fb7f485..a109cbbc5 100644 --- a/src/dipdup/projects/base/configs/dipdup.swarm.yaml.j2 +++ b/src/dipdup/projects/base/configs/dipdup.swarm.yaml.j2 @@ -18,5 +18,3 @@ sentry: prometheus: host: 0.0.0.0 - -logging: ${LOGLEVEL:-INFO} diff --git a/src/dipdup/projects/base/pyproject.toml.j2 b/src/dipdup/projects/base/pyproject.toml.j2 index 77a96fb5b..91305752e 100644 --- a/src/dipdup/projects/base/pyproject.toml.j2 +++ b/src/dipdup/projects/base/pyproject.toml.j2 @@ -5,12 +5,12 @@ version = "{{ project.version }}" description = "{{ project.description }}" license = { text = "{{ project.license }}" } authors = [ - { name = "{{ project.name }}", email = "{{ project.email }}" } + { name = "{{ project.name }}", email = "{{ project.email }}" }, ] readme = "README.md" requires-python = ">=3.11,<3.12" dependencies = [ - "dipdup>={{ project.dipdup_version }},<{{ project.dipdup_version | int + 1 }}" + "dipdup>={{ project.dipdup_version }},<{{ project.dipdup_version | int + 1 }}", ] [tool.pdm.dev-dependencies] @@ -29,9 +29,9 @@ _mypy = "mypy --no-incremental --exclude {{ project.package }} ." [tool.pdm.scripts.all] help = "Run all checks" -composite = ["fmt", "lint"] +composite = ["format", "lint"] -[tool.pdm.scripts.fmt] +[tool.pdm.scripts.format] help = "Format code with isort and black" composite = ["_isort", "_black"] @@ -55,6 +55,7 @@ skip-string-normalization = true [tool.ruff] line-length = {{ project.line_length }} target-version = 'py311' +extend-select = ["B", "C4", "FA", "G", "PTH", "RUF", "TCH"] [tool.mypy] python_version = "3.11" diff --git a/src/dipdup/projects/demo_uniswap/models/__init__.py.j2 b/src/dipdup/projects/demo_uniswap/models/__init__.py.j2 index 2dac7983f..b6e372c82 100644 --- a/src/dipdup/projects/demo_uniswap/models/__init__.py.j2 +++ b/src/dipdup/projects/demo_uniswap/models/__init__.py.j2 @@ -46,7 +46,7 @@ class Token(CachedModel): # volume in token units volume = fields.DecimalField(decimal_places=18, max_digits=96, default=0) # volume in derived USD - volume_usd = fields.DecimalField(decimal_places=18, max_digits=96, default=0) + volume_usd = fields.DecimalField(decimal_places=18, max_digits=96, default=0, index=True) # volume in USD even on pools with less reliable USD values untracked_volume_usd = fields.DecimalField(decimal_places=18, max_digits=96, default=0) # fees in USD @@ -173,7 +173,7 @@ class Tick(Model): # fee_growth_outside_1x128 = fields.BigIntField() -# NOTE: Cached, but with custom logic; see `{{ project.package }}.utils.position` +# NOTE: Cached, but with custom logic; see `demo_uniswap.utils.position` class Position(Model): id = fields.BigIntField(pk=True) # owner of the NFT @@ -336,7 +336,7 @@ class Swap(Model): # pointer to transaction transaction_hash = fields.TextField() # timestamp of transaction - timestamp = fields.BigIntField() + timestamp = fields.DatetimeField(index=True) # pool swap occured within pool: fields.ForeignKeyRelation[Pool] = fields.ForeignKeyField('models.Pool', related_name='swaps') # allow indexing by tokens @@ -410,4 +410,4 @@ class Flash(Model): # amount token1 paid for flash amount1_paid = fields.DecimalField(decimal_places=18, max_digits=96, default=0) # index within the txn - log_index = fields.BigIntField() \ No newline at end of file + log_index = fields.BigIntField() diff --git a/src/dipdup/projects/demo_uniswap/sql/on_reindex/01_create_mv_token_price.sql.j2 b/src/dipdup/projects/demo_uniswap/sql/on_reindex/01_create_mv_token_price.sql.j2 deleted file mode 100644 index 7b7182b10..000000000 --- a/src/dipdup/projects/demo_uniswap/sql/on_reindex/01_create_mv_token_price.sql.j2 +++ /dev/null @@ -1,32 +0,0 @@ -CREATE MATERIALIZED VIEW - token_price -AS - -( - SELECT - to_timestamp(timestamp) as ts, - token0_id as token_id, - abs(amount_usd/amount0) as price, - amount_usd as volume - FROM swap - WHERE - amount_usd!=0 - AND - amount0!=0 -) UNION ( - SELECT - to_timestamp(timestamp) as ts, - token1_id as token_id, - abs(amount_usd/amount1) as price, - amount_usd as volume - FROM swap - WHERE - amount_usd!=0 - AND - amount1!=0 -) - -WITH DATA; - -CREATE INDEX token_price_ts ON token_price(ts); -CREATE INDEX token_price_token_id ON token_price(token_id); diff --git a/src/dipdup/projects/demo_uniswap/sql/on_reindex/10_create_mv_quotes_1m.sql.j2 b/src/dipdup/projects/demo_uniswap/sql/on_reindex/10_create_mv_quotes_1m.sql.j2 deleted file mode 100644 index 4bb98bae2..000000000 --- a/src/dipdup/projects/demo_uniswap/sql/on_reindex/10_create_mv_quotes_1m.sql.j2 +++ /dev/null @@ -1,25 +0,0 @@ -CREATE MATERIALIZED VIEW - quotes_1m -AS - -SELECT - time_bucket('1 minute'::INTERVAL, ts) AS bucket, - token_id, - candlestick_agg( - ts, - price, - volume - ) as candlestick -FROM token_price - -GROUP BY - bucket, - token_id -ORDER BY - bucket, - token_id - -WITH DATA; - -CREATE INDEX quotes_1m_bucket ON quotes_1m(bucket); -CREATE INDEX quotes_1m_token_id ON quotes_1m(token_id); diff --git a/src/dipdup/projects/demo_uniswap/sql/on_reindex/11_create_mv_quotes_1d.sql.j2 b/src/dipdup/projects/demo_uniswap/sql/on_reindex/11_create_mv_quotes_1d.sql.j2 deleted file mode 100644 index e78aac772..000000000 --- a/src/dipdup/projects/demo_uniswap/sql/on_reindex/11_create_mv_quotes_1d.sql.j2 +++ /dev/null @@ -1,25 +0,0 @@ -CREATE MATERIALIZED VIEW - quotes_1d -AS - -SELECT - time_bucket('1 day'::INTERVAL, ts) AS bucket, - token_id, - candlestick_agg( - ts, - price, - volume - ) as candlestick -FROM token_price - -GROUP BY - bucket, - token_id -ORDER BY - bucket, - token_id - -WITH DATA; - -CREATE INDEX quotes_1d_bucket ON quotes_1d(bucket); -CREATE INDEX quotes_1d_token_id ON quotes_1d(token_id); diff --git a/src/dipdup/projects/demo_uniswap/sql/on_reindex/20_create_ca_quotes_1m.sql.j2 b/src/dipdup/projects/demo_uniswap/sql/on_reindex/20_create_ca_quotes_1m.sql.j2 index 032ee3c68..40b88f340 100644 --- a/src/dipdup/projects/demo_uniswap/sql/on_reindex/20_create_ca_quotes_1m.sql.j2 +++ b/src/dipdup/projects/demo_uniswap/sql/on_reindex/20_create_ca_quotes_1m.sql.j2 @@ -1,25 +1,36 @@ --- CREATE MATERIALIZED VIEW --- candlestick_1m --- WITH (timescaledb.continuous) AS --- --- SELECT --- time_bucket('1 minute'::INTERVAL, to_timestamp(timestamp)) AS bucket, --- token0_id as token_id, --- candlestick_agg( --- to_timestamp(timestamp), --- abs(amount_usd/amount0), --- amount_usd --- ) as candlestick --- FROM swap --- WHERE --- amount_usd!=0 --- AND --- amount0!=0 --- --- GROUP BY --- bucket, --- token0_id --- ORDER BY --- bucket, --- token0_id --- WITH NO DATA; +CREATE MATERIALIZED VIEW + candlestick_1m +WITH (timescaledb.continuous) AS + +SELECT + time_bucket('1 minute'::INTERVAL, timestamp) AS bucket, + token0_id as token_id, + candlestick_agg( + timestamp, + abs(amount_usd/amount0), + amount_usd + ) as candlestick +FROM swap + WHERE + amount_usd!=0 + AND + amount0!=0 + +GROUP BY + bucket, + token0_id +ORDER BY + bucket, + token0_id +WITH NO DATA; + +CREATE INDEX candlestick_1m_bucket ON candlestick_1m(bucket); +CREATE INDEX candlestick_1m_token_id ON candlestick_1m(token_id); + +SELECT add_continuous_aggregate_policy( + 'candlestick_1m', + start_offset => INTERVAL '1 hour', + end_offset => INTERVAL '0 minutes', + schedule_interval => INTERVAL '1 minute', + initial_start := '2018-07-01' +); diff --git a/src/dipdup/projects/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1d.sql.j2 b/src/dipdup/projects/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1d.sql.j2 deleted file mode 100644 index 88fb1704f..000000000 --- a/src/dipdup/projects/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1d.sql.j2 +++ /dev/null @@ -1,25 +0,0 @@ --- CREATE MATERIALIZED VIEW --- candlestick_1d --- WITH (timescaledb.continuous) AS --- --- SELECT --- time_bucket('1 day'::INTERVAL, to_timestamp(timestamp)) AS bucket, --- token0_id as token_id, --- candlestick_agg( --- to_timestamp(timestamp), --- abs(amount_usd/amount0), --- amount_usd --- ) as candlestick --- FROM swap --- WHERE --- amount_usd!=0 --- AND --- amount0!=0 --- --- GROUP BY --- bucket, --- token0_id --- ORDER BY --- bucket, --- token0_id --- WITH NO DATA; diff --git a/src/dipdup/projects/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1h.sql.j2 b/src/dipdup/projects/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1h.sql.j2 new file mode 100644 index 000000000..19916fc29 --- /dev/null +++ b/src/dipdup/projects/demo_uniswap/sql/on_reindex/21_create_ca_quotes_1h.sql.j2 @@ -0,0 +1,36 @@ +CREATE MATERIALIZED VIEW + candlestick_1h +WITH (timescaledb.continuous) AS + +SELECT + time_bucket('1 hour'::INTERVAL, timestamp) AS bucket, + token0_id as token_id, + candlestick_agg( + timestamp, + abs(amount_usd/amount0), + amount_usd + ) as candlestick +FROM swap + WHERE + amount_usd!=0 + AND + amount0!=0 + +GROUP BY + bucket, + token0_id +ORDER BY + bucket, + token0_id +WITH NO DATA; + +CREATE INDEX candlestick_1h_bucket ON candlestick_1h(bucket); +CREATE INDEX candlestick_1h_token_id ON candlestick_1h(token_id); + +SELECT add_continuous_aggregate_policy( + 'candlestick_1h', + start_offset => INTERVAL '2 hour', + end_offset => INTERVAL '0 minutes', + schedule_interval => INTERVAL '1 hour', + initial_start := '2018-07-01' +); diff --git a/src/dipdup/projects/demo_uniswap/sql/on_reindex/22_create_ca_quotes_1d.sql.j2 b/src/dipdup/projects/demo_uniswap/sql/on_reindex/22_create_ca_quotes_1d.sql.j2 new file mode 100644 index 000000000..af98eaca3 --- /dev/null +++ b/src/dipdup/projects/demo_uniswap/sql/on_reindex/22_create_ca_quotes_1d.sql.j2 @@ -0,0 +1,36 @@ +CREATE MATERIALIZED VIEW + candlestick_1d +WITH (timescaledb.continuous) AS + +SELECT + time_bucket('1 day'::INTERVAL, timestamp) AS bucket, + token0_id as token_id, + candlestick_agg( + timestamp, + abs(amount_usd/amount0), + amount_usd + ) as candlestick +FROM swap + WHERE + amount_usd!=0 + AND + amount0!=0 + +GROUP BY + bucket, + token0_id +ORDER BY + bucket, + token0_id +WITH NO DATA; + +CREATE INDEX candlestick_1d_bucket ON candlestick_1d(bucket); +CREATE INDEX candlestick_1d_token_id ON candlestick_1d(token_id); + +SELECT add_continuous_aggregate_policy( + 'candlestick_1d', + start_offset => INTERVAL '2 days', + end_offset => INTERVAL '0 minutes', + schedule_interval => INTERVAL '1 hour', + initial_start := '2018-07-01' +); diff --git a/src/dipdup/templates/replay.yaml.j2 b/src/dipdup/templates/replay.yaml.j2 index 2a93448a5..26fe7cb76 100644 --- a/src/dipdup/templates/replay.yaml.j2 +++ b/src/dipdup/templates/replay.yaml.j2 @@ -1,4 +1,6 @@ -# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay +# To refresh existing project run `dipdup init --base --force` after modifying this file. +# To generate a new project from this replay run `dipdup new --replay `. +# spec_version: 2.0 replay: {% for k, v in project.items() %}{{ ' ' + k + ': ' + v }} diff --git a/src/dipdup/test.py b/src/dipdup/test.py index ec5262d85..2d93e05fb 100644 --- a/src/dipdup/test.py +++ b/src/dipdup/test.py @@ -204,5 +204,6 @@ async def run_in_tmp( stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, ) - await proc.communicate() - assert proc.returncode == 0 + res = await proc.communicate() + if proc.returncode != 0: + raise Exception(f'`dipdup` failed: {res[0].decode()}\n{res[1].decode()}') diff --git a/tests/test_demos.py b/tests/test_demos.py index 7e5043f73..db706a837 100644 --- a/tests/test_demos.py +++ b/tests/test_demos.py @@ -162,14 +162,14 @@ async def assert_run_dao() -> None: test_args = ('config', 'package', 'cmd', 'assert_fn') test_params = ( ('demo_token.yml', 'demo_token', 'run', assert_run_token), - ('demo_token.yml', 'demo_token', 'init', partial(assert_init, 'demo_token')), + ('demo_token.yml', 'demo_token', 'init', None), ('demo_nft_marketplace.yml', 'demo_nft_marketplace', 'run', assert_run_nft_marketplace), - ('demo_nft_marketplace.yml', 'demo_nft_marketplace', 'init', partial(assert_init, 'demo_nft_marketplace')), + ('demo_nft_marketplace.yml', 'demo_nft_marketplace', 'init', None), ('demo_auction.yml', 'demo_auction', 'run', assert_run_auction), - ('demo_auction.yml', 'demo_auction', 'init', partial(assert_init, 'demo_auction')), + ('demo_auction.yml', 'demo_auction', 'init', None), ('demo_token_transfers.yml', 'demo_token_transfers', 'run', partial(assert_run_token_transfers, 4, '-0.01912431')), # FIXME: Why so many token transfer tests? - ('demo_token_transfers.yml', 'demo_token_transfers', 'init', partial(assert_init, 'demo_token_transfers')), + ('demo_token_transfers.yml', 'demo_token_transfers', 'init', None), ( 'demo_token_transfers_2.yml', 'demo_token_transfers', @@ -184,24 +184,31 @@ async def assert_run_dao() -> None: partial(assert_run_token_transfers, 2, '-0.02302128'), ), ('demo_token_balances.yml', 'demo_token_balances', 'run', assert_run_balances), - ('demo_token_balances.yml', 'demo_token_balances', 'init', partial(assert_init, 'demo_token_balances')), + ('demo_token_balances.yml', 'demo_token_balances', 'init', None), ('demo_big_maps.yml', 'demo_big_maps', 'run', assert_run_big_maps), - ('demo_big_maps.yml', 'demo_big_maps', 'init', partial(assert_init, 'demo_big_maps')), + ('demo_big_maps.yml', 'demo_big_maps', 'init', None), ('demo_domains.yml', 'demo_domains', 'run', assert_run_domains), - ('demo_domains.yml', 'demo_domains', 'init', partial(assert_init, 'demo_domains')), + ('demo_domains.yml', 'demo_domains', 'init', None), ('demo_dex.yml', 'demo_dex', 'run', assert_run_dex), - ('demo_dex.yml', 'demo_dex', 'init', partial(assert_init, 'demo_dex')), + ('demo_dex.yml', 'demo_dex', 'init', None), ('demo_dao.yml', 'demo_dao', 'run', assert_run_dao), - ('demo_dao.yml', 'demo_dao', 'init', partial(assert_init, 'demo_dao')), + ('demo_dao.yml', 'demo_dao', 'init', None), ('demo_factories.yml', 'demo_factories', 'run', assert_run_factories), - ('demo_factories.yml', 'demo_factories', 'init', partial(assert_init, 'demo_factories')), + ('demo_factories.yml', 'demo_factories', 'init', None), ('demo_events.yml', 'demo_events', 'run', assert_run_events), - ('demo_events.yml', 'demo_events', 'init', partial(assert_init, 'demo_events')), + ('demo_events.yml', 'demo_events', 'init', None), ('demo_raw.yml', 'demo_raw', 'run', assert_run_raw), - ('demo_raw.yml', 'demo_raw', 'init', partial(assert_init, 'demo_raw')), + ('demo_raw.yml', 'demo_raw', 'init', None), ('demo_evm_events.yml', 'demo_evm_events', 'run', assert_run_evm_events), - ('demo_evm_events.yml', 'demo_evm_events', 'init', partial(assert_init, 'demo_evm_events')), + ('demo_evm_events.yml', 'demo_evm_events', 'init', None), ('demo_evm_events_node.yml', 'demo_evm_events', 'run', assert_run_evm_events), + # NOTE: Smoke tests for small tools. + ('demo_dex.yml', 'demo_dex', ('config', 'env', '--compose', '--internal'), None), + ('demo_dex.yml', 'demo_dex', ('config', 'export', '--full'), None), + ('demo_dex.yml', 'demo_dex', ('package', 'tree'), None), + ('demo_dex.yml', 'demo_dex', ('report', 'ls'), None), + ('demo_dex.yml', 'demo_dex', ('self', 'env'), None), + ('demo_dex.yml', 'demo_dex', ('schema', 'export'), None), ) @@ -209,8 +216,8 @@ async def assert_run_dao() -> None: async def test_run_init( config: str, package: str, - cmd: str, - assert_fn: Callable[[], Awaitable[None]], + cmd: str | tuple[str, ...], + assert_fn: Callable[[], Awaitable[None]] | None, ) -> None: config_path = TEST_CONFIGS / config env_config_path = TEST_CONFIGS / 'test_sqlite.yaml' @@ -223,12 +230,14 @@ async def test_run_init( exists=cmd != 'init', ), ) - await run_in_tmp(tmp_package_path, env, cmd) + await run_in_tmp(tmp_package_path, env, *((cmd,) if isinstance(cmd, str) else cmd)) + if not assert_fn: + return + await stack.enter_async_context( tortoise_wrapper( f'sqlite://{tmp_package_path}/db.sqlite3', f'{package}.models', ) ) - await assert_fn()