Add balances index (#871)

Co-authored-by: Vladimir Bobrikov <[email protected]>

CHANGELOG.md

@@ -4,6 +4,16 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog], and this project adheres to [Semantic Versioning].
 
+## Unreleased
+
+### Added
+
+- tezos.tzkt.token_balances: Added new index.
+
+### Fixed
+
+- tezos.tzkt.token_transfers: Fixed token_id handler in token transfers index.
+
 ## [7.0.2] - 2023-10-10
 
 ### Added

docs/1.getting-started/7.indexes.md

@@ -18,6 +18,7 @@ Multiple indexes are available for different workloads. Every index is linked to
 | [tezos.tzkt.operations](../2.indexes/5.tezos_tzkt_operations.md)                       | Tezos          | TzKT       | typed operations            |
 | [tezos.tzkt.operations_unfiltered](../2.indexes/6.tezos_tzkt_operations_unfiltered.md) | Tezos          | TzKT       | untyped operations          |
 | [tezos.tzkt.token_transfers](../2.indexes/7.tezos_tzkt_token_transfers.md)             | Tezos          | TzKT       | TZIP-12/16 token transfers  |
+| [tezos.tzkt.token_balances](../2.indexes/8.tezos_tzkt_token_balances.md)               | Tezos          | TzKT       | TZIP-12/16 token balances   |
 
 Indexes can join multiple contracts considered as a single application. Also, contracts can be used by multiple indexes of any kind, but make sure that they are independent of each other and that indexed data don't overlap.
 

docs/2.indexes/8.tezos_tzkt_token_balances.md

@@ -0,0 +1,19 @@
+---
+title: "Token balances"
+description: "This index allows indexing token balances of contracts compatible with FA1.2 or FA2 standards."
+network: "tezos"
+---
+
+# `tezos.tzkt.token_balances` index
+
+This index allows indexing token balances of contracts compatible with [FA1.2](https://gitlab.com/tzip/tzip/-/blob/master/proposals/tzip-7/README.md) or [FA2](https://gitlab.com/tzip/tzip/-/blob/master/proposals/tzip-12/tzip-12.md) standards. You can either index transfers and cumulatively calculate balances or use this index type to fetch the latest balance information directly.
+
+```yaml [dipdup.yaml]
+{{ #include ../src/demo_token_balances/dipdup.yaml }}
+```
+
+Callback receives `TzktTokenBalanceData` model that optionally contains the owner, token, and balance values
+
+```python
+{{ #include ../src/demo_token_balances/handlers/on_balance_update.py }}
+```

src/demo_token_balances/.dockerignore

@@ -0,0 +1,22 @@
+# Ignore all
+*
+
+# Add metadata and build files
+!demo_token_balances
+!pyproject.toml
+!pdm.lock
+!README.md
+
+# Add Python code
+!**/*.py
+**/.*_cache
+**/__pycache__
+
+# Add configs and scripts (but not env!)
+!**/*.graphql
+!**/*.json
+!**/*.sql
+!**/*.yaml
+!**/*.yml
+!**/*.j2
+!**/.keep

src/demo_token_balances/.gitignore

@@ -0,0 +1,29 @@
+# Ignore all
+*
+!*/
+
+# Add metadata and build files
+!demo_token_balances
+!.gitignore
+!.dockerignore
+!py.typed
+!**/Dockerfile
+!**/Makefile
+!**/pyproject.toml
+!**/pdm.lock
+!**/README.md
+!**/.keep
+
+# Add Python code
+!**/*.py
+**/.*_cache
+**/__pycache__
+
+# Add configs and scripts (but not env!)
+!**/*.graphql
+!**/*.json
+!**/*.sql
+!**/*.yaml
+!**/*.yml
+!**/*.j2
+!**/*.env.default

src/demo_token_balances/README.md

@@ -0,0 +1,61 @@
+# demo_token_balances
+
+TzBTC FA1.2 token balances
+
+## Installation
+
+This project is based on [DipDup](https://dipdup.io), a framework for building featureful dapps.
+
+You need a Linux/macOS system with Python 3.11 installed. Use our installer for easy setup:
+
+```bash
+curl -Lsf https://dipdup.io/install.py | python3
+```
+
+See the [Installation](https://dipdup.io/docs/installation) page for all options.
+
+## Usage
+
+Run the indexer in-memory:
+
+```bash
+dipdup run
+```
+
+Store data in SQLite database:
+
+```bash
+dipdup -c . -c configs/dipdup.sqlite.yml run
+```
+
+Or spawn a docker-compose stack:
+
+```bash
+cp deploy/.env.default deploy/.env
+# Edit .env before running
+docker-compose -f deploy/compose.yaml up
+```
+
+## Development setup
+
+We recommend [PDM](https://pdm.fming.dev/latest/) for managing Python projects. To set up the development environment:
+
+```bash
+pdm install
+$(pdm venv activate)
+```
+
+Some tools are included to help you keep the code quality high: black, ruff and mypy.
+
+```bash
+# Format code
+pdm fmt
+
+# Lint code
+pdm lint
+
+# Build Docker image
+pdm image
+```
+
+Inspect the `pyproject.toml` file. It contains all the dependencies and tools used in the project.

src/demo_token_balances/configs/dipdup.compose.yaml

@@ -0,0 +1,22 @@
+database:
+  kind: postgres
+  host: ${POSTGRES_HOST:-db}
+  port: 5432
+  user: ${POSTGRES_USER:-dipdup}
+  password: ${POSTGRES_PASSWORD}
+  database: ${POSTGRES_DB:-dipdup}
+
+hasura:
+  url: http://${HASURA_HOST:-hasura}:8080
+  admin_secret: ${HASURA_SECRET}
+  allow_aggregations: true
+  camel_case: true
+
+sentry:
+  dsn: ${SENTRY_DSN:-""}
+  environment: ${SENTRY_ENVIRONMENT:-""}
+
+prometheus:
+  host: 0.0.0.0
+
+logging: ${LOGLEVEL:-INFO}

src/demo_token_balances/configs/dipdup.sqlite.yaml

@@ -0,0 +1,5 @@
+database:
+  kind: sqlite
+  path: ${SQLITE_PATH:-/tmp/demo_token_balances.sqlite}
+
+logging: ${LOGLEVEL:-INFO}

src/demo_token_balances/configs/dipdup.swarm.yaml

@@ -0,0 +1,22 @@
+database:
+  kind: postgres
+  host: ${POSTGRES_HOST:-demo_token_balances_db}
+  port: 5432
+  user: ${POSTGRES_USER:-dipdup}
+  password: ${POSTGRES_PASSWORD}
+  database: ${POSTGRES_DB:-dipdup}
+
+hasura:
+  url: http://${HASURA_HOST:-demo_token_balances_hasura}:8080
+  admin_secret: ${HASURA_SECRET}
+  allow_aggregations: false
+  camel_case: true
+
+sentry:
+  dsn: ${SENTRY_DSN:-""}
+  environment: ${SENTRY_ENVIRONMENT:-""}
+
+prometheus:
+  host: 0.0.0.0
+
+logging: ${LOGLEVEL:-INFO}

src/demo_token_balances/configs/replay.yaml

@@ -0,0 +1,15 @@
+# Run `dipdup new --replay configs/replay.yaml` to generate new project from this replay
+spec_version: 2.0
+replay:
+  dipdup_version: 7
+  template: demo_token_balances
+  package: demo_token_balances
+  version: 0.0.1
+  description: TzBTC FA1.2 token balances
+  license: MIT
+  name: John Doe
+  email: [email protected]
+  postgres_image: postgres:15
+  postgres_data_path: /var/lib/postgresql/data
+  hasura_image: hasura/graphql-engine:latest
+  line_length: 120

src/demo_token_balances/deploy/.env.default

@@ -0,0 +1,12 @@
+# This env file was generated automatically by DipDup. Do not edit it!
+# Create a copy with .env extension, fill it with your values and run DipDup with `--env-file` option.
+#
+HASURA_HOST=hasura
+HASURA_SECRET=
+LOGLEVEL=INFO
+POSTGRES_DB=dipdup
+POSTGRES_HOST=db
+POSTGRES_PASSWORD=
+POSTGRES_USER=dipdup
+SENTRY_DSN=""
+SENTRY_ENVIRONMENT=""

src/demo_token_balances/deploy/Dockerfile

@@ -0,0 +1,9 @@
+FROM dipdup/dipdup:7
+# FROM ghcr.io/dipdup-io/dipdup:7
+# FROM ghcr.io/dipdup-io/dipdup:next
+
+# COPY --chown=dipdup pyproject.toml README.md .
+# RUN pip install .
+
+COPY --chown=dipdup . demo_token_balances
+WORKDIR demo_token_balances

src/demo_token_balances/deploy/compose.sqlite.yaml

@@ -0,0 +1,19 @@
+version: "3.8"
+name: demo_token_balances
+
+services:
+  dipdup:
+    build:
+      context: ..
+      dockerfile: deploy/Dockerfile
+    command: ["-c", "dipdup.yaml", "-c", "configs/dipdup.sqlite.yaml", "run"]
+    restart: always
+    env_file: .env
+    ports:
+      - 46339
+      - 9000
+    volumes:
+      - sqlite:${SQLITE_PATH:-/tmp/demo_token_balances.sqlite}
+
+volumes:
+  sqlite:

src/demo_token_balances/deploy/compose.swarm.yaml

@@ -0,0 +1,92 @@
+version: "3.8"
+name: demo_token_balances
+
+services:
+  dipdup:
+    image: ${IMAGE:-ghcr.io/dipdup-io/dipdup}:${TAG:-7}
+    depends_on:
+      - db
+      - hasura
+    command: ["-c", "dipdup.yaml", "-c", "configs/dipdup.swarm.yaml", "run"]
+    env_file: .env
+    networks:
+      - internal
+      - prometheus-private
+    deploy:
+      mode: replicated
+      replicas: ${INDEXER_ENABLED:-1}
+      labels:
+        - prometheus-job=${SERVICE}
+        - prometheus-port=8000
+      placement: &placement
+        constraints:
+          - node.labels.${SERVICE} == true
+    logging: &logging
+      driver: "json-file"
+      options:
+        max-size: "10m"
+        max-file: "10"
+        tag: "\{\{.Name\}\}.\{\{.ImageID\}\}"
+
+  db:
+    image: postgres:15
+    volumes:
+      - db:/var/lib/postgresql/data
+    env_file: .env
+    environment:
+      - POSTGRES_USER=dipdup
+      - POSTGRES_DB=dipdup
+      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+    networks:
+      - internal
+    deploy:
+      mode: replicated
+      replicas: 1
+      placement: *placement
+    logging: *logging
+
+  hasura:
+    image: hasura/graphql-engine:latest
+    depends_on:
+      - db
+    environment:
+      - HASURA_GRAPHQL_DATABASE_URL=postgres://dipdup:${POSTGRES_PASSWORD}@demo_token_balances_db:5432/dipdup
+      - HASURA_GRAPHQL_ADMIN_SECRET=${HASURA_SECRET}
+      - HASURA_GRAPHQL_ENABLE_CONSOLE=true
+      - HASURA_GRAPHQL_DEV_MODE=false
+      - HASURA_GRAPHQL_LOG_LEVEL=warn
+      - HASURA_GRAPHQL_ENABLE_TELEMETRY=false
+      - HASURA_GRAPHQL_UNAUTHORIZED_ROLE=user
+      - HASURA_GRAPHQL_STRINGIFY_NUMERIC_TYPES=true
+    networks:
+      - internal
+      - traefik-public
+    deploy:
+      mode: replicated
+      replicas: 1
+      labels:
+        - traefik.enable=true
+        - traefik.http.services.${SERVICE}.loadbalancer.server.port=8080
+        - "traefik.http.routers.${SERVICE}.rule=Host(`${HOST}`) && (PathPrefix(`/v1/graphql`) || PathPrefix(`/api/rest`))"
+        - traefik.http.routers.${SERVICE}.entrypoints=http,${INGRESS:-ingress}
+        - "traefik.http.routers.${SERVICE}-console.rule=Host(`${SERVICE}.${SWARM_ROOT_DOMAIN}`)"
+        - traefik.http.routers.${SERVICE}-console.entrypoints=https
+        - traefik.http.middlewares.${SERVICE}-console.headers.customrequestheaders.X-Hasura-Admin-Secret=${HASURA_SECRET}
+        - traefik.http.routers.${SERVICE}-console.middlewares=authelia@docker,${SERVICE}-console
+      placement: *placement
+    logging: *logging
+
+volumes:
+  db:
+
+networks:
+  internal:
+  traefik-public:
+    external: true
+  prometheus-private:
+    external: true