Release/2.0 acacia (#33)

* Fix/13 improve auth security (#14) * Support running a single test or multiple. Add JWT dependency. * Implement JWT authorization. Use a new consent message for signature verification. * Update permission tests to support JWT auth. * Implement JWT tests * Add README instructions for enabling JWT auth in CouchDb * Fix incorrect merge issues. * Add support for generating and verifying refresh token * More refresh token unit tests. Add access token support. Db refactor. * Fix permission unit tests with new refactor. * Update server endpoints to use new auth. Create server integration tests for auth. * Hash database name server side before creating * Verify context name when verifying refresh token. Don't generate access token if refresh token is invalid. * Confirm expected user is admin of database before deleting it. * Ensure database names begin with "v" * Return an access token and host when requesting a refresh token. * Support deleteDatabase endpoint * Expand unit tests * Split controller into auth and user. * Fix broken test * Fix broken delete database calls * Remove JWT test no longer required * Support invalidating device ID * Invalidating devices now working correctly with tests * Implement garbage collection * Fix building DSN for public credentials * Add CouchDB configuration note around basic auth * Add more docs * Fix edge case issues identified during testing * Fix yarn dependencies * Fix example .env files and remove duplicate. * Describe how authentication works * Rename to authorization * Add missing account-node dev dependency * Fix minor issues with tests. Better config docs in README. * Resolve feedback for further review. * Fix missed merge issue * Support running single test * Add support for saving user databases and getting the info as an authorized user. More HTTP status improvements and better checking for valid request params. * Add missing require for PouchDB * Cleanup didsToUsernames * Destructure some vars * Add cors requirement to README * Add isTokenValid endpoint to verify a refresh token is still valid and obtain the expiry. * Support the new decentralized did-client implementation (#32) * adding acacia testnet * Update sample.env * Fix server tests. Add support for deleting all databases for a user. (#35) * Update dependencies to acacia rc2 * Update account node dependency for tests * Fix issue with documenting how to configure HMAC key in couchdb * Add better comments * Update config to use testnet storage node * Add docker and docker-compose (#27) * Remove body-parser as deprecated * Add Dockerfile * Replace require with import for the js work without babel * Add docker-compose * Update node to latest 14 * Add gh workflow to build and publish docker image * Rearrange dockerfile to improve image size * Uncomment dockerhub * Add docker info and single node config to readme * Update to latest verida dependencies. Remove module so tests run. * Remove lambda support * Details on how to do the Docker build and push it. * Update REARME.md: enable_cors = true * update Docker build command * Changes to make deployment work. Yarn build must happen in Docker container * Support signing all responses using the storage node VDA private key. (#42) * Add support for /user/usage details (#43) * add logging to make debugging Docker image somewhat possible * fix logging message * update README to correct authentication info. Add logging to dbInit exception handler * logging for failure of /auth/public; clean up pakage.json * update docs on HASH_KEY not being needed * Feature/37 implement vip 3 (#44) * Add scaffolding for new did storage support. Refactor routes. * Initial progress on create and get endpoints * Updates working. Get all versions working. * Deletion now working. All tests passing. * Support field validation for create, including proof verification * Tests destroy database before starting. Fix create and update tests to work with new verification. * Fix missing versionId field on index * Upgrade to latest ethers version * Update error messages. Wrap versions response in a versions parameter. * Fix handling of all versions being wrapped in a versions parameter. * Ensure signature verification for update, delete * Implement storage limits and status information for users and the system (#45) * Remove redundant hash key Co-authored-by: Chris <[email protected]> * Update README.md Remove Docker compose (moved to infrastructure) Remove redundant value in windows config. * Force lowercase username when generating usernames * Add comments about running tests * Include permissions in database info * Update dependencies to next release candidate * delay start of docker image until $DB_HOST reolves * Add support for a NODE_URI * typo * remove checking for network now SN supports different endpoint * Move signature tests to the end to avoid issues with testing servers with invalid signatures. * Support internal and external DB hostnames * Fix hostname generation. * Fix build external host * Add console errors and warnings to help with debugging * Support looking up a DID document and caching it * Include public key in system status * Feature/49 server replication (#50) * Rename applicationName to contextName. Add replicator permissions. * Add generateHash utility method * Refactor so context databases are in their own database that will be replicated, instead of all stored in a single database. Other minor fixes and test improvements. * Implement untested auth/replicationCreds * Implementation complete, untested. * Add comments for next steps * First pass at replication test * Fix missing endpoint variable * Bug fixes and enhanced logging * Bug fixes from testing * Bug fixes * Support generating correct CouchURI for server * Bug fixes * Fix syntax error * First tests passing * Fix clearing databases. Add warnings. * Ensure replicator user has did context role * Bug fixes * Try basic auth header * Update replication document creation * Create new replicater user for replication instead of using admin * Bug fix replication * Update working tests * All tests written so far, pass! * Remove commented test code * Code cleanup * Code cleanup. Replication test cleanup. Verify timestamp when fetching replication creds. * Add comments in .env file Co-authored-by: Chris <[email protected]> * Feature/49 server replication (#51) * Rename applicationName to contextName. Add replicator permissions. * Add generateHash utility method * Refactor so context databases are in their own database that will be replicated, instead of all stored in a single database. Other minor fixes and test improvements. * Implement untested auth/replicationCreds * Implementation complete, untested. * Add comments for next steps * First pass at replication test * Fix missing endpoint variable * Bug fixes and enhanced logging * Bug fixes from testing * Bug fixes * Support generating correct CouchURI for server * Bug fixes * Fix syntax error * First tests passing * Fix clearing databases. Add warnings. * Ensure replicator user has did context role * Bug fixes * Try basic auth header * Update replication document creation * Create new replicater user for replication instead of using admin * Bug fix replication * Update working tests * All tests written so far, pass! * Remove commented test code * Code cleanup * Code cleanup. Replication test cleanup. Verify timestamp when fetching replication creds. * Replication tests all passing with remote 3x servers * Remove console output Co-authored-by: Chris <[email protected]> * Feature/52 user db list replicated (#54) * Ensure user databases are replicated * Add checkReplication tests confirming databases are re-created * Fix user database issues. Add debug logging. * Bug fix database creation via checkReplication() * Basic replication unit tests passing * Remove excess debug output * Fix trailing slash issue * Fix trailing slash issues * Adding cache checks for DID documents * Fix database list database incorrect config * Handle missing DID document * Throw error if unknown check databse error * Improve service endpoint checks on checkReplication() * Ensure user database list database has replicator local role * Fix issue with user database list database not being recreated. Minor refactor. * Handle garbage collection where token already deleted * Handle user database list database document conflict * Fix caching of couch instance not respecting external v internal * Add debug output to nodes about cache status * Add error message when unable to resolve DID * Add debug logging * Unify storage node checks and hashes to use the hostname only * Make endpointUri more resiliant * Upgrade encryption utils to latest * Support recoverying from replication credential errors * Add debug logging * Add storage node HTTP timeouts. Add more debugging. * Fix replicater creds to have replicaterUsername has id * Fix replicater username to only use hostname * Fix incorrect username being loaded * Fix replication to start working upon creation of new databases, only update password if it changes, re-use existing password where possible. * Check replication credentials after all databases have been checked. * Refactor how endpoint security checks occur. Cleanup console output. * Improve debug output * More debug output for testing * Remove redundant record variable * Add delete failed replication meethod * Fix incorrect delete replication code * More debugging * More debug output * Update order of code * Change order so that failed replication entries are removed first, so they are auto created later. * Add more debug output for testing= * Force update of remote credentials if they are definitely invalid * Add debug output * Add more debug output * Fix issue with fetchReplicaterCredential setting the wrong key for the credential cache * Fix incorrect username * Restructure how replication fails are fixed * Add more debug output, fix replication status id parameter * More debug output * More debugging * Fix sample.env comment * Reduce debug output * d * Refactor replication and improve debug output * Handle unable to fetch DID error * Add checkreplication debug header * Lots of bug fixes and refactor * Bug fix incorrect instance reference * Force cache refresh if service endpoint not found * Handle missing did document * set winning_revs_only: true * Remove winning_revs_only * Remove permissions check * Add debug logging * More debug output * Debug to find error * Support ignore cache option when fetching DID document * Correctly handle empty DIDs * Update CHANGELOG in preparation for v2.0.0 release
verida · Jan 14, 2023 · 0e50140 · 0e50140
1 parent 30e0a93
commit 0e50140
Show file tree

Hide file tree

Showing 42 changed files with 7,808 additions and 4,481 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -0,0 +1,11 @@
+node_modules
+
+.git
+.github
+
+dist
+.env*
+docker-compose.yml
+Dockerfile
+*.md
+claudia.*
diff --git a/.github/workflows/build-docker.yml b/.github/workflows/build-docker.yml
@@ -0,0 +1,94 @@
+name: build-docker-image
+
+on:
+  push:
+    branches:
+      - 'main'
+    tags:
+      - 'v*'
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+        # qemu and buildx are for multi-platform images and caching
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      # set up local caching
+      - name: Cache Docker layers
+        uses: actions/cache@v3
+        with:
+          path: /tmp/.buildx-cache
+          key: ${{ runner.os }}-buildx-${{ github.sha }}
+          restore-keys: |
+            ${{ runner.os }}-buildx-
+
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v4
+        with:
+          images: |
+            verida/storage-node
+            ghcr.io/verida/storage-node
+          tags: |
+            type=ref,event=branch
+            type=ref,event=pr
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+            type=sha
+
+      # get the correct AW role to push image to ECR
+      - name: Configure AWS credentials from Testnet account
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          role-duration-seconds: 1500
+          role-to-assume: arn:aws:iam::737954963756:role/github-testnet-deploy-role
+          aws-region: us-east-1
+
+      # login to push to DockerHub
+      - name: Login to DockerHub
+        if: github.event_name != 'pull_request'
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      # login to push to GHCR
+      - name: Login to GHCR
+        if: github.event_name != 'pull_request'
+        uses: docker/login-action@v2
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build and push
+        uses: docker/build-push-action@v3
+        with:
+          platforms: linux/amd64 # ,linux/arm64,linux/arm/v7
+          context: .
+          push: ${{ github.event_name != 'pull_request' }}
+          cache-from: type=local,src=/tmp/.buildx-cache
+          cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max
+          tags: ${{ steps.meta.output.tags }}
+          labels: ${{ steps.meta.output.labels }}
+
+      # Temp fix
+      # https://github.com/docker/build-push-action/issues/252
+      # https://github.com/moby/buildkit/issues/1896
+      - name: Move cache
+        run: |
+          rm -rf /tmp/.buildx-cache
+          mv /tmp/.buildx-cache-new /tmp/.buildx-cache
diff --git a/.gitignore b/.gitignore
@@ -4,6 +4,7 @@ dist
 .DS_Store
 .vscode
 curl_tests
+dbdata
 
 .env
-.env*
+.env*
diff --git a/.nvmrc b/.nvmrc
@@ -1 +1 @@
-14.17.1
+14.20.0
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,17 @@
+2023-01-13 (2.0.0)
+--------------------
+
+- Support blockchain DID registry
+- Support refresh and access tokens
+- Support server side replication, including cehcks and recovery
+- Support device disconnections
+- Support maintaining database of context specific databases to enable monitoring of usage
+- Support database deletion
+- Support create, update, retreive and delete DID documents
+- Support `/lookup/did` endpoint to provide a DID document caching service
+- Remove lambda support
+- Support docker containers
+
 
 2021-09-17 (1.2.0)
 --------------------

diff --git a/Dockerfile b/Dockerfile
@@ -0,0 +1,33 @@
+FROM node:14.20.0-slim as node
+
+ENV NODE_ENV=production
+EXPOSE 5000
+
+RUN mkdir /storage-node && chown -R node:node /storage-node
+WORKDIR /storage-node
+
+# this ensures we fix simlinks for npx and yarn
+RUN corepack disable && corepack enable
+
+RUN apt-get update \
+    && apt-get -qq install -y --no-install-recommends \
+    git ca-certificates dnsutils \
+    && rm -rf /var/lib/apt/lists/*
+
+# new stage
+FROM node AS source
+
+USER node
+# Copy the current directory
+COPY --chown=node:node . .
+
+# we need devDependencies to build. Setting --production=false ignores NODE_ENV (deliberatly)
+RUN yarn install --production=false --frozen-lockfile
+# we have to build inside the container in case we are on a Mac building a linux image
+RUN yarn build
+
+
+### prod stage
+# Note: use --init option when running the container to have better signal forwarding
+FROM source as prod
+CMD ["node", "--trace-warnings", "./dist/server.js"]
diff --git a/README.md b/README.md
@@ -10,6 +10,23 @@ Key features:
 - Adding a second layer of security by managing per-database ACL validation rules
 - Providing applications with user's database connection strings
 
+## How Authorization Works
+
+This is the login flow:
+
+1. The Verida Account makes a request to the storage node API to obtain an auth JWT to be signed (`/auth/generateAuthJwt`). This prevents replay attacks.
+2. The Verida Account signs a consent message using their private key. This consent message proves the user wants to unlock a specific application context
+3. The Verida Account submits the signed authorization request (`/auth/authenticate`). Assuming the signed AuthJWT is valid, the storage node returns a refresh token and an access token
+4. The Verida Account can then use the access token to either; 1) make storage node requests (ie: create database) or 2) directly access CouchDB as an authenticated user (using `Bearer` token auth)
+5. When the access token expires, the Verida Account can use the refresh token to request a new access token (`/auth/connect`)
+6. If a refresh token is close to expiry, the Verida Account can use the active refresh token to obtain a new refresh token (`/auth/regenerateRefreshToken`)
+
+When a Verida Account authenticates, it can designate an `authenticate` requst to be linked to a particular device by specifying the `deviceId` in the request.
+
+This allows a specific device to be linked to a refresh token. A call to `/auth/invalidateDeviceId` can be used to invalidate any refresh tokens linked to the specified `deviceId`. This allows the Verida Vault to remotely log out an application that previously logged in.
+
+Note: This only invalidates the refresh token. The access token will remain valid until it expires. It's for this reason that access tokens are configured to have a short expiry (5 minutes by default). CouchDB does not support manually invalidating access tokens, so we have to take this timeout approach to invalidation.
+
 ## Usage
 
 ```bash
@@ -30,7 +47,6 @@ This server is running on the Verida Testnet and is accessible by any applicatio
 
 A `sample.env` is included. Copy this to `.env` and update the configuration:
 
-- `HASH_KEY`: A unique hash key that is used as entropy when generating an alpha numeric username from a DID. Set this to a unique value when first running the server. DO NOT change this key once the server is up and running as you will end up with a mismatch of usernames. If you run multiple servers in front of a cluster of CouchDB instances, all servers must use the same `HASH_KEY`.
 - `DID_SERVER_URL`: URL of a Verida DID Server endpoint.
 - `DB_PROTOCOL`: Protocol to use when connecting to CouchDB (`http` or `https`).
 - `DB_USER`: Username of CouchDB Admin (has access to create users and databases).
@@ -40,12 +56,15 @@ A `sample.env` is included. Copy this to `.env` and update the configuration:
 - `DB_REJECT_UNAUTHORIZED_SSL`: Boolean indicating if unauthorized SSL certificates should be rejected (`true` or `false`). Defaults to `false` for development testing. Must be `true` for production environments otherwise SSL certificates won't be verified.
 - `DB_PUBLIC_USER`: Alphanumeric string for a public database user. These credentials can be requested by anyone and provide access to all databases where the permissions have been set to `public`.
 - `DB_PUBLIC_PASS`: Alphanumeric string for a public database password.
+- `ACCESS_TOKEN_EXPIRY`: Number of seconds before an access token expires. The protocol will use the refresh token to obtain a new access token. CouchDB does not support a way to force the expiry of an issued token, so the access token expiry should always be set to 5 minutes (300)
+- `REFRESH_TOKEN_EXPIRY`: Number of seconds before a refresh token expires. Users will be forced to re-login once this time limit is reached. This should be set to 7 days (604800).
+- `ACCESS_JWT_SIGN_PK`: The access token private key. The base64 version of this must be specified in the CouchDB configuration under `jwt_keys/hmac:_default`
+- `REFRESH_JWT_SIGN_PK`: The refresh token private key
 
 ### Setting up environment variables on Windows
 
 * On a powershell execute the following ( replica of `.env` )
 ```bash
-$env:HASH_KEY="this_is_not_prod_hash_key"
 $env:DID_SERVER_URL="https://dids.testnet.verida.io:5001"
 $env:DID_CACHE_DURATION=3600
 $env:DB_PROTOCOL="http"
@@ -63,27 +82,83 @@ $env:DB_PUBLIC_PASS="784c2n780c9cn0789"
 - CORS must be enabled so that database requests can come from any domain name
 - A valid user must be enforced for security reasons
 
+[Ensure `{chttpd_auth, jwt_authentication_handler}` is added to the list of the active `chttpd/authentication_handlers`](https://docs.couchdb.org/en/stable/api/server/authn.html?highlight=jwt#jwt-authentication)
+
+
+
 ```
-[httpd]
-WWW-Authenticate = Basic realm="administrator"
+[couchdb]
+single_node=true
+
+[chttpd]
+authentication_handlers = {chttpd_auth, jwt_authentication_handler}, {chttpd_auth, cookie_authentication_handler}, {chttpd_auth, default_authentication_handler}
 enable_cors = true
 
 [chttpd_auth]
 require_valid_user = true
 
+[jwt_auth]
+required_claims = exp
+
+[jwt_keys]
+hmac:_default = <base64 secret key>
+
 [cors]
 origins = *
 credentials = true
 methods = GET, PUT, POST, HEAD, DELETE
 headers = accept, authorization, content-type, origin, referer, x-csrf-token
 ```
 
-## Lambda deployment
+The `hmac:_default` key is a base64 encoded representation of the access token JWT private key
+
+## Generating JWT key
+
+Note: A secret key (string) suitable for `jwt_keys` can be base64 encoded with the following:
+
+```
+const secretKey = 'secretKey'
+const encodedKey = Buffer.from(secretKey).toString('base64')
+```
+
+This can be tested via curl:
+
+```
+curl -H "Host: localhost:5984" \
+ -H "accept: application/json, text/plain, */*" \
+ -H "authorization: Bearer <bearer_token>" \
+  "http://localhost:5984/_session"
+```
+
+Where:
+
+- `bearer_token` - A bearer token generated via the `test/jwt` unit test
+- `localhost` - Replace this with the hostname of the server being tested
+
+## Docker
+
+You can spin up storage node API on your machine with Docker:
+```shell
+docker run --init --env-file=.env verida/storage-node:latest
+```
+
+### Deploying a new Docker Image to Docker Hub
+
+Note that this uses the experimental `buildx` command to build both AMD64 (Intel/AMD servers) and ARM64 (Mac) images.
+
+* Login (details in BitWarden)
+```
+docker buildx build --platform linux/amd64,linux/arm64 --push -t verida/storage-node:latest .
+```
+
+
+## Tests
 
-We use [Claudia.js](https://claudiajs.com/) to turn our Express app into an Express-on-Lambda app.
+Run tests with `yarn run tests`
 
-Before doing any Lambda deployments you **MUST** translate your `.env` file (or one for production) to JSON as `.env.prod.json`.
-See the [Claudia Docs for information](https://claudiajs.com/news/2016/11/24/claudia-2.2.0-environment-vars.html).
+Note: The tests in `server.js` require the server to be running locally. The other tests operate fine without the server running.
 
-Verida staff can see the [internal Verida repo]( https://github.com/verida/infrastructure/blob/develop/storage_node.md) for docs on this. 
+Common issues when running tests:
 
+1. `Bad key`: The key in CouchDB configuration for `jwt_keys/hmac:_default` is not a valid Base64 encoded key
+2. `HMAC error`: The key in CouchDB configuration for `jwt_keys/hmac:_default` does not match `ACCESS_JWT_SIGN_PK` in `.env`
diff --git a/claudia.au1.testnet.json b/claudia.au1.testnet.json
diff --git a/claudia.dev.json b/claudia.dev.json
diff --git a/claudia.sg1.testnet.json b/claudia.sg1.testnet.json
diff --git a/claudia.use1.testnet.json b/claudia.use1.testnet.json
diff --git a/lambda.js b/lambda.js