Make configuration changes to Dokku easier to test

[StevenMaude]

In #4499, to test a Dokku configuration change, I went through the process of running job-server with Dokku in Codespaces. It took me a little while to get that working, but it does work.

To have more confidence in such changes in future, it would be useful to do one or more of the following:

• document how to get a minimal setup job-server-in-Dokku setup running from scratch, for local or Codespaces use
• add a dev container configuration to run job-server with Dokku
• have a CI job start job-server within Dokku, and check for warnings/errors in the logs, to automatically test our configuration end-to-end

[StevenMaude]

I do have some notes on Dokku already that I can check and edit, and then add here.

[StevenMaude]

This is roughly how I configured Dokku in a dev container in Codespaces. It may not bear much resemblance to our real setup (for example, PostgreSQL is installed via a Dokku plugin here, for convenience when I was trying to get things working as quickly as possible).

Not everything here is a perfect guide, and not everything may be absolutely correct. There should be enough for someone following these instructions to get close to a working setup.

Starting Dokku

1. Add a dev container configuration that installs Docker-in-Docker to the job-server repository, as a file named .devcontainer/devcontainer.json

   {
       "name": "job-server",
       "features": {
           "ghcr.io/devcontainers/features/docker-in-docker:2": {}
       }
   }

2. Start a codespace.
3. Set up Dokku with docker-compose.yml based on the Dokku documentation. Note that not all the configured ports may be necessary here (possibly just 8199:8000, but it's the configuration that I had working):

   version: "3.8"
   
   services:
     dokku:
       image: dokku/dokku:0.32.4
       container_name: dokku
       network_mode: bridge
       ports:
         - "3022:22"
         - "8080:80"
         - "8443:443"
         - "8000:999"
         - "8199:8000"
       volumes:
         - "/var/lib/dokku:/mnt/dokku"
         - "/var/run/docker.sock:/var/run/docker.sock"
       environment:
         DOKKU_HOSTNAME: dokku.me
         DOKKU_HOST_ROOT: /var/lib/dokku/home/dokku
         DOKKU_LIB_HOST_ROOT: /var/lib/dokku/var/lib/dokku
       restart: unless-stopped

4. Run docker-compose up

Install PostgreSQL inside the Dokku container

1. Enter the container via docker exec -it dokku
2. dokku apps:create job-server
3. sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git
4. dokku postgres:create jobserver --image-version "13"
5. dokku postgres:link jobserver job-server (got an error because the Docker image doesn't exist yet)

Configure SSH outside of the Dokku container

1. Set ~/.ssh/config in host:

   Host dokku.me
     HostName 127.0.0.1
     Port 3022
   

2. ssh [email protected] then works
3. ssh-keygen -t ed25519 -C "jobserver@codespaces"
4. Copy the .pub public key to /var/lib/dokku

Configure SSH inside the Dokku container

1. dokku ssh-keys:add MYKEY /var/lib/dokku/<publickey.pub>
2. You can then delete the key from /var/lib/dokku

Configure Git inside the Dokku container

⚠️ Not probably necessary as I was actually testing with a built Docker image; see below.

1. dokku git:set job-server deploy-branch main

Configure Git outside of the Dokku container

⚠️ Not probably necessary as I was actually testing with a built Docker image; see below.

1. git remote add dokku [email protected]:job-server
2. git push dokku <yourbranch>:main

Build the Docker image

Outside the Dokku container

1. Copy the local copy of the job-server repository to /var/lib/dokku/job-server.
2. Copy dotenv-sample in job-server directory to .env
3. You could set these environment variables to bogus values for the purposes of only testing Dokku, and not job-server.

Inside the Dokku container

1. Go to /var/lib/dokku/job-server/docker
2. Run the build commands from the justfile; the export commands and the docker compose build. You want to build the prod image.
3. docker image ls should show a job-server image.
4. Run dokku git:from-image job-server job-server:latest