Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue #3474] Doc updates for auth #3491

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
+302 −283
Open

[Issue #3474] Doc updates for auth #3491

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
082e038
Update dev docs for auth setup
acouch Jan 10, 2025
a3bd9f9
Update development.md
acouch Jan 13, 2025
d102e4c
Update README.md
acouch Jan 13, 2025
e993966
Remove empty file
acouch Jan 13, 2025
dd2d464
Update README.md
acouch Jan 13, 2025
84828d9
Update README.md for formatting
acouch Jan 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 73 additions & 58 deletions documentation/api/development.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,87 +4,56 @@ This application is dockerized. Take a look at [Dockerfile](../../api/Dockerfile

A very simple [docker-compose.yml](../../docker-compose.yml) has been included to support local development and deployment.

## Prerequisites
Several components like tests, linting, and scripts can be run either inside of the Docker container, or outside on your machine.

1. Install the version of Python specified in [pyproject.toml](../../api/pyproject.toml)
[pyenv](https://github.com/pyenv/pyenv#installation) is one popular option for installing Python,
or [asdf](https://asdf-vm.com/).
- If using pyenv run `pyenv local <version>` to ensure that version will be used in subsequent steps
2. Ensure that `python -V` and `python3 -V` are picking up that version.
- If not, run `pyenv init -` and/or restart your shell to ensure it was run automatically
3. After installing and activating the right version of Python, install
[poetry](https://python-poetry.org/docs/#installation) and follow the instructions to add poetry to your path if necessary.

```bash
curl -sSL https://install.python-poetry.org | python3 -
```

4. You'll also need [Docker Desktop](https://www.docker.com/products/docker-desktop/)
**Running in Docker is the default**, but on some machines like the M1 Mac, running natively may be desirable for performance reasons.

## **Note:** All the following commands should be run from the `/api` directory.
## Docker

## Database setup: Run Migrations/Seeds
This section covers development using Docker. There are a number of Docker commands included in the [Makefile](../../api/Makefile) which are helpful for local development. Run `make help` for a list of commands.

1. If you haven't done local development before you'll need to execute the migrations and seed the DB with data using the steps in [database-local-usage.md](database/database-local-usage.md)
### Setup

## OpenSearch setup
Run `make init && make run logs` to start the local containers. The application will be available at `http://localhost:8080/docs`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like directing to the docs site is a little confusing here. Would just saying "http://localhost:8080" make more sense?


1. Run `make init-opensearch` setup the OpenSearch Container
2. Run `make populate-search-opportunities` to push data previously seeded in the DB into the search index
This stands up the following services:

If your DB or OpenSearch end up in an odd place, you can reset all the persistent storage using `make volume-recreate`
* Flask API (http://localhost:8080)
* Postgres database
* OpenSearch node
* OpenSearchdashboard (http://localhost:5601)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* OpenSearchdashboard (http://localhost:5601)
* OpenSearch Dashboard (http://localhost:5601)

* localstack
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be nice to have a brief note on what localstack is used for in this context. I had to look it up but seems like s3?

* mock oauth server (http://localhost:5001)

## Run the application
### Seed data

1. Make sure you have [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed & running.
2. Run `make setup-local` to install dependencies
3. Run `make init start` to build the image and start the container.
4. Navigate to `localhost:8080/docs` to access the Swagger UI.
5. Run `make run-logs` to see the logs of the running API container
6. Run `make stop` when you are done to delete the container.
Run `make db-seed-local && populate-search-opportunities` to create local data in the database and make it available in the API.

## Some Useful Commands
### API Authenticaion

`make test` will run all of the tests. Additional arguments can be passed to this command which will be passed to pytest like so: `make test args="tests/api/route -v"` which would run all tests in the route folder with verbosity increased. See the [Pytest Docs](https://docs.pytest.org/en/7.1.x/reference/reference.html#command-line-flags) for more details on CLI flags you can set.

`make clean-volumes` will spin down the docker containers + delete the volumes.

`make volume-recreate` Deletes the volumes and then re-initializes the persistant portions of the stack. This can be useful to reset your DB, or fix any bad states your local environment may have gotten into.

See the [Makefile](../../api/Makefile) for a full list of commands you can run.

The `make console` command initializes a Python REPL environment pre-configured with database connectivity. This allows developers to perform database queries, utilize factories for data generation, and interact with the application's models directly.

- Writing a query: `dbs.query(Opportunity).all()`
- Saving some factory generated data to the db: `f.OpportunityFactory.create()`

## Docker and Native Development

Several components like tests, linting, and scripts can be run either inside of the Docker container, or outside on your native machine.
Running in Docker is the default, but on some machines like the M1 Mac, running natively may be desirable for performance reasons.

You can switch which way many of these components are run by setting the `PY_RUN_APPROACH` env variable in your terminal.

- `export PY_RUN_APPROACH=local` will run these components natively
- `export PY_RUN_APPROACH=docker` will run these within Docker
This API uses a very simple [ApiKey authentication approach](https://apiflask.com/authentication/#use-external-authentication-library) which requires the caller to provide a static key. This is specified with the `API_AUTH_TOKEN` environment variable.

Note that even with the native mode, many components like the DB and API will only ever run in Docker, and you should always make sure that any implementations work within docker.
### User Authentication

Running in the native/local approach may require additional packages to be installed on your machine to get working.
Run `make setup-env-override-file` to create the `override.env` file which will include the necessary JWT keys for running the user.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn't super clear to me what running the user means here. What about something like running user authentication within the app?


## Environment Variables
### Environment Variables

Most configuration options are managed by environment variables.

Environment variables for local development are stored in the [local.env](../../api/local.env) file. This file is automatically loaded when running. If running within Docker, this file is specified as an `env_file` in the [docker-compose](../../docker-compose.yml) file, and loaded [by a script](../../api/src/util/local.py) automatically when running most other components outside the container.

Any environment variables specified directly in the [docker-compose](../../docker-compose.yml) file will take precedent over those specified in the [local.env](../../api/local.env) file.

## Authentication
### Troubleshooting

This API uses a very simple [ApiKey authentication approach](https://apiflask.com/authentication/#use-external-authentication-library) which requires the caller to provide a static key. This is specified with the `API_AUTH_TOKEN` environment variable.
Errors in standing up the API can originate from an out of date container, database syncronization, or other issues with previously created services. Helper functions are available to rebuild:

* **db-check-migrations** - check if migrations are out of sync
* **volume-recreate** - delete all existing volumes and data
* **remake-backend** - delete all data (`volume-recreate`) and load data (`db-seed-local` and `populate-search-opportunities`)

## VSCode Remote Attach Container Debugging
### VSCode Remote Attach Container Debugging

The API can be run in debug mode that allows for remote attach debugging (currently only supported from VSCode) to the container.

Expand All @@ -105,6 +74,52 @@ The API can be run in debug mode that allows for remote attach debugging (curren

- You should now be able to hit set breakpoints throughout the API

## Local (non-Docker)

Run `export PY_RUN_APPROACH=local` to run API and test functions locally when running commands in the Makefile. For example, `make test` or `make format` will run outside of Docker.

**Note:** even with the native mode, many components like the DB and API will only ever run in Docker, and you should always make sure that any implementations work within docker.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**Note:** even with the native mode, many components like the DB and API will only ever run in Docker, and you should always make sure that any implementations work within docker.
**Note:** even with the native mode, many components like the DB and API will only ever run in Docker, and you should always make sure that any implementations work within Docker.


Running in the native/local approach may require additional packages to be installed on your machine to get working.

### Prerequisites

1. Install the version of Python specified in [pyproject.toml](../../api/pyproject.toml)
[pyenv](https://github.com/pyenv/pyenv#installation) is one popular option for installing Python,
or [asdf](https://asdf-vm.com/).
- If using pyenv run `pyenv local <version>` to ensure that version will be used in subsequent steps
2. Ensure that `python -V` and `python3 -V` are picking up that version.
- If not, run `pyenv init -` and/or restart your shell to ensure it was run automatically
3. After installing and activating the right version of Python, install
[poetry](https://python-poetry.org/docs/#installation) and follow the instructions to add poetry to your path if necessary.

```bash
curl -sSL https://install.python-poetry.org | python3 -
```

4. You'll also need [Docker Desktop](https://www.docker.com/products/docker-desktop/)

**Note:** All the following commands should be run from the `/api` directory.

### Database setup: Run Migrations/Seeds

If you haven't done local development before you'll need to execute the migrations and seed the DB with data using the steps in [database-local-usage.md](database/database-local-usage.md)

### Services

Individual services can be run through Docker, which can be useful in concert with non-Docker application development:

* **OpenSearch**
* Run `make init-opensearch` setup the OpenSearch Container
* Run `make populate-search-opportunities` to push data previously seeded in the DB into the search index

If your DB or OpenSearch end up in an odd place, you can reset all the persistent storage using `make volume-recreate`.

* **Localstack (local s3)**
* Run `make init-localstack`
* **Mock OAuth server**
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't seem to be mentioned in any other documentation. Is it worth mentioning in this file what this is for and how / when to use it, maybe in the "User Authentication" section?

* Run `make init-mock-oath2`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Run `make init-mock-oath2`
* Run `make init-mock-oauth2`


## Next steps

Now that you're up and running, read the [application docs](../../api/README.md) to familiarize yourself with the application.
Loading
Loading