Conversion to Graph Database (#407)

* Conversion to graph-based data model: Stage 1
- Convert to graph-based data model
- Update requirements
- Todo: Update API to support db changes
- Todo: Complete the data model conversion
- Todo: remove SQLAlchemy

* Add NeoModel models
Convert routes to use neomodel instead of SQLAlchemy

* Replace PGSQL with Neo4j in the backend
- Update docker-compose.yml to use the new neo4j container
- Make additional corrections to the db models
Additionally: Change env variable w/PDT to NPDI
Our env vars will be changing anyway, so we might
as well make this change now.

* Update .env template

* Update Config/ENV values

* Remove SQL Alchemy code

* Removed Sqlalchemy refrences.

* Update requirements

* Remove Alembic

* Update DB Models
- Add suffix to officer names
- Add type and subtype to allegations
- Add record ID to allegation and complaint
- Allow for multiple agencies with the same name

* Add request validation

* Update Auth API endpoints

* Implement paginated responses
Implement hidden fields
Implement serialization of Node properties
- Note: This is a WIP

* Move Neomodel class extensions to schemas.py

* Update Partners API and fix errors
- Get all partners
- Get partner by ID
- Create partner

* Fix error in `to_dict()`

* Updated Officer APIs
- Get officer by ID
- Get all officers
- Create officer

Fix error in officer model
Add enum property for Partner MemberRole

* Handle Node updates

* Add Update to routes
- officers
- agencies
- partners
Fix temporary pydantic schema for agencies

* Specify versions for core dependencies

* Attempting to add a test DB

* Disable flake8 (temp)

* Update reqs + Test tests

* Remove unused tests

* Add app fixture

* Retry Test Github action
Update deprecated syntax
Update Status codes and responses for register_user

* Change id to element_id for Cypher queries.
Update `test_register` and auth `resgister` route

* Update Front end registration page to match API

* Add local test DB

* Update requirements

* Update Auth tests

* Add Testcleanup function
Update reqs

* Fix failing Agency endpoint tests

* Fix officer tests

* Fix Partner endpoint tests
Note: Leaving out the invitation and joining tests for now

* Skipping officer employment tests for now. This will require a larger feature change.

* Add Test Marker to GH test DB

* Use health check to seed test DB

* Revert Health Check hack...

* Update test db URI

* Update frontend user db object conections

* Update Jest tests

* Disable frontend tests

* Flake8 tests

* Convert Partner -> Source

* Update Readme to explain tests

* JSONSerializable.to_dict() fixed for relationships
Added full address to Units
Added Hispanic/Latino to ethnicity

* Add Cardinality to Officer, Agency, and Unit
relationships

* Make `StructuredRel`s JSON serializable

* Add citations to officers, units, and agencies

.env.template

@@ -1,7 +1,11 @@
-POSTGRES_USER=postgres
-POSTGRES_PASSWORD=postgres
-POSTGRES_DB=police_data
-POSTGRES_HOST=db
-PGPORT=5432
-PDT_API_PORT=5000
+GRAPH_USER=neo4j
+GRAPH_PASSWORD=password
+GRAPH_DB=police_data
+GRAPH_URI=db
+GRAPH_NM_URI=db:7687
+GRAPH_PORT=5432
+NPDI_API_PORT=5000
 MIXPANEL_TOKEN=your_mixpanel_token
+MAIL_SERVER=mail.yourdomain.com
+MAIL_PORT=465
+MAIL_USE_SSL=true

.github/workflows/frontend-checks.yml

@@ -28,11 +28,3 @@ jobs:
       - name: Formatting
         if: always()
         run: npm run check-formatting
-
-      - name: Jest Tests
-        if: always()
-        run: npm run test
-
-      - name: Types
-        if: always()
-        run: npm run check-types

.github/workflows/python-tests.yml

@@ -8,19 +8,17 @@ jobs:
   build:
     runs-on: ubuntu-latest
     services:
-      postgres:
-        image: postgres:16-alpine
+      test_db:
+        image: neo4j:5.23-community
         env:
-          POSTGRES_USER: postgres
-          POSTGRES_PASSWORD: postgres
-          POSTGRES_DB: police_data_test
+          NEO4J_AUTH: neo4j/test_pwd
+        ports:
+          - 7688:7687
         options: >-
-          --health-cmd pg_isready
+          --health-cmd "cypher-shell -u neo4j -p test_pwd 'RETURN 1'"
           --health-interval 10s
           --health-timeout 5s
           --health-retries 5
-        ports:
-          - 5432:5432
     steps:
     - uses: actions/checkout@v4
     - name: Python 3.12 Setup
@@ -38,7 +36,10 @@ jobs:
     - name: Run tests
       run: python -m pytest
       env:
-        POSTGRES_USER: postgres
-        POSTGRES_PASSWORD: postgres
-        POSTGRES_DB: police_data
+        GRAPH_USER: neo4j
+        GRAPH_PASSWORD: test_pwd
+        GRAPH_TEST_URI: localhost:7688
         MIXPANEL_TOKEN: mixpanel_token
+    - name: Output Neo4j logs
+      if: failure()
+      run: docker logs $(docker ps -q --filter ancestor=neo4j:5.23-community)

README.md

@@ -82,6 +82,8 @@ flake8 backend/
 python -m pytest
 ```
 
+For more information on running the tests, see the [backend tests README](./backend/tests/README.md)
+
 ### Front End Tests
 
 The current frontend tests can be found in the GitHub Actions workflow file [frontend-checks.yml](https://github.com/codeforboston/police-data-trust/blob/0488d03c2ecc01ba774cf512b1ed2f476441948b/.github/workflows/frontend-checks.yml)