Documentation on phasing out the repository model in the release notes

Fix other references to the app module that has been removed
Update the other relevant documentation on seeding the KATalogus database, which is no longer needed

Signed-off-by: Donny Peeters <[email protected]>

boefjes/boefjes/seed.py

@@ -0,0 +1,5 @@
+import logging
+
+logger = logging.getLogger(__name__)
+
+logger.warning("This module has been phased out in v1.16.0 and will be removed in v1.17.0")

boefjes/debian/kat-boefjes.kat-katalogus.service

@@ -13,7 +13,7 @@ ExecStart=/opt/venvs/kat-boefjes/bin/python -m gunicorn \
           --access-logfile - \
           -c /etc/kat/katalogus.gunicorn.conf.py \
           -k uvicorn.workers.UvicornWorker \
-          boefjes.katalogus.api:app
+          boefjes.katalogus.api.root:app
 Restart=on-failure
 RestartSec=3s
 KillMode=mixed

boefjes/packaging/deb/data/usr/bin/update-katalogus-db

@@ -6,6 +6,3 @@ source /etc/kat/boefjes.conf
 cd /opt/venvs/kat-boefjes/lib/python*/site-packages
 
 /opt/venvs/kat-boefjes/bin/python -m alembic --config boefjes/alembic.ini upgrade head
-
-cd boefjes
-/opt/venvs/kat-boefjes/bin/python -m boefjes.seed

docs/source/installation_and_deployment/containers.md

@@ -43,21 +43,7 @@ docker compose --env-file .env-prod -f docker-compose.release-example.yml up -d
 
 The container image run the necessary database migration commands in the
 entrypoint if DATABASE_MIGRATION is set. You manually need to run setup commands
-in the katalogus and rocky containers to initialize everything. In the katalogus
-container we need to create an organisation, we can do this by running the
-following in the katalogus container:
-
-```shell
-python3 -m boefjes.seed
-```
-
-With docker compose you would run this as:
-
-```shell
-docker compose --env-file .env-prod -f docker-compose.release-example.yml exec katalogus python3 -m boefjes.seed
-```
-
-In the rocky container we first need to import the OOI database seed:
+in the rocky container to initialize everything. In the rocky container we first need to import the OOI database seed:
 
 ```shell
 python3 manage.py loaddata OOI_database_seed.json

docs/source/release_notes/1.16.rst

@@ -0,0 +1,87 @@
+============================================
+[DRAFT for: v1.15.1...67f18e3] OpenKAT 1.16
+============================================
+
+This release includes some big optimizations in the new reporting functionality
+that was introduced in 1.14. Measurements show that generating a report on 100
+objects is approximately 20 times faster. The reports in general also got a lot
+of improvements and bugfixes.
+
+New Features
+============
+
+* Add xtdb-cli tool to Octopoes
+* Update several plugins: Wappalizer, dns-records, ssl-certificates, pdio_subfinder and remove the many-ports-open boefje/normalizer
+* Add backup scripts
+* Introduce importing/exporting capabilities in xtdb-multinode-tool
+* More Octopoes Query support for complex path queries
+* Introduce support for running custom built OCI images using only a boefje definition (boefje.json), applied to nmap.
+* Improvements of the design, plugin overview and Report titles.
+* Improvements of several Reports in terms of performance, styling, OOI selection and configuration.
+* More documentation on: Reports, the new OCI image functionality and architecture, IPv6 support in Docker and Octopoes Models.
+
+Bug fixes
+=========
+
+* Fix OOI Add/Edit form
+* Fix version handling when no version is present.
+* Fix aggregate plugin overview table
+* Fix task api status code response for malformed id in the scheduler
+* Fix select all OOIs
+* Fix openssl boefje being stuck on port 80
+* Fix pdf alignment
+* Fix critical vulnerability counter
+* Fix in System Specific Reports
+* fix schema errors on empty / missing schemas
+* Fix improve error handling
+* Fix missing cipher csv in Debian package
+* Fix Update nuclei
+* Fix and improve running boefjes/normalizer
+* Fix the KATalogus plugin API limit
+
+Upgrading
+=========
+
+It is no longer needed to seed the KATalogus database using `python -m boefjes.seed` on upgrades.
+This is because v1.16.0 phases out the `repository` database model in the KATalogus.
+The migration could potentially not be backward compatible for each install,
+So please read the following carefully before triggering an upgrade.
+
+Checking the KATalogus Migration
+================================
+If you are using OpenKAT as a regular user and never called APIs or tweaked the database manually,
+you can move forward with the normal instructions of upgrading :ref:`Debian packages<Upgrading Debian>`
+or upgrading :ref:`containers <Upgrading_Containers>`.
+
+If there is a chance you added entries to the seeded model,
+or you notice the KATalogus being down after the upgrade,
+please check your `katalogus` database to see if any records are returned for the following query:
+
+.. code-block:: sql
+
+  SELECT * from plugin_state join repository r on repository_pk = r.pk where r.id != 'LOCAL'
+
+If this raises an exception saying the `repository` table does not exist,
+the migration was applied successfully and this is not the issue.
+If this returns no records but the logs say:
+"Cannot perform migration: remove plugin_states that refer to nonlocal repositories first.",
+please contact us.
+
+If this returns one or more records, either delete these if these were not created intentionally,
+or contact us if you were in fact using a custom plugin repository.
+We will help migrating your setup towards the new custom built OCI image solution,
+after which the custom plugin repository is no longer needed.
+
+Alternatively, to check if there are repositories other than the 'LOCAL' repository,
+look at the result of the following KATalogus endpoint for your organisations:
+`/v1/organisations/{your_organisation_id}/repositories`.
+Then follow the steps above using the HTTP endpoints to delete the other entries if needed, or contact us.
+
+After these steps, again the normal instructions for upgrading :ref:`Debian packages<Upgrading Debian>`
+or upgrading :ref:`containers <Upgrading_Containers>` should be followed.
+
+Full Changelog
+==============
+
+The full changelog can be found on `Github
+<https://github.com/minvws/nl-kat-coordination/releases/tag/v1.16.0>`_.

mula/.ci/docker-compose.yml

@@ -75,7 +75,7 @@ services:
       target: dev
       args:
         ENVIRONMENT: dev
-    command: uvicorn boefjes.katalogus.api:app --host 0.0.0.0
+    command: uvicorn boefjes.katalogus.api.root:app --host 0.0.0.0
     ports:
       - "127.0.0.1:8003:8000"
     env_file: