diff --git a/boefjes/boefjes/seed.py b/boefjes/boefjes/seed.py new file mode 100644 index 00000000000..a849fd89dc4 --- /dev/null +++ b/boefjes/boefjes/seed.py @@ -0,0 +1,5 @@ +import logging + +logger = logging.getLogger(__name__) + +logger.warning("This module has been phased out in v1.16.0 and will be removed in v1.17.0") diff --git a/boefjes/debian/kat-boefjes.kat-katalogus.service b/boefjes/debian/kat-boefjes.kat-katalogus.service index 4c31061f945..6991822b4c6 100644 --- a/boefjes/debian/kat-boefjes.kat-katalogus.service +++ b/boefjes/debian/kat-boefjes.kat-katalogus.service @@ -13,7 +13,7 @@ ExecStart=/opt/venvs/kat-boefjes/bin/python -m gunicorn \ --access-logfile - \ -c /etc/kat/katalogus.gunicorn.conf.py \ -k uvicorn.workers.UvicornWorker \ - boefjes.katalogus.api:app + boefjes.katalogus.api.root:app Restart=on-failure RestartSec=3s KillMode=mixed diff --git a/boefjes/packaging/deb/data/usr/bin/update-katalogus-db b/boefjes/packaging/deb/data/usr/bin/update-katalogus-db index 6d5407043eb..0d82d3d1b8e 100755 --- a/boefjes/packaging/deb/data/usr/bin/update-katalogus-db +++ b/boefjes/packaging/deb/data/usr/bin/update-katalogus-db @@ -6,6 +6,3 @@ source /etc/kat/boefjes.conf cd /opt/venvs/kat-boefjes/lib/python*/site-packages /opt/venvs/kat-boefjes/bin/python -m alembic --config boefjes/alembic.ini upgrade head - -cd boefjes -/opt/venvs/kat-boefjes/bin/python -m boefjes.seed diff --git a/docs/source/installation_and_deployment/containers.md b/docs/source/installation_and_deployment/containers.md index c91f4cf243c..31691a99d7c 100644 --- a/docs/source/installation_and_deployment/containers.md +++ b/docs/source/installation_and_deployment/containers.md @@ -43,21 +43,7 @@ docker compose --env-file .env-prod -f docker-compose.release-example.yml up -d The container image run the necessary database migration commands in the entrypoint if DATABASE_MIGRATION is set. You manually need to run setup commands -in the katalogus and rocky containers to initialize everything. In the katalogus -container we need to create an organisation, we can do this by running the -following in the katalogus container: - -```shell -python3 -m boefjes.seed -``` - -With docker compose you would run this as: - -```shell -docker compose --env-file .env-prod -f docker-compose.release-example.yml exec katalogus python3 -m boefjes.seed -``` - -In the rocky container we first need to import the OOI database seed: +in the rocky container to initialize everything. In the rocky container we first need to import the OOI database seed: ```shell python3 manage.py loaddata OOI_database_seed.json diff --git a/docs/source/release_notes/1.16.rst b/docs/source/release_notes/1.16.rst new file mode 100644 index 00000000000..49931f5cb38 --- /dev/null +++ b/docs/source/release_notes/1.16.rst @@ -0,0 +1,87 @@ +============================================ +[DRAFT for: v1.15.1...67f18e3] OpenKAT 1.16 +============================================ + +This release includes some big optimizations in the new reporting functionality +that was introduced in 1.14. Measurements show that generating a report on 100 +objects is approximately 20 times faster. The reports in general also got a lot +of improvements and bugfixes. + +New Features +============ + +* Add xtdb-cli tool to Octopoes +* Update several plugins: Wappalizer, dns-records, ssl-certificates, pdio_subfinder and remove the many-ports-open boefje/normalizer +* Add backup scripts +* Introduce importing/exporting capabilities in xtdb-multinode-tool +* More Octopoes Query support for complex path queries +* Introduce support for running custom built OCI images using only a boefje definition (boefje.json), applied to nmap. +* Improvements of the design, plugin overview and Report titles. +* Improvements of several Reports in terms of performance, styling, OOI selection and configuration. +* More documentation on: Reports, the new OCI image functionality and architecture, IPv6 support in Docker and Octopoes Models. + +Bug fixes +========= + +* Fix OOI Add/Edit form +* Fix version handling when no version is present. +* Fix aggregate plugin overview table +* Fix task api status code response for malformed id in the scheduler +* Fix select all OOIs +* Fix openssl boefje being stuck on port 80 +* Fix pdf alignment +* Fix critical vulnerability counter +* Fix in System Specific Reports +* fix schema errors on empty / missing schemas +* Fix improve error handling +* Fix missing cipher csv in Debian package +* Fix Update nuclei +* Fix and improve running boefjes/normalizer +* Fix the KATalogus plugin API limit + +Upgrading +========= + +It is no longer needed to seed the KATalogus database using `python -m boefjes.seed` on upgrades. +This is because v1.16.0 phases out the `repository` database model in the KATalogus. +The migration could potentially not be backward compatible for each install, +So please read the following carefully before triggering an upgrade. + +Checking the KATalogus Migration +================================ +If you are using OpenKAT as a regular user and never called APIs or tweaked the database manually, +you can move forward with the normal instructions of upgrading :ref:`Debian packages` +or upgrading :ref:`containers `. + +If there is a chance you added entries to the seeded model, +or you notice the KATalogus being down after the upgrade, +please check your `katalogus` database to see if any records are returned for the following query: + +.. code-block:: sql + + SELECT * from plugin_state join repository r on repository_pk = r.pk where r.id != 'LOCAL' + +If this raises an exception saying the `repository` table does not exist, +the migration was applied successfully and this is not the issue. +If this returns no records but the logs say: +"Cannot perform migration: remove plugin_states that refer to nonlocal repositories first.", +please contact us. + +If this returns one or more records, either delete these if these were not created intentionally, +or contact us if you were in fact using a custom plugin repository. +We will help migrating your setup towards the new custom built OCI image solution, +after which the custom plugin repository is no longer needed. + +Alternatively, to check if there are repositories other than the 'LOCAL' repository, +look at the result of the following KATalogus endpoint for your organisations: +`/v1/organisations/{your_organisation_id}/repositories`. +Then follow the steps above using the HTTP endpoints to delete the other entries if needed, or contact us. + +After these steps, again the normal instructions for upgrading :ref:`Debian packages` +or upgrading :ref:`containers ` should be followed. + +Full Changelog +============== + +The full changelog can be found on `Github +`_. diff --git a/mula/.ci/docker-compose.yml b/mula/.ci/docker-compose.yml index 13a194dc448..a9410d47654 100644 --- a/mula/.ci/docker-compose.yml +++ b/mula/.ci/docker-compose.yml @@ -75,7 +75,7 @@ services: target: dev args: ENVIRONMENT: dev - command: uvicorn boefjes.katalogus.api:app --host 0.0.0.0 + command: uvicorn boefjes.katalogus.api.root:app --host 0.0.0.0 ports: - "127.0.0.1:8003:8000" env_file: