From a30bc74facf9679e4e367dc637fd7359704603a8 Mon Sep 17 00:00:00 2001 From: Max H <10329648+8ear@users.noreply.github.com> Date: Mon, 17 Dec 2018 20:10:29 +0100 Subject: [PATCH] Release Candidate 1.0.1 (#48) ## Changelog for Release Candidate 1.0.1 This release represents the first production release of MISP-Dockerized in version 1.0.0 and an bug fix for container tagging. ### About MISP dockerized is a project designed to provide an easy-to-use and easy-to-install 'out of the box' MISP instance that includes everything you need to run MISP with minimal host-side requirements. MISP dockerized uses MISP (Open Source Threat Intelligence Platform - https://github.com/MISP/MISP), which is maintained and developed by the MISP project team (https://www.misp-project.org/). MISP-dockerized provides the following features: * 3-steps (in 5 minute or less) ready MISP instance * Built in MISP modules * Built in mail transport via postfix * Prehardened TLS configuration * Default http to https redirect * Easy-to-use script based backup and restore function * Prebuilt management commands to ease maintenance efforts * Periodically built Docker containers * Automated test framework to test new MISP releases ----------------------------- ### Update Informations This release includes the new MISP server container version 2.4.99 In addition, we have changed the following: - the proxy container version to the latest version 1.X - the robot container version to the latest version 2.X - the misp-module container version to the latest version 1.X ### General Changes - We have changed our complete directories. Now you have access to all unsupported and old version of MISP-dockerized. - There is a new NEW_FRESH_INSTAL.sh script - We have added a new UPGRADE.sh script - We have added a new UPDATE.sh script - Added the new MISP-Server Container with MISP Version 2.4.99 (#39) ### Detailed Changes #### Main Repository ##### Detailed Changes, Fixes & Improvements - Deactivated the Ansible script for database and server, because since version 2.4.97 the misp-server container does not require Ansible anymore - Changed misp-robot from 1.0.3-ubuntu to 2.x-debian - Changed misp-server from 2.4.94 to 2.4.99 - Removed unused container content in docker-compose.yml - Added docker-compose.override.yml to add the user configuration - Removed misp-postfix container, because postfix is now directly installed in the misp-server - Changed volume paths from misp-server (done since version 0.3.4) #### Container `misp-dockerized-server` ##### Update Informations Features of the release: - Added a new MISP-Server Container for MISP release 2.4.99 on base of 2.4.97. (closes #39) For more information please have a look at the official changelog at https://github.com/MISP/MISP/releases/tag/v2.4.99 - Changed to a new gitlab-ci job structure for all misp-server container - Built in mail transport via postfix - Built in redis - Built in rsyslog-ng - Built in MariaDB ##### Detailed Changes, Fixes & Improvements - The following base changes were made in addition to the new `misp-server version 2.4.97`: - The base image is now **debian stretch** instead of ubuntu 18.04 - Postfix is now directly installed - the postfix container is no longer necessary - Added a cron shell script that fetches and caches all enabled feeds and pull all remote server events - Changed from syslog-ng to rsyslog, because in debian the syslog-ng doesn't support wildcard configuration for logfiles. - **The external requirement for our Ansible playbook is no longer necessary for this container** - Gitlab-CI Strucutre Included statements in the Gitlab CI main file allow us to improve and simplify our Gitlab CI main file. - Added an echo tag to know from which entrypoint script a message comes. - Added on the misp-server 2.4.97-2.4.99 functions to create self-signed certificate, private key and DH params file #### Container `misp-dockerized-robot` ##### Update Informations Features of the release: - Built in Docker - Built in Ansible - Built in Test framework - Docker Container image is based now on debian:stretch-slim In the past, we decided to use a three-digit version number for small and large changes for this container repository. This has proved unnecessary as the main repository is the central repository for new releases and all changes are already tracked here. We have therefore decided to change the 3-digit version number in all minor repositories to a 2-digit version number. For example: 1.0.1 -> 1.1 The second digit is changed any time we make minor changes. The first digit changes whenever we make major changes, e.g. a new base image. ##### Detailed Changes, Fixes & Improvements - Changed from the base image debian:stable-slim to debian:stretch-slim - Remove unused robot versions: - 1.0.2-debian - 1.0.3-debian - 1.0.4-ubuntu - All folders have been renamed, e.g. from 1.0.0.0-alpin to 1.0-alpin. - We have made working with gitlab-ci easier with new extensions and includes. - We have also reduced the readme as much as possible, as the documentation is now available centrally in our own documentation repository `https://dcso.github.io/MISP-dockerized-docs/`. #### Container `misp-dockerized-proxy` ##### Update Informations Features of the release: - Prehardened TLS configuration - Default http to https redirect - Built in User defined IP restriction - Added the ability to create self-signed certificates. In the past, we decided to use a three-digit version number for small and large changes for this container repository. This has proved unnecessary as the main repository is the central repository for new releases and all changes are already tracked here. We have therefore decided to change the 3-digit version number in all minor repositories to a 2-digit version number. For example: 1.0.1 -> 1.1 The second digit is changed any time we make minor changes. The first digit changes whenever we make major changes, e.g. a new base image. ##### Detailed Changes, Fixes & Improvements - Added an own entrypoint script to create user defined files - Added the possibility to create self-signed certificates - All folders have been renamed, e.g. from 1.0.0.0-alpin to 1.0-alpin. - We have made working with gitlab-ci easier with new extensions and includes. - We have also reduced the readme as much as possible, as the documentation is now available centrally in our own documentation repository `https://dcso.github.io/MISP-dockerized-docs/`. #### Container `misp-dockerized-misp-modules` ##### Update Informations Features of the release: - Pre-Installed MISP modules In the past, we decided to use a three-digit version number for small and large changes for this container repository. This has proved unnecessary as the main repository is the central repository for new releases and all changes are already tracked here. We have therefore decided to change the 3-digit version number in all minor repositories to a 2-digit version number. For example: 1.0.1 -> 1.1 The second digit is changed any time we make minor changes. The first digit changes whenever we make major changes, e.g. a new base image. ##### Detailed Changes, Fixes & Improvements - Pre-Installed debian stretch MISP modules container - All folders have been renamed, e.g. from 1.0.0.0-debian to 1.0-debian. - We have made working with gitlab-ci easier with new extensions and includes. - We have also reduced the readme as much as possible, as the documentation is now available centrally in our own documentation repository `https://dcso.github.io/MISP-dockerized-docs/`. --- .gitlab-ci.yml | 37 ++++++++++++++--------------------- .travis.yml | 14 +++++++------ 1.0.0/.travis/Makefile | 17 ++++++++-------- 1.0.0/.travis/main.sh | 2 +- 1.0.0/.travis/tagging.sh | 2 +- 1.0.0/scripts/build_config.sh | 19 ++++++++++++------ 6 files changed, 46 insertions(+), 45 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index c283c904d..f2cf8e497 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -20,13 +20,10 @@ variables: CURRENT_VERSION: "1.0.0" FOLDER: "./current" ENV_OPTION: "$FOLDER/" - MYmake_travis: "make -C $FOLDER/.travis/" + makefile_travis: "make -C $FOLDER/.travis/" CI: "true" REPOURL: "dockerhub.dcso.de" - -# Include other YML Files for Jobs -# Source: https://docs.gitlab.com/ee/ci/yaml/README.html#include include: #- '0.1.0/.gitlab-ci.yml' #- '0.1.1/.gitlab-ci.yml' @@ -38,8 +35,6 @@ include: #- '0.3.3/.gitlab-ci.yml' #- '0.3.4/.gitlab-ci.yml' - '1.0.0/.gitlab-ci.yml' - - before_script: - apk add --no-cache make bash sudo git curl python py2-pip coreutils grep python3 @@ -50,7 +45,7 @@ before_script: # Choose the Environment Version - bash ./FOR_NEW_INSTALL.sh $CURRENT_VERSION # Pull all latest tagged container - - make -C $FOLDER/.travis/ pull-latest REPOURL=${REPOURL} + - $makefile_travis pull-latest REPOURL=${REPOURL} # On all other pull requests and branches exclude dev and master: Build Environment, but do nothing more. No tests. Tests with our Test Framework are only done in the dev branch. .env_test_only: @@ -80,7 +75,7 @@ before_script: - make -C $FOLDER deploy - docker ps -a # Testing - - make -C $FOLDER/.travis/ test + - $makefile_travis test - make -C $FOLDER delete # https://docs.gitlab.com/ee/ci/yaml/README.html#artifacts-reports artifacts: @@ -100,30 +95,28 @@ before_script: - make -C $FOLDER deploy - docker ps -a # Testing - - make -C $FOLDER/.travis/ test + - $makefile_travis test # prepare retagging - - export server_tag=$(cat $ENV_FOLDER.env |grep MISP_CONTAINER|cut -d = -f 2|sed 's,....$,,') - - export proxy_tag=$(cat $ENV_FOLDER.env |grep PROXY_CONTAINER|cut -d = -f 2|sed 's,....$,,') - - export robot_tag=$(cat $ENV_FOLDER.env |grep ROBOT_CONTAINER|cut -d = -f 2|sed 's,....$,,') - #- export postfix_tag=$(cat $ENV_FOLDER.env |grep POSTFIX_CONTAINER|cut -d = -f 2|sed 's,....$,,') - - export modules_tag=$(cat $ENV_FOLDER.env |grep MISP_MODULES_CONTAINER|cut -d = -f 2|sed 's,....$,,') - - export db_tag=$(cat $ENV_FOLDER.env |grep DB_CONTAINER_TAG|cut -d = -f 2|sed 's,....$,,') - - export redis_tag=$(cat $ENV_FOLDER.env |grep REDIS_CONTAINER_TAG|cut -d = -f 2|sed 's,....$,,') + - export server_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep server|cut -d ':' -f 3) + - export proxy_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep proxy|cut -d ':' -f 3) + - export robot_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep robot|cut -d ':' -f 3) + #- export postfix_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep server|cut -d ':' -f 3) + - export modules_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep modules|cut -d ':' -f 3) + - export db_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep db|cut -d ':' -f 3) + - export redis_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep redis|cut -d ':' -f 3) # retag all existing tags dev 2 public repo - - $MYmake_travis tag server_tag=${server_tag} proxy_tag=${proxy_tag} robot_tag=${robot_tag} postfix_tag=${postfix_tag} modules_tag=${modules_tag} REPOURL=${REPOURL}; + - $makefile_travis tag REPOURL=$REPOURL server_tag=${server_tag} proxy_tag=${proxy_tag} robot_tag=${robot_tag} postfix_tag=${postfix_tag} modules_tag=${modules_tag} db_tag=${modules_tag} redis_tag=${modules_tag} # Push Images to registry - - $MYmake_travis push server_tag=${server_tag} proxy_tag=${proxy_tag} robot_tag=${robot_tag} postfix_tag=${postfix_tag} modules_tag=${modules_tag} REPOURL=${REPOURL}; - # If we need separate Images fro DB and Redsi - #- ./travis/push.sh $(REPOURL)/misp-dockerized-db $(db_tag); - #- ./travis/push.sh $(REPOURL)/misp-dockerized-redis $(redis_tag); + - $makefile_travis push REPOURL=$REPOURL server_tag=${server_tag} proxy_tag=${proxy_tag} robot_tag=${robot_tag} postfix_tag=${postfix_tag} modules_tag=${modules_tag}; - make -C $FOLDER delete # https://docs.gitlab.com/ee/ci/yaml/README.html#artifacts-reports artifacts: + paths: + - ./current/.travis/reports/*.xml reports: junit: ./current/.travis/reports/*.xml - # deactivated: # https://docs.gitlab.com/ee/ci/examples/container_scanning.html # https://docs.gitlab.com/ee/ci/yaml/README.html#artifactsreportscontainer_scanning diff --git a/.travis.yml b/.travis.yml index 74d694e11..535080177 100644 --- a/.travis.yml +++ b/.travis.yml @@ -35,13 +35,15 @@ script: - if [[ "$TRAVIS_BRANCH" == "dev" ]]; then $makefile_travis test; fi after_success: - - export server_tag=$(cat $FOLDER.env |grep MISP_CONTAINER|cut -d = -f 2|sed 's,....$,,') - - export proxy_tag=$(cat $FOLDER.env |grep PROXY_CONTAINER|cut -d = -f 2|sed 's,....$,,') - - export robot_tag=$(cat $FOLDER.env |grep ROBOT_CONTAINER|cut -d = -f 2|sed 's,....$,,') - - export postfix_tag=$(cat $FOLDER.env |grep POSTFIX_CONTAINER|cut -d = -f 2|sed 's,....$,,') - - export modules_tag=$(cat $FOLDER.env |grep MISP_MODULES_CONTAINER|cut -d = -f 2|sed 's,....$,,') + - export server_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep server|cut -d ':' -f 3) + - export proxy_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep proxy|cut -d ':' -f 3) + - export robot_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep robot|cut -d ':' -f 3) + #- export postfix_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep server|cut -d ':' -f 3) + - export modules_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep modules|cut -d ':' -f 3) + - export db_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep db|cut -d ':' -f 3) + - export redis_tag=$(cat $FOLDER/docker-compose.yml |grep image|grep redis|cut -d ':' -f 3) # retag all existing tags dev 2 public repo - - $makefile_travis tag server_tag=${server_tag} proxy_tag=${proxy_tag} robot_tag=${robot_tag} postfix_tag=${postfix_tag} modules_tag=${modules_tag} + - $makefile_travis tag server_tag=${server_tag} proxy_tag=${proxy_tag} robot_tag=${robot_tag} postfix_tag=${postfix_tag} modules_tag=${modules_tag} db_tag=${modules_tag} redis_tag=${modules_tag} # check if branch=master and the current build is no pull request, then push it to docker hub - if [[ "$TRAVIS_BRANCH" == "master" ]]; then if [[ "$TRAVIS_PULL_REQUEST" == "false" ]]; then diff --git a/1.0.0/.travis/Makefile b/1.0.0/.travis/Makefile index 34c5be110..2e4c19572 100644 --- a/1.0.0/.travis/Makefile +++ b/1.0.0/.travis/Makefile @@ -18,10 +18,10 @@ pull-latest: $(foreach c, $(CONTAINER), docker pull $(REPOURL)/misp-dockerized-$(c):latest-dev;) tag: - ./tagging.sh $(REPOURL)/misp-dockerized-server $(server_tag); - ./tagging.sh $(REPOURL)/misp-dockerized-proxy $(proxy_tag); - ./tagging.sh $(REPOURL)/misp-dockerized-robot $(robot_tag); - ./tagging.sh $(REPOURL)/misp-dockerized-misp-modules $(modules_tag); + bash ./tagging.sh $(REPOURL)/misp-dockerized-server $(server_tag); + bash ./tagging.sh $(REPOURL)/misp-dockerized-proxy $(proxy_tag); + bash ./tagging.sh $(REPOURL)/misp-dockerized-robot $(robot_tag); + bash ./tagging.sh $(REPOURL)/misp-dockerized-misp-modules $(modules_tag); @echo "###########################################" docker images @echo "###########################################" @@ -36,8 +36,7 @@ test: @docker cp misp-robot:/srv/MISP-dockerized-testbench/reports/. reports/ push: - ./push.sh $(REPOURL)/misp-dockerized-server $(server_tag); - ./push.sh $(REPOURL)/misp-dockerized-proxy $(proxy_tag); - ./push.sh $(REPOURL)/misp-dockerized-robot $(robot_tag); - ./push.sh $(REPOURL)/misp-dockerized-misp-modules $(modules_tag); - + bash ./push.sh $(REPOURL)/misp-dockerized-server $(server_tag); + bash ./push.sh $(REPOURL)/misp-dockerized-proxy $(proxy_tag); + bash ./push.sh $(REPOURL)/misp-dockerized-robot $(robot_tag); + bash ./push.sh $(REPOURL)/misp-dockerized-misp-modules $(modules_tag); diff --git a/1.0.0/.travis/main.sh b/1.0.0/.travis/main.sh index 8a2b12bc6..66de0e7cf 100755 --- a/1.0.0/.travis/main.sh +++ b/1.0.0/.travis/main.sh @@ -1,7 +1,7 @@ #!/bin/bash # Set an option to exit immediately if any error appears -set -o errexit +set -e # Main function that describes the behavior of the # script. diff --git a/1.0.0/.travis/tagging.sh b/1.0.0/.travis/tagging.sh index 64fe7081d..82878ecce 100755 --- a/1.0.0/.travis/tagging.sh +++ b/1.0.0/.travis/tagging.sh @@ -1,5 +1,5 @@ #!/bin/bash -set -exv +set -ex DOCKER_REPO="$1" tag="$2" diff --git a/1.0.0/scripts/build_config.sh b/1.0.0/scripts/build_config.sh index 3c0b115cd..eb72715e2 100755 --- a/1.0.0/scripts/build_config.sh +++ b/1.0.0/scripts/build_config.sh @@ -90,6 +90,11 @@ function default_container_version() { PROXY_CONTAINER_TAG="$(cat $DOCKER_COMPOSE_FILE |grep image:|grep proxy|cut -d : -f 3)" ROBOT_CONTAINER_TAG="$(cat $DOCKER_COMPOSE_FILE |grep image:|grep robot|cut -d : -f 3)" MISP_MODULES_CONTAINER_TAG="$(cat $DOCKER_COMPOSE_FILE |grep image:|grep modules|cut -d : -f 3)" + #[ -z $(echo $POSTFIX_CONTAINER_TAG|grep dev) ] && POSTFIX_CONTAINER_TAG="$POSTFIX_CONTAINER_TAG-dev" + [ -z $(echo $MISP_CONTAINER_TAG|grep dev) ] && MISP_CONTAINER_TAG="$MISP_CONTAINER_TAG-dev" + [ -z $(echo $PROXY_CONTAINER_TAG|grep dev) ] && PROXY_CONTAINER_TAG="$PROXY_CONTAINER_TAG-dev" + [ -z $(echo $ROBOT_CONTAINER_TAG|grep dev) ] && ROBOT_CONTAINER_TAG="$ROBOT_CONTAINER_TAG-dev" + [ -z $(echo $MISP_MODULES_CONTAINER_TAG|grep dev) ] && MISP_MODULES_CONTAINER_TAG="$MISP_MODULES_CONTAINER_TAG-dev" ### MISP_TAG=$(echo $MISP_CONTAINER_TAG|cut -d - -f 1) ###################### END GLOBAL ########### @@ -436,12 +441,6 @@ if [ "$AUTOMATE_BUILD" = "true" ] # ask no questions only defaults echo "automatic build..." #### - #[ -z $(echo $POSTFIX_CONTAINER_TAG|grep dev) ] && POSTFIX_CONTAINER_TAG="$POSTFIX_CONTAINER_TAG-dev" - [ -z $(echo $MISP_CONTAINER_TAG|grep dev) ] && MISP_CONTAINER_TAG="$MISP_CONTAINER_TAG-dev" - [ -z $(echo $PROXY_CONTAINER_TAG|grep dev) ] && PROXY_CONTAINER_TAG="$PROXY_CONTAINER_TAG-dev" - [ -z $(echo $ROBOT_CONTAINER_TAG|grep dev) ] && ROBOT_CONTAINER_TAG="$ROBOT_CONTAINER_TAG-dev" - [ -z $(echo $MISP_MODULES_CONTAINER_TAG|grep dev) ] && MISP_MODULES_CONTAINER_TAG="$MISP_MODULES_CONTAINER_TAG-dev" - # set hostname to an fix one myHOSTNAME="misp.example.com" IMAGE_MISP_MODULES="image: ${DOCKER_REGISTRY}/misp-dockerized-misp-modules:${MISP_MODULES_CONTAINER_TAG}" @@ -473,6 +472,14 @@ if [ "$AUTOMATE_BUILD" = "true" ] [ "$QUERY_PGP" == "yes" ] && query_pgp_settings # LOG_SETTINGS [ "$QUERY_LOG_SETTINGS" == "yes" ] && query_log_settings + + if [ "$DEV" == true ] + then + IMAGE_MISP_MODULES="image: ${DOCKER_REGISTRY}/misp-dockerized-misp-modules:${MISP_MODULES_CONTAINER_TAG}" + IMAGE_MISP_SERVER="image: ${DOCKER_REGISTRY}/misp-dockerized-server:${MISP_CONTAINER_TAG}" + IMAGE_MISP_PROXY="image: ${DOCKER_REGISTRY}/misp-dockerized-proxy:${PROXY_CONTAINER_TAG}" + IMAGE_MISP_ROBOT="image: ${DOCKER_REGISTRY}/misp-dockerized-robot:${ROBOT_CONTAINER_TAG}" + fi fi ################################### # Write Configuration