THREDDS Test Environment

A packer + ansible project to build and deploy an Amazon Machine Image and Docker Container for use by the various THREDDS projects at Unidata for running automated tests. The AMI is used by our Jenkins server to launch worker nodes for running tests. The Docker Image will be used as the basis for a custom GitHub action for testing the various THREDDS projects. Both image types are based off the latest available image, AMI or Docker, for ubuntu 20.04. The AMI and docker images produced by packer are both called "thredds-test-environment".

Requirements

Packer (download)
Docker, for building the Docker Image (download)
AWS EC2 Credentials, for building the AMI.
- The packer configuration expects the AWS credentials to be stored under a profile named ucar-unidata-profile One way of setting this up would be to have an entry in the file <home-directory>/.aws/credentials that looks like the following:
```
[ucar-unidata-profile]
aws_access_key_id=XXXXXXXXX
aws_secret_access_key=XXXXXXXXX
```
  See the packer docs for more information.

While this project's packer configuration relies heavily on Ansible, you do not need to have Ansible installed locally. We utilize the ansible-local provisioner of Packer, which means Ansible is run on the remote/guest machine, and not on the build machine (the machine running packer). This does mean that part of the Packer build process includes installing Ansible on the remote/guest machine (see packer/provisioners/scripts/bootstrap-common.sh), which does add some time to the total build process (approximately two minutes). However, Since Ansible does not support the use of Windows systems as a control node, the ability create the THREDDS test environment images across all major platforms outweighs the extra cost in time.

Building the images

To generate the golden AMI and Docker images, start off by validating the packer configuration by first moving into the packer/ directory and then running:

packer validate thredds-test-env.json

Once validated, you may run the build using:

packer build --only=<type> thredds-test-env.json

<type> will one or more (separated by commas) of the following:

ami: Provision an AWS EC2 instance and generate an AMI (thredds-test-environment-<iso-timestamp>)
docker-commit: Provision a Docker container and generate and tag a local Docker Image (docker.unidata.ucar.edu/thredds-test-environment:latest).
docker-export: Provision a Docker container and generate a local Docker Image as a file (image.tar).
docker-github-action: Provision a Docker container for use with GitHub Actions and publish to the GitHub Package Repository (runs using manual workflow_dispatch trigger on github actions).
docker-github-action-nexus: Same as docker-github-action, but publish to the Unidata Nexus Repository.

Typically, we would run the following to update the AMI and Github Action Docker Image (hosted on Nexus) at the same time:

packer build --only=ami,docker-github-action-nexus thredds-test-env.json

The Docker Image build takes about 1 hour to create, where as the AMI build takes around 30 minutes. Packer will run the builders in parallel, so the total time to create the thredds-test-environment images is around an hour.

If using docker-commit, then once the image is built you can test out the environment by using:

docker run -i -t --rm docker.unidata.ucar.edu/thredds-test-environment:latest

Project layout

Inside the packer directory is a file called thredds-test-env.json. This contains the packer configuration for the builders (amazon-ebs builder, docker builder), the provisioners (shell scripts, ansible playbooks), and the post-processors (tagging the docker image).

The provisioners directory contains the provisioner configurations used by packer. There are two types of provisioners used by the packer configuration: ansible and scripts. The scripts/ directory contains the following shell scripts:

bootstrap-common.sh: Install and configure ansible (common to both builders)
bootstrap_first_aws.sh: This will run first on AWS. Waits for cloud-init to finish before running bootstrap-common.sh.
bootstrap_last_aws.sh: This runs immediately after bootstrap-common.sh on AWS. Installs a system-wide JRE, which is needed when using the AMS as a Jenkins worker node.
cleanup.sh: Runs after the ansible provisioner and cleans up the apt cache, as well as some general build environment things.

The ansible/ directory contains the ansible playbooks that are used to configure the testing environment. The ansible/ directory is laid out as follows:

site.yml: The main ansible playbook that references all other playbooks used by the build
roles/: Playbooks organized by common tasks

We use the following roles when provisioning our images:

cleanup: General cleanup related tasks, such as remove the temporary build directory and running ldconfig
general-packages: Install general packages needed for the build environment using the OS package manager.
gradle-builds-cache-bootstrap: Pull in and build netCDF-Java to populate the gradle cache for user ubuntu.
init: Initialize the build environment by ensuring the temporary ansible build directory exists.
libnetcdf-and-deps: Configure, build, and install zlib, HDF5, and netCDF-C.
maven: Obtain and install the Apache Maven software project management and comprehension tool.
miniconda: Obtain and install the Anaconda miniconda python distribution.
security:
- Add the ubuntu user.
- Add a default maven-settings.xml file configured to publish to the Unidata artifacts server.
- Configure ssh (uses modified version of a task from Jeff Geerling's ansible-role-security project - see packer/provisioners/ansible/roles/security/README.md).
- Configure a system wide bash environment.
temurin: Obtain and install Temurin 8, 11, and 14.
test-data-mount-prep: Prepare the environment to mount the thredds-test-data datasets when available (currently used on Jenkins worker nodes).
zulu: Obtain and install Zulu 8, 11, and 14.

We also use a role from Ansible Galaxy to setup a Ruby environment (geerlingguy.ruby).