Skip to content

Docker based agents and executors

Jump to bottom
wyvern8 edited this page Jan 18, 2018 · 3 revisions

This is relevant to the usage of the 'Docker' executor, which is a generic way of running adhoc tasks inside containers.

A number of CI based projects have the need to both run agents inside docker, as well as run test/other executions inside docker containers. For example, running a service such as Jenkins inside kubernetes will need to spawn agents, but what if these agents need to in turn run docker based tools?

There are two main approaches to this, and a lack of consensus as to the best way to achieve this. Either allow the agent to connect to the host docker services (docker outside of docker), or spawn sub-containers via embedded docker services (docker in docker). To me the docker in docker seems the safest, although it does somewhat obscure what is running due to its nested nature, however we leave the approach up to you with no warranty either way.

There is an interesting discussion along these lines here: https://gitlab.com/gitlab-org/gitlab-ce/issues/17769

Docker outside of Docker

This is fairly contentious with good reason, as it effectively gives agent containers elevated access on the host machine - an agent can now do whatever it likes with host docker, so if an agent is compromised, there will be much pain and suffering. They way it works is it maps the docker socket of the host into the agent, and uses this as the DOCKER_HOST env var within the agent. Some popular CI tools use this approach or at least have it as an option.

--volume="/var/run/docker.sock:/var/run/docker.sock"

The advantage is that running docker ps for example gives good visibility of what is running at a given point in time without inception style layers of processes to be navigated.

An example of this approach is in package.json script docker-siblings, which may be suitable if your environment is suitably isolated and you want good visibility.

Docker In Docker (dind)

docker:dind https://hub.docker.com/_/docker/ is a container mainly used by the docker development team, but implements current best practice of running docker inside another container, so is a good reference point.

Dind provides a minimal docker environment that can start containers within its own container instance. This means that you will need to hop into this container(s) to diagnose issues in spawned executors. Given that executors all use cloudwatch to log events, this should not generally be required.

An example of this is in the provided docker-compose.yml file, in particular this section:

  docker-in-docker:
    image: docker:dind
    privileged: true
    command: --storage-driver=overlay2
    expose:
      - 2375
      - 2376
  agent:
    image: "node:8"
    user: "node"
    working_dir: /home/node/app
    environment:
      - DOCKER_HOST=tcp://docker-in-docker:2375

As you can see, we run a docker-in-docker container, and then map the DOCKER_HOST env var to point at this from the gtm-agent. This works very well in a Kubernetes deployment in particular, and you can see examples in the reference k8s manifest and helm charts:

Clone this wiki locally