Ansible Amazon Base Repository

• Ansible Amazon Base Repository
  • About
  • Setting Up Ansible Environment
    • Configuring Python VENV and Ansible
    • Editor Configuration
  • Setting Up VirtualBox Environment
    • Install VirtualBox and Vagrant
    • Configuring Test Kitchen
    • Downloading Amazon Linux v2 Vagrant Box
    • Build an Optimized Amazon Linux 2 Vagrant Box
    • Environment Variable Overwrites for Kitchen
  • Running Test Kitchen
  • Running Integration Tests
  • Running Ansible Playbooks on AWS Instances
  • Example Project: Build a Local Docker/Containerd Server VM
  • Ansible References

About

Ansible Amazon Base Repository is an MVP monorepo to rapidly develop scalable, reliable, high-quality components for Amazon Linux instance configuration management.

Setting Up Ansible Environment

Configuring Python VENV and Ansible

Note: This document assumes that you are working on Mac

1. Create a new virtual environment with pyenv

   $ pyenv virtualenv miniconda3-latest ansible
   

2. Activate your new python virtual environment

   $ pyenv activate ansible
   

3. Install poetry

   $ conda install poetry
   

4. Install dependencies

   $ poetry install
   

5. Check ansible

   $ ansible --version
   

Editor Configuration

Code editors are major software development productivity tools. VSCode is a game changer.

VSCode should be configured for typical Python development with the following extensions:

• Ansible
• YAML
• Code Spell Checker
• Trailing Spaces
• Blank line at the End of File
• Markdown All in One
• Markdownlint
• Pylance
• Python
• Test Explorer
• Python Test Explorer
• Docker
• Remote SSH
• Remote Containers
• VirtualBox
• Vagrant
• Ruby
• Ruby Solargraph
• Github
• Github Actions
• Gitlens

VSCode should be configured to associate most of the files in the repository with Ansible. Check that your ansible files have Ansible set in the "Select Language Mode". In VSCode the difference between YAML and Ansible Language Mode is night and day. Example .vscode/settings.json

{
  "files.associations": {
    "kitchen*": "yaml",
    "*.yml": "ansible"
  },
}

Python should be setup with the following:

    "editor.renderWhitespace": "all",
    "editor.rulers": [
        80,
        100,
        120
    ],
    "[python]": {
        "editor.tabSize": 4,
        "editor.insertSpaces": true,
        "editor.formatOnSave": true
    },
    "[yaml]": {
        "editor.insertSpaces": true,
        "editor.tabSize": 2,
        "editor.autoIndent": "none",
        "editor.quickSuggestions": {
            "other": true,
            "comments": false,
            "strings": true
        },
        "editor.formatOnPaste": true
    },
    "python.linting.enabled": true,
    "python.linting.pylintEnabled": true,
    "python.linting.banditEnabled": true,
    "python.linting.banditArgs": [
        "-x",
        "./tests",
        "-r",
    ],
    "python.linting.pylintArgs": [
        "--disable=C0301,C0111,E0402,W0702,W0108,W0703"
    ],
    "python.linting.pycodestyleEnabled": true,
    "python.linting.pycodestyleArgs": [
        "--ignore E501"
    ],
    "python.formatting.provider": "black",
    "python.languageServer": "Pylance",
    "python.envFile": "/Users/current.user/.vspyenv",
    "python.testing.pytestArgs": [
        "-s",
        "-vvvv"
    ],
    "pythonTestExplorer.testFramework": "pytest",
    "markdownlint.config": {
        "MD013": {
            "line_length": 120,
            "tables": false,
            "code_blocks": false
        },
        "MD025": false,
        "MD033": false,
        "MD036": false,
        "MD041": false
    },
    "testExplorer.hideEmptyLog": false,

Setting Up VirtualBox Environment

Access to local VMs running Amazon Linux helps rapidly and safely iterate on Ansible code.

Install VirtualBox and Vagrant

VirtualBox can be installed with Homebrew. However, every once in a while the latest build of VirtualBox has a broken functionality. At the time of writing, VirtualBox 6.1.28 has a broken Host Network Manager. Good build of VirtualBox is 6.1.26. Install Virtual Box from the link.

Vagrant is a HashiCorp Ruby project to provide VirtualBox abstraction. Use Homebrew to install vagrant

$ brew install vagrant

Configuring Test Kitchen

Test Kitchen is a Ruby project to automate Infrastructure as Code development life-cyle.

Install RVM

$ gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB

$ \curl -sSL https://get.rvm.io | bash -s stable --ruby

Add RVM to your profile

# Add RVM
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"

Change directory to anything other than ansible-amazon-base and change back to have RVM pull a ruby version and create a gemset.

$ cd ..
$ cd ansible-amazon-base/
ruby-3.0.0 - #gemset created /home/current.user/.rvm/gems/ruby-3.0.0@ansible-kitchen
ruby-3.0.0 - #generating ansible-kitchen wrappers.............

After RVM configures Gemset, run bundler to install gems

$ bundle install

Downloading Amazon Linux v2 Vagrant Box

Amazon Linux v2 is a feature-rich Linux distribution maintained by Amazon. Amazon page about Amazon Linux Images lists various formats available, including Amazon Linux v2 virtualbox.

Older Amazon Linux v2 vagrant box can be downloaded from HashiCorp Vagrant Cloud.

Import vagrant box:

$ vagrant box add amazon2 <downloaded box>
$ vagrant box list

Latest Amazon Linux v2 box can be built from Chef Bento project. Disabling Amazon SSM is a consideration.

Build an Optimized Amazon Linux 2 Vagrant Box

Default Amazon Linux 2 vagrant box does not come with Ansible installed. Any time we run kitchen with the default Amazon Linux 2 box, kitchen will spend time installing Ansible. In order to save development time, we will build a box that includes Ansible and Docker.

Run kitchen converge with kitchen.box.yml

$ KITCHEN_YAML=kitchen.box.yml kitchen converge box

List running VirtualBox VMs and make note of the full name of the kitchen-ansible-amazon-base-box-amazon VM

$ VBoxManage list vms
"kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e" {9cf5ed90-d3a0-4e95-b742-6c9249c0cf34}

Run vagrant to export kitchen-ansible-amazon-base-box-amazon

$ vagrant package --base kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e
==> kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e: Attempting graceful shutdown of VM...
==> kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e: Clearing any previously set forwarded ports...
==> kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e: Exporting VM...
==> kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e: Compressing package to: /Users/current.user/work/ansible-amazon-base/package.box

Import package.box as amazon2-ansible

$ vagrant box add amazon2-ansible package.box
$ vagrant box list
amazon2         (virtualbox, 0)
amazon2-ansible (virtualbox, 0)

Remove kitchen instance

KITCHEN_YAML=kitchen.box.yml kitchen destroy box

Remove package.box from the local folder

Environment Variable Overwrites for Kitchen

Kitchen will use amazon2-ansible box by default. If you are using a different box, you can set KITCHEN_ANSIBLE_BOX environment variable.

Running Test Kitchen

Change directory to playbooks and run

$ kitchen list

To create a VirtualBox VM and apply an Ansible playbook run

$ kitchen converge docker

To destroy a created VM run

$ kitchen destroy docker

To connect to a VM run

$ kitchen ssh docker

Running Integration Tests

Integration tests are written in TestInfra. To run tests

$ kitchen verify docker

Running Ansible Playbooks on AWS Instances

In order to run Ansible Playbooks on AWS instances, first configure ANSIBLE_ROLES_PATH to point to the roles directory of the repository, e.g.

$ export ANSIBLE_ROLES_PATH=/Users/current.user/work/ansible-amazon-base/roles

Also configure ~/.ansible.cfg to format Ansible output as a more readable YAML

[defaults]
stdout_callback = yaml

Choose a playbook and run Ansible

$ ansible-playbook -v -u ec2-user --private-key ~/.ssh/<instance>.pem -i <instance_ip>, playbooks/gst/gst_jupyter.yml

Example Project: Build a Local Docker/Containerd Server VM

Why bother with Docker Desktop when you can build your own Containerd Server?

Check "Host Network Manager" in the File menu of your VirtualBox. Add an interface and make note of the subnet.

Edit kitchen.yml containerd suit and set private_network to a static IP of your choice.

  - name: containerd
    provisioner:
      name: ansible_playbook
      playbook: ./playbooks/docker/docker.yml
    driver:
      vm_hostname: containerd.local
      network:
        - ['private_network', {ip: '192.168.56.121'}]

Run kitchen converge

$ kitchen converge containerd
...
       PLAY RECAP *********************************************************************
       localhost                  : ok=5    changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

       Downloading files from <containerd-amazon>
       Finished converging <containerd-amazon> (0m5.54s).
-----> Test Kitchen is finished. (1m15.18s)

SSH into your Containerd VM. Change vagrant's user password from 'vagrant' to something secure. Add your SSH key to ~/.ssh/authorized_keys. Change permissions on /var/run/docker.sock.

$ ssh [email protected]
[email protected]'s password:
Last login: Mon Nov 15 05:47:31 2021 from 10.0.2.2

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/

This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento

[vagrant@containerd ~]$ passwd
Changing password for user vagrant.
Changing password for vagrant.
(current) UNIX password:
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[vagrant@containerd ~]$ vi ~/.ssh/authorized_keys
[vagrant@containerd ~]$ sudo chmod a+rw /var/run/docker.sock
exit
logout
Connection to 192.168.56.121 closed.

Create a new Docker context:

$ docker context create containerd1 --docker "host=ssh://[email protected]"
containerd
Successfully created context "containerd1"
$ docker context use containerd1
$ docker context ls
NAME            DESCRIPTION                               DOCKER ENDPOINT                KUBERNETES ENDPOINT   ORCHESTRATOR
containerd1 *                                             ssh://[email protected]
default         Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                          swarm

Start using your new Containerd server:

$ docker version
Client: Docker Engine - Community
 Version:           20.10.10
 API version:       1.41
 Go version:        go1.17.2
 Git commit:        b485636f4b
 Built:             Fri Oct 15 14:45:13 2021
 OS/Arch:           darwin/amd64
 Context:           containerd1
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.14
  Git commit:       b0f5bc3
  Built:            Tue Sep 28 19:56:28 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0
  GitCommit:        84113eef6fc27af1b01b3181f31bbaf708715301
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Ansible References

• Ansible 101 - Episode 1 - Introduction to Ansible
• Ansible 101 - Episode 2 - Ad-hoc Tasks and Inventory
• Ansible 101 - Episode 3 - Introduction to Playbooks
• Ansible 101 - Episode 4 - Your First Real-World Playbook
• Ansible 101 - Episode 5 - Playbook Handlers, Environment Vars, and Variables
• Ansible 101 - Episode 6 - Ansible Vault and Roles
• Make Your Ansible Playbooks Flexible, Maintainable, and Scalable