This project is is for the Udacity Cloud Developer Nanodegree, and specifically course 3 Monolith to Microservices at Scale. In this course we learn best practices on how to develop and deploy microservices, with a focus on different microservice architecture patterns, independent scaling, resiliency, securing microservices, and best practices for monitoring and logging.
The task is to refactor monolith application to microservices and deploy through AWS EKS In this project, we take an existing application named Udagram and refactor it into a microservice architecture with lean services. We build out a CI/CD process using Travis CI that automatically builds and deploys Docker images to a Kubernetes cluster. The Kubernetes cluster is configured to help solve common challenges related to scale and security.
To implement this project you need to have the following:
- An AWS account which you can create via the free tier at AWS Free Tier
- Install the AWS CLI on your development machine and initialize with your AWS credentials
- The Node Package Manager (NPM). You will need to download and install Node from https://nodejs.com/en/download. This will allow you to be able to run
npmcommands to install dependent libraries. The following are two important components of our solution: - Create an account with Docker Hub and possibly install Docker Desktop for CLI ease at the development machine
- The Amazon vended kubectl for interacting with your k8s clusters.
Udagram is a simple cloud application developed alongside the Udacity Cloud Engineering Nanodegree. It allows users to register and log into a web client, post photos to the feed, and process photos using an image filtering microservice.
Following tasks will have to be performed.
- Refactor the API : Decompose the API code so that we can have two separate projects that can be run independent of one another.
- Containerize the Code : Creating Dockerfiles for the frontend and backend applications. Each project should have its own Dockerfile.
- Build CICD Pipeline : Integrate the GitHub repository with Travis CI, and create a build pipeline that will push the generated images on DockerHub.
- Deploy to Kubernetes : Deploy the Docker containers for the API applications and web application as their own pods in AWS EKS.
- Implement Logging : Define a logging strategy and use logs to capture metrics. As the k8s will be set up with autoscale, it is important our logging strategy allows us to segregate events from different pods.
In order The following was implemented as a solution to the project challenge.
The project consists of the following containerized applications:
udagram-frontendFrontend - Angular web application built with Ionic Frameworkudagram-reverseproxyReverse Proxy - Nginx Reverse Proxy to relay requests to internal microservicesudagram-api-feedBackend RESTful API for posting and processing photos to the feed - Node-Express applicationudagram-api-usersBackend RESTful API for handling user login and authentication - Node-Express application
The cloud resources being used for this project consists of:
- Amazon S3 Object storage service used for storing the uploaded pictures.
- Amazon RDS Relational database service used for hosting a PostgreSQL database which contains the feeds and users
- Amazon EKS Elastic Kubernetes Service to start, run, and scale Kubernetes applications in the AWS cloud
- Amazon KMS Key management service to allow for the secrets created on Kubernetes to be encrypted (instead of just base-64 encoded)
Refactoring the API was straightforward. Our previous backend was an Express Node.js web application that was serving the routes \api\v0\users and \api\v0\feed. Refactoring meant creating two separate Express applications, one for each route.
The important modifications were:
Ensuring that credentials are pulled from the enviroment is important, since we our applications will be containerized and available on github as well as docker hub. The following secrets are retrieved by config.ts from the environment when running:
"username": process.env.POSTGRES_USERNAME,
"password": process.env.POSTGRES_PASSWD,
"database": process.env.POSTGRES_DATABASE,
"aws_region": process.env.AWS_REGION,
"host": process.env.AWS_HOST,
"aws_media_bucket": process.env.AWS_S3_BUCKETOur backends will need to use the proper IAM credentials in order to be allowed to make calls via the AWS SDK for JavaScript. We need these calls in order to access our AWS resources like the S3 bucket, RDS, etc. This is again implemented via config.ts with the following:
"aws_profile": process.env.AWS_PROFILEAWS will check the credentials of the provided profile to allow for access to resources. Since we will be having containerized apps on k8s, this will require encrypting and providing our credentials on the containers when they are deployed. This is explained in the Secrets section below.
Our Cross-Origin Resource Sharing had to be modified, since our monolithic app was restricting access from within our server localhost. Now, our clients will be accessing us through their frontend app that will have different IPs. The implemented CORS is as follows:
app.use(function(req, res, next) {
res.header("Access-Control-Allow-Origin", "*");
res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept, Authorization");
next();
});Our frontend communicates with the API through port 8080 and receives the requested data according to the path provided. Since we split our backend into separate apps, we need to set up an API Gateway to channel accordingly to the different applications the different paths.
This was implemented by creating a 3rd server application called udagram-reverseproxy which is an Nginx reverse proxy.
Configuration for this reverse proxy is given in nginx.conf. Our two api apps are defined as the upstream and then our nginx will listen on port 8080 and accordingly route to the right app. Excerpt provided below:
server {
listen 8080;
# Add the header X-Request-ID to the response to the client
# enable logging
access_log dev/stdout;
# add tracking/debugging information in the headers
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
# set up routes
location /api/v0/users {
proxy_pass http://user;
}
location /api/v0/feed {
proxy_pass http://feed;
}
}Our four apps are a separate directory in our project, each with it's own Dockerfile. The apps are containerized as follows:
- Backend applications are implemented with Dockerfiles for udagram-api-feed and udagram-api-users as follows:
- Both these apps use a NodeJS image as a base for the machine
- the package.json file is copied over to the created machine and
npm installis used to install all dependencies - Typescript is installed on the machine with
npm install typescript -g - Our source code is copied over and transpiled with
npm run build - We expose the port we use (8080) and inform docker that the command to be executed on this container is
node ./www/server.js
- Frontend
udagram-frontendis implemented differently, as usingionic servein a production environment is not recommened. As such, our Dockerfile performs the following steps:- Use a ready made image with ionic installed called beevelop/ionic
- Copy over our dependencies
package.json, install them withnpm ci, and copy our source code to this image - Transpile our code with
ionic build. This will result with all required JS of our frontend being available in thewwwdirectory - Use an nginx image for the container and copy into it our side with
COPY --from=ionic /usr/src/app/www /usr/share/nginx/html
- API Gateway Dockerfile is the simplest with again an
nginximage like the fronted, to which we simply copy over our nginx.conf
To facilitate building and pushing on DockerHub all our images, we use Docker Compose and created a docker-compose.yml file to allow easy build and/or deployment with the commands docker-compose -f deployments/docker-compose.yml build or docker-compose -f deployments/docker-compose.yml push
Our Docker images were pushed on Docker Hub, and below is the relevant screenshot from Docker Hub:
Travis CI was used for the CI/CD pipeline.
We provided permissions to Travis CI to see our GitHub repositories, and provided our Docker username and Docker password as secrets in Travis CI.
Each time we commit on Github, a pipeline is started to build the docker images. In order to prevent our k8s cluster from pulling dev images, our .travis.yml will publish these images on Docker Hub only if we are at the master branch.
A screenshot of an executed Travis CI build is provided below:
We used Amazon EKS to create and manage our Kubernetes cluster. Prior to delving into how this was performed, it is important to first explain how security concerns were addressed.
We have protected sharing any AWS credentials and information in our apps by getting these values at runtime from the environment and not storing them in either GitHub or insider the docker image.
On our development machine this works fine, however on kubernetes, the docker images that will be deployed need to have these environment variables.
Kubernetes provides a solution to this by allowing use to create secrets via kubectl. However, this solution is not safe, since kubectl create secret interfaces with etcd on an encoded instead of encrypted variable.
To make sure that our secret creation does not expose our secret, it is important that when our Amazon EKS cluster is created, we have created an Key in Amazon KMS enable Secrets encryption during cluster creation in Amazon EKS. With this option enabled, the Kubernetes secrets are encrypted using the customer master key (CMK) that you select. The CMK must be symmetric, created in the same region as the cluster, and if the CMK was created in a different account, the user must have access to the CMK.
With this option, when kubectl create secret creates a secret, the following secrets write flow is followed to secure us:
We will need to encrypt two sets of secrets to be available for our nodes:
On our development machine, we set the below environment variables to the secret values.
set POSTGRES_DB=xxx
set POSTGRES_PASSWD=xxx
set POSTGRES_USERNAME=xxx
set AWS_REGION=xxx
set AWS_PROFILE=xxx
set AWS_HOST=xxx
set AWS_S3_BUCKET=xxx
set JWT_SECRET=xxx
Then we create a secret named udagram-secrets on our cluster by running the following kubectl command:
kubectl create secret generic udagram-secrets \
--from-literal=POSTGRES_DB=%POSTGRES_DB% \
--from-literal=POSTGRES_PASSWD=%POSTGRES_PASSWD% \
--from-literal=POSTGRES_USERNAME=%POSTGRES_USERNAME \
--from-literal=AWS_REGION=%AWS_REGION% \
--from-literal=AWS_PROFILE=%AWS_PROFILE% \
--from-literal=AWS_HOST=%AWS_HOST% \
--from-literal=AWS_S3_BUCKET=%AWS_S3_BUCKET% \
--from-literal=JWT_SECRET=%JWT_SECRET%
Our two backend API apps are using AWS SDK calls to access the S3 bucket. As such, they will need to be authorized. To do this, we will need to have our aws_access_key_id and aws_secret_access_key passed to these two containers. We can retrieve these from our ~\.aws\credentials file as follows:
- encode profile and keys in base-64 with command
cat ~/.aws/credentials | head -n 3 | base64 - Create a yml file as follows to add the output of above command
apiVersion: v1 kind: Secret metadata: name: aws-secret type: Opaque data: credentials: <OUTPUT OF ABOVE COMMAND>
- Apply this file to k8s with
kubectl apply -f
We can use kubectl get secrets to see the secrets created.
NAME TYPE DATA AGE
aws-secret Opaque 1 5h45m
default-token-kx8gd kubernetes.io/service-account-token 3 3d13h
udagram-secrets Opaque 8 2d14h
Each microservice has a deployment for it's app and service in the k8s directory. The notable parts are:
The secrets are given to each app through the env clause. Below excerpt from api-feed.yml:
kind: Deployment
...
spec:
replicas: 1
selector:
matchLabels:
app: udagram-api-feed
template:
metadata:
labels:
app: udagram-api-feed
spec:
containers:
....
envFrom:
- secretRef:
name: udagram-secrets
volumeMounts:
- name: aws-secret
mountPath: "/root/.aws/"
readOnly: true
....
volumes:
- name: aws-secret
secret:
secretName: aws-secret
The frontend and reverse-proxy services are created with type LoadBalancer to allow for having externally accessible endpoints. Excerpt below from the frontend service part:
apiVersion: v1
kind: Service
metadata:
name: frontend
labels:
service: frontend
spec:
ports:
- protocol: TCP
port: 80
targetPort: 80
selector:
service: frontend
type: LoadBalancer Deploying the apps and services is achieved by running kubectl apply -f .\deployments\k8s which applies everything in the directory
Kubernetes Horizontal Pod Autoscaling was set up to allow for scaling our backend api-feed pod when CPU goes over 50%. Note that HPA does not work out of the box on Amazon EKS clusters. You first need to deploy the Metrics Server to your cluster. This can be done with the command:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
We then set up the HPA on the udagram-api-feed deployment as follows:
kubectl autoscale deployment udagram-api-feed --cpu-percent=50 --min=1 --max=3-
verify Kubernetes pods are deployed properly with
kubectl get pods
-
verify Kubernetes services are properly set up with
kubectl get services
-
verify that horizontal scaling set against CPU usage with
kubectl get hpa
-
verify that user activity is logged with
kubectl logs -l=app=udagram-api-feed --all-containers=trueorkubectl logs <pod>
