Warning: this uses Operator SDK 0.12 while 1+ is out. See https://sdk.operatorframework.io/docs/building-operators/golang/migration/ before upgrading as 0.12 no longer builds due to missing dependencies.

MC Robot

An application that manages multi-cluster service discovery & setup. The goal of this application is to make services in other clusters easily reachable.

The actions that are performed:

1. retrieve a list of k8s nodes
2. retrieve a list of k8s Services with a NodePort
3. publish list of services via PubSub mechanism
4. subscribe to list of services of peers via PubSub mechanism
5. configure custom k8s Service & Endpoints pointing to services in other clusters

This meets the goals because after this is done, you can reach other clusters with

curl http://my-service-gke_my-project_europe-west4_my-cluster.default.svc.cluster.local

MC Robot does not configure network topologies, so make sure to have all clusters can reach each other. Either have all clusters in the same shared network, or publish the external IPs of nodes by setting endpointsUseExternalIPs: true and configure the firewalls correctly.

1. Install (from registry)

First prepare a serviceaccount.json Service Account file with access to Google PubSub, and install it:

kubectl create secret generic mc-robot-credentials --from-file="serviceaccount.json=serviceaccount.json"

Then install the CRD & deploy the operator:

kubectl apply -f https://raw.githubusercontent.com/Q42/mc-robot/master/deploy/0_mc.q42.nl_servicesyncs_crd.yaml
kubectl apply -f https://raw.githubusercontent.com/Q42/mc-robot/master/deploy/1_rbac.yaml
curl -Ss https://raw.githubusercontent.com/Q42/mc-robot/master/deploy/2_operator.yaml | \
  sed "s|REPLACE_IMAGE|docker.pkg.github.com/q42/mc-robot/mc-robot:latest|g" | \
  kubectl apply -f -

2. Install (with custom build)

First install the CRD. Then build the operator & deploy it:

# Install CRD
kubectl apply -f deploy/0_mc.q42.nl_servicesyncs_crd.yaml

# Build operator
export VERSION=v1.0.0
export REGISTRY=quay.io/<user> # or gcr.io/project
operator-sdk build $REGISTRY/mc-robot:$VERSION
docker push $REGISTRY/mc-robot:$VERSION

# Deploy operator
kubectl apply -f deploy/1_rbac.yaml
kubectl create secret mc-robot-credentials --from-file="serviceaccount.json=serviceaccount.json"
sed "s|REPLACE_IMAGE|$REGISTRY/mc-robot:$VERSION|g" deploy/2_operator.yaml | kubectl apply -f -

3. Usage

Create a ServiceSync object like this:

apiVersion: mc.q42.nl/v1
kind: ServiceSync
metadata:
  name: example-servicesync
spec:
  topicURL: "gcppubsub://projects/myproject/topics/mytopic"
  selector:
    matchLabels:
      app: my-app
  endpointsPublishMax: 10

A topic url like gcppubsub://projects/myproject/topics/mytopic must be set. The service sync controller must have access to this topic, which can be configured through Application Default Crecentials with an environment variable GOOGLE_APPLICATION_CREDENTIALS which should point to a file with a service account, and which has access to that topic (reference).

4. Developing

4.1 Testing locally

$ GOFLAGS="-tags=mock" OPERATOR_NAME=mc-robot operator-sdk up local --namespace=default

4.2 Testing remote

Use these convenient methods to change the version & update the running operator:

export REGISTRY=quay.io/<user>
make install
VERSION=v1.0.0-alpha.x make build deploy # repeat after changes

4.3 Generated code

A lot of the code for MC Robot is generated by the operator-sdk framework. Commands that were run:

$ brew install operator-sdk
$ export GO111MODULE=on
$ operator-sdk new mc-robot
$ cd mc-robot
$ operator-sdk add api --api-version=mc.q42.nl/v1 --kind=ServiceSync
$ operator-sdk add controller --api-version=mc.q42.nl/v1 --kind=ServiceSync
$ operator-sdk generate k8s && operator-sdk generate openapi

4.4 ADRs

• Initial architecture

4.5 Working documentation

• How to define, build and run a CRD/controller
• How to write a reconciler
• https://medium.com/faun/writing-your-first-kubernetes-operator-8f3df4453234
• https://flugel.it/building-custom-kubernetes-operators-part-3-building-operators-in-go-using-operator-sdk/
• Inspiration regarding controller-runtime utils: https://github.com/openshift/cluster-ingress-operator/
• Installing kubebuilder & testing with Kubebuilder envtest
• Testing without Kubebuilder envtest