ha.md

OVN central database High-availability

OVN architecture has two central databases that can be clustered. The databases are OVN_Northbound and OVN_Southbound. This document explains how to cluster them and start various daemons for the ovn-kubernetes integration. You will ideally need atleast 3 master nodes for a HA cluster. (You will need a miniumum of OVS/OVN 2.9.2 for clustering.)

Master1 node initialization

To bootstrap your cluster, you need to start on one master node. For a lack of better name, lets call it MASTER1 with an IP address of $MASTER1

On MASTER1, delete any stale OVN databases and stop any ovn-northd running. e.g:

sudo /usr/share/openvswitch/scripts/ovn-ctl stop_nb_ovsdb
sudo /usr/share/openvswitch/scripts/ovn-ctl stop_sb_ovsdb
sudo rm /etc/openvswitch/ovn*.db
sudo /usr/share/openvswitch/scripts/ovn-ctl stop_northd

Start the two databases on that node with:

LOCAL_IP=$MASTER1
sudo /usr/share/openvswitch/scripts/ovn-ctl \
    --db-nb-cluster-local-addr=$LOCAL_IP start_nb_ovsdb

sudo /usr/share/openvswitch/scripts/ovn-ctl \
    --db-sb-cluster-local-addr=$LOCAL_IP start_sb_ovsdb

Master2, Master3... node initialization

Delete any stale databases and stop any running ovn-northd daemons. e.g:

sudo /usr/share/openvswitch/scripts/ovn-ctl stop_nb_ovsdb
sudo /usr/share/openvswitch/scripts/ovn-ctl stop_sb_ovsdb
sudo rm /etc/openvswitch/ovn*.db
sudo /usr/share/openvswitch/scripts/ovn-ctl stop_northd

On master node, with a IP of $LOCAL_IP, start the databases and ask it to join $MASTER1

LOCAL_IP=$LOCAL_IP
MASTER_IP=$MASTER1

sudo /usr/share/openvswitch/scripts/ovn-ctl  \
    --db-nb-cluster-local-addr=$LOCAL_IP \
    --db-nb-cluster-remote-addr=$MASTER_IP start_nb_ovsdb

sudo /usr/share/openvswitch/scripts/ovn-ctl  \
    --db-sb-cluster-local-addr=$LOCAL_IP \
    --db-sb-cluster-remote-addr=$MASTER_IP start_sb_ovsdb

This should get your cluster up and running. You can verify the status of your cluster with:

sudo ovs-appctl -t /var/run/openvswitch/ovnnb_db.ctl \
    cluster/status OVN_Northbound

sudo ovs-appctl -t /var/run/openvswitch/ovnsb_db.ctl \
    cluster/status OVN_Southbound

Start 'ovn-kube -init-master'

On any one of the master nodes, we need to start 'ovnkube -init-master'. (This should ideally be a daemonset with replica count of 1.)

IP1="$MASTER1" IP2="$MASTER2" IP3="$MASTER3"

ovn_nb="tcp://$IP1:6641,tcp://$IP2:6641,tcp://$IP3:6641" ovn_sb="tcp://$IP1:6642,tcp://$IP2:6642,tcp://$IP3:6642"

nohup sudo ovnkube -k8s-kubeconfig kubeconfig.yaml
-loglevel=4
-k8s-apiserver="http://$K8S_APISERVER_IP:8080"
-logfile="/var/log/openvswitch/ovnkube.log"
-init-master="$NODENAME" -cluster-subnets="$CLUSTER_IP_SUBNET"
-init-node="$NODENAME"
-k8s-service-cidr="$SERVICE_IP_SUBNET"
-k8s-token="$TOKEN"
-nodeport
-nb-address="${ovn_nb}"
-sb-address="${ovn_sb}" 2>&1 &

start ovn-northd

On any one of the nodes (ideally via a daemonset with replica count as 1), start ovn-northd. Let the 3 master node IPs be $IP1, $IP2 and $IP3.

IP1="$MASTER1"
IP2="$MASTER2"
IP3="$MASTER3"

export ovn_nb="tcp:$IP1:6641,tcp:$IP2:6641,tcp:$IP3:6641"
export ovn_sb="tcp:$IP1:6642,tcp:$IP2:6642,tcp:$IP3:6642"

sudo ovn-northd -vconsole:emer -vsyslog:err -vfile:info \
    --ovnnb-db="$ovn_nb" --ovnsb-db="$ovn_sb" --no-chdir \
    --log-file=/var/log/openvswitch/ovn-northd.log \
    --pidfile=/var/run/openvswitch/ovn-northd.pid --detach --monitor

Start 'ovn-kube -init-node'

In all other minions (and if needed on other masters), start ovnkube with '-init-node'. For e.g:

IP1="$MASTER1"
IP2="$MASTER2"
IP3="$MASTER3"

ovn_nb="tcp://$IP1:6641,tcp://$IP2:6641,tcp://$IP3:6641"
ovn_sb="tcp://$IP1:6642,tcp://$IP2:6642,tcp://$IP3:6642"

nohup sudo ovnkube -k8s-kubeconfig $HOME/kubeconfig.yaml -loglevel=4 \
    -logfile="/var/log/openvswitch/ovnkube.log" \
    -k8s-apiserver="http://$K8S_APISERVER_IP:8080" \
    -init-node="$MINION_NAME"  \
    -nb-address="${ovn_nb}" \
    -sb-address="${ovn_sb}" \
    -k8s-token="$TOKEN" \
    -init-gateways \
    -k8s-service-cidr= \
    -cluster-subnets="$SERVICE_IP_SUBNET" 2>&1 &