Add provision scripts

Signed-off-by: Matej Feder <[email protected]>

netbox/bootstrap/README.md

@@ -76,5 +76,5 @@ Note that the script is not limited to SCS Landscape. The reader could create ow
 of required initialization in Netbox.
 
 ```bash
-./netbox_init.py --api-url <netbox-url> --api-token <netbox-token> --sync-datasources --execute-scripts --data-dir landscape
+./netbox_init.py --api-url <netbox-url> --api-token <netbox-token> --sync-datasources --execute-scripts --sync-config-templates --data-dir landscape
 ```

netbox/bootstrap/landscape/04_config_templates.yml

@@ -2,7 +2,7 @@ config-templates:
   - name: "sonic"
     description: |
       Configuration template for Edgecore SONiC. Template stores two configuration files,
-      config_db.json and frr.conf, each as a string under its respective key in YAML.
+      config_db.json and frr.conf, each as a string under its respective key in yaml.
     data_source:
       name: "scs-hardware-landscape"
     data_file:

netbox/config_templates/sonic.yml.j2

@@ -6,4 +6,4 @@ config_db.json: |
 frr.conf: |
 {%- filter indent(width=2) %}
 {% include 'netbox/config_templates/frr.conf.j2' %}
-{%- endfilter %}
+{%- endfilter %}

netbox/provisioning/README.md

@@ -0,0 +1,52 @@
+# SONiC Zero Touch Provisioning
+
+## ONIE
+
+ONIE offers various methods for locating a Network Operating System (NOS) like the SONiC installer image.
+For a comprehensive list of supported methods, refer to the [onie user-guide](https://opencomputeproject.github.io/onie/user-guide/index.html).
+
+One such supported method is DHCP. A DHCP server can provide specific details about the location of the installer image
+for ONIE. Basic DHCP scenarios and configuration examples can be found in [onie user-guide](https://opencomputeproject.github.io/onie/user-guide/index.html).
+
+An advanced and more realistic scenario involving DHCP is Vendor Class Identifier matching.
+When ONIE makes a DHCP request, it sets the DHCP vendor class (option 60) to a specific string.
+Based on this, the DHCP server can select and return the appropriate installer image according to the machine type.
+
+See an example for [ISC DHCP server](https://github.com/opencomputeproject/onie/blob/master/contrib/isc-dhcpd/dhcpd.conf)
+where the Vendor Class Identifier is used to select installer image for accton_as7326_56x machine type.
+
+```text
+if option vendor-class-identifier = "onie_vendor:x86_64-accton_as7326_56x-r0" {
+  option default-url "http://10.10.23.254:18080/Edgecore-SONiC_20240625_102433_ec202211_ecsonic_331.bin";
+}
+```
+
+Alternatively, the administrator could define matching class as follows:
+
+```text
+class "onie-vendor-accton_as7326_56x-class" {
+  match if substring(option vendor-class-identifier, 0, 27) = "onie_vendor:x86_64-accton_as7326_56x-r0";
+  option default-url "http://10.10.23.254:18080/Edgecore-SONiC_20240625_102433_ec202211_ecsonic_331.bin";
+}
+```
+
+## Zero Touch Provisioning (ZTP)
+
+Zero Touch Provisioning (ZTP) in SONiC automates the initial setup of network devices without any user intervention.
+When a SONiC device boots up for the first time, the ZTP service sends a DHCP request to obtain a location of the ZTP boot file.
+The DHCP server provides the location of a boot file (utilizing DHCP option 67). This file is retrieved by the ZTP via TFTP,
+HTTP, or another protocol.
+
+The boot file contains information for ZTP to kick-start configuration steps of the device. This process bring the device
+into operational state automatically.
+ZTP service can be used by users to configure a fleet of switches using common configuration templates.
+
+[ISC DHCP server](https://github.com/opencomputeproject/onie/blob/master/contrib/isc-dhcpd/dhcpd.conf) could be configured
+to provide boot file location, e.g. as follows:
+
+```text
+option bootfile-name "http://10.10.23.254:18080/provision.json";
+```
+
+Refer to the [docs](https://github.com/sonic-net/SONiC/blob/master/doc/ztp/ztp.md) for detailed information on how to
+create ZTP boot file.

netbox/provisioning/prerequisites.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+set -e
+
+command_exists() {
+  command -v "$1" &> /dev/null
+}
+
+if ! command_exists jq; then
+  echo "Installing jq..."
+  sudo curl -L "http://10.10.23.254:18080/jq" -o /usr/bin/jq
+  sudo chmod +x /usr/bin/jq
+else
+  echo "jq is already installed"
+fi
+
+if ! command_exists yq; then
+  echo "Installing yq..."
+  sudo curl -L "http://10.10.23.254:18080/yq" -o /usr/bin/yq
+  sudo chmod +x /usr/bin/yq
+else
+  echo "yq is already installed"
+fi
+
+echo "All prerequisites are met"

netbox/provisioning/provision.json

@@ -0,0 +1,21 @@
+{
+ "ztp": {
+    "01-prerequisites-script": {
+      "plugin": {
+        "url": "http://10.10.23.254:18080/prerequisites.sh"
+      }
+    },
+    "02-provisioning-script": {
+      "plugin": {
+        "url": "http://10.10.23.254:18080/provision.sh",
+        "ignore-section-data": true,
+        "args": "--netbox-url http://10.10.21.10:8121 --netbox-token 2386377016882763985994631017085286020450"
+      }
+    },
+    "03-connectivity-check": {
+      "ping-hosts": [
+        "10.10.23.254"
+      ]
+    }
+  }
+}

netbox/provisioning/provision.sh

@@ -0,0 +1,98 @@
+#!/bin/bash
+
+# This script is designed for use with Zero Touch Provisioning (ZTP) to automatically apply the initial device configuration.
+# The device hostname is used as a key to retrieve the config_db.json and frr.conf files from NetBox,
+# which are then saved on the device where the script is executed.
+#
+# Usage: ./provision.sh -u NETBOX_URL -t AUTH_TOKEN
+
+CONFIG_DIR="/etc/sonic"
+TEMP_DIR="/tmp/ztp_config"
+CONFIG_DB_PATH="$CONFIG_DIR/config_db.json"
+FRR_CONF_PATH="$CONFIG_DIR/frr/frr.conf"
+
+# Parse command-line arguments
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        -u|--netbox-url)
+            NETBOX_API="$2"
+            shift # past argument
+            shift # past value
+            ;;
+        -t|--netbox-token)
+            AUTH_TOKEN="$2"
+            shift # past argument
+            shift # past value
+            ;;
+        *)
+            echo "Usage: $0 -u NETBOX_URL -t AUTH_TOKEN"
+            exit 1
+            ;;
+    esac
+done
+
+# Ensure both NETBOX_API and AUTH_TOKEN are set
+if [ -z "$NETBOX_API" ] || [ -z "$AUTH_TOKEN" ]; then
+    echo "Error: Both NETBOX_URL and AUTH_TOKEN must be provided."
+    echo "Usage: $0 -u NETBOX_URL -t AUTH_TOKEN"
+    exit 1
+fi
+
+NETBOX_API="${NETBOX_API%/}/api/dcim/devices/"
+
+validate_commands() {
+    for cmd in "$@"; do
+        if ! command -v "$cmd" &> /dev/null; then
+            echo "Error: $cmd is not installed."
+            exit 1
+        fi
+    done
+}
+
+validate_commands jq yq curl
+
+HOSTNAME=$(hostname)
+
+echo "Querying NetBox information for device: $HOSTNAME ..."
+DEVICE_INFO=$(curl -s -H "Authorization: Token $AUTH_TOKEN" -H "Content-Type: application/json" "${NETBOX_API}?name=${HOSTNAME}")
+
+DEVICE_ID=$(echo "$DEVICE_INFO" | jq -r '.results[0].id')
+if [ -z "$DEVICE_ID" ] || [ "$DEVICE_ID" == "null" ]; then
+    echo "Error: Device not found in NetBox with hostname '$HOSTNAME'"
+    exit 1
+fi
+
+echo "Fetching rendered configuration from NetBox for device ID: $DEVICE_ID ..."
+CONFIG=$(curl -X POST -s -H "Authorization: Token $AUTH_TOKEN" -H "Content-Type: application/json" "${NETBOX_API}${DEVICE_ID}/render-config/")
+
+# Create temporary directory
+mkdir -p "$TEMP_DIR"
+
+# Extract and save config_db.json
+CONFIG_DB_JSON=$(echo "$CONFIG" | jq -r '.content' | yq e '."config_db.json"' -)
+if [ -n "$CONFIG_DB_JSON" ]; then
+    echo "Writing config_db.json to temporary directory..."
+    echo "$CONFIG_DB_JSON" > "$TEMP_DIR/config_db.json"
+else
+    echo "Error: config_db.json not found in the rendered configuration."
+    exit 1
+fi
+
+# Extract and save frr.conf
+FRR_CONF=$(echo "$CONFIG" | jq -r '.content' | yq e '."frr.conf"' -)
+if [ -n "$FRR_CONF" ]; then
+    echo "Writing frr.conf to temporary directory..."
+    echo "$FRR_CONF" > "$TEMP_DIR/frr.conf"
+else
+    echo "Error: frr.conf not found in the rendered configuration."
+    exit 1
+fi
+
+# Move files to the desired location with sudo
+echo "Moving files to $CONFIG_DIR..."
+sudo mv "$TEMP_DIR/config_db.json" "$CONFIG_DB_PATH"
+sudo mv "$TEMP_DIR/frr.conf" "$FRR_CONF_PATH"
+
+rm -rf "$TEMP_DIR"
+
+echo "Configuration applied successfully."

netbox/scripts/sonic_config_diff.py

@@ -0,0 +1,112 @@
+import json
+import difflib
+import yaml
+
+from netmiko import ConnectHandler
+from netmiko.exceptions import NetmikoBaseException
+from dcim.models import Device
+from extras.scripts import Script, ObjectVar, AbortScript
+from jinja2.exceptions import TemplateError
+
+
+class SonicConfigDiff(Script):
+    class Meta:
+        name = "Compare SONiC Configuration with NetBox Stored Configuration"
+        description = "Fetches the current configuration from a SONiC device via SSH and compares it with the configuration stored in NetBox."
+
+    device = ObjectVar(
+        description="Select the device to check",
+        label="Device",
+        model=Device
+    )
+
+    @staticmethod
+    def get_device_connection(device):
+        """Connects to the SONiC device via SSH."""
+        sonic_device = {
+            "device_type": "linux",
+            "host": str(device.primary_ip.address.ip),
+            "username": "admin",
+            "password": "YourPaSsWoRd",  # Use an environment variable or secure storage in production
+            "port": 22,
+        }
+        try:
+            return ConnectHandler(**sonic_device)
+        except NetmikoBaseException as err:
+            raise AbortScript(f"Connection to SONiC device failed: {err}")
+
+    @staticmethod
+    def get_sonic_config_db(connection):
+        try:
+            return connection.send_command("sudo show runningconfiguration all")
+        except NetmikoBaseException as err:
+            raise AbortScript(f"Retrieve SONiC config failed: {err}")
+
+    @staticmethod
+    def get_sonic_frr_conf(connection):
+        try:
+            return connection.send_command("sudo cat /etc/sonic/frr/frr.conf")
+        except NetmikoBaseException as err:
+            raise AbortScript(f"Retrieve SONiC config failed: {err}")
+
+    @staticmethod
+    def get_netbox_config(device):
+        """Fetches the configuration stored in NetBox for the given device."""
+
+        context_data = device.get_config_context()
+        context_data.update({"device": device})
+        if config_template := device.get_config_template():
+            try:
+                rendered_config = config_template.render(context=context_data)
+            except TemplateError as err:
+                raise AbortScript(err)
+        else:
+            raise AbortScript("Define config template for device")
+
+        try:
+            rendered_config_yaml = yaml.safe_load(rendered_config)
+        except yaml.YAMLError as err:
+            raise AbortScript(err)
+
+        config_db = json.dumps(json.loads(rendered_config_yaml["config_db.json"]), indent=4)
+        frr = rendered_config_yaml["frr.conf"]
+
+        return config_db, frr
+
+    def run(self, data, commit):
+        device = data["device"]
+
+        with self.get_device_connection(device) as connection:
+            config_db_json = self.get_sonic_config_db(connection)
+            frr_conf = self.get_sonic_frr_conf(connection)
+
+        netbox_config_db, netbox_frr = self.get_netbox_config(device)
+
+        # Compare configurations
+        diff_config_db = difflib.unified_diff(
+            netbox_config_db.splitlines(),
+            config_db_json.splitlines(),
+            fromfile='NetBox config_db.json configuration',
+            tofile='SONiC running configuration',
+            lineterm='',
+        )
+        diff_frr = difflib.unified_diff(
+            netbox_frr.splitlines(),
+            frr_conf.splitlines(),
+            fromfile='NetBox FRR configuration',
+            tofile='SONiC FRR configuration',
+            lineterm='',
+        )
+        diff_config_db_out = "\n".join(list(diff_config_db))
+        diff_frr_out = "\n".join(list(diff_frr))
+        if diff_config_db_out:
+            self.log_warning("Differences found between NetBox and SONiC config_db configuration:")
+            self.log_info(diff_config_db_out)
+        else:
+            self.log_success("No differences found between NetBox and SONiC config_db configuration.")
+
+        if diff_frr_out:
+            self.log_warning("Differences found between NetBox and SONiC FRR configuration:")
+            self.log_info(diff_frr_out)
+        else:
+            self.log_success("No differences found between NetBox and SONiC FRR configuration.")