Skip to content

Commit

Permalink
update doc
Browse files Browse the repository at this point in the history
  • Loading branch information
bytinbit committed May 17, 2021
1 parent 16562bc commit 137c94c
Show file tree
Hide file tree
Showing 8 changed files with 161 additions and 160 deletions.
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Introduction

The NetTowel Network Unit Testing System or "nuts" in short is the testing component of the NetTowel Project.
The NetTowel Network Unit Testing System or "nuts" in short is the testing component of the NetTowel Project, which is developed at the Institute of Networked Solutions in Rapperswil, Switzerland.
It draws on the concept of unit tests, known from the domain of programming, and applies it to the domain of networking.

One major difference between unit tests in programming and
Expand Down Expand Up @@ -132,13 +132,15 @@ but also be provided as argument to the test method itself.
Each test module implements a context class to provide module-specific functionality to its tests. This context class is a `NutsContext` or a subclass of it.
This guarantees a consistent interface across all tests for test setup and execution.
Currently, the predefined test classes use [nornir](https://nornir.readthedocs.io/en/latest/) in order to communicate
with the network devices, therefore the test classes derive all from a more specific `NornirNutsContext`,
with the network devices. Those test classes derive all from a more specific `NornirNutsContext`,
which provides a nornir instance and nornir-specific helpers. In the example above, it is a class called `CdpNeighborsContext` that derives from `NornirNutsContext`.

If you want to learn more how nuts works but do not have a running network in the background, there's a nuts showcase - an offline test class that displays the basic functionality of nuts. See the [tutorial](https://nuts.readthedocs.io/en/latest/tutorial/firststeps.html) for further information.

## Develop Your Own Test Classes

Nuts is essentially designed as a pytest-plugin and it is possible to add your own, self-written test classes.
A dev documentation on how to write your own test classes is planned for a future release.
A dev documentation on how to write your own test classes is planned for a future release.
Still, it is possible to write your own test classes nevertheless, even if we cannot guarantee that upcoming planned refactorings do not introduce breaking changes.

# Thanks
Expand Down
7 changes: 4 additions & 3 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,19 @@
#
import os
import sys
import datetime

sys.path.insert(0, os.path.abspath("../.."))
sys.setrecursionlimit(1500)

# -- Project information -----------------------------------------------------

project = "NUTS"
copyright = "2020, Matthias Gabriel, Méline Sieber, Urs Baumann"
author = "Matthias Gabriel, Méline Sieber, Urs Baumann"
copyright = f"{datetime.date.today().year}, Méline Sieber, Urs Baumann"
author = "Méline Sieber"

# The full version, including alpha/beta/rc tags
release = "0.1.0"
release = "1.0.0"


# -- General configuration ---------------------------------------------------
Expand Down
6 changes: 0 additions & 6 deletions docs/source/dev/index.rst

This file was deleted.

46 changes: 22 additions & 24 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,41 +6,40 @@ Documentation of NUTS
Introduction
------------

The NetTowel Network Unit Testing System or "NUTS" for short is the
testing component of the NetTowel Project, which is developed at the Institute of Networked Solutions in Rapperswil, Switzerland.
NUTS draws on the concept of unit tests, known from the domain of
software development, and applies it to the domain of networking.

One major difference between unit tests in software development and
network tests is the definition of a test.
In software development, unit tests normally focus on testing edge cases,
since the amount of non-edge cases is not definable.
In the network testing domain, tests are less about edge cases, but more about testing network functionalities with pre-defined test cases. Such a single test case might be "can host A
reach neighbors X, Y, Z?" or "has host A all BGP neighbors configured correctly?" on many different devices.

This is what NUTS tries to achieve:
Use pre-defined test cases together with your network topology, apply this to your actual network and have the tests confirm that the network has the expected functionalities.

How NUTS works
The NetTowel Network Unit Testing System or "nuts" in short is the testing component of the NetTowel Project, which is developed at the Institute of Networked Solutions in Rapperswil, Switzerland.
It draws on the concept of unit tests, known from the domain of programming, and applies it to the domain of networking.

One major difference between unit tests in programming and
network tests is the definition of what a test actually is.
In programming, unit tests normally focus on testing edge cases,
since the amount of non-edge cases is not definable.
In the network testing domain, tests are less about edge cases, but more about testing existing network states with
pre-defined test cases. Such a single test case might be "can host A reach neighbors X, Y, Z?" on many different devices.
This is what nuts tries to achieve:
Apply test cases based on your pre-defined network topology to your actual network and have the tests confirm the correct state.

How nuts works
--------------

In order to run NUTS, two components are required:
In order to run nuts, two components are required:

#. **Inventory of the network**. Currently, these are YAML-files in the form of a `nornir inventory <https://nornir.readthedocs.io/en/latest/tutorial/inventory.html>`__. They contain all details of your network, such as hosts, login-information and other configuration.

#. **Test bundles** in the form of YAML-files that specify the actual tests. A test bundle is a series of tests that are logically related to each other. Each test bundle is structured in a similar way, but has specific fields tailored to the test bundle. Some field values in a test bundle are directly related to your inventory.

Head over to the :doc:`Usage section <tutorial/firststeps>` to see how those two components are structured and how you get NUTS up and running.
Head over to the :doc:`Usage section <tutorial/firststeps>` to see how those two components are structured and how you get nuts up and running.

The project relies on the `pytest framework <https://docs.pytest.org/>`__ to setup and execute the
tests. NUTS itself is written as a custom pytest plugin. In the background, `nornir <https://nornir.readthedocs.io/>`__ executes specific network tasks for the actual tests.
tests. Nuts itself is written as a custom pytest plugin. In the background, `nornir <https://nornir.readthedocs.io/>`__ executes specific network tasks for the actual tests.

Pytest reads in the test bundle (2.) and transforms it into test runs. In the background, nornir uses the network information provided in the inventory (1.), queries the network based on the specific test bundle and passes the results of those queries to pytest. Pytest then evaluates if the expectations defined in the test bundle match those results.
Pytest reads in the test bundle (step 2 above) and transforms it into test runs. In the background, nornir uses the network information provided in the inventory (step 1 above), queries the network based on the specific test bundle and passes the results of those queries to pytest. Pytest then evaluates if the expectations defined in the test bundle match those results.

Enhance NUTS
Enhance nuts
------------
Since NUTS is written as a pytest plugin and in python, you can customize it yourself and write your own test classes. Please see the :doc:`development section <dev/index>` to see how NUTS is structured and how to write your own test classes.
Nuts is written in python and designed as a pytest plugin. It provides some base tests described in :doc:`the section about all test bundles <testbundles/alltestbundles>`, but since it's a plugin, you can write your own, self-written test classes for your custom tests.
A dev documentation on how to write your own test classes is planned for a future release.

We do not support self-written tests at the moment, since planned refactorings might introduce breaking changes. Still, you can already write your own test classes - see the ``nuts/base_tests`` folder in the code repository to learn how test classes are written.

Contents
========
Expand All @@ -49,10 +48,9 @@ Contents
:maxdepth: 2

Home <self>
Installation <installation/index>
Installation <installation/install>
Tutorial <tutorial/firststeps>
Test Bundles <testbundles/alltestbundles>
Development <dev/index>


Indices and tables
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,29 +4,25 @@ Installation of NUTS
Installation Instructions
-------------------------

NUTS is currently not published on the Python Package Index (`PyPI <https://pypi.org/>`_). It has to be cloned and installed manually.
Nuts requires Python 3.7 or higher.

.. todo::

Use public repository to access code and retest the installation instructions.

.. code:: shell
Installation via pip
....................

git clone <public repository>
NUTS is published on the Python Package Index (`PyPI <https://pypi.org/>`_), therefore you can install nuts using ``pip``:

cd nettowel-nuts
.. code:: shell
# create virtual environment
python -m venv .venv
pip install nuts
# activate virtual environment
source .venv/bin/activate
Installation via poetry
.......................

# install NUTS
pip install <your_nuts_directory>
Nuts uses `poetry <https://python-poetry.org/>`__ as a dependency manager.

# install NUTS directly from git without clone first
pip install git+https://github.com/INSRapperswil/Nuts.git
1. `Install poetry <https://python-poetry.org/docs/#installation>`__.
2. Clone the `nuts repository <https://github.com/INSRapperswil/Nuts.git>`__
3. Run ``$ poetry install``

Parsing with ntc-templates
--------------------------
Expand Down Expand Up @@ -54,7 +50,7 @@ Deinstallation

.. code:: shell
pip uninstall nettowel-nuts
pip uninstall nuts
If you installed everything in a virtual environment, you can delete the folder that contains the virtual environment.
2 changes: 1 addition & 1 deletion docs/source/testbundles/alltestbundles.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ A test bundle contains one ore more tests that are logically related to each oth

This section contains all test bundles which have been implemented in NUTS, you can incorporate them in your own bundles. They can be executed with the command ``$ pytest <test>.yaml`` from your project root.

Note that you need an inventory for the tests to work. Please see :doc:`First Steps with NUTS <../tutorial/firststeps>` for more information.
Note that you need an inventory of network devices for the tests to work. Please see :doc:`First Steps with NUTS <../tutorial/firststeps>` for more information.

In some test bundles you can directly pass arguments to the nornir task, i.e. the network query that is executed in the background. For those test bundles we indicate the specific task which is used to query the devices, so that you can look up all available arguments.

Expand Down
45 changes: 33 additions & 12 deletions docs/source/tutorial/firststeps.rst
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
First Steps with NUTS
=====================

This tutorial guides you through a minimal setup of the NetTowel Unit Testing System (NUTS, or nuts).
This tutorial guides you through a minimal setup of the NetTowel Unit Testing System (NUTS, or nuts). A showcase is included in nuts' code that allows you to learn the basic mechanics of nuts without the need of an actual network -- find those details at the end of this tutorial.

Two major components are needed for nuts:

#. A network inventory
#. Test bundles

Here's an overview on how to organise your files so that NUTS can find everything it needs. The root folder also contains the virtual environment with the :doc:`NUTS installation <../installation/index>`.
Here's an overview on how to organise your files so that NUTS can find everything it needs. The root folder also contains the virtual environment with the :doc:`nuts installation <../installation/install>`.

.. code:: shell
Expand All @@ -22,7 +22,7 @@ Here's an overview on how to organise your files so that NUTS can find everythin
1. Network Inventory
--------------------

You must provide information on your network configuration so that NUTS can actually interact with it. Currently, NUTS uses a `nornir inventory <https://nornir.readthedocs.io/en/latest/tutorial/inventory.html>`__ and ``nr-config.yaml`` which uses that inventory.
You must provide information on your network configuration so that nuts can actually interact with it. Currently, NUTS uses a `nornir inventory <https://nornir.readthedocs.io/en/latest/tutorial/inventory.html>`__ and ``nr-config.yaml`` which uses that inventory.

A sample ``hosts.yaml`` might look like this:

Expand Down Expand Up @@ -69,9 +69,9 @@ If you set up the above folders and files, you're ready to write test bundles.
2. Test Bundle
--------------

A test bundle is a collection of tests are logically related to each other, for example tests that all revolve around "information on BGP neighbors". The test bundle describes which test definition should be collected and executed and provides data for those tests. The bundles are written as individual entries in a YAML file.
A test bundle is a collection of tests that are logically related to each other, for example tests that all revolve around "information on BGP neighbors". The test bundle describes which test definition should be collected and executed and provides data for those tests. The bundles are written as individual entries in a YAML file.

Currently only YAML files are supported as test bundle format, but other data sources could be integrated in later versions of nuts.
Currently only YAML files are supported as test bundle format, but other data sources could be integrated in later versions of nuts.

Structure of a Test Bundle
**************************
Expand All @@ -86,7 +86,7 @@ Each test bundle contains the following structure:
test_execution: <additional data used to execute the test> # optional
test_data: <data used to generate the test instances>
``test_module``: Optional. The full path of the python module that contains the test class to be used. This value is optional if the test class is registered in ``index.py`` of the pytest-nuts plugin. Note that it can be relevant in which directory ``pytest`` is started if local test modules are used.
``test_module``: Optional. The full path of the python module that contains the test class to be used. This value is optional if the test class is registered in ``index.py`` of the pytest-nuts plugin. Note that it can be relevant in which directory ``pytest`` is started if local test modules are used. Using ``test_modules`` allows you to write your own test classes. **Note: We currently do not support self-written test modules, since upcoming refactorings might introduce breaking changes.**

``test_class``: Required. The name of the python class which contains the tests that should be executed. Note that currently every test in this class is executed.

Expand Down Expand Up @@ -124,16 +124,37 @@ Notes:
* ``test_data.max_drop: 1``. Maximum one ping attempt is allowed to fail to still count as SUCCESS ping.


We save this file as ``test-definition-ping.yaml`` into the ``tests`` folder.

Run NUTS
--------

If everything is set up as shown above, run the test from the root folder:
We save this file as ``test-definition-ping.yaml`` into the ``tests`` folder. If everything is set up as shown above, run the test from the root folder:

.. code:: shell
$ pytest tests/test-definition-ping.yaml
Pytest's output should then inform you if the test succeeded or not.


Sample Test-Bundle Without a Network
************************************

The sample test bundle above requires a network inventory and a running network in the background. In case you want to learn how nuts works but do not have a network at hand, nuts comes with an offline showcase to display its functionality. Use it as follows:

#. Clone the `nuts repository <https://github.com/INSRapperswil/nuts>`__ and change into the cloned folder.
#. Create a `virtual environment (venv) <https://docs.python.org/3/library/venv.html>`__ in it and activate it.
#. Install nuts in the venv.
#. Run the showcase test bundle.

.. code:: shell
$ git clone https://github.com/INSRapperswil/nuts && cd nuts
$ python -m venv .venv && source .venv/bin/activate
$ pip install .
$ pytest tests/showcase_test/test-expanse.yaml
How it works: Each test module implements a context class to provide module-specific functionality to its tests. This context class is a ``NutsContext`` or a subclass of it. This guarantees a consistent interface across all tests for test setup and execution.

The predefined test classes which depend on a network all use `nornir <https://nornir.readthedocs.io/en/latest/>`__ in order to communicate with the network devices. Those test classes all derive all from a more specific ``NornirNutsContext``, which provides a nornir instance and nornir-specific helpers.

In order for the offline showcase to work, the test class derives from ``NutsContext`` and implements its own context class. See the code in ``nuts/tests/showcase/showcase_expanse.py`` to see the structure of this offline context class.



Loading

0 comments on commit 137c94c

Please sign in to comment.