Enhanced Soft RoCE (RXE) Driver
==================================

The driver build current only supports SLES SP2 and Ubuntu 20.10
environments.

On SLES, if MOFED is installed, it must be removed otherwise the RXE
driver will not be functional:

    zypper -n remove rdma-core
    zypper -n install -r SLES15_Module_Basesystem librdmacm1 libibumad3 libibmad5 infiniband-diags SLES15_Module_Server_Applications libibverbs-utils librdmacm-utils ibacm SLES15_Module_Development_Tools perftest


Build:
In top level rxe:

    make

To run:

    modprobe udp_tunnel
    modprobe ip6_udp_tunnel
    modprobe rdma_ucm
    insmod rxe/rxe/rdma_rxe.ko
    rdma link add rxe0 type rxe netdev <hsn_device>

The build consists of 2 parts: a source tarball and a set of patches.

The source tarball is generated by downloading the linux kernel source
package from
https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git This is
the rdma fork of the linux kernel, where changes for RDMA are made.
The tagged source tarball from the kernel archive is expanded, and
just the RXE driver <linux_src>/drivers/infiniband/sw/rxe is extracted
and tarballed to form the rxe-<version>.tar.gz archive.  The
rxe-<version>.tar.gz file is unmodified source from the tagged rdma
branch kernel.

Patches to be applied to the kernel are located in rxe/patches.
Patches are further categorized into several "patch types" by
directory: compatibility, functionality, and upstream.

Compatibility patches are "backrevs" to allow an upstream driver to
work with an older kernel.  When patching the driver, compatibility
patches should be applied first.  Whenever possible, these should be
reversals of patches to the kernel tree, and should be applied in
reverse order to the application of the patches to the kernel
(i.e. driver should be "backed up").  Only changes that must be
reversed are applied (changes to ib driver interfaces for example), so
some patches may need to be modified to accomodate tree changes that
are upstream which are not undone.  In the series file, each patch has
a comment regarding the date when the change that's being undone was
accepted to the rdma fork.  This will help with maintenance by
allowing visibility to when changes were applied as the environment
rolls forward.

Upstream patches are a way to bring "future enhancements" into the
driver.  This is where we incorporate changes that are made (or
targeted) "upstream" of the latest tagged release of the rdma git.
Critical enhancements like performance improvements and bug fixes may
be pulled in prior to being incorporated in a tagged release from
kernel.org.

Functionality patches are patches for changes in the functionality of
the rxe driver we make specifically to support Cassini.  These changes
should be targeted to be merged upstream but may face acceptance
challenges and need to be supported long-term by the team.

The script "setup_rxe.sh" detects if there is an existing rxe
directory, and if not, extracts the tarball and applies the current
patches for the distribution using a quilt series.

Running "make" in the rxe directory runs setup_rxe.sh, and then runs
the make "module" for the rxe module using the standard kernel
process.

Modifying source in rxe is permitted, and "rebuild" by re-invoking
"make" at the top level will incorporate any changes for testing and
development.

The rxe directory and files are not tracked in git, and should not be
checked in.  Instead a patch should be added to an appropriate patches
subdirectory for any changes, and the setup_rxe.sh should be modified
to apply the patch during source extraction.