Beyond the internal
VPP/Linux configuration generated by Contiv and submitted
to Ligato VPP Agent to facilitate connectivity
between pods and nodes, it is also possible to post additional external
network
configuration from other processes to re-use and extend the same data plane
with custom network features.
Currently, Contiv/VPP provides two interfaces - denoted as external configuration sources
- through which outside processes can request additional configuration
to be applied:
etcd
datastore- gRPC API
A new external configuration source can be easily added in the form of a new Contiv plugin, learn more here.
Contiv/VPP uses etcd
to reflect and delegate Kubernetes state data across
all instances of contiv-vswitch
running in the vswitch (keys prefixed with
/vnf-agent/contiv-ksr/k8s/
). The same etcd
instance is also available for
external NB configuration to be forwarded directly to the VPP agent.
With sufficient privileges, an outside process can connect to etcd
, exposed
as a NodePort
service on port 32379 and put configuration to be applied.
The format of key-value pairs representing configuration items is exactly
the same as the NB API v2 of the vpp-agent.
The proto-based models for values as well as methods to compose keys can be found
here and here for VPP and Linux objects, respectively.
Every key has to be prefixed with /vnf-agent/<node-hostname>/
to specify
the node it is targeted for.
The main limitation of using etcd
as the external configuration source is that
there is no feedback from Contiv/VPP-agent to learn the state of the configuration
from. Also, it is not possible to group a series of configuration changes
into one transaction, they are always delivered to the agent one change at a time.
The advantage of using etcd
is that the configuration is automatically
persisted and can be easily read anytime using etcdctl
, making developer's life
easier in general. Agent is also able to resync against it anytime it needs,
but the external process itself is not able to initiate the resync procedure.
An external configuration can be submitted to Contiv/VPP also via a remote procedure call (RPC), specifically using the gRPC framework, taking advantage of the fact that VPP-Agent NB API is already modeled using Procotol Buffer.
The gRPC API is exposed on every Contiv node on the port 9111. The API is split into two services:
DataResyncService
: allows the client to re-synchronize with Contiv by sending a full snapshot of the external configuration that is supposed to be applied at the given moment. Typically, the resync is triggered on (re-)connect to ensure that client and server operate with the same state moving forward, but it is not mandatory. If not used, the agent will assume an empty configuration state for the gRPC source, or will work with whatever has remained from previous connections.DataChangeService
: allows the client to execute incremental change. The service exposes two methods:Put
to create/update one or multiple configuration items andDel
to remove the set of defined objects.
All RCP methods are blocking and do not return until the corresponding transaction has finalized. The returned error, if not nil, is an instance of TransactionError informing about which items have failed to be (re-)configured. The agent is set to apply the changes in a best-effort mode - i.e. not reverting failed transactions, instead periodically re-trying failed items in the background until they are fixed, changed by another transaction, or the number of retry attempts has reached the configured limit.
TODO: add option to select with-revert behaviour if requested
For go clients, we provide an adapter to hide gRPC API behind the more familiar clientv2 interface (unifying different means of accessing the VPP-agent NB). An example application demonstrating combination of clientv2 with gRPC for Contiv can be found here.
Behind the scenes, the gRPC support for Contiv is provided by the contiv-grpc
plugin. To support (server-initiated) re-sync after vswitch restart also for
the external configuration, the plugin persists an up-to-date snapshot of the
configuration received from gRPC clients to Bolt DB at the file path
/var/bolt/grpc.db
(mounted between contiv-vswitch
and the host).
Internally in Contiv, a support for external configuration is generic enough
to allow developers to easily extend the set of available interfaces beyond etcd
and gRPC
. In fact, the gRPC
interface is already decoupled from the Contiv
core and provided as a separate plugin.
A new source of external configuration can be added as a separate plugin by
implementing the ExternalConfigSource
interface defined by the controller plugin.
The plugin then needs to be injected into the ExtSources
slice inside
the controller's dependencies (member Deps
). To request a change in the external
configuration or trigger a (client-initiated) resync, the plugin should send
events ExternalConfigChange
and ExternalConfigResync
, respectively, into the
main event loop and potentially wait for the result to propagate back to the client.
The events with external configuration changes are received and processed
by the controller plugin. Normally, they are just delegated further straight
into the VPP agent. In case there is a collision between internal (from Contiv)
and external configuration (both modifying the same key), the Controller has
to first merge the colliding values using proto.Merge
, where destination
is the internal value and source is the external one. In most cases this
yields the desired outcome - for example, one may easily add another IP address
to an interface already created by Contiv - the slices of IP addresses are simply
added together. But when it comes to removing/replacing items or un-seting scalar
attributes, the solution based on proto.Merge comes short. Generally, while
external configuration sources allow outside processed to re-use the same data
plane, it is recommended to configure new items in separate namespaces whenever
possible (e.g. separate VRFs, custom links between vswitch and the pods, etc.)