Skip to content

Latest commit

 

History

History
126 lines (104 loc) · 7.29 KB

EXTERNAL_CONFIG.md

File metadata and controls

126 lines (104 loc) · 7.29 KB

Extending Contiv/VPP with external configuration

Beyond the internal VPP/Linux configuration generated by Contiv and submitted to Ligato VPP Agent to facilitate connectivity between pods and nodes, it is also possible to post additional external network configuration from other processes to re-use and extend the same data plane with custom network features.

Currently, Contiv/VPP provides two interfaces - denoted as external configuration sources - through which outside processes can request additional configuration to be applied:

  1. etcd datastore
  2. gRPC API

A new external configuration source can be easily added in the form of a new Contiv plugin, learn more here.

Etcd datastore

Contiv/VPP uses etcd to reflect and delegate Kubernetes state data across all instances of contiv-vswitch running in the vswitch (keys prefixed with /vnf-agent/contiv-ksr/k8s/). The same etcd instance is also available for external NB configuration to be forwarded directly to the VPP agent.

With sufficient privileges, an outside process can connect to etcd, exposed as a NodePort service on port 32379 and put configuration to be applied. The format of key-value pairs representing configuration items is exactly the same as the NB API v2 of the vpp-agent. The proto-based models for values as well as methods to compose keys can be found here and here for VPP and Linux objects, respectively. Every key has to be prefixed with /vnf-agent/<node-hostname>/ to specify the node it is targeted for.

The main limitation of using etcd as the external configuration source is that there is no feedback from Contiv/VPP-agent to learn the state of the configuration from. Also, it is not possible to group a series of configuration changes into one transaction, they are always delivered to the agent one change at a time.

The advantage of using etcd is that the configuration is automatically persisted and can be easily read anytime using etcdctl, making developer's life easier in general. Agent is also able to resync against it anytime it needs, but the external process itself is not able to initiate the resync procedure.

gRPC API

An external configuration can be submitted to Contiv/VPP also via a remote procedure call (RPC), specifically using the gRPC framework, taking advantage of the fact that VPP-Agent NB API is already modeled using Procotol Buffer.

The gRPC API is exposed on every Contiv node on the port 9111. The API is split into two services:

  1. DataResyncService: allows the client to re-synchronize with Contiv by sending a full snapshot of the external configuration that is supposed to be applied at the given moment. Typically, the resync is triggered on (re-)connect to ensure that client and server operate with the same state moving forward, but it is not mandatory. If not used, the agent will assume an empty configuration state for the gRPC source, or will work with whatever has remained from previous connections.
  2. DataChangeService: allows the client to execute incremental change. The service exposes two methods: Put to create/update one or multiple configuration items and Del to remove the set of defined objects.

All RCP methods are blocking and do not return until the corresponding transaction has finalized. The returned error, if not nil, is an instance of TransactionError informing about which items have failed to be (re-)configured. The agent is set to apply the changes in a best-effort mode - i.e. not reverting failed transactions, instead periodically re-trying failed items in the background until they are fixed, changed by another transaction, or the number of retry attempts has reached the configured limit.

TODO: add option to select with-revert behaviour if requested

For go clients, we provide an adapter to hide gRPC API behind the more familiar clientv2 interface (unifying different means of accessing the VPP-agent NB). An example application demonstrating combination of clientv2 with gRPC for Contiv can be found here.

Behind the scenes, the gRPC support for Contiv is provided by the contiv-grpc plugin. To support (server-initiated) re-sync after vswitch restart also for the external configuration, the plugin persists an up-to-date snapshot of the configuration received from gRPC clients to Bolt DB at the file path /var/bolt/grpc.db (mounted between contiv-vswitch and the host).

External configuration source

Internally in Contiv, a support for external configuration is generic enough to allow developers to easily extend the set of available interfaces beyond etcd and gRPC. In fact, the gRPC interface is already decoupled from the Contiv core and provided as a separate plugin.

A new source of external configuration can be added as a separate plugin by implementing the ExternalConfigSource interface defined by the controller plugin. The plugin then needs to be injected into the ExtSources slice inside the controller's dependencies (member Deps). To request a change in the external configuration or trigger a (client-initiated) resync, the plugin should send events ExternalConfigChange and ExternalConfigResync, respectively, into the main event loop and potentially wait for the result to propagate back to the client.

The events with external configuration changes are received and processed by the controller plugin. Normally, they are just delegated further straight into the VPP agent. In case there is a collision between internal (from Contiv) and external configuration (both modifying the same key), the Controller has to first merge the colliding values using proto.Merge, where destination is the internal value and source is the external one. In most cases this yields the desired outcome - for example, one may easily add another IP address to an interface already created by Contiv - the slices of IP addresses are simply added together. But when it comes to removing/replacing items or un-seting scalar attributes, the solution based on proto.Merge comes short. Generally, while external configuration sources allow outside processed to re-use the same data plane, it is recommended to configure new items in separate namespaces whenever possible (e.g. separate VRFs, custom links between vswitch and the pods, etc.)