update docs

app/generate.go

@@ -515,7 +515,7 @@ func resolveGlobs(globs []string) ([]string, error) {
 func walkDir(path, ext string) ([]string, error) {
 	fs := make([]string, 0)
 	err := filepath.Walk(path,
-		func(path string, info os.FileInfo, err error) error {
+		func(path string, _ os.FileInfo, err error) error {
 			if err != nil {
 				return err
 			}

app/gnmi_server.go

@@ -31,7 +31,7 @@ import (
 type streamClient struct {
 	target string
 	req    *gnmi.SubscribeRequest
-	
+
 	stream  gnmi.GNMI_SubscribeServer
 	errChan chan<- error
 }

docs/user_guide/caching.md

@@ -1,26 +1,25 @@
 
-`Caching` refers to the additional step of storing the collected gNMI updates before sending them out to their intended output.
-
-A cache is used to store the received updates when the [`gnmi-server`](gnmi_server.md) functionality is enabled as well as with both `influxdb` and `prometheus` outputs to allow for advanced data pipeline processing.
+`Caching` refers to the process of storing the collected gNMI updates before sending them out to the intended output(s).
 
 By default, `gNMIc` outputs send out the received gNMI updates as they arrive (i.e without storing them).
 
-Caching messages before writing them to a remote location can yield a few benefits like **rate limiting**, **batch processing**, **data replication**, etc.
+A cache is used to store the received updates when the [`gnmi-server`](gnmi_server.md) functionality is enabled and (optionally) when `influxdb` and `prometheus` outputs are enabled to allow for advanced data pipeline processing.
+
+Caching messages before writing them to a remote location allows implementing a few use cases like **rate limiting**, **batch processing**, **data replication**, etc.
 
-Both `influxdb` and `prometheus` outputs support caching messages before exporting.
 Caching support for other outputs is planned.
 
 ### How does it work?
 
-When caching is enabled for a certain output, the received messages are not written directly to the output remote server (for e.g: InfluxDB server), but rather cached locally until the `cache-flush-timer` is reached (in the case of an `influxdb` output) or when the output receives a `Prometheus` scrape request (in the case of a `prometheus` output).
+When caching is enabled for a certain output, the received gNMI updates are not written directly to the output remote server (for e.g: InfluxDB server), but rather cached locally until the `cache-flush-timer` is reached (in the case of an `influxdb` output) or when the output receives a `Prometheus` scrape request (in the case of a `prometheus` output).
 
 The below diagram shows how an InfluxDB output works with and without cache enabled:
 
 <div class="mxgraph" style="max-width:100%;border:1px solid transparent;margin:0 auto; display:block;" data-mxgraph="{&quot;page&quot;:10,&quot;zoom&quot;:1.4,&quot;highlight&quot;:&quot;#0000ff&quot;,&quot;nav&quot;:true,&quot;check-visible-state&quot;:true,&quot;resize&quot;:true,&quot;url&quot;:&quot;https://raw.githubusercontent.com/karimra/gnmic/diagrams/diagrams/influxdb_output_with_without_cache.drawio&quot;}"></div>
 
 <script type="text/javascript" src="https://cdn.jsdelivr.net/gh/hellt/drawio-js@main/embed2.js?&fetch=https%3A%2F%2Fraw.githubusercontent.com%2Fkarimra%2Fgnmic%2Fdiagrams%2F/influxdb_output_with_without_cache.drawio" async></script>
 
-When caching is enabled, the cached gNMI updates are periodically retrieved from the cache in batch then converted to [events](event_processors/intro.md#the-event-format).
+The cached gNMI updates are periodically retrieved from the cache in batch then converted to [events](event_processors/intro.md#the-event-format).
 
 If [processors](event_processors/intro.md) are defined under the output config section, they are applied to the whole list of events at once. This allows for augmentation of messages with values from other messages even if they where received in separate updates or collected from a different target/subscription.
 
@@ -29,7 +28,7 @@ If [processors](event_processors/intro.md) are defined under the output config s
 #### gnmi-server
 
 The gNMI server has caching enabled by default.
-The type of cache and its behavior can be tweaked, see [here](#cache-types)
+The cache type and its behavior can be tweaked, see [here](#cache-types)
 
 ```yaml
 gnmi-server:
@@ -41,7 +40,7 @@ gnmi-server:
 
 #### outputs
 
-Caching can be enabled per output by adding the following configuration snippet under the desired output:
+Caching can be enabled per output by populating the `cache` attribute under the desired output:
 
 ```yaml
 outputs:
@@ -62,23 +61,34 @@ Using a single global cache will be implemented in a future release.
 
 When running multiple instances of `gNMIc` it's possible to synchronize the collected data between all the instances using a distributed cache.
 
-Each output configured with a remote cache will write the collected gNMI update to the cache first, then read back all written data to process it and eventually write it to the output server.
+Each output that is configured with a remote cache will write the collected gNMI updates to the remote cache first, then syncs back all the cached data to its local cache then eventually write it to the output.
 
 <div class="mxgraph" style="max-width:100%;border:1px solid transparent;margin:0 auto; display:block;" data-mxgraph="{&quot;page&quot;:10,&quot;zoom&quot;:1.4,&quot;highlight&quot;:&quot;#0000ff&quot;,&quot;nav&quot;:true,&quot;check-visible-state&quot;:true,&quot;resize&quot;:true,&quot;url&quot;:&quot;https://raw.githubusercontent.com/karimra/gnmic/diagrams/diagrams/distributed_caches.drawio&quot;}"></div>
 
 <script type="text/javascript" src="https://cdn.jsdelivr.net/gh/hellt/drawio-js@main/embed2.js?&fetch=https%3A%2F%2Fraw.githubusercontent.com%2Fkarimra%2Fgnmic%2Fdiagrams%2F/distributed_caches.drawio" async></script>
 
+(1) The received gNMI updates are written to the remote cache.
+
+(2) The output syncs the remote cache data to its local cache.
+
+(3) The locally cached data is written to the remote output periodically or on scape request.
+
 This is useful when different instances collect data from different targets and/or subscriptions. A single instance can be responsible for writing all the collected data to the output or each instance would be writing to a different output.
 
 ### Cache types
 
-`gNMIc` supports 4 cache types. The choice of cache to use depends on the use case you are trying to achieve.
+`gNMIc` supports 4 cache types. There is 1 local cache and 3 distributed caches "flavors".
+
+The choice of cache to use depends on the use case you are trying to implement.
+
+A local cache is local to the `gNMIc` instance i.e not exposed externally,
+while a distributed cache is external to the `gNMIc` instance, potentially shared by multiple `gNMIc` instances and is always combined with a local cache to sync updates between `gNMIc` instances.
 
-#### gNMI cache
+#### gNMI cache (local)
 
 Is an in-memory gNMI cache based on the Openconfig gNMI cache published [here](https://github.com/openconfig/gnmi/tree/master/cache)
 
-This type of cache is ideal when running a single `gNMIc` instance. It is also the default cache type when caching is enabled.
+This type of cache is ideal when running a single `gNMIc` instance. It is also the default cache type for the gNMI server and for an output when caching is enabled.
 
 Configuration:
 
@@ -98,7 +108,7 @@ outputs:
       debug: false
 ```
 
-#### NATS cache
+#### NATS cache (distributed)
 
 Is a cache type that relies on a [NATS server](https://docs.nats.io/) to distribute the collected updates between `gNMIc` instances.
 
@@ -128,7 +138,7 @@ outputs:
       debug: false
 ```
 
-#### JetStream cache
+#### JetStream cache (distributed)
 
 Is a cache type that relies on a [JetStream server](https://docs.nats.io/nats-concepts/jetstream) to distribute the collected updates between `gNMIc` instances.
 
@@ -173,7 +183,7 @@ outputs:
       debug: false      
 ```
 
-#### Redis cache
+#### Redis cache (distributed)
 
 Is a cache type that relies on a [Redis PUBSUB server](https://redis.io/docs/manual/pubsub/) to distribute the collected updates between `gNMIc` instances.
 

docs/user_guide/gnmi_server.md

@@ -122,7 +122,7 @@ In this subscription mode, `gNMIc` server supports the `updates-only` knob.
 
 When a subscription is defined to be `on-change`, data updates are only sent to the client when the value of the data item changes.
 
-In the case of `gNMIc` gNMI server, `on-change` subscriptions depend on the subscription writing data in the local cache,
+In the case of `gNMIc` gNMI server, `on-change` subscriptions depend on the subscription writing data to the local cache,
 if it is a `sample` subscription, each update from a target will trigger an `on-change` update to the server client.
 
 `gNMIc` gNMI server supports `on-change` subscriptions with `heartbeat-interval`.
@@ -206,6 +206,33 @@ gnmi-server:
     # if available, the instance-name and cluster-name will be added as tags,
     # in the format: gnmic-instance=$instance-name and gnmic-cluster=$cluster-name
     tags:
+  # cache configuration
+  cache:
+    # cache type, defaults to `oc`
+    type: oc
+    # string, address of the remote cache server,
+    # irrelevant if type is `oc`
+    address:
+    # string, the remote server username.
+    username:
+    # string, the remote server password.
+    password:
+    # string, expiration period of received messages.
+    expiration: 60s
+    # enable extra logging
+    debug: false
+    # int64, default: 1073741824 (1 GiB). 
+    # Max number of bytes stored in the cache per subscription.
+    max-bytes:
+    # int64, default: 1048576. 
+    # Max number of messages stored per subscription.
+    max-msgs-per-subscription:
+    # int, default 100. 
+    # Batch size used by the JetStream pull subscriber.
+    fetch-batch-size:
+    # duration, default 100ms. 
+    # Wait time used by the JetStream pull subscriber.
+    fetch-wait-time:  
 ```
 
 ### Secure vs Insecure Server
@@ -321,3 +348,50 @@ Enables the collection of Prometheus gRPC server metrics.
 #### debug
 
 Enables additional debug logging.
+
+## Caching
+
+By default, the gNMI server uses Openconfig's gNMI cache as a backend.
+
+Distributed caching is supported using any of the other cache types specified [here](caching.md#cache-types).
+
+<div class="mxgraph" style="max-width:100%;border:1px solid transparent;margin:0 auto; display:block;" data-mxgraph="{&quot;page&quot;:0,&quot;zoom&quot;:1.4,&quot;highlight&quot;:&quot;#0000ff&quot;,&quot;nav&quot;:true,&quot;check-visible-state&quot;:true,&quot;resize&quot;:true,&quot;url&quot;:&quot;https://raw.githubusercontent.com/karimra/gnmic/diagrams/diagrams/gnmi_server_cache.drawio&quot;}"></div>
+
+<script type="text/javascript" src="https://cdn.jsdelivr.net/gh/hellt/drawio-js@main/embed2.js?&fetch=https%3A%2F%2Fraw.githubusercontent.com%2Fkarimra%2Fgnmic%2Fdiagrams%2Fgnmi_server_cache.drawio" async></script>
+
+When a distributed cache is used together with the gNMI server feature, a gNMI client can subscribe to any of the gNMI servers to get gNMI updates collected from all the targets.
+
+On the other hand, if the gNMI client sends a unary RPC (Get, Set), it will have be directed to the gNMI server directly connected to the target.
+
+```yaml
+gnmi-server:
+  #
+  # other gnmi-server attributes
+  #
+  cache:
+    # cache type, defaults to `oc`
+    type: oc # redis, nats or jetstream
+    # string, address of the remote cache server,
+    # irrelevant if type is `oc`
+    address:
+    # string, the remote server username.
+    username:
+    # string, the remote server password.
+    password:
+    # string, expiration period of received messages.
+    expiration: 60s
+    # enable extra logging.
+    debug: false
+    # int64, default: 1073741824 (1 GiB). 
+    # Max number of bytes stored in the cache per subscription.
+    max-bytes:
+    # int64, default: 1048576. 
+    # Max number of messages stored per subscription.
+    max-msgs-per-subscription:
+    # int, default 100. 
+    # Batch size used by the JetStream pull subscriber.
+    fetch-batch-size:
+    # duration, default 100ms. 
+    # Wait time used by the JetStream pull subscriber.
+    fetch-wait-time:
+```