NMEA Multiplexer

Summary of the links used in this document:

Get started, fast
Manual
Note on WebSocket
Demos
Examples
Case Study. Comparison with Node-RED
Extract custom data

Scripts:
The scripts you will need to run the Multiplexer are tomux.sh, and mux.sh. You might need to tweak them to fit your context. Some pre-built examples are given in the RESTNavServer module, which is itself based on this Multiplexer.
Do also see the section Build, package, run, in this document.

Note: This project was tested and used at sea, in the real world.

The main method is in nmea.mux.GenericNMEAMultiplexer.

For the impatient: Get started, fast.

At the center of the picture, we have the NMEA-Multiplexer
- This is also hosting an optional HTTP server, implementing REST and Web interfaces to the data.
Then you have Consumers (aka channels), bringing in NMEA Data
- File channel (to replay logged data)
- Serial channel, to read a Serial port
- TCP, REST, MQTT, WebSockets, etc..., whatever can provide NMEA sentences
- etc...
Then you possibly have Computers, calculating extra data, re-injected into the NMEA-multiplexer
- Current and True Wind calculated with GPS data (SOG, COG)
- Custom ones
- etc...
Then you have Forwarders, spitting out NMEA data (and others, sometimes)
- File Forwarder, to log data
- Character-mode console
- REST Forwarder, to feed other servers
- TCP and others to feed other soft programs (like OpenCPN, SeaWi, etc)
- etc...

NMEA channels management, in and out.

This is - obviously - a software multiplexer (no hardware required), that can read data from multiple sources, compute extra data (like current) if needed, and broadcast them onto (possibly) several output destinations.

For example:

Read one or more serial ports (GPS, boat data), write them into a log file
Read a serial port, read a BME280 sensor (air temperature, atmospheric pressure, relative humidity), and broadcast everything on a TCP port, for OpenCPN.
Read a file of logged data, and send them over TCP to OpenCPN, for replay.
etc...

NMEA

NMEA (National Marine Electronics Association) is one of the oldest IT standards, defining how sensor data should be conveyed.
Another Good Resource, by the author of GPSd.

Note: here is an opened question:

Some software products are using more "modern" technologies to shape the data (like GPSd, SignalK, ..., they use JSON over REST/HTTP), but what is the point? text/plain has always been a supported mime-type, suitable for NMEA Sentences...

The conveyed data are NMEA data, and to be able to communicate with other softs, their (proprietary?) format will eventually have to be converted back to NMEA... Here we chose not to convert anything. NMEA remains NMEA, which everyone understands. And everyone's happy.

NMEA Parsers are available in OpenSource, in several languages. In this project, you'll find at least one in Java, and for ES6. Plus one in Python (WiP), in the Java-TCP-Python module, called NMEASerialTest.py.

This being said... You will see in this project that there is a data-cache that can be used, at least, in the case of the REST Server (also part of this project, and heavily used in the RESTNavServer). This cache can be pinged and used from REST requests; as such, it supports a JSON format..., the requirements sound a lot like the ones SignalK is addressing.

The remaining point would be to agree on the JSON Schema to use, but there is no standard about that yet.

This NMEAMultiplexer will deal with NMEA Data, in input, as well as for output, in order to preserve - as much as possible - compatibility with other programs. Everyone understands NMEA.
The "only" exception to this rule will be the structure of the data cache, which brings us back to the note above...

This NMEAMultiplexer can

use many input protocols (File, Serial, TCP, UDP, WebSockets, Sensors, Computations, ...)
produce many outputs (File, Serial, TCP, UDP, WebSockets...)
and use a REST API on top of that.

At the heart of this lies a small http server (part of this project), designed to run on very small boards, like a Raspberry Pi Zero, and with possibly no Internet access.

Warning ⚠️: It is not designed to scale as an enterprise server!

The operations on the Serial port(s) require libRxTx. This is included in the gradle dependencies. To be able to use it outside gradle, run (on Linux/Debian/Raspberry Pi):

 sudo apt-get install librxtx-java

See how this is used and referred to in mux.sh (the launching script).

For the impatient: go here to get started now. The others will keep reading.

Includes

NMEA Strings Parser
AIS Strings Parser (WiP)
NMEA Strings generator
Serial port Reader / Writer
TCP Reader / Writer
REST Reader / Writer
UDP Reader / Writer
WebSockets client (read/write)
... and more.

Also includes

A tiny (but amazing) HTTP REST server, for admin and external access (through REST services) purpose.

Javadoc

Javadoc can be generated by using

 ../../gradlew javadoc

The generated files will be in build/docs/javadoc. Open the index.html in your browser.

Some definitions

At the center of the picture, there is the Multiplexer (aka MUX).

The MUX reads from channels and writes to forwarders.

A channel is an NMEA data provider, a forwarder is an NMEA data producer, it can be used as input from another Multiplexer.

Just to insist: A channel is an NMEA data provider. This means that it produces (pure) NMEA Sentences!! This is pure NMEA. Nothing fancy like a JSON Object or whatever object representing the same data in another format. NMEA is already a well known and defined format.

In addition, we can have sensors. A sensor is reading data from a transducer, and produces NMEA sentences read by the Mux. In other words, a sensor is talking to the Multiplexer, and can be seen as a channel.

Also, a computer is using NMEA data collected by the Multiplexer to produce other NMEA data that will be broadcasted by the forwarders. For example, True Wind computed with Apparent Wind data and the GPS data.

Note: to compute the required data, we have a cache, where the data required by the computers are pushed. This cache is initialized before starting the computers, with parameters contained in the properties file used at startup.

This cache is necessary to perform damping and smoothing operations - among others.

Finally, we have transformers, that transform NMEA data into another (proprietary) format, and then behave like a a regular forwarder to provide them to whoever is interested. A transformer is also a forwarder. See below examples of transformers.

Examples

Channels (aka Consumers):

Serial reads NMEA data from a Serial Port
TCP reads NMEA data from a TCP server
WebSocket reads NMEA data from a WebSocket server (this is a WebSocket client)
File reads NMEA data from a log file
etc

Forwarders:

Serial writes NMEA data to a Serial Port (not implemented yet)
TCP writes NMEA data to a TCP port
WebSocket writes NMEA data to a WebSocket server (this is also a WebSocket client)
File write NMEA data to a log file
Small Screens, oled, eInk, etc (WiP)
etc

Sensors:

BME280 reads raw data from a BME280 sensor (Temperature, Pressure, Humidity), and produces XDR and MDA NMEA Sentences.
BMP180 reads raw data from a BMP180 sensor (Temperature, Pressure), and produces XDR and MDA NMEA Sentences.
HTU21D-F reads raw data from a HTU21D-F sensor (Temperature, Humidity), and produces XDR NMEA Sentences.
LSM303 reads raw data from a LSM303 sensor (3D magnetometer and accelerometer), and produces XDR NMEA Sentences for pitch and roll, HDM for heading.
HMC5883L reads raw data from a HMC5883L sensor (3D magnetometer), and produces XDR NMEA Sentences for pitch and roll, HDM for heading.
... and the list is not closed.

Note about Sensors: As opposed to what it used to be in the previous version, we kept the code that reads the sensors' data in Python. The values will be reached through the TCP channel. This was done to avoid dependencies on Java frameworks like PI4J or diozero. We've experienced some frustration recently, when PI4J-v1 was deprecated (after WiringPi's deprecation). Its V2 - and diozero as well) have restrictions on the Java SDK. It requires JDK 11, that cannot run on a Raspberry Pi Zero. This may be fixed later, but leaving the sensor code in Python (usually written by the sensors' providers) cuts us some slack, as those already working drivers do not need to be re-written in Java. This presented some redundant aspect...

Computers:

True Wind computer (produces MDA, MWD data)
Current computer (produces VDR data).

Also look into nmea.computers.ExtraDataComputer, instantiated when tw-current is mentioned in the computers list of the mux definition.

Important: Computers require the cache to be initialized. This is set in the properties file:

 init.cache=true

Transformers:
A Transformer is specific kind of Forwarder.

GPSD data, like nmea.forwarders.GPSdServer (See this)
Custom data (like nmea.forwarders.CharacterConsoleWriter)

A word about the Current Computer

A basic approach to compute the current would be to do it by instant triangulation, figuring the Course Made Good (CMG) and comparing it with the GPS Data (Course and Speed Over Ground). A better approach turned out to compute the current over a given period of time. For example, you can perform this calculation by comparing the position you should be at with the CMG only (i.e. as if there was no current) and the one given by the GPS, over periods like 30 seconds, 1 minute, 10 minutes, etc, using a smoothing of the Boat Speed (BSP) and the CMG. The cache is designed to manage several such computations in parallel, they are discriminated by the length of their time-buffer (30 seconds, 5 minutes, etc). The accuracy of such a computations is much higher than the instant triangulation. See this article for details.

Overview

Note: There is no Transformer on the picture above

Note

There is an rmi forwarder. This is a work in progress, but it works. It is feeding an RMI server that can then be accessed by an RMI client. See an example of such a client in samples.rmi.client.SampleRMIClient.

To see it at work (aka Get Started)

See the class nmea.mux.GenericNMEAMultiplexer, it uses the file nmea.mux.properties to define what to read, and what to re-broadcast it to. See it to understand its content (should be clear enough). The properties file used by the nmea.mux.GenericNMEAMultiplexer can be overridden by the System property named mux.properties, like in

-Dmux.properties=nmea.mux.gps.log.small.properties

Those settings can be modified once the mux is started, through the REST API.

To compile and build:

 $> ../../gradlew shadowJar

To run it, modify mux.sh to fit your environment, and run

 $> ./mux.sh

The script above can take the name of the properties (or yaml) file as a parameter, like in $> ./mux.sh nmea.mux.gps.log.properties. The default name for the configuration file is nmea.mux.properties, as seen in nmea.mux.GenericNMEAMultiplexer.java, where it is the default value for the system variable named mux.properties.

See the Manual for more technical details.

Build, package, run

The build is done as said above, with a

 $> ../../gradlew shadowJar

This produces a fat jar (in the build/libs folder), that can be used to run the Multiplexer.

In the modules implementing the NMEA-multiplexer, you can also package for production, using the script to.prod.sh from a terminal (you will be prompted for input). This will produce a tar.gz file, containing everything needed to run smoothly (jar-files, scripts, sample config files). This archive can be exported to wherever it is needed, and it will not require any source file, nor git repository.

To start the Multiplexer, you need to eventually run the script mux.sh. This script will refer to the right jar-file, and use the right java command to start the Multiplexer. It can also take the name of the properties (or yaml) file as a parameter. The script to.mux.sh can help you as well, as it can take care of some cleanup before running mux.sh. Those two scripts can be run in batch mode, like with nohup ./mux.sh &. If you do so, then you will need to kill the Multiplexer process to stop it. This can be achieved by running killmx.sh.

Note: During the packaging step above (to.prod.sh), the script is also archiving some web resources. The Multiplexer this page is about can be exposed through a small REST/HTTP server.
This way to expose the Multiplexer's features is indeed part of the Multiplexer, hence those Web examples. But those pages do not mean to be seen as anything but examples !!

Filtering

The Channels - aka Consumers - support sentence filtering. Filtering can be applied on NMEA Device ID, and on NMEA Sentence ID. Filters can be positive (inclusive) or negative (exclusive).

Positive filters are linked with an or, Negative ones with an and.

A (sentence) filter like "HDM", "GLL", "~RMC", "~XDR" would mean

(HDM or GLL) and (not RMC and not XDR)

Note: the filter above is meaningless, this is just an example. If the sentence has to start with HDM or GLL, there is no point in excluding RMC or XDR... 🙄. They are already implicitly excluded.

"HDM", "GLL" would mean "Only HDM or GLL"

"~RMC", "~XDR" would mean "Everything, but RMC and XDR"

It is the user's responsibility not to have contradiction in the filters, like [ "GLL", "~GLL" ], no verification is done in this area. Such a filter would just return nothing.

Note: This is just providing the possibility to negate an expression. Convenient, but limited. The best would probably be to use regular expressions (RegExp). Big drawback though: for the majority of the users, the RegExp syntax could be complex, too complex, or even scary...

About transformers

There is an example of a transformer in WebSocketProcessor.java. As you would see, it is just implementing the Forwarder interface, and this is where it fits in the picture above. A Transformer is just reworking the data before forwarding them as a regular forwarder would.

The example in WebSocketProcessor.java is transforming the NMEA Data in the format expected by a Pebble (this is a smart watch) application. See it here. Data are expected as a json object, over WebSocket. The expected data look like:

{
    "gpstime": 1290377346000,
    "gpsdatetime": 1290377346000,
    "wp": "RANGI   ",
    "d2wp": 561.6,
    "b2wp": 230,
    "xte": 3.0,
    "lat": -9.1102,
    "lng": -140.21108333333333,
    "gpssolardate": 1290343695340,
    "log": 3013.0,
    "daylog": 12.4,
    "cog": 218,
    "sog": 7.2,
    "awa": -121,
    "aws": 17.8,
    "dbt": 1.7000000476837158,
    "hdg": 220,
    "bsp": 6.6,
    "wtemp": 26.5,
    "atemp": 0.0,
    "D": 10.0,
    "d": -0.9830777902388692,
    "W": 9.01692220976113,
    "leeway": 0.0,
    "cmg": 229,
    "tws": 21.4,
    "twa": -139,
    "twd": 89,
    "cdr": 149,
    "csp": 0.29,
    "prmsl": 0.0,
    "hum": 0.0
}

The transformer reads the data from the cache and generates such an object. Then it is sent to a WebSocket server.

To run this transformer example

Start the websocket server, on a port of your choice:

 $> node wsnmea.js -port:1234

Define your transformer in the properties file:

forward.07.type=wsp
forward.07.wsuri=ws://localhost:1234/

Make sure you have configured the Pebble application as required (WebSocket URI), and you are good to go.

Application list

Press select to start

Scroll...

Choose...

Displayed!

Note: Pebble has been acquired by FitBit... There are other smart watch interfaces (like Samsung Frontier) available in the SmartWatches repo.

WebSockets

For more details and use-cases, see here.

WebSocket protocol is supported, in input, and in output. If needed, you can start your own local WebSocket server, running on nodejs. To install it (once):

 $> npm install

Then, to run it,

 $> node wsnmea.js

or

 $> npm start

An alternative, `Node-RED`.

There is a possibility to use Node-RED as a forwarder as well, that is a cool one. You can create a Node-RED flow that ingests data from a TCP port, and spits them out on a WebSocket one.

Here is the flow to import in Node-RED.

To run it

Start the Mux ($> ./mux.sh)
- Make sure there is a TCP Forwarder (port 7002)
Start the node script wsnmea.parser.js.

 node wsnmea.parser.js

Start Node-RED and run the flow mentioned above.

The Multiplexer reads the NMEA data and forwards them on a TCP Channel.
Node-RED listens to the TCP port, and forwards the payloads to a WebSocket server
The WebSocket server transforms the data it receives into the expected format, and broadcasts them to all the connected clients (Pebbles in this case).

Once everything runs, you can reach http://localhost:9876/data/web/wsconsole.html to see the data as they come through.

REST Admin Interface

The properties files like nmea.mux.properties defines the configuration at startup.

You can remotely manage the input channels and the re-broadcasting ones through a REST interface. The soft includes a dedicated HTTP Server. The http port is driven by a property (in nmea.mux.properties). Same if you want the HTTP server to be started or not.

with.http.server=yes
http.port=9999

This HTTP Server is designed and written to run on small computers (like the Raspberry Pi Zero). It is NOT an enterprise server, and it will NOT scale as one. Think of it as a Micro server.

Examples of Web implementations (HTML, REST, etc) are gathered in another module, NMEA-multiplexer-basic.

Supported REST end-points

List of operations

A full list of the available REST services is available at

 GET /oplist

Examples

All the end points and operations are defined in nmea.mux.GenericNMEAMultiplexer.java. See the List<Operation> named operations.

 GET /mux/serial-ports

returns a payload as:

[
  "/dev/tty.Bluetooth-Incoming-Port",
  "/dev/cu.Bluetooth-Incoming-Port"
]

The list of available serial ports.

 GET /mux/channels

returns a payload like

[
  {
    "cls": "consumers.client.SerialClient",
    "type": "serial",
    "port": "/dev/ttyUSB0",
    "br": 4800
  },
  {
    "cls": "consumers.client.BME280Client",
    "type": "bme280"
  }
]

 GET /mux/forwarders

returns a payload like

[
  {
    "cls": "forwarders.TCPWriter",
    "port": 7001,
    "type": "tcp"
  },
  {
    "cls": "forwarders.ConsoleWriter",
    "type": "console"
  }
]

 DELETE /mux/forwarders/{type}

type is one of

file. requires a body like

{
   "log": "./data.nmea",
   "type": "file"
}