Skip to content

Latest commit

 

History

History
1879 lines (1358 loc) · 68.6 KB

development.md

File metadata and controls

1879 lines (1358 loc) · 68.6 KB
Raw

Development documentation

Preface

The IoT Agent node library as the name suggests is a library that provides a set of functions that can be used by IoT Agents to implement the northbound interface. The library is used by several FIWARE IoT Agents, such as:

a standalone library that can be used by any IoT Agent to implement the northbound interface,

is not a standalone product and should be added as a dependency to package.json of the IoT Agent.

...
"dependencies": {
	"iotagent-node-lib": "*",
}

In order to use the library within your own IoT Agent, you must first you require it before use:

const iotagentLib = require('iotagent-node-lib');

This file contains the documentation for developers who wish to contribute to the IoT Agent node library project and also for those who wish to use the library within their own IoT Agent project.

Contributing

Contributions to this project are welcome. Developers planning to contribute should follow the Contribution Guidelines

Project management

The IoT Agent node library project is managed using npm. The following sections show the available options in detail:

Installing dependencies

This is the first step to be executed after cloning the project. To install them, type the following command:

npm install

Project build

The project is managed using npm.

For a list of available task, type

npm run

The following sections show the available options in detail.

Testing

Mocha Test Runner + Should.js Assertion Library.

The test environment is preconfigured to run BDD testing style.

Module mocking during testing can be done with proxyquire

To run tests, type

npm test

There are additional targets starting with test: prefix to run specific test subsets isolated. For instance, the test:expressions target runs the subset of tests related with expression language feature:

npm run test:expressions

Test requirements

A MongoDB 3.2+ instance is required to run tests. You can deploy one by using the commodity docker-compose-dev.yml:

docker-compose -f docker-compose-dev.yml up -d

To run docker compose you will need docker and docker-compose.

Debug Test

To debug the code while running run tests, type

npm run test:debug

In the console the link to the debugger will be provided. You can connect to it via Chrome, for example, by opening the following url: chrome://inspect.

Additional debug clients are listed on node.js.

Continuous testing

Support for continuous testing by modifying a src file or a test. For continuous testing, type

npm run test:watch

If you want to continuously check also source code style, use instead:

npm run watch

Code Coverage

Istanbul

Analyze the code coverage of your tests.

To generate an HTML coverage report under site/coverage/ and to print out a summary, type

# Use git-bash on Windows
npm run test:coverage

Clean

Removes node_modules and coverage folders, and package-lock.json file so that a fresh copy of the project is restored.

# Use git-bash on Windows
npm run clean

Checking code style

Source code style validation - ESLint

Uses the provided .eslintrc.json flag file. To check source code style, type

npm run lint

Documentation Markdown validation

Checks the Markdown documentation for consistency

# Use git-bash on Windows
npm run lint:md

Documentation Spell-checking

Uses the provided .textlintrc flag file. To check the Markdown documentation for spelling and grammar errors, dead links & etc.

# Use git-bash on Windows
npm run lint:text

Prettify Code

Runs the prettier code formatter to ensure consistent code style (whitespacing, parameter placement and breakup of long lines etc.) within the codebase.

# Use git-bash on Windows
npm run prettier

To ensure consistent Markdown formatting run the following:

# Use git-bash on Windows
npm run prettier:text

Library functions and modules

Stats Registry

The library provides a mechanism for the periodic reporting of stats related to the library's work. In order to activate the use of the periodic stats, it must be configured in the config file, as described in the Configuration section.

The Stats Registry holds two dictionaries, with the same set of stats. For each stat, one of the dictionaries holds the historical global value and the other one stores the value since the last value reporting (or current value).

The stats library currently stores only the following values:

  • deviceCreationRequests: number of Device Creation Requests that arrived to the API (no matter the result).
  • deviceRemovalRequests: number of Removal Device Requests that arrived to the API (no matter the result).
  • measureRequests: number of times the ngsiService.update() function has been invoked (no matter the result).

More values will be added in the future to the library. The applications using the library can add values to the Stats Registry just by using the following function:

iotagentLib.statsRegistry.add('statName', statIncrementalValue, callback);

The first time this function is invoked, it will add the new stat to the registry. Subsequent calls will add the value to the specified stat both to the current and global measures. The stat will be cleared in each interval as usual.

Alarm module

The library provide an alarm module that can be used to track through the logs alarms raised in the IoTAgent. This module provides:

  • Two functions to raise and release and alarm (raise() and release()): every alarm is identified by a name and a description. When the alarm is raised, an error with the text Raising [%s] is logged. When the alarm is released, the corresponding text, Releasing [%s] is logged. If an alarm is raised multiple times, it is only logged once. If its released multiple times it is only released once. Releasing a non-existing alarm has no effect.

  • Functions to list all the raised alarms and clean all the alarms (list() and clean()).

  • A function to instrument other functions, so when one of that functions return an error, an alarm is raised, and when it returns a success an alarm is ceased (intercept()).

All this functions can be accessed through the .alarms attribute of the library.

Transactions

The library implements a concept of transactions, in order to follow the execution flow the library follows when treating requests entering both from the North and the South ports of the IoT Agent.

To follow the transactions, a new Domain is created for each incoming request; in the case of requests received on the North Port of the IoT Agent, this domain is automatically created by a Express middleware, and no further action is needed from the user. For the case of requests received on the South Port of the IoT Agent, the user is responsible of creating an stopping the transaction, using the ensureSouthboundDomain and finishSouthBoundTransaction. In this case, the transaction will last from the invocation to the former to the invocation of the latter.

The Transaction Correlator is used along all the IoT Platform to follow the trace of a transaction between multiple components. To do so, in all the HTTP requests sent to other components of the platform, a custom header named Fiware-Correlator is sent with the correlator of the transaction that generated the request. If a component of the platform receives a request containing this header that starts a transaction, the component will create the transaction with the received correlator, instead of creating a new one. If the header is not present or the transaction originates in the component, the transaction ID in this component will be used as the correlator.

During the duration of a transaction, all the log entries created by the code will write the current Transaction ID and correlator for the operation being executed.

Library overview

In order to use the library, add the following dependency to your package.json file:

"iotagent-node-lib": "*"

In order to use this library, first you must require it:

var iotagentLib = require('iotagent-node-lib');

The library supports four groups of features, one for each direction of the communication: client-to-server and server-to-client (and each flow both for the client and the server). Each feature set is defined in the following sections.

Function reference

WARNING This section is outdated. Functions described here may be outdated and not reflect the current implementation of the IoT Agent Library. You could have a look to iotagentLib.js file to see the current detail of functions implemented.

The following fucntions are available in the library:

iotagentLib.activate()
Signature
function activate(newConfig, callback)
Description

Activates the IoT Agent to start listening for NGSI Calls (acting as a Context Provider). It also creates the device registry for the IoT Agent (based on the deviceRegistry.type configuration option).

Params
  • newConfig: Configuration of the Context Server (described in the Configuration section).
iotagentLib.deactivate()
Signature
function deactivate(callback)
Description

Stops the HTTP server.

Params
iotagentLib.register()
Signature
function registerDevice(deviceObj, callback)
Description

Register a new device in the IoT Agent. This registration will also trigger a Context Provider registration in the Context Broker for all its lazy attributes.

The device Object can have the following attributes:

  • id: Device ID of the device.
  • type: type to be assigned to the device.
  • name: name that will be used for the Entity representing the device in the Context Broker.
  • service: name of the service associated with the device.
  • subservice: name of the subservice associated with th device.
  • lazy: list of lazy attributes with their types.
  • active: list of active attributes with their types.
  • staticAttributes: list of NGSI attributes to add to the device entity 'as is' in updates, queries and registrations.
  • internalAttributes: optional section with free format, to allow specific IoT Agents to store information along with the devices in the Device Registry.

The device id and type are required fields for any registration. The rest of the attributes are optional, but, if they are not present in the function call arguments, the type must be registered in the configuration, so the service can infer their default values from the configured type. If an optional attribute is not given in the parameter list and there isn't a default configuration for the given type, a TypeNotFound error is raised.

If the device has been previously preprovisioned, the missing data will be completed with the values from the registered device.

Params
  • deviceObj: object containing all the information about the device to be registered (mandatory).
iotagentLib.unregister()
Signature
function unregisterDevice(id, service, subservice, callback)
Description

Unregister a device from the Context broker and the internal registry.

Params
  • id: Device ID of the device to register.
  • service: Service of the device to unregister.
  • subservice: Subservice inside the service for the unregistered device.
iotagentLib.update()
Signature
function update(entityName, attributes, typeInformation, token, callback)
Description

Makes an update in the Device's entity in the context broker, with the values given in the 'attributes' array. This array should comply to the NGSI's attribute format.

Params
  • entityName: Name of the entity to register.
  • attributes: Attribute array containing the values to update.
  • typeInformation: Configuration information for the device.
  • token: User token to identify against the PEP Proxies (optional).
iotagentLib.setCommandResult()
Signature
function setCommandResult(entityName, resource, apikey, commandName, commandResult, status, deviceInformation, callback)
Description

Update the result of a command in the Context Broker. The result of the command has two components: the result of the command itself will be represented with the suffix _info in the entity while the status is updated in the attribute with the _status suffix.

Params
  • entityName: Name of the entity holding the command.
  • resource: Resource name of the endpoint the device is calling.
  • apikey: Apikey the device is using to send the values (can be the empty string if none is needed).
  • commandName: Name of the command whose result is being updated.
  • commandResult: Result of the command in string format.
  • deviceInformation: Device information, including security and service information. (optional).
iotagentLib.listDevices()
Signature
function listDevices(callback)
function listDevices(limit, offset, callback)
function listDevices(service, subservice, limit, offset, callback)
Description

Return a list of all the devices registered in the specified service and subservice. This function can be invoked in three different ways:

  • with just one parameter (the callback)
  • with three parameters (service, subservice and callback)
  • or with five parameters (including limit and offset).
Params
  • service: service from where the devices will be retrieved.
  • subservice: subservice from where the devices will be retrieved.
  • limit: maximum number of results to retrieve (optional).
  • offset: number of results to skip from the listing (optional).
iotagentLib.setDataUpdateHandler()
Signature
function setDataUpdateHandler(newHandler)
Description

Sets the new user handler for Entity update requests. This handler will be called whenever an update request arrives with the following parameters: (id, type, service, subservice, attributes, callback). Every object within of the attributes array contains name, type and value attributes, and may also include additional attributes for metadata and datasetId. The handler is in charge of updating the corresponding values in the devices with the appropriate protocol.

Once all the updates have taken place, the callback must be invoked with the updated Context Element. E.g.:

callback(null, {
    type: 'TheType',
    isPattern: false,
    id: 'EntityID',
    attributes: [
        {
            name: 'lumniscence',
            type: 'Lumens',
            value: '432'
        }
    ]
});

In the case of NGSI requests affecting multiple entities, this handler will be called multiple times, one for each entity, and all the results will be combined into a single response.

Params
  • newHandler: User handler for update requests
iotagentLib.setDataQueryHandler()
Signature
function setDataQueryHandler(newHandler)
Description

Sets the new user handler for Entity query requests. This handler will be called whenever a query request arrives, with the following parameters: (id, type, service, subservice, attributes, callback). The handler must retrieve all the corresponding information from the devices and return a NGSI entity with the requested values.

The callback must be invoked with the updated Context Element, using the information retrieved from the devices. E.g.:

callback(null, {
    type: 'TheType',
    isPattern: false,
    id: 'EntityID',
    attributes: [
        {
            name: 'lumniscence',
            type: 'Lumens',
            value: '432'
        }
    ]
});

In the case of NGSI requests affecting multiple entities, this handler will be called multiple times, one for each entity, and all the results will be combined into a single response.

Params
  • newHandler: User handler for query requests.
iotagentLib.setNotificationHandler()
Signature
function setNotificationHandler(newHandler)
Description

Sets the new handler for incoming notifications. The notifications are sent by the Context Broker based on the IoT Agent subscriptions created with the subscribe() function.

The handler must adhere to the following signature:

function mockedHandler(device, data, callback)

The device parameter contains the device object corresponding to the entity whose changes were notified with the incoming notification. Take into account that multiple entities may be modified with each single notification. The handler will be called once for each one of those entities.

The data parameter is an array with all the attributes that were requested in the subscription and its respective values.

The handler is expected to call its callback once with no parameters (failing to do so may cause unexpected behaviors in the IoT Agent).

iotagentLib.setCommandHandler()
Signature
function setCommandHandler(newHandler)
Description

Sets the new user handler for registered entity commands. This handler will be called whenever a command request arrives, with the following parameters: (id, type, service, subservice, attributes, callback). The handler must retrieve all the corresponding information from the devices and return a NGSI entity with the requested values.

The callback must be invoked with the updated Context Element, using the information retrieved from the devices. E.g.:

callback(null, {
    type: 'TheType',
    isPattern: false,
    id: 'EntityID',
    attributes: [
        {
            name: 'lumniscence',
            type: 'Lumens',
            value: '432'
        }
    ]
});

In the case of NGSI requests affecting multiple entities, this handler will be called multiple times, one for each entity, and all the results will be combined into a single response. Only IoT Agents which deal with actuator devices will include a handler for commands.

Params
  • newHandler: User handler for command requests.
iotagentLib.setMergePatchHandler()
Signature
function setMergePatchHandler(newHandler)
Description

Sets the new user handler for NGSI-LD Entity merge-patch requests. This handler will be called whenever a merge-patch request arrives, with the following parameters: (id, type, service, subservice, attributes, callback). The handler must retrieve all the corresponding information from the devices and return a NGSI entity with the requested values.

The callback must be invoked with the updated Context Element, using the information retrieved from the devices. E.g.:

callback(null, {
    type: 'TheType',
    isPattern: false,
    id: 'EntityID',
    attributes: [
        {
            name: 'lumniscence',
            type: 'Lumens',
            value: '432'
        }
    ]
});

In the case of NGSI-LD requests affecting multiple entities, this handler will be called multiple times. Since merge-patch is an advanced function, not all IoT Agents will include a handler for merge-patch.

Params
  • newHandler: User handler for merge-patch requests.
iotagentLib.setProvisioningHandler()
Signature
function setProvisioningHandler (newHandler)
Description

Sets the new user handler for the provisioning of devices. This handler will be called every time a new device is created.

The handler must adhere to the following signature:

function(newDevice, callback)

The newDevice parameter will contain the newly created device. The handler is expected to call its callback with no parameters (this handler should only be used for reconfiguration purposes of the IoT Agent).

iotagentLib.setRemoveDeviceHandler()
Signature
function setRemoveDeviceHandler(newHandler)
Description

Sets the new user handler for the removal of a device. This handler will be called every time a device is removed.

The handler must adhere to the following signature:

function(deviceToDelete, callback)

The deviceToDelete parameter will contain the device to be deleted. The handler is expected to call its callback with no parameters (this handler should only be used for reconfiguration purposes of the IoT Agent).

iotagentLib.setConfigurationHandler()
Signature
function setConfigurationHandler(newHandler)
Description

Sets the new user handler for the configuration updates. This handler will be called every time a new configuration is created or an old configuration is updated.

The handler must adhere to the following signature:

function(newConfiguration, callback)

The newConfiguration parameter will contain the newly created configuration. The handler is expected to call its callback with no parameters (this handler should only be used for reconfiguration purposes of the IoT Agent).

For the cases of multiple updates (a single Device Configuration POST that will create several device groups), the handler will be called once for each of the configurations (both in the case of the creations and the updates).

The handler will be also called in the case of updates related to configurations. In that situation, the newConfiguration parameter contains also the fields needed to identify the configuration to be updated, i.e., service, subservice, resource and apikey.

iotagentLib.setRemoveConfigurationHandler()
Signature
function setRemoveConfigurationHandler(newHandler)
Description

Sets the new user handler for the removal of configuratios. This handler will be called every time a configuration is removed.

The handler must adhere to the following signature:

function(configurationToDelete, callback)

The configurationToDelete parameter will contain the configuration to be deleted. The handler is expected to call its callback with no parameters (this handler should only be used for reconfiguration purposes of the IoT Agent).

iotagentLib.getDevice()
Signature
function getDevice(deviceId, service, subservice, callback)
Description

Retrieve all the information about a device from the device registry.

Params
  • deviceId: ID of the device to be found.
  • service: Service for which the requested device.
  • subservice: Subservice inside the service for which the device is requested.
iotagentLib.getDeviceByName()
Signature
function getDeviceByName(deviceName, service, subservice, callback)
Description

Retrieve a device from the registry based on its entity name.

Params
  • deviceName: Name of the entity associated to a device.
  • service: Service the device belongs to.
  • subservice: Division inside the service.
iotagentLib.getDevicesByAttribute()
Signature
function getDevicesByAttribute(attributeName, attributeValue, service, subservice, callback)
Description

Retrieve all the devices having an attribute named name with value value.

Params
  • name: name of the attribute to match.
  • value: value to match in the attribute.
  • service: Service the device belongs to.
  • subservice: Division inside the service.
iotagentLib.retrieveDevice()
Signature
function retrieveDevice(deviceId, apiKey, callback)
Description

Retrieve a device from the device repository based on the given APIKey and DeviceID, creating one if none is found for the given data.

Params
  • deviceId: Device ID of the device that wants to be retrieved or created.
  • apiKey: APIKey of the Device Group (or default APIKey).
iotagentLib.mergeDeviceWithConfiguration()
Signature
function mergeDeviceWithConfiguration(fields, defaults, deviceData, configuration, callback)
Description

Complete the information of the device with the information in the configuration group (with precedence of the device). The first argument indicates what fields would be merged.

Params
  • fields: Fields that will be merged.
  • defaults: Default values fot each of the fields.
  • deviceData: Device data.
  • configuration: Configuration data.
iotagentLib.getConfiguration()
Signature
function getConfiguration(resource, apikey, callback)
Description

Gets the device group identified by the given (resource, apikey) pair.

Params
  • resource: representation of the configuration in the IoT Agent (dependent on the protocol) .
  • apikey: special key the devices will present to prove they belong to a particular configuration.
iotagentLib.findConfiguration()
Signature
function findConfiguration(service, subservice, callback)
Description

Find a device group based on its service and subservice.

Params
  • service: name of the service of the configuration.
  • subservice: name of the subservice of the configuration.
iotagentLib.getEffectiveApiKey()
Signature
function getEffectiveApiKey(service, subservice, type, callback)
Description

Get the API Key for the selected service if there is any, or the default API Key if a specific one does not exist.

Params
  • service: Name of the service whose API Key we are retrieving.
  • subservice: Name of the subservice whose API Key we are retrieving.
  • type: Type of the device.
iotagentLib.subscribe()
Signature
function subscribe(device, triggers, content, callback)
Description

Creates a subscription for the IoTA to the entity representing the selected device.

Params
  • device: Object containing all the information about a particular device.
  • triggers: Array with the names of the attributes that would trigger the subscription
  • content: Array with the names of the attributes to retrieve in the notification.
iotagentLib.unsubscribe()
Signature
function unsubscribe(device, id, callback)
Description

Removes a single subscription from the selected device, identified by its ID.

Params
  • device: Object containing all the information about a particular device.
  • id: ID of the subscription to remove.
iotagentLib.ensureSouthboundDomain()
Signature
function ensureSouthboundTransaction(context, callback)
Description

Ensures that the current operation is executed inside a transaction with all the information needed for the appropriate platform logging: start date, transaction ID and correlator in case one is needed. If the function is executed in the context of a previous transaction, just the context is changed (and the Transaction ID and start time are kept).

Params
  • context: New context data for the transaction.
iotagentLib.finishSouthBoundTransaction()
Signature
function finishSouthboundTransaction(callback)
Description

Terminates the current transaction, if there is any, cleaning its context.

iotagentLib.startServer()
Signature
function startServer(newConfig, iotAgent, callback)
Description

Start the HTTP server either in single-thread or multi-thread (multi-core) based on the value of multiCore variable (described in the Configuration section). If the value is False (either was directly specified False in the config.js or it was not specified and by default is assigned False), it is a normal (single-thread) behaviour. Nevertheless, if multiCore is True, the IoTAgent is executed in multi-thread environment.

The number of parallel processes is calculated based on the number of available CPUs. In case of some of the process unexpectedly dead, a new process is created automatically to keep always the maximum of them working in parallel.

Note: startServer() initializes the server but it does not activate the library. The function in the Node Lib will call the iotAgent.start() in order to complete the activation of the library. Therefore, it is expected that the IoT Agent implement the iotAgent.start() function with the proper invocation to the iotAgentLib.activate().

Params
  • newConfig: Configuration of the Context Server (described in the Configuration section).
  • iotAgent: The IoT Agent Objects, used to start the agent.
  • callback: The callback function.
iotagentLib.request()
Signature
function request(options, callback)
Description

Make a direct HTTP request using the underlying request library (currently got), this is useful when creating agents which use an HTTP transport for their southbound commands, and removes the need for the custom IoT Agent to import its own additional request library

Params
  • options: definition of the request (see got options for more details). The following attributes are currently exposed.
    • method - HTTP Method
    • searchParams - query string params
    • qs - alias for query string params
    • headers
    • responseType - either text or json. json is the default
    • json - a supplied JSON object as the request body
    • body - any ASCII text as the request body. It takes precedence over json if both are provided at the same time (not recommended).
    • url - the request URL
    • uri - alternative alias for the request URL.
  • callback: The callback currently returns an error Object, the response and body. The body is parsed to a JSON object if the responseType is JSON.

Generic middlewares

This collection of utility middlewares is aimed to be used to north of the IoT Agent Library, as well as in other HTTP-based APIs of the IoT Agents. All the middlewares follow the Express convention of (req, res, next) objects, so this information will not be repeated in the descriptions for the middleware functions. All the middlewares can be added to the servers using the standard Express mechanisms.

iotagentLib.middlewares.handleError()
Signature
function handleError(error, req, res, next)
Description

Express middleware for handling errors in the IoTAs. It extracts the code information to return from the error itself returning 500 when no error code has been found.

iotagentLib.middlewares.traceRequest()
Signature
function traceRequest(req, res, next)
Description

Express middleware for tracing the complete request arriving to the IoTA in debug mode.

iotagentLib.middlewares.changeLogLevel()
Signature
function changeLogLevel(req, res, next)
Description

Changes the log level to the one specified in the request.

iotagentLib.middlewares.ensureType()
Signature
function ensureType(req, res, next)
Description

Ensures the request type is one of the supported ones.

iotagentLib.middlewares.validateJson()
Signature
function validateJson(template)
Description

Generates a Middleware that validates incoming requests based on the JSON Schema template passed as a parameter.

Returns an Express middleware used in request validation with the given template.

Params
  • template: JSON Schema template to validate the request.
iotagentLib.middlewares.retrieveVersion()
Signature
function retrieveVersion(req, res, next)
Description

Middleware that returns all the IoTA information stored in the module.

iotagentLib.middlewares.setIotaInformation()
Signature
function setIotaInformation(newIoTAInfo)
Description

Stores the information about the IoTAgent for further use in the retrieveVersion() middleware.

Params
  • newIoTAInfo: Object containing all the IoTA Information.

DB Models (from API document)

WARNING This section is outdated. DB fields described here may be outdated and not reflect the current implementation of the IoT Agent Library.

The following sections describe the models used in the database to store the information about the devices and the config groups.

Config group model

The table below shows the information held in the Config group provisioning resource and the correspondence between the API resource fields and the same fields in the database model.

You can find the description of the fields in the config group datamodel of the API document.

Payload Field DB Field Note
service service
subservice subservice
resource resource
apikey apikey
timestamp timestamp
entity_type entity_type
trust trust
cbHost cbHost
lazy lazy
commands commands
attributes attributes
static_attributes staticAttributes
internal_attributes internalAttributes
explicitAttrs explicitAttrs
entityNameExp entityNameExp
ngsiVersion ngsiVersion
defaultEntityNameConjunction defaultEntityNameConjunction optional string value to set default conjunction string used to compose a default entity_name when is not provided at device provisioning time.
autoprovision autoprovision

Device model

The table below shows the information held in the Device resource. The table also contains the correspondence between the API resource fields and the same fields in the database model.

You can find the description of the fields in the config group datamodel of the API document.

Payload Field DB Field Note
device_id id
service service
service_path subservice
entity_name name
entity_type type
timezone timezone
timestamp timestamp
apikey apikey
endpoint endpoint
protocol protocol Name of the device protocol, for its use with an IoT Manager. IE: IoTA-UL
transport transport
attributes active
lazy lazy
commands commands
internal_attributes internalAttributes List of internal attributes with free format for specific IoT Agent configuration. I.E:LWM2M mappings from object URIs to attributes
static_attributes staticAttributes
explicitAttrs explicitAttrs
ngsiVersion ngsiVersion

Developing a new IoT Agent

WARNING This section is outdated. Methods and steps described here may be outdated and not reflect the current implementation of the IoT Agent Library. You could have a look to other IoT Agents developed using the IoT Agent Library to get a better idea of how to use it, like the IoT Agent JSON

This section's goal is to show how to develop a new IoT Agent step by step. To do so, a simple invented HTTP protocol will be used, so it can be tested with simple command-line instructions as curl and nc.

Protocol

The invented protocol will be freely adapted from Ultralight 2.0. Whenever a device wants to send an update, it will send a request as the following:

curl -X GET 'http://127.0.0.1:8080/iot/d?i=ULSensor&k=abc&d=t|15,l|19.6' -i

Where:

  • i: is the device ID.
  • k: the API Key for the device's service.
  • d: the data payload, consisting of key-value pairs separated by a pipe (|), with each pair separated by comma (,);

Requirements

This tutorial expects a Node.js v8 (at least) installed and working on your machine. It also expects you to have access to a Context Broker (without any security proxies).

Basic IoT Agent

In this first chapter, we will just develop an IoT Agent with a fully connected North Port. This will send and receive NGSI traffic and can be administered using the IoT Agent's Device Provisioning API. The South Port will remain unconnected and no native protocol traffic will be sent to the devices. This may seem useless (and indeed it is) but it will serve us well on showing the basic steps in the creation of an IoT Agent.

First of all, we have to create the Node project. Create a folder to hold your project and type the following instruction:

npm init

This will create the package.json file for our project. Now, add the following lines to your project file:

  "dependencies": {
    "iotagent-node-lib": "*"
  },

And install the dependencies, executing, as usual:

npm install

The first step is to write a configuration file, that will be used to tune the behavior of our IOTA. The contents can be copied from the following example:

var config = {
    logLevel: 'DEBUG',
    contextBroker: {
        host: 'localhost',
        port: '1026'
    },
    server: {
        port: 4041
    },
    deviceRegistry: {
        type: 'memory'
    },
    types: {},
    service: 'howtoService',
    subservice: '/howto',
    providerUrl: 'http://localhost:4041',
    defaultType: 'Thing'
};

module.exports = config;

Create a config.js file with it in the root folder of your project. Remember to change the Context Broker IP to your local Context Broker.

Now we can begin with the code of our IoT Agent. The very minimum code we need to start an IoT Agent is the following:

var iotAgentLib = require('iotagent-node-lib'),
    config = require('./config');

iotAgentLib.activate(config, function (error) {
    if (error) {
        console.log('There was an error activating the IOTA');
        process.exit(1);
    }
});

The IoT Agent is now ready to be used. Execute it with the following command:

node index.js

The North Port interface should now be fully functional, i.e.: management of device registrations and configurations.

IoT Agent With Active attributes

In the previous section we created an IoT Agent that exposed just the North Port interface, but that was pretty useless (aside from its didactic use). In this section we are going to create a simple South Port interface. It's important to remark that the nature of the traffic South of the IoT Agent itself has nothing to do with the creation process of an IoT Agent. Each device protocol will use its own mechanisms and it is up to the IoT Agent developer to find any libraries that would help him in its development. In this example, we will use Express as such library.

In order to add the Express dependency to your project, add the following line to the dependencies section of the package.json:

    "express": "*",

The require section would end up like this (the standard http module is also needed):

var iotAgentLib = require('iotagent-node-lib'),
    http = require('http'),
    express = require('express'),
    config = require('./config');

And install the dependencies as usual with npm install. You will have to require both express and http in your code as well.

Now, in order to accept connections in our code, we have to start express first. With this purpose in mind, we will create a new function initSouthbound(), that will be called from the initialization code of our IoT Agent:

function initSouthbound(callback) {
    southboundServer = {
        server: null,
        app: express(),
        router: express.Router()
    };

    southboundServer.app.set('port', 8080);
    southboundServer.app.set('host', '0.0.0.0');

    southboundServer.router.get('/iot/d', manageULRequest);
    southboundServer.server = http.createServer(southboundServer.app);
    southboundServer.app.use('/', southboundServer.router);
    southboundServer.server.listen(southboundServer.app.get('port'), southboundServer.app.get('host'), callback);
}

This Express code sets up a HTTP server, listening in the 8080 port, that will handle incoming requests targeting path /iot/d using the middleware manageULRequest(). This middleware will contain all the logic south of the IoT Agent, and the library methods we need in order to progress the information to the Context Broker. The code of this middleware would be as follows:

function manageULRequest(req, res, next) {
    var values;

    iotAgentLib.retrieveDevice(req.query.i, req.query.k, function (error, device) {
        if (error) {
            res.status(404).send({
                message: "Couldn't find the device: " + JSON.stringify(error)
            });
        } else {
            values = parseUl(req.query.d, device);
            iotAgentLib.update(device.name, device.type, '', values, device, function (error) {
                if (error) {
                    res.status(500).send({
                        message: 'Error updating the device'
                    });
                } else {
                    res.status(200).send({
                        message: 'Device successfully updated'
                    });
                }
            });
        }
    });
}

For this middleware we have made use of a function parseUl() that parses the data payload and transforms it in the data object expected by the update function (i.e.: an attribute array with NGSI syntax):

function parseUl(data, device) {
    function findType(name) {
        for (var i = 0; i < device.active.length; i++) {
            if (device.active[i].name === name) {
                return device.active[i].type;
            }
        }

        return null;
    }

    function createAttribute(element) {
        var pair = element.split('|'),
            attribute = {
                name: pair[0],
                value: pair[1],
                type: findType(pair[0])
            };

        return attribute;
    }

    return data.split(',').map(createAttribute);
}

Here as an example of the output of the function return for the UL payload t|15,l|19.6:

[
    {
        "name": "t",
        "type": "celsius",
        "value": "15"
    },
    {
        "name": "l",
        "type": "meters",
        "value": "19.6"
    }
]

The last thing to do is to invoke the initialization function inside the IoT Agent startup function. The next excerpt show the modifications in the activate() function:

iotAgentLib.activate(config, function (error) {
    if (error) {
        console.log('There was an error activating the IOTA');
        process.exit(1);
    } else {
        initSouthbound(function (error) {
            if (error) {
                console.log('Could not initialize South bound API due to the following error: %s', error);
            } else {
                console.log('Both APIs started successfully');
            }
        });
    }
});

Some logs were added in this piece of code to help debugging.

Once the IOTA is finished the last thing to do is to test it. To do so, launch the IoT Agent and provision a new device (an example for provisioning can be found in the examples/howtoProvisioning1.json file). Once the device is provisioned, send a new measure by using the example command:

curl -X GET 'http://127.0.0.1:8080/iot/d?i=ULSensor&k=abc&d=t|15,l|19.6' -i

Now you should be able to see the measures in the Context Broker entity of the device.

IOTA With Lazy attributes

Previous considerations

The IoT Agents also give the possibility for the device to be asked about the value of one of its measures, instead of reporting it. In order to do so, the device must be capable of receiving messages of some kind. In this case, we are going to simulate an HTTP server with nc in order to see the values sent by the IOTA. We also have to decide a syntax for the protocol request for asking the device about a measure. For clarity, we will use the same HTTP GET request we used to report a measure, but indicating the attribute to ask instead of the data payload. Something like:

curl -X GET 'http://127.0.0.1:9999/iot/d?i=ULSensor&k=abc&q=t,l' -i

In a real implementation, the server will need to know the URL and port where the devices are listening, in order to send the request to the appropriate device. For this example, we will assume that the device is listening in port 9999 in localhost. For more complex cases, the mechanism to bind devices to addresses would be IoT-Agent-specific (e.g.: the OMA Lightweight M2M IoT Agent captures the address of the device in the device registration, and stores the device-specific information in a MongoDB document).

Being lazy attributes of a read/write nature, another syntax has to be declared for updating. This syntax will mimic the one used for updating the server:

curl -X GET 'http://127.0.0.1:9999/iot/d?i=ULSensor&k=abc&d=t|15,l|19.6' -i

Both types of calls to the device will be distinguished by the presence or absence of the d and q attributes.

A HTTP request library will be needed in order to make those calls. To this extent, mikeal/request library will be used. In order to do so, add the following require statement to the initialization code:

request = require('request');

and add the request dependency to the package.json file:

  "dependencies": [
  [...]

    "request": "*",

  ]

The require section should now look like this:

var iotAgentLib = require('iotagent-node-lib'),
    http = require('http'),
    express = require('express'),
    request = require('request'),
    config = require('./config');

Implementation

QueryContext implementation

The main step to complete in order to implement the Lazy attributes mechanism in the IoT Agent is to provide handlers for the context provisioning requests. At this point, we should provide two handlers: the /v2/op/update and the /v2/op/query handlers. To do so, we must first define the handlers themselves:

function queryContextHandler(id, type, service, subservice, attributes, callback) {
    var options = {
        url: 'http://127.0.0.1:9999/iot/d',
        method: 'GET',
        qs: {
            q: attributes.join()
        }
    };

    request(options, function (error, response, body) {
        if (error) {
            callback(error);
        } else {
            callback(null, createResponse(id, type, attributes, body));
        }
    });
}

The queryContext handler is called whenever a /v2/op/query request arrives at the North port of the IoT Agent. It is invoked once for each entity requested, passing the entity ID and Type as the parameters, as well as a list of the attributes that are requested. In our case, the handler uses this parameters to compose a request to the device. Once the results of the device are returned, the values are returned to the caller, in the NGSI attribute format.

In order to format the response from the device in a readable way, we created a createResponse() function that maps the values to its correspondent attributes. This function assumes the type of all the attributes is "string" (this will not be the case in a real scenario, where the IoT Agent should retrieve the associated device to guess the type of its attributes). Here is the code for the createResponse() function:

function createResponse(id, type, attributes, body) {
    var values = body.split(','),
        responses = [];

    for (var i = 0; i < attributes.length; i++) {
        responses.push({
            name: attributes[i],
            type: 'string',
            value: values[i]
        });
    }

    return {
        id: id,
        type: type,
        attributes: responses
    };
}
UpdateContext implementation
function updateContextHandler(id, type, service, subservice, attributes, callback) {
    var options = {
        url: 'http://127.0.0.1:9999/iot/d',
        method: 'GET',
        qs: {
            d: createQueryFromAttributes(attributes)
        }
    };

    request(options, function (error, response, body) {
        if (error) {
            callback(error);
        } else {
            callback(null, {
                id: id,
                type: type,
                attributes: attributes
            });
        }
    });
}

The updateContext handler deals with the modification requests that arrive at the North Port of the IoT Agent via /v2/op/update. It is invoked once for each entity requested (note that a single request can contain multiple entity updates), with the same parameters used in the queryContext handler. The only difference is the value of the attributes array, now containing a list of attribute objects, each containing name, type and value. The handler must also make use of the callback to return a list of updated attributes.

For this handler we have used a helper function called createQueryFromAttributes(), that transforms the NGSI representation of the attributes to the UL type expected by the device:

function createQueryFromAttributes(attributes) {
    var query = '';

    for (var i in attributes) {
        query += attributes[i].name + '|' + attributes[i].value;

        if (i != attributes.length - 1) {
            query += ',';
        }
    }

    return query;
}
Handler registration

Once both handlers have been defined, they have to be registered in the IoT Agent, adding the following code to the setup function:

iotAgentLib.setDataUpdateHandler(updateContextHandler);
iotAgentLib.setDataQueryHandler(queryContextHandler);

Where necessary, additional handlers to deal with command actuations and merge-patch operations may also be added when necessary.

iotAgentLib.setCommandHandler(commandHandler);
iotAgentLib.setMergePatchHandler(mergePatchHandler);
IOTA Testing

In order to test it, we need to create an HTTP server simulating the device. The quickest way to do that may be using netcat. In order to start it just run the following command from the command-line (Linux and Mac only):

nc -l 9999

This will open a simple TCP server listening on port 9999, where the requests from the IoT Agent will be printed. In order for the complete workflow to work (and to receive the response in the application side), the HTTP response has to be written in the nc console (although for testing purposes this is not needed).

While netcat is great to test simple connectivity, you will need something just a bit more complex to get the complete scenario working (at least without the need to be incredibly fast sending your response). In order to do so, a simple echo server was created, that answers 42 to any query to its /iot/d path. You can use it to test your attributes one by one (or you can modify it to accept more requests and give more complex responses). Copy the Echo Server script to the same folder of your IoTAgent (as it uses the same dependencies). In order to run the echo server, just execute the following command:

node echo.js

Once the mock server has been started (either nc or the echo server), proceed with the following steps to test your implementation:

  1. Provision a device with two lazy attributes. The following request can be used as an example:
POST /iot/devices HTTP/1.1
Host: localhost:4041
Content-Type: application/json
fiware-service: howtoserv
fiware-servicepath: /test
Cache-Control: no-cache
Postman-Token: 993ac66b-72da-9e96-ab46-779677a5896a

{
    "devices": [
      {
        "device_id": "ULSensor",
        "entity_name": "Sensor01",
        "entity_type": "BasicULSensor",
        "lazy": [
            {
                "name": "t",
                "type": "celsius"
            },
            {
                "name": "l",
                "type": "meters"
            }
        ],
        "attributes": [
        ]
      }
    ]
}
  1. Execute a /v2/op/query or /v2/op/update against one of the entity attributes (use a NGSI client of curl command).
POST /v2/op/query HTTP/1.1
Host: localhost:1026
Content-Type: application/json
Accept: application/json
Fiware-Service: howtoserv
Fiware-ServicePath: /test
Cache-Control: no-cache

{
  entities: [
    {
      id: 'Light:light1'
    }
  ],
  attrs: ['dimming']
}

  1. Check the received request in the nc console is the expected one.

  2. (In case you use netcat). Answer the request with an appropriate HTTP response and check the result of the /v2/op/query or /v2/op/update request is the expected one. An example of HTTP response, for a query to the t and l attributes would be:

HTTP/1.0 200 OK
Content-Type: text/plain
Content-Length: 3

5,6

This same response can be used both for updates and queries for testing purposes (even though in the former the body won't be read).

IoT Agent in multi-thread mode

It is possible that an IoT Agent can be executed in multi-thread approach, which will increase the number of request/seconds that can be manage by the server. It's important to remark that the nature of this functionality in included in the IoT Agent Node Lib but it is not mandatory that you activate this functionality. In this example, we will see how to use this functionality to deploy an IoT Agent in multi-thread environment.

In order to activate the functionality, you have two options, configure the config.js file to add the following line:

/**
 * flag indicating whether the node server will be executed in multi-core option (true) or it will be a
 * single-thread one (false).
 */
config.multiCore = true;

or you can define the proper IOTA_MULTI_CORE environment variable. By default, the first choice is the environment variable and afterward the value of the multiCore in the config.js file. The require section would end up like this (the standard http module is also needed):

var iotAgent = require('../lib/iotagent-implementation'),
    iotAgentLib = require('iotagent-node-lib'),
    config = require('./config');

It is important to mention the purpose of the iotAgent variable. It is the proper implementation of the IoT Agent based on the IoT Agent Node Lib. We will need this variable just to make a callback to the corresponding start() process from the library. The variable config is used to get details of the configuration file and send that information to the Node Lib. The Node Lib will take the decision of single-thread or multi-thread execution base on the value of config.multiCore attribute.

Finally, we can call the corresponding iotagentLib.startServer() like the following code with a callback function to show details about any error during the execution or just print the message about starting the IoTAgent:

iotAgentLib.startServer(config, iotAgent, function (error) {
    if (error) {
        console.log(context, 'Error starting IoT Agent: [%s] Exiting process', error);
    } else {
        console.log(context, 'IoT Agent started');
    }
});

Note: startServer() initializes the server but it does not activate the library. The function in the Node Lib will call the iotAgent.start() in order to complete the activation of the library. Therefore, it is expected that the IoT Agent implement the iotAgent.start() function with the proper invocation to the iotAgentLib.activate().

Configuration management

For some IoT Agents, it will be useful to know what devices or configurations were registered in the Agent, or to do some actions whenever a new device is registered. All this configuration and provisioning actions can be performed using two mechanisms: the provisioning handlers and the provisioning API.

Provisioning handlers

The handlers provide a way for the IoT Agent to act whenever a new device, or configuration is provisioned. This can be used for registering the device in external services, for storing important information about the device, or to listen in new ports in the case of new configuration. For the simple example we are developing, we will just print the information we are receiving whenever a new device or configuration is provisioned.

We need to complete two further steps to have a working set of provisioning handlers. First of all, defining the handlers themselves. Here we can see the definition of the configuration handler:

function configurationHandler(configuration, callback) {
    console.log('\n\n* REGISTERING A NEW CONFIGURATION:\n%s\n\n', JSON.stringify(configuration, null, 4));
    callback(null, configuration);
}

As we can see, the handlers receive the device or configuration that is being provisioned, as well as a callback. The handler MUST call the callback once in order for the IOTA to work properly. If an error is passed as a parameter to the callback, the provisioning will be aborted. If no error is passed, the provisioning process will continue. This mechanism can be used to implement security mechanisms or to filter the provisioning of devices to the IoT Agent.

Note also that the same device or configuration object is passed along to the callback. This lets the IoT Agent change some of the values provisioned by the user, to add or restrict information in the provisioning. To test this feature, let's use the provisioning handler to change the value of the type of the provisioning device to CertifiedType (reflecting some validation process performed on the provisioning):

function provisioningHandler(device, callback) {
    console.log('\n\n* REGISTERING A NEW DEVICE:\n%s\n\n', JSON.stringify(device, null, 4));
    device.type = 'CertifiedType';
    callback(null, device);
}

Once the handlers are defined, the new set of handlers has to be registered into the IoT Agent:

iotAgentLib.setConfigurationHandler(configurationHandler);
iotAgentLib.setProvisioningHandler(provisioningHandler);

Now we can test our implementation by sending provisioning requests to the North Port of the IoT Agent. If we provision a new device into the platform, and then we ask for the list of provisioned devices, we shall see the type of the provisioned device has changed to CertifiedType.