Name Mappings is an advanced global feature of Cygnus. It is global because it is available for all NGSI sinks.
Name Mappings allow changing the notified FIWARE service, FIWARE service path, entity IDs, entity types, attribute names and attribute types, given a mapping. Such a mapping is just a Json within a configuration file detailing how original naming must be replaced by alternative naming.
{
"serviceMappings": [
{
"originalService": "myservice1",
"newService": "new_myservice1",
"servicePathMappings": [
{
"originalServicePath": "/myservicepath1",
"newServicePath": "/new_myservicepath1",
"entityMappings": [
{
"originalEntityId": "myentityid1",
"originalEntityType": "myentitytype1",
"newEntityId": "new_myentityid1",
"newEntityType": "new_myentitytype1",
"attributeMappings": [
{
"originalAttributeName": "myattributename1",
"originalAttributeType": "myattributetype1",
"newAttributeName": "new_myattributename1",
"newAttributeType": "new_myattributetype1"
},
...
]
},
...
]
},
...
]
},
...
]
}
When a notification is sent to Cygnus, a special Flume interceptor called NGSINameMappingsInterceptor intercepts the plain Event's generated by NGSIRestHandler, parses the Event's body and iterate on the configured mappings in order to create a mapped version of the original notification. Once finished, both versions of the notification, original and mapped ones, are put into the channel buy means of a NGSIEvent, a Java object able to handle:
- The original headers sent by
NGSIRestHandler, i.e.fiware-service,fiware-servicepath,fiware-correlatorandtransaction-id. - New headers regarding the mapped FIWARE service and FIWARE service path, i.e.
mapped-fiware-serviceandmapped-fiware-service-path. - The original notification already parsed as a
NotifyContextRequestobject. - The mapped version of the original notification as a
NotifyContextRequestobject.
Please observe no raw bytes about the body are sent.
Whenever a sink takes one of these NGSIEvent's, it is only a matter of deciding if such a sink enables the mappings (enable_name_mappings parameter) or not. If mappings are enabled, then the already parsed NotifyContextRequest, mapped version, is used. If not, then the original version is used.
Please observe the mappings definition is global to all the sinks, at NGSIRestHandler, as a Flume interceptor. Nevertheless, the application is local to the sink, depending on the enable_name_mappings parameter. Thus, if none of your sinks is going to take advantage of the mappings, simply avoid configuring the NGSINameMappingsInterceptor in NGSIRestHandler. That will avoid unnecessary interception and iterations on the mappings and Cygnus will perform faster.
$ cat /path/to/conf/agent.conf
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = hdfs-channel
cygnus-ngsi.sources.http-source ...
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler ...
cygnus-ngsi.sources.http-source.interceptors = ts nmi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.nmi.type = com.telefonica.iot.cygnus.interceptors.NGSINameMappingsInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.nmi.name_mappings_conf_file = /path/to/conf/name_mappings.conf
Additionally, please observe if any of the original names is not present, then the mapping affects all the names of that type. For instance, if originalService is not present in the mapping, then the mapping affects all the FIWARE services:
$ cat /path/to/conf/name_mappings.conf
{
"serviceMappings": [
{
"servicePathMappings": [
{
"originalServicePath": "/myservicepath1",
"newServicePath": "/new_myservicepath1",
"entityMappings": [
{
"originalEntityId": "myentityid1",
"originalEntityType": "myentitytype1",
"newEntityId": "new_myentityid1",
"newEntityType": "new_myentitytype1",
"attributeMappings": [
{
"originalAttributeName": "myattributename1",
"originalAttributeType": "myattributetype1",
"newAttributeName": "new_myattributename1",
"newAttributeType": "new_myattributetype1"
}
]
}
]
}
]
}
]
}
Another relevant behaviour must be noticed: when any of the new names is not present, then the original name is used in the mapping. I.e. there is no mapping. In this example, the original service path is never changed, since newServicePath is missing:
$ cat /path/to/conf/name_mappings.conf
{
"serviceMappings": [
{
"originalService": "myservice1",
"newService": "new_myservice1",
"servicePathMappings": [
{
"originalServicePath": "/myservicepath1",
"entityMappings": [
{
"originalEntityId": "myentityid1",
"originalEntityType": "myentitytype1",
"newEntityId": "new_myentityid1",
"newEntityType": "new_myentitytype1",
"attributeMappings": [
{
"originalAttributeName": "myattributename1",
"originalAttributeType": "myattributetype1",
"newAttributeName": "new_myattributename1",
"newAttributeType": "new_myattributetype1"
}
]
}
]
}
]
}
]
}
Last but not least, the original names support Java-based regular expressions. For instance, if you want all the service paths are re-named equals simply use /.* as value for originalServicePath:
$ cat /path/to/conf/name_mappings.conf
{
"serviceMappings": [
{
"originalService": "myservice1",
"newService": "new_myservice1",
"servicePathMappings": [
{
"originalServicePath": "/.*",
"newServicePath": "/new_all",
"entityMappings": [
{
"originalEntityId": "myentityid1",
"originalEntityType": "myentitytype1",
"newEntityId": "new_myentityid1",
"newEntityType": "new_myentitytype1",
"attributeMappings": [
{
"originalAttributeName": "myattributename1",
"originalAttributeType": "myattributetype1",
"newAttributeName": "new_myattributename1",
"newAttributeType": "new_myattributetype1"
}
]
}
]
}
]
}
]
}
In addition, above mentioned Java-based regular expressions can be also used in new entity IDs. For instance, if we want to rename certain entity IDs described as a string concatenated with a number, and we want to replace the string but maintaining the number:
{
"serviceMappings": [{
"originalService": "service",
"newService": "new_service",
"servicePathMappings": [{
"originalServicePath": "/subservice",
"newServicePath": "/new_subservice",
"entityMappings": [{
"originalEntityType": "myentitytype",
"originalEntityId": "(myentityid)([0-9]*)",
"newEntityId": "new_myentityid$2",
"attributeMappings": []
}]
}]
}]
}
Sumarizing these are some useful examples and their result in a sink like MySQL:
- Service Mapping:
{
"serviceMappings": [
{
"originalService": "city012",
"newService": "database_name",
"servicePathMappings": [
{
"originalServicePath": "/electricidad",
"newServicePath": "/tableprefix",
"entityMappings": [
{
"originalEntityType": "luminaria",
"newEntityType": "tablesufix",
"originalEntityId": "luminaria_1",
"newEntityId": "tableid",
"attributeMappings": []
}
]
}
]
}
]
}
- Headers:
Fiware-Service: city012
Fiware-ServicePath: /electricidad
- Entity:
{
"id":"luminaria1",
"type":"luminaria",
...
}
This case creates in database_name db a table with name tableprefix_tableid_tablesufix
- Service Mapping:
{
"serviceMappings": [
{
"originalService": "city012",
"newService": "database_name",
"servicePathMappings": [
{
"originalServicePath": "/electricidad",
"newServicePath": "/tableprefix",
"entityMappings": [
{
"originalEntityId": "pro(.+)",
"newEntityId": "tablepro",
"originalEntityType": ".+",
"newEntityType": "tablesufix",
"attributeMappings": []
}
]
}
]
}
]
}
- Headers:
Fiware-Service: city012
Fiware-ServicePath: /electricidad
- Entity:
{
"id":"pro1",
"type":"luminaria",
...
}
This case creates in database_name db a table with name tableprefix_tablepro_tablesufix
- Service Mapping:
{
"serviceMappings": [
{
"originalService": "city012",
"newService": "database_name",
"servicePathMappings": [
{
"originalServicePath": "/(.+)",
"newServicePath": "/$1",
"entityMappings": [
{
"originalEntityId": "(.+)",
"newEntityId": "any",
"originalEntityType": "(.+)",
"attributeMappings": []
}
]
}
]
}
]
}
- Headers:
Fiware-Service: city012
Fiware-ServicePath: /electricidad
- Entity:
{
"id":"pro1",
"type":"luminaria",
...
}
This case creates in database_name db a table with name elecricidad_any_luminaria
- Service Mapping:
{
"serviceMappings": [
{
"originalService": "city012",
"newService": "database_name",
"servicePathMappings": [
{
"originalServicePath": "/(.+)",
"newServicePath": "/global",
"entityMappings": [
{
"originalEntityId": "(.+)",
"newEntityId": "any",
"originalEntityType": "(.+)",
"attributeMappings": []
}
]
}
]
}
]
}
- Headers:
Fiware-Service: city012
Fiware-ServicePath: /electricidad
- Entity:
{
"id":"pro1",
"type":"luminaria",
...
}
This case creates db database_name db a table with name global_any_luminaria
Case 5: groups maching service, subservice, entitytype, entityid and attribute name and attribute type
- Service Mapping:
{
"serviceMappings": [
{
"originalService": "city012",
"newService": "database_name",
"servicePathMappings": [
{
"originalServicePath": "/electricidad",
"newServicePath": "/tableprefix",
"entityMappings": [
{
"originalEntityType": "luminaria",
"newEntityType": "tablesufix",
"originalEntityId": "luminaria_1",
"newEntityId": "tableid",
"attributeMappings": [
{
"originalAttributeName": "temperatura",
"originalAttributeType": "string",
"newAttributeName": "new_myattributename1",
"newAttributeType": "new_myattributetype1"
}
]
}
]
}
]
}
]
}
And agent is configured with option data_model=dm-by-attribute
- Headers:
Fiware-Service: city012
Fiware-ServicePath: /electricidad
- Entity:
{
"id":"pro1",
"type":"luminaria",
"temperatura": {
"type": "string",
"value": "13"
}
}
This case creates in database_name db a table with name tableprefix_tableid_tablesufix_new_myattributename1
Case 6: groups maching service, entitytype, entityid and attribute name and attribute type in the same tableprefix
- Service Mapping:
{
"serviceMappings": [
{
"originalService": "city012",
"newService": "database_name",
"servicePathMappings": [
{
"newServicePath": "/tableprefix",
"entityMappings": [
{
"originalEntityType": "luminaria",
"newEntityType": "tablesufix",
"originalEntityId": "luminaria_1",
"newEntityId": "tableid",
"attributeMappings": [
{
"originalAttributeName": "temperatura",
"originalAttributeType": "string",
"newAttributeName": "new_myattributename1",
"newAttributeType": "new_myattributetype1"
}
]
}
]
}
]
}
]
}
And agent is configured with option data_model=dm-by-attribute
- Headers:
Fiware-Service: city012
Fiware-ServicePath: /electricidad
- Entity:
{
"id":"pro1",
"type":"luminaria",
"temperatura": {
"type": "string",
"value": "13"
}
}
This case creates in database_name db a table with name tableprefix_tableid_tablesufix_new_myattributename1
Case 7: groups maching service, entitytype, attribute name and attribute type in the same tableprefix depending on entityid
- Service Mapping:
{
"serviceMappings": [
{
"originalService": "city012",
"newService": "database_name",
"servicePathMappings": [
{
"originalServicePath": "/electricidad",
"newServicePath": "/tableprefix",
"entityMappings": [
{
"originalEntityType": "luminaria",
"newEntityType": "tablesufix",
"originalEntityId": "([cabeco,isa]+)(.+)",
"newEntityId": "$1",
"attributeMappings": [
{
"originalAttributeName": "temperatura",
"originalAttributeType": "string",
"newAttributeName": "new_myattributename1",
"newAttributeType": "new_myattributetype1"
}
]
},
{
"originalEntityType": "(.+)",
"originalEntityId": ".+",
"newEntityId": "othertype",
"attributeMappings": [
]
}
]
}
]
}
]
}
And agent is configured with option data_model=dm-by-attribute
- Headers:
Fiware-Service: city012
Fiware-ServicePath: /electricidad
- Entity:
{
"id":"cabeco_33",
"type":"luminaria",
"temperatura": {
"type": "string",
"value": "13"
}
}
This case creates in database_name db a table with name tableprefix_cabeco_tablesufix_new_myattributename1
- Headers:
Fiware-Service: city012
Fiware-ServicePath: /electricidad
- Entity:
{
"id":"otherthing_22",
"type":"container",
"temperatura": {
"type": "string",
"value": "13"
}
}
This case creates in database_name db a table with name tableprefix_othertype_tablesufix
The contents of /path/to/conf/name_mappings.conf provide the name mappings used by Cygnus at startup time. However, they can be managed using a REST-based API. Using such API, new name mappings can be defined, modified or deleted.
The name mappings API is described in this piece of documentation.
When Cygnus was deployed using a docker image Installation via docker and namemappings files are provided in a volume, then updating these namemappings should be done in a way that i-node of that namemapping file would not be modified. This way cygnus will read namemappings changes without need of be restarted. To a achieve that is recomended not use editors like vim or emacs which uses temporal files while editing, and use simple editors like nano.
Please, check the specific documentation for this custom interceptor in the Flume extensions catalogue for cygnus-ngsi agent.