issues with iot agent json (1.14.0) during registration

[chicco785]

when iot agent tries to register services (already existing):

time=2020-07-29T15:07:05.758Z | lvl=DEBUG | corr=1d9ddc2a-d24a-4a9e-95ed-bba29bda754c | trans=1d9ddc2a-d24a-4a9e-95ed-bba29bda754c | op=IoTAgentNGSI.NorthboundServer | srv=n/a | subsrv=n/a | msg=Using config:

{
    "logLevel": "DEBUG",
    "timestamp": true,
  ....
    "iotManager": {
        "host": "iot-agent-manager-iot-agent-manager",
        "port": 8082,
        "path": "/iot/protocols",
        "protocol": "JSON",
        "description": "JSON IoT Agent (Node.js version) supports HTTP/MQTT/AMQP protocols",
        "agentPath": "/config/iot",
        "url": "http://iot-agent-manager-iot-agent-manager:8082"
    },
    "types": {},
    "service": "EKZ",
    "subservice": "/",
    "providerUrl": "http://json-iot-agent:4041",
    "autocast": true,
    "deviceRegistrationDuration": "P1M",
    "defaultType": "Device",
    "defaultResource": "/iot/json",
    "explicitAttrs": false,
    "iotaVersion": "1.14.0",
    "multiCore": false
}
 | comp=IoTAgent
time=2020-07-29T15:07:05.771Z | lvl=INFO | corr=1d9ddc2a-d24a-4a9e-95ed-bba29bda754c | trans=1d9ddc2a-d24a-4a9e-95ed-bba29bda754c | op=IoTAgentNGSI.ContextServer | srv=n/a | subsrv=n/a | msg=Loading NGSI Contect server routes | comp=IoTAgent
time=2020-07-29T15:07:05.907Z | lvl=DEBUG | corr=c84cf275-e2ae-435c-9e77-b49324f63840 | trans=c84cf275-e2ae-435c-9e77-b49324f63840 | op=IoTAgentNGSI.IOTAMService | srv=n/a | subsrv=n/a | msg=Sending registration to the IOTAM:
{
    "url": "http://iot-agent-manager-iot-agent-manager:8082/iot/protocols",
    "method": "POST",
    "json": {
        "protocol": "JSON",
        "description": "JSON IoT Agent (Node.js version) supports HTTP/MQTT/AMQP protocols",
        "iotagent": "http://json-iot-agent:4041/config/iot",
        "resource": "/iot/json",
        "services": [
            {
                "apikey": "xxxx",
                "entity_type": "AirQualityObserved",
                "resource": "/iot/json",
                "service": "EKZ",
                "service_path": "/EnvironmentManagement/Demo",
                "attributes": [
                    {
                        "type": "Number",
                        "name": "CO",
                        "object_id": "CO"
                    },
                    {
                        "object_id": "NO",
                        "name": "NO",
                        "type": "Number"
                    }
                ],
                "static_attributes": []
            }
        ]
    }
}

time=2020-07-29T15:07:06.095Z | lvl=ERROR | corr=c84cf275-e2ae-435c-9e77-b49324f63840 | trans=c84cf275-e2ae-435c-9e77-b49324f63840 | op=IoTAgentNGSI.Alarms | srv=n/a | subsrv=n/a | msg=Raising [IOTAM-ALARM]: "Wrong status code connecting with the IoTAM" | comp=IoTAgent
time=2020-07-29T15:07:06.095Z | lvl=ERROR | corr=c84cf275-e2ae-435c-9e77-b49324f63840 | trans=c84cf275-e2ae-435c-9e77-b49324f63840 | op=IoTAgentNGSI.IOTAMService | srv=n/a | subsrv=n/a | msg=IOTAM-001: Error updating information in the IOTAM. Status Code [500] | comp=IoTAgent

IoT agent managers reports:

time=2020-07-29T15:07:06.095Z | lvl=ERROR | corr=n/a | trans=n/a | op=IoTAgentNGSI.Alarms | msg=Releasing [MONGO-ALARM]
time=2020-07-29T15:07:06.096Z | lvl=ERROR | corr=n/a | trans=n/a | op=IoTAgentNGSI.Alarms | msg=Raising [MONGO-ALARM]: {"driver":true,"name":"MongoError","index":0,"code":11000,"errmsg":"E11000 duplicate key error collection: iot-agent-manager.configurations index: apikey_1_resource_1_protocol_1 dup key: { : \"xxx", : \"/iot/json\", : \"JSON\" }"}
time=2020-07-29T15:07:06.098Z | lvl=ERROR | corr=n/a | trans=n/a | op=IoTAgentNGSI.Alarms | msg=Releasing [MONGO-ALARM]
time=2020-07-29T15:07:06.100Z | lvl=ERROR | corr=n/a | trans=n/a | op=IoTAgentNGSI.Alarms | msg=Raising [MONGO-ALARM]: {"driver":true,"name":"MongoError","index":0,"code":11000,"errmsg":"E11000 duplicate key error collection: iot-agent-manager.configurations index: apikey_1_resource_1_protocol_1 dup key: { : \"xxxx\", : \"/iot/json\", : \"JSON\" }"}

[fgalan]

As far as I understand IOTAManager is responsing with 500 to IOTA under this circustance.

What do you think expected behavior should be? Responding with 400 with a more specific cause?

[chicco785]

shouldn't be the configuration overwritten? who is the latest source of "correct" configuration?

[fgalan]

shouldn't be the configuration overwritten?

That's an option, for sure. But maybe it's safer delete + create instead of overwrite.

[chicco785]

sure, but when the iot agent starts and send the command, is doing it automatically, so either the logic in the iot-agentlib is changed (delete and recreate), or the logic on iot agent manager overwrite existing services

[fgalan]

Looking again to the trace, which operation is the one which is failing? POST /iot/protocols ?

[chicco785]

yes, correct

 "url": "http://iot-agent-manager-iot-agent-manager:8082/iot/protocols",

and the part generating issue should be the services since then the issue are the conflicting apikeys

[fgalan]

Looking to the index that is causing the duplicated key error at MongoDB:

apikey_1_resource_1_protocol_1

it seems it doesn't have to do with services alone, but with the combination of apikey (which comes from sercice) + resource + protocol.

I think is a good idea not allowing two IOTAs register the same apikey+resource+protocol combination. And, in the case of IOTAManager getting a registration attempt on an existing apikey+resource+protocol is probably due to some misconfiguration of the IOTA (which, at least should use a different apikey). In that case, it seems better to report an error (in the form of 400 response) than overriding, so the admin can be aware of the situation and fix the conflicting IOTA configuration.

[chicco785]

concrete use case:

1. iot agent during and update crash, configuration is updated locally, but not pushed to iot agent manager
2. users don't notice anything because agent is in ha.
3. failed iot agent restarts and tries to push up-do-date configuration (recall that the agent is "the master" of the information while the manager is just a proxy)
4. users query iot agent manager, and get mad because don't understand what happened to your service configuration.

this means that configuration recovery will never work toward iot agent manager, unless you first delete everything.