Moonbeam watcher issues

[boastal]

Description

Moonbeam watcher seems to have issues after a moonbeam upgrade, the problem is solved after restarting the guardian service.

[ncavedale-xlabs]

Here are some logs from testnet:

Guardian logs:

Dec 09 09:23:35 t-guardian-01 guardiand[2224758]: 2024-12-09T09:23:35.330Z        INFO        root.moonbeam_watch        observation confirmed        {"msgId": "16/000000000000000000000000bc976d4b9d57e57c3ca52e1fd136c45ff7955a96/1408", "txHash": "0x7759c6f9008c0af8f010e79c8989fe1116ba9145fdb13229e8172adfd36ca661", "blockHash": "0xef65fc645de142516c397c18331bca624304195c4c6b0eadc90081be355cdc25", "target_blockNum": 9815551, "current_blockNum": "9815551", "finality": "finalized", "current_blockHash": "0xef65fc645de142516c397c18331bca624304195c4c6b0eadc90081be355cdc25"}
Dec 09 13:45:54 t-guardian-01 guardiand[2224758]: 2024-12-09T13:45:54.565Z        ERROR        root.moonbeam_watch        error while processing header subscription        {"error": "websocket: close 1006 (abnormal closure): unexpected EOF"}
Dec 09 13:45:54 t-guardian-01 guardiand[2224758]: 2024-12-09T13:45:54.565Z        ERROR        supervisor        Runnable died        {"dn": "root.moonbeam_watch", "error": "returned error when NODE_STATE_NEW: error while processing message publication subscription: websocket: close 1006 (abnormal closure): unexpected EOF"}
Dec 09 13:45:54 t-guardian-01 guardiand[2224758]: 2024-12-09T13:45:54.567Z        INFO        supervisor        rescheduling supervised node        {"dn": "root.moonbeam_watch", "backoff": "704.495768ms"}
Dec 09 13:45:55 t-guardian-01 guardiand[2224758]: 2024-12-09T13:45:55.272Z        INFO        root.moonbeam_watch        Starting watcher        {"watcher_name": "evm", "url": "ws://t-moonbeam-02.xxxx.xxxx:9944/", "contract": "0xa5B7D85a8f27dd7907dc8FdC21FA5657D5E2F901", "networkName": "moonbeam", "chainID": "moonbeam", "unsafeDevMode": false}
Dec 09 13:45:55 t-guardian-01 guardiand[2224758]: 2024-12-09T13:45:55.277Z        ERROR        supervisor        Runnable died        {"dn": "root.moonbeam_watch", "error": "returned error when NODE_STATE_NEW: failed to determine finality: failed to connect to endpoint: dial tcp 10.80.0.172:9944: connect: connection refused"}
Dec 09 13:45:55 t-guardian-01 guardiand[2224758]: 2024-12-09T13:45:55.278Z        INFO        supervisor        rescheduling supervised node        {"dn": "root.moonbeam_watch", "backoff": "492.910652ms"}
Dec 09 13:45:55 t-guardian-01 guardiand[2224758]: 2024-12-09T13:45:55.771Z        INFO        root.moonbeam_watch        Starting watcher        {"watcher_name": "evm", "url": "ws://t-moonbeam-02.xxxx.xxxx:9944/", "contract": "0xa5B7D85a8f27dd7907dc8FdC21FA5657D5E2F901", "networkName": "moonbeam", "chainID": "moonbeam", "unsafeDevMode": false}
Dec 09 13:45:55 t-guardian-01 guardiand[2224758]: 2024-12-09T13:45:55.774Z        ERROR        supervisor        Runnable died        {"dn": "root.moonbeam_watch", "error": "returned error when NODE_STATE_NEW: failed to determine finality: failed to connect to endpoint: dial tcp 10.80.0.172:9944: connect: connection refused"}

Moonbeam node logs show no errors. You can see the service was restarted (because we deployed a new moonbeam release). After this, the node started to process blocks normally:

Dec 09 13:45:54 t-moonbeam-02 moonbeam[3017404]: 2024-12-09 13:45:54 [Relaychain] :trophy: Imported #18514510 (0x4a3f…71a0 → 0xd292…4249)
Dec 09 13:45:54 t-moonbeam-02 moonbeam[3017404]: 2024-12-09 13:45:54 [Relaychain] :new: Imported #18514510 (0x4a3f…71a0 → 0xc7fd…5393)
Dec 09 13:45:54 t-moonbeam-02 systemd[1]: /etc/systemd/system/moonbeam.service:35: Unknown key name '--      --name' in section 'Service', ignoring.
Dec 09 13:45:54 t-moonbeam-02 systemd[1]: Stopping "Moonbeam service"...
Dec 09 13:46:37 t-moonbeam-02 systemd[1]: moonbeam.service: Deactivated successfully.
Dec 09 13:46:37 t-moonbeam-02 systemd[1]: Stopped "Moonbeam service".
Dec 09 13:46:37 t-moonbeam-02 systemd[1]: moonbeam.service: Consumed 3w 5d 3h 50min 27.822s CPU time.
Dec 09 13:46:37 t-moonbeam-02 systemd[1]: Started "Moonbeam service"

[bruce-riley]

After some offline discussion, it seems like the scope of this issue might be that the Moonbeam watcher doesn't survive a Moonbeam node restart.