Periodically run execution node health checks with a system job

[AlanCoding]

SUMMARY

We have this gap in the heartbeat logic, which @kdelee has complained about.

Basically, if we don't have any reason to believe that an execution node is broken, we won't ever discover this automatically. If an execution node is in an error state, then we periodically re-check it. However, if it fits these two conditions we don't check it

• its last known health check status was successful
• its receptor daemon is connected to the mesh, and we see it from the receptor health check in the receptor status command

This somewhat intentionally will not capture the case where the remote node's receptor is working, but has its ansible-runner install messed up somehow. Just imagine that someone re-installs it incorrectly. We will never mark that node as offline unless someone runs a health check against it manually.

The problem is, it's poor form for the system to have full control over this kind of check. Users will have their own particular reasons that an execution node might get disrupted, and they should be in control of how often this (completely periodic) fallback check should happen, or if it should happen. The best tool for that is a system job.

ISSUE TYPE

• New or Enhanced Feature

COMPONENT NAME

• API

[AlanCoding]

There would be a strong argument for making this run health checks just on execution nodes. So, if anyone wants to vote for that, I think it would tip the scales.