Make sure Falco and Falcosidekick are operating correctly

[marwinski]

What would you like to be added:

It must be ensured that (1) Falco- and Falcosidekick pods are running, (2) Falco can successfully push events to Falcosidekick, and (3) Falcosidekick can successfully send events to the event ingestor or a custom web server.

Why is this needed:

It must be clearly visible for a cluster owner if Falco events are lost. If that happens there would be no point in using Falco and it would pretend a false sense of security.

While we have monitoring through Gardener Managed Resources that Falco and Falcosidekick pods are running (1), this does not mean that events are successfully pushed to an event store (2,3).

Implementation Proposal:

Both Falco and Falcosidekick provide a metrics endpoint which should provide all the necessary information. In one of our installations the metics endpoint of Falcosidekick returned this alarming result:

# curl https://100.64.3.60:2801/metrics --insecure
[...]
# HELP falcosidekick_outputs
# TYPE falcosidekick_outputs counter
falcosidekick_outputs{destination="webhook",status="error"} 1981
falcosidekick_outputs{destination="webhook",status="ok"} 183
[...]

This shows that the vast amount of events (1981) could not be published and are lost while only 183 made it through.

Falco has a metrics endpoint as well but we haven't enabled it so far.