Test check cancellation / db failure during shutdown

[moshloop]

A failed connection to the internal DB (e.g. during DB shutdown) could result in an increase of failed_count

• Ensure failed db calls result in an invalid and not failed metric
• Wait until all running checks have completed (or have been cancelled) before initiating the shutdown of the DB
  • For checks that require a non-trivial implementation to support cancellation, log backlogged bugs

[adityathebe]

Ensure failed db calls result in an invalid and not failed metric

@moshloop Is that invalid though? I think we should consider invalid if the spec is incorrect.

We probably need to add a distinction between an error and a check not passing.