Monitoring Plugin Statistics

See: https://github.com/washort/hindsight/commit/b1187d27187a2125bcf9c541945d68a8e49b3695

Ingestion of the plugins.tsv file

Current Approach (one monolithic message)

Pros

• simple
• compact
• efficient if consuming all the stats

Cons

• size - as the the number of plugins grow the message just keeps getting larger (plan on a system with up to tens of thousands of plugins)
• if one is only interested in a small number of plugins this is most likely less efficient
  • todo: compare the message_matcher selection + processing speed to pulling out the desired data from a single message

Alternate Approach (one message per plugin)

Pros

• consume only what you need
• deterministic number of alerts per message during analysis
• columns can be put in fields with names instead of an array of indices in one field

Cons

• if everything is being consumed this should be more work for the system (measurements needed)
• spammy - more messages per minute
• overall data size is larger than the monolithic message since the header data has to be replicated and the message may now carry the actual column names as part of the data