Consider to move post-register actions to register phase

[kaisecheng]

Currently, this plugin splits the initial setup to two parts, register and post-register (finish_register)

register mainly do two things

• basic config checking that does not require to communicate to Elasticsearch
• initialise connection pool which requires to pass health check getting 200 from Elasticsearch

post-register mainly do server check

• check license
• check support version
• install index template
• setup ilm

Issue

The problem with performing post-register actions after the register phase is that the pipeline perceives itself as having started successfully and begins ingesting data into the queue. Meanwhile, the post-register actions may signal to shut down the pipeline before the plugin taking any event if the setup or checking fails.

When the pipeline uses a memory queue, shutting it down requires draining the queue first. However, the events cannot be consumed in this case because the plugin fails during the post-register phase.

The shutdown process is stuck, and it is not possible to trigger a reload of the pipeline

Perform all checking in register phase
The good part is that we have a complete pipeline, rather than showing 'pipeline started' with only partial success. Once the pipeline starts, it is ready to send events out, no events stuck in the middle

Why was it designed to be two parts?
In the past, one concern was to make the plugin fail fast, perform minimal checking, and then allow ingestion to start.
With a persistent queue, processing events before completing the output setup works fine.

Consideration

Post-register does not do retries at the moment, but we should probably implement that. We need to consider the time taken for retries during the register phase, how long we are willing to wait, and how this will pause the ingestion process

[kaisecheng]

Todo: investigate reason why ILM setup was done post register before moving it to register.