Write checkable create & delete sla history events

[yhabteab]

Why do we need this!

Currently we are generating the SLA history events only when e.g. there were state change and downtime start and end events for the checkables. Under some circumstances (if a checkable is created once and never deleted) this should be sufficient. However, when you e.g. delete a host and create it again a couple of days later and want to generate sla reports for this host at the end of the week, the result can vary depending on which state the host had before it was deleted. In order to be able to generate sla reports as accurately as possible, we decided to track the checkable creation and deletion time on top of the existing info. And since Icinga 2 doesn't really know when an object has been deleted (at least not in a simple way), this PR should take care of it.

Though, Icinga DB doesn't know when an object has been deleted either, it just takes the time the delete event for that object arrived and puts it into the new table. Meaning when you delete checkables while Icinga DB is stopped, the events Icinga DB would write after it is started won't reflect the actual delete/create event. Though, there is no better way to handle this gracefully.

Config sync

The upgrade script for 1.3.0 generates a created_at sla lifecycle entry for all existing hosts and services once it is applied as proposed in #566 (comment). Thus, all special cases such as the implementation of a custom fingerprint type for services¹ and performing an extra query to retrieve host IDs from the database for runtime deleted services, have been removed.

Implementation

The new table sla_history_lifecycle has a primary key over (id, delete_time) where delete_time=0 means "not deleted yet" (the column has to be NOT NULL due to being included in the primary key). id is either the service or host ID for which that sla lifecycle is being generated. This ensures that there can only be row per object that states that the object is currently alive in Icinga 2.

Initial sync

Icinga DB performs a simple INSERT statement for Host and Service types after each initial config dump unconditionally, but matches on hosts and services that don't already have a create_time entry with delete_time = 0 in the sla_lifecycle table, and sets their create_time timestamp to now. Additionally, it also updates the delete_time of each existing sla_lifecycle entries whose host/service IDs cannot be found in the Host/Service tables. It's unlikely, but when a given Checkable doesn't already have a create_time entry in the database, the update query won't update anything. Likewise, the insert statements may also become a no-op if the Checkables already have a create_time entry with delete_time = 0.

Create

Nothing to be done here (all newly created objects will be covered by the bulk INSERT ... SELECT FROM host/service queries after the config dump).

Update

Nothing to be done here (object existed before and continues to exist).

Delete

Nothing to be done here (all deleted objects will be covered by general bulk sea_lifecycle queries after the config dump).

Runtime updates

Upsert

Performs an INSERT with ignore for duplicate keys for both create and update events (these look identical in the runtime update stream). If the object is already marked as alive in sla_history_lifecycle, this will do nothing, otherwise it will mark it as created now (including when an object that was created before this feature was enabled is updated).

Delete

It assumes that there exists a created_at sla_lifecycle entry for that checkable currently going to be deleted, and performs a simple UPDATE statement setting delete_time = now (i.e. updates the PK of the row) marking the alive row for the object as deleted. If, for whatever reason, there is no corresponding created_at entry for this checkable, that update statement becomes a no-op, as the upgrade script and/or the initial config dump should have generated the necessary entries for all existing objects that were created before this feature was available.

Footnotes

1. Before @julianbrost proposed a change in https://github.com/Icinga/icingadb/pull/566#issuecomment-2273088195, services had to implement an additional custom fingerprint type ServiceFingerprint which was used to also retrieve their host IDs when computing the config delta of the initial sync. By introducing this type, the necessity of having to always perform an extra SELECT query to additionally retrieve the host IDs was eliminated, as host ID is always required for the sla lifecycles to work. ↩

[julianbrost]

I just had an idea how we could call that type of SLA history after we didn't really come up with good name for this initially: lifecycle

[Al2Klimov]

Please don’t force-push for now.

[yhabteab]

I've pushed the new transparent Screenshots and also changed the main function a bit, so I won't be pushing anything from now on unless someone requests a change.

[oxzi]

I would change the tone of the first two paragraphs to make it sound more like a technical document and less like a PR post. For example, the "legally binding" part could be dropped, as it must not be the case. Furthermore, Icinga Reporting is (currently) not the "optimal complement", but the only complementing UI.

Please don't take this the wrong way. If I would read this as a technical user - and this is technical documentation -, I would have the feeling that somebody wants to sell me something and not like I am going to find the details here.

[oxzi]

As discussed earlier, please include the state's magic numbers used within the database. This eases debugging and understanding when using this technical document together with the own setup.

[oxzi]

Without the mapping from states to their numeric representation, this is quite cryptic. Thus, either introduce a mapping before or make it more explicit, like listing all acceptable previous_hard_state values.

[yhabteab]

I'm not sure if I understand you correctly, but I don't quite agree that this being cryptic, rather it's the exact opposite. These are just normal Icinga 2 host - service states that have their own documentation except the PENDING state. Apart from that, I have actually listed the possible values that the previous_hard_state column represents and you want me to add another listing here?

[oxzi]

Please try to put yourself in a reader's position, who knows a bit about Icinga, but definitely no implementation details. When reading the comparison in this line, it may be unclear what "greater than" means in this context, as no numerical representation or hierarchy of states was introduced.

Apart from that, I have actually listed the possible values that the previous_hard_state column represents and you want me to add another listing here?

Based on this table, one would assume the following order: PENDING < OK < WARNING < CRITICAL.
And what is about UNKNOWN and the host states UP and DOWN?

Coming back to the if cases, there is

1. event.previous_hard_state is PENDING and
2. previous_hard_state is greater than OK/UP AND previous_hard_state is not PENDING AND checkable is not in DOWNTIME (this one misses the event. prefix, but I guess it was just forgotten?).

Shouldn't it be possible to rewrite the second clause to event.previous_hard_state is not in {OK, UP} AND checkable is not in DOWNTIME as event.previous_hard_state cannot be PENDING due to the first check? Doing so eliminates any order which has to be explained otherwise.