[Investigation] Smart Limits for Detection Rules

[Mikaayenson]

Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

With #3842 we limited the number of versions per rule released down to 2 versions. We've been asked by D&R and our PM @approksiu to investigate the LOE to implement smart limits elastic/kibana#187645 . There are hurdles to implement this feature for both publishing and ingesting rules. This issue tracker will help to understand any technical limitations on our end.

This issue is purely an investigation to provide more insight to what limitations /options we can control from this side.

Desired Solution

Details on options / hurdles to implement smart limits. Examples:

• Limit based on latest over a period of time?
• Limit based on rule changes?
• etc.

Considered Alternatives

We may not need to implement complicated smart limits at all based on some initial testing by @xcrzx who may have a solution that supports the number of rule versions published to a limit that we theoretically would not reach (considering the qualitative balance between the number of rules vs maintaining and publishing a number of high quality rules).

This would be another option if they are successful.

Additional Context

• [Security Solution] Smart limits for the package with prebuilt rules kibana#187645
• Since we rely on epr for historical context, we will need to traverse history and at least ship one time or maintain someway a list of the baseline (initial) rule so that the rule customization feature coming upstream will know how to best diff and provide a better out of box experience for users. We need to consider that some rules may have been released at a stack version (with a rule version) that we no longer release to. We also need to consider other things like rule forks and rule releases against different stack versions.

[Mikaayenson]

Update Oct 24

@xcrzx Was able to test loading rules with a limit of about 20k rules. We may be able to remove the limit altogether starting in 8.17. We still need to wait on the serverless testing to confirm.

Long term, we all agree it would be better to have a distribution mechanism that uses git (where kibana pulls directly from the detection-rules repo branches).