[Request] Documentation for Manual Run of Security rules

[nkhristinin]

Description

What: We introduce manual rule Runs for security solution.
Why: Users will be able to run rule in the past and specify the date range.

Use cases we cover:

1. User form rule details page execute manual rule run and specify time rang
2. They can go to Executions tab and see the backfill group, with information about how much tasks scheduled and in progress/pending
3. They can stop the whole group for manual run
4. They can see in execution log result of rule executions and filter by rule type

There additional issue for UX copy: #5265

A little bit of technical background, how it works, which should help with better naming

Let's say we have rule with 5m interval
rule execution log - it represents the results of a single rule execution. it can be running/succeded/failed.
When the user executes Manual rule runs (14:00-16:00)- it creates the Backfill group (we probably need come up with better naming).
Backfill group - it's something, that contains the start and end date range, status of the whole group, and rule info.
Also Backfill group has scheduled entries - it's a list of tasks of potenial rule executions.
When the task manager is free it starts to schedule those tasks - which execute the rule, and then the result of this execution appears in the rule execution log.
scheduled entry - can be pending/running/error/complete
The whole backfill group also can be pending/running/error - depends on status of scheduled entires.
After all scheduled entries are complete - Backfill group is deleted.
We can Delete/Stop only the whole backfill group, but not individual backfill group.

Background & resources

• PRs:
  • Backfill groups
  • Execution log
  • New alert schema fields:
    • kibana.alert.intended_timestamp:
      Add intended timestamp kibana#191717
    • kibana.alert.rule.execution_type: Execution type field kibana#195884
  • PR that removes feature flag: Remove feature flag for manual rule run kibana#193833
• Issues/metas:
  • Main meta for manual runs: https://github.com/elastic/security-team/issues/2840
  • Design change to auto-refresh button: https://github.com/elastic/security-team/issues/9690
  • New alert fields: https://github.com/elastic/security-team/issues/10459
• Point of contact: @approksiu @e40pud @nkhristinin
• Test environments: Provided in Slack channel

Which documentation set does this change impact?

ESS and serverless

ESS release

8.15 8.16

Serverless release

Monday, July 29, 2024 Tuesday, October 14, 2024

Feature differences

None

API docs impact

• See comment here

Prerequisites, privileges, feature flags

None

Doc plan

• Update ESS and Serverless UI docs:
  • Base functionality
  • Limitations and best practices: What we discussed in Slack:
    • Manual runs will execute with low priority and limited concurrency, meaning they might take longer to complete, especially if there are many rules to backfill. Scheduled rules has higher priority and some limited concurrency. But limit is higher
    • Rule parameters are fixed at the time of manual rule run scheduling, meaning any updates to the rules afterward will not be reflected in the manual rule run. For example:
      1. We create a rule - Name - "Rule 1", interval - 5 min, query: "*"
      2. We execute manual rule run, and let's say it's long process taking hours
      3. Immediately after that we a changing rule name to "Rule 2", interval to 10 hours and query to "host.name:2"
      4. All rule execution which are planned after manual rule run executed will use original parameters - Name - "Rule 1", interval - 5 min, query: "*"
    • Users are responsible for ensuring the correct order of backfill jobs (manual rule runs) if dependencies exist between them. Incorrect scheduling can lead to inconsistent data filling.
  • The kibana.alert.intended_timestamp field has been added to the alert schema. This field appears in documents of alerts that were generated by manual rule runs. They convey the estimated time range of when the alert was created.
• Update rule API docs:
  • ESS
  • Serverless (Content on the Serverless API reference site is generated from OAS files and maintained/updated by dev teams) - https://www.elastic.co/docs/api/doc/kibana/operation/operation-performrulesbulkaction
• Add known issues (noted in the comments) to the 8.16 release notes - Noted at 8.16.0 Release notes #5941

Doc updates

NOTE: The feature is being released in Tech Preview in 8.15, so will need to use that label/admonition for ESS and Serverless docs.

Execution results: Make the following updates:

• In the intro para, add that users can now see how a rule was executed. Now, rules can be executed manually or auto-executed on a (preset?) schedule.
• Update rule-execution-logs.png - new image should show the updated Execution log table and the new Manual runs table.
• Add two bullets to the list of controls that allow users to filter what's in the Execution log table. The new items are:
  • Run type - New filter that allows users to display the execution type of a run. The two options are Manual and Scheduled.
  • Show source event time range: Toggling this setting on displays the Source event time range column. By default, this setting is toggled off.

Manage detection rules:

• Add an item to the list of actions that users can take from the Rules page. A good place might be after the "manage rules" list item.
• Add a new section for manually running rules. The new section should include the following details:
  • There are 3 different places where users can manually run a rule:
    • The rule's details page (go to the actions menu in the top right and select Manual run)
    • On the Rule pages, from the actions menu for an individual rule
    • On the Rule page, from the Bulk actions menu (up to 100 rules can be selected for a bulk-run)
  • When selecting a range of time to manually run a rule, you can only choose a past date and time.
    • If you don't change the default timerange selection, the rule will run today and the start time of the run will be 3hrs in the past.
  • Rules must be enabled for you to manually run them.
  • To stop a manual run, go to the Manual runs table and click Stop run in the Actions column.
    Stop rule run
    

[nkhristinin]

We should mention Manual rule limitations - to users in docs

[e40pud]

API docs impact

We need to update next sections:

1. Bulk action > Request body: add run as a new possible action's value
2. Bulk action > Request body: add new property

Name	Type	Description	Required
run	BulkManualRuleRun[]	Object that describes applying an manual rule run action.	No. Yes, if action is run.

3. We should add new type description similar to BulkDuplicateAction object and BulkEditAction object:

BulkManualRuleRun

• start_date field: (String, Required). Defines start date of the manual rule run.
• end_date field: (String, Optional). Defines end date of the manual rule run.

4. Response payload: add run to the actions list. Also, we should mention that we use attributes.results.updated to return rule objects that were scheduled for manual rule run during the run action execution.

[nkhristinin]

Known issue for Threshold rule:
If manual rule run cover the date range, which was already covered by scheduled rule executions it can produce duplicate errors.

Use case:

Let's say I have 4 events: 13:00, 14:00, 15:00, 16:00
Threshold rule - with 5 minute interval and 10 hours lookback time and threshold = 2
Normal rule execution - give me 2 alerts (14:00, 16:00)
If I run manual rule run from 12:00-16:10
I will have 2 alerts generated at 14:00 and 16:00.
Probably user should expect 0 alerts from manual rule run, as it was already covered by scheduled rule execution

[nkhristinin]

Known issue 2:
Suppression count for custom query rule can be updated wrong
https://github.com/elastic/security-team/issues/9870

[nastasha-solomon]

General notes from today's feature sync:

• @nastasha-solomon will aim to have docs done in time for the Monday, July 29, 2024 Serverless release.
• @nkhristinin will check if it's possible to only enable the feature flag for this feature in ESS.
• @nkhristinin or @e40pud to provide a test env for docs testing and screenshots either this week, or after Nastasha is back from PTO (which is the week of July 15)

[nkhristinin]

Trying to rewrite known Issues:

1. Manual rule run for threshold rule can produce duplicated alerts, if date range was already covered by regular rule execution.

No workaround

2. Manual rule run for custom query rule with suppression can produce wrong (bigger) number of doc count.

No workaround

[nastasha-solomon]

Holding off on merging the ESS and Serverless PRs until I learn more about the new gap fill functionality and how it affects the current manual run docs. Should know more by Wednesday Sep 25.

[nastasha-solomon]

This feature is being released in beta in 8.16. I'll need to update the ESS and Serverless docs to make sure they show the beta label, and not the technical preview label.

[nastasha-solomon]

The feature flag that enabled the following core manual run functionality in Serverless was merged last week via elastic/kibana#193833. Two new alert fields are being introduced as part of the manual run feature and are being released on slightly staggered timelines:

• kibana.alert.intended_timestamp: The PR that adds this field was merged before the feature flag was removed, so the field is included in this week's Serverless release.
• kibana.alert.rule.execution_type: The PR that adds this field was merged today, so it will be included in next week's Serverless release.

Action items for me:

• Once the latest Serverless branch (or set of commits?) is successfully promoted to prod, I can merge the manual run Serverless docs. [Serverless] Manual rule run docs #5589.
• Change the tech preview tag to beta in docs:
  • ESS
  • Serverless
• Apply editorial feedback that I received on Serverless docs to the ESS docs [8.16] Manual rule run docs #5631.
• Open a docs issue for the new kibana.alert.rule.execution_type field and update the alert schema table for Serverless and 8.16. Will need to doc the new field and update the description for the kibana.alert.intended_timestamp, since it will be slightly different with the introduction of the new kibana.alert.rule.execution_type field. (See Slack for more info.) [Request][Serverless][8.16] Document the new kibana.alert.rule.execution.type field being added for manual runs #5922

[nastasha-solomon]

Core docs merged for ESS and Serverless. Docs for the new kibana.alert.rule.execution_type field are being tracked in:

• [Request][Serverless][8.16] Document the new kibana.alert.rule.execution.type field being added for manual runs #5922
• [Request][Serverless][8.16] Document the new kibana.alert.rule.execution.type field being added for manual runs #5940