[OSQuery Manager] Remove or greatly increase OSQueryMaxTimeout

[qcorporation]

Context

One of our customers are running into timeout issues with executing Yara disk scans, which could take hours based on the environment
From the field:

We would like to be able to use this for actions requiring a full system scan, which would, of course, take more than 15 minutes.
The author has pointed out that osqueryMaxTimeout = 15 * time.Minute in osquerybeat.go

The limit of 15 minutes was put there on purpose to avoid long-running queries blocking other possible query executions.
At some point, we need to weigh the benefits and possible implications of changing this hard-coded timeout.

Discussion

• What should that max limit be if any?
• Will we get complaints down the road that the queries never return or the queries never executed (because there is one long-running blocking query), or, for example, the queries triggered by the security alert expired or were delayed by so much that they became useless?

These are the things to consider when increasing the max timeout

Tasks

• Work with the PM team to define a reasonable max time out value or no value at all
• Implement the change in line with the next stack release.
• Create documentation with regards to increasing the timeout and it's implications at the stack level

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[qcorporation]

@jamiehynds, @roxana-gheorghe Can you please weigh in on the questions within the ticket?
@andrewkroh : great to get your feedback on this

cc.ing @mjwolf @ThorbenJ

[ThorbenJ]

I would argue to let the customer decided by making this configurable in the policy (not remove it) and to document the possible impact of increasing the timeout.
The customer I am working with already points out that this issue is not exclusive to Elastic. Other endpoint vendors have similar issues with long running (collection) jobs delaying/blocking automated or detection triggered jobs.

Stretch goals might be to provided more insight to the job queue and job status. To do some sort of multi processing; e.g. one for long jobs and one for (Automated) short jobs - but most important right now, I would argue, is to give end users the choice/control over timeouts.

[jamiehynds]

++ to making it configurable in the policy, but with a maximum value to ensure we don't block other queries. As for a reasonable max, I'm open to suggestions. How do we feel about a 24 hour max, too long/short?

[qcorporation]

I would argue to let the customer decided by making this configurable in the policy (not remove it) and to document the possible impact of increasing the timeout. The customer I am working with already points out that this issue is not exclusive to Elastic. Other endpoint vendors have similar issues with long running (collection) jobs delaying/blocking automated or detection triggered jobs.

Stretch goals might be to provided more insight to the job queue and job status. To do some sort of multi processing; e.g. one for long jobs and one for (Automated) short jobs - but most important right now, I would argue, is to give end users the choice/control over timeouts.

@ThorbenJ, thanks for the feedback; I've added a task to document the implications at the deployment level when there is a code change to address this.
Do you happen to have any telemetry on how long could a Yara file scan take before a user would abandon the results?

[ThorbenJ]

I've shared a link to this with the particular customer I am working with, I will comment if they have an opinion on the max. To me 24h seems reasonable. However I would not make that the default, I think we should keep the current 15min. or there abouts default. And make the config per policy so different hosts with different policies have different timeouts.

[qcorporation]

I've shared a link to this with the particular customer I am working with, I will comment if they have an opinion on the max. To me 24h seems reasonable. However I would not make that the default, I think we should keep the current 15min. or there abouts default. And make the config per policy so different hosts with different policies have different timeouts.

@ThorbenJ Thanks, any information would be great.
I agree; we are not changing the defaults within this ticket, just the maximum limit for the timeout.

Thank you, @mjwolf, for posting the PR. If folks are in agreement, let's try to get it in before the FF

[mjwolf]

I will increase the max timeout to 24h. Two PRs are needed to do this, as Kibana also enforces the maximum timeout that can be set. #42356, elastic/kibana#207276.

This will be a global increase in the max timeout. If we want to set it per-policy or add other things like a work queue, I think that would be more work than can be done before the 9.0 or 8.18 feature freeze.