You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It seems like KICS fails to scan a large repository (about 9000 files).
The reason seems to be unlimited concurrent workers in the Preparing Scan Assets stage.
Assuming there's no way to limit the amount of workers spawned - here's the function that goes crazy in spawning workers: pkg/analyzer/analyzer.go:Analyze
More specifically, this part in the code:
Our use-case is quite unique which is probably why this hasn't been a problem so far:
We're trying to scan a remote repository on a network mount, and I'm assuming the slower responsiveness makes the workers spawn more quickly than the previous ones are able to finish their job.
Even though this use-case isn't the obvious one, I still think it's a problem for this part to be unlimited (assuming I didn't just miss how to limit it).
The other constraining part is that we're running this scan on a Kubernetes pod, which is limited to 1000 PIDs.
Expected Behavior
Limit the amount of workers - if not as a customizable variable then at least so KICS doesn't crash
Actual Behavior
Scan fails due to failing to create a new thread. Here's the log:
Preparing Scan Assets: \runtime: failed to create new OS thread (have 989 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc
Steps to Reproduce the Problem
kics scan -p .
Run on a really big repository or a slow network drive.
Alternatively, artificially limit the maximum number of threads.
Specifications
(N/A if not applicable)
Version: v1.7.12
Platform: Kubernetes
Subsystem: N/A
The text was updated successfully, but these errors were encountered:
Hi,
It seems like KICS fails to scan a large repository (about 9000 files).
The reason seems to be unlimited concurrent workers in the
Preparing Scan Assets
stage.Assuming there's no way to limit the amount of workers spawned - here's the function that goes crazy in spawning workers:
pkg/analyzer/analyzer.go:Analyze
More specifically, this part in the code:
Our use-case is quite unique which is probably why this hasn't been a problem so far:
We're trying to scan a remote repository on a network mount, and I'm assuming the slower responsiveness makes the workers spawn more quickly than the previous ones are able to finish their job.
Even though this use-case isn't the obvious one, I still think it's a problem for this part to be unlimited (assuming I didn't just miss how to limit it).
The other constraining part is that we're running this scan on a Kubernetes pod, which is limited to 1000 PIDs.
Expected Behavior
Limit the amount of workers - if not as a customizable variable then at least so KICS doesn't crash
Actual Behavior
Scan fails due to failing to create a new thread. Here's the log:
Steps to Reproduce the Problem
Run on a really big repository or a slow network drive.
Alternatively, artificially limit the maximum number of threads.
Specifications
(N/A if not applicable)
The text was updated successfully, but these errors were encountered: