add redis_storage param

set 'max_crawl_scale' in values.yaml to indicate max possible scale, used to create crawl-instance-{0, N}
priority classes, each with lower priority
allows crawl instance 0 to preempt crawls with more instances (and lower priorities)
eg. 2nd instance of a crawl can preempt 3rd instance of another, and a new crawl (1st instance)
can preempt 2nd instance of another crawl

- ensure redis pod is deleted last
- start deletion in background as soon as crawl is done
- operator may call finalizer with old state: if not finished but in finalizer, attempt to
cancel, and throw 400 if already canceled
- recreate redis in finalizer from yaml to avoid change event

- support reconciling desired and actual scale
- if desired scale is lower, attempt to gracefully shutdown each instance
via new redis 'stopone' key
- once each instance above > desired scale exit successfully, adjust
the status.scale down to clean up pods. also clean up redis per-instance
state when scaling down

…have been running for >60 seconds, not immediately

…ulset

add placeholder for adding podmetrics as related resources
fix canceled condition

- async add_crawl_errors_to_db() call creates its own redis connection, as other one is supposed to be closed
by caller
- remove unneeded 'sync_db_state_if_finished'
- delete job after crawl finished tasks
- log if crawl finished but not yet deleted on next update

- pods explicitly deleted if spec.restartTime != status.restartTime, then updates status.restartTime
- use force_restart to remove pods for one sync response to force deletion
- update to latest metacontroller v4.11.0
- add --restartOnError flag for crawler

Co-authored-by: Tessa Walsh <[email protected]>

Co-authored-by: Tessa Walsh <[email protected]>

cancel crawl test: just wait until page is found, not necessarily done

…ditional logging for failed crawls, if enabled

- print logs: print logs for default container
- also print pod status on failure
- use mark_finished(... 'canceled') for canceled crawl
- tests: also check other finished states to avoid stuck in infinite loop if crawl fails
- tests: disable disk utilization check, which adds unpredictability to crawl testing!