Dispatch processing externally from web server

[benjimin]

The OWS WPS spec facilitates two modes:

• Synchronous: for processes that can be performed in a few seconds, the result is returned in the initial request response. (This would be ideal for providing interactive online features, but probably isn't applicable for ODC until data loading performance is improved.)
• Asynchronous: the initial response specifies a location to poll for the request status.

The problem is that asynchronous tasks should not be executed by the web host itself directly.

• Processing failures leave dangling promises, where the client is left holding its breath with no signal to stop polling.
• Poor scaling: the server becomes unresponsive (as its few available slots become occupied) suddenly and without warning, even if the demand only increases gradually, causing incoming requests to fail before new resources can spin up.

The solution will be to dispatch tasks to some form of queue, entrusting the core processing to a separate resource. (The implementation could involve SQS and a dask cluster, just for one example.) This will decouple the processing performance from the HTTP responsiveness. The decoupling may also reduce the resources that need be kept idling.

[benjimin]

A queue-based design would also facilitate a serverless implementation, to reduce hosting costs (assuming infrequent demand).