Caveats in FanOut implementation

[jmalins]

Great project with a great name (to a former EE).

The current FanOut implementation has a few caveats that are probably worth documenting, since they may be issues for some usage patterns.

• The state of the running fan-outs in maintained in memory of the dispatcher activity, failure of that process means the sub-states are lost. If sub-activities are stateless, it's not a problem, because the parent can retry, but if the children are stateful or non-idempotent, it could be a problem.

• The DescribeExecution polling of the sub-activity states will be subject to throttling (200 unit bucket / account, refilling at 1 unit/sec). If your use case involves many concurrent parent executions, long-running activities or high fan-out / execution, it could be a problem.

I'm sure there's tons of use cases where neither of these will be a problem. In our case, we considered a similar fanout implementation, but had to give it up after analysis revealed we'd run afoul of the throttle limits at our expected volume.

[derek-pryor]

Thanks for the feedback. You are correct in your observations and I'll be updating the documentation to better convey these limitations.

You comment also made me realize that there is a bug with error handling (when an error is detected in a sub-process). A fix for that is being worked on.

[derek-pryor]

I just added some new functionality that touches on this. PR #4 adds a non-blocking version of fanout that keeps state in a dictionary that is passed to the function. It doesn't completely solve how to handle failure cases, as right now it still will cancel any running sub-stepfunctions on a failure.

In addition a status_delay has been added to slow the status polling process to not hit the throttling limit.