Re-architect the batcher service so that it works as a Lambda

[kenoir]

Part of wellcomecollection/platform#5784

This is the specific task of refactoring the ECS service so that it can operate as an SQS triggered Lambda.

• Batcher Lambda, infrastructure and build #2728
• Add queue and event source for pipeline lambdas (specifically batcher) #2738
• ...

It should be preceded by the RFC described in the linked issue, and followed by an ADR describing the change made.

[paul-butcher]

One of the things I want us to consider is that the whole SQS-in/SNS-out business has its tentacles all through the invocation part of this application. Ideally, we should be able to invoke it not just by Lambda (which would still be SQS), but also from a CLI by passing it a list of strings (probably on STDIN), without having to run pretend AWS services.

Throughout the whole pipeline, we have big chains of Akka stream stuff that might be better, both for readability and reuse, if we crack them apart and name them, preferably restricting the interface parts (SQS/SNS or STDIO or whatever) to the very periphery.

[pollecuttn]

@kenoir can this be closed now?

[kenoir]

@kenoir can this be closed now?

I think this one should include the work to actually shift the batcher code over too? Cc @agnesgaroux.