Add Batched lookups for streaming GRPC endpoints and BigTable

[lofifnc]

This aims to extend the usability for AsyncBatchLookupDoFn it adds:

• grpcLookupBatchStream for batched GRPC endpoints with streaming response.
• BigTableBatchDoFn for batch calls to BigTable including a cache.
  To make this happen the AsyncBatchLookupDoFn had to be changed to gracefully handle partial batched responses.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.48%. Comparing base (cc1e584) to head (d39cba0).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5521      +/-   ##
==========================================
+ Coverage   61.42%   61.48%   +0.05%     
==========================================
  Files         312      312              
  Lines       11104    11121      +17     
  Branches      757      796      +39     
==========================================
+ Hits         6821     6838      +17     
  Misses       4283     4283              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[RustedBones]

grpcSafeBatchLookup for calling endpoints which don't include missing values into the batch response.

I think we can keep the same API for that case. When receiving a batch response missing some entries form the batch request, we should fail fast and report those. It is then up to the user to filter out those exceptions if this is an expected behavior. See #5532

[lofifnc]

grpcSafeBatchLookup for calling endpoints which don't include missing values into the batch response.

I think we can keep the same API for that case. When receiving a batch response missing some entries form the batch request, we should fail fast and report those. It is then up to the user to filter out those exceptions if this is an expected behavior. See #5532

The rule of thump I tried to apply here is fail, if the client you're using is throwing an exception. BigTable not having an entry for a key is kind of something expected.
If we just return a failure I know, that I would make the mistake not adding handling for that and then on the first job run discovering that I'm getting a bunch of exceptions.

But we can merge your PR. And the error handling can be added to the . grpcSafeBatchLookup.

[lofifnc]

@RustedBones I've gone ahead and rebased this brach based on your PR #5532 and dropped the .safeGrpcLookup for now. This resolves all contentious parts, at least for me.

[RustedBones]

LGTM. Thanks for the contribution. It makes a lot of sense to handle missing response in batch in a lenient way instead of failing the job.

[RustedBones]

As a heads-up, we will probably move to the BigtableDataClient in some next minor release, see here. Expect some breaking changes in that part.