You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
key getter does not work when a CaseMapper is required to translate field names:
[info] Cause: java.lang.IllegalStateException: Leaf key field user_name does not exist in record class class org.apache.beam.sdk.extensions.smb.ParquetEndToEndTest$Event
[info] at org.apache.beam.sdk.extensions.smb.ParquetBucketMetadata.toKeyGetters(ParquetBucketMetadata.java:374)
[info] at org.apache.beam.sdk.extensions.smb.ParquetBucketMetadata.extractKeyPrimary(ParquetBucketMetadata.java:240)
[info] at org.apache.beam.sdk.extensions.smb.BucketMetadata.getKeyBytesPrimary(BucketMetadata.java:269)
[info] at org.apache.beam.sdk.extensions.smb.BucketMetadata.primaryComparableKeyBytes(BucketMetadata.java:302)
[info] at org.apache.beam.sdk.extensions.smb.SortedBucketSource$BucketedInput.lambda$createIterator$1(SortedBucketSource.java:551)
[info] at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators$6.transform(Iterators.java:785)
[info] at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
[info] at org.apache.beam.sdk.extensions.smb.BufferedIterator.refill(BufferedIterator.java:42)
[info] at org.apache.beam.sdk.extensions.smb.BufferedIterator.<init>(BufferedIterator.java:35)
[info] at org.apache.beam.sdk.extensions.smb.SortedBucketSource$BucketedInput.lambda$createIterator$2(SortedBucketSource.java:553)
Clean fix would imply to give the bucket metadata a way to transform the key field into the proper chained calls Method[].
As construction of the metadata on the read side is done from json file only, injecting this custom mapping isn't straightforward
key getter does not work when a CaseMapper is required to translate field names:
repro: https://github.com/spotify/scio/compare/parquet-type-caseformat?expand=1
The text was updated successfully, but these errors were encountered: