-
Notifications
You must be signed in to change notification settings - Fork 514
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stable TableRow converted from BQ types #5536
base: main
Are you sure you want to change the base?
Conversation
Coder[TableRow] is destructive (it is a dummy JSON serializer), we should make sure that the TableRow object converted from a BQ model is stable after serialization. We currently have an issue with - long that are serialized as string to avoid overflow - float that are read back as double - json that is read as nested TableRow
val provider: OverrideTypeProvider = | ||
OverrideTypeProviderFinder.getProvider | ||
val s = q"$tree.toString" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we were forcing toString
before converting back to desired type
fdd4fbb
to
d8d30c8
Compare
// f is a field from TableRow. | ||
// Jackson ObjectMapper will fail with such key | ||
key <- Gen.alphaStr.retryUntil(_ != "f") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm wondering what happens if a BQ table has a field named f
. It's probable that we can't use the TableRow
API
d8d30c8
to
d76d226
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #5536 +/- ##
==========================================
+ Coverage 61.43% 61.45% +0.01%
==========================================
Files 312 312
Lines 11106 11106
Branches 750 748 -2
==========================================
+ Hits 6823 6825 +2
+ Misses 4283 4281 -2 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally all the TableRow -> scala
cast logic should be shared with TableRowOps
Coder[TableRow] is destructive (it is a dummy JSON serializer), we should make sure that the TableRow object converted from a BQ model is stable after serialization.
We currently have an issue with
As side effect, I needed to avoid
toString
conversion when converting back a BQ typed model from aTableRow