-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error when serializing physical window functions to proto #13401
Comments
Yes, this seems like a bug @mwylde I don't know why this stopped working or why it isn't covered by tests. Note that just yesterday @buraksenn and @jcsherin completed migrating the last of the window functions in #13201 and we are about to remove BuiltInWindowFunction entirely (in other words I suspect the bug about serializing window functions doesn't exist on If you would like to release this fix in a |
@alamb in the PR we had this discussion #13201 (comment). There was a problem with physical plan roundtrip but I thought since |
take |
Unfortunately this does still exist on main, because
But despite the name, BuiltInWindowExpr actually now wraps a UDWF. |
Yes, this is indeed the case. None of the user-defined window functions are currently serialized. We missed this during conversion. I expect serialization of user-defined window functions to be simpler compared to built-in window functions. In built-in window serialization depended on the internals. You can see that in the serialization code here: datafusion/datafusion/proto/src/physical_plan/to_proto.rs Lines 130 to 148 in 6a4a280
We need to also extend |
Describe the bug
Recently, window functions were migrated to UDFs (as part of #8709). This appears to have broken protobuf serialization of some types of window functions (specifically, those that get planned into BuiltInWindowExpr), producing an error like
An example query of this is
which produces the logical plan
and the physical plan
The error comes from here:
datafusion/datafusion/proto/src/physical_plan/to_proto.rs
Line 124 in ccf6258
As we can see, there is no logic handling the BuiltInWindowExpr case
To Reproduce
Here's a test case that demonstrates the issue:
Expected behavior
We should be able to serialize window functions to protobuf.
Additional context
No response
The text was updated successfully, but these errors were encountered: