Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: serialize user-defined window functions to proto #13421

Merged
merged 11 commits into from
Nov 15, 2024

Conversation

jcsherin
Copy link
Contributor

@jcsherin jcsherin commented Nov 14, 2024

Which issue does this PR close?

Closes #13401.

Rationale for this change

Same as #13401.

What changes are included in this PR?

  1. Serializes user-defined window functions to proto
  2. Adds codec methods to PhysicalExtensionCodec:
pub trait PhysicalExtensionCodec: Debug + Send + Sync {

    fn try_decode_udwf(&self, name: &str, _buf: &[u8]) -> Result<Arc<WindowUDF>> {
        not_impl_err!("PhysicalExtensionCodec is not provided for window function {name}")
    }

    fn try_encode_udwf(&self, _node: &WindowUDF, _buf: &mut Vec<u8>) -> Result<()> {
        Ok(())
    }
}

Are these changes tested?

Yes, roundtrip physical plan for,

  1. An existing user-defined window function and,
  2. A user-defined window function with a custom payload and implements PhysicalExtensionCodec.

Are there any user-facing changes?

No.

@github-actions github-actions bot added physical-expr Physical Expressions proto Related to proto crate labels Nov 14, 2024
Comment on lines +2123 to +2129
fn try_decode_udwf(&self, name: &str, _buf: &[u8]) -> Result<Arc<WindowUDF>> {
not_impl_err!("PhysicalExtensionCodec is not provided for window function {name}")
}

fn try_encode_udwf(&self, _node: &WindowUDF, _buf: &mut Vec<u8>) -> Result<()> {
Ok(())
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested by roundtrip_physical_plan::roundtrip_udwf_extension_codec.

WindowFrameBound::CurrentRow,
);

let nth_value_window =
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the test referred in #13201 (comment)

@mwylde
Copy link

mwylde commented Nov 14, 2024

Thanks! I can confirm this fixes our tests.

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @jcsherin (and @mwylde for reporting and test). This PR looks great to me

@@ -120,6 +120,25 @@ pub fn serialize_physical_window_expr(
window_frame,
codec,
)?
} else if let Some(built_in_window_expr) = expr.downcast_ref::<BuiltInWindowExpr>() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The naming of these structures is wild (the physical exprs aren't actually BuiltInWindowExprs) but that is an issue to fix for another day

@jcsherin
Copy link
Contributor Author

I just merged up from main to fix conflict.

@alamb alamb merged commit d840e98 into apache:main Nov 15, 2024
25 checks passed
@alamb
Copy link
Contributor

alamb commented Nov 15, 2024

Thanks again @jcsherin

@jcsherin jcsherin deleted the fix-proto-serialize-udwf branch November 15, 2024 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
physical-expr Physical Expressions proto Related to proto crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error when serializing physical window functions to proto
3 participants