RFC: Should we remove pyarrow feature from datafusion core

[timsaucer]

Is your feature request related to a problem or challenge?

Currently the only place Python interface code exists in the core DataFusion repo is in datafusion/common/src/pyarrow.rs. This contains methods for converting to/from ScalarValue and pyarrow objects. Since datafusion-python repository was split out from the core repo, it may make sense to remove this module. Rust users who want to use these conversions can instead use the datafusion-python, which arguably has far more support for datafusion-python interactions.

Describe the solution you'd like

Remove the pyarrow module and associated features. This would be a breaking change, but likely has extremely little impact on downstream users.

Describe alternatives you've considered

Leave as is, but this prevents things like updating pyo3 downstream. It must stay pinned at the same version as datafusion core.

Additional context

No response

[andygrove]

This seems logical to me

[Omega359]

agreed, remove.