How to pass Polars DataFrame as an argument to a function?

[chetan-exy2]

Hi, I'm trying to learn pyo3 and currently attempting to port some functionality of Polars to Python. I have the following code that has a struct and the corresponding impl for it. In one of the functions, I want to pass the DataFrame from the polars library https://docs.rs/polars/latest/polars/frame/struct.DataFrame.html# to one of the functions after which I do a simple calculation over the rows.

#[pyclass]
pub struct MyCustomStruct {
    pub my_columns : Vec<String>
}

#[pymethods]
impl MyCustomStruct{ 

    #[new]
    pub fn new() ->  PyResult<Self>{
        let my_struct_object = MyCustomStruct {
                my_columns : Vec::<String>::new(),
                };
        Ok(my_struct_object)
    }

    pub fn find_some_points(&mut self,  dataframe: DataFrame) -> () {
        let mut scores : Vec<f64> = Vec::new();
        for i in 0..dataframe.height(){
            let vector_of_row = vec![sv[0].get(i), sv[1].get(i), sv[2].get(i)];
            let number_of_non_nan_values = vector_of_row.iter().filter(|&n| (*n).gt(&polars::prelude::AnyValue::Float64(0.0))).count();
            let score = (number_of_non_nan_values as f64) / (vector_of_row.len() as f64);
            scores.push(score);
        } 
    }
    }

What I am struggling to understand is how should I wrap the DataFrame object? If I try to run maturin develop on the above code, I obtain an error PyClass is not implemented for DataFrame , which makes sense since it is not a native rust type. However having looked at the type table on https://pyo3.rs/v0.16.4/conversions/tables.html , I realise the answer lies somewhere with passing the object as a PyAny object (Or i could be completely wrong about that). Any help would be appreciated.

[davidhewitt]

Hmm, so what you're asking for is actually very hard - it's tracked in #1444.

Basically, your pyo3 code has no knowledge of the polars Python wrappers.

I think the only way I'm aware for this to happen is for the polars team to provide a stable C API which exposes the polars Python functionality, as well as manual implementations in the polars create which can use these APIs to implement PyClass for DataFrame. You would probably have to link against a dynamic library which exposed this C API too.

At the moment PyO3 has no primitives to help make this happen, and nothing has been designed. I'd be happy to see something added to PyO3 to meet this need, but it's a very very hard problem!