feat: Add column format option to iter rows

[colin-ho]

Addresses: #3634

Add an option to iter_rows to decude the format of columns during iteration, either Python (default), Arrow, or Numpy.

[codspeed-hq]

CodSpeed Performance Report

Merging #3681 will not alter performance

_{Comparing colin/iter-rows-with-col-format (85317a2) with main (03fea9c)}

Summary

✅ 27 untouched benchmarks

[codecov]

Codecov Report

Attention: Patch coverage is 89.47368% with 2 lines in your changes missing coverage. Please review.

Project coverage is 77.91%. Comparing base (c932ec9) to head (85317a2).
Report is 26 commits behind head on main.

Files with missing lines	Patch %	Lines
daft/dataframe/dataframe.py	89.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3681      +/-   ##
==========================================
- Coverage   78.06%   77.91%   -0.16%     
==========================================
  Files         728      727       -1     
  Lines       89967    91219    +1252     
==========================================
+ Hits        70236    71074     +838     
- Misses      19731    20145     +414     

Files with missing lines	Coverage Δ
daft/dataframe/dataframe.py	85.39% <89.47%> (-0.13%)	⬇️

... and 125 files with indirect coverage changes

[colin-ho]

Offline: allow python and arrow column format since numpy is only for numeric. If user wants numpy they should be able to convert themselves

[colin-ho]

Example from the linked issue, which is much faster now with iter_rows(format="arrow"). The nested scalar can be converted to numpy via to_numpy()

import numpy as np
import daft

np.random.seed(0)
n_rows = 1_000
list_size = 100_000
data = {"list": np.random.randint(0, 256, (n_rows, list_size), dtype=np.uint8)}
df = daft.from_pydict(data)

for row in df.iter_rows(column_format="arrow"):
    v = row["list"]
    print(v.values.to_numpy())

[jaychia]

Feel free to re-add me as reviewer once we work in the offline discussions yesterday around using arrow!

[colin-ho]

Feel free to re-add me as reviewer once we work in the offline discussions yesterday around using arrow!

Ready for another pass, thanks!

[jaychia]

Any quick numbers for speedup to expect here?

[jaychia]

Maybe add some comments here also about how we determine the appropriate Python types?

I think for example tensor type gets converted to numpy arrays. Not sure if there is special handling for other logical types.

[colin-ho]

Sounds good!

[colin-ho]

import numpy as np
import daft
import time

np.random.seed(0)
n_rows = 1_000
list_size = 100_000
data = {"list": np.random.randint(0, 256, (n_rows, list_size), dtype=np.uint8)}
df = daft.from_pydict(data)

start_time = time.time()
for row in df.iter_rows(column_format="arrow"):
    v = row["list"]
    np_values = v.values.to_numpy()

print(f"Arrow to numpy: {time.time() - start_time} seconds")

start_time = time.time()
for row in df.iter_rows(column_format="python"):
    pass

print(f"Python: {time.time() - start_time} seconds")

Arrow to numpy: 0.0013179779052734375 seconds
Python: 12.269848108291626 seconds