Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Remove cudf Cython bindings (cudf._lib) in favor of pylibcudf #17317

Open
4 tasks
mroeschke opened this issue Nov 14, 2024 · 0 comments
Open
4 tasks

[FEA] Remove cudf Cython bindings (cudf._lib) in favor of pylibcudf #17317

mroeschke opened this issue Nov 14, 2024 · 0 comments
Assignees
@mroeschke
Labels
feature request New feature or request Python Affects Python cuDF API.

Comments

@mroeschke
Copy link
Contributor

mroeschke commented Nov 14, 2024 •
edited
Loading

Is your feature request related to a problem? Please describe.
With pylibcudf largely complete in containing the necessary APIs for cudf Python (xref #15162), the cudf Cython layer (cudf._lib) should be largely unneeded now as pylibcudf provides a public interface to access libcudf.

Describe the solution you'd like
Ideally to fully remove the cudf._lib directory and stop developing Cython binding for this directory.

Each Cython file in cudf._lib should be able to be converted to a Python in cudf.core._internals (or cudf.core._plc?)

Additional context

Open Questions/Considerations

  1. cudf Cython spill locks whenever calling pylibcudf algorithms that operate on columns (e.g. nans_to_nulls), while there's currently no way to replicate spill locking with using pylibcudf alone.

  2. cudf Cython also defines Cython classes that back cudf Python columns and scalars. Ideally cudf Python should just be able to define these objects in Python, but it's not clear yet (to me) if there's some aspect that needs to be defined in Cython.

Projects using cudf Cython layer
@mroeschke mroeschke added the feature request New feature or request label Nov 14, 2024
@vyasr vyasr added the Python Affects Python cuDF API. label Nov 14, 2024
@mroeschke mroeschke self-assigned this Nov 14, 2024
This was referenced Nov 14, 2024
Merged
Merged
rapids-bot bot pushed a commit that referenced this issue Nov 15, 2024
@mroeschke
Remove cudf._lib.avro in favor of inlining pylicudf (#17319)
d67d017 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17319
This was referenced Nov 15, 2024
Merged
Merged
Merged
Merged
rapids-bot bot pushed a commit that referenced this issue Nov 18, 2024
@mroeschke
Move cudf._lib.unary to cudf.core._internals (#17318)
aeb6a30 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17318
rapids-bot bot pushed a commit that referenced this issue Nov 18, 2024
@mroeschke
Remove cudf._lib.hash in favor of inlining pylibcudf (#17345)
18b40dc 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17345
rapids-bot bot pushed a commit that referenced this issue Nov 18, 2024
@mroeschke
Remove cudf._lib.concat in favor of inlining pylibcudf (#17344)
ba21673 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17344
rapids-bot bot pushed a commit that referenced this issue Nov 18, 2024
@mroeschke
Remove cudf._lib.quantiles in favor of inlining pylibcudf (#17347)
02c35bf 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17347
rapids-bot bot pushed a commit that referenced this issue Nov 18, 2024
@mroeschke
Remove cudf._lib.labeling in favor of inlining pylibcudf (#17346)
302e625 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17346
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mroeschke mroeschke

Labels
feature request New feature or request Python Affects Python cuDF API.
Projects
cuDF Python
Status: Todo
cuDF/Dask/Numba/UCX
Status: In Progress
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vyasr @mroeschke