Pick and plan our first collaborative R/Rust project

[jdhoffa]

I am moving the discussion from Slack over to here, as this is close to where the code will end up living!

We've discussed the idea of collaborating on a project to develop a tool, crate or CLI in Rust. Ideally, we want a tool that would simplify our lives or common tasks for software developers. Let's use this thread to talk through all the ideas, and align on something cool

[jdhoffa]

@jibarozzo mentioned the idea of recreating an R package in Rust, with the goal of performance improvement.

The organization https://github.com/extendr provides extension libraries for R in Rust, we might consider either looking to them for design/ architecture inspiration, or even contributing to their projects.

Their package https://github.com/extendr/rextendr/ helps facilitate calling Rust from R code.

[jdhoffa]

He also mentioned an interesting article by @jonocarroll for wrapping C code:
https://jcarroll.com.au/2023/08/11/wrapping-c-code-in-an-r-package/

Slightly different use-case, but a lot of the same principles apply

[jdhoffa]

In general, I favour this idea as it leverage shared expertise, has clear scope, potential real-world utility, and pre-defined design which would enable immediate progress

[jdhoffa]

Other potential inspiration along this thread: https://github.com/pola-rs/r-polars

[jonocarroll]

Josiah Parry has some great examples of using {rextendr} e.g. https://josiahparry.com/posts/2023-04-13-counting-chars/

[jdhoffa]

@asbates mentioned that Posit’s new IDE, Positron, has some parts written in Rust.
The bulk of that is in the Ark R kernel

we also might considering contributing to that

[jdhoffa]

For visibility, I am using the following GH project to track our goals here:
https://github.com/orgs/r-staceans/projects/2

[jibarozzo]

@jdhoffa Thanks for bringing the discussion here and creating the project.

[jdhoffa]

Happy to!! Excited to even 😊

[maurolepore]

Although not in this book club, @andyquinterom might be a great person to ask about projects in the intersection between R and Rust. He's super knowledgable, kind and active -- and has experience with rextendr

[jibarozzo]

Here's a template for R-Rust packages. https://github.com/r-rust/hellorust?tab=readme-ov-file. The slides provide some details about how R uses C headers to interact with Rust code. Something I hadn't realized.

[jibarozzo]

Hey,
I've thought about this and can't really come up with an immediate project. A lot of the packages I work with or thought of are well maintained. The only package that comes to mind is the mirlyn: A package for library size normalization using repeated iterations of rarefaction for microbial analyses. It's not a big package, maybe a dozen functions all in R. It can get slow depending on the user parameters and dataset (see main functions mirl.R and mirlyn.R). It doesn't have many dependencies, and optimizes by using the parallel package. What do y'all think @jdhoffa @asbates @jonocarroll ?

[asbates]

I'm good with that!

I don't know anything about microbial analysis so will need to lean on you. And to be honest I don't know how much time I'll actually have to work on it.

But if you think it's a good project to try then let's do it.

[jibarozzo]

Cool. I can bring folks up to speed on what it is supposed to do.

[jdhoffa]

Totally happy with that!
Also happy to learn the subject matter from you, and just focus on implementation :-)

[jdhoffa]

Another idea could be a R-Rust interface that exposes multi-threading to large R processes?
Specifically what I have in mind is:

• Build out an interface and API exposed to the user through an R package (I think it would be most cool/ interesting to focus on batching memory intensive data-frame/ tibble processes)
• Use extendr crate to interface with Rust
• Choose a threading model in Rust (I guess probably std::thread would be the simplest to start with, or maybe rayon for data parallelism)
• R isn't thread safe, so we wouldn't want to actually run any R functions through Rust...

We could start with trying to do this for one or two dplyr or tidyverse functions (if possible?)

[jibarozzo]

Damn. Maybe omniverse, rustyverse, hyperverse, blazar, quasar...Trying to keep the them of space qnd speed. Blazar is non existent. There are some similar to quasar. In the 'plyr' theme: rusplyr, fplyr or something like that.

[jdhoffa]

FYI r-staceans/blazr#4

[DrEntropy]

I like this project idea as well. One of the big selling points of rust is 'safe' multithreading, so something that exercises this would really help solidify ones understanding. We could look at data.table for inspiration as well, which uses C but does do low-level parallelism for data operation.

[jdhoffa]

@r-staceans/dlsc PSA our intention is to continue work on this package under the name:
Threadr

(For now)

Please 👍 this comment if you are in favour of this change OR

Comment an alternative suggestion if you are strongly opposed.

Full discussion at r-staceans/blazr#4

[jdhoffa]

⚠️ on Sunday COB, I will change the name of the repo, meaning anyone with a local clone will need to delete their local copy and clone the repo with its new name

[jonocarroll]

Not to detract from the started project (which sounds great) but in case people aren't aware of it - {collapse} is a C/C++ analogue of many {dplyr} functions

https://sebkrantz.github.io/collapse/articles/collapse_intro.html#fast-data-manipulation

I don't think it goes as far as parallel, but there's perhaps things to learn from the choices made there.

To start simple (and to keep overlap with the started project) we could implement some common manipulations e.g. mutate, select, filter in Rust and see how the performance compares to regular {dplyr} and {collapse}, then see what parallelising those looks like.

[jdhoffa]

Super interesting, thanks for sharing!

I think the project's key outcome is just our own learning/ pedagogy (and selfishly, I want to dig into Rust's multi-threading functionality 😈 )
I am generally skeptical that we will make significant performance gains against existing packages, but who knows! if we do that's the cherry on top haha