Migrating from bazelruby/rules_ruby for gems with native extensions

[JasonLunn]

Follow up to #180

How do gem authors use bazel-contrib/rules_ruby to reference parts of the Ruby environment needed to construct native extensions?

For example, if building a native extension in C for MRI, under bazelruby/rules_ruby I might include "@rules_ruby//ruby/runtime:headers" in the deps of the cc_library to expose ruby.h. Similarly, for an native extension targeting JRuby, I might include "@rules_ruby//ruby/runtime:jars" in the deps of the java_library.

[p0deje]

I haven't seen an example of gems building native extensions yet, so you might be the first to encounter this. It's worth asking in the #ruby channel on Bazel Slack though.

If you have an example you could share, I can help figure out what needs to be changed in the ruleset to support your case.

[JasonLunn]

This is a good example of a Ruby C extension that needs to leverage definitions from ruby.h, and this is for the JRuby version of the same gem that needs jruby.jar.

[p0deje]

I will take a look at it, but please share if you have a simpler example I can experiment with (protobuf is quite large). It might be something w/o the Bazel bits, I just need to make sure I understand how to build platform-specific versions.

[JasonLunn]

bundle gem foo --ext=c will generate a trivial gem in the folder foo with a native extension implemented in C nested inside the ext subdirectory.

[p0deje]

@JasonLunn I've started experiment with native extensions in https://github.com/bazel-contrib/rules_ruby/tree/ext and so far I had no issues compiling the gem using rake compile or packaging it in a gem and then use gem install to install it. Can you give me an example of a command I can run to reproduce the issue with missing headers when compiling extensions?

[JasonLunn]

Are are you invoking rake install manually, or something like USE_BAZEL_VERSION=7.1.2 bazel build :install?

1. Running rake install doesn't produce errors, but it seems to pollute the workspace such that a subsequent call to bazel will error out because the copy of BUILD in tmp/arm64-darwin24/stage makes bazel treat it as a subpackage.
2. Running USE_BAZEL_VERSION=7.1.2 bazel build //:install doesn't produce errors, but it doesn't complete installation into the system gems and inspecting bazel-bin I don't think see any evidence of the .c, .h, or .bundle file that I see inside the native_ext-0.1.0 directory that the manual rake install produced.

Independent of addressing the nits above, examples/native_ext/BUILD only references the C source files from the data of an rb_library rule - there's no cc_library rule involved. That would prevent passing copts, alwayslink local_defines and other attributes we depend on today. It may be possible to work around that by relocating this configuration to extconf.rb, but at that point we'd be ceding control of the CC toolchain, which is a step backwards in terms of keeping the build hermetic and configurable with standard bazel flags.

[p0deje]

Are are you invoking rake install manually, or something like USE_BAZEL_VERSION=7.1.2 bazel build :install?

I use bazel build command. The branch is WIP but at some point it was essentially a run_binary() macro.

Running USE_BAZEL_VERSION=7.1.2 bazel build //:install doesn't produce errors, but it doesn't complete installation into the system gems and inspecting bazel-bin I don't think see any evidence of the .c, .h, or .bundle file that I see inside the native_ext-0.1.0 directory that the manual rake install produced.

Hmm, I will look into it. I assumed that gem install would compile native extensions, but I guess it's not happening.

Independent of addressing the nits above, examples/native_ext/BUILD only references the C source files from the data of an rb_library rule - there's no cc_library rule involved. That would prevent passing copts, alwayslink local_defines and other attributes we depend on today. It may be possible to work around that by relocating this configuration to extconf.rb, but at that point we'd be ceding control of the CC toolchain, which is a step backwards in terms of keeping the build hermetic and configurable with standard bazel flags.

So what's the workflow with Bazel then?

1. Do you compile the C code and then somehow package it into a gem?
2. Do you use the compiled code as an input to some other package?

[JasonLunn]

Our CI and release systems are bazel-centric to provide a single point of entry for all the programming languages that Protocol Buffers supports, even though we also support end users that build using language-specific tools like rake as well. This includes being able to pass flags to bazel to change what's built, in cases where we're doing cross compilation (where we'll use standard bazel flags) or where we're testing our alternative FFI-based implementation (where we'll use bazel flags like this one).

[p0deje]

That makes sense, I'm merely trying to distill the complex protobuf library into a small Ruby gem with a set of Bazel targets I need the ruleset to support. My initial thought was that creating a gem with a native extension would fail, but it seems to work fine so far. So now I am wondering what targets I need to create to have them failing. I can add cc_library to compile C-code directly, but I wonder if it's enough to fix this issue.

[JasonLunn]

I think that would fix the issue for C-based rubies. A cc_library target that includes the ruby.h header should do it under MRI. An ideal solution would provide a solve for JRuby as well, so a java_library that needs class definitions from jruby.jar.