Proposal: add JavaScript support via OXC

[shahryarjb]

Requirement

One feature that could significantly enhance Igniter's support for all basic requirements in Phoenix is a robust JavaScript parser or manipulator.
Developing this from scratch would be a large-scale project, requiring considerable time and effort.

Therefore, I suggest considering the Oxc project, which is implemented in Rust, as a potential solution.

---

Where we can use it:

Imagine you want to create a Phoenix component that requires a phx-hook. You would need to modify the app.js file to inject the entire JavaScript file into the Hook object, or alternatively, create a separate JavaScript file, import it into app.js, and then inject it into the hook.

Currently, all of this is done manually in Elixir projects, which involves a lot of documentation and a complex process. This approach is not ideal and can discourage users from using it.

For example, take a look at this object that I injected using the JavaScript spread operator.

hooks: { ...Hooks, CopyMixInstallationHook }

No I want to add GetUserText object.

# This code is from ChatGPT, so it is not totally valid code
let ast = parser.parse().expect("Failed to parse the JS code");
for node in ast.body {
    if let Some(object_literal) = node.as_object_literal() {
        for property in object_literal.properties {
            if property.key == "hooks" {
                property.value.push("GetUserText");
            }
        }
    }
}

Thank you in advance

Refs

• https://discord.com/channels/711271361523351632/1253324141491261472/1297914761438302239
• https://github.com/oxc-project/oxc
• https://oxc.rs/
• https://crates.io/crates/oxc_parser

[TwistingTwists]

https://github.com/TwistingTwists/igniter_js

[shahryarjb]

https://github.com/TwistingTwists/igniter_js

I think if it comes into ash-project org and collaborate, would be good; or even on igniter as core optional installing

[zachdaniel]

Agreed :) Still in the early stages of discovery.

[jimsynz]

I'm happy to work on a Rustler integration if needed.

[zachdaniel]

There is some discussion in the igniter discord channel on the rustler side of things. Would be a wonderful to have more eyes and hands on it

[TwistingTwists]

@jimsynz : Currently I am expanding on ideas on the best way forward to integrate js parser and ast manipulator from elixir.

Here is one approach I have decided to take (unless there are unknown unknowns):
the oxc crate has oxc_parser for parsing js to ast and oxc_ast for ast manipulation.
If we can bind to oxc_ast from elixir, we can get a well maintained + production ready js ast manipulator.

I imagine writing one JS AST manipulator in elixir would be possible, but keeping it upto date would be a hassle.

---

So, the plan is to have ast in rust. Reference that AST struct from elixir. manipulate that ast in js using oxc_ast and give the bindings from elixir.

---

This is the 4 step plan (rough direction) as suggested by @zachdaniel

1. Create a custom Rewrite.Source for js that behaves like the Ex Rewrite source behaves.

2. Create a Zipper data structure for traversing and manipulating AST (this is complicated, I barely understand the Sourceror.Zipper 😆 )

3. Start the work of creating a library of functions to manipulate the zipper, which would be namespaced under something like Igniter.Js.Code.

4. figure out how we can make this available w/o requiring a dependency on rust. There are tools for doing prebuilt binaries, or we'd make it an optional thing that users have to opt into if they want to get js patches in addition to elixir patches.

[TwistingTwists]

Here is what I found about integration rust Struct and elixir

1. create a resource : basically, use Arc<Mutex>

2. expose an nif to take the reference to that resource and do something.

[TwistingTwists]

Here is what I found about integration rust Struct and elixir

1. create a resource : basically, use Arc<Mutex>
2. expose an nif to take the reference to that resource and do something.

same pattern found here as well:

1. create struct using Mutex. Hide it behind Arc
2. Bonus - you can even nest the references to struct if you do Arc<Mutex>

[TwistingTwists]

Here are more code references:

1. create a new resource and return that
2. take resource in 1 as ref do some ops on it.

[zachdaniel]

What is the general solution to #4 above? If we're going to invest a lot of time and energy into this via rustler, we need to have confidence that we can ship this to effectively any platform someone is developing on. The nice thing about pure elixir solutions is that we know they have elixir installed at a minimum 🤣

[TwistingTwists]

What is the general solution to #4 above? If we're going to invest a lot of time and energy into this via rustler, we need to have confidence that we can ship this to effectively any platform someone is developing on. The nice thing about pure elixir solutions is that we know they have elixir installed at a minimum 🤣

This seems to be the easier part.

With RustlerPrecompiled library, this is breeze.

Code change: https://github.com/TwistingTwists/crucible/blob/d686d940aa71c7213dd0ef2ec1b5af0da126988b/lib/native.ex#L6-L11

So, this is already done!! See assets which are precompiled binaries: https://github.com/TwistingTwists/crucible/actions/runs/11458278738
cc @zachdaniel

[TwistingTwists]

What is the general solution to #4 above? If we're going to invest a lot of time and energy into this via rustler, we need to have confidence that we can ship this to effectively any platform someone is developing on. The nice thing about pure elixir solutions is that we know they have elixir installed at a minimum 🤣

This seems to be the easier part.

With RustlerPrecompiled library, this is breeze.

Code change: https://github.com/TwistingTwists/crucible/blob/d686d940aa71c7213dd0ef2ec1b5af0da126988b/lib/native.ex#L6-L11

So, this is already done!! See assets which are precompiled binaries: https://github.com/TwistingTwists/crucible/actions/runs/11458278738 cc @zachdaniel

haven't added support for windows yet. or musl binaries. But this can be done. This is just the barebones apple and linux_x86 support for precompiled binaries.

[zachdaniel]

Windows support will be a hard requirement FWIW

[zachdaniel]

If we want igniter to be truly ubiquitous across the ecosystem, it has to run everywhere elixir runs.

[TwistingTwists]

There is a good news. Can compile for windows, apple and linux (x86 and arm!) woot! 🥳

Here: https://github.com/TwistingTwists/crucible/releases/tag/v0.1.1

[TwistingTwists]

I've had a long holiday in context of diwali in India. Back now.

First update:

Since AST is in Rust, and the AST manipulation is also available via rust methods, my idea is to keep all the AST and AST manipulation in Rust-land.

Here is an API to begin with.

Given a js file path
    - read the js file
    - parse the contents -> return the pointer to AST to Beam
    - write bindings to the manipulation functions so that one can do ____ in existing js file
      - import a new file
      - check if a dependency is already imported

Let's see how this goes.

[shahryarjb]

Since AST is in Rust, and the AST manipulation is also available via rust methods, my idea is to keep all the AST and AST manipulation in Rust-land.

Hi @TwistingTwists
I have a question, if we keep it inside rust how can we validate in elixir? and based on the ast inject a code inside for example app.js hook object

[TwistingTwists]

Since AST is in Rust, and the AST manipulation is also available via rust methods, my idea is to keep all the AST and AST manipulation in Rust-land.

Hi @TwistingTwists I have a question, if we keep it inside rust how can we validate in elixir? and based on the ast inject a code inside for example app.js hook object

The methods for manipulating ast will be present in Rust (oxc_semantic) already does a lot of ast manipulation.

the library will call those methods from elixir (via rustler)

Proposed API on elixir side would be (roughly):

---

For Handling Hooks in app.js

filename = app.js
Crucible.Variable.is_defined("Hooks", filename, "const")
or Crucible.Variable.is_defined("Hooks", filename, "let")

Crucible.Variable.define_if_not_already("const Hooks = {}", filename)

Crucible.Variable.add_kv_in_existing_object("Hooks", filename, key, value)
Crucible.Variable.force_add_kv_in_existing_object("Hooks", filename, key, value)