How to dip your toe in the Airstream?

[evbo]

I like to slowly adopt new ideas and I have a pet project that might be a great example for documenting "real world" use cases of Airstream for frontend libraries beyond laminar.

I'm using the infamous "papaparse" js library for parsing CSV. It has a normal parse mode that reads the entire csv into memory, which can handily return a Scala Future using a Promise that completes when the callback is made after papa parse finishes reading the csv document.

Then, there's a fancy Stream option, that let's you step through the rows of csv one by one. For this, I wasn't sure if Scala Future had a built in mechanism for handling a Stream (do I just keep creating a new promise after each iteration?). So I thought to ask here, how could this simple example leverage Airstream to provide a clean way of channeling each row that gets emitted by papa to some other (React) component that is, e.g., building a table of results?

Also, the Airstream docs are great but I am definitely hopeful someone can recommend a short and sweet approach that doesn't demand too much understanding (yet). In time I will get the concepts down, but for now I'd appreciate quick and dirty solutions to see Airstream in action with how it can be applied to common scenarios like this.

Here's an example papa parse facade and helper function for calling it to parse data as a Future:

@js.native
@JSImport("papaparse", JSImport.Namespace)
object PapaParse extends js.Object {
  def parse(url: File, c: PapaConfig): Unit = js.native
}

object Papa {

  def stream(file: File, header: Boolean, dynamicTyping: Boolean, transform: (String, String) => String): Future[js.Array[js.Object]] = {
    val p = Promise[js.Array[js.Object]]()
    PapaParse.parse(
      file,
      PapaConfig(
        header = true,
        dynamicTyping = false,
        transform = transform,
        step = (results: js.Object, parser: js.Object) => {
          // fancy stream data
          println(s"result is: ${JSON.stringify(results)}")
          // complete a promise for each row of results?...
        },
        error = (err: js.Object) => {
          println(s"error: ${JSON.stringify(err)}")
          p.failure(new Exception(JSON.stringify(err)))
        },
        complete = (result: js.Object) => {
          println(s"result is: ${JSON.stringify(result)}")
          // regular parse result we are returning a future for
          p.success(js.Array(result))
        }
      )
    )

    p.future
  }

[evbo]

Update: I did some more reading and realized for ReactJS, Airstream would require I manually render the component. So I did not at first realize that and it may be trying to mix two different architectures too much for my taste. I may need to reapproach Airstream if I get a chance to use Laminar.

[raquo]

Your "data pipeline" will have two terminations: the source and the sink.

At the source, you want to get events into Airstream. That's easy, just create a custom stream:

val (rowStream, nextRow) = EventStream.withCallback[CsvRow]

and push events to it inside PapaParse's step function:

step = (results: js.Object, parser: js.Object) => {
  nextRow(makeCsvRow(results))
}

Now that you have a rowStream, you can map it or apply any other observable transformations. Suppose you want to accumulate some state:

val resultSignal: Signal[Result] = rowStream.foldLeft(initial = Result.empty){ (acc, nextRow) => acc.updateWithRow(nextRow) }

Finally, for the sink, you want to put your transformed data into React I guess? The Laminar video has a chapter on integrating Laminar components into React, and the approach here is similar, you need to:

1. Pass resultSignal to a react component, or create it internally
2. Create a val owner = new ManualOwner inside your React component
3. When the component is mounted, you call resultSignal.foreach(newResult => doSomeReactStateUpdate(newResult))(owner), and when it's unmounted, you call owner.killSubscriptions() to clean up. Use React lifecycle methods (or hooks?) for that.

That's it, implement doSomeReactStateUpdate to put some data from resultSignal into your react component's state, and render said data from that state.

---

Now, if you don't want any rendering, if you just want to do something with rowStream / resultSignal, you just need to do a similar thing with calling foreach, providing a callback and an owner to it, and making sure to call owner.killSubscriptions when you don't need the stream to run anymore. Ownership is kinda annoying to work with manually, because it's designed to be handled automatically by a library, such as Laminar. But if you don't need any rendering you can still do it manually.

[evbo]

I didn't realize I could just call existing React hooks (doSomeReactStateUpdate in your example, or for instance, setState). I guess that was missed by me in your Rendering React Components in Laminar chapter; I took it to mean that was the only way to integrate with React. But what you're showing here is you can still render in React too, so long as Airstream relies on the Component pattern (now I am reading that correctly!).

[evbo]

and would this work the same if papaparse were executing in a separate thread? e.g. does Airstream place nice with web workers?

[raquo]

I haven't really used web workers in JS so take what I say with a grain of salt. Web workers are not like threads in JVM, you don't get shared memory, just the ability to pass messages between workers and the main "thread". If you want Airstream events to pass from the main thread into a worker, you'll need to transport them via this postPessage mechanism. That means encoding your event values into JSON/string and creating a separate stream / event bus inside the worker that will listen to incoming messages and decode them. And if you want to run an instance of Airstream itself inside a web worker, you won't have access to the DOM so you can't use Laminar in there which means you can't use owners provided by Laminar inside a web worker, you'll need to manage them yourself manually. Otherwise I think stuff should still work, it will just be clunky to use due to the above.