Record/stream Flutter widget?

[kenle]

Hi, is it possible to mix-in a Flutter widget into the Camera stream? For example, I would like to be able to record a Flutter widget overlayed on top of the Camera video. Any help or ideas would be greatly appreciated. Thanks.

[shogo4405]

I’m not familiar with Flutter...
I would not be able to mix-in a Flutter widget.

[kenle]

Oh, I thought this was a plug-in for the Flutter framework, I thought you became familiar with Flutter and I was excited :)

I'm not familiar with the topics of video encoding, streaming, and how I would go about combining a Flutter Widget (which in the back-end is drawn with a RenderObject (OpenGL?).

If I had some clues where to start I would be interesting in adding this feature as it would be real useful for cross-platform (iOS & Android) applications that need graphic overlay on top of video camera.

Can you recommend me some books or search key terms on the topic matter of your code? Technologies such as recording camera data to file, streaming data to RTMP, overlaying graphic data (software such as OBS studio). I've searched before but the topic matter seems wide and I don't know where to start, any guidance on the fundamental theories needed would be appreciated.

Apps such as this, https://youtu.be/znIz_FQfXmQ, seem so standard yet I have trouble Googling how to code such apps :(

Thanks @shogo4405!

[shogo4405]

First, HaishinKit(iOS, Android) is that OS native image buffers are sent via RTMP.

Recently, I made a feature to mix and send two camera images on iOS. shogo4405/HaishinKit.swift#1097
Use a technique, It it possible, one is a camera image, and the other is a captured image of a UI drawn with Flutter. If you can get the data of Flutter Widget by some method, it seems to be possible to make it. I'm not sure how to get Flutter's drawing data.

[kenle]

Thanks for your reply @shogo4405

Currently in Flutter, the only commonly documented way to get Flutter drawing data is by converting the widget to a PNG, for example
The Widget (UI)

RepaintBoundary(
            key: _globalKey,
            child: Text("Hello World"),
);

The code to get the rendered widget as a PNG

RenderRepaintBoundary boundary =
          _globalKey.currentContext.findRenderObject();
      ui.Image image = await boundary.toImage(pixelRatio: 3.0);
      ByteData byteData =
          await image.toByteData(format: ui.ImageByteFormat.png);
      var pngBytes = byteData.buffer.asUint8List();
      var bs64 = base64Encode(pngBytes);

This method seems inefficient to me, for example if I have a timer UI that has an animated (not static) counter you would have to continuously capture a PNG and overlaying that onto the camera image.

Any thoughts? I'm sure there's a more direct way to gain access to the OpenGL calls Flutter makes to draw the widgets, but I'm not at that level of knowledge yet.