add video support to warp-blink-wrtc

[sdwoodbury]

tentative task list as of Dec 2023

• get video test project merged into warp.
• extend the MP4 Logger, allowing it to save AV1 encoded video.
• extend Blink, allowing it to send video streams as well.
• possibly add timing information to blink audio and video packets (or extract it from the RTP packets) to aid in synchronizing audio and video tracks.
• take prototype code and extend host_media to capture video.
• find way to render video frames in Dioxus. Perhaps using wgpu or other OpenGL like library. Must support multiple video streams at once.

---

Some work was done in the branch feat/blink-video. OpenCV makes it easy to capture video frames. The next step is to test passing them through a codec. After that, sending the data over WebRTC should be trivial.

Getting the video to display in a Dioxus app may be a different story. Jon is hopeful that an RTP stream can be sent to the webview.

[sdwoodbury]

I investigated using the H.264 codec to encode frames captured by OpenCV (which happen to be in BGR format). Implementations available were x264 and openh264. The former worked excellent but is GPL licensed. The latter only supports inputs in YUV420 format, which severely degrades image quality.

The Alliance for Open Software has released a free AV1 codec which looks promising. I will try that next.

[sdwoodbury]

got the AOM version of the AV1 codec working in the branch feat/blink-video2. Image quality is acceptable. Will work on adding it to blink next.

[sdwoodbury]

This probably needs to go on hold until @jkelleyrtp figures out how we will be sending video to Dioxus. It may be possible to send the WebRTC stream directly to Dioxus/the webview. If that doesn't work, I believe that Blink would need to provide an HTTP endpoint (for an RTP server) for the video elements.

[sdwoodbury]

I was looking into adding opencv to the various installers and learned that when statically linking, it can take 0.5GB. It's probably worth investigating using nohkwa for camera capture instead of OpenCV.

[sdwoodbury]

nokhwa didn't work but eye-rs works well.