Multi encoder #100

cdiddy77 · 2024-02-12T16:43:31Z

here is my first cut at the multi-encoder, along with a dual streamer which uses it to create both RTMP and file streams. Much of this code is bespoke for my application and will need subsequent modification.

Some of the code is generally useful and/or addresses bugs I found along the way.

Jitpack is back after the fork-sync

some tweaks to project files to allow integration

cdiddy77 · 2024-02-12T16:43:48Z

.gitignore

 .DS_Store
+.vscode


whoops, will fix this

cdiddy77 · 2024-02-12T16:44:11Z

core/src/main/java/io/github/thibaultbee/streampack/internal/data/Frame.kt

+
+    fun clone(): Frame {
+        val newRawBuffer = rawBuffer.duplicate() // Creates a new buffer that shares the content of 'rawBuffer'.
+        // Assuming the MediaFormat object can be shared. If not, you need to deep copy it as well.


comments courtesy of chatGPT. Will remove

cdiddy77 · 2024-02-12T16:45:36Z

core/src/main/java/io/github/thibaultbee/streampack/internal/encoders/MediaCodecEncoder.kt

@@ -150,6 +151,11 @@ abstract class MediaCodecEncoder<T : Config>(
            Handler(handlerThread.looper)
        }
    }
+    private fun releaseHandler() {


currently I believe that the MCE leaks a thread every time the user hits start/stop? This code is not probably optimal. Maybe better to reuse the same handler on subsequent starts

cdiddy77 · 2024-02-12T16:47:06Z

core/src/main/java/io/github/thibaultbee/streampack/internal/encoders/MediaCodecEncoder.kt

+                if(isStopped) {
+                    return
+                }
                isStopped = true
-                mediaCodec?.setCallback(null)
-                mediaCodec?.signalEndOfInputStream()
+                if(hasInputSurface) {
+                    mediaCodec?.signalEndOfInputStream()
+                }
                mediaCodec?.flush()
                mediaCodec?.stop()
+                mediaCodec?.setCallback(null)
+                releaseHandler() // prevent thread leak


per offline discussion, this code remedies a bug where stopStream always results in mediaCodec throwing an exception, in both setCallback (must be stopped before call) or in signalEndOfStream (not allowed for non-surface encoder)

cdiddy77 · 2024-02-12T16:53:52Z

.../main/java/io/github/thibaultbee/streampack/internal/encoders/MultiVideoEncoderTargetInfo.kt

+import io.github.thibaultbee.streampack.streamers.interfaces.settings.IVideoSettings
+import java.util.concurrent.ExecutorService
+
+data class MultiVideoEncoderTargetInfo(


obviously this code is pretty similar to alot of VideoMediaCodecEncoder.CodecSurface. Perhaps someday the two might be one.

cdiddy77 · 2024-02-12T16:55:40Z

jitpack.yml

@@ -0,0 +1,2 @@
+jdk:


yes I will remove this again

cdiddy77 · 2024-02-12T16:58:48Z

.../src/main/java/io/github/thibaultbee/streampack/ext/rtmp/streamers/CameraDualLiveStreamer.kt

+    private val rtmpMuxer: IMuxer = FlvMuxer(writeToFile = false)
+    private val fileMuxer: IMuxer = MP4Muxer()
+
+    private var rtmpAudioStreamId: Int? = null
+    private var rtmpVideoStreamId: Int? = null
+    private var fileAudioStreamId: Int? = null
+    private var fileVideoStreamId: Int? = null
+
+    // Keep video configuration separate for rtmp and file
+    private var rtmpVideoConfig: VideoConfig? = null
+    private var fileVideoConfig: VideoConfig? = null


All these things, instead of being two of everything, should get grouped into a thing that there are two of.

If done right, would lead eventually to a CameraMultiStreamer that could use any N streaming destinations.

cdiddy77 · 2024-02-12T17:00:59Z

.../src/main/java/io/github/thibaultbee/streampack/ext/rtmp/streamers/CameraDualLiveStreamer.kt

+    private val audioEncoderListener = object : IEncoderListener {
+        override fun onInputFrame(buffer: ByteBuffer): Frame {
+            return audioSource.getFrame(buffer)
+        }
+
+        override fun onOutputFrame(frame: Frame) {
+            val fileFrame = frame.clone()
+            rtmpAudioStreamId?.let {
+                try {
+                    [email protected](frame, it)
+                } catch (e: Exception) {
+                    throw StreamPackError(e)
+                }
+            }
+            fileAudioStreamId?.let {
+                try{
+                    [email protected](fileFrame,it)
+                } catch(e:Exception){
+                    throw StreamPackError(e)
+                }
+            }
+        }
+    }


per offline discussion, this construction implies that the audio encoding (and thus audio settings) are shared between the different output destinations. This is something that I can live with for my needs, since it allows for a much simpler implementation.

cdiddy77 · 2024-02-12T17:02:08Z

core/src/main/java/io/github/thibaultbee/streampack/internal/gl/FullFrameRect.kt

@@ -91,8 +91,10 @@ class FullFrameRect(var program: Texture2DProgram) {
     *
     * The appropriate EGL context must be current.
     */
-    fun changeProgram(program: Texture2DProgram) {
-        this.program.release()
+    fun changeProgram(program: Texture2DProgram, releasePrevious: Boolean = true) {


in my implementation, each encoding path has its own fullframerect (since the resolutions might differ) but they all share the same program. So I dont want them to nuke the program out from under eachother

cdiddy77 · 2024-02-12T17:09:23Z

core/src/main/java/io/github/thibaultbee/streampack/internal/gl/EglWindowSurface.kt

-class EglWindowSurface(private val surface: Surface, useHighBitDepth: Boolean = false) {
-    private var eglDisplay: EGLDisplay = EGL14.EGL_NO_DISPLAY
-    private var eglContext: EGLContext = EGL14.EGL_NO_CONTEXT
+class EglWindowSurface(


I broke EglWindowSurface and EgDisplayContext into two different classes bc I need to create the texture outside the scope of any particular surface context.

cdiddy77 · 2024-02-12T17:09:53Z

core/src/main/java/io/github/thibaultbee/streampack/internal/gl/EglDisplayContext.kt

I broke EglWindowSurface and EgDisplayContext into two different classes bc I need to create the texture outside the scope of any particular surface context.

cdiddy77 · 2024-02-12T17:19:55Z

demos/camera/src/main/java/io/github/thibaultbee/streampack/app/utils/StreamerFactory.kt

@@ -25,6 +25,8 @@ import io.github.thibaultbee.streampack.data.AudioConfig
 import io.github.thibaultbee.streampack.data.BitrateRegulatorConfig
 import io.github.thibaultbee.streampack.data.VideoConfig
 import io.github.thibaultbee.streampack.ext.rtmp.streamers.AudioOnlyRtmpLiveStreamer
+//import io.github.thibaultbee.streampack.ext.rtmp.streamers.CameraRtmpLiveStreamer
+import io.github.thibaultbee.streampack.ext.rtmp.streamers.CameraDualLiveStreamer


hmm weird probably this would break the demo app. should remove

ThibaultBee · 2024-11-19T13:15:58Z

As lot of code has changed for the v3, this won't be applicable anymore.

ThibaultBee · 2025-01-21T13:17:04Z

Multiple output is a feature of the version after the v3 (probably v3.1). Work in progress there: https://github.com/ThibaultBee/StreamPack/tree/dev_v3_1 with the DualStreamer.

Charles Parker and others added 5 commits February 9, 2024 08:50

Jitpack is back after the fork-sync

254437e

Merge pull request #2 from sidelineHD/jitpack-back

3d91b97

Jitpack is back after the fork-sync

some tweaks to project files to allow integration

43a4178

Merge pull request #3 from sidelineHD/tweak-to-integrate

598328d

some tweaks to project files to allow integration

Multi video encoder is born

97db807

cdiddy77 commented Feb 12, 2024

View reviewed changes

.gitignore

.DS_Store

.vscode

Copy link

Author

cdiddy77 Feb 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whoops, will fix this

cdiddy77 commented Feb 12, 2024

View reviewed changes

jitpack.yml

@@ -0,0 +1,2 @@

jdk:

Copy link

Author

cdiddy77 Feb 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes I will remove this again

cdiddy77 commented Feb 12, 2024

View reviewed changes

ThibaultBee force-pushed the main branch 3 times, most recently from c86b764 to 994b995 Compare February 28, 2024 17:08

ThibaultBee force-pushed the main branch from b1f6cd7 to cdc1876 Compare March 6, 2024 10:19

ThibaultBee force-pushed the main branch from 91ffc57 to adf88e5 Compare March 27, 2024 16:26

ThibaultBee force-pushed the main branch from e9c792d to df0e210 Compare April 18, 2024 14:50

ThibaultBee force-pushed the main branch from 058377c to ac46c22 Compare May 15, 2024 14:56

ThibaultBee force-pushed the main branch 2 times, most recently from 6a5e4ad to 7e712ac Compare August 2, 2024 20:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi encoder #100

Multi encoder #100

cdiddy77 commented Feb 12, 2024

cdiddy77 Feb 12, 2024

cdiddy77 Feb 12, 2024

cdiddy77 Feb 12, 2024

cdiddy77 Feb 12, 2024

cdiddy77 Feb 12, 2024

cdiddy77 Feb 12, 2024

cdiddy77 Feb 12, 2024

cdiddy77 Feb 12, 2024

cdiddy77 Feb 12, 2024

cdiddy77 Feb 12, 2024

cdiddy77 Feb 12, 2024

cdiddy77 Feb 12, 2024

ThibaultBee commented Nov 19, 2024

ThibaultBee commented Jan 21, 2025

Multi encoder #100

Are you sure you want to change the base?

Multi encoder #100

Conversation

cdiddy77 commented Feb 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThibaultBee commented Nov 19, 2024

ThibaultBee commented Jan 21, 2025