Screen objects rendering performance

[levs42]

Describe the bug

1.9.1 has significant CPU usage increase comparing to 1.8.1. Here's some comparison:

1.8.1:
- Multicam disabled
	- Idle: 15%
	- Streaming: 21%
- Multicam enabled
	- Idle: 53% (main: 5%)
	- Streaming: 56%
1.9.1:
- Multicam disabled
	- Idle: 65% (main: 31%)
	- Streaming: 68%
- Multicam enabled
	- Idle: 80% (main: 31%)
	- Streaming: 83%

There is a gain on the main thread:

Spoiler

1.8.1:

1.9.1:

Most of the gain comes from vImageAlphaBlend_ARGB8888 and createCGImage. I'm thinking about some optimizations that could be done and looking for your opinion:

• To move vImageAlphaBlend_ARGB8888 from the main thread
• To try using CIFilter for screens' rendering instead of vImage to avoid using CPU’s vector processor
• Use existing image buffer from the camera like 1.8.1 to void calling createCGImage
• Something that is related specifically to multicam - didn't identify yet

Any feedback is appreciated.

To Reproduce

Profile CPU usage of 1.8.1 and 1.9.1

Expected behavior

• To have less usage of main thread
• To distribute work across the cores

Version

1.8.1, 1.9.1

Smartphone info.

iPhone 15 Pro Max

Additional context

No response

Screenshots

No response

Relevant log output

No response

[shogo4405]

We've discussed this in another thread, but if you're using Xcode 15.4, I'd like you to measure with the Thread Performance Checker turned OFF. If you're using a different version of Xcode, please provide the environment information.

My test case is iOS17.6.1 + iPhone 15 Pro Max + Multicam enabled, the CPU usage is around 45-55%.

Expected behavior

• To have less usage of main thread
• To distribute work across the cores

As for the expected behavior, in version 2.0.0, the transition from ＠MainActor to a custom @ScreenActor has been made. It is expected to be resolved in version 2.0.0.

[levs42]

Thank you for the advice! I tried the example app with the Thread Performance Checker disabled, and the CPU usage for 1.9.1 + multicam + idle dropped from 80% to 50%. I will start migrating to version 2.0.0 to obtain the actor fix. Regarding multicam and screen rendering performance, it still seems like vImage is taking a considerable CPU toll. I haven't checked the performance with 2.0.0 yet, but version 1.8.1 uses vImage to render multicam on a non-main thread, and this still increases CPU usage from 15% to 50%. As more users will have devices that support multicam, a 50% CPU usage on the fastest iPhone appears quite high. Reducing the usage back to 15-20% by moving the rendering to the GPU would be a significant win. I can try to implement something like ScreenRendererByGPU using Core Image. What do you think?

[shogo4405]

Currently, the iPhone 15 Pro Max (2023) has a 6-core CPU, allowing for a maximum CPU usage of 600%.
I believe it's not an issue since it's using about 50% (with around 10-15% of that being for rendering to the view).

Additionally, on the iPhone XR (2018), which is the minimum requirement for iOS 17, it was around 65-70%. As long as the main thread isn't locked, I think it’s practical to use.

In the future, I’d like to try rendering using Metal. However, my understanding is that the CPU usage isn't so much due to vImage but rather the bottleneck is the YUV to RGB conversion of the camera feed.

Simply switching to Metal-based or other alternative methods won’t necessarily reduce CPU usage or stabilize FPS and other performance aspects; I don't know until it’s implemented.

[levs42]

I'll try to use 2.0.0 + CISourceOverCompositing to compare the performance. However, looks like Metal is the best choice because Core Image might be unpredictable in utilizing GPU over CPU.

It looks like the YUV to RGB conversion is possible using a shader.

Going to close this for now as I don't know when I'll be able to try to add Metal implementation.

[mkrn]

Sorry for adding to a closed thread, but also dealing with this now,
Is there any way to improve performance for this? Profiling:


Total CPU is not huge, but it seems to all happen on the main thread
It seems to slow down the preview significantly, especially when using Image stabilization:

rtmpStream.videoMixerSettings.mode = .offscreen

I'm using a single ImageScreenObject()
Version is 1.9.4
iPhone 13 Pro
CPU is about 50%

[shogo4405]

@mkrn Regarding the lag in the preview with stabilization, are you referring to the following preferredVideoStabilizationMode?

preferredVideoStabilizationMode = .cinematic

[mkrn]

It looks like a staggering amount of time is spent on Alpha blending, I removed the alpha channel from colors in my image but it didn't help...
Perhaps the algorithm can be improved to reuse buffer or image in makeImage ? @shogo4405

[mkrn]

I use

videoCapture.preferredVideoStabilizationMode = .standard  

However, when switching to .off it seems to help with perceived performance

[shogo4405]

As for the issue, is the problem the lag that occurs whenpreferredVideoStabilizationMode = .standardis set? Or is the high CPU usage itself the main issue?

Thanks to mkrn's advice, I was able to reduce the CPU usage to some extent. Thank you.

• I optimized the process and managed to reduce CPU usage. Optimize ScreenObject CPU usage. #1576
• I reviewed the rendering of MTHKView. It seems the CPU usage on the main thread has decreased. Optimize MTHKView.swift #1578

[mkrn]

This is good. I already tested first, and it improved a lot. Waiting for 1.9.5 for testing the second

[floriangbh]

We launched production a week ago and we noticed some lag issues in rendering (whether for the preview, local recording or streaming). This appears when ScreenObject is rendered. That's was not the case before with the old way with the old registerVideoEffect and older version.

The issue appears only when the recording or stream is started, not when both are off and only the preview is displayed.

@mkrn did you test the with the new version ?

[floriangbh]

After some tests, the problem seems to have come from stabilization.
By default it's off, put preferredVideoStabilizationMode = .standard seems to be almost OK. Let's try!
I accidentally removed it while migrating to 1.9.X. Maybe .standard should be the default value?