[BUG] Insertion of overlayer pictures is broken/incompatible with Dolby Vision

[igorbaryshev]

Overview
I was playing around with settings to optimize quality, stumbled upon @BlueSwordM's Beginner Guide.
Tried --enable-overlays 1 option with DoVi and noticed that some frames were broken in the resulting video.
I suspect those frames have to do something with DoVi and overlay frames not jiving well together.

Branch

• master
• testing

Reproduction
Steps to reproduce the behavior:

1. Obtain or build SvtAv1EncApp with DoVi enabled.
2. Extract the contents of marvel.7z from marvel.zip
3. Run SvtAv1EncApp --input marvel.y4m --output marvel.ivf --dolby-vision-rpu marvel.rpu --enable-overlays 1
4. Check the result, you'll notice frames looking similar to what I've shown above.

Expected behavior
Properly working overlayer pictures when used with Dolby Vision RPU.
Or a warning, if it's not possible to make it compatible in the meantime.

Platform:

• Ubuntu 24.10; Windows 11 24H2
• Clang (Ubuntu):

Ubuntu clang version 19.1.1 (1ubuntu1)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-19/bin

• Clang (Windows):

clang version 19.1.6
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\llvm\bin

• Build Command (Ubuntu): ./build.sh static enable-dovi

Version String:

• Ubuntu:

SVT-AV1-PSY v2.3.0-A-11-ga3f3ca19 (release)
PSY Release: A

• Windows:

SVT-AV1-PSY f560b40-dirty (release)
PSY Release: A

Additional context / Relevant Files
The marvel.7z also contains an H.265 encoded marvel.mkv source with Dolby Vision from which the RPU data can be extracted with dovi_tool, and example .ivf outputs with --enable-overlays 1 and --enable-overlays 0.