RealTime分支对720x480 10bit片源编码会coredump

[jijiwawa]

问题现象

[root@localhost build]# ./uAVS3_gop -f ../bin/encoder_ra.cfg -p InputFile=/home/video_files/720x480_420p_10bit.yuv -p SourceWidth=720 -p SourceHeight=480 -p slice_sao_enable_Y=0 -p slice_sao_enable_Cb=0 -p slice_sao_enable_Cr=0 -p WQuant=0 -p SpeedLevel=5 -p RateControl=0 -p OutputFile=/home/test_files/out1.avs3 -p SampleBitDepth=10 -p CodingBitDepth=10 -p threads-gop=1
Parsing Configfile ../bin/encoder_ra.cfg......................................
Parsing command line string 'InputFile = /home/video_files/720x480_420p_10bit.yuv '.
Parsing command line string 'SourceWidth = 720 '.
Parsing command line string 'SourceHeight = 480 '.
Parsing command line string 'slice_sao_enable_Y = 0 '.
Parsing command line string 'slice_sao_enable_Cb = 0 '.
Parsing command line string 'slice_sao_enable_Cr = 0 '.
Parsing command line string 'WQuant = 0 '.
Parsing command line string 'SpeedLevel = 5 '.
Parsing command line string 'RateControl = 0 '.
Parsing command line string 'OutputFile = /home/test_files/out1.avs3 '.
Parsing command line string 'SampleBitDepth = 10 '.
Parsing command line string 'CodingBitDepth = 10 '.
Parsing command line string 'threads-gop = 1 '.

WPP threads of one frame: 1 (allowed 1-15)
FRM threads of one frame: 1 (allowed 1-33)

Version: 1.1.9_release, SHA-1: ea36d7b
Total Memory: 241.93

Input YUV file : /home/video_files/720x480_420p_10bit.yuv
Output AVS bitstream : /home/test_files/out1.avs3
Output YUV file :

========== Frame Info Log Start ==========
Num POC T QP Bits PSNR(y,u,v) Time

Segmentation fault (core dumped)

环境信息

CPU:Kunpeng-920
OS:openEuler release 22.03 LTS
gcc:gcc version 10.3.1 (GCC)

复现步骤

git clone https://github.com/uavs3/uavs3e.git -b RealTime
cd uavs3e/
sh +x version.sh
mkdir build
cd build/
cmake -DCOMPILE_10BIT=1 -DCMAKE_BUILD_TYPE=Debug  ..
make -j
# 以下2个命令均出现coredump
 ./uAVS3_gop  -f ../bin/encoder_ra.cfg -p InputFile=/home/video_files/720x480_420p_10bit.yuv -p SourceWidth=720 -p SourceHeight=480 -p slice_sao_enable_Y=0 -p slice_sao_enable_Cb=0 -p slice_sao_enable_Cr=0 -p WQuant=0 -p SpeedLevel=5 -p RateControl=0 -p OutputFile=/home/test_files/out1.avs3 -p SampleBitDepth=10 -p CodingBitDepth=10 -p threads-gop=1

./uAVS3_gop  -f ../bin/encoder_ra.cfg -p InputFile=/home/video_files/720x480_420p_10bit.yuv -p SourceWidth=720 -p SourceHeight=480  -p SpeedLevel=5 -p RateControl=0 -p OutputFile=/home/test_files/out1.avs3 -p SampleBitDepth=10 -p CodingBitDepth=10 -p threads-gop=1

[jijiwawa]

堆栈信息如下，初步分析是uiBitSize传入为0 导致
#1 0x00000000004411bc in xSetEdgeFilterParam (h=h@entry=0xffffe9337980, uiBitSize=uiBitSize@entry=0,
b8x=b8x@entry=28, b8y=b8y@entry=0, idir=idir@entry=0, flag=flag@entry=3)
at uavs3e/uAVS3lib/loopfilter.c:123
#2 0x00000000004412e4 in set_cu_deblk_flag (h=0xffffe9337980, uiBitSize=,
uiPositionInPic=) at uavs3e/uAVS3lib/loopfilter.c:178
#3 0x0000000000441420 in set_cu_deblk_flag (h=0xffffe9337980, uiBitSize=,
uiPositionInPic=) at uavs3e/uAVS3lib/loopfilter.c:172
#4 0x0000000000441420 in set_cu_deblk_flag (h=0xffffe9337980, uiBitSize=,
uiPositionInPic=) at uavs3e/uAVS3lib/loopfilter.c:172
#5 0x0000000000441420 in set_cu_deblk_flag (h=0xffffe9337980, uiBitSize=,
uiPositionInPic=) at uavs3e/uAVS3lib/loopfilter.c:172
#6 0x0000000000441420 in set_cu_deblk_flag (h=0xffffe9337980, uiBitSize=,
uiPositionInPic=) at uavs3e/uAVS3lib/loopfilter.c:172
#7 0x0000000000441420 in set_cu_deblk_flag (h=h@entry=0xffffe9337980, uiBitSize=uiBitSize@entry=5,
uiPositionInPic=uiPositionInPic@entry=28) at uavs3e/uAVS3lib/loopfilter.c:172
#8 0x0000000000450b10 in encode_lcu_row (param=0xffffe9337980)
at uavs3e/uAVS3lib/slice.c:198
#9 0x000000000041d23c in avs3_threadpool_thread (pool=0x69de50)
at uavs3e/uAVS3lib/tools/threadpool.c:188
#10 0x0000fffff7e0209c in ?? () from /usr/lib64/libc.so.6
#11 0x0000fffff7e678dc in ?? () from /usr/lib64/libc.so.6

[dujiangpku]

待测试后修复

[jijiwawa]

进一步分析发现uAVS3lib\loopfilter.c文件 155行在这个场景uiPositionInPic特定参数，获取的cu->bitsize为0，导致了172 set_cu_deblk_flag递归导致传入的uiBitSize为0，从而导致了xSetEdgeFilterParam传入uiBitSize为0，导致了99行代码左移了负值，导致edge_size过大，导致了123行deblk_flag数组访问越界