Unable to allocate a surface from internal buffer pool

[Yasso9]

Description:

When executing the wl-screenrec command, the following error is encountered:

Using output DP-3
Opening libva device from DRM device /dev/dri/renderD128
[AVHWFramesContext @ 0x560bd1d2ec80] Failed to create surface: 14 (the requested RT Format is not supported).
[AVHWFramesContext @ 0x560bd1d2ec80] Unable to allocate a surface from internal buffer pool.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ffmpeg::Error(12: Cannot allocate memory)', src/main.rs:1256:14
stack backtrace:
   0:     0x560bcfc73846 - <unknown>
   1:     0x560bcfc3a47f - <unknown>
   2:     0x560bcfc71347 - <unknown>
   3:     0x560bcfc73655 - <unknown>
   4:     0x560bcfc74a23 - <unknown>
   5:     0x560bcfc747e4 - <unknown>
   6:     0x560bcfc74f16 - <unknown>
   7:     0x560bcfc74e24 - <unknown>
   8:     0x560bcfc73c46 - <unknown>
   9:     0x560bcfc74bb2 - <unknown>
  10:     0x560bcfbdc7e3 - <unknown>
  11:     0x560bcfbdcc23 - <unknown>
  12:     0x560bcfcb0ef1 - <unknown>
  13:     0x560bcfcd3b70 - <unknown>
  14:     0x560bcfcd4cb5 - <unknown>
  15:     0x560bcfcd4fef - <unknown>
  16:     0x560bcfcb85ae - <unknown>
  17:     0x560bcfcda513 - <unknown>
  18:     0x560bcfcee219 - <unknown>
  19:     0x560bcfc6cfa5 - <unknown>
  20:     0x560bcfcbee55 - main
  21:     0x7f312be45cd0 - <unknown>
  22:     0x7f312be45d8a - __libc_start_main
  23:     0x560bcfbe1a35 - <unknown>
  24:                0x0 - <unknown>

Steps to Reproduce:

Executing the wl-screenrec command results in the memory allocation error mentioned above.

Environment:

• Operating System: Arch Linux
• CPU: AMD Ryzen 9 7900X
• GPU: AMD Radeon RX 6700 XT
• GPU Driver: amdgpu
• Kernel: 6.6.8-arch1-1

Observations:

The error persists across both AUR packages (wl-screenrec and wl-screenrec-git), and the use of the --no-hw flag did not resolve the issue.

[JoshElias]

I have this same error. I've made sure vaapi is enabled. Could be we need to set the --codec?

[Yasso9]

I have tested with all --codec values (auto, avc, hevc, vp8, vp9) on both iGPU and GPU and I still have the same error.

[Yasso9]

I tested on another computer with an intel CPU and Iris Xe iGPU, and it works with vaapi drivers installed. I have the same system on both computer.

[russelltg]

I've improved the error messages in latest git, curious to see what you get now. It's also possible (but maybe unlikely) that this was the root cause as #60, which I just pushed a fix for, so please retest.

[Yasso9]

Thanks for your answer

I just reinstalled wl-screenrec-git, and I still have the same error, but I have one more message before the program exit

Using output DP-1
Opening libva device from DRM device /dev/dri/renderD128
[AVHWFramesContext @ 0x5832e5fa57c0] Failed to create surface: 14 (the requested RT Format
 is not supported).
[AVHWFramesContext @ 0x5832e5fa57c0] Unable to allocate a surface from internal buffer poo
l.
failed to create encoder: Failed to create vaapi frame context for capture surfaces of for
mat X2RGB10LE 3840x2160

Here are some more information if you need them:

I have 2 graphics cards :

• an iGPU, the one from the ryzen 9 7900x, on /dev/dri/renderD128
• a GPU, an AMD 6700XT, /dev/dri/renderD129

I tested to run the program on both GPU using the arguments --dri-device /dev/dri/renderD129 and --dri-device /dev/dri/renderD128. I have the same error on both.

When I try to run wf-recorder with the arg --device /dev/dri/renderD128 I have the following error that look like the one on wl-screenrec :

using VA-API, trying to enable DMA-BUF capture...
enabled DMA-BUF capture, device /dev/dri/renderD128
selected region 0,0 0x0
Setting codec option: qp=25
[AVHWFramesContext @ 0x77fdcc2e5d80] Failed to create surface: 14 (the requested RT Format
 is not supported).
[AVHWFramesContext @ 0x77fdcc2e5d80] Unable to allocate a surface from internal buffer poo
l.
Cannot create hw frames context: Cannot allocate memory

But when using --device /dev/dri/renderD129 it works perfectly, whereas wl-screenrec launch the same error

using VA-API, trying to enable DMA-BUF capture...
compositor running on different device, disabling DMA-BUF
selected region 0,0 0x0
Failed to find AV format for0Using video filter: hwupload,scale_vaapi=format=nv12:out_rang
e=full
[hevc_vaapi @ 0x7f0b2c002000] No quality level set; using default (25).
[hevc_vaapi @ 0x7f0b2c002000] Driver does not support some wanted packed headers (wanted 0
xd, found 0x1).
Choosing sample format fltp for audio codec aac
Output #0, matroska:
  Stream #0:0: Video: hevc (Main), vaapi(pc), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31
  Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s
Using PulseAudio device: alsa_output.pci-0000_03_00.1.hdmi-stereo-extra3.monitor

If you need more information about my system, just ask, I will try to do my best to answer. I really like the program on my Intel laptop, more than wf-recorder, so it can be pretty cool if it works on my desktop too.

[russelltg]

compositor running on different device, disabling DMA-BUF

it seems they don't use dmabuf at all in this situation, which wl-screenrec always uses. This is somewhat related to #15.

I think the only option to get wl-screenrec (barring wl-screenrec shm support, which I have little interest in adding as wf-recorder already fits this usecase well) is to drop your display to 8-bit mode instead of 10.

[Yasso9]

Ah, you're right, I removed bitdepth,10 on my hyprland monitor config, and now it works like a charm. I didn't have great use of the 10 bits support anyway. Thanks a lot for your help !