D3D11 vs Other backends

[st0rmbtw]

D3D11 gives the best performance among all the rendering backends that LLGL supports.

D3D12 is much slower than D3D11 (even though it's supposed to be faster according to tests on the internet and because it gives more control).
Vulkan is also slower than D3D11 but faster than D3D12.

Is it possible to achieve the D3D11's performance with the other backends?

In my project I get:

Backend	FPS
D3D11	~6000 😲
D3D12	850-1000
Vulkan	~2700
OpenGL	?*
Metal	?**

*: I didn't manage to test OpenGL because I can't attach to OpenGL renderer via RenderDoc for some reason. I get this error:

failed to create extended OpenGL profile
in 'ErrUnsupportedGLProc': illegal use of unsupported OpenGL procedure: glCreateShader

**: Didn't manage to test Metal properly as well, because I have a poor GPU on Mac.

Some notes

WriteTexture is very slow on D3D12 (On D3D11 it takes less than 1 ms, while on D3D12 it takes ~2 seconds to write the same amount of data to a texture)
WriteTexture is also slower on Vulkan than on D3D11 but faster than on D3D12.

Vulkan reports a ton of validation errors every frame for some reason.
Vulkan crashes often when I recreate large buffers.

Tests were done with the following conditions

OS: Windows 11 23H2
CPU: Ryzen 2700X
GPU: RTX 2060
Compiler: clang 18.1.8
Compiler options: -g0, -O3, -fomit-frame-pointer, -fno-math-errno, -ffinite-math-only, -fno-signed-zeros, -fno-trapping-math, -fassociative-math
Debug layer: Disabled

[LukasBanana]

I'm curiously following your project :) I'm also working on a project that's a lot larger than the examples included in this repository, in hope to have a benchmark/demo project to showcase and test LLGL with a larger use case, but that project is far from ready to be published.

I can also confirm that the D3D11 backend is the fastest LLGL has and beating it is a tough task. Having said that, in my project I'm seeing D3D12 at about 50% speed compared to D3D11, so 850 FPS compared to 6000 sounds extreme and so does ~2 seconds sound in the WriteTexture call. These calls are indeed poorly synchronized with the GPU, so they use a worst case approach that stalls the GPU, but 2 seconds like a different problem. Can you share some more details about these calls specifically? What type of texture data and size?

I have no trouble debugging the GL backend in RenderDoc. I assume you have the latest version? Can you share your RenderDoc parameters?

Vulkan is still in an early state and I'm trying to address validation errors as I get reports, i.e. I see people actively using it.
Can you share some details about what buffer sizes you re-create? I can certainly spin up a unit test for that.

[st0rmbtw]

Oh, nice to hear that you're following my project 😊. Feel free to use it as a test field for LLGL.

In my project, I recreate buffers (tile data) of a chunk when it gets modified (when a block is set or removed), you can see the code here: https://github.com/st0rmbtw/TerrariaCloneCpp/blob/6b38f18012b489192c0ebe373278f31e41b3e163/src/world/chunk.cpp#L102-L124

Honestly, I don't know exactly how long WriteTexture takes on D3D12, I judged by feelings :). But it really muuuch slower than on D3D11. I use it here: https://github.com/st0rmbtw/TerrariaCloneCpp/blob/6b38f18012b489192c0ebe373278f31e41b3e163/src/renderer/world_renderer.cpp#L120

Though 6000 fps on D3D11 is real, I measured it via RenderDoc.

I have no trouble debugging the GL backend in RenderDoc. I assume you have the latest version? Can you share your RenderDoc parameters?

I think it may be something with compilation flags because I can debug it when it is not built for release (i.e. zero optimizations).

[LukasBanana]

I hope you aren't using a debug build for the performance test. You mentioned debug layer is disbaled, so I presume you did not, but just to be sure, because I see you're using DebugDevice flags here, which is great for runtime validation, I still want to emphasize that it's tremendously slower on D3D12 than D3D11 to my experience.

With GL acting weird in optimized code, I would add some logs/printfs into GLCoreExtensionLoader.cpp to see what extensions are available when launching through RenderDoc and if ARB_shader_objects is called to initialize those function pointers.

A little bit off topic:
While it's certainly desirable to get D3D12 and all of its feature set on par with D3D11 - and one day hopefully beyond that - I'd like to point out that the RenderSystem::Write... functions are generally not tuned for the highest performance, their CommandBuffer counterparts are, such as UpdateBuffer. These of course come with limitations and textures don't even have such functions (you would have to update a buffer and then copy that buffer into the texture). Not only are they favorable for performance, you also have to keep in mind that the RenderSystem functions should not be inlined with command buffer recording, especially not in D3D12, Vulkan, and Metal. In other words, command buffers cannot rely on buffer modifications via WriteBuffer while they are being recorded because those commands have not been submitted to the GPU yet. Even if you use an immediate context (i.e. CommandBufferFlags::ImmediateSubmit), those commands will only be submitted to the GPU once recording is finished. So when you update a buffer via WriteBuffer more than once during command buffer recording, its previous content will be overwritten before a command buffer was able to use it. Maybe an example can illustrate that better:

cmd->Begin();
cmd->SetResource(0, *buffer);
renderer->WriteBuffer(*buffer, /* Value A */);
cmd->Dispatch(1,1,1); // Wants to read value A from "buffer"
renderer->WriteBuffer(*buffer, /* Value B */);  <-- BOOM: overrides value A before the command buffer had a chance to read it
cmd->Dispatch(1,1,1); // Reads same value as previous Dispatch
cmd->End();

I see you prepared your project for large buffer updates. Calling WriteBuffer during command recording is not a problem per se, but you have to keep track of where you are writing to and point your shaders to the next location until you submit your command buffer or reach End() on an immediate context. This could look something like this:

uint64_t currentOffset = 0;
cmd->Begin();
cmd->SetResource(0, *buffer);

renderer->WriteBuffer(*buffer, currentOffset, &valueA, sizeof(valueA));
cmd->SetUniform(0, currentOffset); // Tell shader from where to read in "buffer" for next Dispatch
currentOffset += sizeof(valueA);

cmd->Dispatch(1,1,1); // Reads value A

renderer->WriteBuffer(*buffer, currentOffset, &valueB, sizeof(valueB));
cmd->SetUniform(0, currentOffset); // Update offset where to read from
currentOffset += sizeof(valueB);

cmd->Dispatch(1,1,1); // Reads value B
cmd->End();
queue->Submit(*cmd); // This is where "buffer" is finally being accessed by the GPU and all writes are becoming visible

With UpdateBuffer, LLGL already does that for you in the APIs that require explicit command buffer submission, including pooling when a buffer exceeded its size. The size limitation comes directly from Vulkan's vkCmdUpdateBuffer() specification as these updates may be part of the command buffer memory.

I updated the documentation on those functions in 34e5041.

[st0rmbtw]

I hope you aren't using a debug build for the performance test.

No, I used a release build. I provided the optimisation flags I used for the tests in the description.
I define the DEBUG macro for debug build and if it's defined I initialize the validation layer.

the RenderSystem functions should not be inlined with command buffer recording, especially not in D3D12, Vulkan, and Metal

What's the recommended way to use it then? End recording of a command buffer before using a function and begin recording again after? Doesn't often ending and beginning command buffer recording hit performance?

Would splitting the data into chunks of 65536 bytes and uploading each of them to a buffer using CommandBuffer::UpdateBuffer be more efficient?

[LukasBanana]

End recording of a command buffer before using a function and begin recording again after? Doesn't often ending and beginning command buffer recording hit performance?

There are multiple ways to achieve this:

1. Your suggestion would indeed be a way. I honestly can't tell you what the performance impact is, that's something only empirical data can tell. I can imagine the D3D11 driver does this type of scheduling.
2. If you have to update buffers larger than 2^16 bytes during command recording, you can use the approach I illustrated above using segments of a larger buffer and point your shaders to the correct offset.
3. You could also allocate multiple buffers of the same size and then update the buffer and swap it out with the previous one using SetResource.
4. If your buffers are only slightly larger than the 2^16 limit, you could also encode the update with multiple UpdateBuffer commands, since the destination offset has 64 bit. Only the length of each update is limited to 16 bit.
5. Maybe you even have a way of delaying the update to the next frame, so you can call WriteBuffer between command recordings.

I recently added a new example for a mini puzzle game that deals with a similar problem to make it all work in WebGL where SSBOs are not supported, i.e. you are restricted to uniform buffers that are quite limited in size, too. Take a look at the InstanceBuffer class and how it splits up draw calls when storage buffers (aka SSBOs) are not supported.

[LukasBanana]

I am pleased the announce that I have managed to beat the D3D11 backend performance with the D3D12 backend for the first time. My side project is not ready to be presented, but I can share the performance differences (it's a very small scene but it has a deferred renderer, frustum culling, terrain generation, shadow-mapping, post-processing, and so on):

D3D11 (1280x768):

• Average frame rate: 1668.61 Hz (0.60ms)
• Maximum frame rate: 2900.23 Hz (0.34ms)
• Minimum frame rate: 231.42 Hz (4.32ms)

D3D12 (1280x786):

• Average frame rate: 1799.21 Hz (0.56ms)
• Maximum frame rate: 3652.30 Hz (0.27ms)
• Minimum frame rate: 251.58 Hz (3.97ms)

D3D11 (3440x1440):

• Average frame rate: 691.09 Hz (1.45ms)
• Maximum frame rate: 1051.08 Hz (0.95ms)
• Minimum frame rate: 102.72 Hz (9.74ms)

D3D12 (3440x1440):

• Average frame rate: 775.25 Hz (1.29ms)
• Maximum frame rate: 3005.71 Hz (0.33ms)
• Minimum frame rate: 421.19 Hz (2.37ms)

The big performance killer for D3D12 backend in LLGL at the moment are all the Write functions from the RenderSystem because they are synchronized instead of clever scheduling. I hope to improve that over the holidays. Until then, if you can encode all your buffer writes during the frame only with UpdateBuffer from the CommandBuffer interface, performance should be pretty good.
I also made some optimizations with resource barriers which I haven't submitted yet, but these are minor improvements.
You can also tweak the performance potentially by raising the number of native command buffers from 2 (default) to 3 (any higher value is capped at the moment).

[LukasBanana]

My first guess would be that your variable index format for triangle-strip topology makes the draw calls (slightly) more expensive (world_renderer.cpp:62):

    pipelineDesc.indexFormat = LLGL::Format::Undefined;
    pipelineDesc.primitiveTopology = LLGL::PrimitiveTopology::TriangleStrip;

This is probably not obvious and fine to use it that way, but for D3D12, LLGL needs to create two PSOs internally and then swap out the correct one on the first draw call depending on what index buffer is used (16 or 32 bits). Since you don't even use an index buffer (as far as I can tell) you could just set it to Format::R32UInt to let LLGL create only a single one. This will also ensure that the PSO is bound when SetPipelineState is called and not at the first draw call.

I have a few extra ideas for performance improvements:

1. Use static samplers if you can. Probably not a big performance improvement, but it spares you extra internal heap management. Each SetResource needs to put the resource view into a unique place of an internal heap pool. If you know a sampler won't be swapped out for a given PSO and its layout, let LLGL make use of D3D12's static sampler descriptors.
2. On the same note, consider using resource heaps. This allows you to optimize binding multiple resources at once. This is perhaps something you might want to do when you're getting closer to finishing your project, because it makes your PSO layout a bit more rigid. Swapping out resources with a single SetResource call is simply more convenient IMHO, but the resource heaps are optimized to swap out several resource views at once.

As mentioned earlier, I hope to find some time until the end of the year to improve the resource barrier mechanism of the D3D12 backend. I'll keep you posted.

[st0rmbtw]

I applied your suggestions and FPS has increased by around 150. Not much :(

you could just set it to Format::R32UInt

Wouldn't R16UInt be better as it uses less memory?

As mentioned earlier, I hope to find some time until the end of the year to improve the resource barrier mechanism of the D3D12 backend. I'll keep you posted.

Okay, thank you! Good luck!

Merry Christmas BTW :)

[LukasBanana]

150 more FPS actually sounds like a lot but the reduced frame time in milliseconds would be a better measure, I assume this equates to ~6.6ms less, which is also quite a lot comparatively speaking.

I'm not sure if using R16UInt would make a difference if you don't have an indexed draw call. Perhaps it does if D3D12 takes the information of the strip cutoff value into account, but I can't tell how this affects the API.

Merry Christmas to you as well :)

[st0rmbtw]

I assume this equates to ~6.6ms less, which is also quite a lot comparatively speaking.

There were around 1000 FPS before changes and 1150 FPS after, so it's

1000 / 1000 - 1000 / 1150 = 1 - 0.86 = 0.14

0.14 ms less now, isn't?

[LukasBanana]

Yes, if you take your previous FPS into account, the difference would be 0.14ms, I simply looked at the 150 FPS on its own, that's why comparing FPS is not as straightforward as comparing frame time because it's non-linear (I found this article that explains it hopefully better than I me: https://www.mvps.org/directx/articles/fps_versus_frame_time.htm).

If your frame is only taking 1ms to render, 0.14ms (14%) is not a neglectable change in performance.

I know you are comparing this with the D3D11 renderer and that's where we want to get, but we should do these comparisons in frametime instead of framerate. In my side project, I am also showing min, max, and average frametime alongside the FPS.