[WIP] Wayland: Implement native sub-windows

[Riteo]

WIP as some features are still not multiwin aware but otherwise everything seems very usable.

Thanks to all the improvements the backend got over the years, it's now finally possible to have a stable native multi-window experience.

The existing Godot API is still not compatible with all of Wayland expectations, so I distilled the bare minimum into two three, hopefully small enough additions:

• WINDOW_EVENT_FORCE_CLOSE, which is a surefire way of cleaning up a window. It basically makes the window call hide on itself, because the compositor might close a window (e.g. a popup) whether we like it or not. This should be useful for other platforms too.
• WINDOW_FLAG_POPUP_WM_HINT, which lets the DisplayServer know that this is supposed to be a popup, as there's no failproof heuristic for that. This should also come in handy for X11 compositors as the backend has issues telling them apart too.
• FEATURE_SELF_FITTING_WINDOWS, which lets nodes know that they should not try to adjust windows and popups to fit in the screen, as they don't have enough data and the compositor does that for them already. I think that this is the only addition that would be of use only for this backend.

This PR includes also a lot of core improvements and fixes to stuff that became more evident while testing multiple windows, including a complete overhaul of pointer/tablet handling. I'm definitely going to backport/separate some of those. It does not even include the FIFO additions (#101454) or fixes for some leftover issues that @arlo-phoenix found in the original PR so once everything is done the Wayland experience should become even more solid.

I'll document everything properly once it's done, I plan to squash this PR and treat it properly this time.

So, yeah, keeping this here for reference and testing for whoever feels adventurous. I'll keep chipping at it in the meantime.

Here's a picture of it running on sway:

Known issues

• There's some error spam while starting the editor. It's an engine bug: At startup ERROR: editor/progress_dialog.cpp:169 - Do not use progress dialog (task) while flushing the message queue or using call_deferred()! #100900

[scgm0]

When a subwindow appears, it will cause the main window to exit maximized mode.

2025-01-19.23-10-21.mp4

[Riteo]

Hi @scgm0, thank you for testing! I think I know why. I'll push a fix soon.

[bruvzg]

WINDOW_EVENT_FORCE_CLOSE, which is a surefire way of cleaning up a window. It basically makes the window call hide on itself.

This will be useful for Windows/macOS/X11 display server popup auto close logic, currently these are using WINDOW_EVENT_CLOSE_REQUEST, but expect window to be closed unconditionally, if it's not, it can stay opened in a partially broken state.

[Riteo]

This will be useful for Windows/macOS/X11 display server popup auto close logic, currently these are using WINDOW_EVENT_CLOSE_REQUEST, but expect window to be closed unconditionally, if it's not, it can stay opened in a partially broken state.

Hey that's exactly what I was trying to avoid! I'm really glad this will be useful for other backends too! :D

[Riteo]

@scgm0 I pushed quite a few changes, including a fix for the issue you mentioned.

[scgm0]

@scgm0 I pushed quite a few changes, including a fix for the issue you mentioned.

New issue, the borderless setting of the main window is not working

[Riteo]

New issue, the borderless setting of the main window is not working

Does this happen only on the main window? Because unfortunately that's a known issue with the decoration library we're using, see #93472. I had it disabled until this last push to make debugging simpler.

I think I might need to find a workaround since the upstream bug report went nowhere unfortunately.

[scgm0]

New issue, the borderless setting of the main window is not working

Does this happen only on the main window? Because unfortunately that's a known issue with the decoration library we're using, see #93472. I had it disabled until this last push to make debugging simpler.

I think I might need to find a workaround since the upstream bug report went nowhere unfortunately.

But before you fix the maximized window, the main window can be borderless, which is a bit strange.

[Riteo]

But before you fix the maximized window, the main window can be borderless, which is a bit strange.

It stopped working because I also pushed other things along the fix, including "Re-add libdecor support". I usually work in bursts and push everything at once.

[scgm0]

But before you fix the maximized window, the main window can be borderless, which is a bit strange.

It stopped working because I also pushed other things along the fix, including "Re-add libdecor support". I usually work in bursts and push everything at once.

I see, so would libdecor support be of any benefit?

[Riteo]

I see, so would libdecor support be of any benefit?

Ye it is, as some compositors do not have their own decorations, like GNOME. We could implement our own decorations instead of using a library but it's not exactly simple. I'd like to look into that eventually though as it allows fancier features.

[Riteo]

Heads up to whoever might be testing this PR: expect instability and crashes with popups in the latest batch of commits, I'm doing some experiments

[Riteo]

update: things should have come back to a decent state, but popups are a bit iffy when they get pushed back by the compositor, as godot tries to do some fitting itself. I'll need to find a good way to stop popups from trying to be smart.

[Riteo]

Folks, I think I have a plan!

It's quite simple, we add a feature flag for "self-fitting windows" (not a great name I know) and if that's available we avoid "adjusting" popup windows. Now things should work great!

Edit (I don't want to spam too much) here's what it allows:

Not bad for not having screen-space window handling :P

[Riteo]

Heads up: I fixed an issue with child toplevels being shrunk to their minimum size. I thought it was my project metadata being funky but no, it was an issue in the window adjusting code I special cased for the new feature flag 😅