muffin: doesn't unredirect fullscreen proton-based games

[user1-github]

Issue
It seems that fullscreen unredirection doesn't work even if "disable compositing for full-screen windows" is enabled in system settings. This causes stuttering in games.

Steps to reproduce
Enter any game, make sure vsync is disabled in game and you'll see that there is no screen tearing.

I have attached xsession-errors if you need it.
xsession-errors.txt

Cinnamon version - 4.4.3
GPU and Driver - AMD Radeon RX 580, Mesa 19.3.0-rc6 - kisak-mesa PPA

Original Post: linuxmint/cinnamon#9006

Update:
This only affects games running on Steam Proton. Fullscreen unredirection works properly only with native linux games.

Update 2
Ok, something is really bizzare about this. Earlier today when I was testing the issue, fullscreen unredirection worked with native linux games, but now it doesn't work with native linux games as well.

[clefebvre]

Moving towards Cinnamon as it's not a release blocker and we've no reliable way to reproduce.

Note: I'm pretty sure muffin doesn't "force" unredirection for all Windows. I know it's confusing and I'd be interested in revisiting this... but there's an actual formula at play here to decide whether something is composited or not, it's not just following that setting. @itzexor might shine in on this.

[pharmville]

For the few steam games I have tried inside cinnamon, the checkbox to "disable compositing for fullscreen windows" doesn't have any effect at all. Maybe instead of detection, just introduce a shortcut key to toggle the compositor on/off, like kwin does. Without a reliable method to disable the compositor, I cannot use gsync inside of cinnamon, because gsync requires the compositor to be off. Therefore, when playing games I am forced to use a different desktop environment, because there doesn't appear to be a way to enable gsync if cinnamon is running. Inside nvidia-settings there is a toggle called "Enable GSYNC visual indicator" so I am using the visual indicator to be able to know when gsync is running successfully or not. No matter what I try, gsync never activates inside of cinnamon for any fullscreen game. That leads me to believe cinnamon is currently incompatible with gsync. I have been able to get gsync working with gnome, kde, and xfce. I havent tried with mate, but I suspect it would work fine with compositor off. TLDR: Feature request - gsync compatibility to games running inside cinnamon.

[mtwebster]

We've been thinking about adding the ability to specify per-application whether or not to composite - similar to how in windows, you tweak nvidia settings either globally or specifically (can you still do that? It's been a while). So at least if something in the automatic detection fails, you can still have what you want.

[pharmville]

Thanks for considering a solution in the future, among your other priorities. I love the fact that you might make it configurable. While gnomes autodetection seems to work fine, I doubt it would catch every use-case, and I would rather have the choice to configure, as opposed to no choice at all. I feel that you are considering the best possible solution.

[Barafu]

I can confirm that it also affects Blizzard games running in Lutris. No Gsync, forced V-sync, no screen update by flipping. When I switch off V-sync in Cinnamon settings, game runs with or withou Vsync, according to its own settings. But it still can not update by flip, and thus use Gsync.

[jeffkeller87]

Can confirm this on Linux Mint 20. Checkbox to "disable compositing for fullscreen windows" seemingly has no effect. Unable to use GSYNC and must switch to another DE with compositing disabled.

[jeffkeller87]

I have found that "disable compositing for fullscreen windows" works for native games if you restart Cinnamon (Alt+F2, r) after toggling it on. However, this setting does not persist after a reboot which I have documented here: linuxmint/cinnamon#9803.

Even with these workarounds for native games, G-Sync still does not work for proton-based games.

[NikitaKarnauhov]

I can confirm this issue on Fedora 34 with muffin-4.8.1. Not only proton games are affected but native games and applications are also affected. E.g. VRRTest sometimes works and at other times doesn't.

I debugged a bit and found out that offscreen windows sometimes prevent fullscreen unredirection from triggering. They may appear on top of the window stack but fail (obviously) to fulfill unredirection requirements.

Example of such window (in the case of a proton game):

$ xprop -id 0x7a0002a
_GTK_EDGE_CONSTRAINTS(CARDINAL) = 170
_NET_WM_STATE(ATOM) = _NET_WM_STATE_SKIP_PAGER, _NET_WM_STATE_SKIP_TASKBAR
_NET_WM_ALLOWED_ACTIONS(ATOM) = _NET_WM_ACTION_CHANGE_DESKTOP, _NET_WM_ACTION_ABOVE, _NET_WM_ACTION_BELOW
STEAM_GAME(CARDINAL) = 251290

$ xwininfo -id 0x7a0002a

xwininfo: Window id: 0x7a0002a (has no name)

  Absolute upper-left X:  -1
  Absolute upper-left Y:  -1
  Relative upper-left X:  -1
  Relative upper-left Y:  -1
  Width: 1
  Height: 1
  Depth: 24
  Visual: 0x21
  Visual Class: TrueColor
  Border width: 0
  Class: InputOutput
  Colormap: 0x7a00001 (installed)
  Bit Gravity State: ForgetGravity
  Window Gravity State: NorthWestGravity
  Backing Store State: NotUseful
  Save Under State: no
  Map State: IsViewable
  Override Redirect State: yes
  Corners:  +-1+-1  -2560+-1  -2560-1440  +-1-1440
  -geometry 1x1+-1+-1

This window has gone away after the game was closed but another window appeared on "top" and broke unredirection for native apps after that:

$ xprop -id 52428821      
_GTK_EDGE_CONSTRAINTS(CARDINAL) = 170
_NET_WM_STATE(ATOM) = _NET_WM_STATE_SKIP_PAGER, _NET_WM_STATE_SKIP_TASKBAR
_NET_WM_ALLOWED_ACTIONS(ATOM) = _NET_WM_ACTION_CHANGE_DESKTOP, _NET_WM_ACTION_ABOVE, _NET_WM_ACTION_BELOW
WM_HINTS(WM_HINTS):
		Client accepts input or input focus: True
		Initial state is Normal State.
		window id # of group leader: 0x3200001
XdndAware(ATOM) = BITMAP
_NET_WM_WINDOW_TYPE(ATOM) = _NET_WM_WINDOW_TYPE_NORMAL
_NET_WM_SYNC_REQUEST_COUNTER(CARDINAL) = 52428823, 52428824
_NET_WM_USER_TIME_WINDOW(WINDOW): window id # 0x3200016
WM_CLIENT_LEADER(WINDOW): window id # 0x3200001
_NET_WM_PID(CARDINAL) = 352359
WM_LOCALE_NAME(STRING) = "en_US.utf8"
WM_CLIENT_MACHINE(STRING) = "jinx"
WM_NORMAL_HINTS(WM_SIZE_HINTS):
		program specified location: 0, 0
		program specified minimum size: 0 by 0
		program specified base size: 0 by 0
		window gravity: NorthWest
WM_PROTOCOLS(ATOM): protocols  WM_DELETE_WINDOW, WM_TAKE_FOCUS, _NET_WM_PING, _NET_WM_SYNC_REQUEST
WM_CLASS(STRING) = "cinnamon", "Cinnamon"
WM_ICON_NAME(STRING) = "cinnamon"
_NET_WM_ICON_NAME(UTF8_STRING) = "cinnamon"
WM_NAME(STRING) = "cinnamon"
_NET_WM_NAME(UTF8_STRING) = "cinnamon"

$ xwininfo -id 52428821   

xwininfo: Window id: 0x3200015 "cinnamon"

  Absolute upper-left X:  -200
  Absolute upper-left Y:  -200
  Relative upper-left X:  -200
  Relative upper-left Y:  -200
  Width: 1
  Height: 1
  Depth: 24
  Visual: 0x2b
  Visual Class: TrueColor
  Border width: 0
  Class: InputOutput
  Colormap: 0x3200014 (not installed)
  Bit Gravity State: NorthWestGravity
  Window Gravity State: NorthWestGravity
  Backing Store State: NotUseful
  Save Under State: yes
  Map State: IsViewable
  Override Redirect State: yes
  Corners:  +-200+-200  -2759+-200  -2759-1639  +-200-1639
  -geometry 1x1+-200+-200

Maybe offscreen windows should be ignored when choosing the window for unredirection? I've changed the code a bit (see patch) and everything works ok so far for me.

[cozybuild]

I tested the patch and it works very well. For me when playing games (fullscreen) and sharing the screen in parallel it's easy to reproduce. Frametime is very unstable without the patch and I have a lot of screen stuttering. With the patch it's smooth and stable.

Game + screenshare with patch:

Unpatched (vanilla muffin 4.8.1) Game + screenshare