feat: add ei_text support

[hojjatabdollahi]

Description

Adds Text capability to the seat. Also adds TextKeySym and TextUtf8 as EiInputEvents.

Checklist

• I agree to smithay's Developer Certificate of Origin.

[ids1024]

This may not be changed by this PR, but testing with https://github.com/pop-os/cosmic-osk, virtual-keyboard input doesn't really seem "isolated"? Shift or caps lock on a physical keyboard seems to capitalize input when pressing a key on the OSK. And holding shift on the OSK likewise affects input on the physical keyboard.

That's not supposed to happen here, right?

Also, since https://wayland.app/protocols/virtual-keyboard-unstable-v1#zwp_virtual_keyboard_v1:request:modifiers is sent from the client to the server, is the virtual-keyboard protocol not supposed to update the xkb state automatically on handling key, and only on when the client sends modifiers?

May need to compare other implementations.

[ids1024]

I guess we also need to pass modifiers to the keyboard grab if the last key press was on a different isolated keyboard state, or non-isolated? (Likewise input_forward for normal input.)

We probably need to track somewhere what the last send modifiers were.

[hojjatabdollahi]

I added something to track it. It gets reset on set_focus.

[hojjatabdollahi]

This may not be changed by this PR, but testing with https://github.com/pop-os/cosmic-osk, virtual-keyboard input doesn't really seem "isolated"? Shift or caps lock on a physical keyboard seems to capitalize input when pressing a key on the OSK. And holding shift on the OSK likewise affects input on the physical keyboard.

That's not supposed to happen here, right?

I think the idea was that shortcuts that the compositor should intercept (such as super+t) work even if you're holding shift on the physical keyboard. Modifiers otherwise go to the client.

But with the latest commit I implemented last_sent_modifier, and now the modifiers on physical keyboard doesn't affect the osk, but I don't know if that's desirable.

Also, since https://wayland.app/protocols/virtual-keyboard-unstable-v1#zwp_virtual_keyboard_v1:request:modifiers is sent from the client to the server, is the virtual-keyboard protocol not supposed to update the xkb state automatically on handling key, and only on when the client sends modifiers?

May need to compare other implementations.

Gnome and KDE don't use zwp_virtual_keyboard. I think the only implementation would be wl_roots and it just uses the shared keyboard state for the seat (so do Gnome and KDE with their virtual keyboards).

So, maybe for the virtual keyboard we want to do the same as everybody else and share it with the physical keyboard.

Also had to add pop-os/cosmic-comp@990525a to make sure esc works in cosmic-osk.

What do you think?

[ids1024]

Yeah, I guess if zwp_virtual_keyboard_v1 is basically a wlroots protocol, we want to be consistent with their implementation since that is the behavior clients expect. Even if the protocol looks like it should behave differently (the spec isn't as clear as official Wayland protocols usually are).

Though the client can also use zwp_virtual_keyboard_v1::keymap to set a different keymap from what the compositor is otherwise using, so shift level and group don't necessarily even mean the same thing outside the virtual keyboard? Mixing up the group index with isolated keyboard using different keymaps presumably wouldn't make much sense.

For libei, Gnome is probably the implementation to reference, but hopefully the intended behavior of libei is better specified.

[ids1024]

Oh, it's separate from this, but I guess sending ei_keyboard.modifiers is also something that we need to add. Looks like we should be send that to all libei sender clients with an ei_keyboard any time the compositor's modifier state changes.

[hojjatabdollahi]

Alright, I'm rewriting everything to be more in line with the protocol and other compositors.

• I'm keeping a map of held modifiers keyed by the source device, that enables us to have a consistent final compiled modifiers list.
• When a modifier changes, we send a wl_keyboard.modifiers and ei_keyboard.modifiers to all the clients, everybody needs to update their internal state based on compositors state. So pressing shift on a physical keyboard should show up on cosmic-osk too.
• I'm removing all the isolated keyboard stuff, if you're typing on a remote machine and a person there hold down a key, then depending on your remote desktop software (if it's sending keysym or keycode) they're gonna be affected by the modifier (that's the intended behavior).
• The clients (vk, ei) must release everything when they go away. So probably a cosmic-osk PR is needed here.

Concerns:

The reason we had isolated keyboard stuff, was because if a remote user holds down "super" you can't do anything anymore on your local machine! You can't even click on "quit" (on Rustdesk for example) to kick the remote user off because that just drags the window. You can't type, that would just do all the super+* shortcuts. But apparently that's by design.
Also buggy clients may mess up your modifiers list and you have no way of resetting it. For libei clients, when they get disconnected the server can clear all their modifiers (when removing the device). For virtual keyboard, it's up to the osk app, for example if I press super (latch) on the osk, if for some reason the keyboard interface is going to hide or go away the app is required to release that, otherwise we're gonna have the super key held down while trying to bring osk back up and clicking on it to release.

[ids1024]

I've made some changes to cosmic-osk to test a keymap that's different from what the compositor has. I need to do a bit of testing, but if it shares modifiers, including group, with different keymaps, that seems like a bug in wlroots, or the design of the virtual-keyboard protocol...

I guess it's not really explicit with libei/xdg-desktop-portal if the modifier state should be independent. Libei itself provides a way to expose devices, but isn't explict about what exactly they represent, and https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.impl.portal.RemoteDesktop.html#org-freedesktop-impl-portal-remotedesktop-connecttoeis doesn't really provide additional details.

But we probably want ei_keyboard to not be isolated from the main keyboard (probably should verify that's what Gnome does).

[ids1024]

If we decide we don't need IsolatedKeyboardState for anything related to libei, we can just leave any changes to virtual-keyboard out of this PR. It may also need various improvements, but those can be a separate PR. Then we can hopefully merge the libei part sooner.