New IO keyboard/mouse/gamepad event API (1.87) recap

[ocornut]

EDIT 2022/02/29: We discovered a backward-compatibility regression in 18700:

• Direct legacy reads from io.KeysDown[] won't work with new backends.
• And in particular io.KeysDown[GetKeyIndex[XXX]) will buffer overflow in old and new backends.

Reads via IsKeyDown() function are always working, this only affect direct array access This was fixed in 18708 (commit 5659db5)

---

(This is a followup on #4858 now that things have settled)

We (@thedmd @ocornut) have completely revamped IO API in 1.87 (the way backend communicate with dear imgui to submit inputs)

All backends have been updated to use new API.

Both user code and old backends should be backward compatible (apart from a few rare edge cases) but we encourage you to transition.

Additions:

• New functions: io.AddKeyEvent(), io.AddKeyAnalogEvent(), io.AddKeyModsEvent() removed jan 27, io.AddMousePosEvent(), io.AddMouseButtonEvent(), io.AddMouseWheelEvent(), io.SetKeyEventNativeData() (for legacy support).
• New enums: ImGuiKey now contains a full keyset (e.g. ImGuiKey_F1) this allow using full range of keyboard inputs in a portable manner, which was previously impossible as we encouraged use of native indices. This will make it easier for people to share code/extensions publicly or across multiple codebases.
• New enums: ImGuiKey also contains gamepad axises/buttons ImGuiKey_GamepadDpadUp) unifying input sources.
• Input queue now trickle conflicting inputs over multiple frames (unless io.ConfigInputTrickleEventQueue == false) which massively improve input experience with low framerate.
• Both SDL and GLFW backends have been fixed to submit translated keys, so ImGuiKey_A will be submitted when pressing the key which user would need to press to emit an "A" character, allowing for translated shortcuts.

Those changes will constitute the bulk of 1.87.
If you want to help, please upgrade today :)

Transition Guide

If you are using a standard backend (RECOMMENDED) just update it to latest!
If you have are using a custom backend:

Keyboard:

• Backend writing to io.KeyMap[], io.KeysDown[] -> backend should call io.AddKeyEvent()
• If your backend now call io.AddKeyEvent() and you want legacy user code to still work with legacy indices, also call io.SetKeyEventNativeData().
• User code calling IsKeyPressed(MY_NATIVE_KEY_XXX) -> call IsKeyPressed(ImGuiKey_XXX) (OLD CODE WILL STILL WORK*)
• User code calling IsKeyPressed(GetKeyIndex(ImGuiKey_XXX)) -> call IsKeyPressed(ImGuiKey_XXX) (OLD CODE WILL STILL WORK*)
• All keyboard related functions taking int user_key_index now take ImGuiKey key: IsKeyDown(), IsKeyPressed(), IsKeyReleased(), GetKeyPressedAmount().
• Basically the trick we took advantage of is that we previously only supported native keycode from 0 to 511, so ImGuiKey values can still express a legacy native keycode, and new named keys are all >= 512. Luckily that old constraint allowed us to save the day!
• Added GetKeyName() helper function.
• Obsoleted GetKeyIndex(): it is now unnecessary and will now return the same value.
• For all new calls to IO functions, the Dear ImGui context should be bound/current.

(* until IMGUI_DISABLE_OBSOLETE_KEYIO is set. In a few versions, IMGUI_DISABLE_OBSOLETE_FUNCTIONS will automatically set it)
(* and unless the backend forgot to call io.SetKeyEventNativeData())

Mouse:

• Backend writing to io.MousePos -> backend should call io.AddMousePosEvent()
• Backend writing to io.MouseDown[] -> backend should call io.AddMouseButtonEvent()
• Backend writing to io.MouseWheel, io.MouseWheelH -> backend should call io.AddMouseWheelEvent()
• (Docking Branch w/ Multi-viewports) Backend writing to io.MouseHoveredViewport-> backend should call io.AddMouseViewportEvent()
• For all new calls to IO functions, the Dear ImGui context should be bound/current.

Gamepad/Nav:

• Backend writing to io.NavInputs[] -> backend should call io.AddKeyEvent()/io.AddKeyAnalogEvent() with ImGuiKey_GamepadXXX values.

What is input queue trickling?

TL;DR; if you submit "Key A went down, then went up, then went down, then went up" during the same Dear ImGui frame, previously events would be missed and it was the application/backend responsibility to submit those events wisely. We moved that responsibility to the core library now. Because most of Dear ImGui code and widgets are reading inputs levels (rather than events), we spread those events over multiple frames when necessary. This makes dear imgui easier to use at low framerate (e.g. 10 FPS).

Backends polling inputs vs handling events

One gotcha is that if you submit events it is preferable to submit then in the right order.
If you transition to new API make sure you are submitting events in the order they come from your OS / platform layer. Normally this should be the default. But because we are punks, our old backends had a tendency to be polling some input state instead of reacting to events, which effectively "lost" their order (e.g. which have changed first, Mouse Position or Mouse Button state?). We have now transitioned all standard backends to be reacting on events rather than polling. If you used your own backend maybe you already behaved better.

Demo

You can see pressed keys in the demo + drawing a small section of a keyboard for no other reason that it being fun.

Reference changes (e.g. to platform backends)

ALWAYS REFER TO FINAL VERSION AS SOME MINOR STUFF MAY HAVE BEEN RENAMED/CHANGED SINCE THE INTERMEDIARY COMMITS. Those references commits are here to help understanding what changed, if you need to update a custom backend.

Core lib

• Core: Rename ImGuiKey_KeyPadEnter > ImGuiKey_KeypadEnter (afffcd5)
• Core: Add extra keys, io.AddKeyEvent() and more (3b66929, bf08c13)
• Core: Add io.AddKeyModsEvent() and updated all Backends accordingly. (790132a, e8172fd)
• Core: Add AddMousePosEvent(), AddMouseButtonEvent(), AddMouseWheelEvent() api + updated all Backends. (b8e56dc)
• Core: Add Input Queue with automatic trickling of fast events. (7374b96, 90a6961, 7ad42ff)
• Core: Add AddKeyAnalogEvent() and support for ImGuiKey_GamepadXXXX. (f33bb99)

Win32 backend

• Win32: update to use io.AddKeyEvent(), add full key map (746c9f7)
• Win32: maintain MouseButtonsDown mask instead of using IsAnyMouseDown() which will be obsoleted (7f8a89c)
• Win32: reorganize to update mouse inputs using WM_MOUSEMOVE/WM_MOUSELEAVE instead of polling so we can submit events in the right order + fallback to provide it when focused but not hovered/captured + update MousePos before Key Modifiers (bf4de2a)
• Win32: update to use AddMousePosEvent(), AddMouseButtonEvent(), AddMouseWheelEvent(). (b8e56dc)
• Win32: add full gamepad support using io.AddKeyEvent(), io.AddKeyAnalogEvent(), stopped writing to io.NavInputs[]. (9f8c599)

GLFW backend

• GLFW: submitting translated keys (100ede5)
• GLFW: update to use io.AddKeyEvent(), add full key map (ecd212c)
• GLFW: reorganize to update mouse inputs using glfwSetCursorPosCallback() (breaking if user install their callback themselves) instead of polling so we can submit events in the right order + fallback to provide it when focused but not hovered/captured + update MousePos before MouseButtons. (200a8f1)
• GLFW: update to use AddMousePosEvent(), AddMouseButtonEvent(), AddMouseWheelEvent(). (b8e56dc)
• GLFW: add full gamepad support using io.AddKeyEvent(), io.AddKeyAnalogEvent(), stopped writing to io.NavInputs[]. (3d85433)

SDL backend

• SDL: update to use io.AddKeyEvent(), add full key map (fe646ea)
• SDL: maintain MouseButtonsDown mask instead of using IsAnyMouseDown() which will be obsoleted (7f8a89c)
• SDL: reorganize to update mouse inputs using SDL_MOUSEMOTION/SDL_WINDOWEVENT_LEAVE instead of polling so we can submit events in the right order + fallback to provide it when focused but not hovered/captured + update MousePos before MouseButtons (98ce013).
• SDL: update to use AddMousePosEvent(), AddMouseButtonEvent(), AddMouseWheelEvent(). (b8e56dc)
• SDL: add full gamepad support using io.AddKeyEvent(), io.AddKeyAnalogEvent(), stopped writing to io.NavInputs[]. (39c3412)

Other backends

• OSX: update to use io.AddKeyEvent(), add full key map (ee436aa)
• OSX: add full gamepad support using io.AddKeyEvent(), io.AddKeyAnalogEvent(), stopped writing to io.NavInputs[]. (5ea47d9)
• Android: update to use io.AddKeyEvent(), add full key map (1797135)
• Allegro: update to use io.AddKeyEvent(), add full key map (1bfe4a7)
• GLUT: update to use io.AddKeyEvent(), add full key map (da1864d)

List of keys

ImGuiKey_None = 0,
ImGuiKey_Tab = 512,             // == ImGuiKey_NamedKey_BEGIN
ImGuiKey_LeftArrow,
ImGuiKey_RightArrow,
ImGuiKey_UpArrow,
ImGuiKey_DownArrow,
ImGuiKey_PageUp,
ImGuiKey_PageDown,
ImGuiKey_Home,
ImGuiKey_End,
ImGuiKey_Insert,
ImGuiKey_Delete,
ImGuiKey_Backspace,
ImGuiKey_Space,
ImGuiKey_Enter,
ImGuiKey_Escape,
ImGuiKey_LeftCtrl,
ImGuiKey_LeftShift,
ImGuiKey_LeftAlt,
ImGuiKey_LeftSuper,
ImGuiKey_RightCtrl,
ImGuiKey_RightShift,
ImGuiKey_RightAlt,
ImGuiKey_RightSuper,
ImGuiKey_Menu,
ImGuiKey_0,
ImGuiKey_1,
ImGuiKey_2,
ImGuiKey_3,
ImGuiKey_4,
ImGuiKey_5,
ImGuiKey_6,
ImGuiKey_7,
ImGuiKey_8,
ImGuiKey_9,
ImGuiKey_A,
ImGuiKey_B,
ImGuiKey_C,
ImGuiKey_D,
ImGuiKey_E,
ImGuiKey_F,
ImGuiKey_G,
ImGuiKey_H,
ImGuiKey_I,
ImGuiKey_J,
ImGuiKey_K,
ImGuiKey_L,
ImGuiKey_M,
ImGuiKey_N,
ImGuiKey_O,
ImGuiKey_P,
ImGuiKey_Q,
ImGuiKey_R,
ImGuiKey_S,
ImGuiKey_T,
ImGuiKey_U,
ImGuiKey_V,
ImGuiKey_W,
ImGuiKey_X,
ImGuiKey_Y,
ImGuiKey_Z,
ImGuiKey_F1,
ImGuiKey_F2,
ImGuiKey_F3,
ImGuiKey_F4,
ImGuiKey_F5,
ImGuiKey_F6,
ImGuiKey_F7,
ImGuiKey_F8,
ImGuiKey_F9,
ImGuiKey_F10,
ImGuiKey_F11,
ImGuiKey_F12,
ImGuiKey_Apostrophe,        // '
ImGuiKey_Comma,             // ,
ImGuiKey_Minus,             // -
ImGuiKey_Period,            // .
ImGuiKey_Slash,             // /
ImGuiKey_Semicolon,         // ;
ImGuiKey_Equal,             // =
ImGuiKey_LeftBracket,       // [
ImGuiKey_Backslash,         // \ (this text inhibit multiline comment caused by backslash)
ImGuiKey_RightBracket,      // ]
ImGuiKey_GraveAccent,       // `
ImGuiKey_CapsLock,
ImGuiKey_ScrollLock,
ImGuiKey_NumLock,
ImGuiKey_PrintScreen,
ImGuiKey_Pause,
ImGuiKey_Keypad0,
ImGuiKey_Keypad1,
ImGuiKey_Keypad2,
ImGuiKey_Keypad3,
ImGuiKey_Keypad4,
ImGuiKey_Keypad5,
ImGuiKey_Keypad6,
ImGuiKey_Keypad7,
ImGuiKey_Keypad8,
ImGuiKey_Keypad9,
ImGuiKey_KeypadDecimal,
ImGuiKey_KeypadDivide,
ImGuiKey_KeypadMultiply,
ImGuiKey_KeypadSubtract,
ImGuiKey_KeypadAdd,
ImGuiKey_KeypadEnter,
ImGuiKey_KeypadEqual,

// Gamepad (some of those are expected to be analog values from 0.0f to 1.0f) ..............// NAVIGATION action
ImGuiKey_GamepadStart,          // Menu (Xbox)          + (Switch)      Start/Options (PS)  // --
ImGuiKey_GamepadBack,           // View (Xbox)          - (Switch)      Share (PS)          // --
ImGuiKey_GamepadFaceUp,         // Y (Xbox)             X (Switch)      Triangle (PS)       // -> ImGuiNavInput_Input
ImGuiKey_GamepadFaceDown,       // A (Xbox)             B (Switch)      Cross (PS)          // -> ImGuiNavInput_Activate
ImGuiKey_GamepadFaceLeft,       // X (Xbox)             Y (Switch)      Square (PS)         // -> ImGuiNavInput_Menu
ImGuiKey_GamepadFaceRight,      // B (Xbox)             A (Switch)      Circle (PS)         // -> ImGuiNavInput_Cancel
ImGuiKey_GamepadDpadUp,         // D-pad Up                                                 // -> ImGuiNavInput_DpadUp
ImGuiKey_GamepadDpadDown,       // D-pad Down                                               // -> ImGuiNavInput_DpadDown
ImGuiKey_GamepadDpadLeft,       // D-pad Left                                               // -> ImGuiNavInput_DpadLeft
ImGuiKey_GamepadDpadRight,      // D-pad Right                                              // -> ImGuiNavInput_DpadRight
ImGuiKey_GamepadL1,             // L Bumper (Xbox)      L (Switch)      L1 (PS)             // -> ImGuiNavInput_FocusPrev + ImGuiNavInput_TweakSlow
ImGuiKey_GamepadR1,             // R Bumper (Xbox)      R (Switch)      R1 (PS)             // -> ImGuiNavInput_FocusNext + ImGuiNavInput_TweakFast
ImGuiKey_GamepadL2,             // L Trigger (Xbox)     ZL (Switch)     L2 (PS) [Analog]
ImGuiKey_GamepadR2,             // R Trigger (Xbox)     ZR (Switch)     R2 (PS) [Analog]
ImGuiKey_GamepadL3,             // L Thumbstick (Xbox)  L3 (Switch)     L3 (PS)
ImGuiKey_GamepadR3,             // R Thumbstick (Xbox)  R3 (Switch)     R3 (PS)
ImGuiKey_GamepadLStickUp,       // [Analog]                                                 // -> ImGuiNavInput_LStickUp
ImGuiKey_GamepadLStickDown,     // [Analog]                                                 // -> ImGuiNavInput_LStickDown
ImGuiKey_GamepadLStickLeft,     // [Analog]                                                 // -> ImGuiNavInput_LStickLeft
ImGuiKey_GamepadLStickRight,    // [Analog]                                                 // -> ImGuiNavInput_LStickRight
ImGuiKey_GamepadRStickUp,       // [Analog]
ImGuiKey_GamepadRStickDown,     // [Analog]
ImGuiKey_GamepadRStickLeft,     // [Analog]
ImGuiKey_GamepadRStickRight,    // [Analog]

// Keyboard Modifiers
// - This is mirroring the data also written to io.KeyCtrl, io.KeyShift, io.KeyAlt, io.KeySuper, in a format allowing
//   them to be accessed via standard key API, allowing calls such as IsKeyPressed(), IsKeyReleased(), querying duration etc.
// - Code polling every keys (e.g. an interface to detect a key press for input mapping) might want to ignore those
//   and prefer using the real keys (e.g. ImGuiKey_LeftCtrl, ImGuiKey_RightCtrl instead of ImGuiKey_ModCtrl).
// - In theory the value of keyboard modifiers should be roughly equivalent to a logical or of the equivalent left/right keys.
//   In practice: it's complicated; mods are often provided from different sources. Keyboard layout, IME, sticky keys and
//   backends tend to interfere and break that equivalence. The safer decision is to relay that ambiguity down to the end-user...
ImGuiKey_ModCtrl,
ImGuiKey_ModShift,
ImGuiKey_ModAlt,
ImGuiKey_ModSuper,

ImGuiKey_COUNT,                 // No valid ImGuiKey is ever greater than this value

Questions?

(Will be updated over time)

Q: It is ok to call io.AddXXXEvent() functions every frame with unchanged data?
A: Yes.

[RT222]

Awesome changes! I'm in the process of updating my custom backend to this API, it will integrate much better in my callback based event system.

One thing though, If I understand correctly, ImGuiKey_Menu is supposed to represent this key :

If yes, then the SDL backend should use SDLK_APPLICATION instead of SDLK_MENU.

[thedmd]

Yes, this is a bug. It should be SDLK_APPLICATION.

[Flix01]

[OT] Not sure if somebody's interested... but when I saw the new "keyboard rectangle" in imgui_demo.cpp, I've soon tried to "extend" it. Here is what I've done so far [gist link].

show/hide screenshot

[thedmd]

I love it. I'm also so picky creature that I differentiate between ANSI and ISO layouts. :)

[ocornut]

IMPORTANT: If you have updated backends between January 10 and January 27
We have made a breaking change to the new IO api c906c65
io.AddKeyModsEvent() was replaced with normal calls to io.AddKeyEvent() with ImGuiKey_ModXXX values.

All backends have been updated. This only applies to you if you updated standard or custom backend between January 10 and January 27. If you use standard backends, simply update. If you use a custom backend change code above.

[Flix01]

...as far as I can see, in the glfw backend, when we "translate untranslated" keys, numbers in the keypad (= numpad) are converted to (or detected as) normal numbers of the main keyboard block (...well at least in my custom glfw backend).
If I comment out the "translation code" they are detected correctly.
This seems to affect only numbers (other keypad keys work correctly).
It's OK for me, but I'd like to know if this is the expected behavior...

[ocornut]

Not expected behavior; will fix soon! Thank you!

[PeterJohnson]

What's the recommended way to remap keys in the new API? From what I can tell, it looks like doing so after this change requires customizing the backend, whereas previously that could be done simply by modifying io.KeyMap to point to a different io.KeysDown?

[PathogenDavid]

@PeterJohnson What/why were you remapping before? Everything in KeyMap was used for pretty well-established UI behaviors so it seems odd to me that you'd want to customize them in the first place.

[PeterJohnson]

For my use case, there is another application running on the system that has a global (system-wide) listener on the spacebar and Enter keys. While it's relatively easy to avoid the use of spacebar for numeric entry, the use of Enter for committing changes to InputText-based controls is hardcoded, so the easiest workaround was to provide users a way to remap another key (e.g. right ctrl) to the Enter key. Is not acceptable for my use case to commit the value for every keypress.

I agree this is a relatively unusual use case and modifying the backend (GLFW in my case) to support key remap is probably the right answer.

[PathogenDavid]

Ah yeah, the driver station? 😅 That makes sense.

I haven't tested it super thoroughly, but this might be good enough for your use-case:

//...
ImGui::NewFrame();

if (ImGui::IsKeyPressed(ImGuiKey_RightCtrl))
{
    io.AddKeyEvent(ImGuiKey_Enter, true);
    io.AddKeyEvent(ImGuiKey_Enter, false);
}

//...

Note that this will break using right control for shortcuts. (IE: Right Control + V to paste.)

---

If you want to truly remap it you could install your own glfwSetKeyCallback callback and handle the translation there.

Another alternative would be to manipulate ImGuiIO::KeysData instead, but then you're mucking with Dear ImGui's internal state which is obviously not super ideal.

(Or like you said you could just modify the GLFW backend, but then you have to deal with merging changes all the time.)

(If you want me to elaborate on one of these alternatives it would be better to open a separate issue.)

[caxapexac]

@ocornut its really cool update but what about key/mouse intercompatibility #4858 (comment) ?

[ocornut]

We discovered a backward-compatibility regression in 1.87

• Direct legacy reads from io.KeysDown[] won't work with new backends.
• And in particular io.KeysDown[GetKeyIndex[XXX]) will buffer overflow in old and new backends.

Reads via IsKeyDown() function are working.

We'll see if we can come up with a fix. But changing your app to use IsKeyDown() will fix things.

[ocornut]

We discovered a backward-compatibility regression in 1.87

Fixed by commit 5659db5.

[rickyviking]

@ocornut I think that the doc about porting (for instance)

Backend writing to io.MousePos -> backend should call io.AddMousePosEvent()

should mention clearly the added constraint

"Can only add events to current context."

which is now only present as source code assertion
I have a multi-context application which worked OK with the legacy io.MousePos API without the need to switch ImGui::SetCurrentContext(), while switching current context is needed after moving to io.AddMousePosEvent()

[ocornut]

should mention clearly the added constraint

Thanks for the notice. You are right and I'll add a few comments here and there but I'm not sure they have much value in comparison with the existing assert which makes this explicit for the few users who would be affected.