Skip to content

Mac: Keyboard and mouse controls for Tears of the Kingdom - #330

Open
sujitv19196 wants to merge 1 commit into
seedds:mainfrom
sujitv19196:totk-keyboard-mouse
Open

sujitv19196 wants to merge 1 commit into
seedds:mainfrom
sujitv19196:totk-keyboard-mouse

Conversation

@sujitv19196

@sujitv19196 sujitv19196 commented Oct 26, 2026 •

Copy link
Copy Markdown

Keyboard and mouse controls for Tears of the Kingdom on the Mac

The Mac front end mapped the keyboard onto a controller layout and had no mouse camera, which makes Tears of the Kingdom hard to play without a controller. This PR adds keyboard and mouse controls in which the mouse turns the camera by angle instead of pretending to be the right stick. They are opt in (DM_KBM=1): most players use a controller, so the game still starts with the classic keys.

It is one commit on top of current main, so it can also be cherry-picked onto other forks of the repo: git cherry-pick <sha>.

Minimal and kept to its own files

The feature lives almost entirely in three new files. It touches shared code only through a few one-line hooks, so it stays easy to rebase and to carry on other branches.

File Change
dm/host/front_kbm_mac.m / .h New. The whole host half: the PC layout, mouse capture, the mouse camera's block in the state page, the DM_KBM / DM_MOUSE_SENS / DM_MOUSE_INVERT_Y settings and the build gate. It installs its own event monitor for clicks, the scroll wheel and modifier keys.
dm/games/0100F2C0115B6000/kbm_patches.py New. The rewriter's camera stubs, with the encoders they need defined locally.
dm/host/front_mac.m +6 / −1: an include and four hooks (kbm_start, kbm_input in push, kbm_key in on_key, kbm_release). Nothing is renamed: kbm_start takes pointers to the front end's own key state.
dm/host/Makefile +2: SRC += $(sort $(wildcard front_*_mac.m)), on its own line, so a new front end file does not touch the SRC line.
dm/games/0100F2C0115B6000/patches.py +11, appended at the end: a 1.4.3 FRAME_STEP entry that runs the existing frame step stub and then the mouse camera.
Game note, docs/env_switches.tsv A "Keyboard and mouse" section and three rows.
docs/agents/game-patches.md Generated (dm/tools/docs_patches.py). If it conflicts, re-run the script rather than merging by hand.

Compared with the earlier version of this PR, loader.c, dm.h, hle.h, patch_nso.py and nso_asm.py are no longer changed:

  • The mouse functions that were in loader.c and hle.h are now static in front_kbm_mac.m.
  • The MOUSE_CAMERA fixup, its module_t field and the patch_nso.py plumbing are gone. The host gates on the build ID through the existing dm_main_build(), as AGENTS.md describes for host code tied to fixed addresses.
  • The new encoders (BC, CBZW, LDRS, STRS, STRW) live in kbm_patches.py. They can move to nso_asm.py if a second user appears.

How the mouse camera works

  • The stubs: the rewriter patches each camera's per-frame update (kbm_patches.py, mouse_stub). At that point, a stub:
    • takes the mouse counts the host summed;
    • adds them to that camera's yaw and pitch, within the game's own bounds;
    • runs the instruction it replaced.
  • Cameras covered: Chase, Aiming (bow, throws), LockOn (Target and Ultrahand's view) and SpecialPower (the other abilities). In Recall the stub turns SpecialPower's base yaw.
  • Ultrahand: two stubs in its move turn and raise a held object.
  • Ascend: its aiming node stamps the state page, so the mouse buttons become A / B there.
  • Movement nothing takes: counts that no camera takes within 1/8 s are dropped.
  • Where the state lives: at +0x800 in the frame step state page. That keeps it clear of the frame step's own fields at the start of the page: main now uses +0x40 for dynamic resolution and +0x48..+0x98 for the screen shape, which the earlier version of this PR would have overlapped.
  • Which build: the stubs patch fixed addresses found on 1.4.3, so they run only on that build (277178b7…). There the frame step comes from a FRAME_STEP entry that calls the same stub() at the six functions' 1.4.3 addresses, where MATCH_FRAME_STEP finds them. Every other build keeps the matched frame step alone.
  • Details: the game note's "Keyboard and mouse" section and the kbm_patches.py comment list every address and field.

The layout

Input Normal play Context
WASD move right stick while a D-pad key is held (quick menus)
Mouse camera right stick while Q is held (ability wheel); moves a held Ultrahand object
Left click Y (attack) A in menus and abilities, X in quick menus, fire / throw in an aim mode
Right click ZL (shield) B in menus, quick menus, the ability wheel, while aiming
Middle click R3
Space / Shift / Ctrl or C X (jump) / B (run) / L3 (crouch)
E or Return A R in menus and with an object held (rotate), Y in quick menus
Q L (abilities) ZL with an object held
F / G bow mode / throw mode (toggle) left click fires or throws; the key again or right click leaves through the game's own ZR+B / R+B
R D-pad up shakes the right stick with an object held (pulls stuck objects apart; a mouse shake does too)
1 2 3 4 D-pad right / left / up / down (weapons, shields, abilities, whistle) the scroll wheel picks in them, one item per notch
Scroll wheel with an object held: D-pad down / up (push / pull); horizontal scroll D-pad left / right
Esc / Tab or M + / −
  • Opting in: DM_KBM=1 (for example a line in ~/Library/Application Support/nxnative/0100F2C0115B6000/env.txt) turns the keyboard and mouse controls on. Without it, the game starts with the classic keys, as before. A click in the window captures the mouse, and ⌘Tab releases it.
  • Environment switches: DM_KBM=1 turns the controls on. DM_MOUSE_SENS sets the sensitivity (degrees per count, 0.1 by default). DM_MOUSE_INVERT_Y=1 inverts the vertical axis. All three are listed in docs/env_switches.tsv.

The default and other games are unchanged

  • Without DM_KBM=1: Tears of the Kingdom plays as on main. The stubs are in, but with a sensitivity of 0 they run only the instruction they replaced.
  • Other games: the controls can switch on only for the 1.4.3 main build with the frame step state page and DM_KBM=1. Every other game keeps the classic keys and the mouse as the touchscreen.
  • iOS: the _mac suffix keeps the files out of the iOS build.

Checked

  • Builds: dmhost builds for Tears of the Kingdom with no new warnings. dm/ios/tools/compile_check.py passes (265 files).
  • Tests: test_patches, test_docs and test_env_switches pass.
  • In game: the controls were played on a MacBook Pro (M2 Max, macOS 26.6) before this restructure: the camera, aiming, Ultrahand, menus, quick menus, the ability wheel, and the bow and throw toggles. The restructure moved code into the new files and moved the mouse state to +0x800. It has not been played in game again yet.
  • Not tested: the scope, lock-on aiming, and the horse and climbing cameras.

🤖 Generated with Claude Code

@sujitv19196
sujitv19196 marked this pull request as draft October 26, 2026 19:41
@sujitv19196
sujitv19196 marked this pull request as ready for review October 26, 2026 23:28
@sujitv19196 sujitv19196 changed the title Keyboard and mouse controls for Tears of the Kingdom on the Mac Mac: Keyboard and mouse controls for Tears of the Kingdom Oct 26, 2026
@github-actions github-actions Bot added host The Horizon host: SVCs, services, threads, memory (dm/host) rewriter The NSO rewriter and per-build patches mac The Mac front end and app (dm/mac, front_mac.m) tools Tools and scripts (dm/tools, scripts) docs Documentation game: 0100F2C0115B6000 The Legend of Zelda: Tears of the Kingdom labels Nov 3, 2026
ltrdpper pushed a commit to ltrdpper/BananaNative that referenced this pull request Nov 16, 2026
…per-frame budget' (seedds#330) from 1982tito/BananaNative:prune-budget into android
… Mac

DM_KBM=1 turns on keyboard and mouse controls for Tears of the Kingdom 1.4.3
on the Mac: the mouse turns the camera by angle (Chase, Aiming, LockOn and
Ultrahand, SpecialPower and Recall), moves objects held by Ultrahand, and
the keyboard takes a PC layout (quick menus on 1-4 and the scroll wheel,
the ability wheel pointed with the mouse, bow and throw toggles on F and G,
Ascend's aiming on the mouse buttons). Without DM_KBM the game and every
other game run as before.

The feature lives in files of its own:
- dm/host/front_kbm_mac.m / .h: the layout and the mouse camera's host half
  (its state page block, settings and build gate); front_mac.m calls it
  through four one-line hooks, and the Makefile takes front_*_mac.m by
  wildcard.
- dm/games/0100F2C0115B6000/kbm_patches.py: the camera stubs, with their
  state at +0x800 in the frame step state page, clear of the frame step's
  own fields; patches.py runs them from a 1.4.3 FRAME_STEP entry beside
  the same frame step stub the matched rule places.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0100F2C0115B6000 The Legend of Zelda: Tears of the Kingdom host The Horizon host: SVCs, services, threads, memory (dm/host) mac The Mac front end and app (dm/mac, front_mac.m) rewriter The NSO rewriter and per-build patches tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant