EOS R (DIGIC 8): MMU ROM-patching + serial-flash RE [WIP] - #281
Draft
jed2nd wants to merge 34 commits into
Draft
Conversation
The EOS R (and other Digic 7/8 cameras) have no recorded MPU init spells, so they fall back to the generic set and cannot boot in qemu-eos. Add a capture facility for the R that records Canon's MPU send/recv handshake during boot, in the dm-spy log format that extract_init_spells.py consumes. The capture rides on my_task_dispatch_hook (installed in boot_pre, runs safely from early boot). It does only a safe pointer-compare each dispatch until Canon's InitializeIntercom publishes the genuine recv callback (mpu_recv_cbr == &mpu_recv), then swaps in a logger that drains both MPU ring buffers per received message and timestamps them with the microsecond timer. Creating a task this early is unsafe (no platform does it) and hangs the boot, hence the dispatch-hook approach. mpu_capture_dump(), wired to the Debug menu's "Don't click me!" item, writes ML/LOGS/MPULOG.TXT plus ML/LOGS/INTERCOM.TXT (the live MPU MMIO register addresses from the intercom struct, needed for the qemu R model). Guarded by CONFIG_R; uses only stubs the R already has.
Add an is_camera("R","1.8.0") branch with the serial-flash function
stubs RE'd from ROM0 via Ghidra (SF_CreateSerial 0xE03C002A,
readSerialFlashWithQuad 0xE03C066A for large reads, SF_Destroy
0xE03C1EE4; 16MB flash). Also add a small crash-breadcrumb marker so a
hang during dumping leaves the last-reached step on the card.
Note: reading serial flash from a running ML task currently deadlocks
on the R because the SIO/SPI channel is in active use by Canon
firmware; this is recorded for future work on the serial-flash path.
Image capture is gated off on the R (CONFIG_IMAGE_CAPTURE_NOT_WORKING)
due to a suspected null-pointer crash "if it fails to AF lock". Add a
breadcrumb-instrumented probe to find out where it breaks.
Result on hardware: take_a_pic(AF_DONT_CHANGE) runs lens_take_picture
cleanly end to end (A->Z) and the camera captures and saves a real
image. So the crash is confined to the AF setup path; capture itself
works when AF button reassignment is skipped.
One remaining gap: lens_info.job_state never becomes non-zero
("K: job_state timeout"), so ML doesn't know the exposure happened.
This is a property-wiring issue to chase next (also explains the long
"saving" spinner at power-off).
- lens.c: SHOOT_BC breadcrumbs through lens_take_picture (CONFIG_R),
written cumulatively to ML/LOGS/SHOOT.TXT.
- debug.c: "Take test pic (no AF)" Debug item -> take_a_pic(AF_DONT_CHANGE).
mpu_capture_arm required mpu_recv_cbr == &mpu_recv, but at runtime Canon's recv callback can be a different (still valid) ROM handler, so the install silently failed and the capture only snapshotted the ring buffers at dump time. Hook whatever valid ROM handler is currently installed instead, so a runtime capture (arm -> act -> dump) records continuously.
…ister a half-press on R)
ML programmatic capture on the EOS R issued call("Release"), which takes the
picture but leaves the shooting job unfinalized -> the camera hangs on
"saving..." at power-off (the SRM-buffer facet). The physical shutter was
always clean.
SetEventIrRemoteReleaseBtn (0xE0190214) drives the same CameraConductor
remote-release pipeline a wireless remote / the physical shutter uses, which
finalizes the job properly. lens_take_picture now uses it on the R
(1 = press/SW2-on, 0 = release) instead of call("Release").
Verified on hardware: captures a picture AND powers off cleanly, no hang.
Also adds a Debug menu probe ("Take pic (IR remote)") used to validate it.
Port ML's MMU-based ROM patcher to the EOS R, modeled on the M50. This
unlocks ROM patching on the R generally (most ML features depend on it),
and specifically lets us hook ReadBlockSerialFlash to dump the TUNE
serial-flash region the qemu boot stalls without.
- internals.h: +CONFIG_RPC, +CONFIG_MMU
- features.h: +CONFIG_SGI_HANDLERS, +CONFIG_MMU_REMAP
- consts.h: CANON_ORIG_MMU_TABLE_ADDR=0xe0000000 (ROM start),
DRYOS_SGI_HANDLERS_PTR=0x402c (M50 value, TODO verify on R)
- include/platform/mmu_patches.h: created (empty patch arrays)
- stubs.S: 6 reverse-engineered Canon stubs:
change_mmu_tables 0xE0554466 (verified: TTBR0/1, CONTEXTIDR<-cpu, TLBIALL, SCTLR.M=1)
memcpy_dryos 0xE0563064 (thumb veneer -> ARM aeabi memcpy 0xE0563068, verified)
dcache_clean_multicore 0xE008E23E (low conf; R folds broadcast into dcache_clean@0xc1100730)
_request_RPC 0xDF002BAA (from the 0xDF000000 AMP region, dumped from running qemu)
clear_RPC_request 0xDF002BCA
task_create_ex 0xE05172FA (= task_create for now; cpu_id param TODO)
- init.c/debug.c: passive ReadBlockSerialFlash capture probe (TUNE dump groundwork);
its patch_hook_function path is cache-patch-only, so MMU_REMAP bodies route to the
unavailable branch -- the hook moves to mmu_patches.h once remap is validated.
Builds clean for R.180. Runtime validation (does the remap boot? is
DRYOS_SGI_HANDLERS_PTR correct? do the RPC stubs behave?) needs a camera test.
… MMU remap Install a read-only function_hook_patch (early_code_patches[]) that replaces ReadBlockSerialFlash @0xE03C10C4 with sfread_wrapper. The wrapper calls the real read through sfread_tramp (a naked-thumb trampoline that replays RBSF's 8 overwritten prologue bytes relocated, then bx to RBSF+8), then copies the just-read bytes out when addr is in the TUNE region [0xF09C0000,0xF0A00000). Replaces the dead patch_hook_function-based probe (DIGIC 4/5 only). The detour patches ICU code in MMU-remapped RAM, never the flash: no write/erase/program/ install path. Capture lands in sfread_tune_buf, dumped to ML/LOGS/TUNE.BIN via the Debug menu. Hook target verified by qemu gdb trace: the firmware calls RBSF with the absolute address 0xF09C0000 (len=0x50 combo-package header) during PROPAD_CreateFROMPropertyHandle, so the range check is correct.
…u-validated) Root cause (found via camera-free qemu debug -- ML now boots in qemu-eos for the R): generic_mmu_space had __attribute__((aligned(0x10000))), which forced a large linker padding hole in BSS (removing the 86KB buffer freed 257KB of footprint). That bloated ml_reserved_mem (_bss_end - user_mem_start) past the R's tight user_mem budget, so my_create_init_task (boot-d678.c) aborted into an infinite loop -> solid red LED. It was never an mmu_init logic bug. Fix: drop the aligned(0x10000) attribute and add MMU_PAGE_SIZE of slack. calc_mmu_globals already aligns the L1 table / 64KB page WITHIN the buffer via start_adjust, so the buffer needs no external alignment. Footprint 590KB, 542KB user_mem left (> Canon's 512KB floor). Also: mpu_cap_buf 128K->4K (vestigial -- MPU spell capture is complete) for headroom. qemu-validated: memory fix + full remap (table copy, M50 L1 fixup, SGI registration, TTBR switch) all boot to WINSYS (285 serial lines). The only qemu fault is dcache_clean_multicore (R inter-core regs 0xc1100730/0xe00c7b28 the incomplete R model doesn't emulate; stub verified correct -> works on hardware). Hardware camera test pending. SF-capture detour temporarily off + sfread_tune_buf shrunk: the 256KB static buffer also overruns the budget; capture must stream to card (FIO append) -- follow-up.
…KB window) Now the MMU remap memory fix is in (a2c38d3), re-enable the early_code_patches[] detour that replaces ReadBlockSerialFlash @0xE03C10C4 with sfread_wrapper to capture TUNE. Fix a real OOB bug: the capture clamped against SF_TUNE_SIZE (256KB) while writing into the smaller sfread_tune_buf -> any TUNE read past the buffer corrupted memory. Now clamps to SF_TUNE_BUFSZ and skips off>=BUFSZ reads. Buffer set to 16KB (the full 256KB can't be static BSS without overrunning the R user_mem budget); captures the start of TUNE (firmware reads from offset 0). sfread_tune_captured tracks the high-water mark. If SFREAD.TXT shows reads past 16KB, switch to a Canon AllocateMemory dynamic buffer (_AllocateMemory @0xE0552814). Footprint 598KB (fits, ~10KB margin). qemu can't exercise the detour firing (RBSF only called with the SF struct installed -> camera-only) but the build boots + the patch installs. Camera test: boot 0bf5088f (bare remap) first, then this (9bb8b958) -> Dump SF reads -> TUNE.BIN.
…onfirmed) CAMERA-CONFIRMED: the MMU remap WORKS on the R -- ML boots with full UI. The earlier red LED was dcache_clean_multicore (Canon inter-core cpu1 cache-broadcast @0xE008E23E, writes 0xc1100730): it crashes when called this early in mmu_init (inter-core HW not yet up). Bisected on hardware -- build with it red-LEDs, build without it boots ML fully. It is also unnecessary on the R: the remap is cpu0-only (CONFIG_INIT1_HIJACK is not defined, so cpu1 never takes the remapped tables), and dcache_clean already covers cpu0 coherency. Override the Canon stub with a no-op in function_overrides.c (stub commented in stubs.S to avoid a duplicate symbol) -- covers all 5 callers including the early_code_patches detour-install path. Re-enabled the RBSF detour. First DIGIC 8 R with hardware-working ROM patching. Next: camera-test the TUNE capture.
boot_pre_init_task now stores mmu_init()'s return into sfread_diag_mmu_ret so the SFDIAG.TXT dump can report whether the remap fully installed (it does return 0 on hardware). Diagnostic-only; no behavioural change.
Mirrors the shutter RE playbook for ISO. PROP_ISO is deny-listed and prop_get_value isn't stubbed on the R, so both probes register a runtime PROP_ISO slave (prop_register_slave/_prop_cleanup) -- safe post-boot, can't hit the early-boot crash the deny list guards against. - ISO read map: logs PROP_ISO value+len per change while dialing ISO, plus a delivery counter to confirm whether the R broadcasts ISO dial changes. -> ML/LOGS/ISOMAP.TXT - ISO write sweep: writes raw ISO codes 72..120 (ISO 100..6400) via the raw Canon stub at the live length (no PROP_LEN red box), watch the ISO display. -> ML/LOGS/ISOSWEEP.TXT Both are Debug-menu, boot-safe (no property_whitelist.h changes); diagnostics to be stripped before any PR.
ISOMAP.TXT from the camera revealed the R's PROP_ISO is a 4-byte value with the ISO code in BYTE 1 (val = code<<8), code 15 = ISO 100, +3 per stop (1/3-stop units) -- the same high-byte trick the shutter uses. v1 wrote the code in byte 0, which the R ignores, so the display stayed at ISO 100. v2 sweeps codes 15..33 (ISO 100..6400) in byte 1 to confirm writes drive the real ISO. The read map also showed the R DOES broadcast PROP_ISO on the dial (unlike the shutter dial), so ISO read works once the handler decodes byte 1.
Camera data (ISOMAP/ISOSWEEP) confirmed the R's PROP_ISO is 4 bytes with the ISO code in byte 1 (val = code<<8), code 15 = ISO 100, +3 per stop. The R also broadcasts PROP_ISO on the dial, so read works once the handler decodes byte 1. - lens.c: ml_raw_to_r_iso / r_iso_to_ml_raw (ML APEX raw <-> R byte-1 code), r_set_rawiso; PROP_HANDLER(PROP_ISO) decodes byte 1 on R; prop_set_rawiso routes through r_set_rawiso on R. - property_whitelist.h: PROP_ISO out of prop_handler_deny[] (read + length cache) and into prop_write_allow[] (write). A runtime PROP_ISO slave delivered fine, so the boot handler is expected safe. - debug.c: 'ISO set test (real)' probe drives the production lens_set_rawiso path and logs the decoded read-back (-> ISOSET.TXT). Awaiting camera validation (boot + read tracks dial + production write).
SFDIAG proved the passive RBSF detour is live on cpu0 but fires 0 times on a normal boot -- Canon reads the property-DB serial flash during its own pre-ML init, before mmu_init installs the detour. So perform the read ourselves: a new Debug item 'SF active read TUNE' calls sfread_tramp (the validated clean RBSF) directly on cpu0 for 0xF09C0000, poisons the buffer first (0xAA) to tell a real read from a no-op, and dumps TUNE.BIN + SFACTIVE.TXT (return, non-blank count). Read-only flash I/O, no brick risk. Also bumped the SFDIAG buffer 300->420 so its last two lines stop truncating.
Last exposure prop to close out Path 1. PROP_APERTURE isn't deny-listed (read handler registers) but its R encoding is unverified -- shutter used the hi-byte, ISO byte 1, so aperture may not be plain byte 0 either. Two boot-safe Debug probes (runtime PROP_APERTURE slave, same pattern as ISO): - Aperture read map: logs PROP_APERTURE value+len per change while dialing, plus the lens raw_aperture min/max range. -> ML/LOGS/APMAP.TXT - Aperture write test: writes f/8 in 4 byte-layout/length variants via the raw Canon stub (PROP_APERTURE isn't in prop_write_allow), watch the f-number. -> ML/LOGS/APFMT.TXT
The aperture-probe build (_bss_end 0x144900) overran the R's user_mem budget by ~20 bytes -> my_create_init_task aborts -> solid red LED before boot (the same failure mode as the MMU footprint overrun). It booted intermittently because it sat right at the edge. The passive SF-read log fires 0 times on the R, so its 8KB buffer is effectively dead weight; shrinking it to 2KB reclaims 6KB and restores ~6KB of margin (_bss_end 0x143100, below the rock-solid ISO build). Budget: ml_reserved = _bss_end - user_mem_start(0xb0078); remaining = user_mem_len(0x114884) - ml_reserved must stay >= MINIMUM_USER_MEM_LEFT(0x80010).
Camera data (APMAP/APFMT) confirmed the R's PROP_APERTURE is 2 bytes with the Av code in byte 1 (val = code<<8), 3 per stop: code 9=f/2.8, 18=f/8, 27=f/22 (= round((ml_raw-8)*3/8)). Same byte-1 / *3/8 family as PROP_ISO; writing byte 1 drove the lens and the R clamps to the lens range itself. - lens.c: ml_raw_to_r_aperture / r_aperture_to_ml_raw + r_set_rawaperture (write code<<8, len 2, no min/max coerce -- raw_aperture_min/max are 0 on the R, the R clamps); PROP_HANDLER(PROP_APERTURE) decodes byte 1; prop_set_rawaperture routes through r_set_rawaperture on R. - property_whitelist.h: PROP_APERTURE added to prop_write_allow (read handler was already registered -- not deny-listed). - debug.c: 'Aperture set test' probe drives the production lens_set_rawaperture path and logs the decoded read-back (-> APSET.TXT). Awaiting camera validation.
These were one-time reverse-engineering tools to map the R's exposure-property encodings. All three props (shutter, ISO, aperture) are now validated and the features are on PR reticulatedpines#280, so the probes have served their purpose. Removing them (scale map / write sweep / format test / dial read / hi-byte map / read map / set test, plus the runtime PROP_ISO and PROP_APERTURE probe slaves) drops _bss_end from 0x143600 to 0x13f600 -- ~16KB reclaimed, restoring ~20KB of user_mem budget margin (the probe build had been ~20 bytes over -> red LED). Kept: the SF capture/active-read probes (Path 2) and the other capture probes.
RE of the serial-flash driver (ROM0) explains the ret=17:
- RBSF @0xE03C10C4 returns 17 ("SerialFlash is not install") when the handle at
*(0xE03C14B4)+16 (struct 0x62bc, handle 0x62cc) is 0.
- InstallSerialFlash sets that handle to 1 (str r6,[r4,reticulatedpines#16], r6=1) + 32MB mode.
- UninstallSerialFlash sets it back to 0 -- and does NOTHING else; the SIO
controller config from boot stays intact.
So Canon's teardown is just a flag. v2 re-pokes handle=1 before calling
sfread_tramp; if the leftover SIO state is still valid the read should now
succeed. SFACTIVE.TXT reports handle before/after + the non-blank byte count.
Also fixed the %u -> %d/%x (ML printf has no %u, which is why v1 logged '=u').
… dump Poking handle=1 then calling RBSF crashed/rebooted the camera: re-setting the flag isn't enough. InstallSerialFlash also reconfigures the SIO registers, and the SF string pool has powerDownSerialFlash/DeepPowerDown -- so post-boot the SF chip is almost certainly in deep power-down with its SIO controller torn down, and RBSF on it faults. Replaced the crashing poke+read with a READ-ONLY dump of the SF driver struct (RAM @ 0x62bc) so we can see exactly what state Canon left, to ground a proper Install+ReleaseDeepPowerDown re-init later. No RBSF, no poke, no crash. SF driver fully mapped in PATH2_QEMU_STATE.md.
The 23 CONFIG_R diagnostic menu items (capture tests, ROM/EEPROM/FROM/SF dumps, the Path 2 probes, etc.) were one-time reverse-engineering tools. With all the features validated and shipped, they're just clutter in the Debug menu. Removed the whole #ifdef CONFIG_R menu block; the linker then GC's the now-unreferenced probe handlers, dropping _bss_end 0x13f700 -> 0x127a00 (~97KB reclaimed). The standard ML debug items and the Bootflag submenu are untouched.
The CONFIG_R diagnostic wrote ML/LOGS/INTRV.TXT once a second whenever the intervalometer was enabled -- ~20k file writes over a 6-hour run, racing the camera's own JPG saves to the same card (the exact FIO-during-capture pattern that deadlocks). It was a one-time bring-up probe to see why firing stalled; with the firing path validated it's now just a reliability liability for long unattended runs. Removed. (Display-off between frames is already handled by idle_force_powersave_now(); the firing path is take_a_pic -> the validated IR-remote capture.)
…xposure RE)
Foundation for an adaptive-exposure ('holy grail') intervalometer: we have
exposure read/write, but need a brightness signal on the R. The ambient-light
sensor (lightsensor_raw_value) isn't compiled in for this body, so the probe
logs the two through-the-lens candidates -- PROP_BV (via a runtime slave) and
lens_info.ae -- plus the current Tv/Sv/Av, once every ~2.5s for ~3.5 min, with a
brief half-press each line to force metering. Run it while the light changes
(dimmable lamp at the lens, or dusk/dawn) and see which signal tracks.
-> ML/LOGS/BRIGHT.TXT
…ightness) v1 (PROP_BV via M-mode half-press) never updated -- the R doesn't surface brightness in M mode, and lens_info.ae is just exposure compensation (stayed 0). v2 watches the AUTO/metering properties the camera computes when it actually meters: PROP_SHUTTER_AUTO (the shutter it picks in Av mode -- the prime candidate), PROP_ISO_AUTO, PROP_APERTURE_AUTO, PROP_LV_BV, PROP_BV, via one runtime multi-property slave. Run in Av mode + vary the light; whichever tracks is the adaptive-exposure signal. -> ML/LOGS/BRIGHT.TXT
… Av) v2 showed PROP_SHUTTER_AUTO/ISO_AUTO/AP_AUTO are never delivered on the R (seq=0), PROP_LV_BV sits at the not-in-LiveView sentinel 0x7FFF, and lens_info Tv never moved in Av -- because ML's PROP_SHUTTER handler deliberately ignores Av/P mode (shutter is camera-chosen there). So the metered shutter the camera picks in Av DOES arrive via PROP_SHUTTER, but the handler drops it and the v2 slave was watching PROP_SHUTTER_AUTO instead. v3 watches PROP_SHUTTER directly via the slave and logs raw hex (value is in byte 1, the usual high-byte quirk). Run in Av + vary light; SH's byte 1 should track the scene.
v1-v3 proved a software half-press yields zero metering updates on the R (every prop frozen at seq<=1). Last property-based avenue: PROP_LV_BV, which meters continuously off the LiveView sensor feed. v4 drops the fake half-press and just polls the slaves every ~1.6s for ~2.5min -- run it IN LIVEVIEW with the light varying. Also implicitly tests whether ML LiveView even comes up on the R. If LV_BV tracks, that's the adaptive signal; if not, property-based metering is a dead end on this port and we fall back to a scheduled ramp.
…image The metering properties only surface the user's dial settings on the R, not a metered brightness -- so all the half-press probes were structurally doomed. New approach (how holy-grail apps actually work): in LiveView, average the luma of the live YUV422 image via get_yuv422_vram() -- that IS the scene brightness, off the sensor feed, independent of the meter. Logs avgY (0-255) + the LV buffer w x h (0x0 => LiveView isn't up) + PROP_LV_BV as backup. Run in LiveView, vary the light; avgY should track it directly. Raw histogram would be cleaner but CONFIG_RAW isn't enabled on the R; YUV luma needs no new infrastructure.
jed2nd
force-pushed
the
eosr-mpu-spell-capture
branch
2 times, most recently
from
June 19, 2026 22:01
b0efa2f to
1b7c68f
Compare
Owner
|
Very interesting, sounds like great work - will take me a while to review it! The If it is Aligned change I'll have to consider very carefully. There are a bunch of alignment rules in ARM docs around MMU usage, and it's very important to meet them (hard fault and immediate hang if it fails to parse the tables). Spotting the binary size going too large is good, we really should check that as part of the build process and fail / warn if so. It's tedious work because it's a per cam size limit and involves messing with the build system. More notes later :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft / WIP — the integrated EOS R (DIGIC 8) research branch. This shares everything from the bring-up so the community has the full picture; the clean, mergeable feature subsets are already split into focused PRs (#279, #280). Not all of this branch is merge-ready — what's what is noted below.
Headline: first DIGIC 8 R with working ROM patching (
CONFIG_MMU_REMAP)The R can now have ML patches applied to its (RAM-remapped) ROM — the foundation everything else on D8 needs. Getting there took two HW-confirmed fixes:
generic_mmu_spacecarried an__attribute__((aligned(0x10000)))that forced a huge BSS linker hole, pushing ML's footprint past the R's tightuser_membudget →my_create_init_taskaborts → solid red LED. It was never an mmu_init logic bug. Fix: drop the alignment + add a one-page slack (the L1 table is aligned internally).dcache_clean_multicoreHW crash: Canon's inter-core cache broadcast (0xE008E23E) faults when called this early inmmu_init(inter-core HW not up). No-op'd for the R's cpu0-only remap (HW-bisected: with it = red LED, without = boots with full UI).Confirmed on hardware: ML boots with full UI, and a ROM code-patch detour is live on cpu0 (SFDIAG dump).
Reverse-engineering captured on this branch
ReadBlockSerialFlash@0xE03C10C4, the driver struct @0x62bc (handle at +16),InstallSerialFlash/UninstallSerialFlash, deep-power-down. Canon installs the SF, reads the property DB at boot, then powers it down (handle→0). This RE is what produced the captured property data that lets qemu-eos PR Add --execute-event-proc to ptpcam #18 boot the R past the property wall toK424 READY.What's mergeable vs bring-up
CONFIG_MMU_REMAPROM patching (src/patch_mmu.c,src/mmu_utils.c,platform/R.180/function_overrides.c, stubs).src/init.c/src/debug.c(the exposure-RE probes have already been pruned).Happy to split
CONFIG_MMU_REMAPinto a focused standalone PR if that's preferred over this umbrella WIP.