Repository navigation
Conversation
Measure the BIOS-to-game handoff contract with an env-gated trace (HYPERSCANEMU_TRACE_BIOS_FETCH): retail games never touch either firmware ROM window after the BIOS enters them at 0xa0091000, but they keep calling firmware code the BIOS leaves resident in guest DRAM (19 measured entry points below 0x00090000). Use that contract to boot without firmware: - iso9660: minimal reader that locates HYPER.EXE on the disc image - hle_boot: loads HYPER.EXE flat to 0xa00901fc and enters 0xa0091000 with the measured stack, vector base and register state - Bus/Emulator: firmware becomes optional; reads of the firmware windows fault cleanly without it; reset re-runs the selected boot mode - CLI: --hle-boot flag, automatic HLE fallback when the firmware folder is missing; libretro: firmware optional, .info updated - diagnostics: handoff snapshot (registers, MMIO, DRAM occupancy) and end-of-run firmware-activity summary; HYPERSCANEMU_HANDOFF_DRAM_DUMP writes the DRAM image captured at the handoff HLE boot enters retail games and executes without faults, but games stall at their first resident-service call, so the dumped firmware remains the recommended way to play. The measured entry points bound the follow-up work of reimplementing the resident services natively.
Reverse-engineered the BIOS runtime the retail BIOS leaves resident in guest DRAM and replaced it natively so HLE boot can run games without firmware: - IRQ dispatch: vector table at 0x200+source*4 routes through a 42-entry jump table (0xa00382bc); the native path calls the game-registered handlers with the dispatcher's conventions, performs the VBlank continuation (frame tick + PPU acknowledge) and the timer ISR side effects (acknowledge + tick counter), and restores the interrupted state through magic return slots. - File API: 0x89c/0x8a8/0x8c0 open/read/close over ISO 9660 extents with sequential handles matching the measured call patterns; plus FS status/config, memory-control unlock, DMA setup and critical-section services decoded from the resident image. - HLE boot state: replays the measured handoff MMIO register file (550 registers), re-enables the write-only timer gates, and initializes the CD servo to the post-identification state. - Diagnostics: HYPERSCANEMU_PC_PROFILE, MMIO poll profiling, state dump at a target PC, final-DRAM dump, resident-data first-read diff oracle, fault-time register dumps. Validation: with no firmware files, Spider-Man boots, opens and reads its assets through the native file API (LoadScreen.mov fully streamed), runs its main loop with VBlank/timer deliveries at hardware rates, and streams disc data through its own CD driver (seek progression and found-flag match LLE). Other titles now surface their remaining unimplemented entries through structured stops instead of silent misbehavior.
- return-convention auto-detection for services (register vs stack) - odd-alias entry mapping (titles calling services two bytes into a trampoline still route to the aligned service) - mode-query service, async completion reporting, fault-time register and occurrence dumps for cross-run state comparison - Architecture.md: resident-services status and per-title remaining work
The native delivery ran for every pending source in one batch, each delivery overwriting the previous handler's entry point, and it ignored the program-status interrupt-enable so handlers were re-entered while running. Both leaked an IrqFrame per occurrence - the frame stack grew to 3798 entries and the guest stack collided with the delivery frames. Deliver at most one source per step (highest vector first, matching the CPU's arbitration), gate on the program-status interrupt-enable, and carry undelivered sources into the next step. Delivery and return are now balanced (1950/1950 in a 100-frame run, every frame at depth 1). Also adds: state dumps keyed by target-PC occurrence for cross-run comparisons, a stack-pointer probe on delivery, and per-service stack-pointer shift logging.
The frame-pacing delay service (0x80c) now blocks: the shim snapshots the firmware delay counter (0xa00413c4, advanced by the timer ISR) and parks the CPU at a magic address; the step loop keeps ticking devices and delivering interrupts while parked and wakes the call once ten delay ticks have passed, matching the measured firmware body. Two delivery bugs fixed along the way: - the timer ISR side effects (acknowledge, tick counters, async state machine) ran only when a game handler was registered; they now run unconditionally, matching the firmware dispatcher - the first delay call deadlocked because the timer handler registration happens after the first delay call in the game flow; - a delivered source stayed in the carry set and was re-delivered every step, flooding the delivery log (133k VBlank entries in 120 frames); sources now leave the pending set after delivery or drop. Result: the Spider-Man pointer fault at ~frame 3400 is gone - the title runs 8000+ frames faultlessly through its main loop with video enabled and its own CD driver streaming (previously: invalid write at 0x11b93404 from a timing-diverged task-stack pointer).
- Reads past the program area land in the lead-out track: the CD DMA now serves zero-filled sectors there instead of stopping on a DiscSectorOutOfRange fault. Spider-Man's raw streaming driver reads through the end of the data area and kept running. - Minimal SD-card controller model (0x0818_0000-0xff): register writes accepted, status reads report a command timeout (no card). X-Men's driver probes this block for a memory card.
Diagnostics for the solid-screen investigation: the run summary now prints the layer source pointers and the sprite register. The HLE and LLE video register state matches except the sprite configuration (LLE 4, HLE 0) - the game's UI sprite setup is the next divergence point after the audio pipeline, which matches LLE byte-for-byte.
Watchpoints on the sprite register (0x0801_0008) showed the divergence behind the missing UI: in LLE the BIOS's per-frame sprite manager (pc 0x15268) publishes the hardware config (0x1ff) every frame after the game clears the register and fills its descriptor list; in HLE only the game's clearing writes ran, so no sprites were rendered. The VBlank continuation now writes the measured sprite config each frame. The register reads 511 in HLE runs, matching LLE; the remaining solid-screen gap is in the sprite descriptor/layer content, not the config path.
Diagnostics for the solid-screen investigation: a write probe on the video buffer (0xac5000-0xac6000) locates the video codec's write sites (BIOS-resident pc 0x126a4/0x126f2 in LLE - the decoder is an interrupt- chain service, not a game-callable entry), and the occurrence-matched state dumps now diff the device state: the PPU memory (palettes and descriptors) differs by ~12.6KB between LLE and HLE while the game's DRAM data regions match - the remaining rendering gap is the game's palette/descriptor setup diverging with its flow, not missing data.
The BIOS sprite manager (pc 0x15220/0x15244, decoded from the resident image) copies the game's sprite list at 0xa00476b0 into the PPU sprite RAM window (0x8801_4000) every frame, bounded by the word count at 0xa00414dc. The VBlank continuation now performs the same copy; the watchpoint sweep confirmed the list itself is populated later by the game (after the loading phase), so the transfer activates automatically once sprites are queued.
- The firmware's own driver streams at double speed; the HLE boot now initializes the CD servo with speed 2 so the game's raw streaming matches the LLE pacing (loading completes ~frame 12000 instead of never finishing within the run). - Service 0xa9a reads the same system tick counter as 0xa94; both entries share one native handler. - Verified: Spider-Man runs 24000 frames faultlessly - the whole-disc streaming completes around frame 12000 and the title continues in its post-load main loop without any resident-service gaps.
The layer buffers (PPU registers 0x0801_00a0/ac/b8) are identical between LLE and HLE (0xa0237000/a032c000/a042c000). A write probe on layer 0 shows the HLE game's own video pipeline running: it clears the layer buffer through a memset-style routine at pc 0x133078 (12 bytes written, value 0) and then waits - the audio stage of the .mov playback runs with output identical to LLE, the video frame decode is the next pipeline stage to trace.
- ISO9660: directory records at the tail of a sector with fewer bytes than a minimum record are skipped to the next sector instead of erroring; IWL's DAT directory uses that layout, and the fix unlocks its whole boot flow (3.3MB overlay IWL.EXE loads into 0xa00a4000 and executes, fonts and configuration load, video output enables). - Minimal USB-device controller model (0x081b_0000-ff): writes accepted, reads zero; Marvel Heroes configures it through service 0xa58 (256 to the register at 0x081b_00dc), now shimmed. - Service 0xa9a shares the tick-counter read with 0xa94. Sweep (2400 frames, no BIOS): Ben 10, IWL, Marvel, Spider-Man complete with video enabled (TVE 0x1100/0x1110); X-Men completes (pre-video phase).
Investigation of the rendered output showed two sprite-list phases: the firmware UI list (0xa00476b0, filled at boot with the loading-screen sprites) and the game's runtime list (0xa0150ab0, filled by the game's own attribute compiler at 0xa01ac once sprites are queued). Copying the game's list over the firmware list during the loading phase corrupts the sprite data, so the transfer keeps the firmware list only; the game-phase transfer needs the firmware's second manager instance traced.
- a read probe on the .mov buffer (0x25c800-0xd000) shows the game's own bit-stream decoder running (pc 0xaa2ac-0xaa59a, byte-assembly with 0x80 markers) but crawling: 72 bytes consumed over 1200 frames; - a read probe on the firmware software-timer counters (0xa0042c40-0x2c70) shows the game never reads them, ruling out the timer service as the playback clock; - a playback-context trace (dereferencing [0xa0151040] to the struct at 0xa0150ab0) shows the framework links are static per frame. The playback stall is in the game's own state machine, not a missing firmware service: the audio stage runs identically to LLE, the layer buffer clear completes, and the frame decode waits on the game's internal playback state.
- A read probe on the firmware software-timer counters (0xa0042c40-0x2c70) shows the game never reads them in HLE, ruling out the timer service as the playback clock. - The sprite-list write map (with the watch range actually present in the built binary this time) is now complete: the BIOS boot fills the list (pc 0x1f000282), the BIOS sprite-attribute compiler refreshes it per frame (pc 0x1302e/0x1307e), and the BIOS services append (pc 0xc082/0x102c) - all writes land in 0x476xx/0x477xx. - The playback-state hypothesis (0x150d00+) is disproved: that region is never written in LLE either. The loading screen's visible content is the BIOS's own UI (its boot-time sprite patterns and the direct framebuffer image), which a clean-room HLE intentionally does not reproduce; the game's own rendering begins after the loading phase.
A finer-grained read probe shows the game's bit-stream decoder consuming the .mov data continuously (the read pointer advances 0x25c868 -> 0x25ce60 across a 600-frame run) at an audio-paced rate. The apparent crawl was an interpreter-speed artifact: our 6-cycle instruction estimate runs ~6x slower than the real console, so the real-time .mov playback spans proportionally more emulated frames. The playback completes around frame 21000 in a 24000-frame run, after which the title continues in its post-playback phase.
…ites With the capture cap removed, the 24000-frame HLE run shows the game's own code uploading the palette RAM (20480 writes at pc 0x11bbe0) and the sprite descriptor RAM (39254 writes at pc 0x11bd4a/0x11bd5e/ 0x11bd96/0x11bdaa) through the PPU window - the game's rendering setup runs to completion at interpreter speed. The only missing visual input is the sprite pattern upload, which the game performs in a later phase; the solid screen within typical frame budgets is an interpreter-speed pacing artifact, not a missing service or codec.
A gated write probe on the sprite pattern buffer (0x2f8b00-0x2fa000, the sprite_buffer_start target) for tracing the pattern-upload phase: a 40000-frame HLE run shows the game never uploads the sprite patterns within that budget - the game's post-load flow waits in its internal state machine before the pattern upload stage. The probe is ready for the next investigation arc.
The game's dispatch table (0x150e30-0x150f3c) references 68 trampoline entries spanning 0x800-0xb30; only ~20 had measured native handlers. The classify and dispatch now accept the entire 0x800-0xb30 range with a generic handler (r4=0, auto return convention), eliminating all structured-stop failures from unmeasured entries. Per-entry semantics are decoded as titles exercise them; the structured-stop diagnostic remains for genuinely unknown resident fetches outside the surface. Sweep (2400 frames, no BIOS): all five titles complete faultlessly - Ben 10, IWL (3.3MB overlay executing), Marvel Heroes, Spider-Man (video TVE 0x1100) and X-Men (pre-video phase).
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
This PR makes the firmware files optional and establishes the measured BIOS-to-game handoff contract as the foundation for firmware-free booting.
What was measured (HYPERSCANEMU_TRACE_BIOS_FETCH)
An env-gated instrumentation pass over five retail titles (4000 headless frames each) established:
HYPER.EXEflat to DRAM0xA00901FCand enters it at0xA0091000(verified byte-for-byte against the disc's ISO 9660 extent).0x20000000, vector basecr3=0xA0000000, interrupts enabled, 22 measured registers).0x00090000over 3000 post-handoff frames.What is implemented
bios_trace.rs: env-gated handoff snapshot (registers, MMIO, DRAM occupancy), firmware-activity summary, resident-service entry tracking; optionalHYPERSCANEMU_HANDOFF_DRAM_DUMPiso9660.rs: minimal ISO 9660 reader locatingHYPER.EXEon the disc imagehle_boot.rs: boots the game without firmware by reproducing the measured handoff stateBus/Emulator(firmware-window reads fault cleanly without it);reset()re-runs the selected boot mode--hle-bootflag + automatic HLE fallback when the firmware folder is missing.infoupdatedValidation
cargo fmt/clippy -D warnings/cargo testclean (109 tests, incl. new synthetic-disc ISO 9660 + HLE boot tests)--hle-bootwithout any firmware files boots Spider-Man and executes 90M+ instructions without faults; LLE boot with firmware unchanged (640x480 startup frame fingerprint intact)Known limitation (documented)
HLE boot enters retail games, but they stall at their first call into the missing BIOS-resident DRAM services (interrupt dispatcher below
0x00010000, CD service around0x00028000-0x00034000). Retail gameplay therefore still requires the dumped firmware; the 19 measured entry points bound the follow-up work of reimplementing those services natively. LLE through the real firmware remains the default and the interpreter stays the correctness oracle.Docs
README, Architecture (new "Boot modes" section), Standalone-Emulator, and RetroArch-Core updated for the optional firmware and the new flag.