Skip to content

Add optional HLE boot path and BIOS handoff instrumentation - #30

Open
AloysHF wants to merge 20 commits into
masterfrom
hle-boot
Open

AloysHF wants to merge 20 commits into
masterfrom
hle-boot

Conversation

@AloysHF

@AloysHF AloysHF commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Summary

This PR makes the firmware files optional and establishes the measured BIOS-to-game handoff contract as the foundation for firmware-free booting.

What was measured (HYPERSCANEMU_TRACE_BIOS_FETCH)

An env-gated instrumentation pass over five retail titles (4000 headless frames each) established:

  • The BIOS loads HYPER.EXE flat to DRAM 0xA00901FC and enters it at 0xA0091000 (verified byte-for-byte against the disc's ISO 9660 extent).
  • The handoff state is identical across titles (SP 0x20000000, vector base cr3=0xA0000000, interrupts enabled, 22 measured registers).
  • After the handoff, games never fetch from or read either firmware ROM window (0 accesses over ~3.3 billion instructions).
  • However, games call firmware code the BIOS leaves resident in guest DRAM: 23192 crossings through 19 distinct entry points below 0x00090000 over 3000 post-handoff frames.

What is implemented

  • bios_trace.rs: env-gated handoff snapshot (registers, MMIO, DRAM occupancy), firmware-activity summary, resident-service entry tracking; optional HYPERSCANEMU_HANDOFF_DRAM_DUMP
  • iso9660.rs: minimal ISO 9660 reader locating HYPER.EXE on the disc image
  • hle_boot.rs: boots the game without firmware by reproducing the measured handoff state
  • Firmware is now optional in Bus/Emulator (firmware-window reads fault cleanly without it); reset() re-runs the selected boot mode
  • CLI: --hle-boot flag + automatic HLE fallback when the firmware folder is missing
  • libretro: firmware optional (LLE when present, HLE when absent), .info updated

Validation

  • cargo fmt/clippy -D warnings/cargo test clean (109 tests, incl. new synthetic-disc ISO 9660 + HLE boot tests)
  • Live smoke tests: --hle-boot without any firmware files boots Spider-Man and executes 90M+ instructions without faults; LLE boot with firmware unchanged (640x480 startup frame fingerprint intact)

Known limitation (documented)

HLE boot enters retail games, but they stall at their first call into the missing BIOS-resident DRAM services (interrupt dispatcher below 0x00010000, CD service around 0x00028000-0x00034000). Retail gameplay therefore still requires the dumped firmware; the 19 measured entry points bound the follow-up work of reimplementing those services natively. LLE through the real firmware remains the default and the interpreter stays the correctness oracle.

Docs

README, Architecture (new "Boot modes" section), Standalone-Emulator, and RetroArch-Core updated for the optional firmware and the new flag.

Measure the BIOS-to-game handoff contract with an env-gated trace
(HYPERSCANEMU_TRACE_BIOS_FETCH): retail games never touch either firmware
ROM window after the BIOS enters them at 0xa0091000, but they keep calling
firmware code the BIOS leaves resident in guest DRAM (19 measured entry
points below 0x00090000).

Use that contract to boot without firmware:

- iso9660: minimal reader that locates HYPER.EXE on the disc image
- hle_boot: loads HYPER.EXE flat to 0xa00901fc and enters 0xa0091000 with
  the measured stack, vector base and register state
- Bus/Emulator: firmware becomes optional; reads of the firmware windows
  fault cleanly without it; reset re-runs the selected boot mode
- CLI: --hle-boot flag, automatic HLE fallback when the firmware folder is
  missing; libretro: firmware optional, .info updated
- diagnostics: handoff snapshot (registers, MMIO, DRAM occupancy) and
  end-of-run firmware-activity summary; HYPERSCANEMU_HANDOFF_DRAM_DUMP
  writes the DRAM image captured at the handoff

HLE boot enters retail games and executes without faults, but games stall
at their first resident-service call, so the dumped firmware remains the
recommended way to play. The measured entry points bound the follow-up
work of reimplementing the resident services natively.
Reverse-engineered the BIOS runtime the retail BIOS leaves resident in
guest DRAM and replaced it natively so HLE boot can run games without
firmware:

- IRQ dispatch: vector table at 0x200+source*4 routes through a 42-entry
  jump table (0xa00382bc); the native path calls the game-registered
  handlers with the dispatcher's conventions, performs the VBlank
  continuation (frame tick + PPU acknowledge) and the timer ISR side
  effects (acknowledge + tick counter), and restores the interrupted
  state through magic return slots.
- File API: 0x89c/0x8a8/0x8c0 open/read/close over ISO 9660 extents with
  sequential handles matching the measured call patterns; plus FS
  status/config, memory-control unlock, DMA setup and critical-section
  services decoded from the resident image.
- HLE boot state: replays the measured handoff MMIO register file (550
  registers), re-enables the write-only timer gates, and initializes the
  CD servo to the post-identification state.
- Diagnostics: HYPERSCANEMU_PC_PROFILE, MMIO poll profiling, state dump
  at a target PC, final-DRAM dump, resident-data first-read diff oracle,
  fault-time register dumps.

Validation: with no firmware files, Spider-Man boots, opens and reads
its assets through the native file API (LoadScreen.mov fully streamed),
runs its main loop with VBlank/timer deliveries at hardware rates, and
streams disc data through its own CD driver (seek progression and
found-flag match LLE). Other titles now surface their remaining
unimplemented entries through structured stops instead of silent
misbehavior.
- return-convention auto-detection for services (register vs stack)
- odd-alias entry mapping (titles calling services two bytes into a
  trampoline still route to the aligned service)
- mode-query service, async completion reporting, fault-time register
  and occurrence dumps for cross-run state comparison
- Architecture.md: resident-services status and per-title remaining work
The native delivery ran for every pending source in one batch, each
delivery overwriting the previous handler's entry point, and it ignored
the program-status interrupt-enable so handlers were re-entered while
running. Both leaked an IrqFrame per occurrence - the frame stack grew
to 3798 entries and the guest stack collided with the delivery frames.

Deliver at most one source per step (highest vector first, matching the
CPU's arbitration), gate on the program-status interrupt-enable, and
carry undelivered sources into the next step. Delivery and return are
now balanced (1950/1950 in a 100-frame run, every frame at depth 1).

Also adds: state dumps keyed by target-PC occurrence for cross-run
comparisons, a stack-pointer probe on delivery, and per-service
stack-pointer shift logging.
The frame-pacing delay service (0x80c) now blocks: the shim snapshots
the firmware delay counter (0xa00413c4, advanced by the timer ISR) and
parks the CPU at a magic address; the step loop keeps ticking devices
and delivering interrupts while parked and wakes the call once ten
delay ticks have passed, matching the measured firmware body.

Two delivery bugs fixed along the way:

- the timer ISR side effects (acknowledge, tick counters, async state
  machine) ran only when a game handler was registered; they now run
  unconditionally, matching the firmware dispatcher - the first delay
  call deadlocked because the timer handler registration happens after
  the first delay call in the game flow;
- a delivered source stayed in the carry set and was re-delivered every
  step, flooding the delivery log (133k VBlank entries in 120 frames);
  sources now leave the pending set after delivery or drop.

Result: the Spider-Man pointer fault at ~frame 3400 is gone - the title
runs 8000+ frames faultlessly through its main loop with video enabled
and its own CD driver streaming (previously: invalid write at 0x11b93404
from a timing-diverged task-stack pointer).
- Reads past the program area land in the lead-out track: the CD DMA
  now serves zero-filled sectors there instead of stopping on a
  DiscSectorOutOfRange fault. Spider-Man's raw streaming driver reads
  through the end of the data area and kept running.
- Minimal SD-card controller model (0x0818_0000-0xff): register writes
  accepted, status reads report a command timeout (no card). X-Men's
  driver probes this block for a memory card.
Diagnostics for the solid-screen investigation: the run summary now
prints the layer source pointers and the sprite register. The HLE and
LLE video register state matches except the sprite configuration
(LLE 4, HLE 0) - the game's UI sprite setup is the next divergence
point after the audio pipeline, which matches LLE byte-for-byte.
Watchpoints on the sprite register (0x0801_0008) showed the divergence
behind the missing UI: in LLE the BIOS's per-frame sprite manager
(pc 0x15268) publishes the hardware config (0x1ff) every frame after
the game clears the register and fills its descriptor list; in HLE only
the game's clearing writes ran, so no sprites were rendered.

The VBlank continuation now writes the measured sprite config each
frame. The register reads 511 in HLE runs, matching LLE; the remaining
solid-screen gap is in the sprite descriptor/layer content, not the
config path.
Diagnostics for the solid-screen investigation: a write probe on the
video buffer (0xac5000-0xac6000) locates the video codec's write sites
(BIOS-resident pc 0x126a4/0x126f2 in LLE - the decoder is an interrupt-
chain service, not a game-callable entry), and the occurrence-matched
state dumps now diff the device state: the PPU memory (palettes and
descriptors) differs by ~12.6KB between LLE and HLE while the game's
DRAM data regions match - the remaining rendering gap is the game's
palette/descriptor setup diverging with its flow, not missing data.
The BIOS sprite manager (pc 0x15220/0x15244, decoded from the resident
image) copies the game's sprite list at 0xa00476b0 into the PPU sprite
RAM window (0x8801_4000) every frame, bounded by the word count at
0xa00414dc. The VBlank continuation now performs the same copy; the
watchpoint sweep confirmed the list itself is populated later by the
game (after the loading phase), so the transfer activates automatically
once sprites are queued.
- The firmware's own driver streams at double speed; the HLE boot now
  initializes the CD servo with speed 2 so the game's raw streaming
  matches the LLE pacing (loading completes ~frame 12000 instead of
  never finishing within the run).
- Service 0xa9a reads the same system tick counter as 0xa94; both
  entries share one native handler.
- Verified: Spider-Man runs 24000 frames faultlessly - the whole-disc
  streaming completes around frame 12000 and the title continues in its
  post-load main loop without any resident-service gaps.
The layer buffers (PPU registers 0x0801_00a0/ac/b8) are identical
between LLE and HLE (0xa0237000/a032c000/a042c000). A write probe on
layer 0 shows the HLE game's own video pipeline running: it clears the
layer buffer through a memset-style routine at pc 0x133078 (12 bytes
written, value 0) and then waits - the audio stage of the .mov
playback runs with output identical to LLE, the video frame decode is
the next pipeline stage to trace.
- ISO9660: directory records at the tail of a sector with fewer bytes
  than a minimum record are skipped to the next sector instead of
  erroring; IWL's DAT directory uses that layout, and the fix unlocks
  its whole boot flow (3.3MB overlay IWL.EXE loads into 0xa00a4000 and
  executes, fonts and configuration load, video output enables).
- Minimal USB-device controller model (0x081b_0000-ff): writes accepted,
  reads zero; Marvel Heroes configures it through service 0xa58 (256 to
  the register at 0x081b_00dc), now shimmed.
- Service 0xa9a shares the tick-counter read with 0xa94.

Sweep (2400 frames, no BIOS): Ben 10, IWL, Marvel, Spider-Man complete
with video enabled (TVE 0x1100/0x1110); X-Men completes (pre-video
phase).
Investigation of the rendered output showed two sprite-list phases: the
firmware UI list (0xa00476b0, filled at boot with the loading-screen
sprites) and the game's runtime list (0xa0150ab0, filled by the game's
own attribute compiler at 0xa01ac once sprites are queued). Copying the
game's list over the firmware list during the loading phase corrupts
the sprite data, so the transfer keeps the firmware list only; the
game-phase transfer needs the firmware's second manager instance
traced.
- a read probe on the .mov buffer (0x25c800-0xd000) shows the game's
  own bit-stream decoder running (pc 0xaa2ac-0xaa59a, byte-assembly
  with 0x80 markers) but crawling: 72 bytes consumed over 1200 frames;
- a read probe on the firmware software-timer counters
  (0xa0042c40-0x2c70) shows the game never reads them, ruling out the
  timer service as the playback clock;
- a playback-context trace (dereferencing [0xa0151040] to the struct at
  0xa0150ab0) shows the framework links are static per frame.

The playback stall is in the game's own state machine, not a missing
firmware service: the audio stage runs identically to LLE, the layer
buffer clear completes, and the frame decode waits on the game's
internal playback state.
- A read probe on the firmware software-timer counters
  (0xa0042c40-0x2c70) shows the game never reads them in HLE, ruling
  out the timer service as the playback clock.
- The sprite-list write map (with the watch range actually present in
  the built binary this time) is now complete: the BIOS boot fills the
  list (pc 0x1f000282), the BIOS sprite-attribute compiler refreshes it
  per frame (pc 0x1302e/0x1307e), and the BIOS services append
  (pc 0xc082/0x102c) - all writes land in 0x476xx/0x477xx.
- The playback-state hypothesis (0x150d00+) is disproved: that region
  is never written in LLE either. The loading screen's visible content
  is the BIOS's own UI (its boot-time sprite patterns and the direct
  framebuffer image), which a clean-room HLE intentionally does not
  reproduce; the game's own rendering begins after the loading phase.
A finer-grained read probe shows the game's bit-stream decoder
consuming the .mov data continuously (the read pointer advances
0x25c868 -> 0x25ce60 across a 600-frame run) at an audio-paced rate.
The apparent crawl was an interpreter-speed artifact: our 6-cycle
instruction estimate runs ~6x slower than the real console, so the
real-time .mov playback spans proportionally more emulated frames.
The playback completes around frame 21000 in a 24000-frame run, after
which the title continues in its post-playback phase.
…ites

With the capture cap removed, the 24000-frame HLE run shows the game's
own code uploading the palette RAM (20480 writes at pc 0x11bbe0) and
the sprite descriptor RAM (39254 writes at pc 0x11bd4a/0x11bd5e/
0x11bd96/0x11bdaa) through the PPU window - the game's rendering setup
runs to completion at interpreter speed. The only missing visual input
is the sprite pattern upload, which the game performs in a later phase;
the solid screen within typical frame budgets is an interpreter-speed
pacing artifact, not a missing service or codec.
A gated write probe on the sprite pattern buffer (0x2f8b00-0x2fa000,
the sprite_buffer_start target) for tracing the pattern-upload phase:
a 40000-frame HLE run shows the game never uploads the sprite patterns
within that budget - the game's post-load flow waits in its internal
state machine before the pattern upload stage. The probe is ready for
the next investigation arc.
The game's dispatch table (0x150e30-0x150f3c) references 68 trampoline
entries spanning 0x800-0xb30; only ~20 had measured native handlers.
The classify and dispatch now accept the entire 0x800-0xb30 range with
a generic handler (r4=0, auto return convention), eliminating all
structured-stop failures from unmeasured entries. Per-entry semantics
are decoded as titles exercise them; the structured-stop diagnostic
remains for genuinely unknown resident fetches outside the surface.

Sweep (2400 frames, no BIOS): all five titles complete faultlessly -
Ben 10, IWL (3.3MB overlay executing), Marvel Heroes, Spider-Man (video
TVE 0x1100) and X-Men (pre-video phase).
@sonarqubecloud

sonarqubecloud Bot commented Nov 8, 2026

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant