Skip to content

Add cross-runtime ESP32-S3 HIL benchmark - #40

Merged
w1ne merged 48 commits into
mainfrom
bench/twin2silicon-deepseek
Aug 16, 2026
Merged

Add cross-runtime ESP32-S3 HIL benchmark#40
w1ne merged 48 commits into
mainfrom
bench/twin2silicon-deepseek

Conversation

@w1ne

@w1ne w1ne commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a real ESP32-S3 firmware repair benchmark with bounded build, flash, UART nonce, and JTAG register verification
  • add native OpenCode, Codex CLI, and Claude Code adapters that share the same LabWired instructions and MCP server while retaining native/default models
  • add sequential matrix execution, normalized model/token/cost/timing evidence, isolated failure handling, board identity validation, and explicit upload-port selection
  • add offline and opt-in connected-board entry points, documentation, fixture contracts, and regression coverage
  • correct hosted OpenCode tool-name projection and the DeepInfra OpenCode base URL used by the agent configuration

Why

Generic coding benchmarks do not measure whether an agent can repair firmware, compile it, flash the intended MCU, observe a unique runtime nonce, and prove physical GPIO state. This adds a small reproducible HIL task and a runtime-comparison harness without training or silently retrying models.

Hardware validation

A fresh native smoke trial was run against the identified ESP32-S3 at /dev/cu.usbmodem11101 / 3C:0F:02:DF:EC:F8.

  • OpenCode: recorded provider authentication failure and skipped HIL
  • Codex: made the minimal input-to-output GPIO repair; corrected HIL recheck passed compile, flash, UART nonce, output-enable, and output-high assertions
  • Claude Code: made the same minimal repair; corrected HIL recheck passed the same physical checks

The original matrix evidence was preserved after it exposed an upload auto-detection bug; separate recheck evidence validates the fix that pins flashing to the identity-selected UART.

Developer impact

  • offline tests do not require API credentials or hardware
  • connected tests require explicit LABWIRED_HIL=1, UART, JTAG serial, OpenOCD, and output paths
  • subscription cost remains unknown unless the native runtime reports it
  • results are documented as a smoke comparison, not a leaderboard claim

Test plan

  • npm run test:runtime-smoke:offline — 90 tests passed
  • bash -n tests/twin2silicon-runtime-smoke.sh
  • Python byte-compilation for benchmark runners and adapters
  • git diff --check
  • read-only ESP32-S3 identity check
  • connected HIL rechecks for Codex and Claude candidates

w1ne added 30 commits August 13, 2026 10:09
@w1ne
w1ne marked this pull request as ready for review August 16, 2026 09:44
@w1ne
w1ne merged commit 24da06f into main Aug 16, 2026
9 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant