Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .github/workflows/renovate-hash-sync.yml
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,11 @@ on:
- "package/bcm20702-firmware/bcm20702-firmware.mk"
- "package/libchdr/libchdr.mk"
- "package/lzma-sdk/lzma-sdk.mk"
# rcheevos (RetroAchievements client library). Ordinary github
# tag-archive pin, covered by the generic loop in step 1. Same
# both-lists-or-silent-failure note as rtl8852cu-morrownr above: it must
# appear HERE and in HASH_SYNC_PACKAGES below.
- "package/rcheevos/rcheevos.mk"
# azcopy (case 7). ADDED 2026-08-28, when that case was added -- before
# that an azcopy bump PR deliberately stayed red until a human ran
# package/azcopy/azcopy.hash's regeneration recipe by hand. Same
Expand Down Expand Up @@ -280,7 +285,7 @@ jobs:
# tarballs from the loop's .../archive/<ref>.tar.gz form and Buildroot's
# .../archive/<ref>/<name>.tar.gz form -- see package/dualsensectl/
# dualsensectl.hash.
HASH_SYNC_PACKAGES: "rtl8852cu-morrownr aic8800 xone midilink munt bcm20702-firmware libchdr dualsensectl ltunify"
HASH_SYNC_PACKAGES: "rtl8852cu-morrownr aic8800 xone midilink munt bcm20702-firmware libchdr rcheevos dualsensectl ltunify"

steps:
# A manual run pushes commits to whatever branch it is pointed at, so
Expand Down
10 changes: 10 additions & 0 deletions Config.in
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,16 @@ menu "Main_MiSTer shared libraries"
# independently selectable.
source "$BR2_EXTERNAL_MISTER_PATH/package/lzma-sdk/Config.in"
source "$BR2_EXTERNAL_MISTER_PATH/package/libchdr/Config.in"
# rcheevos (RetroAchievements client library) is the ODD ONE OUT in this
# menu and is sourced last to keep that visible. lzma-sdk and libchdr are
# here to REPLACE code Main_MiSTer already vendors -- that is what "the
# Main_MiSTer shared-lib refactor" means. rcheevos replaces nothing:
# neither Main nor this tree references it yet. It is in this menu anyway
# because the menu's actual contract is "shared libraries this tree
# authors for Main_MiSTer to link, which upstream Buildroot does not
# carry", and a second menu holding one library would say less. It has no
# dependency on either sibling and toggles independently.
source "$BR2_EXTERNAL_MISTER_PATH/package/rcheevos/Config.in"
endmenu

menu "Archival / on-device update tooling"
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -678,7 +678,7 @@ board/mister/de10nano/
rootfs-overlay/ init scripts, sshd wiring, MiSTer-specific files
post-build.sh /MiSTer.version stamping, parity fixups
post-image.sh linux.img assembly + contract checks
package/ 16 packages: Realtek Wi-Fi, xone, libchdr, lzma-sdk, midilink, munt…
package/ 23 packages: Wi-Fi/controller drivers, libchdr, lzma-sdk, rcheevos, midilink, munt…
scripts/ the verification suite, hash-sync, SD-card builder, kernel export
docs/ ADRs, parity audits, the kernel reconciliation, user docs
.github/ 9 workflows + 4 composite actions
Expand Down
28 changes: 24 additions & 4 deletions TASKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -1054,10 +1054,30 @@ is), never whether it has been proven yet.
because that check runs at the end of an ~80-minute image build. Fixed by teaching the
step to refresh `*_LICENSE_FILES` hashes and diff any that changed; see
`scripts/hash-sync-ip7z-src.sh`'s header.
Still unproven: the generic github-package loop (all 12 github-sourced pins, including
libchdr) and the sdcard-payload step have never run against a real PR — treat those
regexes/URLs as reviewed-by-hand, not proven. See `docs/renovate.md`'s "Unverified /
what to check on first run".
Still unproven **against a real PR**: the generic github-package loop (every
github-sourced pin in `HASH_SYNC_PACKAGES`, libchdr and rcheevos included) and the
sdcard-payload step. Partial credit for the loop as of 2026-09-16, when
`package/rcheevos` was added — it was exercised standalone, over the network, on BOTH
of its branches:
- *already-current*: against the real pin, it derived the archive URL and the
`<pkgdir>-<version>.tar.gz` filename and agreed with the hand-computed sha256.
- *refreshed*: against a fixture with the pin moved back to `v12.4.0` and a
deliberately wrong tarball hash, it followed the new ref and rewrote the line to
`7fb1a43b…`, independently confirmed by fetching Buildroot's own
`.../archive/<ref>/<name>.tar.gz` URL form, and it left the three licence-file
lines alone as documented. **This is the branch that matters**: an
`already-current` run never executes the rewrite at all, so it could not have
caught a stale-write bug of the kind that once left `linux.hash` untouched while
the job reported success three times.

So the loop's own URL/filename/hash/rewrite logic is no longer "reviewed-by-hand, not
proven". The `paths:` trigger is proven too, incidentally and on a real PR: #182 (the
PR that added `package/rcheevos`) changed exactly ONE file matching this workflow's
`paths:` filter — the new `package/rcheevos/rcheevos.mk` — and the workflow fired,
then correctly skipped itself as a non-Renovate branch. What remains unproven is the
rest of the workflow AROUND the loop — the commit-back and the outcomes gate —
which neither a skipped run nor a standalone run reaches, so the item stays open. See
`docs/renovate.md`'s "Unverified / what to check on first run".
**Done when:** a real Renovate PR has exercised each of the two still-unproven paths
at least once with a passing (or fixed-then-passing) run, and `docs/renovate.md` drops
the "unverified" caveat for each path once proven.
Expand Down
19 changes: 17 additions & 2 deletions docs/buildroot-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -762,8 +762,9 @@ glibc; it does not generate any locale *data*. That is a separate knob — §5.4
The BR2_EXTERNAL half of the Main_MiSTer shared-lib refactor (no task ID —
referenced by name): Main stops vendoring `lib/{lzma,zstd,miniz,libchdr}` and
links Buildroot-provided shared libraries; the upstream half (zstd, minizip-ng)
is §5.4. Both packages are authored under `package/`; see
`docs/main-shared-libs.md`.
is §5.4. All three packages are authored under `package/`; see
`docs/main-shared-libs.md`. The third, rcheevos, shares the menu but not the
refactor — it replaces no vendored code.

- `BR2_PACKAGE_LZMA_SDK=y` — 7-Zip LZMA SDK 26.03 as `liblzma-sdk.so.<ver>`;
the full-version SONAME is the deliberate loud-ABI-event policy: the Main
Expand All @@ -773,6 +774,20 @@ is §5.4. Both packages are authored under `package/`; see
Findzstd pkg-config fallback (the tag cannot configure against Buildroot's
zstd); system zlib/zstd/lzma-sdk via our 3 patches; exports `chd_*` ONLY
(version script), so no symbol collisions with minizip-ng et al.
- `BR2_PACKAGE_RCHEEVOS=y` — the RetroAchievements client library,
`librcheevos.so.12.5.0` (273 KiB stripped in the rootfs — 279,580 bytes). **NOTHING LINKS IT YET** — unlike the
two above it replaces no vendored Main code; it is shipped so a consumer
can link it, and its only CI cover is the presence assertion in
`scripts/ci-tests.sh` (a library with no consumer has no other way to fail
visibly). Full-version SONAME for the same loud-ABI-event reason as
lzma-sdk, but on harder evidence: upstream has changed caller-allocated
public struct layouts in *patch* releases (v10.7.1, v10.3.3, v6.0.1). Built
`-DRC_SHARED -fvisibility=hidden` so only the 264 `RC_EXPORT` entry points
escape rather than 477 — the suppressed ones include `md5_*` and `AES_*`,
which would otherwise interpose against Main's own `lib/md5`. A consumer
must keep the `-DRC_CLIENT_SUPPORTS_HASH` that `rcheevos.pc` supplies: it
gates public declarations in `rc_client.h`, not just implementation
(`package/rcheevos/rcheevos.mk`).

### 5.6 graphics / fonts

Expand Down
21 changes: 14 additions & 7 deletions docs/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -2421,13 +2421,20 @@ idiom** and should get the same treatment when next touched.
<a id="renovate-hash-sync-safety-model"></a>
### Safety model, the refresh cases: where each refreshed value legitimately comes from

1. **The 14 github-sourced packages** (`package/*/*.mk` + their `.hash`): the
12 driver/firmware pins plus `libchdr` (a userspace shared library — the
Main_MiSTer shared-lib refactor) and `dualsensectl` (a userspace CLI —
the DualSense operator tool). Neither of the last two is a driver, but
both have the exact same `$(call github,...)` archive shape;
`dualsensectl` differs only in pinning a `v`-prefixed tag rather than a
commit SHA, which the loop handles without special-casing. Their own `.hash` file
1. **The github-sourced packages named in `HASH_SYNC_PACKAGES`**
(`package/*/*.mk` + their `.hash`): the driver/firmware pins plus
`libchdr` and `rcheevos` (userspace shared libraries — the first from the
Main_MiSTer shared-lib refactor, the second the RetroAchievements client
library) and `dualsensectl` + `ltunify` (userspace CLIs — the DualSense
operator tool and the Logitech pairing tool). None of the last four is a
driver, but all have the exact same `$(call github,...)` archive shape;
`dualsensectl` and `rcheevos` differ only in pinning a `v`-prefixed tag
rather than a commit SHA, which the loop handles without special-casing —
demonstrated on 2026-09-16, when the loop was run standalone against the
real `rcheevos` pin and derived the same URL, filename and sha256 the pin
was written with. Count the roster in `HASH_SYNC_PACKAGES` itself rather
than trusting a number here; that env var is the single source of truth
and a hardcoded count here has gone stale before. Their own `.hash` file
headers already say the hash is "locally computed" — GitHub publishes no
signed manifest for a commit/tag archive tarball, so `sha256sum` of a
freshly-fetched tarball from the ACTUAL pinned owner/repo/ref IS the
Expand Down
103 changes: 100 additions & 3 deletions docs/main-shared-libs.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@ Three of the five packages are upstream Buildroot, enabled straight from
authored in this tree under `package/` and sourced via the
"Main_MiSTer shared libraries" menu in the top-level `Config.in`.

A **sixth** library, `rcheevos`, is built and shipped from that same menu but
is **not part of this refactor** and is tracked separately below. The
distinction is worth keeping: the five replace code Main already vendors, so
each one's success condition is "Main stops carrying a copy". rcheevos
replaces nothing — it adds a capability the image did not have, and nothing
links it yet.

## The five packages

| Package | Provider | SONAME | Headers | pkg-config | Notes |
Expand All @@ -25,6 +32,89 @@ authored in this tree under `package/` and sourced via the
| lzma-sdk (`BR2_PACKAGE_LZMA_SDK`) | BR2_EXTERNAL (`package/lzma-sdk`) | `liblzma-sdk.so.26.03` | `include/lzma-sdk/` (13 headers, Main's vendored `lib/lzma` set 1:1) | `lzma-sdk.pc` | 7-Zip LZMA SDK 26.03, built `-DZ7_ST` (single-threaded — same as Main's vendored build). **The SONAME is the full SDK version, deliberately**: upstream gives no ABI guarantees between releases and the API embeds caller-allocated structs (`CLzmaDec` by value), so a silent struct-layout change is memory corruption, not an error. A full-version SONAME turns every SDK bump into a *loud* ABI event — an old binary refuses to load with a clean linker error. That matters here because the `MiSTer` binary lives on `/media/fat` and **survives rootfs reflashes**; stale-binary-meets-new-rootfs is the expected failure mode. NOT xz-utils' `liblzma.so.5` — entirely different API. |
| libchdr (`BR2_PACKAGE_LIBCHDR`) | BR2_EXTERNAL (`package/libchdr`) | `libchdr.so.0` (real file `libchdr.so.0.3`) | `include/libchdr/` | `libchdr.pc` | Commit-pinned past `v0.3.0` (the tag can't configure against Buildroot's zstd — no `Findzstd` pkg-config fallback yet). Built against **system zlib/zstd/lzma-sdk** via our patches 0001–0003; the header-only **dr_flac stays bundled** (header-only by design, and `libchdr_flac.c` pokes drflac internals — no `.so` exists to unbundle to). Exports **`chd_*` only** (upstream's version script `src/link.T`), so no `mz_*` or other symbol collision with minizip-ng et al. **"System zlib" here is zlib-ng in `ZLIB_COMPAT` mode** (`BR2_PACKAGE_ZLIB_NG`, defconfig compression block): same `libz.so.1`, same `zlib.h`, so nothing changes at link or `pkg-config` time — and it is the whole point of unbundling, since a *shared* libchdr is what actually reaches system zlib (stock's static libchdr decodes through its own vendored miniz instead). Measured on the rig: CHD audio-hunk decode p90 −10 to −11 %, LZMA-dominated data hunks unchanged (`harness/rig/chd-decode-optimization.md` in Main_MiSTer). |

## The sixth library: rcheevos (not part of the refactor)

| Package | Provider | SONAME | Headers | pkg-config | Notes |
|---|---|---|---|---|---|
| rcheevos (`BR2_PACKAGE_RCHEEVOS`) | BR2_EXTERNAL (`package/rcheevos`) | `librcheevos.so.12.5.0` | `include/rcheevos/` (15 from upstream's `include/`, plus `rc_version.h` from `src/`) | `rcheevos.pc` | RetroAchievements' own client library — achievement/leaderboard evaluation, the RetroAchievements web API marshalling (`rapi`), game-identification hashing (`rhash`), and `rc_client`. Tag-pinned `v12.5.0`. **No consumer links it yet**; it is shipped so one can. 273 KiB as shipped (279,580 bytes, stripped; the staging copy is 330 KiB unstripped). Upstream ships no Unix build system, so the package compiles it directly like `lzma-sdk`. Three sources are excluded (`rc_libretro.c` needs `<libretro.h>`; `rc_client_external.c` and `rc_client_raintegration.c` are empty translation units without their Windows/external-ABI defines). |

Two properties of this one are worth knowing before linking it:

- **The SONAME is the full version, and here that is not a precaution —
upstream has changed caller-allocated public struct layouts in *patch*
releases.** `v10.7.1` added `char owns_self;` to the public `rc_runtime_t`;
`v10.3.3` inserted `const char* display_name;` into the *middle* of
`rc_api_login_response_t`; `v6.0.1` redefined `RC_ALIGNMENT` from `8` to
`sizeof(void*)`, which changes padding library-wide on a 32-bit target like
this one. Minor releases do it too (`v12.4.0` grew `rc_client_user_t`).
Derived by diffing `include/` across all 55 upstream tags. Same
`/media/fat` stale-binary hazard as `lzma-sdk`, with harder evidence: every
bump must be a clean refuse-to-load, so every bump is a rebuild-your-
consumer event.
- **It is built `-DRC_SHARED -fvisibility=hidden`, so only the 264
`RC_EXPORT`-annotated entry points are exported** (477 without). This is
not tidiness: the suppressed symbols include `md5_init`/`md5_append`/
`md5_finish` and six `AES_*` entry points, and ELF interposition is
first-definition-wins, so exporting those into a process that has its own
md5 — which Main does, `lib/md5` — is a silent wrong-function bind rather
than a link error. It is the same guarantee libchdr gets from its version
script, obtained a different way. Patch `0001` is load-bearing for it:
`rc_util.h` is the one public header upstream left un-annotated, so without
it nine declared entry points would be hidden and the installed headers
would promise an API the `.so` does not export.

`rcheevos.pc` carries **`-DRC_CLIENT_SUPPORTS_HASH` and `-DRC_IMPORT` in its
`Cflags`**, and a consumer must not drop them: the first gates *public header
declarations* (`rc_client.h` lines 265 and 333, the
`rc_client_begin_identify_and_load_game` family), so without it a consumer's
`rc_client.h` hides functions the `.so` genuinely exports.

### Its two vendored files, and why neither is unbundled

rcheevos has no `deps/` or `third_party/` directory, so it *looks* like it is
all upstream's own code. It is not, quite: two vendored third-party
implementations are compiled into the shipped `.so`, and both carry their own
grant, which is why `RCHEEVOS_LICENSE` is `MIT, Zlib (md5), Unlicense
(tiny-AES-c)` and not plain MIT.

| File | Origin | Used by | Unbundle to `/usr/lib`? |
|---|---|---|---|
| `src/rhash/md5.{c,h}` | L. Peter Deutsch / Aladdin, 1999–2002; the notice is the **zlib licence verbatim** (upstream Buildroot spells this same file `Zlib (md5)` in `package/rtty`) | 12 files across `rhash`, `rapi` and the runtime | **No** — see below |
| `src/rhash/aes.{c,h}` | `kokke/tiny-AES-c` @ `f06ac37`, "with unused code excised", Unlicense | `hash_encrypted.c` only | **No** — the question is whether to compile it at all |

**md5 → OpenSSL `libcrypto.so.3`** (which this image already ships) was
considered and **rejected**. OpenSSL 3.x deprecates the low-level
`MD5_Init`/`MD5_Update`/`MD5_Final` API in favour of `EVP`, which is a
different shape from the incremental `md5_init`/`md5_append`/`md5_finish` that
rcheevos threads through 12 files — so this is an invasive, permanently-carried
patch re-validated on every monthly bump. The maintenance argument that
normally justifies unbundling does not apply either: `md5.c` has changed
**twice in rcheevos's entire history** (last 2023-02-23) and `md5.h` **once**
(2020-01-04), and MD5 here is content identification, not security, so there is
no CVE stream to track. The one argument that survives is **performance** —
OpenSSL has hand-written ARM assembly MD5 where this is plain C, which could
matter when hashing a full disc image. That is **unmeasured and deliberately
left so**: it cannot be timed honestly under `qemu-arm` (which JITs asm and C
alike), it needs the rig, and nothing links rcheevos yet. Revisit it with a
real measurement when a consumer exists, not before.

**aes** is not an unbundling question at all. `hash_encrypted.c` is **entirely
Nintendo 3DS** — every function in it is `rc_hash_nintendo_3ds*` (CIA/NCCH/3DSX
container decryption) — so on a board with no 3DS core the honest option is not
"link a different AES" but `-DRC_HASH_NO_ENCRYPTED`, which upstream supports as
a first-class switch. **Measured: that saves 8,088 bytes** (279,580 → 271,492
stripped). It is left **on**, for two reasons: 8 KB is noise against the size
budget, and narrowing a library that nothing links yet is the wrong default,
since a consumer cannot turn it back on without a rebuild. Note it would not
reduce "AES in the image" anyway — `libcrypto.so.3` ships a much larger one.
If it is ever wanted off, the define is **ABI-visible** (it gates declarations
in `include/rc_hash.h` lines 100–126 and 141), so it must go in `rcheevos.pc`'s
`Cflags` as well as the build, exactly like `RC_CLIENT_SUPPORTS_HASH`.

Nothing else is vendored: `hash_zip.c` walks the zip central directory and
never inflates, so there is no bundled zlib/miniz to unbundle — unlike
`libchdr`, whose whole patch series exists for that.

## How Main links these

Against the Buildroot staging sysroot, by pkg-config name — no hardcoded
Expand Down Expand Up @@ -73,6 +163,13 @@ Mapping from Main's vendored `lib/` dirs to their replacements
- **The stage-1 initramfs stacks / installer defconfig are deliberately
unchanged** — static busybox, no ABI surface, nothing there links any of
these.
- Renovate manages both new pins (`lib-pin` label); hashes auto-refresh via
`renovate-hash-sync.yml` (generic loop for libchdr, a bespoke
release-asset step for lzma-sdk). See `docs/renovate.md`.
- Renovate manages all three BR2_EXTERNAL pins (`lib-pin` label); hashes
auto-refresh via `renovate-hash-sync.yml` (generic loop for libchdr and
rcheevos, a bespoke release-asset step for lzma-sdk). See
`docs/renovate.md`.
- **rcheevos has no Main-side mapping row above and should not get one**
until something links it. `scripts/check-abi.sh` stays untouched for it for
the same reason it does for the other five, only more so — no shipped
binary DT_NEEDs `librcheevos.so.12.5.0`. Its only CI cover is the
presence-in-rootfs assertion in `scripts/ci-tests.sh`, which is deliberate:
a library with no consumer has no other way to fail visibly.
Loading
Loading