update_all/Downloader are slower on our Python 3.14: upstream's compiled fast path is pinned to /usr/bin/python3.9, and the fallback zipapps recompile from source every run

[mcfbytes]

docs/python-compat.md §3.6 left one item open when P3.9 was resolved: "downloader.sh's
hardcoded /usr/bin/python3.9 fast-path gate". That gate turns out to be half of a
measurable, every-run slowdown in Update All on our image, and the other half is a
consequence of it. Both are now measured.

We ship Python 3.14.7; stock ships 3.9. Nothing is broken — the tools run and
produce correct results — but they take a path upstream intended as the fallback.

Cause 1 — the precompiled fast path is pinned to /usr/bin/python3.9

downloader_bin is not a Python file. It is a Nuitka-compiled native ARMv7 binary,
built in Downloader_MiSTer/src/Dockerfile.nuitka from arm32v7/python:3.9-bullseye
with CFLAGS="-Os -march=armv7-a", then in Dockerfile.downloader_bin:

RUN nuitka --lto=yes --follow-imports --python-flag=no_site --python-flag=-OO \
    --include-package=downloader --nofollow-import-to=orjson __main__.py
RUN upx --best __main__.bin

Note what is absent: --standalone. Nuitka's accelerated mode links the binary against
the CPython runtime it was built with, so downloader_bin needs a real Python 3.9
installed to run at all. Upstream knows this and gates on it — in two independent places:

Downloader_MiSTer/downloader.sh:161
    if [[ -s "${LATEST_BIN_PATH}" && -x /usr/bin/python3.9 ]] ; then

Update_All_MiSTer/src/update_all/constants.py:203
    DOWNLOADER_LATEST_BIN_PYTHON_COMPATIBLE: Final[str] = "/usr/bin/python3.9"

Update_All_MiSTer/src/update_all/downloader_utils.py:28
    if consider_bin and file_system.is_file(DOWNLOADER_LATEST_BIN_PATH) \
       and file_system.is_file(DOWNLOADER_LATEST_BIN_PYTHON_COMPATIBLE):

On this image /usr/bin/python3.9 does not exist, so the test is false every time and
both tools fall through to the pure-Python zipapp. The upstream Dockerfile's own comments
show how much they care about this path — they weigh --standalone against "saving around
150-400ms" and UPX against "around 150ms" — so losing it entirely is not nothing.

Cause 2 — the fallback zipapps ship source only, and zipimport cannot cache bytecode

Both projects build a PEP 441 zipapp containing .py source and nothing else. This is
deliberate in both build scripts:

Update_All_MiSTer/src/build.sh
    find build/src -type f ! -name '*.py' -exec rm -f {} +
    python -m zipapp build/src -o build/update_all.pyz -p "/usr/bin/env python3"

Downloader_MiSTer/src/build.sh
    zip -q -0 -D -X -A -r "${DOWNLOADER_ZIP}" __main__.py ... -x "*/__pycache__/*"

Verified against the live artifacts: the current dont_download2.sh payload is 67 .py
files, 1,056,177 bytes uncompressed, zero .pyc; the Downloader payload is 90 .py
files, 492,032 bytes, zero .pyc.

zipimport will happily load .pyc from inside a zip, but it never writes one — there
is no __pycache__ for zip-imported modules and no supported way to make one. So on our
image every module is parsed and compiled from source on every single invocation, and
the work is thrown away when the process exits. On stock this cost mostly does not arise,
because the fast path above is taken instead.

(Our own stdlib is unaffected — BR2_PACKAGE_PYTHON3_PYC_ONLY=y means it is already
shipped as bytecode. The tax is purely the applications' own modules.)

Measurements

Importing every module in each package, on the real target interpreter
(output/target/usr/bin/python3.14, Python 3.14.7) under qemu-arm -L output/target —
the method docs/python-compat.md §2 established. Best of three, all runs within ±3%.

Variant	Update All (64 modules)	Downloader (87 modules)
As shipped — source-only zipapp	4.03 s	2.46 s
Extracted to a directory, cold (compiles + writes __pycache__)	2.39 s	—
Extracted to a directory, warm __pycache__	0.61 s	0.53 s
Repacked as a .pyc-only zipapp for 3.14	0.63 s	—
Bare interpreter startup (python3 -c pass)	0.15 s	0.15 s

So roughly 3.4 s + 1.9 s ≈ 5.4 s of pure recompilation per update_all run, recurring,
and Update All invokes the Downloader more than once in a typical run.

Two honesty caveats:

1. These are qemu-user numbers, not hardware numbers. The ratio (~6.6x for Update All,
   ~4.7x for the Downloader) should hold on real silicon since both paths are CPU-bound on
   the same interpreter, but the absolute seconds are not a DE10-Nano measurement. I could
   not take one: the rig at 192.168.0.229 rejects our key, which is exactly the
   reflash-wipes-authorized_keys problem in Standardize authorized_keys on /media/fat/config, the location security_fixes.sh already uses #183. An on-rig measurement is owed before
   this is closed.
2. Importing all modules is an upper bound; a real run may import a subset.

Also unmeasured: how much slower interpreted CPython is than the Nuitka -OO/LTO binary on
the Downloader's actual workload (parsing a multi-megabyte db.json, hashing thousands of
files). That is plausibly the larger of the two losses, and nothing below addresses it —
see the revised option 5 for why the upstream route that would have is closed.

Options

1. Ship /usr/bin/python3.9 alongside 3.14 — reject. Buildroot's python3 package is
a single version, so this means a bespoke second-interpreter package carrying an
end-of-life interpreter (3.9 went EOL in October 2025) plus a second stdlib. And it is a
gamble even then: downloader_bin is linked against Debian bullseye's CPython 3.9, so our
Buildroot 3.9 would have to match that ABI closely enough to load. Large bloat, permanent
security surface, uncertain payoff.

2. Symlink /usr/bin/python3.9 → python3.14 — reject, actively harmful. The gate
would pass and downloader_bin would be copied and executed, then fail to load its
libpython3.9 runtime. downloader.sh:171-173 treats that as an error: it prints
WARNING! downloader_bin didn't work as expected and writes
Scripts/.config/downloader/downloader_bin_error.log on every run. We would trade a quiet
slow path for a noisy slow path. This one is worth writing down precisely because it is the
obvious-looking fix.

3. Cache the bytecode on-device — recommended. Extract the zipapp payload once into a
cache directory keyed by the SHA-256 of the payload, keep the .py files, and run the
directory instead of the .pyz. CPython then manages __pycache__ itself with its normal
per-file source-mtime/size validation, and a new upstream release simply produces a new
hash and a new directory. Measured steady state is 0.61 s vs 4.03 s. Cost is about
2 MB of extracted source per tool on exFAT and a one-time ~2.4 s first run. Zero image
bloat beyond a shell script.

4. Repack as a .pyc-only zipapp — equivalent performance, worse ergonomics. Measured
0.63 s, statistically the same as option 3. But we would own magic-number bookkeeping and
lose CPython's automatic invalidation. Prefer 3.

5. Upstream changes — investigated before filing, and largely closed off. This was
originally written as two asks: (a) replace the hardcoded /usr/bin/python3.9 literal with
a capability test, and (b) publish a --standalone Nuitka build. Reading upstream's own
history first showed both are weaker than they looked:

• (b) has already been tried and deliberately dropped. downloader_bin originates in
  MiSTer-devel/Downloader_MiSTer#50,
  where theypsilon built exactly this --standalone variant and measured it: "There is a
  problem also with the binary size, currently at around 13mb. It would be best if we can
  trim it to sub 4mb somehow." He then moved to the accelerated + UPX build that ships
  today, which the latest release puts at 1.45 MB. Proposing --standalone means
  asking him to reverse a measured decision. The Dockerfile comment quoted earlier in this
  issue is the residue of that decision, not an invitation.
• (a) cannot help this image. The binary is Nuitka-accelerated, not standalone, so it
  links libpython3.9 and loads C extension modules from the 3.9 stdlib.
  /usr/bin/python3.9 is therefore an accurate predictor of whether it will run, not a lazy
  one. Turning it into a capability test would correctly handle a 3.9 installed somewhere
  other than /usr/bin, and would do nothing whatsoever for a 3.14 system.

What remains plausible upstream is a third thing that is neither (a) nor (b): have the
zipapp fallback cache its bytecode — extract once into a hashed cache directory under
Scripts/.config/downloader/ and run the directory, so CPython can persist __pycache__.
That is option 3 applied upstream instead of locally. It needs no build-pipeline change,
adds nothing to the download, is a no-op for stock 3.9 users, and matches #50's own stated
goal ("could make downloader 5secs faster").

No upstream PR has been filed, and none should be filed unilaterally. @mcfbytes is
raising this with theypsilon on Discord first, so the shape of any upstream change is his
call rather than ours.

6. Do nothing, document it. ~5 s on a run that is otherwise minutes of network I/O is
not a catastrophe, and it costs no maintenance. This is the honest baseline the other
options have to beat.

Recommendation

Take 3 now — it is entirely within our image, needs no upstream cooperation, and the
measurement says it removes ~85% of the interpreter cost. 5 is no longer a PR we can
usefully file on our own initiative; see the revised section above. If the Discord
conversation produces an upstream appetite for caching the zipapp fallback, our local hook
can be dropped in favour of it. Explicitly reject 1 and 2, and record why, so
neither gets re-proposed.

The open design question for 3 is where the hook goes, because upstream's launchers do
the extraction themselves (tail -n +8 "${0}" | xzcat -d -c > /tmp/update_all.pyz) and we
do not own Scripts/update_all.sh:

• (i) A sitecustomize.py path hook that gives zipimport a persistent bytecode cache.
  Transparent, uses a documented CPython startup hook, no wrapper binaries — but it changes
  import machinery for every Python process on the system, so it needs an env-var kill
  switch and careful review.
• (ii) An opt-in helper in the FAT payload. We already ship
  board/mister/de10nano/fat-payload/Scripts/ (check_storage.sh, pair_logitech.sh,
  update_linux_modernization.sh), so a update_all_cached.sh alongside them is
  established practice. Safe and obvious, but opt-in, so most users never get the win.
• (iii) Nothing, if 5 lands.

My inclination is to prototype (i) behind an env var, measure it on the rig, and ship (ii)
regardless as the no-magic path.

Work items

• Reproduce the table above on real hardware; unblock the rig first (Standardize authorized_keys on /media/fat/config, the location security_fixes.sh already uses #183).
• Measure a full update_all.sh run end to end, so the 5.4 s is expressed as a
  percentage of something real rather than in isolation.
• Measure the Downloader's interpreted-vs-Nuitka gap on a real db.json, to size
  option 5 against option 3.
• Prototype (i) and (ii); decide.
• Outcome of @mcfbytes' Discord conversation with theypsilon: is there upstream
  appetite for caching the zipapp fallback (the third option under 5)?
• Fold the outcome into docs/python-compat.md and close out its §3.6 open item.

[mcfbytes]

Correction to option 5 — checked upstream history before filing anything, and the
proposal does not survive it. The option 5 section in the body above has been revised;
this comment records why, since body edits are silent.

5(b), the --standalone Nuitka build, has already been tried and deliberately dropped.
downloader_bin originates in
MiSTer-devel/Downloader_MiSTer#50
("Compile downloader into a native binary"), where theypsilon built exactly the variant
this issue was about to propose, with --standalone --python-flag=no_site --include-module=_json --include-package=downloader --follow-imports, and measured it:

After one final attempt, it seems we are saving around 1.5 secs (branch WIP). The trick
was the flag --include-module=_json. […] There is a problem also with the binary size,
currently at around 13mb. It would be best if we can trim it to sub 4mb somehow, by
tweaking the flags further or looking for other approaches.

He then found "a solution that looks promising", which is the accelerated + UPX build
shipping today — the latest release asset is 1.45 MB. So the size comment in
Dockerfile.downloader_bin that this issue quoted approvingly ("will make the
downloader_bin setup more robust in the event of a linux update") is the residue of a
decision already taken against it, not an open invitation. Filing that PR would have asked
the maintainer to reverse a measured call, from someone who had evidently not read his own
issue tracker.

5(a), the capability test, cannot help this image either. I had assumed the
/usr/bin/python3.9 literal was over-strict. It is not: the binary is Nuitka-accelerated
rather than standalone, so it links libpython3.9 and pulls C extension modules from the
3.9 stdlib. The literal is an accurate predictor of whether it will run. Generalising it
would fix a 3.9 installed outside /usr/bin and change nothing at all for a 3.14 system.

What is left. A third option neither (a) nor (b): have the zipapp fallback cache its
bytecode upstream, the same mechanism as option 3 in the body, which costs nothing in
download size and no build-pipeline change. That is worth raising, but as a conversation
rather than an unsolicited PR — @mcfbytes is taking it to theypsilon on Discord.

No upstream PR has been filed. The measurements in the body stand unchanged; only the
upstream half of the plan does.