WASM: Host a WASM testbed site under the QuantEcon org #950

Activity

1. oyamad

   added this to the JupyterLite / WASM browser support milestone on Sep 7, 2026
2. oyamad

   assigned

   mmcky

   on Sep 7, 2026
3. oyamad

   added

   enhancement

   infrastructureCI, build, packaging, and repo toolingCI, build, packaging, and repo tooling

   on Sep 7, 2026
4. oyamad

   added a parent issue on Sep 7, 2026

   • WASM: browser support for QuantEcon.py (Numba + JupyterLite) — tracking issue #925
5. mmcky commented on Sep 9, 2026

   

   mmcky

   on Sep 9, 2026

   Contributor

   More actions

   Priority raised — #950 is now the lead item of #925. The argument that moved it is not the one this issue was opened on. Being able to build a patched numba and run it in a real kernel is not just how we check our own claims; it is how we design and test the patches we contribute upstream. Two of the campaign's five gaps (#927, #944) are blocked on upstream fixes we have already diagnosed but cannot build or test, so today we can only report bugs. With this testbed we can send fixes with browser evidence attached. #925's work plan and sequencing are updated.

   To make the first contribution concrete, here is a source audit of the #927 symbol mismatch, done 2026-09-09. The fix is a single alias, and it is complete.

   numba/core/base.py::_load_global_helpers registers every helper from both helper dicts under a uniform name:

   for c_helpers in (_helperlib.c_helpers, _dynfunc.c_helpers):
       for py_name, c_address in c_helpers.items():
           c_name = "numba_" + py_name
           ll.add_symbol(c_name, c_address)

   Checking each family against the C symbol it actually resolves to:

   Helper family	Entries	C symbol	Registered as	Match
   _helperlib.c_helpers	158	numba_<key> (via _declpointer(#func, &numba_##func) in _helpermod.c)	numba_<key>	Yes
   _nrt.c_helpers	25	NRT_<name>	NRT_<name>	Yes
   _dynfunc.c_helpers	1 — make_generator	Numba_make_generator (via _declpointer(#func, &Numba_##func) in _dynfuncmod.c)	numba_make_generator	No

   So make_generator is the sole helper in numba whose registered name differs in case from its C symbol — it is the only entry in _dynfunc.c_helpers, and the capital Numba_ prefix is an outlier that no other helper uses. Natively this never surfaces, because ll.add_symbol bridges the two spellings inside the JIT's symbol table and the C symbol stays hidden. On Emscripten the generated side module resolves helper imports through the dynamic loader instead, and 0002-export-runtime-helpers-for-wasm.patch exports only the capital spelling — so exactly one import goes unresolved, and because the wasm linker runs with --allow-undefined it fails at first call rather than at link time. That is the hang.

   The consequence for the patch is that no survey of other helpers is needed: aliasing this one name is sufficient and nothing else is exposed. Candidate, to be added to patch 0002 after the definition of Numba_make_generator in _dynfunc.c:

   #ifdef __EMSCRIPTEN__
   __attribute__((visibility("default"), alias("Numba_make_generator")))
   PyObject *numba_make_generator(Py_ssize_t gen_state_size, void *initial_state,
                                  PyCFunctionWithKeywords nextfunc,
                                  gen_finalizer_t finalizer,
                                  EnvironmentObject *env);
   #endif

   Caveats, stated plainly because they are the point of this issue. The audit above is source-verified, not browser-verified — it is read from numba 0.62.1 sources and the emscripten-forge patch set, and the browser build is 0.67.0. Neither the alias above nor the thin-forwarding-function alternative has been compiled, and __attribute__((alias)) under the wasm backend is precisely the kind of thing that needs to be tried rather than reasoned about. There is also a design question worth settling before we file: the mismatch is latent in numba itself, so the durable fix may belong upstream in numba (make _dynfunc.c consistent with _helperlib.c) rather than in an emscripten-forge patch that compensates for it. @anutosh491 offered on 2026-08-19 to take this one; that offer is still open and worth answering either way.

   All of which is the case for building this first: right now every sentence in the paragraph above ends in "we cannot check that yet".

   Refs: #925, #927, #928, #933
6. mmcky commented on Sep 9, 2026

   

   mmcky

   on Sep 9, 2026

   Contributor

   More actions

   Implementation mechanics: verified against live sources

   Verified 2026-09-09 by downloading the actual artifacts and running the actual tools — micromamba 2.9.0, jupyterlite-xeus 5.0.0, jupyterlite-core 0.8.3, empack 6.0.1, rattler-index 0.31.5 — rather than from documentation. The implementation notes hold up better than I expected: most of what follows confirms them, one of their open questions is now answered, and the whole channel mechanic was proven end to end. Two things are wrong, both in ways that would silently ship stock numba rather than fail loudly.

   Two corrections

   The artifact is a .tar.bz2, not a .conda. Every emscripten-forge pixi build task passes --package-format tar-bz2; emci/rattler_build.py hardcodes the same; all 2735 packages in the emscripten-wasm32 repodata are .tar.bz2 and packages.conda is present-but-empty (count 0); emci/upload.py rejects .conda outright. The real file is numba-0.67.0-py313h02b136b_1.tar.bz2, 1,318,840 bytes, sha256 6842b3db…7584d; the .conda URL 404s. This matters more than a filename nit, because a *.conda glob in the build workflow matches nothing, and copying zero packages into channel/ still produces a valid, correctly-indexed, empty channel — so the build succeeds and ships stock numba. That is exactly the failure the notes warn about under "Make the patched package win the solve", reached through a different door. The size figure is also good news for "commit the package into the repo": 1.3 MB per patched build, so twenty candidate rebuilds cost about 26 MB of history.

   "Make the patched package win the solve" reaches the right remedy by the wrong mechanism. The note says listing the channel first is not enough because "at equal version the solver prefers the higher build number". The solver is micromamba, and under its default flexible priority and under strict, libsolv runs prune_to_highest_prio before prune_to_best_version: the first-listed channel wins regardless of version or build number. This was demonstrated both ways — a local build _0 beating upstream's _1, and a synthetic foo 1.0 beating foo 2.0. Build number only decides under channel_priority: disabled. The practical upshot is that the note's parenthetical second option is the stronger one and should be promoted to the primary: pin the build in environment.yml as numba=0.67.0=py313h02b136b_2, which passes through jupyterlite-xeus verbatim and converts "the patched build is in use" from a runtime assertion into a build-time invariant — a missing patched package then fails the build instead of quietly resolving to stock. Bump build: number as well, since jupyterlite-xeus does not pass --no-rc and a stray runner condarc or CONDA_CHANNEL_PRIORITY can flip the mode to disabled, which is the one setting where channel order stops mattering. Setting MAMBA_CHANNEL_PRIORITY=strict in the workflow closes that off directly.

   One trivial nit while here: the recipe path is recipes/recipes_emscripten/numba/, with a leading recipes/.

   The open question in the notes, answered: yes, local paths work

   The notes hedge with "If the xeus build's solver turns out not to accept local paths, the fallback is a quantecon channel on prefix.dev". That fallback is not needed. The solver is micromamba, invoked as a subprocess — jupyterlite_xeus/create_conda_env.py runs micromamba create --yes --no-pyc --prefix <p> --relocate-prefix "" --root-prefix <r> --platform=emscripten-wasm32 -c <chan>… <specs>, passing every channels: entry through verbatim as -c. A grep for rattler, libmambapy or conda.api across the whole 5.0.0 sdist returns zero hits.

   And it was proven end to end, not merely read: downloading the real numba-0.67.0-py313h02b136b_1.tar.bz2, injecting a marker file, rebuilding it as build _2, indexing with rattler-index fs, and running micromamba create --platform=emscripten-wasm32 against the local channel installed that package into a real emscripten-wasm32 prefix, resolving llvmlite 0.49.0, numpy 2.4.6 and python 3.13.1 cleanly from emscripten-forge and python_abi from conda-forge. This also confirms the note under Notes that mixing a recipe-built package with stock emscripten-forge packages is safe: channel priority prunes per package name, so a one-package local channel shadows nothing else. Keep the local channel to exactly the packages under test for that reason.

   The one consequence of it being micromamba: the build job must put micromamba on PATH before jupyter lite build imports the addon, because shutil.which("micromamba") is evaluated at module import and raises if absent. Installing jupyterlite-xeus from conda-forge pulls it automatically; installing from PyPI does not.

   Details behind the notes, worth recording because of how they fail

   • The noarch/ subdir the notes already specify is not optional, and its absence is fatal and misdiagnosable. Without a readable <channel>/noarch/repodata.json micromamba aborts the entire solve — even when the platform subdir is fine and another channel could satisfy the spec — with Download error (37) … Subdir …/noarch not loaded!, which reads like a network fault rather than a channel-layout error. A missing or empty platform subdir, by contrast, is tolerated silently. Worth a comment in the workflow so nobody "fixes" it by adding retries.
   • Write the channel as an absolute file:// URL generated by the workflow, e.g. file://${GITHUB_WORKSPACE}/channel. The notes' ./channel is correctly formed — a bare channel with no leading ./ is typed as an anaconda.org channel name and silently resolves nothing — but a relative path resolves against the jupyter lite build process CWD, not against environment.yml's directory, so it breaks the first time the build moves into a subdirectory.
   • rattler-index flag surface has moved. rattler-index fs <dir> is right, but --write-zst and --write-shards no longer exist in 0.31.5 — they are now [index-config] keys in a TOML passed with --config, and the rattler-build docs page describing the flags is stale. Defaults write sharded repodata that micromamba then ignores for file:// channels, i.e. four extra files churning on every index run. It is also a Rust CLI available from conda-forge for linux-64 and not on PyPI. The step can be skipped entirely: rattler-build publish <pkg>.tar.bz2 --to file://$PWD/channel does copy, index and noarch-init in one command.
   • conda-forge is required, not merely conventional. emscripten-forge-4x ships no python_abi record while numba pins python_abi 3.13.* *_cp313, so the three-channel order in the notes is load-bearing.
   • "The browser never contacts a channel" is true for the packed environment and false for dynamic installs. Packages are baked at build time into {output}/xeus/{env}/kernel_packages/*.tar.gz and the worker fetches only from the Pages origin. But jupyterlite-xeus copies the channel list verbatim into empack_env_meta.json, which mambajs uses to solve in-browser %pip / %conda install calls — so a file:// entry lands in the deployed site's metadata and would break dynamic installs there. Harmless for the smoke notebooks as specified; worth knowing before anyone adds an install cell.
   • A build-time provenance guard is cheaper than the runtime one. The notes' environment-check cell and ctypes marker are right and should stay, but grepping the resulting conda-meta/numba-*.json for the expected build string and the local channel path fails the job an hour before the browser would.

   The site itself was built and driven, not estimated

   The site also builds and runs. From this repo's ci/wasm/environment.yml verbatim: 47 packages, 214.0 MB installed prefix, built site 159.7 MB (116.0 MB with --no-sourcemaps, 90.8 MB as the Pages tar.gz) against GitHub's documented 1 GB published-site limit — roughly 6x headroom. A first-time visitor is served 96.65 MB over 133 requests, about 72-76 MB on the wire since Pages gzips application/wasm but not the already-compressed package chunks; that puts the 100 GB/month soft bandwidth limit at roughly 1,300 cold visits. jupyter lite build took 46.8-48.5 s. In headless Chromium against the built site: import quantecon 4.05 s, first @njit compile 1.49-1.57 s, simplex_grid(3, 4) 0.50 s on a cold cache. The wasm32 memory ceiling is tighter than #925 records — xpython.js declares maximum: 32768 pages, a hard 2 GiB, and a numpy allocation actually failed at 1792 MiB.

   On proving which numba is loaded, there is something better than the ctypes probe. empack ships a per-package /conda-meta/<name>-<version>-<build>.json into the Emscripten filesystem carrying build, build_number, channel and the source url. Confirmed by reading it from a live browser kernel. numba.__version__ yields only 0.67.0 and importlib.metadata no more, so /conda-meta is the provenance cell; the #927 ctypes symbol probe stays as the behavioural marker.

   Two additions to the record

   The Motivation's read of upstream CI checks out; here are the numbers behind it. build_recipes.yaml builds recipes changed between origin/main and HEAD and runs the recipe's own test_numba.py inside three real headless browser backends via Playwright. On emscripten-forge/recipes#6293 the whole job — a genuine from-source rebuild with all ten patches applied, emcc compiling numba._dynfunc / _dispatcher / _helperlib, plus all three in-browser runs — took 3 min 20 s. The "only as part of a PR" qualifier is exact rather than approximate: the workflow's push trigger is restricted to branches: [main], so pushing a patch branch to a fork builds nothing; a pull request is required, though one opened within the fork against the fork's own main is enough and needs no upstream involvement. And the boundary falls precisely where the Motivation puts it — quantecon is absent from the recipe's test environment (#932) and the recipe tests never touch the xeus-python/JupyterLite stack, so "support_enumeration and vertex_enumeration return the correct equilibria under the patched build" is evidence that only this testbed can produce. That is the sharpest single argument for building it, and it is worth promoting into the Proposal explicitly, because it is also the argument that survives if a maintainer asks why the recipe's own tests are not sufficient.

   The #944 trigger is an in-session cache restore, not a browser session boundary. Numba's _object_getbuffer_hook nulls _compiled_object, _unserialize re-sets it, and _get_compiled_object then raises at codegen.py:632 — the exact frame in the traceback. The failure was reproduced in a single native CPython process by monkeypatching only patch 0006's two codegen.py hunks, with no browser and no second session; the session boundary in the existing repro is incidental, its only job being to make the callee a cache hit rather than an in-memory hit. This does not change the case for the testbed, but it does change two things downstream. A regression test for #944 can live in an ordinary pytest file rather than requiring browser orchestration. And a deployed site is the worst place to observe the bug: CI visits cold on every run and will never see it, while real visitors warm their cache and will — so the site can read green while being broken for humans. Deliberate reproduction needs a Playwright persistent context with page.reload() inside one BrowserContext, or a human with a warm profile, and the smoke notebooks should say which state they are asserting rather than leaving it to chance.

   One item to validate during implementation

   Whether /drive is actually mounted on a Pages-hosted JupyterLite. Patch 0006 sets default_cache_dir = "/drive/.cache/numba" if sys.platform == "emscripten" and os.path.isdir("/drive") and otherwise falls back to CACHE_DIR = "", i.e. MEMFS, which is destroyed on every kernel restart. Whether jupyterlite-xeus still gates that mount on serviceWorker?.enabled || crossOriginIsolated could not be settled from source — the gate appears to have been removed in 2024 but the current computation is unclear, and GitHub Pages cannot send COOP/COEP headers.

   This is deliberately left open rather than guessed at, and it is cheap to settle: os.path.isdir('/drive') and numba.config.CACHE_DIR from a cell against a locally served build, then again against the first deployed site. Treat it as a gate on the warm-cache work rather than a background risk — if /drive turns out not to be mounted, the persistent Numba cache does not exist on the deployed site, #944 is unobservable there no matter how the notebooks are written, and the cache-state cell in 00_environment.ipynb should say so plainly instead of implying a warm cache that cannot form. Both probes belong in that notebook permanently, since the answer can change under jupyterlite-xeus upgrades.

   Decisions taken

   Name. The testbed repo will be QuantEcon/test-quanteconpy-wasm — ruled in, and available. It sits under the test-{name} entry in QEP-3's type registry ("test double or CI target for tooling, pilot-scoped or standing"), as a standing target for this repo's WASM work, alongside the eight existing test-* repos. One follow-up for whoever files the rename audit: status-projects calls the project quantecon-py-wasm, so the name segments differ by a hyphen (quanteconpy-wasm here). Not worth blocking on, but worth choosing deliberately.

   Sequencing. Report #927 upstream first — it has never been filed, and searches of emscripten-forge/recipes issues and pull requests for numba_make_generator, make_generator and generator numba all return zero results, so there is currently nothing upstream to assign, label or find. Filing the report commits us to no patch and creates the reference an eventual pull request can point at. Then stand up the testbed, prove each candidate patch there, and only then open the upstream patch pull requests citing the testbed URL. The reasoning behind that order is the paragraph above on what only the testbed can demonstrate: a recipe-level test can show the generator MWE running and the symbol resolving, but not support_enumeration returning the correct equilibria, and the latter is what a reviewer should be shown.

   Registration. status-projects needs only a links.repos addition — the quantecon-py-wasm row already carries a hosted WASM testbed as deliverable 5.

   🤖 Generated with Claude Code