stdlib: load the standard library from S-lineage lib/StLib sources

[pyrex41]

Summary

Retire the community klambda/stlib.kl overlay and make the S-lineage Lib/StLib Shen sources the standard-library source of truth, matching how Tarver's S41.2 (2026-07-11) refresh actually ships the stdlib (as sources loaded into the image at install time, not as a precompiled KLambda blob). This is the follow-up logged by the kernel-refresh PR (#36).

Headline fix: loading the stdlib through the kernel's own (load)/define pipeline registers each function's arity property + shen.*lambdatable* entry, so a bare top-level (filter …) and a first-class (fn filter) now resolve instead of raising "fn: filter is undefined". The old stlib.kl booted as raw KL defuns and never called stlib.initialise, so those functions had runtime arity = -1.

Mechanism (chosen: load-from-source at boot)

boot.lua load_stdlib() runs upstream's own install.shen — its file order, factorise toggles, and the (package stlib …) + (map (fn systemf) …) externals block — with two mechanical rewrites:

1. Its relative (load "Sub/file.shen") paths are made absolute against the vendored dir, so no process chdir is needed (a chdir would invalidate the KLDIR-relative reads the fasl kernel-key hashing does — a real trap I hit).
2. Its (tc +) toggles are neutralised to (tc -): the stdlib loads without typechecking. This matches the pre-refresh behaviour (the old stlib.kl registered no stdlib type signatures either), is markedly faster, and keeps the native typecheck drivers deferred at boot. Functions are still fully defined and arity-registered.

load_stdlib() is called from initialise() — the single post-kernel-init chokepoint every boot path runs (CLI, port specs, kernel certification) — so the stdlib is present for all of them. SHEN_NO_STDLIB=1 opts out (kernel-only embed); SHEN_STDLIB_DIR overrides the location.

Why not build-time bake / fasl cache?

I investigated caching the loaded stdlib. It is not fasl-serialisable on the refreshed kernel: the refresh registers lambda entries via set(shen.*lambdatable*, …) / set(shen.*sigf*, …) with function-valued globals, which the fasl set recorder can't kdata_ser. (The old kernel used put(name, shen.lambda-form, …), which the fasl special-cases via its "lf" record.) Rather than rewrite core fasl machinery, load-from-source at boot is the low-risk choice; each CI suite boots once, so the cost is a non-issue there. Making stdlib fasl-cacheable is a clean future optimisation.

What changed

• Vendored the S-lineage stdlib sources under lib/StLib/ (28 files, byte-identical to mirror pyrex41/shen-upstream tag s41.2-pristine-20260711 / commit 11fc51b, formerly pyrex41/shen-s41.1). See lib/StLib/PROVENANCE.md.
• Deleted klambda/stlib.kl; dropped it from the boot FILES list (kernel is now 19 vendored .kl).
• boot.lua: load_stdlib() + find_stdlib_dir(); wired into initialise().
• Single-file bundle: make-bundle.lua embeds the lib/StLib tree; load_stdlib materialises it to a temp dir at boot, so the bundle stays self-contained (and is smaller — the compact .shen sources replace the 366 KB stlib.kl; kernel blob 1.48 MB → 652 KB, bundle 5.3 MB → 2.6 MB).
• Tests: new test/stdlib_spec.lua locks the (fn filter) / bare-(filter …) regression; test/library_spec.lua's "filter/take absent" assertions flipped to present-and-correct; registered the new spec in scripts/run-tests.lua.
• Docs: lib/StLib/PROVENANCE.md (new), klambda/PROVENANCE.md, klambda/README.md, README.md.

Test status (LuaJIT 2.1)

Gate	Result
Port spec suite (make test)	482 / 0 across 13 specs (was 471/12; +stdlib_spec, +flipped library_spec)
Kernel certification (make certify)	134 / 134 (100%)
(fn filter) / top-level (filter …)	resolve (regression test test/stdlib_spec.lua)
CLI (bin/shen -e '(filter …)')	works
Single-file bundle	builds, boots standalone (no repo present), stdlib live
typecheck_lazy_spec	8/8 — lazy native drivers stay deferred (tc- load)

Notes / trade-offs

• No stdlib type signatures (loaded with tc -): a tc + program calling e.g. filter won't see filter's type. Unchanged from the pre-refresh overlay. A typed stdlib (tc +) is a deliberate future option — slower, and it eagerly initialises the native typecheck drivers.
• Boot cost: load_stdlib adds ~0.6–0.9 s to a cold boot (it is not cached; see "Why not build-time bake"). Each CI suite and CLI start pays it once. Making it cacheable is the natural follow-up.

Note for sibling ports

Any port tracking this kernel that loaded a precompiled stlib.kl should switch to loading the Lib/StLib sources through its own define path — that is what registers arities/lambda entries and makes (fn <stdlib-fn>) work. Load with tc - for parity/speed unless a typed stdlib is wanted. Watch for the chdir vs relative-kernel-path trap if your loader chdirs.

🤖 Generated with Claude Code