Update vendored kernel to Tarver's S41.2 (2026-07-11 refresh)

README.md

@@ -14,9 +14,10 @@ primitives on a host runtime and (b) translating the kernel's `.kl` files into
 that host. This port does both by **compiling KLambda to Lua source** that
 LuaJIT then trace-compiles to machine code.
 
-It targets **Shen 41.2** (via the KLambda in `ShenOSKernel-41.2/klambda`) and
-passes the official 41.2 kernel test suite (134/134). Earlier versions were
-certified against the Shen 22.4 kernel test suite.
+It targets **Shen 41.2** (the KLambda is vendored under `klambda/`; see
+[`klambda/PROVENANCE.md`](klambda/PROVENANCE.md)) and passes the official 41.2
+kernel test suite (134/134). Earlier versions were certified against the Shen
+22.4 kernel test suite.
 
 ## Quick start
 
@@ -64,7 +65,7 @@ not a tree-walker:
 | `runtime.lua`  | data representation, symbol interning, the KLambda reader |
 | `compiler.lua` | KLambda → Lua source compiler (statement-based codegen, tail-call → loop lowering) |
 | `prims.lua`    | runtime env: the primitive set, apply/curry machinery, native overrides, loader |
-| `boot.lua`     | kernel loading, the bytecode + fasl caches, `shen.initialise` |
+| `boot.lua`     | kernel loading + initialisation, the bytecode + fasl caches |
 | `shen.lua`     | the public embedding API (`require("shen")`) |
 | `lua_interop.lua` | the Lua ⇄ Shen bridge (`lua.call`, `lua.function`, marshaling) |
 | `repl.lua`     | the interactive REPL (multiline input, error translation, backtraces) |
@@ -103,8 +104,9 @@ designed around what LuaJIT's tracing JIT rewards:
 * **`typecheck_native.lua` — the t-star driver.** The ~16 CPS driver functions
   are machine-translated from `klambda/t-star.kl` through the same translator
   (they share the goal vocabulary); the four that escape it (entry,
-  signature lookup, datatype search, spy display) are hand-ported. The 162
-  kernel signatures are harvested from `init.kl` into a native table. The
+  signature lookup, datatype search, spy display) are hand-ported. The 161
+  kernel signatures are harvested from the `(declare …)` forms in `types.kl`
+  into a native table (pre-refresh kernels harvest from `init.kl` instead). The
   native driver performs the **byte-identical inference sequence** to the
   legacy engine (431,741 inferences on the reference typecheck, exactly).
 * **Legacy native overrides** (`prims.lua`): native Prolog deref core with

boot.lua

@@ -1,5 +1,7 @@
 -- boot.lua : load the full Shen KLambda kernel into the Lua runtime and
--- run (shen.initialise).  Returns the prims module P with everything live.
+-- initialise it. Returns the prims module P with everything live. (On the
+-- S41.2 2026-07-11 kernel the kernel self-initialises at load time; see FILES
+-- and initialise() below.)
 local R = require("runtime")
 local C = require("compiler")
 local P = require("prims")
@@ -69,11 +71,39 @@ local function find_kldir()
 end
 local KLDIR = find_kldir() .. "/"
 P.KLDIR = KLDIR   -- resolved .kl directory (trailing /), for typecheck_native
+-- Boot order for the S41.2 (2026-07-11 refresh) kernel. The first 15 entries
+-- are the refreshed KLambda modules. The refreshed kernel initialises itself
+-- at LOAD time: declarations.kl runs top-level forms — (set *property-vector*
+-- (vector 20000)), the environment `set`s, (shen.initialise-arity-table ...),
+-- (put shen shen.external-symbols ...) and (shen.build-lambda-table ...) — that
+-- the removed init.kl used to run from shen.initialise (see initialise()).
+--
+-- Order is NOT upstream Sources/make.shen order. make.shen relies on the
+-- factorise pass + a macros bootstrap that runs last; shen-lua compiles KL
+-- directly, so what matters is that a module's LOAD-TIME side effects see
+-- their dependencies already defined:
+--   * declarations' top-level init calls put/vector/hash/shen.lambda-entry
+--     (sys), so sys precedes declarations;
+--   * types.kl's 161 top-level (declare ...) forms actually RUN the type
+--     checker at load (each declare infers the signature's variance), so every
+--     function `declare` reaches transitively must already be defined:
+--     shen.prolog-vector (macros.kl), shen.*sigf* + the arity table
+--     (declarations.kl), and — new in the refresh — shen.rectify-type and the
+--     rest of the inference machinery (t-star.kl). Pre-refresh t-star trailed
+--     types; the refresh moved shen.rectify-type into t-star, so t-star must
+--     now precede types. Hence the tail: macros declarations t-star types.
+--
+-- The trailing four are the community ShenOSKernel extensions + stlib, which
+-- Tarver's refresh no longer ships as KLambda (stlib is now lazy Shen sources
+-- under Lib/StLib). shen-lua keeps vendoring them on top so the standard
+-- library and the CLI launcher stay available and the kernel test suite still
+-- certifies. They are pure defuns/defmacros and reference only public kernel
+-- functions (get/put/… — never the removed dict.* or shen.initialise-*), so
+-- they load unchanged against the refreshed kernel. See klambda/PROVENANCE.md.
 local FILES = {
-  "toplevel","core","sys","dict","sequent","yacc","reader","prolog",
-  "track","load","writer","macros","declarations","types","t-star","init",
-  "extension-features","extension-expand-dynamic","extension-launcher",
-  "compiler","stlib"
+  "yacc","core","load","prolog","reader","sequent","sys","toplevel",
+  "track","writer","backend","macros","declarations","t-star","types",
+  "extension-features","extension-expand-dynamic","extension-launcher","stlib"
 }
 
 -- ---- standard streams ----------------------------------------------------
@@ -93,7 +123,7 @@ P.GLOBALS["*implementation*"] = rawget(_G, "jit") and "LuaJIT" or _VERSION
 P.GLOBALS["*port*"]           = "shen-lua"
 P.GLOBALS["*porters*"]        = "shen-lua contributors"
 P.GLOBALS["*os*"]             = (package.config and package.config:sub(1,1) == "\\") and "Windows" or "Unix"
-P.GLOBALS["*release*"]        = "0.1"  -- port release; kernel *version* comes from init.kl ("41.2")
+P.GLOBALS["*release*"]        = "0.1"  -- port release; kernel *version* comes from declarations.kl ("41.2")
 
 -- ---- kernel bytecode cache -------------------------------------------------
 -- Loading the kernel from .kl sources costs ~0.8s (read + parse + KL->Lua
@@ -282,10 +312,11 @@ local function read_cache(path, key)
 end
 
 -- ---- load the kernel -----------------------------------------------------
--- Loads the 21 .kl modules that make up Shen 41.2 (core + stlib + extensions;
--- same boot list as shen-cl 41.2 — the opt-in
--- extension-programmable-pattern-matching.kl is vendored but not booted).
--- The 41.2 KLambda sources are vendored under `klambda/` so the repository
+-- Loads the 19 .kl modules in FILES (see above): the 15 refreshed S41.2
+-- (2026-07-11) KLambda modules plus the vendored community stlib + 3 booted
+-- extensions. The opt-in extension-programmable-pattern-matching.kl is
+-- vendored but not booted.
+-- The KLambda sources are vendored under `klambda/` so the repository
 -- is self-contained. You can still override with SHEN_KL_DIR (e.g. to point
 -- at a full ShenOSKernel checkout during development).
 
@@ -800,13 +831,20 @@ end
 
 -- ---- initialise ----------------------------------------------------------
 local function initialise()
-  -- (shen.initialise) sets up the environment, lambda-form tables, etc.
-  local sym = R.intern("shen.initialise")
+  -- Kernel environment setup (env globals, *property-vector*, arity table).
+  --
+  -- On the S41.2 (2026-07-11 refresh) kernel this all happens at LOAD time via
+  -- top-level forms in declarations.kl — there is no `shen.initialise` function
+  -- to call, so load_kernel() has already done it by the time we get here.
+  --
+  -- Older kernels (pre-refresh community ShenOSKernel, reachable via
+  -- SHEN_KL_DIR) instead define `shen.initialise` and expect it to be called
+  -- once, post-load. Preserve that path when the function is present.
+  local r
   local fn = P.F["shen.initialise"]
-  if not fn then error("shen.initialise not defined after kernel load") end
-  local r = fn()
+  if fn then r = fn() end
   -- Register the lua.* interop entries in Shen's own arity/lambda-form
-  -- tables (needs the *property-vector* that shen.initialise just created).
+  -- tables (needs the *property-vector* the kernel just created).
   require("lua_interop").post_initialise()
   return r
 end

build/make-bundle.lua

@@ -70,7 +70,8 @@ do
     if nm then kl_names[#kl_names + 1] = nm end
   end
   p:close()
-  assert(#kl_names >= 21, "klambda/ looks incomplete (" .. #kl_names .. " files)")
+  -- 20 vendored .kl: 15 refreshed S41.2 kernel modules + stlib + 4 extensions.
+  assert(#kl_names >= 20, "klambda/ looks incomplete (" .. #kl_names .. " files)")
 end
 
 -- ---- 3. emit ---------------------------------------------------------------

klambda/PROVENANCE.md

@@ -1,21 +1,105 @@
 # Provenance of the vendored KLambda kernel
 
-## Release
+The vendored `klambda/` tree has **two lineages**. Read both sections — they
+come from different upstreams and are updated independently.
 
-- **Release**: ShenOSKernel-41.2
-- **Tag/URL**: https://github.com/Shen-Language/shen-sources/releases/tag/shen-41.2
-- **Zip SHA-256**: `49f1b85d02348d9b3ebc461570c5c56cc066270ab81e35d5257625fb9d17fe82`
+## 1. Kernel proper — S41.2 (2026-07-11 refresh)
 
-All `.kl` files from the release `klambda/` directory are vendored here
-**byte-identical** to the release (verified with `cmp` against the extracted
-zip). This includes the new opt-in extension
-`extension-programmable-pattern-matching.kl`, which is vendored but **not**
-on the boot list — shen-lua boots the same 21 modules as shen-cl 41.2.
+- **Canonical source**: `pyrex41/shen-s41.1` — the designated mirror of Mark
+  Tarver's shenlanguage.org uploads (private repo).
+  - Tag: `s41.2-pristine-20260711`
+  - Commit: `11fc51bdf53a4dcb505adeec6ec8352754cbe50f`
+    ("Pristine import of the 2026-07-11 S41.2 refresh from shenlanguage.org")
+- **Upstream origin** (what the mirror imported): Mark Tarver's `S41.2.zip`,
+  the reference SBCL/Windows distribution.
+  - URL: https://www.shenlanguage.org/Download/S41.2.zip
+  - Last-Modified: 2026-07-11
+  - Zip SHA-256: `51becbfd60fa8c93c3f8ae5b20b948eaa84c4b1d14ad2f5d2a056002a53ee836`
 
-## compiler.kl caveat
+These 15 files are vendored **byte-identical** to `KLambda/` in the mirror at
+that tag — equivalently, to the zip's `KLambda/` directory (both verified with
+`cmp`):
 
-`compiler.kl` is **not** part of the ShenOSKernel release. It is shen-cl's
-generated KL->Lisp compiler (a shen-cl build artifact), vendored here from a
-freshly generated 41.2 shen-cl build
-(`cl-source/ShenOSKernel-41.2/klambda/compiler.kl`,
-SHA-256 `ed30a08be4c8916b1e844d437fb9d65a36476e1a99419340b5523fc81a7c3e44`).
+    yacc core load prolog reader sequent sys t-star toplevel
+    track types writer backend declarations macros
+
+> **Caveat — the "41.2" version number was reused.** Upstream re-uploaded a
+> *restructured* kernel under the same `41.2` version. This is a **different
+> lineage** from the community `ShenOSKernel-41.2`
+> (github.com/Shen-Language/shen-sources, tag `shen-41.2`,
+> zip SHA-256 `49f1b85d02348d9b3ebc461570c5c56cc066270ab81e35d5257625fb9d17fe82`)
+> that this file previously described. We call the current one
+> **"S41.2 (2026-07-11 refresh)"** to disambiguate. `(version)` still reports
+> `"41.2"` (it is set in `declarations.kl`).
+
+### What changed vs the community ShenOSKernel-41.2 we used to vendor
+
+- **New: `backend.kl`** — a `cl.*` KLambda→Common-Lisp backend. Irrelevant to
+  the Lua runtime, but on the upstream boot list, so vendored and booted (it is
+  pure defuns; it defines functions that are never called under Lua).
+- **Removed: `compiler.kl`** — was shen-cl's generated KL→Lisp compiler (a
+  shen-cl build artifact, never part of any ShenOSKernel release). shen-lua has
+  its own Lua compiler (`compiler.lua`) and never used it at runtime. Dropped.
+- **Removed: `dict.kl`** — the dictionary layer is gone. Property vectors are
+  now plain vectors: `*property-vector*` is `(vector 20000)` and `get`/`put`
+  index it via `hash` + `shen.change-pointer-value` / `shen.remove-pointer`,
+  rather than the old `shen.dict` / `shen.<-dict` / `shen.dict->`. Internal to
+  `get`/`put`; callers are unaffected.
+- **Removed: `init.kl`** — its work moved into `declarations.kl` and
+  `toplevel.kl`. There is **no `shen.initialise` function** any more: the
+  kernel initialises itself at LOAD time via top-level forms in
+  `declarations.kl` — `(set *property-vector* (vector 20000))`, the environment
+  `set`s, `(shen.initialise-arity-table …)`, `(put shen shen.external-symbols …)`
+  and `(shen.build-lambda-table …)`. `shen.initialise-lambda-forms` /
+  `-signedfuncs` / `-environment` are gone; `shen.initialise-lambda-tables`
+  (~renamed) and the arity table remain.
+- **Removed: `stlib.kl`** — the standard library is no longer shipped as a
+  precompiled KLambda blob; upstream now ships it as **lazy Shen sources** under
+  `Lib/StLib/` to be loaded on demand. See section 2 for how shen-lua bridges
+  this gap.
+- Other renames observed: `hush` → `shen.hush`, `input+` → `shen.input-h+` /
+  `shen.process-input+`, plus new `shen.rdecons`, `shen.shen`, pointer helpers.
+
+## 2. Standard library + extensions — community ShenOSKernel-41.2 (retained)
+
+Because Tarver's refresh no longer ships a precompiled `stlib.kl` and shen-lua
+does not yet have a lazy `Lib/StLib` loader, these five files are **retained
+byte-identical from the community `ShenOSKernel-41.2`** release
+(zip SHA-256 `49f1b85d02348d9b3ebc461570c5c56cc066270ab81e35d5257625fb9d17fe82`)
+so the standard library and the CLI launcher stay available and the kernel test
+suite still certifies 134/134:
+
+- `stlib.kl` — the precompiled standard library (`filter`, `take`, `drop`,
+  `reduce`, string/list/vector helpers, …).
+- `extension-features.kl`, `extension-expand-dynamic.kl` — booted.
+- `extension-launcher.kl` — booted; provides the launcher the CLI can use.
+- `extension-programmable-pattern-matching.kl` — vendored, opt-in, **not** booted.
+
+These are all pure `defun`/`defmacro` and reference only public kernel functions
+(`get`/`put`/`arity`/…) — never the removed `dict.*`, `shen.<-dict`, or
+`shen.initialise-*` functions — so they load unchanged against the refreshed
+kernel. They are library/Shen-level code, effectively version-stable.
+
+> **Known limitation (pre-existing, not introduced by the refresh).** stlib
+> functions get an `F` entry from their `defun` and a compiler arity from the
+> boot prescan, so they compile as direct calls and work in loaded programs and
+> the test suite. But `stlib.initialise` is never called, so their runtime
+> `arity` *property* stays `-1`; a bare `(fn filter)` / top-level `(filter …)`
+> reference that resolves through the lambda table fails with "fn: filter is
+> undefined". This behaves identically on the pre-refresh kernel. The clean fix
+> is a lazy `Lib/StLib` loader (see the PR's "remaining work").
+
+## Boot order
+
+The boot order lives in `boot.lua` (`FILES`) with a full rationale. It is **not**
+upstream `Sources/make.shen` order: make.shen relies on the factorise pass and a
+macros bootstrap that runs last, whereas shen-lua compiles KLambda directly, so
+what matters is that each module's LOAD-TIME side effects (the `declare` forms in
+`types.kl`, the init forms in `declarations.kl`) see their dependencies already
+defined. The resulting tail is `… macros declarations t-star types`.
+
+## Overriding
+
+Set `SHEN_KL_DIR=/path/to/some/other/klambda` to boot a different KLambda tree
+(e.g. a full ShenOSKernel checkout during development). `boot.lua` uses it
+instead of this vendored directory.

klambda/README.md

@@ -1,27 +1,41 @@
 # Vendored KLambda for Shen 41.2
 
-These are the KLambda kernel sources for **Shen 41.2** (from `ShenOSKernel-41.2`).
+These are the KLambda kernel sources for **Shen 41.2**. They are included
+directly in this repository so that `shen-lua` is self-contained: you can
+`git clone` and run without needing a separate ShenOSKernel checkout.
 
-They are included directly in this repository so that `shen-lua` is self-contained:
-you can `git clone` and run without needing a separate ShenOSKernel checkout.
-
-See [PROVENANCE.md](PROVENANCE.md) for the exact release tag, checksum, and the
-`compiler.kl` caveat (it is a shen-cl build artifact, not part of the release).
+The tree has **two lineages** — the refreshed kernel proper from
+shenlanguage.org, and the community standard library + extensions retained on
+top. See [PROVENANCE.md](PROVENANCE.md) for exact sources, checksums, and the
+full list of what the refresh added/removed.
 
 ## Files
-There are 22 `.kl` files:
-- core, toplevel, sys, dict, sequent, yacc, reader, prolog, track, load, writer, macros, declarations, types, t-star, init
-- compiler (shen-cl build artifact — see PROVENANCE.md)
+
+There are 20 `.kl` files.
+
+**Kernel proper — S41.2 (2026-07-11 refresh), 15 files, byte-identical to
+`shenlanguage.org/Download/S41.2.zip`:**
+- yacc, core, load, prolog, reader, sequent, sys, t-star, toplevel,
+  track, types, writer, backend, declarations, macros
+
+**Standard library + extensions — retained from community ShenOSKernel-41.2:**
 - stlib
-- extension-features, extension-expand-dynamic, extension-launcher
+- extension-features, extension-expand-dynamic, extension-launcher (booted)
 - extension-programmable-pattern-matching (vendored, opt-in; NOT on the boot list)
 
+The actual boot order is defined in `boot.lua` (`FILES`), not by this list.
+
+> Removed relative to the pre-refresh vendored set: `compiler.kl` (a shen-cl
+> build artifact), `dict.kl`, and `init.kl` — see PROVENANCE.md.
+
 ## License
-See the LICENSE in the root of this repository and the original ShenOSKernel distribution.
+See the LICENSE in the root of this repository and the original Shen distribution.
 
 ## Overriding
-If you want to use a different set of KLambda files (e.g. a development tree or a different Shen version), set the environment variable:
+To use a different set of KLambda files (e.g. a development tree or a different
+Shen version), set the environment variable:
 
     SHEN_KL_DIR=/path/to/some/other/klambda
 
-The `boot.lua` loader will use that location instead of the vendored `klambda/` directory.
+The `boot.lua` loader will use that location instead of the vendored `klambda/`
+directory.