stdlib: load the standard library from S-lineage lib/StLib sources

README.md

@@ -281,8 +281,9 @@ browser. One `rules.shen`, two runtimes, no client/server drift. See its
 
 ## Certification / Testing
 
-The port loads and initialises the full 41.2 kernel (including `stlib` and the new
-extensions) and **passes the official 41.2 kernel test suite, 134/134**:
+The port loads and initialises the full 41.2 kernel plus the standard library
+(loaded at boot from the S-lineage Shen sources under `lib/StLib/`) and the
+extensions, and **passes the official 41.2 kernel test suite, 134/134**:
 
 ```sh
 luajit run-kernel-tests.lua    # => "passed ... 134 / failed ... 0 / pass rate ... 100%"

boot.lua

@@ -93,17 +93,19 @@ P.KLDIR = KLDIR   -- resolved .kl directory (trailing /), for typecheck_native
 --     types; the refresh moved shen.rectify-type into t-star, so t-star must
 --     now precede types. Hence the tail: macros declarations t-star types.
 --
--- The trailing four are the community ShenOSKernel extensions + stlib, which
--- Tarver's refresh no longer ships as KLambda (stlib is now lazy Shen sources
--- under Lib/StLib). shen-lua keeps vendoring them on top so the standard
--- library and the CLI launcher stay available and the kernel test suite still
--- certifies. They are pure defuns/defmacros and reference only public kernel
--- functions (get/put/… — never the removed dict.* or shen.initialise-*), so
--- they load unchanged against the refreshed kernel. See klambda/PROVENANCE.md.
+-- The trailing three are the community ShenOSKernel extensions, which Tarver's
+-- refresh no longer ships as KLambda. shen-lua keeps vendoring them on top so
+-- the CLI launcher etc. stay available; they are pure defuns/defmacros
+-- referencing only public kernel functions, so they load unchanged.
+--
+-- NOTE: stlib is NOT here. The standard library is no longer a precompiled
+-- klambda/stlib.kl; it is loaded from the S-lineage Shen sources under
+-- lib/StLib/ by load_stdlib() (below), which the refresh's own install.shen
+-- drives. See lib/StLib/PROVENANCE.md and klambda/PROVENANCE.md.
 local FILES = {
   "yacc","core","load","prolog","reader","sequent","sys","toplevel",
   "track","writer","backend","macros","declarations","t-star","types",
-  "extension-features","extension-expand-dynamic","extension-launcher","stlib"
+  "extension-features","extension-expand-dynamic","extension-launcher"
 }
 
 -- ---- standard streams ----------------------------------------------------
@@ -830,6 +832,96 @@ local function install_fasl()
 end
 
 -- ---- initialise ----------------------------------------------------------
+-- ---- standard library (S-lineage lib/StLib sources) ----------------------
+-- Tarver's S41.2 refresh ships the standard library as Shen SOURCES under
+-- Lib/StLib (loaded into the SBCL image at install time), not as a precompiled
+-- stlib.kl. shen-lua vendors those sources under lib/StLib/ and loads them the
+-- same way: through the kernel's own (load ...) / define pipeline. Unlike raw
+-- stlib.kl defuns (which the pre-refresh port booted as a kernel module), the
+-- define path registers each function's arity property + shen.*lambdatable*
+-- entry, so `(fn filter)` and a bare top-level `(filter ...)` now resolve
+-- instead of raising "fn: filter is undefined".
+--
+-- We run upstream's own install.shen (its factorise toggles, the package +
+-- systemf externals block, everything) with two mechanical rewrites:
+--   1. its relative (load "Sub/file.shen") paths are made absolute against the
+--      vendored directory, so no process chdir is needed (a chdir would
+--      invalidate the KLDIR-relative reads the fasl kernel-key hashing does);
+--   2. its (tc +) toggles are neutralised to (tc -), i.e. the stdlib is loaded
+--      WITHOUT typechecking. This matches the pre-refresh behaviour (the old
+--      precompiled stlib.kl registered no stdlib type signatures either — its
+--      stlib.initialise was never called), it is markedly faster, and it keeps
+--      the native typecheck drivers deferred at boot (a tc+ load would trigger
+--      the typechecker and translate them eagerly). Functions are still fully
+--      defined and arity-registered — only their type signatures are skipped.
+-- SHEN_NO_STDLIB=1 skips the whole thing (a kernel-only embed); SHEN_STDLIB_DIR
+-- overrides the location.
+local STDLIB_LOADED = false
+local function find_stdlib_dir()
+  local env = os.getenv("SHEN_STDLIB_DIR")
+  if env and env ~= "" then return env end
+  -- module-relative first (chdir-independent: boot.lua sits at the repo root)
+  local src = debug.getinfo(1, "S").source
+  local here = src:match("^@(.*)[/\\][^/\\]*$")
+  local candidates = {}
+  if here then candidates[#candidates+1] = here .. "/lib/StLib" end
+  candidates[#candidates+1] = "lib/StLib"
+  for _, d in ipairs(candidates) do
+    local f = io.open(d .. "/install.shen", "r")
+    if f then f:close(); return d end
+  end
+  -- Single-file bundle: no lib/StLib on disk, but make-bundle.lua embedded the
+  -- whole tree as P.STDLIB_SOURCES (relpath -> content). Materialise it once to
+  -- a temp dir and load from there (reuses the ordinary file-based load path).
+  if P.STDLIB_SOURCES then
+    local base = os.tmpname(); os.remove(base)
+    os.execute("mkdir -p '" .. base .. "'")
+    for rel, content in pairs(P.STDLIB_SOURCES) do
+      local full = base .. "/" .. rel
+      local sub = full:match("^(.*)/[^/]*$")
+      if sub then os.execute("mkdir -p '" .. sub .. "'") end
+      local fh = io.open(full, "wb")
+      if fh then fh:write(content); fh:close() end
+    end
+    return base
+  end
+  return nil
+end
+
+local function load_stdlib(verbose)
+  if STDLIB_LOADED then return end
+  if os.getenv("SHEN_NO_STDLIB") == "1" then return end
+  local dir = find_stdlib_dir()
+  if not dir then
+    if verbose then io.stderr:write("  stdlib: lib/StLib not found; skipping (kernel-only)\n") end
+    return
+  end
+  local script = read_file(dir .. "/install.shen")
+  if not script then return end
+  -- Rewrite the relative (load "X") targets to absolute so no chdir is needed.
+  -- All install.shen load paths are relative; the replacement text is escaped
+  -- for gsub's % handling.
+  local prefix = ("(load \"" .. dir .. "/"):gsub("%%", "%%%%")
+  script = script:gsub('%(load "', prefix)
+  script = script:gsub("%(tc %+%)", "(tc -)")   -- load without typechecking (see above)
+  local hush0 = P.GLOBALS["*hush*"]
+  P.GLOBALS["*hush*"] = true       -- suppress the ~20 "loaded" echoes
+  local ok, err = pcall(function()
+    local forms = P.F["read-from-string"](script)
+    while R.is_cons(forms) do
+      P.F["eval"](forms[1])
+      forms = forms[2]
+    end
+  end)
+  P.GLOBALS["*hush*"] = hush0
+  if not ok then
+    error("stdlib load failed: " .. tostring(P.F["error-to-string"](err)), 0)
+  end
+  STDLIB_LOADED = true
+  if verbose then io.stderr:write("  loaded stdlib from " .. dir .. "\n") end
+end
+P.load_stdlib = load_stdlib
+
 local function initialise()
   -- Kernel environment setup (env globals, *property-vector*, arity table).
   --
@@ -846,6 +938,11 @@ local function initialise()
   -- Register the lua.* interop entries in Shen's own arity/lambda-form
   -- tables (needs the *property-vector* the kernel just created).
   require("lua_interop").post_initialise()
+  -- Load the standard library from its S-lineage Shen sources (lib/StLib).
+  -- Done here, at the single post-kernel-init chokepoint every boot path runs
+  -- (shen.boot, run-kernel-tests, the port specs), so the stdlib is present
+  -- for all of them. SHEN_NO_STDLIB=1 opts out.
+  load_stdlib()
   return r
 end
 

build/make-bundle.lua

@@ -70,8 +70,24 @@ do
     if nm then kl_names[#kl_names + 1] = nm end
   end
   p:close()
-  -- 20 vendored .kl: 15 refreshed S41.2 kernel modules + stlib + 4 extensions.
-  assert(#kl_names >= 20, "klambda/ looks incomplete (" .. #kl_names .. " files)")
+  -- 19 vendored .kl: 15 refreshed S41.2 kernel modules + 4 extensions
+  -- (the standard library is no longer a .kl — see the stdlib payload below).
+  assert(#kl_names >= 19, "klambda/ looks incomplete (" .. #kl_names .. " files)")
+end
+
+-- Standard-library Shen sources (lib/StLib). The refresh ships stdlib as
+-- sources, loaded at boot by boot.lua load_stdlib; embed the whole tree
+-- (relative paths, incl. install.shen) so the bundle stays self-contained.
+-- On boot with no lib/StLib on disk, load_stdlib materialises these to a temp
+-- dir and loads from there.
+local stdlib_names = {}
+do
+  local p = io.popen("cd '" .. root .. "/lib/StLib' && find . -type f | sed 's|^\\./||'")
+  for line in p:lines() do
+    if line ~= "" then stdlib_names[#stdlib_names + 1] = line end
+  end
+  p:close()
+  assert(#stdlib_names >= 20, "lib/StLib looks incomplete (" .. #stdlib_names .. " files)")
 end
 
 -- ---- 3. emit ---------------------------------------------------------------
@@ -106,6 +122,11 @@ for _, nm in ipairs(kl_names) do
   emit(("kl[%q] = %s\n"):format(nm, quote(read_file(root .. "/klambda/" .. nm .. ".kl"))))
 end
 
+emit("\nlocal stdlib = {}\n")
+for _, rel in ipairs(stdlib_names) do
+  emit(("stdlib[%q] = %s\n"):format(rel, quote(read_file(root .. "/lib/StLib/" .. rel))))
+end
+
 emit("\nlocal KERNEL = " .. quote(blob) .. "\n")
 
 emit([[
@@ -122,6 +143,7 @@ end
 local P = require("boot")
 P.KERNEL_CACHE_DATA = KERNEL   -- boot.load_kernel() takes the embedded path
 P.KL_SOURCES = kl              -- source fallback for a foreign LuaJIT
+P.STDLIB_SOURCES = stdlib      -- lib/StLib tree; load_stdlib materialises it
 return require("shen")
 ]])
 

klambda/PROVENANCE.md

@@ -54,40 +54,36 @@ that tag — equivalently, to the zip's `KLambda/` directory (both verified with
   `-signedfuncs` / `-environment` are gone; `shen.initialise-lambda-tables`
   (~renamed) and the arity table remain.
 - **Removed: `stlib.kl`** — the standard library is no longer shipped as a
-  precompiled KLambda blob; upstream now ships it as **lazy Shen sources** under
-  `Lib/StLib/` to be loaded on demand. See section 2 for how shen-lua bridges
-  this gap.
+  precompiled KLambda blob; upstream now ships it as **Shen sources** under
+  `Lib/StLib/`. shen-lua vendors those sources under `lib/StLib/` and loads them
+  at boot (see section 2 and `lib/StLib/PROVENANCE.md`); `stlib.kl` is gone.
 - Other renames observed: `hush` → `shen.hush`, `input+` → `shen.input-h+` /
   `shen.process-input+`, plus new `shen.rdecons`, `shen.shen`, pointer helpers.
 
-## 2. Standard library + extensions — community ShenOSKernel-41.2 (retained)
+## 2. Extensions — community ShenOSKernel-41.2 (retained); stdlib moved out
 
-Because Tarver's refresh no longer ships a precompiled `stlib.kl` and shen-lua
-does not yet have a lazy `Lib/StLib` loader, these five files are **retained
-byte-identical from the community `ShenOSKernel-41.2`** release
-(zip SHA-256 `49f1b85d02348d9b3ebc461570c5c56cc066270ab81e35d5257625fb9d17fe82`)
-so the standard library and the CLI launcher stay available and the kernel test
-suite still certifies 134/134:
+**Extensions.** Tarver's refresh does not ship these; they are **retained
+byte-identical from the community `ShenOSKernel-41.2`** release (zip SHA-256
+`49f1b85d02348d9b3ebc461570c5c56cc066270ab81e35d5257625fb9d17fe82`) so the CLI
+launcher etc. stay available:
 
-- `stlib.kl` — the precompiled standard library (`filter`, `take`, `drop`,
-  `reduce`, string/list/vector helpers, …).
 - `extension-features.kl`, `extension-expand-dynamic.kl` — booted.
 - `extension-launcher.kl` — booted; provides the launcher the CLI can use.
 - `extension-programmable-pattern-matching.kl` — vendored, opt-in, **not** booted.
 
-These are all pure `defun`/`defmacro` and reference only public kernel functions
+They are pure `defun`/`defmacro` referencing only public kernel functions
 (`get`/`put`/`arity`/…) — never the removed `dict.*`, `shen.<-dict`, or
-`shen.initialise-*` functions — so they load unchanged against the refreshed
-kernel. They are library/Shen-level code, effectively version-stable.
-
-> **Known limitation (pre-existing, not introduced by the refresh).** stlib
-> functions get an `F` entry from their `defun` and a compiler arity from the
-> boot prescan, so they compile as direct calls and work in loaded programs and
-> the test suite. But `stlib.initialise` is never called, so their runtime
-> `arity` *property* stays `-1`; a bare `(fn filter)` / top-level `(filter …)`
-> reference that resolves through the lambda table fails with "fn: filter is
-> undefined". This behaves identically on the pre-refresh kernel. The clean fix
-> is a lazy `Lib/StLib` loader (see the PR's "remaining work").
+`shen.initialise-*` — so they load unchanged against the refreshed kernel.
+
+**Standard library.** No longer a `.kl` file here. It is loaded at boot from the
+S-lineage Shen sources vendored under `lib/StLib/` — see
+[`lib/StLib/PROVENANCE.md`](../lib/StLib/PROVENANCE.md) and `boot.lua`
+`load_stdlib`. Loading through the kernel's own `define` path (rather than as
+raw `stlib.kl` defuns) registers each function's `arity` property + lambda-table
+entry, which **fixes** the long-standing quirk where a bare `(fn filter)` /
+top-level `(filter …)` raised "fn: filter is undefined" (the pre-refresh
+`stlib.kl` never called `stlib.initialise`, so those functions had runtime
+`arity` = -1). A regression test lives in `test/stdlib_spec.lua`.
 
 ## Boot order
 

klambda/README.md

@@ -5,28 +5,29 @@ directly in this repository so that `shen-lua` is self-contained: you can
 `git clone` and run without needing a separate ShenOSKernel checkout.
 
 The tree has **two lineages** — the refreshed kernel proper from
-shenlanguage.org, and the community standard library + extensions retained on
-top. See [PROVENANCE.md](PROVENANCE.md) for exact sources, checksums, and the
-full list of what the refresh added/removed.
+shenlanguage.org, and the community extensions retained on top. The standard
+library is no longer here; it lives as Shen sources under `../lib/StLib/` (see
+that directory's PROVENANCE.md). See [PROVENANCE.md](PROVENANCE.md) for exact
+sources, checksums, and the full list of what the refresh added/removed.
 
 ## Files
 
-There are 20 `.kl` files.
+There are 19 `.kl` files.
 
 **Kernel proper — S41.2 (2026-07-11 refresh), 15 files, byte-identical to
 `shenlanguage.org/Download/S41.2.zip`:**
 - yacc, core, load, prolog, reader, sequent, sys, t-star, toplevel,
   track, types, writer, backend, declarations, macros
 
-**Standard library + extensions — retained from community ShenOSKernel-41.2:**
-- stlib
+**Extensions — retained from community ShenOSKernel-41.2:**
 - extension-features, extension-expand-dynamic, extension-launcher (booted)
 - extension-programmable-pattern-matching (vendored, opt-in; NOT on the boot list)
 
 The actual boot order is defined in `boot.lua` (`FILES`), not by this list.
 
 > Removed relative to the pre-refresh vendored set: `compiler.kl` (a shen-cl
-> build artifact), `dict.kl`, and `init.kl` — see PROVENANCE.md.
+> build artifact), `dict.kl`, `init.kl`, and `stlib.kl` — the standard library
+> now loads from `../lib/StLib/` Shen sources. See PROVENANCE.md.
 
 ## License
 See the LICENSE in the root of this repository and the original Shen distribution.