From 203475710412cc54f041641df8e5ad6bf2e253a7 Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Mon, 3 Aug 2026 02:44:36 -0500 Subject: [PATCH 1/2] trace: bound the temporal history and stop arming it on dead code (#827) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The per-name assignment history behind `prev of x`, ` is x at ` and `state_at` was append-only, uncapped, and held a reference to every value it recorded. Any long-running program that so much as mentioned `prev of` grew linearly until the machine died. It froze a 4 GB box for ~20 minutes with no OOM kill — the kernel thrashed rather than killing, so the box had to be power-cycled. LeakSanitizer never saw a byte: everything was reachable from the history table and freed at exit. This is unbounded RETENTION, not a leak. Repro (the `prev of` is in a function that is NEVER CALLED): define never_called(v) as: return prev of v x is 0 i is 0 loop while i < N: x is i * 1.5 i is i + 1 peak RSS, /usr/bin/time -f %M, N=200k / 800k / 1.6M before 9088 / 28032 / 53120 kB (~33 B per assignment, no plateau) after 2944 / 2944 / 2944 kB (= the no-temporal-query floor) live `prev of x` in the loop, N=1.6M: 203008 kB -> 2944 kB live `what is x at L`, N=1.6M: 53248 kB -> 2944 kB Two independent defects, both fixed, with NO change to any answer. (B) Unbounded retention. A backward query returns the LATEST assignment whose line is <= L, which makes most entries provably unreachable: entry i is dead exactly when some later entry j has line[j] <= line[i], since any L admitting i admits j too and j wins for being later. What survives are the strict suffix minima of the line sequence — so the live entries are line-sorted and can never outnumber the distinct source lines that assign that name. Bounded by program TEXT, not by runtime: a loop reassigning one name a billion times keeps one entry and pins one value. Maintenance is one pop-while at append. Two facts pruning would otherwise lose are carried explicitly, which is why the answers are identical: - `prev of x at L` wants the value of the assignment immediately preceding (in EXECUTION order) the one that answers L — usually a pruned entry — so every live entry stores its own predecessor. - `when is x at L` counts assignments with line <= L, pruned ones included; counting is order-independent, so a per-name (line -> count) histogram carries it exactly. Also bounded. The observer snapshot for `where/why/how at L` moved inside the live entry (trace_record_obs always targets the newest, which is always live). Backward queries became a binary search over the sorted live array, retiring the periodic line-floor segment index that existed only to make scanning an unbounded array survivable. (A) Whole-program arming. g_trace_hist was set by a source scan, so a `prev of v` in a function nothing calls armed recording for every name. Both history-reading forms compile to a NAMED opcode carrying a compile-time identifier, so the reachable name set is exactly known: the compiler arms only those names and assignments to any other name record nothing. `state_at` (queries every name), an open tape, and turning recording on without naming a name (REPL, `record_history of 1`) still arm the wildcard. No site outside trace.c writes g_trace_hist now. Semantics unchanged; tape untouched. tests/test_temporal_pruning.eigs (22 checks) pins the four cases a naive prune breaks — including the backward-line-jump counterexample a line-keyed table gets wrong — and passes on the PRE-FIX binary too. A 200-seed differential fuzz of every query form against the pre-fix binary diverges zero times, JIT on and EIGS_JIT_OFF. `A` records are written independently of the history table, so tapes before and after are byte-identical and a pre-fix tape replays byte-identically on the fixed binary — no format bump (#411). Gates: release suite 3541/3541; ASan+UBSan with detect_leaks=1 3539/3539, leak tally still 0; make dap + DAP suite 30/30; test_replay.sh 24/24; test_trace_on_fail.sh 7/7; freestanding-check; jit-smoke; embed_stack_soak; --lint clean. Also ~30% faster on the live `prev of` loop (0.45s -> 0.31s, n=3) — it stopped reallocating. New gate [70d] (tests/test_temporal_memory.sh): peak RSS ceiling AND flatness across an 8x iteration range for dead-code, live-prev and live-at programs, under ulimit -v. Validated red-then-green — 6/6 fail on the pre-fix binary, and it also catches an answer-preserving but unbounded prune (`>` instead of `>=`: 13952 -> 90496 kB) that the semantic tests pass. Closes #827 Closes #827 Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 59 +++++ docs/TRACE.md | 56 ++++- src/builtins.c | 6 +- src/compiler.c | 17 +- src/repl.c | 4 +- src/trace.c | 380 ++++++++++++++++++++----------- src/trace.h | 47 +++- tests/run_all_tests.sh | 32 ++- tests/test_temporal_memory.sh | 166 ++++++++++++++ tests/test_temporal_pruning.eigs | 100 ++++++++ 10 files changed, 705 insertions(+), 162 deletions(-) create mode 100755 tests/test_temporal_memory.sh create mode 100644 tests/test_temporal_pruning.eigs diff --git a/CHANGELOG.md b/CHANGELOG.md index 05105af8..da0ca2ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,65 @@ All notable changes to EigenScript are documented here. +## [Unreleased] + +### Fixed + +- **The temporal assignment history is bounded, and it no longer arms on + dead code (#827).** The per-name history that backs `prev of x`, + ` is x at ` and `state_at` was append-only, uncapped, and held + a *reference* to every value it recorded. Any long-running program that + so much as mentioned `prev of` grew linearly until the machine died — + `dynamics`' orbit lab retained 3.9 MB per rendered frame and hit 859 MB + by frame 300; the minimal repro froze a 4 GB box for ~20 minutes with no + OOM kill (power-cycle to recover). LeakSanitizer never saw a byte of it: + every allocation was reachable from the history table and freed at exit, + so this was unbounded *retention*, not a leak. Two independent defects, + both fixed, with **no change to any temporal answer**: + - **Unbounded retention.** The history is now pruned at append time, + because a backward query makes most entries provably unreachable: + entry `i` is dead exactly when some later entry `j` has + `line[j] <= line[i]` (any `L` that admits `i` admits `j` too, and `j` + wins for being later). What survives are the strict suffix minima of + the line sequence, so the live entries are line-sorted and can never + outnumber the distinct source lines that assign that name — bounded by + program *text*, not by runtime. A loop reassigning one name a billion + times now keeps one entry and pins one value. Two facts pruning would + otherwise lose are carried explicitly so the answers are identical: + each live entry stores its own execution-order predecessor (which is + what `prev of x at L` returns, and it is usually a pruned entry), and + a per-name `(line -> count)` histogram carries `when is x at L`, which + counts pruned assignments. Backward queries became a binary search + over the sorted live array, retiring the periodic line-floor segment + index that existed only to make scanning an unbounded array + survivable. Measured, 1.6M iterations: a live `prev of` went + 203,008 kB -> 2,944 kB, a live `at` query 53,248 kB -> 2,944 kB, both + now flat in iteration count and equal to the no-temporal-query floor. + - **Whole-program arming.** `g_trace_hist` was set by a source scan, so + a `prev of v` inside a function nothing ever called switched on + recording for every name in the program. Both history-reading forms + compile to a NAMED opcode carrying a compile-time identifier, so the + reachable name set is exactly known: the compiler now arms only those + names, and assignments to any other name record nothing. `state_at` + (it queries every name), an open tape, and turning recording on + without naming a name (the REPL, `record_history of 1`) still arm the + wildcard. The dead-code repro went 53,120 kB -> 2,944 kB. + + **Semantics are unchanged and the tape is untouched.** The pruning drops + only entries no query could reach, which is why + `tests/test_temporal_pruning.eigs` — including the backward-line-jump + counterexample that a line-keyed table gets wrong — passes on the + pre-fix binary too, and why a 200-seed differential fuzz of every query + form (`what`/`who`/`when`/`where`/`why`/`how`/`prev`, with and without + `at`, plus `state_at`) against the pre-fix binary shows zero + divergences on both execution tiers. `A` records are written + independently of the history table, so tapes recorded before and after + are byte-identical and a pre-fix tape replays byte-identically on the + fixed binary — no format-version bump (#411). New gates: suite [70c] + (semantics) and [70d] (`tests/test_temporal_memory.sh` — peak RSS + ceiling *and* flatness across an 8x iteration range, which goes red on + the pre-fix binary and on an answer-preserving-but-unbounded prune). + ## [0.35.0] - 2026-08-02 ### Added diff --git a/docs/TRACE.md b/docs/TRACE.md index 94f75cf5..19f6600c 100644 --- a/docs/TRACE.md +++ b/docs/TRACE.md @@ -289,6 +289,18 @@ in-run time travel. *first* temporal query starts recording at that point, so assigns executed earlier are not visible to it. Aliasing `state_at` through a dict or eval-built string also hides it from the compiler's scan. +- The gate is **per name**, not whole-program (#827). Both history-reading + forms — `prev of x` and ` is x at L` — compile to a NAMED opcode + carrying a compile-time identifier, so the set of names a temporal query + can ever reach is known exactly, and assignments to any other name record + nothing. This is what stops a `prev of v` sitting in a function nothing + ever calls from taxing every assignment in the program. Three things + force the wildcard instead, because they can reach a name the compiler + cannot enumerate: `state_at` (it queries every tracked name), an open + tape (`EIGS_TRACE` or an embed sink), and turning recording on without + naming a name (the REPL, `record_history of 1`). Arming only ever widens + within a session — a name armed mid-run by `eval` starts recording from + that point, the same edge the whole-program gate already had. - When the compiled program contains a `where`/`why`/`how ... at` query, each history entry also stamps an observer snapshot (entropy, dH) at assign time, so the observer-derived @@ -297,14 +309,42 @@ in-run time travel. no such query in the program, no per-assign cost. - `state_at of line` walks every tracked name's history backward and returns a dict of each binding's value at or before `line`. -- Backward queries (`at`, `state_at`) are pruned by a periodic - line-floor index: each 64-entry segment of a name's history caches - its minimum line stamp, so segments that cannot contain a hit are - skipped in one compare. Loop-heavy histories — thousands of assigns - stamped with the same few lines, the debugger-scrub worst case — - resolve in O(history/64) instead of O(history). The index adds one - `int` per 64 history entries and an O(1) min-update per assign. -- Per-assign cost of the history: one cache line + a pointer compare. +- **A backward query is TEMPORAL, not line-keyed.** ` is x at L` + returns the value from the most recent assignment whose line is `<= L` + — which is *not* "the value at the greatest line `<= L`". Assign at + line 12, then at line 5, then ask at L=15: the answer is the line-5 + value, because that assignment happened later. Any representation that + keys the history by line answers the line-12 value and is wrong. +- **The history is bounded by the program TEXT, not by runtime** (#827). + It used to be append-only and uncapped, holding a reference to every + value ever assigned: a program that merely mentioned `prev of` grew + linearly until the machine died. It is now pruned at append time, with + no change to any answer, because most entries are provably unreachable: + + entry i is dead <=> some later entry j has line[j] <= line[i] + + (any `L` that admits `i` also admits `j`, and `j` wins for being later). + What survives are the strict suffix minima of the line sequence, so the + live entries are sorted by line and can never outnumber the distinct + source lines that assign that name. A loop that reassigns one name a + billion times keeps ONE entry — and pins one value instead of a billion. + Two facts that pruning would otherwise lose are carried explicitly, so + the answers are identical: each live entry stores its own + execution-order predecessor (`prev of x at L` wants a value that is + usually pruned), and a per-name `(line -> count)` histogram carries + `when is x at L`, which counts pruned assignments too. + **Nothing about the tape changed**: `A` records are written by + `trace_assign` independently of the history table, one per assignment + as before, and an open tape arms every name anyway. Tapes recorded + before and after #827 are byte-identical, so no format-version bump + (#411) — this was a retention bug, not a format one. +- Backward queries (`at`, `state_at`) are therefore a binary search over + a line-sorted array — `O(log D)` where `D` is the number of distinct + assigning lines. This replaced the periodic line-floor segment index, + which existed only to make scanning an unbounded array survivable. +- Per-assign cost of the history: one cache line + a pointer compare, + plus the pop-while that retires the entries the new assignment kills + (amortized O(1) — an entry is pushed once and popped once). - **The history is per-thread; the tape is per-process** (#739). The history table is keyed by *interned name pointer*, and the intern table lives on `EigsThread`, so two threads' `x` were never the same diff --git a/src/builtins.c b/src/builtins.c index 92eb1b92..749fb80c 100644 --- a/src/builtins.c +++ b/src/builtins.c @@ -3046,8 +3046,10 @@ Value* builtin_record_history(Value *arg) { } int prev = g_trace_hist; int on = (arg->data.num != 0.0) ? 1 : 0; - g_trace_hist = on; - g_trace_obs_hist = on; + /* #827: no name to narrow on — a self-hosted compiler calling this is + * standing in for the whole-program arming, so it gets the wildcard. */ + if (on) { trace_arm_history_all(); g_trace_obs_hist = 1; } + else trace_history_disable(); return make_num((double)prev); } diff --git a/src/compiler.c b/src/compiler.c index 2b67d3dc..564742f3 100644 --- a/src/compiler.c +++ b/src/compiler.c @@ -1908,7 +1908,7 @@ static void compile_node_inner(Compiler *c, ASTNode *node) { * seen; recording then starts at the aliasing program's own * temporal queries, or never. Documented in TRACE.md.) */ if (strcmp(node->data.ident.name, "state_at") == 0) - g_trace_hist = 1; + trace_arm_history_all(); /* #827: state_at queries every name */ /* Try local slot resolution for params (fast path) */ if (c->enclosing) { uint32_t h = node->name_hash; @@ -2842,9 +2842,18 @@ static void compile_node_inner(Compiler *c, ASTNode *node) { ASTNode *at_expr = node->data.interrogate.at_expr; /* `prev of x` and every `at ` form answer from the - * per-assign history — enable recording. */ - if (kind == 6 || at_expr) - g_trace_hist = 1; + * per-assign history — enable recording. #827: arm only the NAME + * this query can reach. Both history-reading forms compile to a + * NAMED opcode carrying a compile-time identifier, so the reachable + * set is exact; a non-ident operand never reads the history at all + * (bare OP_INTERROGATE) but arms the wildcard anyway — widening is + * the safe direction. */ + if (kind == 6 || at_expr) { + if (expr && expr->type == AST_IDENT) + trace_arm_history_name(expr->data.ident.name); + else + trace_arm_history_all(); + } if (at_expr && expr && expr->type == AST_IDENT) { /* ` is x at ` — operand value is not needed; only diff --git a/src/repl.c b/src/repl.c index 8d4ff3a0..da9dd529 100644 --- a/src/repl.c +++ b/src/repl.c @@ -585,7 +585,9 @@ static void repl_interactive(Env *env) { * by line — `x is 5` would record nothing and a later `prev of x` finds * no history. Interactive sessions record from the start (the piped * path is left untouched: byte-identical output is its contract). */ - g_trace_hist = 1; + /* #827: a REPL line can name any binding assigned by an earlier line, + * so the narrow per-name arming cannot apply here — wildcard. */ + trace_arm_history_all(); g_trace_obs_hist = 1; hist_load(); diff --git a/src/trace.c b/src/trace.c index 9a7fb3ae..eeb31244 100644 --- a/src/trace.c +++ b/src/trace.c @@ -69,33 +69,67 @@ int g_trace_current_line = 0; * not a debug-tape feature. Per-assign cost is ~one cache line read + * a pointer-equality compare. */ +/* ----- #827: the history is REACHABILITY-PRUNED, not append-only. + * + * Every backward query (`what/prev is x at L`, `state_at of L`) answers with + * the LATEST assignment whose line stamp is <= L — a temporal-backward walk, + * not "the greatest line <= L". That is the whole contract, and it makes most + * recorded entries provably unreachable: + * + * entry i is dead <=> exists j > i with line[j] <= line[i] + * + * because any L that admits i (line[i] <= L) also admits the later j, and j + * wins for being later. The surviving entries are exactly the strict suffix + * minima of the line sequence, so read left-to-right their lines are STRICTLY + * INCREASING — and therefore the live history for one name can never exceed + * the number of distinct source lines that assign it. Bounded by program TEXT, + * independent of how long the program runs. + * + * Maintenance is one pop-while at append: a new entry stamped `line` kills + * exactly the trailing live entries whose line is >= it. Nothing that could + * ever have been an answer is dropped, so NO query changes its answer — the + * two facts the raw array carried that pruning would otherwise lose are kept + * explicitly: + * + * - `prev of x at L` wants the value of the assignment immediately + * preceding (in EXECUTION order) the one that answers L. That predecessor + * is usually a pruned entry, so each live entry carries its own + * `prev_value` — captured at append time, exact regardless of pruning. + * - `when is x at L` wants the COUNT of assignments with line <= L, which + * depends on the pruned entries too. Counting is order-independent, so a + * per-name (line -> count) histogram carries it exactly. Its size is also + * bounded by the number of distinct assigning lines. + * + * The observer snapshot (`where/why/how is x at L`) rides inside the live + * entry: it is patched onto the most recent entry by trace_record_obs, which + * always targets a live one (the newest entry is always live). + * + * Because the live array is sorted by line, the backward walk is a binary + * search — which replaced the old periodic line-floor segment index (that + * index existed only to make scanning an unbounded array survivable). */ typedef struct { int line; EigsSlot value; + EigsSlot prev_value; /* value of the immediately preceding assign */ + uint8_t has_prev_value; + /* Observer-state snapshot, captured only while g_trace_obs_hist is set. + * obs_valid == 0 marks assigns whose slot had no observer state + * (untracked immediates, e.g. writes inside `unobserved` blocks) and + * every entry recorded before the flag flipped on mid-run (eval/REPL + * compiling a historical observer query). */ + uint8_t obs_valid; + double entropy; + double dH; + double last_entropy; } HistoryEntry; -/* Observer-state snapshot per assignment, captured only while - * g_trace_obs_hist is set. Parallel to the history array: history[i] - * pairs with obs[i - obs_start] when i >= obs_start (obs_start is the - * history index of the first captured assign — the flag can flip on - * mid-run when eval/REPL compiles a historical observer query). - * valid == 0 marks assigns whose slot had no observer state (untracked - * immediates, e.g. writes inside `unobserved` blocks). */ +/* (line -> assignment count) histogram, kept sorted by line so `when is x + * at L` is an exact sum over a bounded array. `count` is 64-bit: a hot loop + * can genuinely assign one line more than 2^31 times. */ typedef struct { - double entropy; - double dH; - double last_entropy; - uint8_t valid; -} ObsHistEntry; - -/* Phase 4 snapshot cache: a periodic line-floor index over the history. - * Segment k summarizes history[k< count) histogram, sorted by line */ + int lc_count; + int lc_cap; } PrevEntry; +/* ----- #827 (defect A): per-name arming. + * + * `g_trace_hist` is a whole-program flag, so a `prev of v` inside a function + * that is never called used to switch on recording for EVERY name in the + * program. The set of names a temporal query can ever reach is known at + * compile time, though: `prev of x` and ` is x at L` both compile to a + * NAMED opcode carrying the identifier, and only those named forms consult + * the history (the bare value form, OP_INTERROGATE, never does). So the + * compiler arms the names it actually mentions, and an assignment to any + * other name records nothing. + * + * Sound by construction: a name no temporal query names cannot be asked + * about, so skipping it cannot change an answer. Two things force the + * wildcard instead — `state_at` (queries every name) and a tape being open + * (EIGS_TRACE / an embed sink) — plus anything that turns recording on + * without naming a name (the REPL, `record_history of 1`). + * + * The set is process-global because it records compile-time facts, while the + * table it filters is per-thread (#739). `g_arm_gen` bumps whenever the set + * changes, so each PrevEntry caches its decision and rechecks only after a + * mid-run arming (eval / REPL / a later `record_history`). */ +static char **g_arm_names = NULL; +static int g_arm_count = 0; +static int g_arm_cap = 0; +static int g_arm_all = 0; +static uint32_t g_arm_gen = 1; + +static int arm_set_has(const char *name) { + for (int i = 0; i < g_arm_count; i++) + if (strcmp(g_arm_names[i], name) == 0) return 1; + return 0; +} + +void trace_arm_history_all(void) { + g_trace_hist = 1; + if (g_arm_all) return; + g_arm_all = 1; + g_arm_gen++; +} + +void trace_arm_history_name(const char *name) { + g_trace_hist = 1; + if (!name || g_arm_all) return; + if (arm_set_has(name)) return; + if (g_arm_count >= g_arm_cap) { + int nc = g_arm_cap ? g_arm_cap * 2 : 8; + char **nn = realloc(g_arm_names, (size_t)nc * sizeof(char *)); + if (!nn) { trace_arm_history_all(); return; } /* OOM: never narrow */ + g_arm_names = nn; + g_arm_cap = nc; + } + size_t len = strlen(name) + 1; + char *copy = malloc(len); + if (!copy) { trace_arm_history_all(); return; } /* OOM: never narrow */ + memcpy(copy, name, len); + g_arm_names[g_arm_count++] = copy; + g_arm_gen++; +} + +void trace_history_disable(void) { + g_trace_hist = 0; + g_trace_obs_hist = 0; +} + /* g_prev_tab / g_prev_cap / g_prev_count are bridge macros onto EigsThread * (eigenscript.h), reached only with a thread attached. Every read path below * that can run during teardown or from atexit guards on `eigs_current` first. */ @@ -170,6 +267,35 @@ static void prev_grow(void) { g_prev_cap = new_cap; } +/* Bump the (line -> count) histogram for `when is x at L`. Sorted insert; + * after the first few assigns this is a pure binary-search hit. */ +static void lc_bump(PrevEntry *e, int line) { + int lo = 0, hi = e->lc_count - 1; + while (lo <= hi) { + int mid = (int)(((unsigned)lo + (unsigned)hi) >> 1); + if (e->lc[mid].line == line) { e->lc[mid].count++; return; } + if (e->lc[mid].line < line) lo = mid + 1; + else hi = mid - 1; + } + if (e->lc_count >= e->lc_cap) { + int nc = e->lc_cap ? e->lc_cap * 2 : 8; + LineCount *nl = realloc(e->lc, (size_t)nc * sizeof(LineCount)); + if (!nl) return; /* `when at L` under-counts rather than aborting */ + e->lc = nl; + e->lc_cap = nc; + } + memmove(&e->lc[lo + 1], &e->lc[lo], + (size_t)(e->lc_count - lo) * sizeof(LineCount)); + e->lc[lo].line = line; + e->lc[lo].count = 1; + e->lc_count++; +} + +static void hist_drop(HistoryEntry *h) { + slot_decref(h->value); + if (h->has_prev_value) slot_decref(h->prev_value); +} + static void prev_record_assign(const char *name, EigsSlot value) { if (!eigs_current || !name) return; if (g_prev_count * PREV_LOAD_DEN >= g_prev_cap * PREV_LOAD_NUM) { @@ -181,6 +307,13 @@ static void prev_record_assign(const char *name, EigsSlot value) { e->name = name; g_prev_count++; } + /* #827 defect A: names no temporal query can name record nothing. */ + if (__builtin_expect(e->armed_gen != g_arm_gen, 0)) { + e->armed_gen = g_arm_gen; + e->armed = (uint8_t)(g_arm_all || arm_set_has(name)); + } + if (!e->armed) return; + if (e->has_current) { /* Shift current -> prev; drop the old prev. */ if (e->has_prev) slot_decref(e->prev); @@ -191,8 +324,16 @@ static void prev_record_assign(const char *name, EigsSlot value) { e->current = value; e->has_current = 1; - /* Append to history for `at ` queries. Stamp with the - * current VM line as cached by trace_line. */ + /* Stamp with the current VM line as cached by trace_line. */ + int line = g_trace_current_line; + lc_bump(e, line); + + /* Reserve BEFORE pruning, so an allocation failure leaves the history + * exactly as it was. Pruning first and then failing to append would + * retire entries that are only unreachable once the new one exists — + * turning an OOM into a stale (wrong) answer instead of an unchanged + * one. Capacity reserved for the pre-prune count always covers the + * post-prune count plus this entry, since pruning only shrinks. */ if (e->hist_count >= e->hist_cap) { int new_cap = e->hist_cap ? e->hist_cap * 2 : 8; HistoryEntry *nh = realloc(e->history, (size_t)new_cap * sizeof(HistoryEntry)); @@ -200,55 +341,31 @@ static void prev_record_assign(const char *name, EigsSlot value) { e->history = nh; e->hist_cap = new_cap; } - slot_incref(value); - int idx = e->hist_count; - e->history[idx].line = g_trace_current_line; - e->history[idx].value = value; - e->hist_count++; - - /* Maintain the periodic line-floor index. */ - if (!e->seg_dead) { - int seg = idx >> HIST_SEG_SHIFT; - if (seg >= e->seg_cap) { - int nc = e->seg_cap ? e->seg_cap * 2 : 4; - int *ns = realloc(e->seg_min, (size_t)nc * sizeof(int)); - if (!ns) { - e->seg_dead = 1; /* keep history; queries go linear */ - } else { - e->seg_min = ns; - e->seg_cap = nc; - } - } - if (!e->seg_dead) { - if ((idx & (HIST_SEG - 1)) == 0 || g_trace_current_line < e->seg_min[seg]) - e->seg_min[seg] = g_trace_current_line; - } + + /* #827 defect B: retire the entries this assignment makes unreachable — + * every trailing live entry stamped at or after `line` (see the header + * comment on HistoryEntry). This is what bounds the history. */ + while (e->hist_count > 0 && e->history[e->hist_count - 1].line >= line) { + e->hist_count--; + hist_drop(&e->history[e->hist_count]); } - /* Observer-state capture for `where/why/how ... at `. The - * OBSERVE op ran before this hook and left a tracked Value with - * carried-over (dirty) observer state on the stack, so forcing - * freshness here computes exactly what a live interrogative at - * this point would see. */ - if (__builtin_expect(g_trace_obs_hist, 0)) { - if (!e->obs) e->obs_start = idx; - int oi = idx - e->obs_start; - if (oi >= 0) { - if (oi >= e->obs_cap) { - int nc = e->obs_cap ? e->obs_cap * 2 : 8; - while (nc <= oi) nc *= 2; - ObsHistEntry *no = realloc(e->obs, (size_t)nc * sizeof(ObsHistEntry)); - if (no) { e->obs = no; e->obs_cap = nc; } - } - if (oi < e->obs_cap) { - /* #262 Step E: observer state lives on the Env slot, not the - * Value — leave the snapshot empty here; OBSERVE_NAME_POST fills - * it from the fresh slot via trace_record_obs (runs after the - * SET that created this history entry). */ - e->obs[oi].valid = 0; - } - } + HistoryEntry *h = &e->history[e->hist_count++]; + slot_incref(value); + h->line = line; + h->value = value; + /* The execution-order predecessor — `prev of x at L`'s answer. The + * current->prev shift above already put it in e->prev. */ + h->has_prev_value = e->has_prev; + if (e->has_prev) { + h->prev_value = e->prev; + slot_incref(h->prev_value); } + /* #262 Step E: observer state lives on the Env slot, not the Value — + * leave the snapshot empty here; OBSERVE_NAME_POST fills it from the + * fresh slot via trace_record_obs (runs after the SET that created this + * entry), which is why the newest entry must stay live. */ + h->obs_valid = 0; } /* #262 Phase-3 D2: patch the observer snapshot for `name`'s most recent @@ -265,15 +382,12 @@ void trace_record_obs(const char *name, double entropy, double dH, double last_entropy) { if (!eigs_current || !name || !g_prev_tab) return; PrevEntry *e = prev_lookup_slot(g_prev_tab, g_prev_cap, name); - if (!e->name || e->hist_count == 0 || !e->obs) return; - int idx = e->hist_count - 1; - int oi = idx - e->obs_start; - if (oi < 0 || oi >= e->obs_cap) return; - ObsHistEntry *o = &e->obs[oi]; - o->entropy = entropy; - o->dH = dH; - o->last_entropy = last_entropy; - o->valid = 1; + if (!e->name || e->hist_count == 0) return; + HistoryEntry *h = &e->history[e->hist_count - 1]; + h->entropy = entropy; + h->dH = dH; + h->last_entropy = last_entropy; + h->obs_valid = 1; } int trace_query_prev(const char *interned_name, EigsSlot *out) { @@ -285,39 +399,21 @@ int trace_query_prev(const char *interned_name, EigsSlot *out) { return 1; } -/* Walk history backward — entries are appended in execution order, so - * the array is monotone-ish but not strictly sorted (a backward jump - * could in principle re-execute earlier lines; the latest such write - * is the answer, which is exactly what backward scan from the end - * gives us). +/* The latest live assignment stamped at or before `line`. * - * The line-floor index prunes the walk: a segment whose minimum line - * stamp exceeds the query line cannot contain a hit, so it is skipped - * in one compare. When a segment's floor is ≤ the query line it holds - * at least one qualifying entry, and the backward scan inside it finds - * the latest one — which is the global answer, since all later segments - * were ruled out. Loop-heavy histories (many assigns stamped with the - * same few lines) skip in O(H/SEG); the residual scan is ≤ SEG entries. */ + * The live array is sorted strictly increasing by line (#827 — see the + * HistoryEntry header), and pruning removed only entries that no query could + * reach, so the answer is the LAST entry with line <= L: one binary search. + * This replaces the old periodic line-floor segment index, which existed to + * make a backward scan over an unbounded append-only array survivable. */ static int find_hist_idx_at_or_before(PrevEntry *e, int line) { - int i = e->hist_count - 1; - if (e->seg_min && !e->seg_dead) { - while (i >= 0) { - int seg = i >> HIST_SEG_SHIFT; - int seg_start = seg << HIST_SEG_SHIFT; - if (e->seg_min[seg] > line) { - i = seg_start - 1; - continue; - } - for (; i >= seg_start; i--) { - if (e->history[i].line <= line) return i; - } - } - return -1; - } - for (; i >= 0; i--) { - if (e->history[i].line <= line) return i; + int lo = 0, hi = e->hist_count - 1, ans = -1; + while (lo <= hi) { + int mid = (int)(((unsigned)lo + (unsigned)hi) >> 1); + if (e->history[mid].line <= line) { ans = mid; lo = mid + 1; } + else hi = mid - 1; } - return -1; + return ans; } int trace_query_at(int kind, const char *interned_name, int line, EigsSlot *out) { @@ -333,44 +429,43 @@ int trace_query_at(int kind, const char *interned_name, int line, EigsSlot *out) } if (kind == 2) { - /* `when is x at L` — count of assignments with line ≤ L. */ - int count = 0; - for (int i = 0; i < e->hist_count; i++) { - if (e->history[i].line <= line) count++; - } + /* `when is x at L` — count of assignments with line ≤ L. Summed + * from the histogram, which counts pruned assignments too (#827). */ + long long count = 0; + for (int i = 0; i < e->lc_count && e->lc[i].line <= line; i++) + count += e->lc[i].count; *out = slot_from_num((double)count); return 1; } int idx = find_hist_idx_at_or_before(e, line); if (idx < 0) return 0; + HistoryEntry *h = &e->history[idx]; if (kind >= 3 && kind <= 5) { /* where/why/how — read the observer snapshot captured at that * assign. Mirrors the live INTERROGATE formulas. */ - if (!e->obs || idx < e->obs_start || - idx - e->obs_start >= e->obs_cap) return 0; - ObsHistEntry *o = &e->obs[idx - e->obs_start]; - if (!o->valid) return 0; - double r = (kind == 3) ? o->entropy - : (kind == 4) ? o->dH - : observer_settledness(o->dH); /* #412: how = f(dH) */ + if (!h->obs_valid) return 0; + double r = (kind == 3) ? h->entropy + : (kind == 4) ? h->dH + : observer_settledness(h->dH); /* #412: how = f(dH) */ *out = slot_from_num(r); return 1; } if (kind == 0) { /* `what is x at L` — value at most recent assign ≤ L. */ - *out = e->history[idx].value; + *out = h->value; slot_incref(*out); return 1; } if (kind == 6) { /* `prev of x at L` — value at the assign immediately preceding - * the one that produced `x`'s state at L. */ - if (idx < 1) return 0; - *out = e->history[idx - 1].value; + * the one that produced `x`'s state at L. Carried on the entry + * because that predecessor is usually pruned (#827). */ + if (!h->has_prev_value) return 0; + *out = h->prev_value; slot_incref(*out); return 1; } @@ -489,7 +584,7 @@ void trace_set_sink(void (*cb)(const char *, size_t, void *), void *ud) { g_last_line = -1; g_line_dirty = 0; g_trace_enabled = 1; - g_trace_hist = 1; + trace_arm_history_all(); /* a tape records every name's assigns */ emit_header(); } else { sink_flush(); @@ -543,7 +638,7 @@ void trace_init(void) { } setvbuf(g_trace_fp, NULL, _IOFBF, 64 * 1024); g_trace_enabled = 1; - g_trace_hist = 1; + trace_arm_history_all(); /* a tape records every name's assigns */ emit_header(); #endif /* !EIGENSCRIPT_FREESTANDING */ } @@ -1060,10 +1155,9 @@ void trace_thread_release(void) { if (e->has_prev) slot_decref(e->prev); if (e->has_current) slot_decref(e->current); for (int j = 0; j < e->hist_count; j++) - slot_decref(e->history[j].value); + hist_drop(&e->history[j]); free(e->history); - free(e->seg_min); - free(e->obs); + free(e->lc); } free(tab); } @@ -1087,6 +1181,18 @@ void trace_shutdown(void) { trace_thread_release(); + /* #827: drop the compile-time armed-name set and fall back to the + * wildcard. Widening, never narrowing — a state opened after a + * process-wide teardown must not inherit a narrowing whose compile-time + * evidence has been freed. (Recording still needs g_trace_hist.) */ + for (int i = 0; i < g_arm_count; i++) free(g_arm_names[i]); + free(g_arm_names); + g_arm_names = NULL; + g_arm_count = 0; + g_arm_cap = 0; + g_arm_all = 1; + g_arm_gen++; + replay_shutdown(); } diff --git a/src/trace.h b/src/trace.h index 9baa4982..eebed4db 100644 --- a/src/trace.h +++ b/src/trace.h @@ -42,9 +42,35 @@ extern int g_trace_enabled; * records) gates on this, so programs that never ask temporal questions * pay nothing per assign. Profiling the DMG-shaped dispatch workload * showed the always-on variant cost ~17.8M trace_line + 2.5M - * trace_assign calls per 500k interpreted steps (~1/3 of runtime). */ + * trace_assign calls per 500k interpreted steps (~1/3 of runtime). + * + * #827: this flag alone is too coarse — it is whole-program, so a + * `prev of v` inside a function nothing ever calls used to arm recording + * for every name. Set it only through the two arming entry points below, + * which also record WHICH names a temporal query can reach. Never write + * `g_trace_hist = 1` directly: an armed flag with no armed name records + * nothing. */ extern int g_trace_hist; +/* #827: turn history recording on. + * + * trace_arm_history_name(n) — narrow: record only assignments to `n` (plus + * any other armed name). Use when the query's target is a compile-time + * identifier, which is every form that actually reads the history + * (`prev of x`, ` is x at L` — both compile to a NAMED opcode). + * + * trace_arm_history_all() — wildcard: record every name. Required by + * `state_at` (it queries all names), by an open tape, and by anything that + * turns recording on without naming a name (the REPL, `record_history of 1`). + * Widening is always safe; narrowing after the fact is not, so the arming + * set only ever grows within a session. + * + * trace_history_disable() — the `record_history of 0` opt-out; clears both + * g_trace_hist and g_trace_obs_hist. */ +void trace_arm_history_all(void); +void trace_arm_history_name(const char *name); +void trace_history_disable(void); + /* Source line currently being executed. Written by OP_LINE (a plain global * store — cheaper than a call; the JIT also stamps it via a flat-address * write, so it can't be __thread), read by trace_assign to stamp history @@ -136,6 +162,12 @@ int trace_query_prev(const char *interned_name, EigsSlot *out); * value last bound to `name` at or before `line`. Returns 1 + fills * *out on hit, 0 on miss. * + * TEMPORAL-BACKWARD, not line-keyed: assign at line 12, then at line 5, + * then query at L=15 — the answer is the line-5 value, because it is the + * most recent assignment whose line is <= 15. A line-keyed map would + * wrongly answer with the line-12 value. #827's pruning preserves this + * exactly (see the HistoryEntry comment in trace.c). + * * `kind` mirrors the interrogative encoding: * 0 (what), 6 (prev) → return historical slot * 2 (when) → return assignment count up to that line (as immediate num) @@ -152,13 +184,12 @@ int trace_query_at(int kind, const char *interned_name, int line, EigsSlot *out) * are omitted. Result is a fresh VAL_DICT owned by the caller; returns * NULL only on allocation failure. * - * Cost is O(N · (H/64 + 64)) where N = distinct names and H = avg history - * depth: each name's backward walk consults a periodic line-floor index - * (min line stamp per 64-entry segment) that skips whole segments which - * cannot contain a hit, then scans at most one segment linearly. Histories - * dominated by loop re-assigns — the debugger-scrub worst case — skip in - * O(H/64). If the index allocation ever fails the name falls back to the - * plain O(H) backward scan. */ + * Cost is O(N · log D) where N = distinct names and D = the number of + * distinct source lines that assign a given name: #827 prunes each name's + * history down to the entries a backward query can actually reach, leaving + * a line-sorted array that binary-searches. D is a property of the program + * TEXT, so neither the cost nor the memory grows with how long the program + * runs — a loop that reassigns one name a billion times keeps one entry. */ struct Value *trace_state_at(int line); /* #736: 1 for a runtime-internal binding name (the `__name__` form the diff --git a/tests/run_all_tests.sh b/tests/run_all_tests.sh index 67fbf255..dbcccdf0 100755 --- a/tests/run_all_tests.sh +++ b/tests/run_all_tests.sh @@ -2766,8 +2766,8 @@ check_eigs_suite "dispatch rebind (module scope + paren form)" test_dispatch_reb check_eigs_suite "dispatch rebind (fn-body write-through)" test_dispatch_rebind_fn.eigs "All tests passed" 2 check_eigs_suite "dispatch rebind (eval escape)" test_dispatch_rebind_eval.eigs "All tests passed" 1 -# [70] Temporal interrogatives (prev of, at, state_at) + the line-floor -# index in trace.c (deep loop histories must skip segments correctly). +# [70] Temporal interrogatives (prev of, at, state_at). Deep loop histories +# must still answer early-line queries correctly after #827's pruning. echo "[70] Temporal Interrogatives (23 checks)" TT_OUTPUT=$(./eigenscript ../tests/test_temporal.eigs 2>&1); TT_OUTPUT_RC=$? if rc_ok "$TT_OUTPUT_RC" "$TT_OUTPUT" && echo "$TT_OUTPUT" | grep -q "All tests passed"; then @@ -2809,6 +2809,34 @@ rm -f "$PRV_FILE" # freeze at the OSR point (OP_LINE must stamp g_trace_current_line in the JIT). check_eigs_suite "JIT temporal at/state_at under OSR (g_trace_current_line)" test_jit_temporal_osr.eigs "All tests passed" 2 +# [70c] #827 — the assignment history is reachability-pruned, so an entry no +# backward query can reach is dropped at append time. These pin the four cases +# a naive prune silently breaks: the backward-line-jump counterexample (`at L` +# is a TEMPORAL walk, not "greatest line <= L"), `prev of x at L` when the +# execution-order predecessor was pruned, `when is x at L` counting pruned +# assignments, and alternating lines (which a same-line-run collapse misses). +# These answers are UNCHANGED by #827 — the file passes on the pre-fix binary +# too. It is the semantic half; the memory half is [70d]. +check_eigs_suite "temporal history pruning keeps every answer (#827)" test_temporal_pruning.eigs "All tests passed" 22 + +# [70d] #827 — and the history must stay BOUNDED. Peak RSS at two iteration +# counts 8x apart, ceiling + flatness, for a dead-code `prev of`, a live +# `prev of`, and a live `at` query. Pre-fix this ran 203 MB and climbing; it +# froze a 4 GB box. Not a leak — every byte was reachable and freed at exit, +# so no sanitizer sees it. Skips on sanitizer builds (ASan overhead swamps it). +echo "[70d] Temporal history is bounded (#827)" +TMEM_OUTPUT=$(bash "$TESTS_DIR/test_temporal_memory.sh" 2>&1); TMEM_RC=$? +echo "$TMEM_OUTPUT" | grep -E "^ (PASS|FAIL|SKIP|baseline|dead|live|at_live)" +TMEM_N=$(echo "$TMEM_OUTPUT" | sed -n 's/^TEMPORAL_MEM: \([0-9]*\) passed.*/\1/p') +TMEM_F=$(echo "$TMEM_OUTPUT" | sed -n 's/^TEMPORAL_MEM: [0-9]* passed, \([0-9]*\) failed.*/\1/p') +TOTAL=$((TOTAL + TMEM_N + TMEM_F)) +PASS=$((PASS + TMEM_N)) +FAIL=$((FAIL + TMEM_F)) +if [ "$TMEM_RC" -ne 0 ]; then + echo " FAIL: temporal history memory gate (rc=$TMEM_RC)" +fi +echo "" + # [98] Cross-thread channel dict-key survival (#293). echo "[98] Cross-thread Channel Dict Keys (7 checks)" XCD_OUTPUT=$(./eigenscript ../tests/test_chan_dict_xthread.eigs 2>&1); XCD_OUTPUT_RC=$? diff --git a/tests/test_temporal_memory.sh b/tests/test_temporal_memory.sh new file mode 100755 index 00000000..b0aaca80 --- /dev/null +++ b/tests/test_temporal_memory.sh @@ -0,0 +1,166 @@ +#!/bin/bash +# Bounded-history gate for the temporal interrogatives (#827). +# +# WHAT BROKE: the per-name assignment history that backs `prev of x`, +# ` is x at ` and `state_at` was append-only with no cap, and it +# held a REFERENCE to every assigned value. Any long-running program that so +# much as mentioned `prev of` grew linearly forever — and the whole-program +# compile flag armed it even when the `prev of` sat in a function that was +# never called. It froze a 4 GB box: ~20 minutes of thrash with no OOM kill, +# power-cycle to recover. +# +# WHAT THIS MEASURES: peak RSS of the same program at two iteration counts a +# factor of 8 apart. Both a CEILING and FLATNESS are asserted — a ceiling +# alone would pass a slow leak, and flatness alone would pass a program that +# allocated a fixed enormous block. `/usr/bin/time -f %M` under a `ulimit -v` +# cap so a regression on a small box fails the test instead of taking the box +# down with it. +# +# WHY NOT A SANITIZER: this is not a leak — every byte was reachable from the +# history table and correctly freed at exit. LeakSanitizer reported nothing +# for the whole life of the bug. Unbounded RETENTION is only visible to +# process-level memory accounting. +# +# THE FOUR PROGRAMS, and the defect each one covers: +# base no temporal query at all — the floor every other case must meet +# dead `prev of` inside a function that is NEVER CALLED (defect A: +# arming was whole-program, so dead code taxed everything) +# live `prev of x` evaluated every iteration (defect B: legitimate use +# was unbounded too — this is the half that freezes machines) +# at_live `what is x at ` — the backward-query form, which needs the +# line-stamped history rather than just the depth-2 chain +# +# PLANTED-FAULT VALIDATION (#827): reverting the prune in prev_record_assign +# to the answer-preserving-but-weaker `> line` (keeping same-line duplicates) +# leaves every temporal answer correct and takes `live` from 2944 kB flat to +# 13952 kB / 90496 kB — this gate goes red, the semantic tests do not. That is +# the whole reason this file exists separately from test_temporal_pruning.eigs. +set -u + +EIGS="${EIGS:-./eigenscript}" +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT + +PASS=0 +FAIL=0 +ok() { echo " PASS: $1"; PASS=$((PASS+1)); } +bad() { echo " FAIL: $1${2:+ ($2)}"; FAIL=$((FAIL+1)); } + +if ! [ -x /usr/bin/time ]; then + echo " SKIP: /usr/bin/time not available (peak RSS unmeasurable)" + echo "TEMPORAL_MEM: 0 passed, 0 failed (skipped)" + exit 0 +fi + +# MUST NOT run against a sanitizer build: ASan's redzones and quarantine grow +# RSS on their own, which would both raise the floor past the ceiling and +# manufacture growth between the two iteration counts. Same rationale as +# test_http_rss_growth.sh. +if grep -qa "__asan_init" "$EIGS" 2>/dev/null; then + echo " SKIP: sanitizer build — peak RSS is dominated by ASan overhead" + echo "TEMPORAL_MEM: 0 passed, 0 failed (skipped)" + exit 0 +fi + +cat > "$TMP/base.eigs" <<'EOF' +N is num of (env_get of "N") +x is 0 +i is 0 +loop while i < N: + x is i * 1.5 + i is i + 1 +print of "done" +EOF + +# The `prev of` is in a function no call site reaches. +cat > "$TMP/dead.eigs" <<'EOF' +define never_called(v) as: + return prev of v + +N is num of (env_get of "N") +x is 0 +i is 0 +loop while i < N: + x is i * 1.5 + i is i + 1 +print of "done" +EOF + +cat > "$TMP/live.eigs" <<'EOF' +N is num of (env_get of "N") +x is 0 +i is 0 +loop while i < N: + x is i * 1.5 + p is prev of x + i is i + 1 +print of "done" +EOF + +cat > "$TMP/at_live.eigs" <<'EOF' +N is num of (env_get of "N") +x is 0 +i is 0 +loop while i < N: + x is i * 1.5 + i is i + 1 +q is what is x at 5 +print of "done" +EOF + +N_SMALL=200000 +N_BIG=1600000 + +# Peak RSS in KB, or empty on failure. The ulimit -v cap is the safety belt: +# a reverted fix hits it and dies instead of thrashing the box. +peak_kb() { + local prog="$1" n="$2" out rc + out=$( ulimit -v 1500000; N="$n" /usr/bin/time -f "PEAK_KB %M" \ + "$EIGS" "$prog" 2>&1 >/dev/null ) + rc=$? + if [ $rc -ne 0 ]; then + echo "" ; return 1 + fi + echo "$out" | sed -n 's/^PEAK_KB \([0-9]*\)$/\1/p' | tail -1 +} + +BASE_BIG=$(peak_kb "$TMP/base.eigs" "$N_BIG") +if [ -z "$BASE_BIG" ]; then + bad "baseline program did not run" "check $EIGS" + echo "TEMPORAL_MEM: $PASS passed, $FAIL failed" + exit 1 +fi +echo " baseline (no temporal query) at N=$N_BIG: ${BASE_BIG} kB" + +# CEILING: the floor plus a generous allowance. The bounded history is a +# handful of entries per name, so every case should land ON the baseline; 4x +# the baseline leaves room for allocator noise while sitting far below the +# pre-fix numbers (dead 53120 kB, live 203008 kB at this N). +CEIL=$(( BASE_BIG * 4 )) +# FLATNESS: peak must not grow with iteration count. Pre-fix, an 8x iteration +# increase multiplied peak RSS ~7x; bounded, the two are identical. +SLOP=$(( BASE_BIG / 2 )) + +for prog in dead live at_live; do + SMALL=$(peak_kb "$TMP/$prog.eigs" "$N_SMALL") + BIG=$(peak_kb "$TMP/$prog.eigs" "$N_BIG") + if [ -z "$SMALL" ] || [ -z "$BIG" ]; then + bad "$prog: run failed" "likely the ulimit -v cap — unbounded retention" + continue + fi + echo " $prog: N=$N_SMALL ${SMALL} kB N=$N_BIG ${BIG} kB" + if [ "$BIG" -le "$CEIL" ]; then + ok "$prog peak RSS under ceiling (${BIG} <= ${CEIL} kB)" + else + bad "$prog peak RSS over ceiling" "${BIG} kB > ${CEIL} kB" + fi + GROWTH=$(( BIG - SMALL )) + if [ "$GROWTH" -le "$SLOP" ]; then + ok "$prog peak RSS flat in iteration count (+${GROWTH} kB over 8x)" + else + bad "$prog peak RSS grows with iteration count" "+${GROWTH} kB over 8x, slop ${SLOP} kB" + fi +done + +echo "TEMPORAL_MEM: $PASS passed, $FAIL failed" +[ "$FAIL" -eq 0 ] diff --git a/tests/test_temporal_pruning.eigs b/tests/test_temporal_pruning.eigs new file mode 100644 index 00000000..3e32852f --- /dev/null +++ b/tests/test_temporal_pruning.eigs @@ -0,0 +1,100 @@ +# test_temporal_pruning.eigs — #827. +# +# The assignment history is reachability-pruned (trace.c): an entry that no +# backward query could ever reach is dropped at append time, which is what +# bounds the history. These checks pin the four cases a naive prune breaks. +# +# LINE-NUMBER SENSITIVE: every `at ` below hardcodes a line of this +# file. Append new checks at the END, before the final if/else, and re-verify +# the annotated line numbers with `grep -n`. + +pass_count is 0 +fail_count is 0 + +define check(args) as: + label is args[0] + cond is args[1] + if cond: + pass_count is pass_count + 1 + else: + fail_count is fail_count + 1 + print of ("FAIL: " + label) + +# ---- (1) The backward-jump counterexample. +# +# `at L` walks the history TEMPORALLY BACKWARD — the most recent assignment +# whose line is <= L — which is NOT "the value at the greatest line <= L". +# `back` is assigned at line 38, and then again at the LOWER line 32 (calling +# a function defined above the call site). A line-keyed table would answer the +# line-38 value for L=40; the correct answer is the line-32 one, because it +# happened later. +define bump() as: # line 31 + back is 200 # line 32 <- assigned SECOND, lower line + +back is 100 # line 34 (declare before the call) +bump of 0 # line 35 +check of ["backward-jump seeds", (what is back at 40) == 200] + +back is 300 # line 38 <- assigned THIRD, higher line +bump of 0 # line 39 <- assigns at line 32 + +check of ["backward: latest wins at L=40", (what is back at 40) == 200] +check of ["backward: latest wins at L=34", (what is back at 34) == 200] +check of ["backward: latest wins at L=33", (what is back at 33) == 200] +check of ["backward: miss before first", (what is back at 31) == null] +check of ["backward: when counts all", (when is back at 40) == 4] +check of ["backward: when below line 34", (when is back at 33) == 2] + +# ---- (2) `prev of x at L` after the predecessor was pruned. +# +# `r` is assigned three times at the SAME line (57) inside a loop. Only the +# last of those is reachable, so the middle one is pruned — but it is exactly +# the answer `prev of r at 60` must give, so each surviving entry has to carry +# its own execution-order predecessor. +r is 0 +j is 0 +loop while j < 3: + r is (j + 1) * 7 # line 57: 7, 14, 21 + j is j + 1 + +check of ["pruned prev: at line", (prev of r at 60) == 14] +check of ["pruned prev: live chain", (prev of r) == 14] +check of ["pruned what: latest", (what is r at 60) == 21] +check of ["pruned when: counts pruned", (when is r at 60) == 4] + +# ---- (3) A long same-line run must stay exact (and must not grow). +big is 0 +k is 0 +loop while k < 5000: + big is k # line 69 + k is k + 1 + +check of ["long run: what", (what is big at 75) == 4999] +check of ["long run: prev", (prev of big at 75) == 4998] +check of ["long run: when", (when is big at 75) == 5001] +check of ["long run: miss before", (what is big at 65) == null] +bs is state_at of 75 +check of ["long run: state_at", bs["big"] == 4999] + +# ---- (4) Alternating lines — the case a same-line-run collapse would miss. +# +# `alt` alternates between lines 85 and 87. Only the last pair is reachable. +alt is 0 +m is 0 +loop while m < 400: + alt is m # line 85 + # (this comment holds line 86 so `alt` alternates 85/87) + alt is m + 1000 # line 87 + m is m + 1 + +check of ["alt: at 85", (what is alt at 85) == 399] +check of ["alt: at 87", (what is alt at 87) == 1399] +check of ["alt: prev at 87", (prev of alt at 87) == 399] +check of ["alt: prev at 85", (prev of alt at 85) == 1398] +check of ["alt: when at 85", (when is alt at 85) == 401] +check of ["alt: when at 87", (when is alt at 87) == 801] + +if fail_count == 0: + print of "All tests passed" +else: + print of fail_count From 121ce4fc2162af379992c669e337d4d674fa3709 Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Mon, 3 Aug 2026 03:17:33 -0500 Subject: [PATCH 2/2] trace: close the MT race in #827's armed-name set; skip the RSS gate off Linux MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects in my own #827 fix, found by reviewing the new shared state and by CI's macOS legs. 1. USE-AFTER-FREE under `spawn`. #827 filters the (per-thread, #739) history table through a PROCESS-global armed-name set that the compiler grows with `realloc`. Single-threaded that is fine — compile, then run. But a worker calling `eval`/`load_file` compiles *concurrently* with other workers recording assignments, so the realloc lands under a reader walking the array: a UAF, not the benign torn-int race `g_trace_hist` already had. Nothing in the suite or CI's TSan leg exercises eval-on-a-worker with a temporal query, so this would not have been caught. Fix: `spawn` widens to the wildcard as its last single-threaded act, before the first `pthread_create` — so the value is published to every worker by the same happens-before #297 relies on, and from then on the filter reads two ints and the name array is never touched again. It uses a separate entry point (`trace_arm_history_all_mt`) that does NOT set `g_trace_hist`: a program with no temporal query must not start recording just because it made a thread. Verified — a spawning program with no temporal query stays at the 2944 kB floor, flat from 200k to 1.6M iterations, and a spawning program that does use `prev of` / `at` / `when` still answers correctly. The narrowing is a per-assign CPU optimization for the single-threaded long-running programs #827 was actually about; giving it up under MT costs nothing that matters, because the history is bounded either way. 2. macOS: the new RSS gate is Linux-only and must SKIP, not fail. BSD `/usr/bin/time` has no `-f` and no `%M`, and `ulimit -v` is a no-op there, so both macOS legs went red on `[70d]`. The gate now probes the exact invocation (`/usr/bin/time -f ... true`) rather than the platform name — a Linux box without GNU time skips for the same reason — and additionally requires `/proc`, since the thresholds are calibrated against Linux VmRSS accounting. Same platform reasoning as test_http_rss_growth.sh. Validated in both directions, because a gate that skips when it shouldn't is a dead gate: with a stub that mimics BSD `time` it prints SKIP and exits 0; on Linux it still runs all 6 checks; and a genuinely missing binary still FAILS loudly rather than skipping. Gates re-run on the final tree: release suite 3541/3541; ASan+UBSan with detect_leaks=1, leak tally still 0; make dap + DAP suite 30/30; freestanding-check; jit-smoke. Closes #827 Co-Authored-By: Claude Opus 5 (1M context) --- src/builtins.c | 12 ++++++++++++ src/trace.c | 11 +++++++++-- src/trace.h | 14 +++++++++++++- tests/run_all_tests.sh | 1 + tests/test_temporal_memory.sh | 21 +++++++++++++++++---- 5 files changed, 52 insertions(+), 7 deletions(-) diff --git a/src/builtins.c b/src/builtins.c index 749fb80c..02d0c0b9 100644 --- a/src/builtins.c +++ b/src/builtins.c @@ -3526,6 +3526,18 @@ Value* builtin_spawn(Value *arg) { * READ it (the value is already published to all workers via the first * write's happens-before through their pthread_create). */ if (!g_vm_multithreaded) g_vm_multithreaded = 1; + /* #827: #739's per-thread history is filtered by a PROCESS-global armed-name + * set that the compiler grows. Single-threaded that is fine (compile, then + * run), but a worker calling eval/load_file compiles concurrently with other + * workers recording assignments — a realloc of the name array under a + * reader is a use-after-free, not just a torn read. So the last + * single-threaded act before the first spawn is to widen to the wildcard, + * permanently: from here the filter reads only the two ints (the same + * benign shape as g_trace_hist itself) and the name array is never touched + * again. Costs nothing that matters — the history is bounded either way + * now; the narrowing is a per-assign CPU optimization for the + * single-threaded long-running programs #827 was actually about. */ + trace_arm_history_all_mt(); int pc_rc = pthread_create(&h->tid, NULL, thread_entry, h); if (pc_rc != 0) { /* The thread never started. Returning a live-looking handle here diff --git a/src/trace.c b/src/trace.c index eeb31244..2a51d0a4 100644 --- a/src/trace.c +++ b/src/trace.c @@ -191,13 +191,20 @@ static int arm_set_has(const char *name) { return 0; } -void trace_arm_history_all(void) { - g_trace_hist = 1; +/* Widen to the wildcard WITHOUT enabling recording. Separate from + * trace_arm_history_all because `spawn` calls it: a program with no temporal + * query must not start recording just because it made a thread. */ +void trace_arm_history_all_mt(void) { if (g_arm_all) return; g_arm_all = 1; g_arm_gen++; } +void trace_arm_history_all(void) { + g_trace_hist = 1; + trace_arm_history_all_mt(); +} + void trace_arm_history_name(const char *name) { g_trace_hist = 1; if (!name || g_arm_all) return; diff --git a/src/trace.h b/src/trace.h index eebed4db..852b7520 100644 --- a/src/trace.h +++ b/src/trace.h @@ -66,8 +66,20 @@ extern int g_trace_hist; * set only ever grows within a session. * * trace_history_disable() — the `record_history of 0` opt-out; clears both - * g_trace_hist and g_trace_obs_hist. */ + * g_trace_hist and g_trace_obs_hist. + * + * trace_arm_history_all_mt() — widen to the wildcard WITHOUT enabling + * recording. `spawn` calls this as its last single-threaded act: the armed-name + * set is process-global and the compiler grows it, so a worker running + * eval/load_file would realloc it under another worker reading it (a UAF, not + * a torn read). After widening, the filter reads only two ints and the name + * array is never touched again. A program with no temporal query must not + * start recording just because it made a thread, hence the separate entry + * point. The narrowing is a per-assign CPU optimization for the + * single-threaded long-running programs #827 was about; the history is + * bounded either way. */ void trace_arm_history_all(void); +void trace_arm_history_all_mt(void); void trace_arm_history_name(const char *name); void trace_history_disable(void); diff --git a/tests/run_all_tests.sh b/tests/run_all_tests.sh index dbcccdf0..86d0dcb3 100755 --- a/tests/run_all_tests.sh +++ b/tests/run_all_tests.sh @@ -2829,6 +2829,7 @@ TMEM_OUTPUT=$(bash "$TESTS_DIR/test_temporal_memory.sh" 2>&1); TMEM_RC=$? echo "$TMEM_OUTPUT" | grep -E "^ (PASS|FAIL|SKIP|baseline|dead|live|at_live)" TMEM_N=$(echo "$TMEM_OUTPUT" | sed -n 's/^TEMPORAL_MEM: \([0-9]*\) passed.*/\1/p') TMEM_F=$(echo "$TMEM_OUTPUT" | sed -n 's/^TEMPORAL_MEM: [0-9]* passed, \([0-9]*\) failed.*/\1/p') +TMEM_N=${TMEM_N:-0}; TMEM_F=${TMEM_F:-0} TOTAL=$((TOTAL + TMEM_N + TMEM_F)) PASS=$((PASS + TMEM_N)) FAIL=$((FAIL + TMEM_F)) diff --git a/tests/test_temporal_memory.sh b/tests/test_temporal_memory.sh index b0aaca80..3fd3b3ed 100755 --- a/tests/test_temporal_memory.sh +++ b/tests/test_temporal_memory.sh @@ -46,8 +46,21 @@ FAIL=0 ok() { echo " PASS: $1"; PASS=$((PASS+1)); } bad() { echo " FAIL: $1${2:+ ($2)}"; FAIL=$((FAIL+1)); } -if ! [ -x /usr/bin/time ]; then - echo " SKIP: /usr/bin/time not available (peak RSS unmeasurable)" +# GNU time only. BSD/macOS `/usr/bin/time` has no `-f` and no `%M`, and its +# maxrss units differ, so probe the exact invocation rather than the platform +# name — a Linux box without GNU time skips for the same reason. +if ! /usr/bin/time -f "PEAK_KB %M" true >/dev/null 2>&1; then + echo " SKIP: GNU /usr/bin/time -f not available (peak RSS unmeasurable)" + echo "TEMPORAL_MEM: 0 passed, 0 failed (skipped)" + exit 0 +fi + +# Linux only: the thresholds below are calibrated against Linux VmRSS +# accounting, and `ulimit -v` (the safety belt that turns a regression into a +# failed test instead of a dead box) is a no-op on macOS. Same platform +# reasoning as test_http_rss_growth.sh. +if [ ! -r /proc/self/status ]; then + echo " SKIP: /proc not available (RSS accounting differs on this platform)" echo "TEMPORAL_MEM: 0 passed, 0 failed (skipped)" exit 0 fi @@ -115,8 +128,8 @@ N_BIG=1600000 # a reverted fix hits it and dies instead of thrashing the box. peak_kb() { local prog="$1" n="$2" out rc - out=$( ulimit -v 1500000; N="$n" /usr/bin/time -f "PEAK_KB %M" \ - "$EIGS" "$prog" 2>&1 >/dev/null ) + out=$( ulimit -v 1500000 2>/dev/null || true + N="$n" /usr/bin/time -f "PEAK_KB %M" "$EIGS" "$prog" 2>&1 >/dev/null ) rc=$? if [ $rc -ne 0 ]; then echo "" ; return 1