Wuggy 2.0: graph-based core reengineering, sampling modes, terminology

.gitignore

@@ -146,4 +146,7 @@ build_documentation
 .DS_Store
 
 # Claude Code
-.claude/
+.claude/
+
+# Wuggy binary chain caches (written next to language data files)
+*.graph.pkl

dev-requirements.txt

@@ -1,9 +1,8 @@
 autopep8==2.0.4
 pycodestyle==2.11.0
 pylint==3.0.3
-python-Levenshtein==0.25.0
 setuptools==69.1.0
 wheel==0.42.0
 twine==5.0.0
 pdoc3==0.10.0
-statsmodels==0.14.1
+statsmodels==0.14.1

documentation/design/extract-graph-kernel.prompt.md

@@ -0,0 +1,108 @@
+# Task prompt: sketch the extraction of a domain-agnostic graph kernel
+
+Paste everything below the line into a new agent session, working from
+the repository root on branch `claude/wuggy-2.0-reengineering`.
+
+---
+
+You are working in the Wuggy repository (a multilingual pseudoword
+generator) on the local, unpushed branch `claude/wuggy-2.0-reengineering`.
+The 2.0 rewrite turned the core bigram store into an interned integer DAG
+with NumPy edge arrays, boolean-mask filters, exact path counting, and
+uniform/weighted sampling. That core is in
+`wuggy/utilities/segmentgraph.py` (class `SegmentGraph`, vertex namedtuple
+`Vertex`). Background on why this is worth doing is in
+`documentation/design/positional-graph-abstraction.md` — read it first.
+
+## Goal
+
+Sketch — and, if it comes out clean, implement — the extraction of a
+**domain-agnostic weighted positional-DAG kernel** out from under the
+Wuggy-specific `SegmentGraph`, with **no change in observable behavior**.
+The deliverable is primarily a design: the module boundary, the kernel's
+public interface, and exactly which state and methods move versus stay.
+Implementing the refactor is welcome if it is verifiable against the
+existing harness, but a crisp proposal with a skeleton is an acceptable
+stopping point — do not force a half-finished rewrite.
+
+## The intended seam
+
+The kernel (proposed name `PositionalGraph`, in
+`wuggy/utilities/positionalgraph.py`) should own everything that has no
+knowledge of language:
+
+- symbol interning and the `(position, symbol)` vertex identity, treating
+  each symbol payload as an **opaque hashable** — the kernel must not
+  reference `.letters`, `.hidden`, `Segment`, `SegmentH`, `separator`,
+  `transform`, or any `language_plugin`;
+- the flat edge arrays (`edge_src/dst/freq/pos`), the `edge_mask`, the
+  position slices, and the view/mask machinery (`_view`,
+  `_invalidate_caches`, `_edges`, `_adjacency`);
+- generic filter *primitives*: an edge-mask filter and a vertex-predicate
+  / vertex-pass-array filter, plus the numeric `frequency_filter` band
+  logic (which is already language-free);
+- `prune` / `_prune_in_place`, `set_start_vertices`, `get_frequencies`,
+  `count_paths`, `_completion_counts`, and all of `generate` /
+  `_generate_exhaustive` / `_generate_sampled` (the three search modes);
+- cache serialization of the array state, with the linguistic payload
+  reconstruction left to the subclass (see `to_cache_state` /
+  `from_cache_state`, which currently reach into `language_plugin.Segment`
+  — that reconstruction must move to the Wuggy layer).
+
+`SegmentGraph` then becomes a thin layer (subclass or composition — you
+decide, argue the trade-off) that adds only the linguistic parts:
+
+- `load()` parsing a plugin data file via `language_plugin.separator` and
+  `language_plugin.transform`;
+- `attribute_filter` and `segmentset_filter`, re-expressed in terms of the
+  kernel's generic vertex-predicate primitive (`segmentset_filter`'s use
+  of `segment.letters` is the clearest linguistic dependency;
+  `attribute_filter`'s `getattr(segment, attribute)` is nearly generic);
+- `build_limit_frequencies` (uses named fields);
+- the Segment/SegmentH reconstruction half of the cache round-trip.
+
+Watch for: the mapping-compat methods (`__len__`, `__contains__`, `keys`,
+`items`, `display`) are used by cold paths and tests — keep them working,
+and decide which layer they belong to. `WuggyGenerator` refers to
+`segment_graph`/`segment_graphs` and calls `attribute_filter`,
+`frequency_filter`, `prune`, `set_start_vertices`, `generate(mode=...)`,
+`get_frequencies`, `count_paths`; that public surface must not change.
+
+## Hard constraints
+
+- **No behavioral change.** This is a pure refactor.
+- Keep the existing public API of `SegmentGraph` and `WuggyGenerator`
+  intact (the rename to graph terminology already happened this branch;
+  do not revert or re-alias it).
+- Python only (numpy + rapidfuzz are the runtime deps). No new deps.
+
+## How to verify
+
+There is an equivalence harness and a sampling test suite. Set up a venv
+with `numpy` and `rapidfuzz`, then:
+
+1. `python tests/synthetic_plugin/make_data.py` (regenerates committed
+   synthetic data deterministically; should be a no-op).
+2. Capture behavior on the current commit **before** refactoring:
+   `python tests/capture_behavior.py --out /tmp/before.json`.
+3. Do the extraction.
+4. `python tests/capture_behavior.py --out /tmp/after.json` and confirm
+   `diff /tmp/before.json /tmp/after.json` is empty — this proves edge
+   dumps (type and token weighting), start vertices, exhaustive
+   generation sets, and all statistics are byte-identical.
+5. `python tests/test_sampling.py` must pass (path counts, set-equality
+   of all three generate modes, seeded determinism, weighted bias,
+   generator API integration).
+
+If you implement it, commit on the same branch with a message explaining
+the seam; do not push. If you only sketch it, write the proposal to
+`documentation/design/graph-kernel-extraction-plan.md` with the concrete
+method/state partition and the proposed kernel interface.
+
+## Deliverable
+
+Either a verified refactor (both checks green) or a written extraction
+plan precise enough to implement without rediscovering the seam — the
+exact list of what moves, what stays, the kernel's public signature, and
+how `SegmentGraph` re-expresses `attribute_filter` / `segmentset_filter`
+on top of it.

documentation/design/positional-graph-abstraction.md

@@ -0,0 +1,121 @@
+# The positional-graph abstraction
+
+*Design note, written during the 2.0 reengineering (branch
+`claude/wuggy-2.0-reengineering`).*
+
+## Context
+
+The 2.0 rewrite replaced Wuggy's dict-of-dicts bigram store with an
+interned integer DAG held in flat arrays
+([`wuggy/utilities/segmentgraph.py`](../../wuggy/utilities/segmentgraph.py),
+class `SegmentGraph`). Filters became boolean masks over the edge array,
+`prune()` became a single backward-reachability sweep, and generation
+gained exact path counting (`count_paths`) plus uniform and weighted
+sampling-without-replacement modes, all driven by one backward dynamic
+programming sweep over the layers.
+
+A consequence worth recording: almost none of that machinery is about
+language. It is a general data structure that happens to be carrying
+subsyllabic segments today.
+
+## What the structure actually is
+
+Strip away the segments and lexicons and it is:
+
+> **A weighted, layered DAG — equivalently an acyclic weighted
+> finite-state automaton — over positional symbols, supporting
+> composable hard-constraint masks, exact path counting, and unbiased
+> sampling without replacement, all driven by one backward DP.**
+
+Each *position* is a layer; each vertex is a `(position, symbol)` pair;
+edges connect consecutive layers with weights equal to co-occurrence
+counts (or summed token frequency). Every attested and every generable
+sequence is a path from a start vertex to a vertex in the final layer.
+
+## The four reusable primitives
+
+1. **Compact representation of a combinatorial set as paths.** A set of
+   sequences too large to materialize is held as a small graph.
+
+2. **Constrain without materializing.** Each filter is a boolean AND
+   over the edge array; the admissible set is narrowed by intersecting
+   masks, never by enumerating and rejecting. This is what beats
+   rejection sampling when the admissible set is a tiny fraction of the
+   whole.
+
+3. **Count exactly.** `count_paths` answers a #P-style question that is
+   tractable *because the graph is a DAG* — one DP sweep. The weighted
+   version of the same sweep is the **partition function** of the
+   positional distribution.
+
+4. **Sample uniformly or by weight, without replacement, exactly.**
+
+## Applications, sorted by how cleanly they map
+
+### Maps directly (same machinery, different alphabet)
+
+- **Template-matched generation in general.** Wuggy is one instance of
+  "generate items resembling a template, varying a controlled proportion
+  of elements, drawn from attested statistics." The same engine drives
+  procedural names, style-matched synthetic/fuzz test corpora, or
+  de-novo biological sequence candidates (peptides, oligos, SMILES
+  fragments) under positional-composition constraints and attested
+  transition frequencies.
+
+- **Wordlikeness / typicality scoring, not just generation.** Transition
+  frequencies are already computed as a statistic; exposed as a
+  normalized scorer (transition-product ÷ partition function) it is a
+  general "how typical is this string for this lexicon" function —
+  psycholinguistic norming, narrow-domain spell/OCR/ASR candidate
+  rescoring, structural anomaly detection (low weight = atypical).
+
+- **Design-space analytics for experiments.** "How many stimuli survive
+  these controls?" answered instantly, plus quantities the partition
+  function unlocks: **entropy of the constrained sub-language** and **KL
+  divergence between two constrained sub-languages** — e.g. how much a
+  frequency band or a segment restriction actually shrinks and reshapes
+  the space.
+
+### Maps with a modest extension
+
+- **Layered constraint satisfaction / solution counting.** Read each
+  position as a CSP variable and the masks as a constraint store:
+  crossword filling, positionally-constrained scheduling, configurable-
+  product (feature-model) enumeration. Uniform sampling of *solutions*
+  and exact solution-counting are cheap here for the same DAG reason.
+
+## Honest scope boundaries
+
+- The model is **first-order and acyclic.** Higher-order context means
+  vertices become `(position, k-gram)` — same machinery, more vertices.
+  True cycles (a non-positional automaton) break the layered backward
+  sweep and would need a general topological-order DP, or matrix methods
+  in the cyclic case.
+
+- The weighting is a **product of raw frequencies, not a smoothed
+  probability model.** Fine for ranking and relative comparison, but
+  unseen transitions are hard zeros; treat it as a partition function
+  over attested mass, not a calibrated language model.
+
+- The masks express **hard** constraints well; soft/weighted constraints
+  beyond the frequency weighting are not modeled.
+
+## Implication: the extraction seam
+
+The core already contains no linguistics. This argues for a clean seam
+between:
+
+- a **domain-agnostic kernel** — call it a `PositionalGraph` /
+  weighted-DAG module — owning interning, edge arrays, masks, counting,
+  and the three sampling modes; and
+
+- the **Wuggy-specific layer** — segments, ONC syllabification,
+  lexicality, OLD20, plugin-driven parsing — sitting on top of it.
+
+That is also exactly the boundary worth compiling if the counting-and-
+sampling kernel ever becomes a Rust / `extendr` core: the kernel is the
+part worth reusing and the part worth making fast, and it is separable
+from everything that makes Wuggy specifically about pseudowords.
+
+See [`extract-graph-kernel.prompt.md`](extract-graph-kernel.prompt.md)
+for a task prompt that sketches this extraction.

requirements.txt

@@ -1,2 +1,3 @@
-Levenshtein >= 0.12.0
+numpy >= 1.22
+rapidfuzz >= 3.0
 statsmodels >= 0.12.1

setup.py

@@ -8,7 +8,7 @@
 
 setuptools.setup(
     name="wuggy",
-    version="1.2.0",
+    version="2.0.0",
     author="Emmanuel Keuleers",
     author_email="E.A.Keuleers@tilburguniversity.edu",
     description="Wuggy: A multilingual pseudoword generator",