Wuggy 2.0: graph-based core reengineering, sampling modes, terminology

[uncle-hedgehog]

Major version release (2.0.0). Reengineers the pseudoword-generation core for efficiency, adds new generation capabilities, splits the core into a reusable kernel plus a linguistic layer, and aligns the code's vocabulary with the graph model described in the Wuggy literature. The public API of WuggyGenerator is preserved; internals, dependencies, and some structure/attribute names change.

What changed

Core representation (SegmentGraph, was BigramChain). The dict-of-dicts store keyed by namedtuples is replaced by a layered DAG over interned integer (position, segment) vertices held in flat NumPy arrays. Filters (attribute / frequency / segmentset) are now boolean masks over shared edge arrays instead of full graph copies; prune() (was clean()) is a single backward-reachability sweep instead of a recursive copy-until-fixpoint; generation is an iterative shuffled DFS. Parsed graphs are cached to a binary .graph.pkl next to the data file, keyed by mtime/size, so the corpus text is parsed once per version.

Kernel extraction (PositionalGraph). The graph mechanics are split into a domain-agnostic kernel (wuggy/utilities/positionalgraph.py) — interning, edge arrays, masks, pruning, exact counting, and the three generation modes — with SegmentGraph reduced to the linguistic layer on top (plugin-driven parsing, segment-field filters, cache reconstruction of plugin namedtuples). The kernel treats symbol payloads as opaque hashables and contains no linguistics; see documentation/design/ for the rationale (it is a weighted positional DAG / acyclic WFA with exact counting and unbiased constrained sampling, reusable beyond pseudowords).

Generator hot paths. Set-based sequence/lexicon caches (were O(n) list scans), getattr instead of eval, one-level match copies instead of deepcopy, streamed file parsing.

Levenshtein statistics. old20, ned1, and ld1nn now use rapidfuzz's batched SIMD distance; python-Levenshtein is dropped. Runtime deps are now numpy + rapidfuzz.

New generation capabilities. count_paths() returns the exact number of candidates a (filtered) subgraph can produce without generating them. generate() gains three modes, plumbed through generate_advanced(mode=) and generate_classic/generate_gui (search_mode=):

• exhaustive — the classic shuffled DFS (default, unchanged);
• uniform — every candidate exactly once in unbiased random order;
• weighted — every candidate exactly once, drawn proportional to the product of its transition frequencies (most word-like first).

Both sampling modes draw without replacement via a shared backward DP plus a prefix trie of drawn paths, so each candidate is yielded exactly once and the concentric-search drain-then-widen loop still terminates correctly.

Terminology rename (breaking, no aliases). The code now matches the "directed graph of subsyllabic segments" description in the literature: BigramChain->SegmentGraph, Link->Vertex, clean->prune, startkeys->start_vertices, WuggyGenerator.bigramchain(s)->segment_graph(s), *_subchain->*_subgraph.

Why a major version

Internals are replaced wholesale; dependencies change (numpy + rapidfuzz replace python-Levenshtein); all edge frequencies are floats now; word-lexicon buckets are sets; generate_classic match dicts are no longer deep-copied; and the token-frequency fix changes numeric output when token=True.

Verification

A committed harness (tests/capture_behavior.py over a deterministic synthetic language) captures order-independent behavior — edge dumps in both weighting modes, start vertices, exhaustive generation sets (~870k sequences across 9 references), and every statistic. Output is byte-identical across the 1.x baseline, the core rewrite, the rename, and the kernel extraction. tests/test_sampling.py verifies exact path counts, set-equality of all three generate modes, seeded determinism, weighted-order bias, and generator API integration.

Benchmarks (synthetic 40k-word lexicon): candidate evaluation loop ~4.6x, ned1+old20 ~4.1x, band filtering ~3.5x, warm load ~8.8x.

Notes for reviewers

• Shares the first commit (7525616, the token=True fix) with Fix token frequency being discarded in BigramChain.load #38. If Fix token frequency being discarded in BigramChain.load #38 merges first, this reduces to the 2.0 commits on top; if you prefer, that commit can be dropped here once Fix token frequency being discarded in BigramChain.load #38 lands.
• Language-plugin authors: custom statistics that referenced generator.bigramchain must use generator.segment_graph. The official plugins inherit from BaseLanguagePlugin (updated here); any that reference the old attribute directly need a matching update in the wuggy_language_plugin_data repo when 2.0 ships.
• Sampling modes use memory proportional to the number of candidates drawn (documented in the code); exhaustive mode remains the choice for memory-free full drains.

🤖 Generated with Claude Code