diff --git a/papers/disjoint-worlds/ADDENDUM_battery_and_holdout_limits_2026_09_01.md b/papers/disjoint-worlds/ADDENDUM_battery_and_holdout_limits_2026_09_01.md new file mode 100644 index 00000000..c480a397 --- /dev/null +++ b/papers/disjoint-worlds/ADDENDUM_battery_and_holdout_limits_2026_09_01.md @@ -0,0 +1,257 @@ +# ADDENDUM — three limits on the disjoint-worlds reads: what the battery and the held-out set do and do not license + +Fathom Lab · 2026-09-01 · a bounding document, not a retraction. + +**These three limits bound the arc's published reads. They do not overturn them.** The gemma +cross-family paired read of 0.7857 (`b31v2_result.json`) and the label-free read of 0.5714 +(`b34v3_result.json`, 0.5263 on the 57 genuinely-unseen concepts per +`b34v3_fresh_split_addendum.json`) were computed by the code that is committed, against gates +frozen before the runs, with nulls at or near chance. Nothing below contradicts any of those +numbers, and nothing below is evidence that the reads are artifacts. What the three limits do is +narrow the population those numbers describe and name one claim whose label is wider than its +method. Each was verified against the committed code for this document rather than taken from a +summary; where our verification disagreed with the account that prompted it, the verification is +what is written here. + +Nothing in the 2026-08 arc is edited by this addendum. `run_g0clear.py`, every `*_result.json`, +every certificate and every seal stay exactly as committed. The successor work lives in new files: +`build_concept_pool.py`, `concept_pool.json`, `PREREG_b52_pooled_battery_2026_09_01.md`. + +--- + +## Limit 1 — the selection that produced the battery is not recorded, so the extraction term is UNCHECKABLE + +### What is in the code + +`run_g0clear.py` line 31 opens a triple-quoted string literal `_BANK`, which runs to line 65 and +holds **465 whitespace-separated words** typed directly into the source across category blocks +(animals, fruit, food, tools, vehicles, furniture, clothing, nature, materials, buildings, +instruments, emotions, roles, body parts, colours, abstractions). Lines 66-67 are the entire +filtering chain: + +``` +_seen = set() +CONCEPTS = [c for c in _BANK if not (c in _seen or _seen.add(c))] +``` + +An order-preserving dedup. Verified by re-running the literal through `.split()` and a counter: it +removes exactly three words — **chicken, orange, mushroom** — each of which the author wrote into +two different category blocks. 465 becomes 462. + +There is nothing else. Verified by reading the whole 188-line file: **no tokenization filter, no +check against any target model's vocabulary, no frequency threshold, no part-of-speech filter, no +representation-quality gate.** `CONCEPTS` is used unchanged from line 67 onward, and the only +other thing that ever touches it is `CONCEPTS = CONCEPTS[:40]` under `--smoke`. + +The docstring at line 5 and the receipt field `parent_baseline.N` in +`g0clear_result_llama3b.json` both record the parent bank as **110**. The parent bank at +`run_thought_transfer.py` lines 31-40 contains **121** words, with no duplicates. Verified by +counting. All 121 appear in the 462. **341 words are net new, and how they were chosen is recorded +nowhere.** + +### Why this matters, stated precisely + +A read scored over a battery has two terms a reader needs: how well the reader identified the +items, and what share of candidate items reached the battery in the first place. Call the second +the **extraction term**. For these reads it is not merely unmeasured but **UNCHECKABLE**: no pool +was recorded, no rejection log exists, there is no source corpus and no seed to replay, so no +later work can reconstruct what was considered and refused. Provenance of the *file* is complete — +it is in git, it is byte-stable, every receipt names it. Provenance of the *selection* is absent. + +This is not evidence that the battery is biased. It is the statement that the question cannot be +asked of it. UNCHECKABLE is the verdict, and the absence of a rejection log is not itself a +contradiction of anything. + +### One measurable consequence, since a filter that was absent can be priced after the fact + +The extraction *chain* is unrecoverable, but the effect of one filter it lacked can be measured +now. Using the tokenizers of the four models this arc compares, **35 of the 462 words are +multi-token in Llama-3.2 (3B and 1B) and in Qwen2.5**; 2 of 462 are multi-token in gemma-2. **6 of +the 70 held-out concepts** are multi-token in at least one target: apricot, gorilla, tambourine, +peacock, cactus, melon. (Reproduced by `build_concept_pool.py`, which records the full list in +`concept_pool.json` under `comparison_to_2026_08_battery`.) + +The consequence should not be overstated, and the smaller version is the true one. +`CONCEPT_TEMPLATES` never puts the concept at the end of a sentence — every template continues +past the `{c}` slot — and `extract_multi` reads the **last** token of the whole sentence. So a +multi-token concept does not mean the read was taken on a word-piece. What it means is that +`template.format(c=concept)` and the neutral contrast `template.format(c="object")` are then +different lengths, so the two hidden states being subtracted sit at different absolute positions; +and that a string which is one token in gemma and two in Llama is not the same stimulus in the two +models being compared. Both are confounds of unmeasured size. Neither is shown here to have moved +any published number, and no claim is made that it did. + +### What would have made it checkable + +A pool recorded as an artifact before the selection, and a filter chain that counts what it +removes at every stage. That is `build_concept_pool.py`, committed alongside this addendum. It +draws 547,373 candidate types by a stated rule from a version-pinned corpus, applies eight named +filters each recording its rule, its removal count and a sha256 of what it left, and samples the +battery from the survivors under a recorded seed. Its extraction term is +**462 / 547,373 = 0.00084403**, with 462 / 5,199 = 0.088863 of the eligible set. The same quantity +for the 2026-08 battery is UNCHECKABLE, and `concept_pool.json` records it with that word rather +than with a number. + +--- + +## Limit 2 — the held-out set has been scored repeatedly, and the arc's hyperparameters were selected on a sibling split + +### What is in the code + +`split_concepts(seed=0)` in `run_g0clear.py` partitions the 462 concepts into 323 anchors, 69 +`SEL_dirs` and 70 `FIN_dirs`. Verified by reproducing the split: 323 / 69 / 70. + +Nine committed scripts import that function and score its `fin` set. Verified by reading each +one's use of the variable, not by grepping the name: + +| script | committed receipt | what it scores on FIN | +|---|---|---| +| `run_g0clear.py` | `g0clear_result_llama3b.json` | the locked `pc_cos` positive control | +| `run_b31v2.py` | `b31v2_result.json` | paired-MLP top-1, the 0.7857 cell | +| `run_b34.py` | `b34_result.json` | label-free top-1 | +| `run_b34v2.py` | `b34v2_result.json` | label-free top-1 | +| `run_b36.py` | `b36_result.json` | steering gain per concept | +| `run_g0_stage1.py` | `thought_transfer_g0clear_result_llama3b.json` | zero-anchor read top-1 and steer gains | +| `run_g0_stage1b.py` | `writelayer_decouple_result_llama3b.json` | native and transferred steer gains | +| `run_read_verify.py` | `read_verify_result.json` | read top-1 across a sweep | +| `run_rung2_read.py` | `rung2_read_result_Qwen2_5_1_5B_Instruct.json` | read top-1 | + +That is **nine, not seven**; `run_g0_stage1.py` and `run_g0_stage1b.py` also score it and were not +in the account that prompted this check. Two further scripts, `run_g0_stage3_truthaxis.py` and +`run_rung3_steerlayer.py`, take only `tr` from the split and are not on the list. + +### The map is never fitted on FIN. Two hyperparameters are selected on a slice of it. + +The cross-model map is fitted on anchors in every one of the nine. **In the two headline read +experiments — `run_b31v2.py` and `run_b34v3.py` — `fin` appears only in scoring**, and that is +worth saying plainly because the loose word for this problem is "leakage" and this is not leakage +in the training sense. + +Three write-side scripts do more than score it, and the account that prompted this check did not +mention it: + +- `run_g0_stage1.py` line 85: `alpha = P.lock_dose(..., fin[:5], ...)` — the injection dose is + chosen on the first five held-out concepts. +- `run_b36.py` lines 99-100: `dose_pool = list(fin)[:8]`, then `lock_dose` over it. +- `run_g0_stage1b.py` lines 26, 42, 68: `N_NATIVE_SEL = 12` with the comment *"held-out concepts + used to pick the steer-optimal layer"*; `native_sel = fin[:12]`; `f_star` is the write fraction + maximising mean gain over `native_sel`. + +In all three, `lock_dose` and the layer sweep score the **native** target direction `vecsB[c]` — +not a transferred one — so neither the map nor the transfer quantity is tuned on held-out data. +The bound is narrower than "leakage" and it is real: a hyperparameter was selected using items +that were subsequently scored, in the write-side experiments. **The two headline reads are clean +on this point.** + +### The layer and k the arc runs on were selected on SEL_dirs + +`g0clear_result_llama3b.json` records `locked: {layer: 11, k: 150}`, chosen as the argmax of +`pc_cos` over `SEL_dirs`. **Seventeen committed scripts in this directory hard-code that exact +receipt filename and read `k` (and in several cases the layer) out of it** — counted by grep and +confirmed by reading each call site: `run_b31v2`, `run_b34v2`, `run_b34v3`, `run_b35a`, +`run_b35b`, `run_b35c`, `run_b35c_nullreplicate`, `run_b37`, `run_b38`, `run_b39`, `run_b41`, +`run_b42`, `run_b44`, `run_b46`, `run_read_verify`, `run_rung2_read`, `diag_b35c_collapse`. Three +more (`run_g0_stage1`, `run_g0_stage1b`, `run_rung3_steerlayer`) read a `g0clear_result_.json` +chosen by command-line tag, which may resolve to one of the axis receipts instead. + +`run_b34v3.py` draws a fresh split at seed 343 with `n_fin = 70` and `n_tr = 462 - 70 = 392`, so +its training set is every concept not in its held-out set. Reproducing both splits: **60 of the 69 +`SEL_dirs` concepts fall inside b34v3's 392-concept training set, and 9 fall inside its 70-concept +held-out set.** The claim that SEL_dirs "sits inside" b34v3's training set is therefore true of 60 +of 69 and false of 9, and the precise version is the one that belongs in the record. The +hyperparameters b34v3 runs under were selected on concepts that are, for the most part, its own +training items — and for 9 items, its own test items. + +Two smaller observations, recorded because they are checkable and neither is a defect: the locked +pair sits at two corners of the swept grid (`layer 11` is the lowest of `[11, 14, 17, 20]`, `k 150` +the highest of `[60, 90, 120, 150]`), so the selection surface's optimum may lie outside what was +swept; and `grid_max_sel` equals `pc_cos_sel_at_locked` at 0.9131, as it must by construction. + +### The bound, stated without inflation + +A set that is scored across nine committed experiments is not a fresh held-out set for the ninth. +Each individual experiment's gate was frozen before its own run, and the arc's per-experiment +discipline is not in question. What accumulates across the arc is a different quantity: the +program's own choices — which method to keep, which successor to spawn, which map class to +abandon — were made by people who had seen this set's numbers many times. That is a bound on the +*arc*, not a defect in any single run, and no correction to any single number follows from it. +It is also not measurable retrospectively, so it is stated and left stated. + +The arc already carries the precedent for handling this properly. `run_b34v3.py` was written +specifically to draw a fresh split, its frozen prereg asserted the new held-out set was disjoint +in membership from `split_concepts(0)`, verification falsified that at 13 of 70, and +`b34v3_fresh_split_addendum.json` persists the true overlap together with the decisive read +recomputed on the 57 genuinely-unseen concepts (gemma 0.5263, llama 0.6667). We reproduced the +split independently and obtained 13 of 70, matching the committed receipt exactly. The prereg was +left frozen and wrong, and the correction was published rather than buried. That is the right +handling, and it is the reason the successor prereg +(`PREREG_b52_pooled_battery_2026_09_01.md`) measures freshness with a gate instead of asserting it. + +--- + +## Limit 3 — the "open-vocabulary" readout scores against a closed list of 462 + +### What is in the code + +`run_b35c.py` builds its candidate set at line 46: + +``` +vocab = fin + tr # the full open candidate set +``` + +`fin` is 70 held-out concepts, `tr` is the remaining 392 — the same hand-typed 462 imported as +`from run_g0clear import CONCEPTS as C`. `read462` then takes, for each held-out query, the +`argmin` of the distance from the mapped source point to `vocabB`, the 462 target-space points. +Chance is recorded as `1/462 = 0.00216`. + +**So "open vocabulary" here names a 462-way closed-set identification, not open recall.** The +candidate set is larger than the 70-way set it replaced, and it now includes the anchors the map +was fitted toward, which is genuinely the adversarial direction. It is still a closed list, and it +is still the same hand-authored list whose selection provenance Limit 1 records as UNCHECKABLE. An +open-vocabulary readout in the ordinary sense would score against a vocabulary the battery did not +define — a tokenizer vocabulary, a frequency list, an unrestricted lexicon — and no committed +script in this directory does that. + +### Being fair about what was claimed + +The code says what it does. `run_b35c.py`'s own docstring states that each query is scored +"against ALL 462 concept points in the target space", and +`PREREG_b35c_open_vocab_2026_08_03.md` states it too: *"the query is scored against all 462 +concepts — the 392 anchors the MLP trained toward plus the 70 held-outs"*, with the new chance +written out as 1/462. Nothing was concealed at any point. The overclaim is in the **name**: the +prereg's framing sentence is "Real reading has no shortlist", and what the design then does is +replace a 70-item shortlist with a 462-item shortlist drawn from the same literal. + +`b35c_result.json` returned `INVALID__null_artifact` — `G2_null` failed at `max_null462 = 0.01429` +against a bar of 0.00648, exactly the single-coincidental-hit failure the prereg predicted in +advance and said it would not respond to by moving the bar. **No open-vocabulary claim is +published from this run**, and `FINDING_b35bc_generality_invalids_2026_08_03.md` carries the +invalids. The limit recorded here is therefore about vocabulary in this arc generally, not about a +standing published number: wherever the phrase "open vocabulary" appears in this program's prose, +it should be read as "462-way closed set over the 2026-08 battery" until a run scores against a +vocabulary the battery did not author. + +--- + +## What is now measurable that was not + +Three artifacts are committed with this addendum. + +- **`build_concept_pool.py`** derives a candidate pool from a version-pinned corpus by a stated + rule, then applies eight named filters, each recording what it removed and why and what that + cost. Every stage carries a sha256 of its survivors. The battery is a seeded sample of the + eligible set, not a choice. +- **`concept_pool.json`** is that record: pool 547,373 → eligible 5,199 → battery 462, survival + ratio 0.00084403, byte-reproducible across processes (verified by re-running under different + `PYTHONHASHSEED` values and comparing bytes). +- **`PREREG_b52_pooled_battery_2026_09_01.md`** preregisters the read on that battery with the + extraction term reported beside it, a fresh split seed, a measured rather than asserted + freshness gate, six null cells, and a two-sided gate requiring the pooled read to land within + 0.15 of the published 0.7857 — a bar derived from the binomial standard error at n = 70 rather + than chosen. It commits in advance to publishing a failure as a failure, and it names the + outcome in which the hand-authored battery turns out to have been load-bearing as the most + informative thing the design can produce. + +None of this recovers an extraction term for the existing reads. That number does not exist and +cannot be made to exist, and the honest form of the sentence is that these reads are real, +preregistered, null-controlled, and unpriced on one axis that their successors will be priced on. diff --git a/papers/disjoint-worlds/PREREG_b52_pooled_battery_2026_09_01.md b/papers/disjoint-worlds/PREREG_b52_pooled_battery_2026_09_01.md new file mode 100644 index 00000000..6630e1fe --- /dev/null +++ b/papers/disjoint-worlds/PREREG_b52_pooled_battery_2026_09_01.md @@ -0,0 +1,181 @@ +# PREREG — B52: the read re-run on a battery whose extraction term is known + +Fathom Lab · 2026-09-01 · frozen before any cell of this experiment runs. No number below is a +measurement; every number below is a threshold or an arithmetic derivation of one. + +Every published read in this arc is scored over the 462-concept battery at `run_g0clear.py` +lines 31-67. That battery is a hand-authored string literal whose only filter is an +order-preserving dedup removing three words, and no pool of candidates was ever recorded. The +**extraction term** — the share of candidate concepts that survived into the scored set — is +therefore not computable for those reads, and `ADDENDUM_battery_and_holdout_limits_2026_09_01.md` +records that as UNCHECKABLE rather than as a defect in the reads themselves. B52 does not revisit +those reads. It runs the same machinery on a battery whose extraction term **is** computable, and +reports the term beside the read. + +The successor battery already exists as a committed artifact: `concept_pool.json`, produced by +`build_concept_pool.py`. It records a pool of 547,373 candidate types drawn by a stated rule from +a pinned corpus, then eight named filters each carrying its rule, its survivor count, its removal +count and a sha256 of what it left. B52 scores the 462-word battery that receipt names, and no +other. + +## What is fixed before any cell runs + +**Battery.** The `battery.words` list of the committed `concept_pool.json`, identified by +`battery.sha256`. The runner reads the list from the receipt; it does not rebuild it. Rebuilding +is permitted only to *verify* the sha256, never to produce a different list. + +**Pool rule and filter chain.** Frozen in `build_concept_pool.py` as committed. Its constants — +`FREQ_FLOOR = 500`, `MIN_LEN, MAX_LEN = 3, 14`, `PROPER_CAP_SHARE = 0.5`, `POOL_SEED = 52`, +`BATTERY_N = 462`, `TARGET_MODELS`, `_SUFFIXES` — are frozen by this document. Changing any of +them after this prereg is committed VOIDS the run; a changed chain needs a new prereg and a new +receipt, not an edit. + +**Extraction.** `run_g0clear.extract_multi` unchanged: 12 `CONCEPT_TEMPLATES`, last-token hidden +state, differenced against the same sentence rendered with the neutral word `object`. Source +`meta-llama/Llama-3.2-3B-Instruct`. Targets `meta-llama/Llama-3.2-1B-Instruct`, +`google/gemma-2-2b-it`, `Qwen/Qwen2.5-1.5B-Instruct` — the three b31v2 and b34v3 scored, with +b31v2's committed per-model layer rule. + +**Split.** 462 concepts partitioned at **seed 5201** into 323 anchors / 69 selection / 70 +held-out — the same proportions and the same integer sizes as `split_concepts`, so chance is +1/70 = 0.0143 and the read is numerically comparable to the published ones. Seed 5201 has not +been used anywhere in this arc; the seeds already spent are 0, 31, 343 and 8080. **The seed is +not shopped.** The held-out set's overlap with previously-scored held-out sets was deliberately +not computed before this document was frozen; the runner computes it and G1 judges it. If G1 +fails, the run is VOID and a successor prereg draws one new seed, publishing the discarded seed +so that any shopping is visible in the record. + +**Arms, per target.** + +- **PAIRED** — the b31v2 M1 two-layer MLP fit on the 323 true anchor pairs, top-1 identification + of each held-out concept among the 70. +- **LABEL-FREE** — the b34v3 method unchanged: shuffle the target's anchor rows under the split + seed, discover the correspondence with `TransferMap.fit`, fit ONE MLP on the discovered + pseudo-pairs, read the 70 held-outs. +- **NULL** — for each arm and each target, the pairing-shuffled twin: same architecture, same + training budget, correspondence destroyed. Six null cells. + +**Hyperparameters, both ways.** The published reads take `layer` and `k` from +`g0clear_result_llama3b.json` (`layer 11`, `k 150`), selected on the *old* battery's SEL_dirs. +B52 runs the gated arms under those same committed values, so the reproduction gate compares like +with like. It **also** re-selects `(layer, k)` on B52's own 69-concept selection split and reports +the held-out read under the re-selected pair as a **diagnostic**, not a gate. Reporting both +prices the hyperparameter carry-over that the addendum names, without letting an ungated number +into a verdict. + +## Asserted invariants — recorded, deliberately not gated + +`styxx.protocol` refuses a gate no outcome row depends on, on the principle that a leg which +cannot fail must not gate. The following are facts of the frozen artifacts, so gating them would +be decoration; they are required fields of `b52_result.json` and their absence VOIDS the run. + +- `extraction_term.pool` = 547373, `extraction_term.eligible` = 5199, `extraction_term.battery` + = 462, `extraction_term.survival_ratio_battery_over_pool` = 0.00084403, + `extraction_term.sampling_fraction_battery_over_eligible` = 0.088863 — copied verbatim from + `concept_pool.json`, which is the artifact that makes them true. +- The per-filter ladder of `concept_pool.json` reproduced into the result, so a reader of the + result alone can see what each stage cost. +- The same quantities for the 2026-08 battery, recorded as `UNCHECKABLE` with the reason. + +## Gates (frozen; scored by `styxx.protocol` against `b52_result.json`) + +```gates +{"gates": {"G0_battery_integrity": {"metric": "battery_sha256_matches_pool_receipt", "op": ">=", "value": 1, + "power_basis": "an identity check, not a statistical bar. It can only fail if the scored word list is not the one concept_pool.json records, which is the single failure that would make every other number in this run unpriced again. Achievable by construction at zero compute; falsifiable if a runner substitutes a list.", + "metric_means": "1 if sha256 of the newline-joined scored battery equals concept_pool.json battery.sha256, else 0"}, + "G1_heldout_freshness": {"metric": "n_fin_previously_scored", "op": "<=", "value": 7, + "power_basis": "10 percent of a 70-item held-out set. The bar exists because b34v3 asserted membership disjointness in a frozen prereg and verification falsified it at 13 of 70 (b34v3_fresh_split_addendum.json); this program does not assert freshness again, it measures it. Computable before any forward pass, so failure costs no compute and produces a VOID rather than a tempting re-draw.", + "metric_means": "count of the 70 held-out concepts that also appear in the seed-0 FIN-70 or the seed-343 FIN-70 of the 2026-08 battery"}, + "G2_machinery": {"metric": "targets.llama_1b.read_paired_top1", "op": ">=", "value": 0.53, + "power_basis": "b31v2's own G0 bar verbatim (PREREG_b31v2_content_transport_2026_08_01.md: the committed 0.586 minus 0.05 slack), reused so the same-family sanity check is not re-derived on this run's convenience. b31v2_result.json measured 0.8000 against it on the old battery, so the bar is known reachable by this machinery.", + "metric_means": "same-family paired-MLP top-1 over the 70 held-out concepts"}, + "G3_decisive_paired": {"metric": "targets.gemma_2b.read_paired_top1", "op": ">=", "value": 0.143, + "power_basis": "b31v2's G1 bar verbatim: 10x chance at 1/70, binomial p < 1e-6 at n=70. Reachability on THIS battery is exactly what is unknown and is the reason the gate is here.", + "metric_means": "cross-family paired-MLP top-1 for gemma over the 70 held-out concepts"}, + "G4_decisive_labelfree": {"metric": "targets.gemma_2b.read_labelfree_top1", "op": ">=", "value": 0.143, + "power_basis": "b34v3's G1_bar verbatim (10x chance). Reused unchanged so the label-free arm is judged against the bar its own finding was judged against, not a new one.", + "metric_means": "cross-family label-free top-1 for gemma over the 70 held-out concepts"}, + "G5_null_bounded": {"metric": "max_null_top1", "op": "<=", "value": 0.0286, "agg": "max", "over": "nulls", + "power_basis": "2x chance, the null bar of both PREREG_b31v2 and PREREG_b34v3. b34v3 passed it at exactly the boundary and disclosed the knife edge; the bar is kept rather than loosened so a repeat of that is visible as a repeat. A 70-item null at 1/70 expects one hit, so two hits fail and that strictness is deliberate.", + "metric_means": "the largest of the six pairing-shuffled null reads (two arms x three targets)"}, + "G6_reproduces_old_read": {"metric": "abs_delta_paired_gemma_vs_b31v2", "op": "<=", "value": 0.15, + "power_basis": "derived, not chosen. b31v2_result.json records gemma paired top-1 = 0.7857 at n=70; the binomial standard error there is sqrt(0.7857*0.2143/70) = 0.0491, and two independent 70-item batteries differ with standard error sqrt(2)*0.0491 = 0.0694. The bar 0.15 is 2.16 of those, so ordinary sampling variation between two batteries cannot fail this gate while a battery effect larger than about a fifth of the read can. This gate is allowed to fail and its failure is a finding, not a miss.", + "metric_means": "absolute difference between this run's gemma paired top-1 and the 0.7857 recorded in b31v2_result.json"}}, + "outcomes": [{"when": {"G0_battery_integrity": false}, "verdict": "INVALID__battery_is_not_the_recorded_one"}, + {"when": {"G0_battery_integrity": true, "G1_heldout_freshness": false}, "verdict": "VOID__heldout_set_not_fresh"}, + {"when": {"G0_battery_integrity": true, "G1_heldout_freshness": true, "G2_machinery": false}, "verdict": "INVALID__machinery_broken"}, + {"when": {"G0_battery_integrity": true, "G1_heldout_freshness": true, "G2_machinery": true, "G5_null_bounded": false}, "verdict": "INVALID__null_fired"}, + {"when": {"G0_battery_integrity": true, "G1_heldout_freshness": true, "G2_machinery": true, "G5_null_bounded": true, "G3_decisive_paired": false}, "verdict": "POOLED_READ_BELOW_BAR__hand_authored_battery_was_load_bearing"}, + {"when": {"G0_battery_integrity": true, "G1_heldout_freshness": true, "G2_machinery": true, "G5_null_bounded": true, "G3_decisive_paired": true, "G6_reproduces_old_read": false}, "verdict": "POOLED_READ_CLEARS_BAR__BATTERIES_NOT_COMPARABLE"}, + {"when": {"G0_battery_integrity": true, "G1_heldout_freshness": true, "G2_machinery": true, "G5_null_bounded": true, "G3_decisive_paired": true, "G6_reproduces_old_read": true, "G4_decisive_labelfree": false}, "verdict": "PAIRED_READ_REPRODUCES_POOLED__labelfree_below_bar"}, + {"when": {"G0_battery_integrity": true, "G1_heldout_freshness": true, "G2_machinery": true, "G5_null_bounded": true, "G3_decisive_paired": true, "G6_reproduces_old_read": true, "G4_decisive_labelfree": true}, "verdict": "READ_REPRODUCES_ON_POOLED_BATTERY__extraction_term_priced"}], + "smoke_verdict": "INVALID__smoke_plumbing_only"} +``` + +## What each outcome licenses, written before it is known which one fires + +- **`READ_REPRODUCES_ON_POOLED_BATTERY__extraction_term_priced`** — both reads clear their + original bars on a battery that is a mechanically-derived sample of a recorded pool, and the + paired read lands within 0.15 of 0.7857. The reads then carry an extraction term, which we know of no other + read in this arc to carry: E = 0.00084403 of the recorded pool, 0.088863 of the eligible set. That is + the whole claim. It does not retroactively supply an E for the 2026-08 reads, and no document + may report it as if it did. +- **`PAIRED_READ_REPRODUCES_POOLED__labelfree_below_bar`** — the paired ceiling survives the + battery change and the label-free protocol does not. Licensed reading: label-free discovery is + more battery-sensitive than paired transport. This is a partial negative and gets reported with + the same prominence as a pass. +- **`POOLED_READ_CLEARS_BAR__BATTERIES_NOT_COMPARABLE`** — the read is real at 10x chance but is + far from the published level. **The two batteries are then not interchangeable and neither + number generalises to the other.** The pooled battery is broader in word class than the + hand-authored one (no offline part-of-speech tagger exists in this environment, so no filter + restricts it to concrete singular nouns) and that is the leading hypothesis a follow-up must + test. Stated now so it cannot be produced later as an explanation that was available all along. +- **`POOLED_READ_BELOW_BAR__hand_authored_battery_was_load_bearing`** — the decisive cross-family + read does not clear 10x chance on a pooled battery while the same machinery clears the + same-family bar and the nulls stay down. This is a **full result and the most informative + outcome this design can produce**: it would mean the published reads are bounded to the + properties of one hand-authored word list. It gets a FINDING of its own, and this program does + not re-run the cell with a different pool rule in the hope of a better number — the pool rule is + frozen above and a second pool needs a second prereg saying why. +- **`INVALID__*` / `VOID__*`** — no read is reported from a run that fires one of these, not even + as context. + +## VOID conditions, in addition to the gated ones + +The run is VOID — no numbers reported, not even as diagnostics — if any of the following holds. + +1. Any frozen constant of `build_concept_pool.py` differs from the committed version at run time, + or `concept_pool.json` does not reproduce byte-identically from it. +2. The split seed changes after any read has been computed, or any read is computed before G1 is + evaluated. +3. A target's extraction fails and its cell is dropped. The gated targets are fixed at three; a + two-target run is a different experiment. +4. Any concept is added to, removed from, or substituted into the battery by hand at any point. +5. The MLP architecture, training budget, or readout metric differs from b31v2's committed + apparatus. This run is about the battery; changing the machinery in the same firing makes the + comparison uninterpretable. +6. `b52_result.json` omits any asserted invariant listed above. + +## Rounding convention + +Every reported read and every null is rounded to four decimal places, and `max_null_top1` is the +maximum of the *rounded* per-cell values. `styxx.protocol`'s composition check compares the quoted +maximum against a recomputation over the declared `nulls` population at 1e-12, so a runner that +rounds one and not the other refuses loudly before any verdict. That refusal is the intended +behaviour and must not be worked around by relaxing the tolerance. + +## The failure this document promises to publish + +If the pooled battery does not reproduce the read, that is written up as a FINDING with the same +prominence as a pass, on the same day, with the numbers unhedged. The addendum accompanying this +prereg already records that the extraction term for the existing reads is UNCHECKABLE; a B52 that +comes back negative would sharpen that from "unpriced" to "priced and battery-dependent", and the +arc is more useful having said so than having not run it. + +## Discipline + +One GPU-bearing extraction pass over 462 concepts on four models, then CPU-only fits. Smoke uses a +40/10 split and writes `_smoke`-suffixed files, INVALID-only by type. Result `b52_result.json`, +scored by `styxx.protocol.Experiment(this_prereg).score(result)` so the verdict is computed rather +than chosen. `run_g0clear.py` and the 462-word `_BANK` are not touched by this experiment, and no +frozen artifact of the 2026-08 arc is edited by it. diff --git a/papers/disjoint-worlds/build_concept_pool.py b/papers/disjoint-worlds/build_concept_pool.py new file mode 100644 index 00000000..64858019 --- /dev/null +++ b/papers/disjoint-worlds/build_concept_pool.py @@ -0,0 +1,521 @@ +# -*- coding: utf-8 -*- +"""build_concept_pool.py — a concept battery that is a RECORDED SUBSET of a RECORDED POOL. + +WHY THIS EXISTS +--------------- +The battery this arc's published reads are scored over (`run_g0clear._BANK`, 462 concepts) is a +hand-authored string literal. Its only filter is an order-preserving dedup that removes three +words. No pool of candidates was ever written down, so the EXTRACTION TERM — the share of +candidate concepts that survived into the battery at all — is not computable after the fact. +Provenance of the file is complete; provenance of the SELECTION is absent. See +`ADDENDUM_battery_and_holdout_limits_2026_09_01.md`. + +This script builds the successor. It does not touch `run_g0clear.py` and does not regenerate the +462-word battery, which stays exactly as it is. + +WHAT IT GUARANTEES +------------------ +1. A POOL exists as an artifact BEFORE any selection, derived by a stated rule from a pinned, + nameable source. +2. Every filter is applied in a stated order and records its name, its rule, its survivor count + and exactly how many candidates it removed. +3. Every stage's surviving set is fingerprinted (sha256 over the sorted, newline-joined list), so + a stage can be verified without the receipt enumerating half a million strings. +4. The final battery is a SEEDED uniform sample of the eligible set. The seed is in the receipt. +5. The survival ratio final/pool is reported, and so is every intermediate ratio. + +WHAT IT DOES NOT GUARANTEE +-------------------------- +The filters below are justified in their `why` strings and their costs are stated in their `cost` +strings. They are still choices made by a person. What changes is that each choice is now a named +rule with a counted price attached, so a reader can disagree with a specific filter and recompute +what it cost. That is the whole of the improvement; it is not a claim that the resulting battery +is unbiased. + +No GPU. No model weights are loaded — only four tokenizer files. Runs in roughly two minutes on +CPU, most of it counting the corpus. + + python build_concept_pool.py # writes concept_pool.json + python build_concept_pool.py --dump-stages # also writes concept_pool_stages.json (large) + python build_concept_pool.py --smoke # first corpus shard only; writes nothing +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import sys +from collections import Counter +from pathlib import Path + +import numpy as np + +HERE = Path(__file__).resolve().parent + +# --------------------------------------------------------------------------------------------- +# PINS. Everything the output depends on, named and version-locked. A pool whose source drifts is +# a pool that cannot be replayed, which is the defect this file exists to fix. +# --------------------------------------------------------------------------------------------- +CORPUS_REPO = "wikitext" +CORPUS_CONFIG = "wikitext-103-raw-v1" +CORPUS_SPLIT = "train" +CORPUS_REVISION = "b08601e04326c79dfdd32d625aee71d232d685c3" +CORPUS_LICENCE = "CC BY-SA 3.0 (WikiText is derived from verified-good Wikipedia articles)" +CORPUS_CITE = "Merity, Xiong, Bradbury, Socher (2016), Pointer Sentinel Mixture Models, arXiv:1609.07843" + +# The models the arc actually compares. A concept must be one token in EVERY one of them or the +# same string is not the same stimulus across the comparison being made. +TARGET_MODELS = [ + "meta-llama/Llama-3.2-3B-Instruct", # source A in b31v2 / b34v3 / g0clear + "meta-llama/Llama-3.2-1B-Instruct", # target, same family + "google/gemma-2-2b-it", # target, the decisive cross-family cell (b31v2's `hf`) + "Qwen/Qwen2.5-1.5B-Instruct", # target, context cell +] +# gemma-2-2b-it and gemma-2-2b carry byte-identical vocabularies (verified by comparing +# get_vocab() on both), so F6 is insensitive to which of the two is named here. The -it variant +# is named because it is the model b31v2_result.json records as the decisive target. + +# The neutral contrast word `run_g0clear.extract_multi` differences against. A concept that is not +# the same token length as this word changes the sentence length of the contrast pair. +NEUTRAL_WORD = "object" + +FREQ_FLOOR = 500 # corpus occurrences, all case forms +MIN_LEN, MAX_LEN = 3, 14 # characters +PROPER_CAP_SHARE = 0.5 # >= this share of non-lowercase surface forms => treated as a name +BATTERY_N = 462 # matched to the 2026-08 battery so the reproduction gate is like-for-like +POOL_SEED = 52 # b52; recorded in the receipt + +# Inflectional suffixes whose stripped base, if itself a survivor, makes the pair near-duplicate. +# Deliberately EXCLUDES -er/-est/-or: agent nouns are distinct concepts (a driver is not a drive) +# and stripping them produced pure false positives in development (beer<-bee, forest<-fore). +_SUFFIXES = ("ies", "es", "s", "ing", "ed", "ly") + + +def _sha(items) -> str: + return hashlib.sha256("\n".join(items).encode("utf-8")).hexdigest() + + +# --------------------------------------------------------------------------------------------- +# STAGE 0 — THE POOL +# --------------------------------------------------------------------------------------------- +def build_pool(smoke: bool = False): + """POOL RULE (stated before any filter runs): + + Every distinct case-folded string obtained by splitting the pinned corpus split on + whitespace. WikiText is distributed space-tokenized (punctuation is space-separated), so a + whitespace split yields the corpus's own token inventory, not a tokenization of ours. + + What this rule already excludes, and therefore never prices: anything the corpus does not + contain, and any multi-word concept. Both are real narrowings and they are the pool's + boundary rather than a filter's cost. Multi-word concepts are out of scope for this battery + because the extraction template renders one word into one slot. + """ + from huggingface_hub import snapshot_download + import pyarrow.parquet as pq + + root = Path(snapshot_download( + repo_id=CORPUS_REPO, repo_type="dataset", revision=CORPUS_REVISION, + allow_patterns=[f"{CORPUS_CONFIG}/{CORPUS_SPLIT}-*.parquet"])) + shards = sorted((root / CORPUS_CONFIG).glob(f"{CORPUS_SPLIT}-*.parquet")) + if not shards: + raise SystemExit(f"no {CORPUS_SPLIT} shards under {root / CORPUS_CONFIG}") + if smoke: + shards = shards[:1] + + lower = Counter() # surface form was entirely lowercase + initcap = Counter() # surface form was Xxxx + other = Counter() # anything else (ALLCAPS, MiXeD) + lines = 0 + for shard in shards: + for batch in pq.ParquetFile(shard).iter_batches(batch_size=20000, columns=["text"]): + for s in batch.column("text").to_pylist(): + if not s: + continue + lines += 1 + for w in s.split(): + k = w.lower() + if w.islower(): + lower[k] += 1 + elif w[:1].isupper() and w[1:].islower(): + initcap[k] += 1 + else: + other[k] += 1 + + types = sorted(set(lower) | set(initcap) | set(other)) + total = {k: lower[k] + initcap[k] + other[k] for k in types} + meta = { + "rule": ("distinct case-folded whitespace-delimited strings of the pinned corpus split; " + "no filtering of any kind at this stage"), + "source": {"repo_id": CORPUS_REPO, "repo_type": "dataset", "config": CORPUS_CONFIG, + "split": CORPUS_SPLIT, "revision": CORPUS_REVISION, + "licence": CORPUS_LICENCE, "citation": CORPUS_CITE, + "shards": [s.name for s in shards]}, + "lines_read": lines, + "token_occurrences": int(sum(total.values())), + "pool_size": len(types), + "sha256_pool": _sha(types), + "not_priced_here": ("strings absent from this corpus, and every multi-word concept: both " + "are excluded by the pool's own boundary and no filter below charges " + "for them"), + } + return types, total, initcap, other, meta + + +# --------------------------------------------------------------------------------------------- +# THE FILTER CHAIN. Order matters and is fixed here; each entry states why it exists and what it +# costs. A filter with no stated cost is a filter no reader can price, which is the failure mode +# this file was written against. +# --------------------------------------------------------------------------------------------- +def apply_chain(types, total, initcap, other, tokenizers, log_cap=25): + stages = [] + survivors = list(types) + + def step(name, rule, why, cost, keep_fn, cap=None): + nonlocal survivors + cap = log_cap if cap is None else cap + before = list(survivors) + kept, dropped = [], [] + for t in before: + (kept if keep_fn(t) else dropped).append(t) + survivors = kept + stages.append({ + "filter": name, "rule": rule, "why": why, "cost": cost, + "n_in": len(before), "n_removed": len(dropped), "n_out": len(kept), + "removed_fraction_of_input": round(len(dropped) / len(before), 6) if before else 0.0, + "removed_examples": dropped[:cap], + "removed_enumerated": len(dropped) <= cap, + "sha256_survivors": _sha(kept), + }) + return stages[-1] + + ascii_alpha = re.compile(r"^[a-z]+$") + step( + "F1_ascii_alphabetic", + "keep t matching ^[a-z]+$ after case-folding", + "The battery renders a bare English word into an English sentence template. Strings " + "carrying digits, punctuation, or non-ASCII letters are not bare English words, and the " + "corpus's whitespace inventory is full of them (numerals, the @-@ hyphen marker, " + "transliterations).", + "Refuses accented loanwords (cafe with an acute) and every hyphenated compound, which " + "WikiText writes with @-@ separators. Those are real concepts and this filter will not " + "admit them.", + lambda t: bool(ascii_alpha.match(t)), + ) + + step( + "F2_length_3_to_14", + f"keep {MIN_LEN} <= len(t) <= {MAX_LEN}", + "One- and two-character strings in this corpus are dominated by initials, roman numerals " + "and unit abbreviations rather than concepts; strings longer than 14 characters are " + "dominated by concatenation artefacts and chemical names.", + "Loses genuine short concepts (ox, ax, pi) and long ones (thunderstorm is fine at 12, but " + "e.g. responsibility at 14 is at the boundary and anything longer is refused outright).", + lambda t: MIN_LEN <= len(t) <= MAX_LEN, + ) + + step( + "F3_frequency_floor", + f"keep total corpus occurrences (all case forms) >= {FREQ_FLOOR}", + "The concept vector is a mean over 12 template renderings of a sentence-final hidden " + "state. For a type the model has seen rarely, that mean is dominated by tokenizer " + "segmentation and noise rather than by a stable representation, and a battery of such " + "items measures the extraction pipeline instead of the model.", + "This is by far the largest narrowing in the chain and it BIASES the battery toward " + "common English. A read scored on this battery licenses no claim about rare or technical " + "concepts. Separately, WikiText frequency is a PROXY for the target models' pretraining " + "frequency, which is not observable; the substitution is a known and unmeasured " + "approximation.", + lambda t: total[t] >= FREQ_FLOOR, + ) + + from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS + stop = frozenset(ENGLISH_STOP_WORDS) + step( + "F4_not_function_word", + "drop t in sklearn.feature_extraction.text.ENGLISH_STOP_WORDS " + f"({len(stop)} entries, Glasgow IR stop list, shipped under scikit-learn's BSD-3 licence)", + "The template 'The {c} was the first thing everyone noticed.' is meaningless for " + "determiners, prepositions, pronouns and auxiliaries. Such items would contribute " + "template-artefact vectors, not concept vectors.", + "This list is hand-curated and idiosyncratic — scikit-learn's own documentation warns " + "against using it uncritically — so this filter reintroduces exactly the kind of " + "unaccountable hand-authorship the pool is meant to remove. Two mitigations: it is a " + "published, citable, version-pinned list rather than one written here, and every word it " + "removes is enumerated below. Its cost is measurable: on the 2026-08 battery it would " + "have refused 'fire' and 'back', both contentful concepts.", + lambda t: t not in stop, + cap=400, + ) + + step( + "F5_not_proper_noun_dominant", + f"drop t where (initcap + other) / total >= {PROPER_CAP_SHARE}", + "Proper nouns are entities. A battery of entities measures name recall; this arc's claims " + "are about concept representation. Nothing in the 2026-08 chain prevented an entity from " + "entering the battery — it happens to contain none, but by authorship, not by rule.", + "Sentence-initial capitalisation inflates the ratio for every word, and the 0.5 bar is a " + "judgement call with no power analysis behind it. It removes polysemous words that are " + "also frequent names. Measured casualties in this run: 'brown' (a colour in the 2026-08 " + "battery), and the contentful nouns 'academy', 'abbey', 'admiral', 'act', 'march'. This " + "filter is the least principled in the chain and is flagged as such.", + lambda t: (initcap[t] + other[t]) / total[t] < PROPER_CAP_SHARE, + ) + + def single_token_everywhere(t): + return all(len(tk.encode(" " + t, add_special_tokens=False)) == 1 + for tk in tokenizers.values()) + + step( + "F6_single_token_in_every_target", + "keep t where tokenizer.encode(' ' + t) has length 1 for EVERY model in TARGET_MODELS", + "This is the filter the 2026-08 chain lacks entirely. Two reasons it belongs. (a) " + "`extract_multi` differences template.format(c=concept) against template.format(c=" + f"'{NEUTRAL_WORD}'), reads the LAST token of each sentence, and subtracts. The neutral " + "word is one token in all four models; a multi-token concept therefore makes the two " + "sentences different lengths, so the two hidden states being subtracted sit at different " + "absolute positions. (b) A string that is one token in gemma and three in Llama is not " + "the same stimulus in the two models, which is the comparison the cross-family reads " + "actually make.", + "Narrows hard toward high-frequency Anglo-Saxon vocabulary present in all four BPE " + "vocabularies, which is a further and unquantified restriction of what 'a concept' means " + "here. Measured on the 2026-08 battery: 35 of its 462 words are multi-token in " + "Llama-3.2 and in Qwen2.5 (2 of 462 in gemma-2), and 6 of its 70 held-out concepts are " + "multi-token in at least one target — so this filter would have changed the existing " + "battery, not merely tidied it. Note the read is sentence-final rather than on the " + "concept token itself, so the defect this filter removes is a length/position confound, " + "NOT a read taken on a word-piece; the smaller claim is the true one.", + single_token_everywhere, + ) + + surviving = set(survivors) + + def inflection_base(t): + for suf in _SUFFIXES: + if not t.endswith(suf): + continue + b = t[:-len(suf)] + if len(b) >= MIN_LEN and b in surviving: + return b + if suf == "ies": + b2 = t[:-3] + "y" + if len(b2) >= MIN_LEN and b2 in surviving: + return b2 + if suf in ("ing", "ed"): + b2 = t[:-len(suf)] + "e" + if len(b2) >= MIN_LEN and b2 in surviving: + return b2 + return None + + st = step( + "F7_no_inflection_of_a_survivor", + f"drop t if stripping one of {list(_SUFFIXES)} (with -ies->-y and -e restoration) yields a " + "base of length >= 3 that is itself a survivor of F6", + "Identification is argmin over the battery. A battery holding both 'friend' and 'friends' " + "asks the reader to discriminate near-duplicates, which measures morphological " + "resolution rather than concept identification and is not the quantity any gate in this " + "arc is about.", + "Partial and asymmetric. It cannot see irregular inflection (men, children, ran all " + "survive), and it produces false positives where a suffix is accidental: 'early' is " + "removed here because 'ear' survives. Nominalised gerunds that are genuine concepts " + "(building, painting, meeting) are removed alongside true duplicates. Verified on the " + "2026-08 battery: it would have removed none of its 462 words, which is why -er/-est are " + "excluded from the suffix list.", + lambda t: inflection_base(t) is None, + ) + st["removed_examples"] = [f"{t}<-{inflection_base(t)}" + for t in st["removed_examples"]] + + step( + "F8_exact_duplicate", + "order-preserving dedup", + "This is the ONLY filter the 2026-08 chain has (run_g0clear.py:66-67), where it removed " + "three words. It is kept here so the comparison is explicit rather than implied.", + "Nothing: case-folding at pool construction already made types unique, so this removes " + "zero. A stage that removes zero is worth recording precisely because the 2026-08 chain " + "consisted of nothing else.", + _keep_first_occurrence(), + ) + + return survivors, stages + + +def _keep_first_occurrence(): + """The 2026-08 chain's one filter, verbatim in spirit: keep a word the first time only.""" + seen = set() + return lambda t: not (t in seen or seen.add(t)) + + +def select_battery(eligible, n=BATTERY_N, seed=POOL_SEED): + """SELECTION RULE: seeded uniform sample without replacement from the sorted eligible set. + + Sorted first so the sampling frame does not depend on dict or set iteration order, which is + not stable across Python processes. This is the step that must be mechanical: the moment a + human picks which survivors go in, the extraction term stops being computable again. + """ + frame = sorted(set(eligible)) + if len(frame) < n: + raise SystemExit(f"eligible set has {len(frame)} < {n}; loosen a filter or lower BATTERY_N") + idx = np.random.default_rng(seed).permutation(len(frame))[:n] + return sorted(frame[i] for i in idx), frame + + +def old_battery(): + """The committed 2026-08 battery, read out of run_g0clear.py without importing torch.""" + src = (HERE / "run_g0clear.py").read_text(encoding="utf-8") + raw = re.search(r'_BANK = """(.*?)"""', src, re.S).group(1).split() + seen = set() + return raw, [c for c in raw if not (c in seen or seen.add(c))] + + +def main(argv=None): + ap = argparse.ArgumentParser() + ap.add_argument("--smoke", action="store_true", + help="first corpus shard only; writes nothing") + ap.add_argument("--dump-stages", action="store_true", + help="also write concept_pool_stages.json with full per-stage survivor lists") + args = ap.parse_args(argv) + + from transformers import AutoTokenizer + import sklearn + import transformers as _tf + + print("loading tokenizers (no weights)...", flush=True) + tokenizers, tok_pins = {}, {} + for m in TARGET_MODELS: + tokenizers[m] = AutoTokenizer.from_pretrained(m, local_files_only=True) + tok_pins[m] = {"vocab_size": len(tokenizers[m]), + f"tokens_for_{NEUTRAL_WORD}": + len(tokenizers[m].encode(" " + NEUTRAL_WORD, + add_special_tokens=False))} + bad = {m: p for m, p in tok_pins.items() if p[f"tokens_for_{NEUTRAL_WORD}"] != 1} + if bad: + print(f"NOTE: neutral word {NEUTRAL_WORD!r} is not single-token in {sorted(bad)} — " + f"the F6 justification is weaker for those models and this is recorded, not fixed", + flush=True) + + print("counting the pinned corpus...", flush=True) + types, total, initcap, other, pool_meta = build_pool(smoke=args.smoke) + print(f"POOL = {pool_meta['pool_size']} types " + f"from {pool_meta['token_occurrences']} occurrences", flush=True) + + survivors, stages = apply_chain(types, total, initcap, other, tokenizers) + for s in stages: + print(f" {s['filter']:34s} {s['n_in']:>7d} -> {s['n_out']:>7d} " + f"(-{s['n_removed']})", flush=True) + + battery, frame = select_battery(survivors) + raw_old, old = old_battery() + + pool_n = pool_meta["pool_size"] + receipt = { + "artifact": "concept_pool.json", + "produced_by": "build_concept_pool.py", + "purpose": ("a candidate pool recorded BEFORE the selection, so the extraction term " + "final/pool is computable for every battery derived from it"), + "prereg": "PREREG_b52_pooled_battery_2026_09_01.md", + "addendum": "ADDENDUM_battery_and_holdout_limits_2026_09_01.md", + "smoke": args.smoke, + "seed": POOL_SEED, + "deterministic": ("every stage sorts before it samples; the only randomness is " + f"numpy.random.default_rng({POOL_SEED}).permutation over the sorted " + "eligible frame"), + "environment_pins": { + "python": sys.version.split()[0], + "numpy": np.__version__, + "transformers": _tf.__version__, + "scikit_learn": sklearn.__version__, + }, + "pool": pool_meta, + "target_models": tok_pins, + "neutral_contrast_word": NEUTRAL_WORD, + "filters": stages, + "eligible": { + "n": len(frame), + "sha256": _sha(frame), + "words": frame, + }, + "battery": { + "n": len(battery), + "rule": (f"uniform sample without replacement of {BATTERY_N} from the sorted eligible " + f"set at seed {POOL_SEED}; size matched to the 2026-08 battery so the " + f"reproduction gate compares like with like (same N, same 1/70 chance under " + f"the same 70/392 split)"), + "sha256": _sha(battery), + "words": battery, + }, + "survival": { + "pool": pool_n, + "eligible": len(frame), + "battery": len(battery), + "survival_ratio_battery_over_pool": round(len(battery) / pool_n, 8), + "eligible_ratio_eligible_over_pool": round(len(frame) / pool_n, 8), + "sampling_fraction_battery_over_eligible": round(len(battery) / len(frame), 6), + "means": ("survival_ratio is the extraction term E for this battery: the share of " + "recorded candidates that reached the scored set. For the 2026-08 battery " + "the same quantity is UNCHECKABLE, because no pool was recorded."), + }, + "comparison_to_2026_08_battery": { + "file": "run_g0clear.py lines 31-67", + "raw_tokens_in_literal": len(raw_old), + "after_its_only_filter": len(old), + "its_filters": ["order-preserving dedup (removed 3: chicken, orange, mushroom)"], + "its_pool": None, + "its_survival_ratio": "UNCHECKABLE — no pool artifact exists", + "old_words_reaching_this_eligible_set": len(set(old) & set(frame)), + "old_words_in_this_battery": len(set(old) & set(battery)), + "old_words_refused_by_F6_single_token": sorted( + c for c in old + if any(len(tk.encode(" " + c, add_special_tokens=False)) != 1 + for tk in tokenizers.values())), + }, + "known_limits_of_this_artifact": [ + "The pool is one English corpus of Wikipedia prose. Concept frequency in WikiText is " + "a proxy for frequency in the target models' pretraining, which is not observable.", + "Filter thresholds (frequency floor 500, length 3-14, cap share 0.5) are judgement " + "calls stated in advance rather than values derived from a power analysis. They are " + "frozen here so a later run cannot move them after seeing a read.", + "No part-of-speech filter is applied: no offline tagger is available in this " + "environment. The successor battery is therefore broader in word class than the " + "hand-authored one, which is exactly what the reproduction gate in the b52 prereg " + "exists to price.", + "F4 and F5 are the two stages that most resemble the defect being repaired — a " + "curated list and a hand-set threshold. They are logged item-by-item and by count so " + "the disagreement is at least computable.", + ], + } + + if args.smoke: + print(json.dumps({k: receipt[k] for k in ("survival", "comparison_to_2026_08_battery")}, + indent=2), flush=True) + print("\nSMOKE: nothing written.", flush=True) + return 0 + + out = HERE / "concept_pool.json" + # newline="\n" so the receipt is byte-identical off Windows too; the prereg's VOID condition + # is byte-reproducibility of this file, and platform line endings would break it silently. + out.write_text(json.dumps(receipt, indent=2, ensure_ascii=False) + "\n", + encoding="utf-8", newline="\n") + print(f"\nwrote {out.name}", flush=True) + + if args.dump_stages: + dump = {"note": "full per-stage survivor lists; fingerprints match concept_pool.json", + "pool_sha256": pool_meta["sha256_pool"], + "pool_words": types, + "stages": [{"filter": s["filter"], "sha256_survivors": s["sha256_survivors"]} + for s in stages]} + d = HERE / "concept_pool_stages.json" + d.write_text(json.dumps(dump, ensure_ascii=False) + "\n", + encoding="utf-8", newline="\n") + print(f"wrote {d.name}", flush=True) + + s = receipt["survival"] + print(f"\nPOOL {s['pool']} -> ELIGIBLE {s['eligible']} -> BATTERY {s['battery']}", flush=True) + print(f"SURVIVAL RATIO (battery/pool) = {s['survival_ratio_battery_over_pool']}", flush=True) + print(f"eligible/pool = {s['eligible_ratio_eligible_over_pool']} " + f"battery/eligible = {s['sampling_fraction_battery_over_eligible']}", flush=True) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/papers/disjoint-worlds/concept_pool.json b/papers/disjoint-worlds/concept_pool.json new file mode 100644 index 00000000..4c5bc385 --- /dev/null +++ b/papers/disjoint-worlds/concept_pool.json @@ -0,0 +1,6315 @@ +{ + "artifact": "concept_pool.json", + "produced_by": "build_concept_pool.py", + "purpose": "a candidate pool recorded BEFORE the selection, so the extraction term final/pool is computable for every battery derived from it", + "prereg": "PREREG_b52_pooled_battery_2026_09_01.md", + "addendum": "ADDENDUM_battery_and_holdout_limits_2026_09_01.md", + "smoke": false, + "seed": 52, + "deterministic": "every stage sorts before it samples; the only randomness is numpy.random.default_rng(52).permutation over the sorted eligible frame", + "environment_pins": { + "python": "3.12.10", + "numpy": "2.4.4", + "transformers": "4.57.3", + "scikit_learn": "1.8.0" + }, + "pool": { + "rule": "distinct case-folded whitespace-delimited strings of the pinned corpus split; no filtering of any kind at this stage", + "source": { + "repo_id": "wikitext", + "repo_type": "dataset", + "config": "wikitext-103-raw-v1", + "split": "train", + "revision": "b08601e04326c79dfdd32d625aee71d232d685c3", + "licence": "CC BY-SA 3.0 (WikiText is derived from verified-good Wikipedia articles)", + "citation": "Merity, Xiong, Bradbury, Socher (2016), Pointer Sentinel Mixture Models, arXiv:1609.07843", + "shards": [ + "train-00000-of-00002.parquet", + "train-00001-of-00002.parquet" + ] + }, + "lines_read": 1165029, + "token_occurrences": 101425671, + "pool_size": 547373, + "sha256_pool": "a7f7961aff5a638e07aee23e74edd1250619455f42274d34eed05c1c1ff9f2de", + "not_priced_here": "strings absent from this corpus, and every multi-word concept: both are excluded by the pool's own boundary and no filter below charges for them" + }, + "target_models": { + "meta-llama/Llama-3.2-3B-Instruct": { + "vocab_size": 128256, + "tokens_for_object": 1 + }, + "meta-llama/Llama-3.2-1B-Instruct": { + "vocab_size": 128256, + "tokens_for_object": 1 + }, + "google/gemma-2-2b-it": { + "vocab_size": 256000, + "tokens_for_object": 1 + }, + "Qwen/Qwen2.5-1.5B-Instruct": { + "vocab_size": 151665, + "tokens_for_object": 1 + } + }, + "neutral_contrast_word": "object", + "filters": [ + { + "filter": "F1_ascii_alphabetic", + "rule": "keep t matching ^[a-z]+$ after case-folding", + "why": "The battery renders a bare English word into an English sentence template. Strings carrying digits, punctuation, or non-ASCII letters are not bare English words, and the corpus's whitespace inventory is full of them (numerals, the @-@ hyphen marker, transliterations).", + "cost": "Refuses accented loanwords (cafe with an acute) and every hyphenated compound, which WikiText writes with @-@ separators. Those are real concepts and this filter will not admit them.", + "n_in": 547373, + "n_removed": 93169, + "n_out": 454204, + "removed_fraction_of_input": 0.170211, + "removed_examples": [ + "!", + "\"", + "#", + "$", + "%", + "&", + "'", + "'-pentahydroxy", + "'03", + "'25", + "'458", + "'90", + "'a", + "'a'nga", + "'a's", + "'a'ā", + "'a.k.", + "'aa", + "'aah", + "'aaj", + "'aak", + "'aaknal", + "'aal", + "'aala", + "'aar" + ], + "removed_enumerated": false, + "sha256_survivors": "406361e3927d231c24feda09cf02b9a84deda6470335cac488bba8098e68004e" + }, + { + "filter": "F2_length_3_to_14", + "rule": "keep 3 <= len(t) <= 14", + "why": "One- and two-character strings in this corpus are dominated by initials, roman numerals and unit abbreviations rather than concepts; strings longer than 14 characters are dominated by concatenation artefacts and chemical names.", + "cost": "Loses genuine short concepts (ox, ax, pi) and long ones (thunderstorm is fine at 12, but e.g. responsibility at 14 is at the boundary and anything longer is refused outright).", + "n_in": 454204, + "n_removed": 9220, + "n_out": 444984, + "removed_fraction_of_input": 0.020299, + "removed_examples": [ + "a", + "aa", + "aaaaarrrrrrggghhh", + "aappilattorsuaq", + "aardrijkskundig", + "aargauerstalden", + "ab", + "abcdefghijklmnopqrstuvwxyz", + "abgeordnetenhaus", + "abhilashitartha", + "abhinavaabhidaana", + "aboriginalising", + "aboveaveragenetwork", + "absentmindedness", + "absolutelegends", + "abstentionalism", + "abwehrflammenwerfer", + "ac", + "acaciaepycnanthae", + "acanthaclisinae", + "acanthobothrium", + "acanthocephalan", + "acanthocephalans", + "acanthocephales", + "acanthopholidae" + ], + "removed_enumerated": false, + "sha256_survivors": "1621b9d0f556bc672a9aeff687cba5a44ed76d61dab944b571d36df91f9fc757" + }, + { + "filter": "F3_frequency_floor", + "rule": "keep total corpus occurrences (all case forms) >= 500", + "why": "The concept vector is a mean over 12 template renderings of a sentence-final hidden state. For a type the model has seen rarely, that mean is dominated by tokenizer segmentation and noise rather than by a stable representation, and a battery of such items measures the extraction pipeline instead of the model.", + "cost": "This is by far the largest narrowing in the chain and it BIASES the battery toward common English. A read scored on this battery licenses no claim about rare or technical concepts. Separately, WikiText frequency is a PROXY for the target models' pretraining frequency, which is not observable; the substitution is a known and unmeasured approximation.", + "n_in": 444984, + "n_removed": 433625, + "n_out": 11359, + "removed_fraction_of_input": 0.974473, + "removed_examples": [ + "aaaa", + "aaaaa", + "aaaaaa", + "aaaaaaargh", + "aaaaaahhhhhh", + "aaaaab", + "aaaah", + "aaaahh", + "aaaannnd", + "aaab", + "aaabc", + "aaagh", + "aaah", + "aaai", + "aaalac", + "aaarrrrr", + "aaas", + "aaat", + "aaata", + "aaathe", + "aaaw", + "aab", + "aaba", + "aabar", + "aabb" + ], + "removed_enumerated": false, + "sha256_survivors": "31ed794674fbdec2f2b5738955ec1f8734ef1970f339a0d2f0cf8aa773733e63" + }, + { + "filter": "F4_not_function_word", + "rule": "drop t in sklearn.feature_extraction.text.ENGLISH_STOP_WORDS (318 entries, Glasgow IR stop list, shipped under scikit-learn's BSD-3 licence)", + "why": "The template 'The {c} was the first thing everyone noticed.' is meaningless for determiners, prepositions, pronouns and auxiliaries. Such items would contribute template-artefact vectors, not concept vectors.", + "cost": "This list is hand-curated and idiosyncratic — scikit-learn's own documentation warns against using it uncritically — so this filter reintroduces exactly the kind of unaccountable hand-authorship the pool is meant to remove. Two mitigations: it is a published, citable, version-pinned list rather than one written here, and every word it removes is enumerated below. Its cost is measurable: on the 2026-08 battery it would have refused 'fire' and 'back', both contentful concepts.", + "n_in": 11359, + "n_removed": 258, + "n_out": 11101, + "removed_fraction_of_input": 0.022713, + "removed_examples": [ + "about", + "above", + "across", + "after", + "afterwards", + "again", + "against", + "all", + "almost", + "alone", + "along", + "already", + "also", + "although", + "always", + "among", + "amongst", + "amount", + "and", + "another", + "any", + "anyone", + "anything", + "anyway", + "anywhere", + "are", + "around", + "back", + "became", + "because", + "become", + "becomes", + "becoming", + "been", + "before", + "behind", + "being", + "below", + "beside", + "besides", + "between", + "beyond", + "bill", + "both", + "bottom", + "but", + "call", + "can", + "cannot", + "con", + "could", + "cry", + "describe", + "detail", + "done", + "down", + "due", + "during", + "each", + "eight", + "either", + "eleven", + "else", + "elsewhere", + "empty", + "enough", + "etc", + "even", + "ever", + "every", + "everyone", + "everything", + "everywhere", + "except", + "few", + "fifteen", + "fifty", + "fill", + "find", + "fire", + "first", + "five", + "for", + "former", + "formerly", + "forty", + "found", + "four", + "from", + "front", + "full", + "further", + "get", + "give", + "had", + "has", + "have", + "hence", + "her", + "here", + "herself", + "him", + "himself", + "his", + "how", + "however", + "hundred", + "inc", + "indeed", + "interest", + "into", + "its", + "itself", + "keep", + "last", + "latter", + "least", + "less", + "ltd", + "made", + "many", + "may", + "meanwhile", + "might", + "mill", + "mine", + "more", + "moreover", + "most", + "mostly", + "move", + "much", + "must", + "myself", + "name", + "namely", + "neither", + "never", + "nevertheless", + "next", + "nine", + "nobody", + "none", + "nor", + "not", + "nothing", + "now", + "nowhere", + "off", + "often", + "once", + "one", + "only", + "onto", + "other", + "others", + "otherwise", + "our", + "ourselves", + "out", + "over", + "own", + "part", + "per", + "perhaps", + "please", + "put", + "rather", + "same", + "see", + "seem", + "seemed", + "seems", + "serious", + "several", + "she", + "should", + "show", + "side", + "since", + "six", + "sixty", + "some", + "somehow", + "someone", + "something", + "sometime", + "sometimes", + "somewhere", + "still", + "such", + "system", + "take", + "ten", + "than", + "that", + "the", + "their", + "them", + "themselves", + "then", + "there", + "thereafter", + "thereby", + "therefore", + "these", + "they", + "thick", + "thin", + "third", + "this", + "those", + "though", + "three", + "through", + "throughout", + "thus", + "together", + "too", + "top", + "toward", + "towards", + "twelve", + "twenty", + "two", + "under", + "until", + "upon", + "very", + "via", + "was", + "well", + "were", + "what", + "whatever", + "when", + "whenever", + "where", + "whereas", + "whereby", + "wherein", + "whether", + "which", + "while", + "who", + "whole", + "whom", + "whose", + "why", + "will", + "with", + "within", + "without", + "would", + "yet", + "you", + "your", + "yourself" + ], + "removed_enumerated": true, + "sha256_survivors": "fd950c2af5a4ee9958c7b76292587ac93f735cec18c982471fd6aecdcb4d87ac" + }, + { + "filter": "F5_not_proper_noun_dominant", + "rule": "drop t where (initcap + other) / total >= 0.5", + "why": "Proper nouns are entities. A battery of entities measures name recall; this arc's claims are about concept representation. Nothing in the 2026-08 chain prevented an entity from entering the battery — it happens to contain none, but by authorship, not by rule.", + "cost": "Sentence-initial capitalisation inflates the ratio for every word, and the 0.5 bar is a judgement call with no power analysis behind it. It removes polysemous words that are also frequent names. Measured casualties in this run: 'brown' (a colour in the 2026-08 battery), and the contentful nouns 'academy', 'abbey', 'admiral', 'act', 'march'. This filter is the least principled in the chain and is flagged as such.", + "n_in": 11101, + "n_removed": 2783, + "n_out": 8318, + "removed_fraction_of_input": 0.250698, + "removed_examples": [ + "aaa", + "aaron", + "abbey", + "abc", + "abdul", + "aberdeen", + "aboriginal", + "abraham", + "abrams", + "abu", + "abyss", + "academy", + "acc", + "accolades", + "according", + "ace", + "aclu", + "act", + "activision", + "adam", + "adams", + "additionally", + "adelaide", + "admiral", + "admiralty" + ], + "removed_enumerated": false, + "sha256_survivors": "2340ad564df9c93415d49e1fc90a0d7d62f0c33731418d488d5e4011d984c325" + }, + { + "filter": "F6_single_token_in_every_target", + "rule": "keep t where tokenizer.encode(' ' + t) has length 1 for EVERY model in TARGET_MODELS", + "why": "This is the filter the 2026-08 chain lacks entirely. Two reasons it belongs. (a) `extract_multi` differences template.format(c=concept) against template.format(c='object'), reads the LAST token of each sentence, and subtracts. The neutral word is one token in all four models; a multi-token concept therefore makes the two sentences different lengths, so the two hidden states being subtracted sit at different absolute positions. (b) A string that is one token in gemma and three in Llama is not the same stimulus in the two models, which is the comparison the cross-family reads actually make.", + "cost": "Narrows hard toward high-frequency Anglo-Saxon vocabulary present in all four BPE vocabularies, which is a further and unquantified restriction of what 'a concept' means here. Measured on the 2026-08 battery: 35 of its 462 words are multi-token in Llama-3.2 and in Qwen2.5 (2 of 462 in gemma-2), and 6 of its 70 held-out concepts are multi-token in at least one target — so this filter would have changed the existing battery, not merely tidied it. Note the read is sentence-final rather than on the concept token itself, so the defect this filter removes is a length/position confound, NOT a read taken on a word-piece; the smaller claim is the true one.", + "n_in": 8318, + "n_removed": 499, + "n_out": 7819, + "removed_fraction_of_input": 0.05999, + "removed_examples": [ + "abbot", + "advisories", + "aground", + "airfield", + "airfields", + "airmen", + "airplay", + "altitudes", + "ambushed", + "amidships", + "amphibious", + "animators", + "annexation", + "annexed", + "antiquity", + "archaeologist", + "archaeologists", + "archaeology", + "arches", + "archipelago", + "arid", + "aristocracy", + "armament", + "armoured", + "artefacts" + ], + "removed_enumerated": false, + "sha256_survivors": "98f52e37e51db36e8f663375a494fb4dfd6383939ca8fa926a0f9d56dff68505" + }, + { + "filter": "F7_no_inflection_of_a_survivor", + "rule": "drop t if stripping one of ['ies', 'es', 's', 'ing', 'ed', 'ly'] (with -ies->-y and -e restoration) yields a base of length >= 3 that is itself a survivor of F6", + "why": "Identification is argmin over the battery. A battery holding both 'friend' and 'friends' asks the reader to discriminate near-duplicates, which measures morphological resolution rather than concept identification and is not the quantity any gate in this arc is about.", + "cost": "Partial and asymmetric. It cannot see irregular inflection (men, children, ran all survive), and it produces false positives where a suffix is accidental: 'early' is removed here because 'ear' survives. Nominalised gerunds that are genuine concepts (building, painting, meeting) are removed alongside true duplicates. Verified on the 2026-08 battery: it would have removed none of its 462 words, which is why -er/-est are excluded from the suffix list.", + "n_in": 7819, + "n_removed": 2620, + "n_out": 5199, + "removed_fraction_of_input": 0.335081, + "removed_examples": [ + "abandoned<-abandon", + "abandoning<-abandon", + "abilities<-ability", + "absolutely<-absolute", + "absorbed<-absorb", + "academics<-academic", + "accepted<-accept", + "accepting<-accept", + "accepts<-accept", + "accessed<-access", + "accidentally<-accidental", + "accidents<-accident", + "acclaimed<-acclaim", + "accompanying<-accompany", + "accomplished<-accomplish", + "accounted<-account", + "accounting<-account", + "accounts<-account", + "accurately<-accurate", + "achieved<-achieve", + "achievements<-achievement", + "achieving<-achieve", + "acids<-acid", + "acknowledged<-acknowledge", + "acquired<-acquire" + ], + "removed_enumerated": false, + "sha256_survivors": "32ed3165e5774b72af45ddbcc1cc96d19370f49814a1ea846debab9d9c7b50de" + }, + { + "filter": "F8_exact_duplicate", + "rule": "order-preserving dedup", + "why": "This is the ONLY filter the 2026-08 chain has (run_g0clear.py:66-67), where it removed three words. It is kept here so the comparison is explicit rather than implied.", + "cost": "Nothing: case-folding at pool construction already made types unique, so this removes zero. A stage that removes zero is worth recording precisely because the 2026-08 chain consisted of nothing else.", + "n_in": 5199, + "n_removed": 0, + "n_out": 5199, + "removed_fraction_of_input": 0.0, + "removed_examples": [], + "removed_enumerated": true, + "sha256_survivors": "32ed3165e5774b72af45ddbcc1cc96d19370f49814a1ea846debab9d9c7b50de" + } + ], + "eligible": { + "n": 5199, + "sha256": "32ed3165e5774b72af45ddbcc1cc96d19370f49814a1ea846debab9d9c7b50de", + "words": [ + "abandon", + "abdomen", + "ability", + "able", + "aboard", + "abolished", + "abolition", + "abortion", + "abroad", + "abruptly", + "absence", + "absent", + "absolute", + "absorb", + "absorption", + "abstract", + "abundance", + "abundant", + "abuse", + "academic", + "accelerated", + "accent", + "accept", + "acceptable", + "acceptance", + "access", + "accessible", + "accession", + "accident", + "accidental", + "acclaim", + "accommodate", + "accommodation", + "accompanied", + "accompany", + "accomplish", + "accordance", + "accordingly", + "account", + "accumulated", + "accuracy", + "accurate", + "accusations", + "accused", + "accusing", + "achieve", + "achievement", + "acid", + "acknowledge", + "acoustic", + "acquire", + "acquisition", + "acquitted", + "acre", + "acted", + "acting", + "action", + "activated", + "active", + "activism", + "activist", + "activity", + "actor", + "actress", + "acts", + "actual", + "acute", + "adapt", + "adaptation", + "add", + "addiction", + "addition", + "additional", + "address", + "adequate", + "adjacent", + "adjoining", + "adjusted", + "administered", + "administration", + "administrative", + "administrator", + "admiration", + "admired", + "admission", + "admit", + "admitted", + "adopt", + "adoption", + "ads", + "adult", + "advance", + "advancement", + "advantage", + "advent", + "adventure", + "adverse", + "advertised", + "advertisement", + "advertising", + "advice", + "advised", + "adviser", + "advisor", + "advisory", + "advocacy", + "advocate", + "aerial", + "aesthetic", + "affair", + "affect", + "affection", + "affiliate", + "affinity", + "affirmed", + "afford", + "aforementioned", + "afraid", + "aft", + "aftermath", + "afternoon", + "afterward", + "age", + "agency", + "agenda", + "agent", + "aggregate", + "aggression", + "aggressive", + "ago", + "agree", + "agreement", + "agricultural", + "agriculture", + "ahead", + "aid", + "aide", + "aim", + "air", + "airborne", + "aircraft", + "airline", + "airplane", + "airports", + "akin", + "alarm", + "albeit", + "album", + "alcohol", + "alcoholic", + "alert", + "alien", + "aligned", + "alignment", + "alike", + "alive", + "allegations", + "alleged", + "allegiance", + "alliance", + "allocated", + "allow", + "alloy", + "ally", + "alongside", + "alphabet", + "altar", + "alter", + "alterations", + "alternate", + "alternative", + "altitude", + "alto", + "altogether", + "aluminium", + "aluminum", + "alumni", + "amassed", + "amateur", + "amazing", + "ambassador", + "ambient", + "ambiguous", + "ambition", + "ambitious", + "ambulance", + "ambush", + "amended", + "amendments", + "amid", + "amidst", + "amino", + "ammunition", + "amnesty", + "amounted", + "amounts", + "amusement", + "analogous", + "analyses", + "analysis", + "analysts", + "analyzed", + "anarchist", + "ancestor", + "ancestral", + "ancestry", + "anchor", + "ancient", + "anger", + "angle", + "angry", + "angular", + "animal", + "animated", + "animation", + "anime", + "ankle", + "anniversary", + "announce", + "announcement", + "annual", + "anonymous", + "answer", + "ant", + "antagonist", + "anthem", + "anthology", + "anti", + "antibodies", + "anticipated", + "anticipation", + "anxiety", + "anxious", + "anybody", + "anymore", + "apart", + "apartment", + "apology", + "app", + "apparatus", + "apparent", + "appeal", + "appear", + "appearance", + "applauded", + "application", + "applied", + "appoint", + "appointment", + "appreciate", + "appreciation", + "apprentice", + "approach", + "appropriate", + "approval", + "approve", + "approximately", + "aquatic", + "arbitration", + "arc", + "arcade", + "arch", + "archaeological", + "architect", + "architectural", + "architecture", + "archive", + "area", + "aren", + "arena", + "arguably", + "argue", + "argument", + "arise", + "arm", + "armor", + "armour", + "arose", + "arrange", + "arrangement", + "array", + "arrest", + "arrival", + "arrive", + "arrows", + "art", + "article", + "artifacts", + "artificial", + "artillery", + "artist", + "artistic", + "artwork", + "ascent", + "ash", + "aside", + "ask", + "asleep", + "aspect", + "assassin", + "assassination", + "assault", + "assembled", + "assert", + "assess", + "assessment", + "asset", + "assigned", + "assignment", + "assist", + "assistance", + "assistant", + "associate", + "associations", + "assume", + "assumption", + "assured", + "asteroid", + "astronomers", + "astronomical", + "astronomy", + "asylum", + "athlete", + "atmosphere", + "atmospheric", + "atom", + "atomic", + "atop", + "attached", + "attachment", + "attack", + "attackers", + "attain", + "attempt", + "attend", + "attendance", + "attention", + "attitude", + "attract", + "attraction", + "attractive", + "attributed", + "attributes", + "auction", + "audience", + "audio", + "audition", + "augmented", + "aunt", + "authentic", + "authenticity", + "author", + "authorised", + "authority", + "authorized", + "autism", + "autobiography", + "automatic", + "automatically", + "automobile", + "autonomous", + "autonomy", + "autumn", + "auxiliary", + "availability", + "available", + "avant", + "average", + "avoid", + "awaiting", + "awarded", + "aware", + "awareness", + "away", + "awesome", + "awkward", + "axe", + "baby", + "bachelor", + "backdrop", + "backed", + "backgrounds", + "backing", + "backlash", + "backs", + "backstage", + "backstory", + "backup", + "backwards", + "bacteria", + "bacterial", + "bad", + "badge", + "bag", + "bail", + "balance", + "bald", + "ball", + "ballet", + "balloon", + "ballot", + "ban", + "band", + "bank", + "bankruptcy", + "banned", + "banner", + "baptism", + "bar", + "bare", + "bark", + "barn", + "barracks", + "barrage", + "barred", + "barrel", + "barrier", + "basal", + "base", + "baseball", + "baseman", + "basement", + "basic", + "basically", + "basin", + "basis", + "basketball", + "bass", + "bat", + "batch", + "bathroom", + "battery", + "batting", + "battle", + "battlefield", + "beaches", + "beam", + "bear", + "beat", + "beaten", + "beautiful", + "beauty", + "bed", + "bedroom", + "bee", + "beef", + "beer", + "began", + "begin", + "beginning", + "begun", + "behalf", + "behavior", + "behaviour", + "beings", + "belief", + "believe", + "bells", + "belly", + "belong", + "beloved", + "belt", + "bench", + "beneath", + "beneficial", + "benefit", + "bent", + "best", + "beta", + "betrayal", + "betrayed", + "better", + "bias", + "biblical", + "bicycle", + "bid", + "bigger", + "biggest", + "bike", + "bikini", + "billed", + "billion", + "bills", + "bin", + "binary", + "binding", + "biography", + "biological", + "biology", + "bird", + "birth", + "birthday", + "bishops", + "bit", + "bite", + "bitter", + "bizarre", + "black", + "blade", + "blame", + "bland", + "blank", + "blast", + "bleeding", + "blend", + "blew", + "blind", + "block", + "blockade", + "blog", + "blonde", + "blood", + "bloody", + "blow", + "blown", + "blue", + "blunt", + "board", + "boat", + "body", + "boiler", + "bold", + "bomb", + "bomber", + "bonds", + "bone", + "bonus", + "book", + "boom", + "boost", + "boot", + "border", + "bore", + "born", + "borne", + "borough", + "borrowed", + "boss", + "botanical", + "bottle", + "bought", + "bound", + "boundary", + "bounty", + "bout", + "bow", + "bowling", + "box", + "boxer", + "boy", + "boycott", + "boyfriend", + "brain", + "brake", + "branch", + "brand", + "brass", + "brave", + "brawl", + "breach", + "bread", + "break", + "breakdown", + "breakfast", + "breakout", + "breakthrough", + "breakup", + "breast", + "breath", + "breathe", + "bred", + "breed", + "brewery", + "brick", + "bride", + "bridge", + "brief", + "bright", + "brilliant", + "bring", + "broad", + "broadcast", + "broadcaster", + "broader", + "broke", + "broken", + "bronze", + "brother", + "brought", + "brutal", + "bubble", + "budget", + "build", + "built", + "bulk", + "bullet", + "bunch", + "bunker", + "burden", + "burial", + "buried", + "burn", + "burnt", + "burst", + "bus", + "busiest", + "business", + "businessman", + "bust", + "busy", + "butterfly", + "button", + "buy", + "bypass", + "cabin", + "cabinet", + "cable", + "cage", + "cake", + "calcium", + "calculated", + "calculations", + "calendar", + "caliber", + "called", + "calling", + "calls", + "calm", + "came", + "cameo", + "camera", + "camouflage", + "camp", + "campaign", + "campus", + "cancel", + "cancellation", + "cancelled", + "cancer", + "candidacy", + "candidate", + "cane", + "cannon", + "canon", + "canopy", + "canvas", + "cap", + "capability", + "capable", + "capacity", + "capital", + "capitalism", + "capped", + "captains", + "captive", + "captivity", + "capture", + "car", + "carbon", + "card", + "care", + "career", + "careful", + "cargo", + "carriage", + "carried", + "carrier", + "carry", + "cartoon", + "cartridge", + "carved", + "case", + "cash", + "cassette", + "cast", + "castle", + "casual", + "casualty", + "cat", + "catalog", + "catalogue", + "catch", + "catcher", + "catchy", + "category", + "cattle", + "caught", + "cause", + "caution", + "cavalry", + "cave", + "cavity", + "cease", + "ceiling", + "celebrate", + "celebration", + "celebrity", + "celestial", + "cell", + "cellular", + "cement", + "censorship", + "census", + "cent", + "center", + "central", + "centre", + "century", + "ceremonial", + "ceremony", + "certain", + "certificate", + "certification", + "certified", + "chain", + "chair", + "chairman", + "challenge", + "chamber", + "champion", + "chance", + "change", + "channels", + "chaos", + "chaotic", + "chapel", + "chapter", + "character", + "characteristic", + "characterized", + "charge", + "charitable", + "charity", + "charm", + "chart", + "charter", + "chase", + "cheap", + "cheaper", + "check", + "cheek", + "cheese", + "chef", + "chemical", + "chemistry", + "chess", + "chest", + "chicken", + "chicks", + "chief", + "child", + "childhood", + "children", + "chip", + "chloride", + "chocolate", + "choice", + "choir", + "choose", + "chord", + "chorus", + "chose", + "chosen", + "chronic", + "church", + "cinema", + "cinematic", + "circle", + "circuit", + "circular", + "circulated", + "circulation", + "circumstances", + "citation", + "cited", + "cites", + "citing", + "citizen", + "citizenship", + "city", + "civic", + "civilian", + "civilization", + "clad", + "claim", + "clan", + "clarified", + "clash", + "class", + "classic", + "classical", + "classification", + "classified", + "clause", + "claws", + "clay", + "clean", + "clear", + "clearance", + "clergy", + "clerk", + "clever", + "client", + "cliff", + "climate", + "climax", + "climb", + "clinic", + "clinical", + "clip", + "clock", + "clone", + "close", + "closer", + "closest", + "closure", + "cloth", + "cloud", + "club", + "clues", + "cluster", + "coach", + "coal", + "coalition", + "coast", + "coastal", + "coaster", + "coastline", + "coat", + "cockpit", + "code", + "coffee", + "coffin", + "cognitive", + "coin", + "coincide", + "cold", + "collaborated", + "collaboration", + "collapse", + "collar", + "colleague", + "collect", + "collection", + "collective", + "collector", + "colleges", + "collegiate", + "collided", + "collision", + "colonial", + "colony", + "color", + "colour", + "column", + "columnist", + "combat", + "combination", + "combine", + "come", + "comeback", + "comedian", + "comedic", + "comedy", + "comfort", + "comfortable", + "comic", + "command", + "commander", + "commemorate", + "commenced", + "comment", + "commentary", + "commentator", + "commerce", + "commercial", + "commissioned", + "commissioners", + "commissions", + "commit", + "commitment", + "committed", + "committees", + "committing", + "common", + "communicate", + "communication", + "communism", + "community", + "commuter", + "compact", + "companion", + "company", + "comparable", + "comparatively", + "compare", + "comparison", + "compartments", + "compatible", + "compelled", + "compelling", + "compensate", + "compensation", + "compete", + "competition", + "competitive", + "competitor", + "compilation", + "compiled", + "complained", + "complaint", + "complement", + "complete", + "completion", + "complex", + "complexity", + "complicated", + "complications", + "comply", + "component", + "compose", + "composer", + "composite", + "composition", + "compound", + "comprehensive", + "comprise", + "compromise", + "compulsory", + "computer", + "computing", + "concealed", + "conceded", + "conceived", + "concentrate", + "concentration", + "concept", + "conception", + "concern", + "concert", + "concession", + "conclude", + "conclusion", + "concrete", + "concurrency", + "concurrent", + "condemned", + "condition", + "conduct", + "conductor", + "cone", + "conferences", + "confessed", + "confession", + "confidence", + "confident", + "configuration", + "confined", + "confirm", + "confirmation", + "confiscated", + "conflict", + "confront", + "confrontation", + "confused", + "confusing", + "confusion", + "congestion", + "congregation", + "congressional", + "conjunction", + "connect", + "connection", + "connector", + "conquer", + "conquest", + "conscience", + "conscious", + "consciousness", + "consecutive", + "consensus", + "consent", + "consequence", + "conservation", + "conservative", + "consider", + "considerable", + "considerably", + "consideration", + "consist", + "consistent", + "console", + "consolidated", + "consort", + "conspicuous", + "conspiracy", + "constant", + "constellation", + "constituency", + "constitute", + "constitutional", + "constraints", + "construct", + "construction", + "consul", + "consultant", + "consulted", + "consulting", + "consume", + "consumer", + "consumption", + "contact", + "contain", + "contaminated", + "contemporary", + "contempt", + "contender", + "content", + "contention", + "contest", + "contestant", + "context", + "continent", + "contingent", + "continually", + "continuation", + "continue", + "continuity", + "continuous", + "contract", + "contractor", + "contrary", + "contrast", + "contribute", + "contribution", + "contributor", + "control", + "controlled", + "controller", + "controlling", + "controversial", + "controversy", + "convened", + "convention", + "conventional", + "conversation", + "conversion", + "convert", + "convex", + "convey", + "convicted", + "conviction", + "convince", + "convoy", + "cooking", + "cool", + "cooler", + "cooperation", + "cooperative", + "coordinate", + "coordination", + "coordinator", + "cope", + "copied", + "copper", + "copy", + "copyright", + "coral", + "cord", + "core", + "corn", + "corner", + "corporate", + "corporations", + "corpse", + "correct", + "correspond", + "correspondence", + "correspondent", + "corridor", + "corrupt", + "corruption", + "cosmic", + "cost", + "costume", + "cottage", + "cotton", + "couldn", + "councils", + "counsel", + "count", + "countdown", + "counter", + "counterpart", + "country", + "countryside", + "coup", + "couple", + "courage", + "course", + "court", + "courthouse", + "courtyard", + "cousin", + "cover", + "coverage", + "cow", + "crab", + "crack", + "craft", + "crash", + "crater", + "cream", + "create", + "creation", + "creative", + "creator", + "creature", + "credit", + "crest", + "crew", + "cricket", + "crime", + "criminal", + "crisis", + "criteria", + "critic", + "critical", + "criticised", + "criticism", + "criticized", + "criticizing", + "critique", + "crop", + "cross", + "crossover", + "crowd", + "crowned", + "crucial", + "crude", + "cruel", + "cruise", + "cruiser", + "crush", + "crust", + "crying", + "crystals", + "cubic", + "cuisine", + "cult", + "cultivated", + "cultivation", + "cultural", + "culture", + "cure", + "curious", + "currency", + "current", + "curriculum", + "curse", + "curtain", + "curve", + "custody", + "custom", + "customary", + "customer", + "cut", + "cutter", + "cutting", + "cycle", + "cylinder", + "cylindrical", + "dam", + "damage", + "dance", + "dancer", + "danger", + "dangerous", + "dark", + "darker", + "darkness", + "dash", + "data", + "database", + "date", + "daughter", + "dawn", + "day", + "daylight", + "daytime", + "dead", + "deadline", + "deal", + "dealer", + "dealt", + "death", + "debate", + "debris", + "debt", + "debut", + "decade", + "decay", + "deceased", + "decent", + "deception", + "decide", + "decision", + "decisive", + "deck", + "declaration", + "declare", + "decline", + "decorated", + "decoration", + "decorative", + "decrease", + "decree", + "dedicated", + "dedication", + "deemed", + "deep", + "deeper", + "deer", + "defeat", + "defence", + "defend", + "defendant", + "defender", + "defense", + "defensive", + "deficit", + "define", + "definitely", + "definition", + "definitive", + "degree", + "deity", + "del", + "delay", + "delegate", + "delegation", + "deleted", + "deliberate", + "delicate", + "delight", + "deliver", + "delivery", + "deluxe", + "demand", + "demise", + "demo", + "democracy", + "demographic", + "demolished", + "demolition", + "demon", + "demonstrate", + "demonstration", + "denial", + "denied", + "denounced", + "dense", + "density", + "dental", + "deny", + "depart", + "departments", + "departure", + "depend", + "dependent", + "depict", + "depiction", + "depleted", + "deploy", + "deployment", + "deposit", + "depot", + "depressed", + "depression", + "deprived", + "depth", + "deputy", + "der", + "derivative", + "derived", + "derives", + "des", + "descendant", + "descended", + "descending", + "descent", + "described", + "describes", + "describing", + "description", + "desert", + "deserved", + "design", + "designated", + "designation", + "designer", + "desire", + "desk", + "desperate", + "destination", + "destined", + "destroy", + "destroyer", + "destruction", + "destructive", + "detached", + "detachment", + "detailed", + "detailing", + "details", + "detained", + "detect", + "detection", + "detective", + "detention", + "determination", + "determine", + "devastated", + "devastating", + "develop", + "developer", + "development", + "developmental", + "device", + "devised", + "devoted", + "devotion", + "diagnosed", + "diagnosis", + "dialect", + "dialogue", + "diameter", + "diamond", + "diary", + "dictator", + "did", + "didn", + "die", + "diesel", + "diet", + "differ", + "difference", + "different", + "difficult", + "difficulty", + "diffusion", + "dig", + "digging", + "digital", + "dignity", + "dimension", + "dimensional", + "diminished", + "dining", + "dinner", + "dinosaur", + "dioxide", + "diplomat", + "diplomatic", + "direct", + "direction", + "director", + "dirt", + "dirty", + "disability", + "disabled", + "disagreed", + "disagreement", + "disappearance", + "disappeared", + "disappointed", + "disappointing", + "disappointment", + "disaster", + "disastrous", + "disc", + "discarded", + "discharge", + "discipline", + "disco", + "discontinued", + "discouraged", + "discover", + "discovery", + "discrimination", + "discuss", + "discussion", + "disease", + "disguise", + "dish", + "disk", + "dislike", + "dismiss", + "dismissal", + "disorder", + "dispatch", + "dispersed", + "displaced", + "displacement", + "display", + "disposal", + "dispute", + "disrupt", + "disruption", + "dissolution", + "dissolved", + "distance", + "distant", + "distinct", + "distinction", + "distinctive", + "distinguish", + "distorted", + "distracted", + "distress", + "distribute", + "distribution", + "distributor", + "district", + "disturbance", + "disturbed", + "disturbing", + "ditch", + "dive", + "divers", + "diverse", + "diversity", + "diverted", + "divide", + "divine", + "divisions", + "divorce", + "dock", + "doctors", + "doctrine", + "document", + "documentary", + "documentation", + "does", + "doesn", + "dog", + "doing", + "doll", + "dollar", + "dolphins", + "domain", + "dome", + "domestic", + "dominance", + "dominant", + "dominate", + "domination", + "donated", + "donation", + "door", + "dorsal", + "dose", + "double", + "doubt", + "dove", + "download", + "downloadable", + "downs", + "downstream", + "downtown", + "downward", + "dozen", + "draft", + "drag", + "dragged", + "drain", + "drainage", + "drama", + "dramatic", + "dramatically", + "draw", + "drawn", + "dreams", + "dress", + "drew", + "dried", + "drift", + "drink", + "drive", + "driven", + "driver", + "drop", + "dropped", + "dropping", + "drought", + "drove", + "drowned", + "drowning", + "drug", + "drum", + "drummer", + "drunk", + "dry", + "dual", + "dubbed", + "duck", + "duel", + "dug", + "dull", + "dungeon", + "duo", + "duration", + "dust", + "duty", + "dwarf", + "dwelling", + "dying", + "dynamic", + "dynasty", + "eager", + "eagle", + "ear", + "earlier", + "earliest", + "earn", + "earthquake", + "ease", + "easier", + "easily", + "east", + "eastern", + "easy", + "eat", + "eaten", + "eccentric", + "echoed", + "echoes", + "eclipse", + "ecological", + "ecology", + "economic", + "economically", + "economy", + "ecosystem", + "edge", + "edible", + "edit", + "edition", + "editor", + "editorial", + "educated", + "education", + "educational", + "effect", + "effective", + "effectiveness", + "efficiency", + "efficient", + "effort", + "egg", + "ego", + "eighteen", + "eighth", + "eighty", + "ejected", + "elaborate", + "elder", + "eldest", + "elect", + "election", + "electoral", + "electric", + "electrical", + "electricity", + "electro", + "electron", + "electronic", + "elegant", + "element", + "elementary", + "elephant", + "elevated", + "elevation", + "elevator", + "eligibility", + "eligible", + "eliminate", + "elimination", + "elite", + "elk", + "embarked", + "embassy", + "embedded", + "emblem", + "embrace", + "emerge", + "emergence", + "emergency", + "emission", + "emotion", + "emotional", + "emphasis", + "emphasize", + "employ", + "employee", + "employer", + "employment", + "enable", + "enacted", + "enclosed", + "enclosure", + "encounter", + "encourage", + "encouragement", + "end", + "endangered", + "endemic", + "endorsed", + "endorsement", + "endurance", + "endured", + "enduring", + "enemy", + "energetic", + "energy", + "enforce", + "enforcement", + "engage", + "engagement", + "engine", + "engineer", + "enhance", + "enjoy", + "enjoyable", + "enlarged", + "enlisted", + "enormous", + "enrolled", + "enrollment", + "ensemble", + "ensued", + "ensuing", + "ensure", + "enter", + "entertaining", + "enthusiasm", + "enthusiastic", + "entire", + "entirety", + "entitled", + "entity", + "entrance", + "entry", + "environment", + "environmental", + "envisioned", + "enzyme", + "epic", + "epidemic", + "episode", + "equal", + "equality", + "equation", + "equilibrium", + "equipment", + "equipped", + "equity", + "equivalent", + "era", + "erected", + "erosion", + "error", + "erupted", + "eruption", + "escape", + "escort", + "especially", + "essay", + "essence", + "essential", + "est", + "establish", + "establishment", + "estate", + "estimate", + "eternal", + "ethanol", + "ethical", + "ethics", + "ethnic", + "evacuate", + "evacuation", + "evaluated", + "evaluation", + "evening", + "event", + "eventual", + "everybody", + "everyday", + "evidence", + "evident", + "evil", + "evolution", + "evolutionary", + "evolve", + "exact", + "exaggerated", + "exam", + "examination", + "examine", + "example", + "excavation", + "exceed", + "excellence", + "excellent", + "exception", + "exceptional", + "excess", + "excessive", + "exchange", + "excited", + "excitement", + "exciting", + "excluded", + "excluding", + "exclusion", + "exclusive", + "execute", + "execution", + "executive", + "exercise", + "exhausted", + "exhibit", + "exhibition", + "exile", + "exist", + "existence", + "exit", + "exotic", + "expand", + "expansion", + "expect", + "expectations", + "expedition", + "expelled", + "expense", + "expensive", + "experience", + "experiment", + "experimental", + "expert", + "expired", + "explain", + "explanation", + "explicit", + "exploded", + "exploit", + "exploitation", + "exploration", + "explore", + "explorer", + "explosion", + "explosive", + "export", + "exposed", + "exposure", + "expressed", + "expresses", + "expressing", + "expression", + "extend", + "extension", + "extensive", + "extent", + "exterior", + "external", + "extinct", + "extinction", + "extra", + "extract", + "extraction", + "extraordinary", + "extreme", + "eye", + "fabric", + "facade", + "face", + "facial", + "facilitate", + "facility", + "fact", + "faction", + "facto", + "factor", + "factory", + "faculty", + "fail", + "failure", + "fair", + "fairy", + "faith", + "faithful", + "fake", + "fall", + "fallen", + "false", + "familiar", + "family", + "famous", + "fan", + "fantastic", + "far", + "fare", + "farewell", + "farm", + "farmer", + "farther", + "fascist", + "fashion", + "fashionable", + "fast", + "faster", + "fastest", + "fat", + "fatal", + "fatalities", + "fate", + "father", + "fault", + "fauna", + "favor", + "favorable", + "favorite", + "favour", + "favourable", + "favourite", + "fear", + "feast", + "feat", + "feathers", + "feature", + "fed", + "federal", + "fee", + "feedback", + "feel", + "feet", + "fell", + "fellow", + "felt", + "female", + "feminine", + "feminist", + "fence", + "ferry", + "fertile", + "festivals", + "feud", + "feudal", + "fever", + "fewer", + "fiction", + "fictional", + "field", + "fierce", + "fifth", + "fight", + "fighter", + "figure", + "file", + "filled", + "filling", + "film", + "filmmaker", + "filter", + "fin", + "final", + "finale", + "finalist", + "finance", + "financial", + "finding", + "finds", + "fine", + "finest", + "finger", + "finish", + "firearms", + "fired", + "fires", + "firing", + "firm", + "fiscal", + "fish", + "fisheries", + "fishermen", + "fit", + "fitness", + "fitted", + "fitting", + "fix", + "fixture", + "flag", + "flagship", + "flame", + "flank", + "flash", + "flashback", + "flat", + "flattened", + "flaws", + "fled", + "flee", + "fleet", + "flesh", + "flew", + "flexible", + "flight", + "floating", + "flood", + "floor", + "flow", + "flower", + "flown", + "flu", + "fluid", + "fluoride", + "fly", + "focus", + "fog", + "fold", + "folk", + "folklore", + "follow", + "followers", + "fond", + "food", + "fool", + "foot", + "footage", + "football", + "forbidden", + "force", + "fore", + "forecast", + "foreign", + "foreigners", + "foremost", + "forensic", + "forest", + "forestry", + "forget", + "forgotten", + "form", + "formal", + "format", + "formation", + "formidable", + "formula", + "forth", + "forthcoming", + "fortified", + "fortress", + "forts", + "fortune", + "forward", + "fossil", + "fought", + "foul", + "foundations", + "founded", + "founder", + "founding", + "fountain", + "fourteen", + "fourth", + "fraction", + "fragile", + "fragment", + "frame", + "framework", + "franchise", + "fraternity", + "fraud", + "free", + "freedom", + "freeway", + "freezing", + "freight", + "frequency", + "frequent", + "fresh", + "freshman", + "freshwater", + "friction", + "fried", + "friend", + "friendship", + "frog", + "frontal", + "frontier", + "fronts", + "frozen", + "fruit", + "frustrated", + "frustration", + "fuck", + "fuel", + "fulfill", + "fully", + "fun", + "function", + "functional", + "fund", + "fundamental", + "fundraising", + "funeral", + "fungi", + "fungus", + "funk", + "funnel", + "funny", + "fur", + "furious", + "furniture", + "fused", + "fusion", + "future", + "futuristic", + "gain", + "gal", + "galaxy", + "galleries", + "gallons", + "gambling", + "game", + "gameplay", + "gamers", + "gamma", + "gang", + "gap", + "garage", + "garbage", + "gardens", + "garnered", + "gas", + "gasoline", + "gates", + "gateway", + "gather", + "gauge", + "gave", + "gay", + "gear", + "gender", + "gene", + "genera", + "generally", + "generals", + "generate", + "generation", + "generator", + "generic", + "generous", + "genetic", + "genetically", + "genius", + "genome", + "genre", + "gentle", + "gentleman", + "genuine", + "genus", + "geographic", + "geographical", + "geological", + "geometry", + "gesture", + "gets", + "getting", + "ghosts", + "giant", + "gift", + "girl", + "girlfriend", + "given", + "gives", + "giving", + "glaciers", + "glad", + "glands", + "glass", + "global", + "goal", + "goalkeeper", + "goddess", + "gods", + "goes", + "going", + "gold", + "golf", + "gone", + "gonna", + "good", + "gorge", + "gospel", + "gossip", + "got", + "governance", + "governed", + "governing", + "government", + "governmental", + "governor", + "grade", + "gradient", + "gradual", + "graduate", + "graduation", + "grain", + "grammar", + "grandfather", + "grandmother", + "grandson", + "granite", + "granted", + "granting", + "grants", + "graph", + "graphic", + "graphical", + "grass", + "grave", + "gravel", + "gravitational", + "gravity", + "grazing", + "great", + "greater", + "greatest", + "greeted", + "grew", + "grey", + "grid", + "grief", + "grip", + "groove", + "gross", + "ground", + "group", + "grow", + "grown", + "growth", + "guarantee", + "guard", + "guess", + "guest", + "guidance", + "guided", + "guidelines", + "guides", + "guilt", + "guilty", + "guitar", + "guitarist", + "gun", + "gunfire", + "guys", + "habit", + "habitat", + "hail", + "hair", + "half", + "halftime", + "halfway", + "halls", + "halt", + "hand", + "handful", + "handheld", + "handle", + "hang", + "happen", + "happiness", + "happy", + "harbour", + "hard", + "hardcore", + "harder", + "hardest", + "hardware", + "harm", + "harmful", + "harmony", + "harsh", + "harvest", + "hat", + "hatch", + "hate", + "hatred", + "haul", + "haunted", + "having", + "hazard", + "head", + "headline", + "headquartered", + "headquarters", + "healing", + "health", + "healthcare", + "healthy", + "hear", + "heard", + "heart", + "heat", + "heavier", + "heavily", + "heavy", + "hectares", + "height", + "heir", + "held", + "helicopter", + "helium", + "helm", + "helmet", + "help", + "herd", + "hero", + "heroic", + "heroine", + "hiatus", + "hidden", + "hide", + "hierarchy", + "high", + "higher", + "highest", + "highlight", + "highway", + "hind", + "hip", + "hire", + "historian", + "historic", + "historical", + "history", + "hit", + "hitter", + "hitting", + "hockey", + "hold", + "holder", + "hole", + "holiday", + "hollow", + "homage", + "home", + "homeland", + "homeless", + "hometown", + "homosexual", + "homosexuality", + "honest", + "honor", + "honorable", + "honorary", + "honour", + "hook", + "hop", + "hope", + "horizon", + "horizontal", + "horn", + "horror", + "horse", + "horsepower", + "hospital", + "host", + "hostage", + "hostile", + "hostility", + "hotel", + "hour", + "house", + "household", + "hub", + "huge", + "hull", + "human", + "humanitarian", + "humanity", + "humor", + "humorous", + "humour", + "hundreds", + "hung", + "hunger", + "hungry", + "hunted", + "hunters", + "hunting", + "hurricane", + "hurt", + "husband", + "hybrid", + "hydraulic", + "hydrogen", + "hypothesis", + "ice", + "icon", + "iconic", + "idea", + "ideal", + "identical", + "identification", + "identified", + "identify", + "identity", + "ideological", + "ideology", + "ignored", + "ill", + "illegal", + "illness", + "illuminated", + "illusion", + "illustrate", + "illustration", + "image", + "imagery", + "imagination", + "imagine", + "immediate", + "immense", + "immigrant", + "immigration", + "imminent", + "immortal", + "immune", + "immunity", + "impact", + "impaired", + "impedance", + "impending", + "implement", + "implementation", + "implications", + "implied", + "implies", + "import", + "importance", + "important", + "impose", + "impossible", + "impressed", + "impression", + "impressive", + "imprisoned", + "imprisonment", + "improve", + "improvement", + "improvised", + "inability", + "inactive", + "inadequate", + "inappropriate", + "inaugural", + "inauguration", + "incarnation", + "inception", + "inch", + "incident", + "inclined", + "include", + "inclusion", + "income", + "incomplete", + "inconsistent", + "incorporate", + "incorporation", + "incorrect", + "increase", + "incredible", + "incredibly", + "incumbent", + "incurred", + "independence", + "independent", + "index", + "indicate", + "indication", + "indie", + "indigenous", + "indirect", + "individual", + "indoor", + "induced", + "industrial", + "industry", + "ineffective", + "inevitable", + "inexperienced", + "infamous", + "infant", + "infected", + "infection", + "infectious", + "inferior", + "infinite", + "inflation", + "inflicted", + "influence", + "influential", + "influenza", + "influx", + "inform", + "informal", + "information", + "infrared", + "infrastructure", + "ing", + "ingredients", + "inhabitants", + "inhabited", + "inherent", + "inheritance", + "inherited", + "initial", + "initiate", + "initiative", + "injured", + "injury", + "ink", + "inland", + "inmates", + "inner", + "inning", + "innocence", + "innocent", + "innovation", + "innovative", + "input", + "inquiry", + "insane", + "inscription", + "insect", + "inserted", + "inside", + "insight", + "insisted", + "insistence", + "insists", + "inspection", + "inspiration", + "inspired", + "inspiring", + "instability", + "install", + "installation", + "installment", + "instance", + "instant", + "instead", + "instituted", + "institution", + "instructed", + "instruction", + "instructor", + "instrument", + "instrumental", + "insufficient", + "insurance", + "insurgents", + "intact", + "intake", + "integral", + "integrated", + "integration", + "integrity", + "intellectual", + "intelligence", + "intelligent", + "intended", + "intending", + "intense", + "intensified", + "intensity", + "intensive", + "intent", + "intention", + "intentionally", + "inter", + "interact", + "interaction", + "intercept", + "interception", + "interchange", + "interested", + "interesting", + "interests", + "interface", + "interfere", + "interference", + "interim", + "interior", + "intermediate", + "internal", + "interpretation", + "interpreted", + "interrupted", + "intersection", + "intersects", + "interval", + "intervene", + "intervention", + "interview", + "intimate", + "introduce", + "introduction", + "invade", + "invaders", + "invasion", + "invented", + "invention", + "inventory", + "invested", + "investigate", + "investigation", + "investigators", + "investment", + "investors", + "invisible", + "invitation", + "invited", + "invites", + "involve", + "involvement", + "ion", + "iron", + "irregular", + "irrigation", + "island", + "isn", + "isolated", + "isolation", + "issue", + "item", + "jacket", + "jail", + "jaw", + "jazz", + "jealous", + "jet", + "job", + "join", + "joint", + "joke", + "journalism", + "journalist", + "journals", + "journey", + "joy", + "judge", + "judgement", + "judgment", + "judicial", + "judiciary", + "juice", + "jump", + "junction", + "jungle", + "junior", + "jurisdiction", + "jury", + "just", + "justices", + "justified", + "justify", + "juvenile", + "keen", + "keeper", + "keeping", + "keeps", + "kept", + "key", + "keyboard", + "kick", + "kickoff", + "kidnapped", + "kidney", + "kids", + "kill", + "killer", + "kilograms", + "kilometers", + "kilometres", + "kind", + "kingdoms", + "kings", + "kiss", + "kit", + "kitchen", + "knee", + "knew", + "knife", + "knock", + "knockout", + "knots", + "know", + "knowledge", + "known", + "lab", + "label", + "labelled", + "labor", + "laboratories", + "lack", + "ladder", + "laid", + "lakes", + "land", + "landmark", + "landscape", + "lane", + "language", + "lap", + "large", + "larger", + "largest", + "larvae", + "laser", + "lasted", + "lasting", + "lasts", + "late", + "later", + "lateral", + "latest", + "latitude", + "laugh", + "launch", + "lava", + "law", + "lawn", + "lawsuit", + "lawyer", + "lay", + "layer", + "layout", + "lead", + "leader", + "leadership", + "leaf", + "leagues", + "leak", + "learn", + "lease", + "leather", + "leave", + "lecture", + "led", + "left", + "leg", + "legal", + "legend", + "legendary", + "legislation", + "legislative", + "legislature", + "legitimate", + "leisure", + "length", + "lengthy", + "lens", + "lent", + "leopard", + "lesbian", + "lesser", + "lesson", + "let", + "lethal", + "letter", + "letting", + "level", + "liaison", + "liberation", + "liberties", + "library", + "licence", + "license", + "lie", + "life", + "lifelong", + "lifespan", + "lifestyle", + "lifetime", + "lift", + "light", + "lighter", + "like", + "likelihood", + "likewise", + "limb", + "limestone", + "limit", + "limitations", + "line", + "lineage", + "linear", + "linebacker", + "liner", + "lineup", + "linguistic", + "link", + "lip", + "liquid", + "liquor", + "list", + "listen", + "listeners", + "lit", + "literacy", + "literally", + "literary", + "literature", + "lithium", + "little", + "live", + "liver", + "livestock", + "lizard", + "load", + "loan", + "lobby", + "local", + "localized", + "locate", + "location", + "lock", + "log", + "logic", + "logical", + "logistics", + "logo", + "lonely", + "long", + "longer", + "longest", + "longtime", + "look", + "loop", + "loose", + "lose", + "loss", + "lost", + "lot", + "lottery", + "loud", + "lounge", + "love", + "lover", + "low", + "lower", + "lowest", + "loyal", + "loyalty", + "luck", + "lumber", + "lunar", + "lunch", + "lung", + "lure", + "luxury", + "lying", + "lyric", + "machine", + "machinery", + "magazine", + "magic", + "magical", + "magistrate", + "magnetic", + "magnificent", + "magnitude", + "maid", + "maiden", + "mail", + "main", + "mainland", + "mainstream", + "maintain", + "maintenance", + "major", + "majority", + "make", + "maker", + "makeup", + "malaria", + "male", + "mammals", + "man", + "manage", + "management", + "manager", + "mandate", + "mandatory", + "maneuver", + "manga", + "mankind", + "manned", + "manner", + "manpower", + "mansion", + "mantle", + "manual", + "manufacture", + "manufacturer", + "manuscript", + "map", + "marble", + "marched", + "marching", + "margin", + "marginal", + "marijuana", + "maritime", + "marked", + "marker", + "market", + "marking", + "marks", + "marriage", + "married", + "marry", + "martial", + "mascot", + "mask", + "mass", + "massacre", + "massive", + "mast", + "master", + "masterpiece", + "mat", + "match", + "mate", + "material", + "maternal", + "mathematical", + "mathematics", + "matrix", + "matter", + "mature", + "maturity", + "maximum", + "maybe", + "mayor", + "meal", + "mean", + "meant", + "meantime", + "measure", + "measurement", + "meat", + "mechanic", + "mechanical", + "mechanism", + "medal", + "media", + "median", + "medical", + "medication", + "medicine", + "medieval", + "meditation", + "medium", + "meet", + "melee", + "melody", + "melting", + "member", + "membership", + "membrane", + "memoir", + "memorable", + "memory", + "men", + "mental", + "mention", + "mentor", + "menu", + "mercenaries", + "merchandise", + "merchant", + "mere", + "merge", + "merger", + "merit", + "mess", + "message", + "messenger", + "met", + "metal", + "metallic", + "metaphor", + "meter", + "method", + "metre", + "metric", + "mice", + "microphone", + "mid", + "middle", + "midfielder", + "midnight", + "midst", + "migrated", + "migration", + "mild", + "mile", + "milestone", + "military", + "militia", + "milk", + "million", + "min", + "mind", + "mineral", + "miners", + "mini", + "miniature", + "minimal", + "minimum", + "ministers", + "minor", + "minority", + "minute", + "mirrors", + "missed", + "missile", + "missing", + "mission", + "missionary", + "mistake", + "mistaken", + "mistress", + "mix", + "mixture", + "mob", + "mobile", + "mobility", + "mock", + "mod", + "mode", + "model", + "moderate", + "modern", + "modest", + "modification", + "modified", + "module", + "moist", + "moisture", + "molecular", + "molecules", + "moment", + "momentum", + "monarch", + "monarchy", + "monastery", + "monetary", + "money", + "monitor", + "monk", + "monkey", + "monopoly", + "monster", + "month", + "monument", + "mood", + "moons", + "moral", + "morale", + "morality", + "morning", + "morphology", + "mortality", + "mortar", + "mosque", + "mother", + "motif", + "motion", + "motivated", + "motivation", + "motives", + "motor", + "motorcycle", + "motto", + "mound", + "mountain", + "mounted", + "mounting", + "mounts", + "mourning", + "mouse", + "mouth", + "moved", + "movement", + "moves", + "movie", + "moving", + "mph", + "mud", + "multi", + "multiplayer", + "multiple", + "municipal", + "municipality", + "murder", + "murderer", + "muscle", + "muscular", + "museums", + "mushroom", + "music", + "musical", + "musician", + "mutations", + "mutual", + "muzzle", + "mysterious", + "mystery", + "myth", + "mythology", + "naked", + "named", + "names", + "naming", + "narrative", + "narrator", + "narrow", + "nation", + "nationalism", + "nationalist", + "nationally", + "nationwide", + "native", + "natural", + "nature", + "naval", + "nave", + "navigation", + "near", + "nearby", + "nearest", + "necessarily", + "necessary", + "necessity", + "neck", + "need", + "negative", + "neglected", + "negotiate", + "negotiations", + "neighbor", + "neighborhood", + "neighbouring", + "neighbours", + "neo", + "neon", + "nephew", + "nerve", + "nervous", + "nest", + "net", + "network", + "neutral", + "neutrality", + "neutron", + "new", + "newer", + "newspaper", + "nice", + "nickel", + "nickname", + "niece", + "night", + "nightclub", + "nineteen", + "nineteenth", + "ninety", + "ninth", + "nitrogen", + "noble", + "noir", + "noise", + "nominal", + "nominate", + "nomination", + "nominee", + "non", + "noon", + "norm", + "normal", + "northeast", + "northeastern", + "northern", + "northwest", + "nose", + "notable", + "notably", + "note", + "notice", + "notified", + "notion", + "notorious", + "novel", + "novelist", + "nuclear", + "nuclei", + "nucleus", + "nude", + "number", + "numerical", + "numerous", + "nurse", + "nursery", + "nutrients", + "oath", + "object", + "objections", + "objective", + "obligations", + "obliged", + "obscure", + "observation", + "observe", + "observers", + "obsessed", + "obsolete", + "obstacles", + "obtain", + "obvious", + "occasion", + "occasional", + "occupation", + "occupied", + "occupy", + "occur", + "occurred", + "occurrence", + "occurring", + "oceans", + "odd", + "offence", + "offense", + "offensive", + "offer", + "office", + "officer", + "official", + "offs", + "offset", + "offshore", + "offspring", + "oil", + "old", + "older", + "oldest", + "olive", + "omitted", + "ones", + "ongoing", + "online", + "onset", + "onstage", + "onwards", + "open", + "opener", + "opera", + "operate", + "operation", + "operational", + "operative", + "operator", + "opinion", + "opponent", + "opportunity", + "oppose", + "opposite", + "opposition", + "opted", + "optical", + "optimistic", + "option", + "optional", + "oral", + "orbit", + "orbital", + "orchestra", + "order", + "ordinance", + "ordinary", + "ore", + "organ", + "organic", + "organisation", + "organised", + "organism", + "organization", + "organize", + "orientation", + "oriented", + "origin", + "original", + "originated", + "originating", + "ought", + "outbreak", + "outcome", + "outdoor", + "outer", + "outfit", + "outlet", + "outline", + "outlook", + "outpost", + "output", + "outright", + "outside", + "outskirts", + "outward", + "oval", + "overall", + "overcome", + "overhaul", + "overhead", + "overlap", + "overlooked", + "overlooking", + "overly", + "overnight", + "overs", + "overseas", + "oversee", + "overthrow", + "overtime", + "overturned", + "overview", + "overwhelmed", + "overwhelming", + "owed", + "owing", + "owl", + "owned", + "owner", + "ownership", + "owns", + "oxidation", + "oxide", + "oxygen", + "pace", + "pack", + "package", + "pagan", + "page", + "paid", + "pain", + "painful", + "paint", + "painter", + "pair", + "pale", + "panel", + "panic", + "pants", + "paper", + "par", + "para", + "parade", + "parallel", + "paranormal", + "parasites", + "parent", + "parental", + "parish", + "parking", + "parks", + "parliamentary", + "parody", + "partial", + "participant", + "participate", + "participation", + "particle", + "particular", + "partition", + "partly", + "partner", + "partnership", + "parts", + "party", + "pass", + "passage", + "passenger", + "passion", + "passionate", + "passive", + "past", + "pastor", + "patch", + "patent", + "paternal", + "path", + "patient", + "patriotic", + "patrol", + "patron", + "pattern", + "paved", + "pay", + "payment", + "peace", + "peaceful", + "peak", + "peasant", + "pedestrian", + "peer", + "pen", + "penalty", + "pending", + "penetrate", + "penned", + "pension", + "people", + "perceived", + "percent", + "percentage", + "perception", + "percussion", + "perfect", + "perform", + "performance", + "performer", + "perimeter", + "period", + "periodic", + "periodically", + "permanent", + "permission", + "permit", + "permitted", + "persecution", + "persisted", + "persistent", + "person", + "persona", + "personal", + "personality", + "personnel", + "perspective", + "persuade", + "pet", + "petition", + "petroleum", + "petty", + "phase", + "phenomena", + "phenomenon", + "philosopher", + "philosophical", + "philosophy", + "phone", + "photo", + "photograph", + "photographer", + "photographic", + "photography", + "phrase", + "physical", + "physician", + "physicist", + "physics", + "piano", + "pick", + "picture", + "pie", + "piece", + "pier", + "piercing", + "pig", + "pigeon", + "pile", + "pilgrimage", + "pillar", + "pilot", + "pin", + "pink", + "pinned", + "pioneer", + "pipe", + "pipeline", + "pirate", + "pistol", + "pit", + "pitch", + "pitcher", + "place", + "placement", + "plague", + "plain", + "plan", + "plane", + "planet", + "planetary", + "planned", + "planning", + "plant", + "plantation", + "plaque", + "plasma", + "plastic", + "plate", + "plateau", + "platform", + "platinum", + "play", + "playable", + "player", + "playoff", + "playwright", + "plea", + "pleaded", + "pleasant", + "pleased", + "pleasure", + "pledged", + "plenty", + "plot", + "plug", + "plus", + "pneumonia", + "pocket", + "poem", + "poet", + "poetic", + "poetry", + "point", + "poison", + "poker", + "polar", + "pole", + "police", + "policy", + "polished", + "political", + "politician", + "politics", + "poll", + "pollution", + "pool", + "poor", + "pop", + "populace", + "popular", + "popularity", + "populated", + "population", + "porch", + "port", + "portable", + "portal", + "portion", + "portrait", + "portray", + "portrayal", + "pose", + "position", + "positive", + "possess", + "possession", + "possibility", + "possible", + "possibly", + "post", + "postal", + "poster", + "postponed", + "postseason", + "pot", + "potassium", + "potential", + "pottery", + "pound", + "poverty", + "powder", + "power", + "powerful", + "practical", + "practice", + "practitioners", + "praise", + "pray", + "prayer", + "pre", + "preceded", + "precedent", + "preceding", + "precious", + "precipitation", + "precise", + "precision", + "precursor", + "predator", + "predecessor", + "predict", + "predictable", + "prediction", + "predominantly", + "prefer", + "preference", + "preferred", + "preferring", + "pregnancy", + "pregnant", + "prejudice", + "preliminary", + "premiere", + "premise", + "premium", + "preparation", + "prepare", + "prescribed", + "preseason", + "presence", + "present", + "presentation", + "preservation", + "preserve", + "presidency", + "presidential", + "presidents", + "press", + "pressure", + "prestige", + "prestigious", + "presumably", + "presumed", + "pretty", + "prevailed", + "prevailing", + "prevalent", + "prevent", + "prevention", + "preview", + "previous", + "prey", + "price", + "pride", + "priest", + "primarily", + "primary", + "primitive", + "princes", + "principal", + "principle", + "print", + "prior", + "priority", + "prison", + "prisoner", + "privacy", + "private", + "privilege", + "prizes", + "pro", + "probability", + "probable", + "probably", + "probe", + "problem", + "problematic", + "procedure", + "proceed", + "process", + "procession", + "proclaimed", + "proclamation", + "produce", + "producer", + "product", + "production", + "productive", + "profession", + "professional", + "professor", + "profile", + "profit", + "profitable", + "profound", + "program", + "programme", + "progress", + "progression", + "progressive", + "prohibited", + "prohibition", + "project", + "projection", + "prolific", + "prolonged", + "prominence", + "prominent", + "promise", + "promote", + "promotion", + "promotional", + "prompted", + "prompting", + "promptly", + "prone", + "pronounced", + "pronunciation", + "proof", + "propaganda", + "propelled", + "proper", + "property", + "prophet", + "proportion", + "proposal", + "propose", + "props", + "propulsion", + "prose", + "prosecution", + "prosecutor", + "prospect", + "prosperity", + "protagonist", + "protect", + "protection", + "protective", + "protector", + "protein", + "protest", + "protesters", + "proto", + "protocol", + "prototype", + "proud", + "prove", + "proven", + "provide", + "province", + "provincial", + "provision", + "provisional", + "provoked", + "proximity", + "psi", + "psychedelic", + "psychiatric", + "psychological", + "psychology", + "pub", + "public", + "publication", + "publicity", + "publish", + "publisher", + "puck", + "pull", + "pulp", + "pulse", + "pump", + "punch", + "punished", + "punishment", + "punk", + "punt", + "pupil", + "puppet", + "purchase", + "pure", + "purple", + "purpose", + "pursue", + "pursuit", + "push", + "puts", + "putting", + "puzzle", + "pyramid", + "qualification", + "qualified", + "qualify", + "quality", + "quantity", + "quantum", + "quarry", + "quarter", + "quarterback", + "question", + "quick", + "quiet", + "quit", + "quite", + "quote", + "race", + "racial", + "racism", + "racist", + "radar", + "radial", + "radiation", + "radical", + "radio", + "radioactive", + "radius", + "rage", + "raid", + "rail", + "railroad", + "railway", + "rain", + "rainfall", + "raise", + "rally", + "ramp", + "ran", + "random", + "range", + "rank", + "rap", + "rape", + "rapid", + "rapper", + "rare", + "rat", + "rate", + "ratified", + "ratio", + "rational", + "rays", + "reach", + "react", + "reaction", + "reactor", + "read", + "reader", + "readily", + "ready", + "real", + "realised", + "realism", + "realistic", + "reality", + "realize", + "realm", + "rear", + "reason", + "reasonable", + "reasonably", + "rebel", + "rebellion", + "rebounds", + "rebuild", + "rebuilt", + "recall", + "receive", + "receiver", + "recent", + "reception", + "recession", + "recipes", + "recipient", + "recognise", + "recognition", + "recognize", + "recommend", + "recommendation", + "reconciliation", + "reconnaissance", + "reconstructed", + "reconstruction", + "record", + "recounted", + "recover", + "recovery", + "recreational", + "recruit", + "recruitment", + "rectangular", + "recurring", + "red", + "redemption", + "redesigned", + "redevelopment", + "reduce", + "reduction", + "reef", + "reel", + "refer", + "referee", + "reference", + "referendum", + "referred", + "referring", + "refined", + "reflect", + "reflection", + "reform", + "refuge", + "refugee", + "refusal", + "refuse", + "regain", + "regard", + "regardless", + "regime", + "region", + "regional", + "registered", + "registration", + "registry", + "regret", + "regular", + "regulate", + "regulation", + "rehabilitation", + "rehearsal", + "reign", + "reinforce", + "reinforcements", + "reject", + "rejection", + "relate", + "relation", + "relationship", + "relative", + "relaxed", + "relay", + "release", + "relegated", + "relevant", + "reliable", + "reliance", + "relics", + "relied", + "relief", + "relieve", + "religion", + "religious", + "relocated", + "relocation", + "reluctant", + "rely", + "remain", + "remainder", + "remake", + "remarkable", + "remarked", + "remarks", + "rematch", + "remember", + "reminded", + "reminiscent", + "remix", + "remnants", + "remote", + "removal", + "remove", + "renamed", + "render", + "rendition", + "renewable", + "renewal", + "renewed", + "renovated", + "renovation", + "renowned", + "rent", + "reopened", + "repair", + "repeat", + "repertoire", + "repetitive", + "replace", + "replacement", + "replay", + "replica", + "replied", + "reply", + "report", + "reporter", + "represent", + "representation", + "representative", + "reproduce", + "reproduction", + "reproductive", + "reputation", + "request", + "require", + "requirement", + "rescue", + "research", + "researcher", + "resemblance", + "resemble", + "reservation", + "reserve", + "reservoir", + "residence", + "resident", + "residential", + "resign", + "resignation", + "resist", + "resistance", + "resistant", + "resolution", + "resolve", + "resonance", + "resort", + "resource", + "respect", + "respective", + "respiratory", + "respond", + "response", + "responsibility", + "responsible", + "rest", + "restart", + "restaurant", + "restoration", + "restore", + "restricted", + "restriction", + "result", + "resume", + "resurrection", + "retail", + "retailers", + "retain", + "retaliation", + "retire", + "retirement", + "retreat", + "retrieve", + "retro", + "retrospective", + "return", + "reunion", + "reunited", + "reveal", + "revelation", + "revenge", + "revenue", + "reverse", + "reverted", + "review", + "reviewer", + "revised", + "revision", + "revival", + "revive", + "revolt", + "revolutionary", + "revolver", + "revolves", + "reward", + "rewrite", + "rhythm", + "ribbon", + "rice", + "rich", + "rid", + "ride", + "rider", + "ridge", + "ridiculous", + "riff", + "rifle", + "rift", + "right", + "rigid", + "rim", + "ring", + "riot", + "rip", + "rise", + "risen", + "risk", + "rites", + "ritual", + "rival", + "rivalry", + "rivers", + "roads", + "roadway", + "robbery", + "robot", + "robust", + "rock", + "rocket", + "rode", + "role", + "roll", + "roller", + "romance", + "romantic", + "roof", + "rookie", + "room", + "root", + "rope", + "rose", + "roster", + "rotating", + "rotation", + "rough", + "round", + "route", + "routine", + "row", + "royalty", + "rubber", + "rugby", + "ruin", + "rule", + "ruler", + "rumors", + "rumours", + "run", + "runner", + "running", + "runway", + "rural", + "rushed", + "rushing", + "sac", + "sack", + "sacred", + "sacrifice", + "sad", + "safe", + "safety", + "sage", + "said", + "sail", + "sailor", + "sake", + "salary", + "sale", + "salmon", + "salt", + "salvage", + "salvation", + "sample", + "sanctuary", + "sand", + "sandwich", + "sang", + "sank", + "sat", + "satellite", + "satire", + "satisfaction", + "satisfied", + "satisfy", + "save", + "saw", + "say", + "scale", + "scandal", + "scarce", + "scary", + "scattered", + "scenario", + "scene", + "scenery", + "schedule", + "scheme", + "scholar", + "scholarship", + "school", + "science", + "scientific", + "scientist", + "scope", + "score", + "scorer", + "scrap", + "scrapped", + "scratch", + "screaming", + "screen", + "screenplay", + "screw", + "script", + "scrutiny", + "sculpture", + "sea", + "seal", + "search", + "season", + "seasonal", + "seat", + "second", + "secondary", + "secret", + "section", + "sector", + "secular", + "secure", + "security", + "sediment", + "seed", + "seeing", + "seek", + "seemingly", + "seen", + "sees", + "segment", + "seize", + "seizure", + "seldom", + "select", + "selection", + "selective", + "self", + "sell", + "seller", + "semi", + "send", + "senior", + "sensation", + "sense", + "sensitive", + "sensitivity", + "sent", + "sentence", + "sentiment", + "separate", + "separation", + "sequel", + "sequence", + "serial", + "series", + "seriously", + "servant", + "serve", + "service", + "session", + "set", + "setting", + "settle", + "settlement", + "settlers", + "seven", + "seventeen", + "seventh", + "seventy", + "severe", + "severity", + "sex", + "sexual", + "sexuality", + "sexy", + "shade", + "shaft", + "shake", + "shale", + "shall", + "shallow", + "shame", + "shape", + "share", + "shark", + "sharp", + "shattered", + "shear", + "shed", + "sheep", + "sheer", + "sheet", + "shell", + "shelter", + "shield", + "shift", + "ship", + "shipment", + "shipped", + "shipping", + "shirt", + "shit", + "shock", + "shoes", + "shoot", + "shooter", + "shop", + "shopping", + "shore", + "shoreline", + "short", + "shortage", + "shortened", + "shorter", + "shot", + "shoulder", + "showcase", + "showed", + "shower", + "showing", + "shown", + "shows", + "shrine", + "shut", + "shuttle", + "shy", + "siblings", + "sic", + "sick", + "sided", + "sides", + "siege", + "sight", + "sign", + "signal", + "signature", + "significance", + "significant", + "silence", + "silent", + "silicon", + "silk", + "silly", + "silver", + "similar", + "similarity", + "simple", + "simpler", + "simplicity", + "simplified", + "simply", + "simulation", + "simultaneous", + "sin", + "sing", + "singer", + "single", + "sink", + "sister", + "sit", + "sitcom", + "site", + "sitting", + "situated", + "situation", + "sixteen", + "sixth", + "size", + "skating", + "skeleton", + "skeptical", + "sketch", + "ski", + "skill", + "skin", + "skull", + "slate", + "slaughter", + "slave", + "slavery", + "sleep", + "sleeve", + "slender", + "slide", + "slight", + "slip", + "slipped", + "slogan", + "slope", + "slot", + "slow", + "slower", + "small", + "smaller", + "smallest", + "smell", + "smile", + "smoke", + "smooth", + "snakes", + "snow", + "soap", + "soccer", + "social", + "socialism", + "socialist", + "societies", + "sociology", + "sodium", + "soft", + "software", + "soil", + "solar", + "sold", + "soldier", + "sole", + "solid", + "solitary", + "solo", + "solution", + "solve", + "somebody", + "somewhat", + "son", + "song", + "songwriter", + "soon", + "sophisticated", + "sophomore", + "sorry", + "sort", + "sought", + "soul", + "sound", + "soundtrack", + "soup", + "source", + "southeast", + "southeastern", + "southern", + "southwest", + "southwestern", + "sovereign", + "sovereignty", + "space", + "spacecraft", + "span", + "spanning", + "spare", + "spark", + "sparse", + "spawned", + "speak", + "speaker", + "special", + "specialist", + "specialized", + "species", + "specific", + "specifically", + "specifications", + "specified", + "specimen", + "spectacular", + "spectators", + "spectral", + "spectrum", + "speculated", + "speculation", + "speech", + "speed", + "spell", + "spend", + "spent", + "sperm", + "sphere", + "spherical", + "spiders", + "spin", + "spinal", + "spine", + "spinning", + "spiral", + "spirit", + "spiritual", + "spite", + "split", + "splitting", + "spoke", + "spoken", + "spokesman", + "spokesperson", + "sponsor", + "sponsorship", + "spontaneous", + "sport", + "spot", + "spotted", + "spread", + "spring", + "sprint", + "spun", + "spur", + "squad", + "square", + "stability", + "stable", + "stack", + "stadium", + "staff", + "stage", + "stained", + "staircase", + "stairs", + "stake", + "stalled", + "stamp", + "stance", + "stand", + "standard", + "stanza", + "star", + "starred", + "starring", + "start", + "starter", + "state", + "statement", + "static", + "station", + "stationary", + "statistical", + "statistics", + "statue", + "status", + "statute", + "statutory", + "stay", + "steadily", + "steady", + "steal", + "stealth", + "steam", + "steel", + "steep", + "steering", + "stellar", + "stem", + "step", + "stepped", + "stereo", + "stereotypes", + "stern", + "stick", + "sticky", + "stiff", + "stint", + "stock", + "stole", + "stolen", + "stomach", + "stones", + "stood", + "stop", + "stopped", + "stopping", + "storage", + "store", + "storm", + "story", + "storyline", + "storytelling", + "straight", + "strain", + "strand", + "strange", + "strategic", + "strategy", + "streak", + "stream", + "streets", + "strength", + "strengthen", + "stress", + "stretch", + "strict", + "strike", + "strikeouts", + "striker", + "string", + "strip", + "stripped", + "stroke", + "strong", + "stronger", + "strongest", + "stronghold", + "struck", + "structural", + "structure", + "struggle", + "stuck", + "stud", + "student", + "studied", + "studio", + "study", + "stuff", + "stunning", + "stunt", + "stupid", + "style", + "sub", + "subdivision", + "subject", + "submarine", + "submerged", + "submission", + "submit", + "submitted", + "subordinate", + "subplot", + "subsequent", + "subsidiary", + "substance", + "substantial", + "substitute", + "substrate", + "subtle", + "suburb", + "suburban", + "subway", + "succeed", + "success", + "successful", + "succession", + "successive", + "successor", + "sudden", + "sued", + "suffer", + "sufficient", + "sugar", + "suggest", + "suggestion", + "suicide", + "suit", + "suitable", + "suite", + "sulfur", + "sum", + "summarized", + "summary", + "summer", + "summit", + "summoned", + "sung", + "sunk", + "sunlight", + "sunny", + "superb", + "superhero", + "superior", + "superiority", + "supernatural", + "supervised", + "supervision", + "supervisor", + "supplement", + "supplied", + "supply", + "support", + "supporter", + "supportive", + "supposed", + "suppress", + "suppression", + "sur", + "sure", + "surf", + "surface", + "surge", + "surgeon", + "surgery", + "surgical", + "surname", + "surpassed", + "surplus", + "surprise", + "surrender", + "surround", + "surveillance", + "survey", + "survival", + "survive", + "survivors", + "susceptible", + "suspect", + "suspended", + "suspension", + "suspicion", + "suspicious", + "sustain", + "sustainable", + "swamp", + "sweep", + "swept", + "swim", + "swimming", + "swing", + "switch", + "sword", + "sworn", + "symbol", + "symbolic", + "symbolism", + "symmetry", + "sympathetic", + "sympathy", + "symptoms", + "synagogue", + "syndrome", + "synth", + "synthesis", + "synthesized", + "synthetic", + "systematic", + "systems", + "table", + "tablets", + "tackle", + "tactic", + "tactical", + "tail", + "taken", + "takeover", + "takes", + "taking", + "tale", + "talent", + "talk", + "tall", + "tallest", + "tally", + "tank", + "tape", + "target", + "task", + "taste", + "taught", + "tax", + "taxa", + "taxation", + "taxi", + "tea", + "teach", + "teacher", + "team", + "teammate", + "tear", + "technical", + "technique", + "technological", + "technology", + "teen", + "teenage", + "teenager", + "teeth", + "telephone", + "telescope", + "televised", + "television", + "tell", + "temperature", + "temple", + "tempo", + "temporarily", + "temporary", + "tenants", + "tend", + "tendency", + "tender", + "tennis", + "tens", + "tense", + "tension", + "tent", + "tenth", + "tenure", + "term", + "terminal", + "terminated", + "terrain", + "terrestrial", + "terrible", + "territorial", + "territory", + "terror", + "terrorism", + "terrorist", + "test", + "testified", + "testimony", + "text", + "textile", + "texture", + "thank", + "theater", + "theatrical", + "theft", + "thematic", + "theme", + "theological", + "theology", + "theorem", + "theoretical", + "theory", + "therapy", + "thermal", + "thesis", + "thickness", + "thing", + "think", + "thirds", + "thirteen", + "thirty", + "thorough", + "thought", + "thousand", + "threat", + "threaten", + "threw", + "thriller", + "throat", + "throne", + "throw", + "thrown", + "thrust", + "ticket", + "tidal", + "tide", + "tie", + "tier", + "tight", + "tiles", + "till", + "timber", + "time", + "timeline", + "tin", + "tiny", + "tip", + "tired", + "tissue", + "title", + "tobacco", + "today", + "told", + "tolerance", + "toll", + "tomb", + "ton", + "tone", + "tongue", + "tonnes", + "took", + "tool", + "tooth", + "topic", + "topped", + "topping", + "tops", + "torch", + "torn", + "tornado", + "torpedo", + "torture", + "toss", + "total", + "touch", + "touchdown", + "tough", + "tour", + "tourism", + "tourist", + "tournament", + "tow", + "tower", + "town", + "toxic", + "toxicity", + "toy", + "trace", + "track", + "tract", + "trade", + "trademark", + "traders", + "tradition", + "traditional", + "traffic", + "trafficking", + "tragedy", + "tragic", + "trail", + "trailer", + "train", + "trainer", + "trait", + "tram", + "transactions", + "transfer", + "transferred", + "transferring", + "transform", + "transformation", + "transit", + "transition", + "translated", + "translation", + "transmission", + "transmitted", + "transport", + "trap", + "trapped", + "trauma", + "travel", + "travelers", + "travelled", + "travelling", + "traverse", + "treason", + "treasure", + "treasurer", + "treat", + "treatment", + "tree", + "tremendous", + "trench", + "trend", + "trial", + "triangle", + "triangular", + "tribal", + "tribe", + "tribute", + "trick", + "tried", + "trigger", + "trilogy", + "trio", + "trip", + "triple", + "triumph", + "troop", + "tropical", + "trouble", + "trough", + "trout", + "truck", + "true", + "truly", + "trumpet", + "truncated", + "trunk", + "trusted", + "trustees", + "truth", + "try", + "tube", + "tuberculosis", + "tune", + "tunnel", + "turbine", + "turf", + "turmoil", + "turn", + "turret", + "turtle", + "twentieth", + "twice", + "twin", + "twist", + "tying", + "type", + "typical", + "tyre", + "ultimate", + "ultra", + "unable", + "unanimous", + "unavailable", + "unaware", + "uncertain", + "uncertainty", + "unchanged", + "uncle", + "unclear", + "uncomfortable", + "uncommon", + "unconscious", + "uncovered", + "und", + "undefeated", + "undergo", + "undergone", + "undergraduate", + "underground", + "underlying", + "underneath", + "underside", + "understand", + "understood", + "undertake", + "undertaken", + "underwater", + "underway", + "underwent", + "unemployment", + "unexpected", + "unfinished", + "unhappy", + "unidentified", + "unified", + "uniform", + "unions", + "unique", + "unit", + "unite", + "unity", + "universally", + "universe", + "universities", + "unknown", + "unless", + "unlikely", + "unlimited", + "unnamed", + "unnecessary", + "unofficial", + "unpopular", + "unprecedented", + "unrelated", + "unrest", + "unsigned", + "unstable", + "unsuccessful", + "unsure", + "unused", + "unusual", + "unveiled", + "unwilling", + "upcoming", + "update", + "upgrade", + "upheld", + "upper", + "upright", + "uprising", + "ups", + "upset", + "upstream", + "upward", + "uranium", + "urban", + "urged", + "urging", + "urine", + "usage", + "use", + "useful", + "user", + "usual", + "utility", + "utilized", + "utterly", + "vacant", + "vacation", + "vaccine", + "vacuum", + "vague", + "valid", + "valleys", + "valuable", + "value", + "vampire", + "variable", + "variant", + "variation", + "varied", + "variety", + "various", + "vary", + "vast", + "vault", + "vector", + "vegetables", + "vegetation", + "vehicle", + "vein", + "velocity", + "venom", + "venture", + "venue", + "verbal", + "verdict", + "verse", + "version", + "versus", + "vertical", + "vessel", + "veteran", + "veto", + "viable", + "vibe", + "vicinity", + "vicious", + "victim", + "victorious", + "victory", + "video", + "view", + "viewer", + "vigorous", + "village", + "villagers", + "villain", + "vintage", + "vinyl", + "violated", + "violation", + "violence", + "violent", + "violin", + "viral", + "virtual", + "virtue", + "virus", + "visibility", + "visible", + "vision", + "visit", + "visitor", + "visual", + "vital", + "vitamin", + "vocal", + "vocalist", + "voice", + "void", + "volcanic", + "volcano", + "voltage", + "volume", + "voluntary", + "volunteer", + "von", + "vote", + "voters", + "voyage", + "vulnerable", + "wage", + "wagon", + "wait", + "wake", + "walk", + "wall", + "want", + "wardrobe", + "wards", + "warehouse", + "warfare", + "warm", + "warmer", + "warn", + "warrant", + "warriors", + "wartime", + "washed", + "wasn", + "waste", + "watch", + "water", + "watershed", + "wave", + "wax", + "way", + "weak", + "weaken", + "weaker", + "weakness", + "wealth", + "wealthy", + "weapon", + "weaponry", + "wear", + "weather", + "web", + "website", + "wedding", + "week", + "weekend", + "weigh", + "weight", + "weird", + "welcome", + "welding", + "welfare", + "went", + "weren", + "wet", + "whale", + "wheat", + "wheel", + "wheelchair", + "whilst", + "whip", + "white", + "wholly", + "wide", + "widened", + "widening", + "wider", + "widespread", + "widow", + "width", + "wife", + "wild", + "wilderness", + "willing", + "willingness", + "win", + "wind", + "window", + "wine", + "wing", + "winner", + "winning", + "winter", + "wire", + "wireless", + "wisdom", + "wise", + "wish", + "wit", + "witch", + "withdraw", + "withdrawal", + "withdrawn", + "withdrew", + "withstand", + "witness", + "wives", + "wolves", + "woman", + "women", + "won", + "wonderful", + "wood", + "wooden", + "woodland", + "wool", + "word", + "wore", + "work", + "worker", + "workshop", + "worlds", + "worldwide", + "worn", + "worried", + "worry", + "worse", + "worship", + "worst", + "worth", + "worthy", + "wouldn", + "wound", + "wrapped", + "wreck", + "wrestler", + "write", + "writer", + "written", + "wrong", + "wrote", + "wrought", + "yard", + "year", + "yellow", + "yield", + "yoga", + "young", + "younger", + "youngest", + "youth", + "zinc", + "zombie", + "zone" + ] + }, + "battery": { + "n": 462, + "rule": "uniform sample without replacement of 462 from the sorted eligible set at seed 52; size matched to the 2026-08 battery so the reproduction gate compares like with like (same N, same 1/70 chance under the same 70/392 split)", + "sha256": "61679f9378dd3cc2d8407d1d3d5b7dd0f165c3accfbb3abeccc2d414aacfd11a", + "words": [ + "accelerated", + "acclaim", + "accuracy", + "adapt", + "additional", + "administrator", + "advised", + "affiliate", + "agent", + "airborne", + "alliance", + "altar", + "aluminium", + "angular", + "announcement", + "answer", + "anymore", + "appropriate", + "approximately", + "assassin", + "assess", + "assignment", + "associations", + "audience", + "augmented", + "autonomous", + "bachelor", + "band", + "bank", + "barred", + "beam", + "beer", + "beginning", + "beings", + "bells", + "belt", + "bicycle", + "bigger", + "billed", + "bitter", + "blood", + "bore", + "bow", + "boy", + "bread", + "breathe", + "breed", + "brilliant", + "bring", + "bubble", + "busiest", + "buy", + "cancellation", + "canon", + "caught", + "celebrate", + "celebration", + "celestial", + "cellular", + "ceremonial", + "change", + "charitable", + "cheap", + "choice", + "choose", + "civilian", + "clash", + "clear", + "cluster", + "coastal", + "companion", + "company", + "comparison", + "compartments", + "competitor", + "complaint", + "comprise", + "confined", + "confusing", + "conscious", + "consent", + "consistent", + "conspiracy", + "consulted", + "contact", + "contaminated", + "contestant", + "continuation", + "continuity", + "contribute", + "corporate", + "countdown", + "creator", + "crossover", + "crying", + "decoration", + "deeper", + "define", + "deleted", + "deliberate", + "delicate", + "dental", + "deny", + "depict", + "depot", + "depth", + "design", + "devastating", + "did", + "dignity", + "dining", + "dioxide", + "diplomat", + "disappointment", + "discovery", + "disease", + "distinction", + "dock", + "documentary", + "domestic", + "drawn", + "drunk", + "duck", + "dull", + "dying", + "earliest", + "easy", + "economic", + "ejected", + "electoral", + "electrical", + "eliminate", + "elk", + "emerge", + "emergency", + "end", + "ensemble", + "enthusiastic", + "entrance", + "era", + "escape", + "establish", + "evening", + "everyday", + "exception", + "excluding", + "execution", + "exist", + "expelled", + "exploration", + "explore", + "extensive", + "external", + "fashion", + "fashionable", + "favorable", + "feathers", + "feedback", + "festivals", + "feudal", + "fictional", + "fishermen", + "fitting", + "flight", + "floating", + "football", + "fossil", + "fraction", + "fraternity", + "fresh", + "freshman", + "fried", + "fronts", + "frustrated", + "fully", + "fungi", + "future", + "gain", + "gambling", + "gear", + "generals", + "gesture", + "girl", + "granting", + "gravel", + "grazing", + "groove", + "hang", + "heroine", + "hide", + "higher", + "honorary", + "humanitarian", + "humour", + "hung", + "identity", + "illuminated", + "image", + "imagine", + "immune", + "improve", + "inability", + "incomplete", + "independence", + "industrial", + "industry", + "influential", + "inserted", + "instrument", + "integration", + "intensity", + "interchange", + "interface", + "interfere", + "isolated", + "jealous", + "judiciary", + "juvenile", + "kitchen", + "knots", + "landmark", + "latest", + "law", + "leader", + "leopard", + "lesser", + "letting", + "license", + "liquid", + "liquor", + "literature", + "little", + "live", + "lock", + "logical", + "logo", + "lot", + "lottery", + "machine", + "make", + "manga", + "map", + "marry", + "massive", + "mean", + "mental", + "mentor", + "messenger", + "metal", + "mile", + "mineral", + "mirrors", + "missed", + "monkey", + "morality", + "mother", + "motor", + "movement", + "mud", + "museums", + "named", + "nation", + "nave", + "navigation", + "neighbours", + "nervous", + "newspaper", + "nineteen", + "noir", + "northeast", + "northeastern", + "nuclei", + "objective", + "obtain", + "occupy", + "oceans", + "odd", + "office", + "old", + "operation", + "operative", + "opposite", + "orchestra", + "organize", + "outlook", + "overhaul", + "overlap", + "painful", + "paint", + "panel", + "para", + "parallel", + "partnership", + "patient", + "peak", + "penetrate", + "periodically", + "permanent", + "persona", + "personnel", + "philosophical", + "piano", + "pitcher", + "place", + "placement", + "plain", + "planned", + "positive", + "possession", + "postal", + "pottery", + "power", + "preceding", + "precious", + "prejudice", + "preserve", + "primary", + "priority", + "prisoner", + "privacy", + "problematic", + "product", + "profession", + "profile", + "proportion", + "propose", + "prosperity", + "prototype", + "provision", + "public", + "pursuit", + "qualified", + "quarry", + "radial", + "radiation", + "radical", + "rain", + "range", + "rank", + "rape", + "reality", + "realize", + "rebuild", + "rectangular", + "redesigned", + "relics", + "remnants", + "rent", + "repeat", + "representative", + "resident", + "resolution", + "restart", + "reunited", + "reverted", + "ribbon", + "rich", + "ridge", + "riff", + "rifle", + "rim", + "rock", + "roller", + "routine", + "salvage", + "sanctuary", + "sang", + "scarce", + "scattered", + "scene", + "seen", + "seldom", + "sentence", + "shoot", + "shooter", + "shopping", + "sister", + "sixth", + "slogan", + "slow", + "software", + "solitary", + "somewhat", + "songwriter", + "sought", + "southern", + "spanning", + "specialized", + "speech", + "spinal", + "spoke", + "sponsorship", + "spotted", + "sprint", + "stairs", + "stern", + "strain", + "stud", + "style", + "subsequent", + "substance", + "summary", + "summit", + "supervised", + "suppression", + "surgery", + "survey", + "suspect", + "synthesized", + "systems", + "tank", + "tape", + "technological", + "telescope", + "televised", + "tempo", + "terrible", + "terrorism", + "terrorist", + "testified", + "throne", + "tide", + "tip", + "tissue", + "torture", + "touch", + "tourist", + "toy", + "traffic", + "translation", + "treason", + "trend", + "triangular", + "trilogy", + "turf", + "twice", + "twist", + "unavailable", + "unexpected", + "unfinished", + "unified", + "unique", + "unsigned", + "use", + "useful", + "vague", + "vibe", + "vinyl", + "virus", + "voice", + "volcano", + "wartime", + "waste", + "weaponry", + "weird", + "whilst", + "white", + "willingness", + "witness", + "word", + "worldwide", + "yield" + ] + }, + "survival": { + "pool": 547373, + "eligible": 5199, + "battery": 462, + "survival_ratio_battery_over_pool": 0.00084403, + "eligible_ratio_eligible_over_pool": 0.00949809, + "sampling_fraction_battery_over_eligible": 0.088863, + "means": "survival_ratio is the extraction term E for this battery: the share of recorded candidates that reached the scored set. For the 2026-08 battery the same quantity is UNCHECKABLE, because no pool was recorded." + }, + "comparison_to_2026_08_battery": { + "file": "run_g0clear.py lines 31-67", + "raw_tokens_in_literal": 465, + "after_its_only_filter": 462, + "its_filters": [ + "order-preserving dedup (removed 3: chicken, orange, mushroom)" + ], + "its_pool": null, + "its_survival_ratio": "UNCHECKABLE — no pool artifact exists", + "old_words_reaching_this_eligible_set": 260, + "old_words_in_this_battery": 15, + "old_words_refused_by_F6_single_token": [ + "apricot", + "banjo", + "beaver", + "blueberry", + "bookcase", + "broom", + "cactus", + "cello", + "clarinet", + "crocodile", + "donkey", + "fisher", + "giraffe", + "gorilla", + "harp", + "lighthouse", + "melon", + "octopus", + "otter", + "parrot", + "peacock", + "penguin", + "pliers", + "radish", + "sandal", + "saxophone", + "snail", + "sparrow", + "swan", + "tambourine", + "toad", + "turnip", + "wasp", + "watermelon", + "zebra" + ] + }, + "known_limits_of_this_artifact": [ + "The pool is one English corpus of Wikipedia prose. Concept frequency in WikiText is a proxy for frequency in the target models' pretraining, which is not observable.", + "Filter thresholds (frequency floor 500, length 3-14, cap share 0.5) are judgement calls stated in advance rather than values derived from a power analysis. They are frozen here so a later run cannot move them after seeing a read.", + "No part-of-speech filter is applied: no offline tagger is available in this environment. The successor battery is therefore broader in word class than the hand-authored one, which is exactly what the reproduction gate in the b52 prereg exists to price.", + "F4 and F5 are the two stages that most resemble the defect being repaired — a curated list and a hand-set threshold. They are logged item-by-item and by count so the disagreement is at least computable." + ] +}