diff --git a/TODO.sota-2026/01-r8-teacher-yallamorph.md b/TODO.sota-2026/01-r8-teacher-yallamorph.md index 484fbdb..73bde0e 100644 --- a/TODO.sota-2026/01-r8-teacher-yallamorph.md +++ b/TODO.sota-2026/01-r8-teacher-yallamorph.md @@ -1,8 +1,6 @@ # 01 — r8 teacher: run-009-yallamorph (YallaMorph/CamelMorph aux stream) -Status: TRAINING IN FLIGHT (launched 2026-10-01 18:07, app ap-f1R8ChDikMKUGyBuJGO6; -26,289 steps on A100-80GB; supervisor /tmp/r9-supervisor.sh v4 relaunches on -kill-storm deaths — training is checkpoint-resumable and EVAL_DONE-idempotent) +Status: CLOSED — VERIFIED NEGATIVE on both surfaces (2026-10-02); r7 stays canonical Literature basis: YallaMorph (arXiv 2609.10153, EMNLP 2026) — 663,804 controlled morphological-generation instances over 4,795 lemmas, constructed from CamelMorph MSA via CAMeL Tools. The GitHub repo ships @@ -71,12 +69,41 @@ Recipe = r7 verbatim (train_arabic_r7.py) + one new aux stream: 5. [x] Launch `modal run --detach` (+ supervisor; one double-launch incident from `modal app list` name truncation — grep prefix "rababa-ara", dupes stopped, volume verified clean). -6. [ ] ID gate: windowed zero-skip SadeedDiac-25 full 1,200-para DER - ≤ 2.389 (r7 2.2864 + 0.1 tolerance). -7. [ ] OOD gate: eval_wikinews_multiref improves over 17.3794/11.8273. -8. [ ] Canonical replacement only if ID improves outright; else record - as ablation. Student distillation from r9 only after canonical - call. +6. [x] ID gate: windowed zero-skip SadeedDiac-25 full 1,200-para DER + ≤ 2.389 (r7 2.2864 + 0.1 tolerance) — **FAILED: 2.4895**. +7. [x] OOD gate: WikiNews-2024 multi-ref — **FAILED: 17.4265/12.1093** + vs r7's 17.3794/11.8273 (worse on both axes; no trade). +8. [x] Canonical call: **r7 stays canonical.** run-009 recorded as + ablation. No student distillation from r9. + +## Result (measured 2026-10-02) — NEGATIVE, first negative teacher-side result + +| surface | r9 | r7 | verdict | +|---|---|---|---| +| SadeedDiac-25 Total DER (CE) | 2.4895 | 2.2864 | +0.20pp worse — gate fail | +| Morph DER | 1.5054 | 1.3343 | worse | +| WikiNews-2024 WER / DER | 17.4265 / 12.1093 | 17.3794 / 11.8273 | worse on both | + +**Mechanism (data-backed):** the aux corpus's vocalization convention +is far denser than the benchmark's — marks per 100 letters: fatha +43.0 vs 28.4 (1.5×), damma 11.9 vs 7.1 (1.7×), shadda 8.6 vs 4.3 +(2.0×), tanwīn 0.3 vs 2.0 (nearly absent). CamelMorph paradigm forms +are fully-explicit isolated words; SadeedDiac-style text is partially +vocalized running text. At 25% dose, the aux stream taught convention +drift, not morphology: the teacher over-marks the plain stream. + +**Lesson (refines the knowledge-injection template):** r6's qalsadi +aux worked because it was running text IN benchmark convention; r9's +paradigm tables were out-of-context AND differently-conventioned. +Knowledge injection is delivery-vehicle-sensitive. A future teacher +lever from lexical resources must be rendered into benchmark- +convention running text (e.g. lexically-guided text selection from +the corpus, not paradigm tables). + +The lineage: r5→r6→r7 all-positive; r9 first negative. Teacher-side +levers are NOT closed (r7's news mix remains the proven axis), but +this instantiation is dead: no re-run, no dose tuning, no convention +normalization retry — recorded and closed. ## Gates / risks diff --git a/docs/RESULTS.md b/docs/RESULTS.md index de150ce..4c1a861 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1002,3 +1002,26 @@ Sinkhorn embeddings, engram memory, and representation displacement (the only arm with a measured mechanistic premise) have all been run to verdict. The frontier mover remains teacher-side data only — run-009-yallamorph (TODO.sota-2026/01) is the active lever. + +## run-009-yallamorph teacher — NEGATIVE on both surfaces; r7 stays canonical (2026-10-02) + +The 2026-09-30 sweep's identified teacher-side lever (YallaMorph/ +CamelMorph morphological paradigm aux, 25% dose, r7-init) measured +**worse on both surfaces**: SadeedDiac-25 windowed zero-skip DER +2.4895 (r7: 2.2864 — gate bar 2.389 failed) and WikiNews-2024 +multi-ref 17.4265/12.1093 WER/DER (r7: 17.3794/11.8273) — no ID +improvement and no OOD trade. **r7 remains canonical**; run-009 is +recorded as the lineage's first negative teacher-side result. + +Mechanism, data-backed: the paradigm corpus's vocalization convention +is 1.5–2.0× denser than benchmark text (fatha 43.0 vs 28.4, damma +11.9 vs 7.1, shadda 8.6 vs 4.3 marks per 100 letters; tanwīn nearly +absent) and its forms are isolated words rather than running text. +At 25% dose the aux stream injected convention drift — the teacher +over-marks the plain stream — instead of morphology. The knowledge- +injection template survives with a sharper edge: r6's aux worked as +running text in benchmark convention; delivery vehicle matters as +much as the knowledge. Paradigm-table aux is closed (no dose tuning, +no convention-normalization retry recorded as future work; a lexical +lever, if ever revisited, must be rendered into benchmark-convention +running text). The frontier mover on record remains the r7 news mix. diff --git a/docs/paper.adoc b/docs/paper.adoc index f839c49..2686656 100644 --- a/docs/paper.adoc +++ b/docs/paper.adoc @@ -303,7 +303,7 @@ zip through the runtime itself — sha-pinned to the index) agree on re-derives exactly by the same tooling from the published prediction files, which is how the discrepancy was caught. -Two structural findings sit on it, one now scoped by a cross-lingual counterexample. First, *pretrained width is load-bearing*: SVD width-stitching of the pretrained ByT5-small fails at both tested ratios (3.8× narrow: 82.96 — worse than the from-scratch collapse; 2× narrow: 78.23). *Depth, by contrast, was compressible under our Arabic recipe*: a verbatim layer copy of encoder layers 12→6 trained to 5.78 full-set — a shippable rung at 63% of the parameters whose 1.21pp depth cost at 6 epochs is CI-separated from the full-depth peer. The Hebrew replication of that depth cut, single-variable against its own full-depth lineage (logit-KD recipe), collapsed instead: 77.48 DER versus the full-depth 30.38 (+53.77pp [51.64, 55.92]) despite normal training convergence. Depth-compressibility is therefore *not* a universal property of pretrained ByT5-small — it held under sequence-KD with Muon on Arabic and catastrophically failed under logit-KD on Hebrew; whether the boundary is the distillation regime or the language is open. Width surgery destroys the pretrained representation; depth surgery spends it — but only where the training regime lets it. Second, the epochs lever is real but secondary: doubling 3→6 epochs moves the rung 4.82→4.57 (−0.25pp) — most of the residual is not undertraining. Two pre-registered causal tests closed the domain-coverage attribution. The swap direction — 8k news-domain units replaced by classical-register Tashkeela at constant 30k budget — came back *negative* (5.81, −0.98pp vs control). The add direction — 48k total with the full cleaned Tashkeela corpus (5× classical coverage, all other levers held, 39,018 steps) — came back *flat-negative*: 4.8231 full-set, delta vs teacher 2.3717 [2.194, 2.554], statistically indistinguishable from the 2.1 rung (4.5701 [1.91, 2.35]) with the point estimate 0.25pp worse. A third lever, on-policy distillation (GKD — training on student-generated mistakes scored by the teacher), also came back negative: 6.0036 [3.109, 3.743], 1.43pp *worse* than the off-policy rung it was meant to improve. The residual has now resisted every lever tested — corpus scale, register mix, on-policy correction, memory layers (real but 0.70pp), epochs (0.25pp), and two optimizer-recipe arms imported from the 2025 frontier-LLM literature — and we report it as a property of the compression itself rather than a shortfall of any single method. The optimizer-recipe arms are instructive because they transfer negatively at our scale: head-wise Muon on Q/K projections (reported positive at 671B scale in DeepSeek-V4.1-Flash) scores 4.8164, separated-worse than its vanilla peer by +0.2267pp [0.016, 0.411] under a paired between-students bootstrap on identical data; the Sinkhorn-balanced embedding update is statistically indistinguishable from AdamW (−0.0903pp [−0.267, 0.069]). One measurement caveat travels with all such arm verdicts: each is a single training seed, and a recent small-model distillation audit shows per-seed variance large enough to swallow sub-point deltas — with bimodal collapse in some KD variants — so our paired bootstrap's protection extends to prediction resampling but not the seed axis (Sumit et al. 2026, arXiv 2608.27729); future arms run multi-seed or carry this caveat. Both directions of the classical-corpus lever fail: the residual is *not* a classical-domain coverage deficit, and it is not an optimizer artifact. A final arm tested the residual in representation space, on the one mechanism whose premise we could measure before spending training compute: the teacher's news-domain fine-tune shifts its encoder representations in a direction that is domain-general in early and mid layers (cosine 0.59–0.95 between the shift measured on classical vs news text, layers 0–8), so we regressed the student's encoder hidden states toward ridge-projected, extrapolated teacher targets along that direction — and it was the *most* harmful lever of all (5.8627, +1.3926pp [1.156, 1.646] worse than its vanilla peer), despite healthy training and a pre-registered loss budget. A domain-general direction is necessary but not sufficient: pulling a 300M byte student's representations toward projected 580M targets displaces what its decoder relies on. With that, the residual is closed on evidence rather than exhaustion. It reframes as a teacher–student interaction the corpus cannot reach — the remaining lever on our record is teacher-side: every frontier move in this table's upper rows came from the teacher's data, not the student's training. The next teacher rung is in flight on exactly that axis: systematic morphological paradigm coverage from the YallaMorph/CamelMorph resource (Reda et al. 2026, arXiv 2609.10153 — 663,804 controlled morphological-generation instances) added as an auxiliary stream to the news-domain mix that produced the current rung. +Two structural findings sit on it, one now scoped by a cross-lingual counterexample. First, *pretrained width is load-bearing*: SVD width-stitching of the pretrained ByT5-small fails at both tested ratios (3.8× narrow: 82.96 — worse than the from-scratch collapse; 2× narrow: 78.23). *Depth, by contrast, was compressible under our Arabic recipe*: a verbatim layer copy of encoder layers 12→6 trained to 5.78 full-set — a shippable rung at 63% of the parameters whose 1.21pp depth cost at 6 epochs is CI-separated from the full-depth peer. The Hebrew replication of that depth cut, single-variable against its own full-depth lineage (logit-KD recipe), collapsed instead: 77.48 DER versus the full-depth 30.38 (+53.77pp [51.64, 55.92]) despite normal training convergence. Depth-compressibility is therefore *not* a universal property of pretrained ByT5-small — it held under sequence-KD with Muon on Arabic and catastrophically failed under logit-KD on Hebrew; whether the boundary is the distillation regime or the language is open. Width surgery destroys the pretrained representation; depth surgery spends it — but only where the training regime lets it. Second, the epochs lever is real but secondary: doubling 3→6 epochs moves the rung 4.82→4.57 (−0.25pp) — most of the residual is not undertraining. Two pre-registered causal tests closed the domain-coverage attribution. The swap direction — 8k news-domain units replaced by classical-register Tashkeela at constant 30k budget — came back *negative* (5.81, −0.98pp vs control). The add direction — 48k total with the full cleaned Tashkeela corpus (5× classical coverage, all other levers held, 39,018 steps) — came back *flat-negative*: 4.8231 full-set, delta vs teacher 2.3717 [2.194, 2.554], statistically indistinguishable from the 2.1 rung (4.5701 [1.91, 2.35]) with the point estimate 0.25pp worse. A third lever, on-policy distillation (GKD — training on student-generated mistakes scored by the teacher), also came back negative: 6.0036 [3.109, 3.743], 1.43pp *worse* than the off-policy rung it was meant to improve. The residual has now resisted every lever tested — corpus scale, register mix, on-policy correction, memory layers (real but 0.70pp), epochs (0.25pp), and two optimizer-recipe arms imported from the 2025 frontier-LLM literature — and we report it as a property of the compression itself rather than a shortfall of any single method. The optimizer-recipe arms are instructive because they transfer negatively at our scale: head-wise Muon on Q/K projections (reported positive at 671B scale in DeepSeek-V4.1-Flash) scores 4.8164, separated-worse than its vanilla peer by +0.2267pp [0.016, 0.411] under a paired between-students bootstrap on identical data; the Sinkhorn-balanced embedding update is statistically indistinguishable from AdamW (−0.0903pp [−0.267, 0.069]). One measurement caveat travels with all such arm verdicts: each is a single training seed, and a recent small-model distillation audit shows per-seed variance large enough to swallow sub-point deltas — with bimodal collapse in some KD variants — so our paired bootstrap's protection extends to prediction resampling but not the seed axis (Sumit et al. 2026, arXiv 2608.27729); future arms run multi-seed or carry this caveat. Both directions of the classical-corpus lever fail: the residual is *not* a classical-domain coverage deficit, and it is not an optimizer artifact. A final arm tested the residual in representation space, on the one mechanism whose premise we could measure before spending training compute: the teacher's news-domain fine-tune shifts its encoder representations in a direction that is domain-general in early and mid layers (cosine 0.59–0.95 between the shift measured on classical vs news text, layers 0–8), so we regressed the student's encoder hidden states toward ridge-projected, extrapolated teacher targets along that direction — and it was the *most* harmful lever of all (5.8627, +1.3926pp [1.156, 1.646] worse than its vanilla peer), despite healthy training and a pre-registered loss budget. A domain-general direction is necessary but not sufficient: pulling a 300M byte student's representations toward projected 580M targets displaces what its decoder relies on. With that, the residual is closed on evidence rather than exhaustion. It reframes as a teacher–student interaction the corpus cannot reach — the remaining lever on our record is teacher-side: every frontier move in this table's upper rows came from the teacher's data, not the student's training. That axis is not uniformly fertile: the next teacher rung — systematic morphological paradigm coverage from the YallaMorph/CamelMorph resource (Reda et al. 2026, arXiv 2609.10153) added as a 25% auxiliary stream to the news-domain mix — measured *worse* on both surfaces (in-domain 2.4895 vs 2.2864; out-of-domain 17.43/12.11 vs 17.38/11.83), the lineage's first negative teacher-side result. The mechanism is measurable: the paradigm corpus vocalizes 1.5–2.0× more densely than benchmark text (shadda 2.0×, damma 1.7×, tanwīn nearly absent), so a 25% dose injects convention drift rather than morphology. The knowledge-injection lever survives with a sharper edge — the r6 auxiliary stream worked as running text in benchmark convention; paradigm tables are out-of-context and differently-conventioned, and the delivery vehicle matters as much as the knowledge. == The decode protocol is part of the measurement [[section-decode]]