eval/score.py computes corpus-pooled MER, ScriptAcc, JA-CER/WER and EN-WER, with 6 unit tests. It is sound for what it does, but it reports only aggregates — which hides exactly the behaviour the paper is about.
Switch-point analysis
Every metric is pooled over whole utterances. The interesting question is what happens at the boundary. Worth adding:
- Error rate as a function of distance from a switch point (do errors cluster at boundaries?)
- ScriptAcc split by switch type — intra-sentential vs inter-sentential, and by embedded-span length (single word vs phrase)
- Whether a katakana substitution predicts further errors downstream in the same utterance
This would turn §7's aggregate claims into a mechanistic account of why encoder LoRA helps, which is currently asserted rather than shown.
Normalisation audit
ScriptAcc depends on alignment between reference and hypothesis English words. Worth documenting and testing the edge cases:
- Words dropped entirely vs. transliterated — currently both count against ScriptAcc, but they are different failures and arguably deserve separate reporting
- Partial transliteration (
meeting → ミーティング会議)
- Romaji output — Latin script but not English (
ミーティング → miitingu would score as correct today, which is wrong)
- Digit/word stylisation (
92% vs ninety-two percent), which the dissertation found accounted for an entire EN-CER figure
Done when
The harness reports switch-point-stratified metrics, and the romaji false-positive case is either handled or documented as a known limitation in the paper's methodology section.
eval/score.pycomputes corpus-pooled MER, ScriptAcc, JA-CER/WER and EN-WER, with 6 unit tests. It is sound for what it does, but it reports only aggregates — which hides exactly the behaviour the paper is about.Switch-point analysis
Every metric is pooled over whole utterances. The interesting question is what happens at the boundary. Worth adding:
This would turn §7's aggregate claims into a mechanistic account of why encoder LoRA helps, which is currently asserted rather than shown.
Normalisation audit
ScriptAcc depends on alignment between reference and hypothesis English words. Worth documenting and testing the edge cases:
meeting→ミーティング会議)ミーティング→miitinguwould score as correct today, which is wrong)92%vsninety-two percent), which the dissertation found accounted for an entire EN-CER figureDone when
The harness reports switch-point-stratified metrics, and the romaji false-positive case is either handled or documented as a known limitation in the paper's methodology section.