Skip to content

Harden the scoring harness: switch-point metrics and a normalisation audit #11

Description

@Awshesh12

eval/score.py computes corpus-pooled MER, ScriptAcc, JA-CER/WER and EN-WER, with 6 unit tests. It is sound for what it does, but it reports only aggregates — which hides exactly the behaviour the paper is about.

Switch-point analysis

Every metric is pooled over whole utterances. The interesting question is what happens at the boundary. Worth adding:

  • Error rate as a function of distance from a switch point (do errors cluster at boundaries?)
  • ScriptAcc split by switch type — intra-sentential vs inter-sentential, and by embedded-span length (single word vs phrase)
  • Whether a katakana substitution predicts further errors downstream in the same utterance

This would turn §7's aggregate claims into a mechanistic account of why encoder LoRA helps, which is currently asserted rather than shown.

Normalisation audit

ScriptAcc depends on alignment between reference and hypothesis English words. Worth documenting and testing the edge cases:

  • Words dropped entirely vs. transliterated — currently both count against ScriptAcc, but they are different failures and arguably deserve separate reporting
  • Partial transliteration (meetingミーティング会議)
  • Romaji output — Latin script but not English (ミーティングmiitingu would score as correct today, which is wrong)
  • Digit/word stylisation (92% vs ninety-two percent), which the dissertation found accounted for an entire EN-CER figure

Done when

The harness reports switch-point-stratified metrics, and the romaji false-positive case is either handled or documented as a known limitation in the paper's methodology section.

Metadata

Metadata

Assignees

Labels

evaluationEvaluation harness, metrics, benchmarks

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions