Skip to content

Fix Jev task questions, ordinal typing, and extreme request lengths - #20

Merged
sileod merged 8 commits into
mainfrom
audit-jev-data-quality
Sep 25, 2026
Merged

sileod merged 8 commits into
mainfrom
audit-jev-data-quality

Conversation

@sileod

@sileod sileod commented Sep 25, 2026

Copy link
Copy Markdown
Owner

Summary

Audit-driven fixes for tasksource-jev-typed-decisions:

  • give ambiguous classification families explicit task questions instead of the generic criterion prompt
    • CrowdFlower configs, including economic-news and tweet_global_warming
    • TuringBench, I2D2, args_me, Touche23-ValueEval, starcon, subjectivity
    • MOH, VUAC, TroFi
    • AmbigNQ
  • support config-specific question metadata in shared Tasksource mappings
  • mark XGLUE WPR as ordinal so its Bad → Fair → Good → Excellent → Perfect scale can emit score examples
  • filter pathological sequence lengths on the complete rendered Jev request
    • default 128 KiB UTF-8 cap
    • optional exact token cap when a tokenizer is supplied
    • record per-split dropped-row counts in the build report
    • include the new limits in the shard fingerprint
  • add regressions for the question metadata, ordinal flag, build fingerprint, byte cap, and exact-token cap

Audit notes

The NusaX choice/score mixture was investigated and is intentional: ordinal tasks deliberately produce both representations, so it is unchanged.

Legacy mojibake / HTML noise is also unchanged. The Jev recast already has an explicit raw-text robustness path; this PR does not apply blanket text normalization.

No source labels are changed by this PR.

@sileod
sileod merged commit de3dd00 into main Sep 25, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant