Skip to content

Broaden synthetic Jev with interpretive reading tasks - #23

Merged
sileod merged 1 commit into
mainfrom
interpretive-synthetic-data
Oct 1, 2026
Merged

sileod merged 1 commit into
mainfrom
interpretive-synthetic-data

Conversation

@sileod

@sileod sileod commented Sep 29, 2026

Copy link
Copy Markdown
Owner

What changed

  • Add 14 low-weight general reading skills to the existing 36-domain synthetic sampler, with no benchmark labels or special generation path. Arithmetic, chronology, entity lookup, and literal retrieval remain in the procedural segment.
  • Keep general questions near 10% of the deterministic 4,000-spec draw: 606 of 5,890 questions, with every general skill present in at least 19 domains.
  • Strengthen the DeepSeek critic with a per-question exact evidence quote and checks for skill match, support, and one best answer. Rename prompt files to stable names without v2.
  • Correct metadata-leak validation so ordinary words such as “evidence” and “difficulty” are accepted.

Live check

A natural 120-state Albert run produced 117 validated states, 94 critic-approved states, and 119 Jev decisions in 582 seconds with one Albert key. The independent auditor answered 118/119 questions, disagreed on 12 overall and one of the 16 retained general questions, and reported no confident disagreements. The short-run rate extrapolates to about 17,700 retained decisions/day; production rate limits and sustained quota may change that.

Manual reading found skill drift and unsupported inferences that Jev/auditor agreement did not catch. A focused 16-state probe drove tighter skill instructions and the critic evidence checks. The stricter critic rejected an ambiguous negotiation implication, but missed a prediction about tomorrow based only on today's idle crew. The pilot note documents the remaining quality risk. This PR broadens the candidate distribution and improves screening; it does not certify an unattended 4,000-state export as clean training data.

Verification

  • PYTHONPATH=.:src pytest -q: 217 passed
  • git diff --check: clean
  • Live pilot and focused probe artifacts under ignored .synthetic_runs/

@sileod
sileod merged commit 2d8714b into main Oct 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant