Skip to content

Broaden synthetic Jev generation and support dual Albert keys - #22

Merged
sileod merged 1 commit into
mainfrom
reduce-graded-synthetic-share
Sep 29, 2026
Merged

sileod merged 1 commit into
mainfrom
reduce-graded-synthetic-share

Conversation

@sileod

@sileod sileod commented Sep 29, 2026

Copy link
Copy Markdown
Owner

Why

The first audited synthetic pilots overproduced subjective scores and revealed malformed yes/no questions, rubric mismatches, and choice questions with multiple defensible answers. The user asked for broad, simpler data rather than benchmark-specific examples. A second Albert credential was provided to test optional load balancing.

Changes

  • Reduce expected score share to about 15% in the audited and 4,000-state configs; keep all 36 domains, 24 skills, and all difficulty/ambiguity levels.
  • Add configurable difficulty, ambiguity, and distractor weights. Pair named score rubrics with compatible skills.
  • Tighten generation, deterministic validation, and the Albert critic around explicit decision standards, single-proposition yes/no questions, and one defensible choice answer.
  • Rotate generation and Albert critic calls across distinct environment-supplied keys when ALBERT_API_KEY_2 is present. No key value enters config, manifest, or raw response artifacts; raw generation records only a slot number.
  • Record pilot findings in docs/jev/synthetic-pilot-2026-09-29.md.

Evidence

  • 4,000 deterministic specs: 5,944 decisions, 45.6% choice, 39.2% noul, 15.3% score; all 36 domains and 24 skills.
  • Live revised pilot: 24 generated, 23 structurally valid, 19 critic-passing states, 22 real Jev decisions; 3 auditor disagreements. The one confident flag appears to be an auditor error, so flags remain diagnostic.
  • Two-key generation: 32 states in 69 s (16 per key slot), versus 89 s with one key; neither short run hit a visible rate limit.
  • PYTHONPATH=.:src pytest -q: 215 passed.

Limits

The live pilots are small. The speed comparison does not establish separate sustained quotas, and the retained corpus still needs manual semantic review before being called clean supervision. The second key was used transiently; production needs ALBERT_API_KEY_2 supplied in its environment.

@sileod
sileod merged commit 07ad61b into main Sep 29, 2026
2 checks passed
@sileod
sileod deleted the reduce-graded-synthetic-share branch September 29, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant