Skip to content

Validate structured decisions and fix provider compatibility - #1

Merged
RodrigoAlbe merged 1 commit into
mainfrom
fix/validated-decisions
Sep 26, 2026
Merged

RodrigoAlbe merged 1 commit into
mainfrom
fix/validated-decisions

Conversation

@RodrigoAlbe

Copy link
Copy Markdown
Owner

Problem and behavior

Missing answers previously became negative decisions with full confidence, and pooled token logprobs could overwrite different batch answers with the same value. This change validates every answer locally and preserves the model's JSON decision. Invalid, refused, or truncated batches now fail explicitly.

Changes

  • Validate question definitions, exact answer fields, categorical membership, finite numeric bounds, and duplicate JSON keys. Export typed response/refusal/incomplete/transport errors.
  • Disable diagnostic logprobs by default. Preserve opted-in token data in raw_response, require a token position in extraction helpers, and identify confidence as model-reported. No inferred class distribution or calibrated-confidence claim.
  • Separate provider request/envelope handling. Send native schemas for supported configurations; otherwise include the schema in the prompt and validate locally. Avoid unsupported Groq logprob parameters and allow explicit response-mode overrides.
  • Give sync/async calls the same retry behavior, honor Retry-After, reuse the sync client within a retry cycle, and avoid sleeping after exhaustion.
  • Report unknown cost as None. Replace hard-coded benchmark comparisons with JSON reports, dataset hashes, measured latency/tokens/failures, and optional labeled accuracy/Brier metrics.
  • Update README, agent instructions, and plugin descriptions, including migration notes. Extend CI to Windows and distribution builds; include the benchmark module in the source distribution.

Validation

  • 166 tests passed on Python 3.9 with the minimum supported httpx==0.24.0.
  • 166 tests passed against the built wheel in an isolated environment with current dependencies.
  • Wheel and source distribution builds succeeded; inspected source distribution contents.
  • Ruff undefined/unused-name checks and git diff --check passed.
  • HTTP behavior is tested with httpx.MockTransport; no live inference calls or paid benchmarks were executed.

Compatibility and remaining work

Callers must handle explicit invalid-response errors and optional cost; logprobs no longer rewrite answers or confidence. Groq/Gemini diagnostic opt-in currently fails locally. max_retries retains its existing meaning of total attempts. Native schema support for custom models can be selected explicitly and must be verified against the endpoint.

The bundled benchmark scenarios are still unlabeled smoke examples. This PR adds support for independently labeled datasets; it does not claim empirical calibration, real-provider performance, or introduce automatic confidence-based abstention. No package release or merge is included.

@RodrigoAlbe
RodrigoAlbe marked this pull request as ready for review September 26, 2026 16:02
@RodrigoAlbe
RodrigoAlbe merged commit 8567575 into main Sep 26, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant