feat: add fleet.judge module for LLM-as-a-judge grading (ENG-1527)

[jarredFleetSo]

Summary

• Creates the fleet/judge/ package that provides LLM-as-a-judge grading capability, enabling verifiers to use from fleet.judge import Rubric, Criterion as designed
• Implements standalone judge grading via configurable LLM endpoints (Anthropic or OpenAI-compatible), enabling offline/airgapped Docker Worlds evaluation
• Mirrors the orchestrator's prompt construction logic so judge grading works entirely client-side without calling the Fleet orchestrator

New Files

File	Purpose
fleet/judge/__init__.py	Public exports: Rubric, Criterion, JudgeClient, JudgeEndpointConfig, JudgeResult
fleet/judge/models.py	Data classes — Criterion(name, max, levels), Rubric(criteria), CriterionResult, JudgeResult (with __float__), JudgeEndpointConfig
fleet/judge/client.py	JudgeClient with async grade() + sync grade_sync(), Anthropic/OpenAI backends via raw httpx, JSON extraction with trailing comma repair
fleet/judge/prompt.py	build_structured_rubric_prompt() mirroring orchestrator logic
tests/test_judge.py	36 tests covering all models, JSON extraction, prompt construction, env var config, response parsing

Configuration

# Docker Worlds / airgapped — configure via env vars
JUDGE_ENDPOINT=https://customer-llm.internal/v1
JUDGE_API_KEY=sk-...
JUDGE_API_FORMAT=openai
JUDGE_MODEL=gpt-4o

Key Design Decisions

• Plain classes (not Pydantic) — Criterion("name", max=10, levels={...}) matches verifier usage exactly
• No new dependencies — uses only httpx (already a dep) for HTTP calls
• Config priority: explicit config → JUDGE_* env vars → ANTHROPIC_API_KEY → helpful error
• float(result) returns normalized score (0.0-1.0) for verifier compatibility

Related

• Orchestrator PR: fleet-ai/theseus#4078 — pluggable LLM endpoint in judge service
• Existing PR: feat: pluggable LLM provider interface for judge #78 — earlier attempt at provider abstraction (single-file approach)
• Issue: ENG-1527

Test plan

• 36 unit tests passing
• Verify from fleet.judge import Rubric, Criterion works in verifier context
• Integration test with actual LLM endpoint

🤖 Generated with Claude Code

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6bdb38e. Configure here.}

[cursor]

JudgeResult lacks .score attribute for verifier compatibility

High Severity

JudgeResult is designed for "verifier compatibility" via __float__, but the verifier framework's local execution path checks hasattr(result, "score") before ever calling float(). Since JudgeResult only has normalized_score, total_score, and max_score — but no .score attribute — the verifier raises a ValueError, which is silently caught, returning 0.0. A verifier returning a JudgeResult directly will always silently produce a zero score in local execution.

 

^{Reviewed by Cursor Bugbot for commit 6bdb38e. Configure here.}

[cursor]

Unused rubric parameter in _build_user_message

Low Severity

_build_user_message accepts a rubric parameter that is never referenced in the function body. The caller build_structured_rubric_prompt explicitly passes rubric=rubric, but the function only uses submission, ground_truth, problem, context, and conversation. This is either dead code or indicates missing logic that was intended to include rubric information in the user message.

 

^{Reviewed by Cursor Bugbot for commit 6bdb38e. Configure here.}