Kestrel

One agent, eight kinds of task.
Checked, not just generated.

---

Kestrel handles all eight task categories in a single pipeline — factual questions, math, sentiment, summarization, entity extraction, code debugging, logic puzzles, and code generation.

Two ideas do most of the work:

• A large share of the tasks never leave the container. A quantized 4B model ships inside the image and runs on CPU. It was benchmarked category by category and given only the work it genuinely handles well, so those tasks cost nothing at all.
• Answers are checked with code, not with another model. Generated code is executed against the examples in the prompt. Length and format constraints are parsed out of the prompt and counted in Python. An interpreter is a better judge of whether code works than the model that wrote it — and no language model reliably counts its own output.

Submitted image: ghcr.io/py-tr/kestrel:v80-a82 (profile A82)

How it works

1. Reads /input/tasks.json on startup.

2. A deterministic router (app/router.py) classifies each task by pattern matching — no model call, no cost, and it cannot hallucinate a category. It also parses out the task's stated output requirements.

3. Routing. The category decides whether the task leaves the container at all:

   	categories
   Local — baked-in 4B, zero API cost	factual, sentiment, code_gen
   Remote — model chosen per category	math, logic, summarization, ner, code_debug

   Remote calls use a minimal prompt suffix, a tight per-category max_tokens, and reasoning disabled. ner requests are batched. If a model is unavailable, the call falls back to another permitted one.

4. Verification.

   • Code — generated code is executed against examples extracted from the prompt (app/solvers/exec_verify.py). On failure it goes back to the model once, with the failing case attached.
   • Format — when a task states a length or shape ("in exactly two sentences"), the requirement is parsed and the output is counted. A non-compliant answer is rewritten.
   • Math — deterministic zero-token templates (app/solvers/math_templates.py) answer only where a prompt matches one unambiguous pattern. Any ambiguity returns None and the task goes remote. Precision over coverage: a wrong template answer costs accuracy, and accuracy is worth more than the tokens it saves.

5. Writes /output/results.json and exits 0.

A deadline guard guarantees valid results are always written, even when individual calls hang or fail.

Results

• 100% on the graded task set.
• 31/32 on a held-out set written from scratch — genuinely new tasks, not rewordings of the originals. That second number is the harder test, and the one that shows the agent generalizes rather than memorizes.

Run

docker run --rm --cpus=2 --memory=3.8g \
  -v "$PWD/input:/input:ro" -v "$PWD/output:/output" \
  -e FIREWORKS_API_KEY=... \
  -e ALLOWED_MODELS=... \
  ghcr.io/py-tr/kestrel:v80-a82

Credentials, base URL, and the permitted model list are read from the environment at run time. Nothing is downloaded while running — the model ships inside the image.

Build

Dockerfile.local is the one that builds the submitted agent. It bakes the GGUF model into the image, and is what v80-a82 was built from. Place the model at model.gguf in the repo root first.

docker build -f Dockerfile.local --build-arg PROFILE=A82 -t kestrel:local .

The plain Dockerfile bakes no model and produces a remote-only agent. It is kept for measuring the API path in isolation — it is not the submission.

Develop

pip install -r requirements.txt

python evalkit/smoke.py A82    # end-to-end against a local mock proxy
python evalkit/chaos.py        # failure modes: dead endpoints, malformed replies,
                               # oversized and adversarial inputs
python evalkit/probe.py        # measure real API behavior (needs .env)

Behaviour is entirely config-driven. Profiles live in config.json; the active one is baked at build time (--build-arg PROFILE=...) and selected at run time via AGENT_PROFILE.

Because profiles are additive, the tree contains modules a given profile may not enable. A82 — the submitted profile — does not use app/solvers/pot.py, gates.py, summ_check.py, or logic_gate.py; those back experimental profiles that were built and measured, but not shipped. The features block of the active profile in config.json is the authoritative statement of what actually runs.

License

MIT