Intelligence Gating Suite

A compact, shareable benchmark package for checking whether a local or hosted LLM is good enough at agentic tool use, a small amount of programming, and image understanding before you rely on it.

Scope and Intent (please read first)

This is a personal, opinionated probe, shared for reference only — not a comprehensive or authoritative benchmark. The author built it to test models in the environments they personally might actually use, with a deliberately narrow focus:

• Tool calling — the primary target: strict single-tool selection, multi-step tool chains, error recovery, argument boundaries, and no-tool cases.
• A little programming — polluted-context program rewrites and loop/degeneration resistance, not a coding benchmark.
• Image understanding — schema-strict visual extraction, alias-aware semantic grounding, and flexible reverse-prompt description.

It is not trying to measure general intelligence, knowledge, math, or creative writing. A small case count (51 total) means single-run differences of a few cases carry no statistical weight — treat results as a directional signal, not a leaderboard. Use it, fork it, change the cases to fit your own environment.

This package is generated from a larger private research workspace by build_public_suite.py. It intentionally includes only sanitized prompt suites, reviewed image assets, deterministic evaluators, and lightweight runners. It contains no model outputs, run artifacts, local logs, databases, model weights, API keys, or private payloads.

What Is Included

• raw/prompts/agentic-gate-v1/suite.json: active text agentic gate, agentic-gate-v1@0.2.2, 35 cases.
• raw/prompts/agentic-vision-schema-v1/cases.json: 5 strict schema-visible multimodal cases.
• raw/prompts/agentic-vision-semantic-v1/cases.json: 5 alias-aware semantic multimodal cases.
• raw/prompts/vision-reverse-prompt-v0/cases.json: 6 flexible image prompt-reconstruction cases.
• raw/assets/: generated UI/OCR/chart/photo/painting images used by the vision suites.
• scripts/evaluate_agentic_gate.py: deterministic text-gate evaluator.
• scripts/run_agentic_gate_retry.py: OpenAI-compatible text-gate runner with deterministic retry.
• scripts/run_vision_gate.py: OpenAI-compatible multimodal runner and deterministic scorer.
• scripts/run_codex_agentic_gate.py and scripts/run_codex_vision_gate.py: optional Codex CLI adapters.
• CASE_INDEX.md: case list and lane mapping.
• MANIFEST.json: package metadata and SHA-256 checksums.

The current package total is 35 + 10 + 6 = 51 cases.

Quick Validation

Run the deterministic self-test:

python3 scripts/evaluate_agentic_gate.py \
  --suite raw/prompts/agentic-gate-v1/suite.json \
  --self-test \
  --out tmp/self-test.json

Dry-run the multimodal suite loading and asset resolution:

python3 scripts/run_vision_gate.py \
  --suite raw/prompts/agentic-vision-schema-v1/cases.json \
  --asset-root . \
  --model dry-run \
  --dry-run

python3 scripts/run_vision_gate.py \
  --suite raw/prompts/agentic-vision-semantic-v1/cases.json \
  --asset-root . \
  --model dry-run \
  --dry-run

python3 scripts/run_vision_gate.py \
  --suite raw/prompts/vision-reverse-prompt-v0/cases.json \
  --asset-root . \
  --model dry-run \
  --dry-run

Running Against an OpenAI-Compatible Server

Text gate:

python3 scripts/run_agentic_gate_retry.py \
  --base-url http://127.0.0.1:8000/v1 \
  --api-key EMPTY \
  --model YOUR_MODEL_ID \
  --suite raw/prompts/agentic-gate-v1/suite.json \
  --out-root raw/runs/local \
  --label YOUR_MODEL-agentic-gate-full35 \
  --timeout 2400 \
  --max-retries 3 \
  --max-tokens 2048 \
  --write-done

Vision schema plus semantic lanes:

python3 scripts/run_vision_gate.py \
  --base-url http://127.0.0.1:8000/v1 \
  --api-key EMPTY \
  --model YOUR_MODEL_ID \
  --suite raw/prompts/agentic-vision-schema-v1/cases.json \
  --asset-root . \
  --out-root raw/runs/local \
  --label YOUR_MODEL-vision-schema-v1 \
  --write-done

python3 scripts/run_vision_gate.py \
  --base-url http://127.0.0.1:8000/v1 \
  --api-key EMPTY \
  --model YOUR_MODEL_ID \
  --suite raw/prompts/agentic-vision-semantic-v1/cases.json \
  --asset-root . \
  --out-root raw/runs/local \
  --label YOUR_MODEL-vision-semantic-v1 \
  --write-done

Reverse-prompt lane:

python3 scripts/run_vision_gate.py \
  --base-url http://127.0.0.1:8000/v1 \
  --api-key EMPTY \
  --model YOUR_MODEL_ID \
  --suite raw/prompts/vision-reverse-prompt-v0/cases.json \
  --asset-root . \
  --out-root raw/runs/local \
  --label YOUR_MODEL-vision-reverse-prompt-v0 \
  --max-tokens 1024 \
  --write-done

Scoring Conventions

• Text cases pass via the deterministic evaluator; retries are deterministic with score caps, and per-case results record clean versus recovered passes and hard failures.
• Vision schema cases require strict JSON with exact leaf matching; semantic cases require strict JSON validity with alias-aware fact matching; reverse-prompt cases are scored with flexible must-capture matching and are also suitable for LLM-judge or human review.
• Request errors are infrastructure results, not model scores.

Package Integrity

MANIFEST.json lists SHA-256 checksums for every file in the package. Regenerate or audit with the checksums after download.

License

MIT — see LICENSE. The prompt cases, image assets, and scripts are all covered. Attribution is appreciated but not required.

Package version: 2026-06-13 (generated 2026-06-13T02:20:50.897221+00:00).