From eb9f36023538072c8440ba98a69422343077bce6 Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:45:07 +0800 Subject: [PATCH 01/14] docs: add benchmark methodology --- benchmarks/README.md | 133 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 benchmarks/README.md diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..5f08657 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,133 @@ +# CQO Benchmark Framework + +The benchmark exists to answer one question honestly: + +> Does Codex Quota Optimizer reduce avoidable agent work **while the task still passes the same acceptance criteria**? + +It is not a token-savings calculator and it does not scrape private OpenAI usage data. + +## What we measure + +Record only behavior you can directly observe from the session or tool output: + +- acceptance criteria passed / failed, +- files inspected, +- files changed, +- searches, +- focused checks, +- broad test-suite checks, +- subagents spawned, +- model escalations, +- repeated file reads, +- wall-clock time when available, +- model role / reasoning level used, +- notes about unusual conditions. + +Do **not** estimate hidden tokens, remaining quota, or percentage savings. + +If a client directly exposes token counts, keep them in the raw notes/source evidence rather than converting them into unsupported savings claims. + +## Two benchmark modes + +### 1. Controlled behavior benchmark + +Use the same starting model and reasoning setting for both variants. + +Purpose: isolate CQO's effects on: + +- context discipline, +- change-surface discipline, +- verification scope, +- subagent use, +- repeated exploration. + +### 2. Full-policy benchmark + +Let each variant use its normal policy. CQO may route to a cheaper model/reasoning role when appropriate. + +Purpose: evaluate the complete CQO workflow, including routing. + +Always record the actual model role/reasoning used so the comparison stays transparent. + +## Fair-run protocol + +For every public comparison: + +1. Start both variants from the **same repository commit**. +2. Use the **same task prompt and acceptance criteria**. +3. Use equivalent permissions/tool access. +4. Record a shared `pair_id` for the baseline and CQO run. +5. Do not reuse the modified working tree from the first run. +6. Prefer alternating run order across repeated pairs. +7. For public claims, use multiple pairs when practical; one pair is a case study, not a universal benchmark. +8. If either side fails acceptance, do not claim efficiency superiority from that pair. +9. Preserve raw notes or session evidence when publishing a case study. + +## Result format + +Store one JSON object per line: + +```json +{"case_id":"checkout-bug","pair_id":"p1","variant":"baseline","benchmark_mode":"controlled","repo_commit":"abc123","acceptance_passed":true,"model_role":"balanced","reasoning":"medium","files_inspected":9,"files_changed":2,"searches":5,"focused_checks":1,"broad_checks":1,"subagents":0,"model_escalations":0,"repeated_reads":2,"wall_seconds":180,"notes":"Example shape only; replace with observed data."} +``` + +Then record the CQO run with the same `case_id` and `pair_id`. + +### Required fields + +- `case_id` +- `pair_id` +- `variant`: `baseline` or `cqo` +- `benchmark_mode`: `controlled` or `full-policy` +- `acceptance_passed` + +### Optional numeric metrics + +- `files_inspected` +- `files_changed` +- `searches` +- `focused_checks` +- `broad_checks` +- `subagents` +- `model_escalations` +- `repeated_reads` +- `wall_seconds` + +Unknown values should be omitted rather than guessed. + +## Generate a report + +```bash +python benchmarks/report.py path/to/results.jsonl +``` + +Markdown output: + +```bash +python benchmarks/report.py path/to/results.jsonl --format markdown +``` + +JSON output for further analysis: + +```bash +python benchmarks/report.py path/to/results.jsonl --format json +``` + +The report only computes efficiency deltas for matched pairs where **both runs passed acceptance**. + +Negative deltas mean CQO used less of that observable resource. + +## Recommended first real-world cases + +Build the first public case set from real repositories and real tasks: + +1. **XS — documentation / mechanical patch** +2. **S — localized bug fix** +3. **M — small cross-file feature** +4. **L — debugging or migration task** + +Do not manufacture results. The repository intentionally ships the framework before publishing benchmark numbers. + +## Case study template + +Use [case-study-template.md](./case-study-template.md) when publishing a measured example. From cf39321d7209ae425cf87c5ee0123c4b24d26175 Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:45:08 +0800 Subject: [PATCH 02/14] chore: add benchmark result schema --- benchmarks/result.schema.json | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 benchmarks/result.schema.json diff --git a/benchmarks/result.schema.json b/benchmarks/result.schema.json new file mode 100644 index 0000000..2529163 --- /dev/null +++ b/benchmarks/result.schema.json @@ -0,0 +1,33 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "CQO benchmark result", + "type": "object", + "required": [ + "case_id", + "pair_id", + "variant", + "benchmark_mode", + "acceptance_passed" + ], + "properties": { + "case_id": {"type": "string", "minLength": 1}, + "pair_id": {"type": "string", "minLength": 1}, + "variant": {"enum": ["baseline", "cqo"]}, + "benchmark_mode": {"enum": ["controlled", "full-policy"]}, + "repo_commit": {"type": "string"}, + "acceptance_passed": {"type": "boolean"}, + "model_role": {"type": "string"}, + "reasoning": {"type": "string"}, + "files_inspected": {"type": "integer", "minimum": 0}, + "files_changed": {"type": "integer", "minimum": 0}, + "searches": {"type": "integer", "minimum": 0}, + "focused_checks": {"type": "integer", "minimum": 0}, + "broad_checks": {"type": "integer", "minimum": 0}, + "subagents": {"type": "integer", "minimum": 0}, + "model_escalations": {"type": "integer", "minimum": 0}, + "repeated_reads": {"type": "integer", "minimum": 0}, + "wall_seconds": {"type": "number", "minimum": 0}, + "notes": {"type": "string"} + }, + "additionalProperties": true +} From 8257c4346f9e91b9b6a50299316588e61fa1cfe9 Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:45:11 +0800 Subject: [PATCH 03/14] docs: add benchmark case study template --- benchmarks/case-study-template.md | 58 +++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 benchmarks/case-study-template.md diff --git a/benchmarks/case-study-template.md b/benchmarks/case-study-template.md new file mode 100644 index 0000000..f0da6f7 --- /dev/null +++ b/benchmarks/case-study-template.md @@ -0,0 +1,58 @@ +# CQO Case Study + +## Case + +- Repository: +- Commit: +- Task: +- Benchmark mode: Controlled / Full-policy +- Number of matched pairs: +- Run order: + +## Acceptance criteria + +- [ ] +- [ ] + +## Environment + +- Codex surface: +- Baseline model / reasoning: +- CQO model / reasoning: +- Permissions / tools: +- CQO version: + +## Results + +Only include matched pairs where both variants passed the same acceptance criteria. + +| Observable metric | Baseline | CQO | Delta | +|---|---:|---:|---:| +| Files inspected | | | | +| Files changed | | | | +| Searches | | | | +| Focused checks | | | | +| Broad checks | | | | +| Subagents | | | | +| Model escalations | | | | +| Repeated reads | | | | +| Wall seconds | | | | + +## What changed + +Describe concrete behavior differences. Avoid turning them into unsupported token or quota percentages. + +## Quality check + +- Baseline acceptance passed: +- CQO acceptance passed: +- Any regressions: +- Any extra work CQO introduced: + +## Evidence + +Link or attach raw session notes, command logs, or other reproducible evidence where appropriate. + +## Limitations + +State what this case does **not** prove. One case is not a universal savings claim. From d9e067da24f79a7a533439c6215c0add64012bee Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:45:39 +0800 Subject: [PATCH 04/14] feat: add paired benchmark report tool --- benchmarks/report.py | 279 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 279 insertions(+) create mode 100644 benchmarks/report.py diff --git a/benchmarks/report.py b/benchmarks/report.py new file mode 100644 index 0000000..9069975 --- /dev/null +++ b/benchmarks/report.py @@ -0,0 +1,279 @@ +#!/usr/bin/env python3 +"""Generate an honest CQO benchmark report from paired JSONL observations.""" + +from __future__ import annotations + +import argparse +import json +import statistics +import sys +from collections import defaultdict +from pathlib import Path +from typing import Any + +VARIANTS = {"baseline", "cqo"} +MODES = {"controlled", "full-policy"} +NUMERIC_METRICS = ( + "files_inspected", + "files_changed", + "searches", + "focused_checks", + "broad_checks", + "subagents", + "model_escalations", + "repeated_reads", + "wall_seconds", +) + + +class BenchmarkError(ValueError): + pass + + +def load_jsonl(path: Path) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + with path.open("r", encoding="utf-8") as f: + for lineno, line in enumerate(f, start=1): + raw = line.strip() + if not raw: + continue + try: + row = json.loads(raw) + except json.JSONDecodeError as exc: + raise BenchmarkError( + f"{path}:{lineno}: invalid JSON: {exc.msg}" + ) from exc + validate_row(row, path, lineno) + rows.append(row) + return rows + + +def validate_row(row: dict[str, Any], path: Path | None = None, lineno: int | None = None) -> None: + where = "" + if path is not None and lineno is not None: + where = f"{path}:{lineno}: " + + required = ("case_id", "pair_id", "variant", "benchmark_mode", "acceptance_passed") + missing = [key for key in required if key not in row] + if missing: + raise BenchmarkError(f"{where}missing required field(s): {', '.join(missing)}") + + if row["variant"] not in VARIANTS: + raise BenchmarkError(f"{where}variant must be baseline or cqo") + if row["benchmark_mode"] not in MODES: + raise BenchmarkError(f"{where}benchmark_mode must be controlled or full-policy") + if not isinstance(row["acceptance_passed"], bool): + raise BenchmarkError(f"{where}acceptance_passed must be boolean") + + for metric in NUMERIC_METRICS: + if metric not in row: + continue + value = row[metric] + if not isinstance(value, (int, float)) or isinstance(value, bool) or value < 0: + raise BenchmarkError(f"{where}{metric} must be a non-negative number") + + +def pair_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: + grouped: dict[tuple[str, str, str], dict[str, dict[str, Any]]] = defaultdict(dict) + for row in rows: + key = (row["case_id"], row["pair_id"], row["benchmark_mode"]) + variant = row["variant"] + if variant in grouped[key]: + raise BenchmarkError( + f"duplicate {variant} record for case={key[0]} pair={key[1]} mode={key[2]}" + ) + grouped[key][variant] = row + + pairs: list[dict[str, Any]] = [] + for (case_id, pair_id, mode), variants in sorted(grouped.items()): + baseline = variants.get("baseline") + cqo = variants.get("cqo") + matched = baseline is not None and cqo is not None + acceptance_eligible = bool( + matched + and baseline["acceptance_passed"] + and cqo["acceptance_passed"] + ) + pairs.append( + { + "case_id": case_id, + "pair_id": pair_id, + "benchmark_mode": mode, + "baseline": baseline, + "cqo": cqo, + "matched": matched, + "acceptance_eligible": acceptance_eligible, + } + ) + return pairs + + +def metric_delta(pair: dict[str, Any], metric: str) -> float | int | None: + if not pair["acceptance_eligible"]: + return None + baseline = pair["baseline"] + cqo = pair["cqo"] + if metric not in baseline or metric not in cqo: + return None + return cqo[metric] - baseline[metric] + + +def summarize(rows: list[dict[str, Any]]) -> dict[str, Any]: + pairs = pair_rows(rows) + eligible = [pair for pair in pairs if pair["acceptance_eligible"]] + + metrics: dict[str, Any] = {} + for metric in NUMERIC_METRICS: + deltas = [ + delta + for pair in eligible + if (delta := metric_delta(pair, metric)) is not None + ] + metrics[metric] = { + "paired_observations": len(deltas), + "median_delta": statistics.median(deltas) if deltas else None, + "deltas": deltas, + } + + baseline_total = sum(1 for row in rows if row["variant"] == "baseline") + cqo_total = sum(1 for row in rows if row["variant"] == "cqo") + baseline_pass = sum( + 1 for row in rows if row["variant"] == "baseline" and row["acceptance_passed"] + ) + cqo_pass = sum( + 1 for row in rows if row["variant"] == "cqo" and row["acceptance_passed"] + ) + + return { + "records": len(rows), + "pairs_total": len(pairs), + "pairs_matched": sum(1 for pair in pairs if pair["matched"]), + "pairs_acceptance_eligible": len(eligible), + "baseline": {"runs": baseline_total, "passed": baseline_pass}, + "cqo": {"runs": cqo_total, "passed": cqo_pass}, + "metrics": metrics, + "pairs": pairs, + "interpretation": ( + "Negative metric deltas mean CQO used less of that observed resource. " + "Efficiency deltas are excluded unless both variants passed acceptance." + ), + } + + +def format_number(value: Any) -> str: + if value is None: + return "n/a" + if isinstance(value, float) and not value.is_integer(): + return f"{value:.2f}" + return str(int(value)) + + +def markdown_report(summary: dict[str, Any]) -> str: + lines = [ + "# CQO Benchmark Report", + "", + f"- Records: {summary['records']}", + f"- Matched pairs: {summary['pairs_matched']} / {summary['pairs_total']}", + f"- Acceptance-eligible pairs: {summary['pairs_acceptance_eligible']}", + f"- Baseline acceptance: {summary['baseline']['passed']} / {summary['baseline']['runs']}", + f"- CQO acceptance: {summary['cqo']['passed']} / {summary['cqo']['runs']}", + "", + "> Negative deltas mean CQO used less of that observable resource. " + "No efficiency delta is counted unless both runs passed acceptance.", + "", + "| Metric | Paired observations | Median CQO − baseline |", + "|---|---:|---:|", + ] + for metric in NUMERIC_METRICS: + item = summary["metrics"][metric] + lines.append( + f"| {metric.replace('_', ' ')} | {item['paired_observations']} | " + f"{format_number(item['median_delta'])} |" + ) + + lines.extend( + [ + "", + "## Pair status", + "", + "| Case | Pair | Mode | Baseline pass | CQO pass | Included in efficiency deltas |", + "|---|---|---|---|---|---|", + ] + ) + for pair in summary["pairs"]: + baseline = pair["baseline"] + cqo = pair["cqo"] + lines.append( + "| " + + " | ".join( + [ + str(pair["case_id"]), + str(pair["pair_id"]), + str(pair["benchmark_mode"]), + "yes" if baseline and baseline["acceptance_passed"] else ("no" if baseline else "missing"), + "yes" if cqo and cqo["acceptance_passed"] else ("no" if cqo else "missing"), + "yes" if pair["acceptance_eligible"] else "no", + ] + ) + + " |" + ) + + return "\n".join(lines) + "\n" + + +def text_report(summary: dict[str, Any]) -> str: + lines = [ + "CQO Benchmark Report", + f"Matched pairs: {summary['pairs_matched']} / {summary['pairs_total']}", + f"Acceptance-eligible pairs: {summary['pairs_acceptance_eligible']}", + f"Baseline acceptance: {summary['baseline']['passed']} / {summary['baseline']['runs']}", + f"CQO acceptance: {summary['cqo']['passed']} / {summary['cqo']['runs']}", + "Observable median deltas (CQO - baseline):", + ] + for metric in NUMERIC_METRICS: + item = summary["metrics"][metric] + lines.append( + f" {metric}: {format_number(item['median_delta'])} " + f"(n={item['paired_observations']})" + ) + lines.append("Negative deltas mean less observed work by CQO.") + lines.append("Pairs failing acceptance are excluded from efficiency deltas.") + return "\n".join(lines) + "\n" + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Summarize paired CQO benchmark observations without inventing savings." + ) + parser.add_argument("results", type=Path, help="JSONL benchmark results file") + parser.add_argument( + "--format", + choices=("text", "markdown", "json"), + default="text", + help="Output format", + ) + return parser + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + try: + rows = load_jsonl(args.results) + if not rows: + raise BenchmarkError("results file contains no records") + summary = summarize(rows) + except (OSError, BenchmarkError) as exc: + print(f"benchmark error: {exc}", file=sys.stderr) + return 2 + + if args.format == "json": + print(json.dumps(summary, ensure_ascii=False, indent=2, default=str)) + elif args.format == "markdown": + print(markdown_report(summary), end="") + else: + print(text_report(summary), end="") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From 7675ac8ae3da3c3f867d7ac77bf6a0d7b495bd89 Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:45:53 +0800 Subject: [PATCH 05/14] test: cover benchmark pairing and honest deltas --- tests/test_benchmark_report.py | 130 +++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 tests/test_benchmark_report.py diff --git a/tests/test_benchmark_report.py b/tests/test_benchmark_report.py new file mode 100644 index 0000000..b13339c --- /dev/null +++ b/tests/test_benchmark_report.py @@ -0,0 +1,130 @@ +import importlib.util +import json +import tempfile +import unittest +from pathlib import Path + +SCRIPT = Path(__file__).resolve().parents[1] / "benchmarks" / "report.py" +spec = importlib.util.spec_from_file_location("benchmark_report", SCRIPT) +benchmark_report = importlib.util.module_from_spec(spec) +assert spec.loader is not None +spec.loader.exec_module(benchmark_report) + + +class BenchmarkReportTests(unittest.TestCase): + def test_only_acceptance_passing_pairs_contribute_deltas(self): + rows = [ + { + "case_id": "case-a", + "pair_id": "p1", + "variant": "baseline", + "benchmark_mode": "controlled", + "acceptance_passed": True, + "files_inspected": 10, + "broad_checks": 1, + }, + { + "case_id": "case-a", + "pair_id": "p1", + "variant": "cqo", + "benchmark_mode": "controlled", + "acceptance_passed": True, + "files_inspected": 4, + "broad_checks": 0, + }, + { + "case_id": "case-b", + "pair_id": "p1", + "variant": "baseline", + "benchmark_mode": "controlled", + "acceptance_passed": True, + "files_inspected": 8, + }, + { + "case_id": "case-b", + "pair_id": "p1", + "variant": "cqo", + "benchmark_mode": "controlled", + "acceptance_passed": False, + "files_inspected": 2, + }, + ] + + summary = benchmark_report.summarize(rows) + + self.assertEqual(summary["pairs_matched"], 2) + self.assertEqual(summary["pairs_acceptance_eligible"], 1) + self.assertEqual(summary["metrics"]["files_inspected"]["median_delta"], -6) + self.assertEqual(summary["metrics"]["files_inspected"]["paired_observations"], 1) + self.assertEqual(summary["metrics"]["broad_checks"]["median_delta"], -1) + + def test_missing_metrics_are_not_guessed(self): + rows = [ + { + "case_id": "case-a", + "pair_id": "p1", + "variant": "baseline", + "benchmark_mode": "full-policy", + "acceptance_passed": True, + }, + { + "case_id": "case-a", + "pair_id": "p1", + "variant": "cqo", + "benchmark_mode": "full-policy", + "acceptance_passed": True, + "searches": 2, + }, + ] + summary = benchmark_report.summarize(rows) + self.assertIsNone(summary["metrics"]["searches"]["median_delta"]) + self.assertEqual(summary["metrics"]["searches"]["paired_observations"], 0) + + def test_jsonl_validation_and_markdown_report(self): + records = [ + { + "case_id": "case-a", + "pair_id": "p1", + "variant": "baseline", + "benchmark_mode": "controlled", + "acceptance_passed": True, + "subagents": 1, + }, + { + "case_id": "case-a", + "pair_id": "p1", + "variant": "cqo", + "benchmark_mode": "controlled", + "acceptance_passed": True, + "subagents": 0, + }, + ] + + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "results.jsonl" + path.write_text( + "\n".join(json.dumps(item) for item in records) + "\n", + encoding="utf-8", + ) + rows = benchmark_report.load_jsonl(path) + report = benchmark_report.markdown_report( + benchmark_report.summarize(rows) + ) + + self.assertIn("Acceptance-eligible pairs: 1", report) + self.assertIn("| subagents | 1 | -1 |", report) + + def test_duplicate_variant_in_pair_is_rejected(self): + row = { + "case_id": "case-a", + "pair_id": "p1", + "variant": "baseline", + "benchmark_mode": "controlled", + "acceptance_passed": True, + } + with self.assertRaises(benchmark_report.BenchmarkError): + benchmark_report.pair_rows([row, dict(row)]) + + +if __name__ == "__main__": + unittest.main() From e50d36a7ee58cae2bee74268e30df64c15469299 Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:46:01 +0800 Subject: [PATCH 06/14] test: add benchmark report fixture --- tests/fixtures/benchmark-results.jsonl | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 tests/fixtures/benchmark-results.jsonl diff --git a/tests/fixtures/benchmark-results.jsonl b/tests/fixtures/benchmark-results.jsonl new file mode 100644 index 0000000..5e5486c --- /dev/null +++ b/tests/fixtures/benchmark-results.jsonl @@ -0,0 +1,2 @@ +{"case_id":"test-case","pair_id":"p1","variant":"baseline","benchmark_mode":"controlled","acceptance_passed":true,"files_inspected":8,"broad_checks":1,"subagents":1} +{"case_id":"test-case","pair_id":"p1","variant":"cqo","benchmark_mode":"controlled","acceptance_passed":true,"files_inspected":3,"broad_checks":0,"subagents":0} From 6c4db7d9b582f08a91a78a0adc6fe08138cf419d Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:46:10 +0800 Subject: [PATCH 07/14] ci: smoke test benchmark report --- .github/workflows/python-cli-tests.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/python-cli-tests.yml b/.github/workflows/python-cli-tests.yml index f1a9347..8647c5c 100644 --- a/.github/workflows/python-cli-tests.yml +++ b/.github/workflows/python-cli-tests.yml @@ -28,3 +28,8 @@ jobs: python3 skills/codex-quota-optimizer/scripts/cqo.py audit --note ci-smoke python3 skills/codex-quota-optimizer/scripts/cqo.py history --limit 1 python3 skills/codex-quota-optimizer/scripts/cqo.py doctor + + - name: Benchmark report smoke test + run: | + python3 benchmarks/report.py tests/fixtures/benchmark-results.jsonl + python3 benchmarks/report.py tests/fixtures/benchmark-results.jsonl --format markdown From 8e128dcf3f179ff905ab5c5a4dc10129dc8893ea Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:46:17 +0800 Subject: [PATCH 08/14] chore: bump plugin manifests to 0.2.2 --- .codex-plugin/plugin.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index b9ce6ba..abb34ed 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "codex-quota-optimizer", - "version": "0.2.1", + "version": "0.2.2", "description": "A zero-friction Codex usage governor with task-aware routing, soft budgets, focused context, local audits, and layered verification.", "author": { "name": "Daniel Chen", From 71fcf94f406f2dd8d4a6071b0e0d8943eeedc683 Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:46:19 +0800 Subject: [PATCH 09/14] chore: bump plugin manifests to 0.2.2 --- plugin.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugin.json b/plugin.json index e78c088..6a162a7 100644 --- a/plugin.json +++ b/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json", "name": "codex-quota-optimizer", - "version": "0.2.1", + "version": "0.2.2", "description": "A zero-friction Codex usage governor with task-aware routing, soft budgets, focused context, local audits, and layered verification.", "author": { "name": "Daniel Chen", From de070a50f642fda4def032d1e1fd6d911858c2a3 Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:46:47 +0800 Subject: [PATCH 10/14] docs: add Chinese benchmark methodology --- benchmarks/README.zh-CN.md | 115 +++++++++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 benchmarks/README.zh-CN.md diff --git a/benchmarks/README.zh-CN.md b/benchmarks/README.zh-CN.md new file mode 100644 index 0000000..4f13702 --- /dev/null +++ b/benchmarks/README.zh-CN.md @@ -0,0 +1,115 @@ +# CQO Benchmark Framework｜基准测试框架 + +这套 Benchmark 只回答一个问题： + +> 在任务仍然通过**同一套验收标准**的前提下，Codex Quota Optimizer 是否减少了可以避免的 Agent 工作？ + +它不是 Token 节省计算器，也不会抓取 OpenAI 私有 Usage 数据。 + +## 记录什么 + +只记录能够从 Session / Tool 输出中直接观察到的行为： + +- 验收标准通过 / 失败 +- 读取文件数 +- 修改文件数 +- 搜索次数 +- 定向检查次数 +- 全量 / 广范围测试次数 +- 启动 Subagent 数 +- 模型升级次数 +- 重复读取次数 +- 可获取时的 Wall-clock time +- 实际使用的模型角色 / reasoning +- 特殊情况备注 + +未知数据直接省略，不猜测隐藏 Token、剩余额度或“节省百分比”。 + +## 两种 Benchmark 模式 + +### 1. Controlled behavior benchmark + +Baseline 与 CQO 使用相同的起始模型和 reasoning。 + +主要隔离测试 CQO 在以下方面的作用： + +- 上下文克制 +- 修改面控制 +- 验证范围 +- Subagent 使用 +- 重复探索 + +### 2. Full-policy benchmark + +Baseline 按正常方式运行；CQO 可以按自己的策略选择更经济的模型角色或 reasoning。 + +主要测试完整 CQO 工作流，包括模型路由。 + +必须记录实际模型角色 / reasoning，确保比较透明。 + +## 公平测试协议 + +每个对外发布的比较都应该： + +1. 从**同一个仓库 Commit**开始。 +2. 使用**同一个任务 Prompt 和验收标准**。 +3. 使用等价的权限和工具。 +4. Baseline / CQO 使用同一个 `pair_id`。 +5. 两次运行都从干净工作区开始。 +6. 多次测试时尽量交替运行顺序。 +7. 单个 Pair 只能称为 Case Study；需要更强结论时使用多个 Pair。 +8. 任意一侧验收失败，该 Pair 都不能用于宣称效率优势。 +9. 对外发布时尽量保留原始 Session Note / Command Log 等证据。 + +## 结果格式 + +每行一个 JSON 对象： + +```json +{"case_id":"checkout-bug","pair_id":"p1","variant":"baseline","benchmark_mode":"controlled","repo_commit":"abc123","acceptance_passed":true,"model_role":"balanced","reasoning":"medium","files_inspected":9,"files_changed":2,"searches":5,"focused_checks":1,"broad_checks":1,"subagents":0,"model_escalations":0,"repeated_reads":2,"wall_seconds":180,"notes":"这里只展示字段结构，请替换为真实观察数据。"} +``` + +CQO Run 使用相同的 `case_id` 与 `pair_id`。 + +### 必填字段 + +- `case_id` +- `pair_id` +- `variant`：`baseline` 或 `cqo` +- `benchmark_mode`：`controlled` 或 `full-policy` +- `acceptance_passed` + +## 生成报告 + +```bash +python benchmarks/report.py path/to/results.jsonl +``` + +Markdown： + +```bash +python benchmarks/report.py path/to/results.jsonl --format markdown +``` + +JSON： + +```bash +python benchmarks/report.py path/to/results.jsonl --format json +``` + +只有 Baseline 与 CQO **都通过验收**的匹配 Pair，才会进入效率 Delta。 + +负数代表 CQO 在该可观察指标上使用得更少。 + +## 第一批真实 Case 建议 + +首批公开 Case 建议覆盖： + +1. **XS — 文档 / 机械修改** +2. **S — 局部 Bug 修复** +3. **M — 小型跨文件功能** +4. **L — Debug 或 Migration** + +仓库会先发布 Benchmark Framework，再逐步加入真实测量结果，避免为了传播先制造漂亮数字。 + +发布真实 Case 时，可使用 [case-study-template.md](./case-study-template.md)。 From fde67ae8267c96f195fc9135ac2e05e57405100d Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:46:53 +0800 Subject: [PATCH 11/14] chore: align cqo version with 0.2.2 --- skills/codex-quota-optimizer/scripts/cqo.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/codex-quota-optimizer/scripts/cqo.py b/skills/codex-quota-optimizer/scripts/cqo.py index 1a4a194..6722e1c 100644 --- a/skills/codex-quota-optimizer/scripts/cqo.py +++ b/skills/codex-quota-optimizer/scripts/cqo.py @@ -18,7 +18,7 @@ from pathlib import Path from typing import Any -VERSION = "0.2.1" +VERSION = "0.2.2" HIGH_RISK_TERMS = { "security", "auth", "authentication", "authorization", "permission", From de3588c4871aecdb21a8fb126389a4887ce9bd6f Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:47:12 +0800 Subject: [PATCH 12/14] docs: add honest benchmark framework --- README.md | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0cea349..0807e11 100644 --- a/README.md +++ b/README.md @@ -322,6 +322,29 @@ These are **behavioral budgets**, not fake quota counters. --- +## Benchmarking without fake savings + +v0.2.2 adds a paired benchmark framework for comparing **baseline Codex vs CQO** without inventing hidden-token or quota-savings numbers. + +Two modes are supported: + +- **Controlled** — same starting model/reasoning on both sides, isolating context, verification, repeated exploration and Subagent discipline. +- **Full-policy** — CQO may use its own routing policy; the actual model role/reasoning must be recorded. + +The quality gate comes first: **a matched pair only contributes efficiency deltas when both runs pass the same acceptance criteria**. + +Observable metrics can include files inspected, searches, focused/broad checks, Subagents, model escalations, repeated reads, and wall time. Unknown values are omitted instead of guessed. + +```bash +python benchmarks/report.py path/to/results.jsonl --format markdown +``` + +See the full [Benchmark methodology](./benchmarks/README.md) · [中文说明](./benchmarks/README.zh-CN.md). + +> The project intentionally ships the measurement method before publishing savings claims. + +--- + ## What it deliberately does not do - ❌ bypass or evade OpenAI usage limits @@ -351,7 +374,8 @@ codex-quota-optimizer/ │ ├── cqo.py # optional local usage governor CLI │ ├── change_scope.py │ └── repo_snapshot.py -├── tests/ # standard-library CLI tests +├── benchmarks/ # paired baseline/CQO benchmark framework +├── tests/ # standard-library CLI + benchmark tests ├── assets/ # Plugin icon + README visuals ├── examples/ ├── install.sh # installs Skill + optional cqo shortcut @@ -370,7 +394,8 @@ codex-quota-optimizer/ - [x] Plugin packaging for dual Skill / Plugin distribution - [x] HOL Codex Plugin Catalog listing - [x] One-line Skills CLI installation -- [ ] Benchmark suite comparing default and optimized workflows +- [x] Paired **Benchmark Framework** comparing baseline and CQO behavior +- [ ] Publish measured real-world benchmark case set Ideas and PRs are welcome. See [CONTRIBUTING.md](./CONTRIBUTING.md). From 3eac79330d6b0710963dd12b33e82057c678782d Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:47:14 +0800 Subject: [PATCH 13/14] docs: add honest benchmark framework in Chinese --- README.zh-CN.md | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/README.zh-CN.md b/README.zh-CN.md index cfe6266..6e96650 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -324,6 +324,29 @@ Repository Snapshot 与 Change Scope 两个脚本同样只在本地运行，不 --- +## Benchmark｜先保证质量，再谈节省 + +v0.2.2 加入了一套 Baseline Codex vs CQO 的成对 Benchmark 框架，重点是比较**可直接观察的 Agent 行为**，而不是猜隐藏 Token 或额度节省百分比。 + +支持两种模式： + +- **Controlled**：两边使用相同起始模型 / reasoning，主要隔离测试上下文、验证范围、重复探索和 Subagent 克制。 +- **Full-policy**：允许 CQO 使用自己的路由策略，但必须记录实际模型角色 / reasoning。 + +最重要的 Gate 是质量：**只有 Baseline 和 CQO 都通过同一套验收标准，该 Pair 才会进入效率 Delta。** + +可以记录的指标包括读取文件数、搜索次数、定向 / 全量检查次数、Subagent、模型升级、重复读取和 Wall time；不知道的数据直接省略，不猜。 + +```bash +python benchmarks/report.py path/to/results.jsonl --format markdown +``` + +完整方法见 [Benchmark 中文说明](./benchmarks/README.zh-CN.md) · [English](./benchmarks/README.md)。 + +> 先发布可复现的测量方法，再发布节省结论。 + +--- + ## 它明确不会做什么 - ❌ 绕过或规避 OpenAI 使用限制 @@ -353,7 +376,8 @@ codex-quota-optimizer/ │ ├── cqo.py # 可选本地 Usage Governor CLI │ ├── change_scope.py │ └── repo_snapshot.py -├── tests/ # Python 标准库 CLI 测试 +├── benchmarks/ # Baseline / CQO 成对 Benchmark 框架 +├── tests/ # Python 标准库 CLI + Benchmark 测试 ├── assets/ # Plugin 图标 + README 视觉资源 ├── examples/ ├── install.sh # 安装 Skill + 可选 cqo 快捷命令 @@ -372,7 +396,8 @@ codex-quota-optimizer/ - [x] Plugin 打包：同时支持 Skill 直装与 Plugin 分发 - [x] HOL Codex Plugin Catalog 收录 - [x] Skills CLI 一行命令安装 -- [ ] Benchmark：对比默认工作流和优化工作流 +- [x] Baseline / CQO 成对 **Benchmark Framework** +- [ ] 发布真实测量的 Real-world Benchmark Case 集合 欢迎提交 Issue / PR。参见 [CONTRIBUTING.md](./CONTRIBUTING.md)。 From 998d0b2b657769a864b118168fb2161415dcfaae Mon Sep 17 00:00:00 2001 From: Daniel Date: Mon, 21 Sep 2026 14:47:24 +0800 Subject: [PATCH 14/14] docs: add 0.2.2 changelog --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 48873ab..1e8009a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,20 @@ All notable changes to this project will be documented here. +## [0.2.2] - 2026-09-21 + +### Added +- Paired Baseline-vs-CQO benchmark framework with controlled and full-policy modes. +- Honest benchmark result schema and case-study template. +- Local benchmark report generator with text, Markdown, and JSON output. +- Acceptance-gated comparisons: efficiency deltas are only calculated when both runs pass the same acceptance criteria. +- Benchmark tests and CI smoke coverage. +- English and Simplified Chinese benchmark methodology. + +### Changed +- CQO CLI/package version aligned to 0.2.2. +- Roadmap now separates the benchmark framework from future published real-world measurements. + ## [0.2.1] - 2026-09-21 ### Added