Add Tier 1 lints, per-workflow ledgers and the weekly improvement cycle (dry-run)

Meta-improvement engine P1 (design §4, §5, §9). The cycle reads trace/1,
never a model; it writes only its own ledger directory and its own trace
records. Proposals stay in the ledger: nothing reaches GitHub until P4.

improve/reader.py groups records into units of work by context.unit_id
(records without one, the knowledge service's and the review bot's, fall
back to run_id:step so nothing in the store is invisible) and reads a
window by JSONL day file.

improve/lints.py is the Tier 1 catalogue: 17 deterministic lints, each with
a version and the campaign finding it comes from (L01 truncation, L02
budget exhaustion, L03 empty final after tools, L04 contract reject, L05
parse-failure rate vs baseline, L06 duplicate rendering, L07 silent
fallback / served-model mismatch, L08 zero-yield pass, L09 cap drops, L10
unresolved anchors, L11 cost/latency robust-z outliers, L12 refusals and
out-of-scope edits, L13 provider errors which quarantine the unit, L14
trace/1-native completeness, L15 repeated identical tool calls, L16
evidence-cap drops, L17 epistemic claims without a backing tool call). A
lint whose evidence shape a producer does not emit yet finds nothing. The
robust z floors the MAD at a tenth of the median so a uniform history does
not flag every change as infinite.

improve/ledger.py keeps one JSON ledger per workflow: per-cycle stats and
lint rates, rolling baseline samples, the proposal state machine (open ->
experiment-registered -> supported|neutral|refuted|underpowered ->
landed|closed, 30 days untouched -> stale), one live proposal per lint or
stage, holds, channel liveness; every change is also a decision record.

improve/cycle.py runs a cycle: preflight (improve_enabled is the kill
switch; holds are reported), lints and baselines per workflow, Tier 1
proposals when a lint's unit rate worsened 1.5x with at least 3 units, the
stale sweep, a Markdown + JSON report and a decision(type=cycle) record,
then the cursor advances. The engine's own records are units of the next
cycle (self-enrolment). is_due/maybe_run_weekly implement the weekly slot
(Monday 05:00 UTC by default); the kb serve scheduler calls it every tick
under its lease, isolated like its other tick items.

workflow_improve is a new read-only L2 task kind with the vetted
workflow-improve playbook (improve.preflight -> improve.lint ->
improve.ledger -> report.final_summary); the intent and chat enums list
it. Settings: improve_enabled, improve_cycle_weekday/hour,
improve_ledger_dir, improve_budget_usd_week and
improve_budget_judge_calls_week (enforced in P3). CLI: improve cycle,
improve ledger, improve lints.

Review round 1 (gpt-6.1-sol-high) fixes: units are attributed whole to
the cycle in which they END (records read from a two-day lookback before
the window; a unit still in flight at the cutoff is deferred untouched),
so a lint that needs both halves of a unit still fires; every ledger
workflow is swept and reported each cycle, including ones with no unit in
the window, so a workflow that stopped producing records still goes stale
and reports a dead channel; a cross-process file lock (trace_store.file_lock,
the same primitive as the migration lock) serializes whole cycles per
ledger directory including the due check, and every ledger
read-modify-write takes the workflow's lock with a per-process temp name.

Review round 2 fixes: a unit counts only with completion evidence (a
terminal decision after its last call, or an hour of silence before the
cutoff) and only once — the cursor keeps the counted ids for the lookback
span, so a unit still running at a cutoff is deferred and never counted
twice; the scheduler, the CLI and the playbook share one default ledger
directory (cycle.DEFAULT_LEDGER_DIR / ledger_dir_for); L11 prices a unit
with the same settings the baseline used; the unresolved-anchor pattern
matches `file.py:?` at a whitespace or text boundary; the CLI entry block
sits after the helpers it calls; the reader defers an unterminated
trailing JSONL line and skips a torn one instead of aborting the cycle.

Review round 3 fixes: a unit that finished before the window counts only
if it was still deferred (incomplete) at the window's start, so the
lookback never pulls in earlier completed work (the first cycle reports
seven days, an explicit --since window means what it says); L07 compares
served and requested models through canonical_model with the configured
aliases, exactly as the served-model guard does.

Review round 4 fix: the cursor persists the ACTUAL deferred set and the
engine's origin instead of re-deriving completion at the window start from
timestamps (a record can become visible after the cutoff: a torn trailing
line); a unit that finished before the window counts when it was deferred
last time or when it became visible inside the engine's contiguous coverage
(never counted before), and is historical otherwise; regression test with a
torn decision line across two cycles. The coverage origin resets to the
window start whenever continuity breaks (an explicit window elsewhere), so
work skipped between windows never leaks into a later contiguous cycle.
A cursor written before the deferred set existed recovers the old
cycle's pending units from their own records at the old cutoff (a unit
complete then was counted then and is never replayed), once, on upgrade.

Tests: test_improve_p1.py — reader grouping and fallbacks, a clean unit
has no findings, every lint fires on its fixture, baseline-dependent lints,
completeness, quarantine, the ledger state machine and stale sweep, two
cycles end to end (cursor, proposal opened on worsening, quarantined unit
out of the baseline, cycle records), kill switch/holds/slot, the scheduler
hook, kind/playbook/planner resolution, the playbook through the executor,
the CLI. Full suite green except test_mcp.py (host mcp 2.x, fails on main).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Author: tzhouam

doc/architecture/SPEC/improve.md

@@ -0,0 +1,36 @@
+# improve/ —— 规范（元改进引擎）
+
+<!-- verified-against: 2026-09-30 -->
+
+`设计：/data/zhoutaichang/copilot/meta-improvement-engine-design.md v1（GPT-6 sol 批准 2026-09-29） · refactor-status: building (P0–P1 已落地)`
+
+## 职责
+面向任意 trace/1 工作流的"取证—实验—提案"循环。**只提案**：引擎的写权限限于追加自己的 trace/1 记录、
+写自己的账本目录、在影子环境跑实验；产线变更一律走 PR。
+
+## 模块
+| 文件 | 阶段 | 作用 |
+|---|---|---|
+| `enroll.py` | P0 | 工作流声明（`workflows/*.yaml` + `IMPROVE_WORKFLOWS_DIRS`）：kind/unit/item_key/fingerprint.covers/capture/shadow_tools；未声明 = Tier 1 |
+| `fingerprint.py` | P0 | 声明式配置指纹（playbook sha、settings、prompt 文件哈希、routing、`resolved_models`、tools、knowledge_snapshot、copilot_sha）；覆盖项缺失 → 无指纹、记 `fingerprint_missing` |
+| `shadow.py` | P0 | 影子四层边界：`harden_scope`（严格 extra、读围栏）、`shadow_env`/`make_executables_dir`/`assert_boundaries`（凭据与可执行白名单）、`make_shadow_clone`（`--shared` 独立克隆、无 remote、钉住的 base ref）、`smoke` |
+| `staging.py` | P0 | 隔离前的条目快照：`stage_item` 复用 `pr.fetch_diff` 的钉住取数，写 `item_staged` 记录；任一必需输入不可得即拒绝 |
+| `reader.py` | P1 | trace/1 → 工作单元（按 `context.unit_id`，RB/知识服务记录回退到 `run_id:step`）；窗口按 JSONL 日期文件读取 |
+| `lints.py` | P1 | Tier 1 的 17 条确定性 lint（L01–L17，每条带版本与出处）；`Baseline` 稳健 z（MAD 下限 = 中位数的 10%）；L13 隔离 |
+| `ledger.py` | P1 | 每工作流 JSON 账本：周期统计、基线样本、提案状态机（open → experiment-registered → supported/neutral/refuted/underpowered → landed/closed；30 天无人触碰 → stale）、hold、通道活性；每次变更同时写 `decision` |
+| `cycle.py` | P1 | 周期：preflight（`improve_enabled` 是 kill switch）→ lints + 基线 → 账本 + Tier 1 提案（lint 恶化 ≥1.5× 且 ≥3 单元）→ stale 清扫 → 报告（`reports/cycle-*.md/json`）+ `decision(type=cycle)`；`is_due`/`maybe_run_weekly` 是周度槽位 |
+| `cli.py` | P0/P1 | `improve migrate-index|rebuild-index|rollback-index|compare-index|verify-index|cycle|ledger|lints` |
+
+## 接入点
+- 执行器：`settings.trace_store_root` 非空时绑定 store 并为每步绑定单元上下文；`settings.improve_shadow` 下拒绝非 read/report 步骤。
+- `tools.dispatch` / `LLM.create` / `HarnessLLM.create` / `run_harness_step` / MCP bridge：全保真采集。
+- `kb serve` 调度器：每 tick 调 `_improve_cycle()`，周度槽位到达即跑一次周期（与知识服务同一租约、异常隔离）。
+- 任务种类 `workflow_improve`（L2，READ_ONLY_KINDS）；playbook `workflow-improve`（preflight → lint → ledger → report）。
+
+## 不变量
+- P1 周期永远 dry-run：提案只在账本与 trace 里，不上 GitHub（P4）。
+- 引擎自身的记录（`playbook=workflow-improve`）是下一周期的 Tier 1 单元（自登记）。
+- lint 只读 trace，不调模型；无证据形态的 lint 不猜。
+
+## 测试
+`test_improve_p0.py`、`test_improve_p0b.py`、`test_improve_p1.py`。

playbooks/workflow-improve.yaml

@@ -0,0 +1,20 @@
+# The meta-improvement engine's weekly cycle (design §4). A vetted reuse
+# playbook, never generated. Every step is read-only toward the target
+# repositories and the trace store (the engine only appends its own records);
+# publication (P4) is a separate `risk: push` step behind the post gate.
+name: workflow-improve
+version: 1
+status: active
+provenance:
+  created_by: human
+  source: meta-improvement-engine-design.md v1 (sol-approved 2026-09-29), P1 slice
+task_kinds: [workflow_improve]
+repos: []
+params: {}
+steps:
+  - {id: preflight, step: improve.preflight}
+  - {id: lint, step: improve.lint}
+  - {id: ledger, step: improve.ledger}
+  - {id: report, step: report.final_summary}
+success: one cycle recorded (decision type=cycle), the per-workflow ledgers
+  updated, the cycle report written; proposals stay in the ledger until P4

src/infermatrix_copilot/chat.py

@@ -43,7 +43,7 @@
 
 You can: chat and answer questions; inspect past runs, logs and reports; read/search \
 the repository; and execute maintenance tasks via the run_task tool. Task kinds: \
-repo_rebase, pr_rebase, pr_debug, pr_review, issue_answer, issue_filter, repo_profile.
+repo_rebase, pr_rebase, pr_debug, pr_review, issue_answer, issue_filter, repo_profile, workflow_improve.
 
 Rules:
 - To execute anything, CALL a tool — never claim work happened without a tool result.
@@ -67,7 +67,7 @@
                 "kind": {"type": "string",
                          "enum": ["repo_rebase", "pr_rebase", "pr_debug",
                                   "pr_review", "issue_answer", "issue_filter",
-                                  "repo_profile"]},
+                                  "repo_profile", "workflow_improve"]},
                 "pr": {"type": "integer"},
                 "issue": {"type": "integer"},
                 "repo": {"type": "string",

src/infermatrix_copilot/config.py

@@ -558,6 +558,17 @@ def _parse_mixture(cls, v):
     # Extra directories of workflow declarations (os.pathsep separated),
     # consulted after the package's builtin improve/workflows/.
     improve_workflows_dirs: str = ""
+    # The weekly cycle (design §4/§10). improve_enabled is THE kill switch;
+    # the slot is `improve_cycle_weekday` (0=Monday) at `improve_cycle_hour`
+    # UTC, after the 04:00 PR-state sweep; the ledgers and reports live under
+    # improve_ledger_dir (default ~/.infermatrix-copilot/improve).
+    improve_enabled: bool = False
+    improve_cycle_weekday: int = 0
+    improve_cycle_hour: int = 5
+    improve_ledger_dir: str = ""
+    # the hard weekly envelope (P3 enforces it per model call)
+    improve_budget_usd_week: float = 20.0
+    improve_budget_judge_calls_week: int = 300
 
     # Metrics (eval/METRICS_RESEARCH.md) — per-run metrics.json: CATQ = Q·S/C.
     # Reference budgets are EXPLICIT deployment assumptions (RQS3e precedent):

src/infermatrix_copilot/engine/steps/improve.py

@@ -45,3 +45,86 @@ async def _stage_items(ctx: StepContext) -> StepResult:
     ctx.state.update(updates)
     return StepResult(True, summary=f"staged {len(snapshots)} item(s) under {shadow_root}",
                       outputs={"staged": sorted(snapshots), "state_updates": updates})
+
+
+def _ledger_dir(ctx: StepContext) -> Path:
+    from ...improve.cycle import ledger_dir_for
+
+    override = str(ctx.params.get("ledger_dir") or "")
+    return Path(override).expanduser() if override else ledger_dir_for(ctx.settings)
+
+
+def _trace_store(ctx: StepContext):
+    from ...trace_store import TraceStore, current_store
+
+    store = current_store()
+    if store is not None:
+        return store
+    root = str(getattr(ctx.settings, "trace_store_root", "") or "")
+    if not root:
+        return None
+    return TraceStore(Path(root).expanduser())
+
+
+@step("improve.preflight", "deterministic", "read",
+      "Kill switch + per-workflow holds; refuses the cycle loudly.")
+async def _preflight(ctx: StepContext) -> StepResult:
+    from ...improve.cycle import CycleRefused, preflight
+    from ...improve.ledger import Ledger
+
+    store = _trace_store(ctx)
+    if store is None:
+        return StepResult(False, FailureKind.BLOCKED, "no trace store (set TRACE_STORE_ROOT)")
+    try:
+        gate = preflight(ctx.settings, Ledger(_ledger_dir(ctx), store))
+    except CycleRefused as exc:
+        return StepResult(False, FailureKind.BLOCKED, str(exc))
+    updates = {"improve_holds": gate["holds"], "improve_ledger_dir": str(_ledger_dir(ctx))}
+    ctx.state.update(updates)
+    return StepResult(True, summary=f"preflight ok; holds: {gate['holds'] or 'none'}",
+                      outputs={"state_updates": updates})
+
+
+@step("improve.lint", "deterministic", "read",
+      "Tier 1 lints + baselines + ledger update over the cycle window (dry-run).")
+async def _lint(ctx: StepContext) -> StepResult:
+    """Runs the whole P1 cycle body (lints, baselines, Tier 1 proposals in
+    the ledger, stale sweep, report). Window from params `since`/`until`
+    (epoch seconds) or the cursor. Publishes `improve_report_path`."""
+    from ...improve.cycle import CycleRefused, run_cycle
+
+    store = _trace_store(ctx)
+    if store is None:
+        return StepResult(False, FailureKind.BLOCKED, "no trace store (set TRACE_STORE_ROOT)")
+    since = ctx.params.get("since")
+    until = ctx.params.get("until")
+    try:
+        report = run_cycle(store, ctx.settings, _ledger_dir(ctx),
+                           since=float(since) if since else None, until=float(until) if until else None,
+                           dry_run=True, force=bool(ctx.params.get("force")))
+    except CycleRefused as exc:
+        return StepResult(False, FailureKind.BLOCKED, str(exc))
+    updates = {"improve_cycle": {k: report[k] for k in ("at", "since", "until", "units", "proposals_opened")},
+               "improve_report_dir": str(_ledger_dir(ctx) / "reports")}
+    ctx.state.update(updates)
+    opened = len(report["proposals_opened"])
+    return StepResult(True, summary=f"cycle over {report['units']} units in {len(report['workflows'])} workflow(s); "
+                                    f"{opened} Tier 1 proposal(s) opened (dry-run)",
+                      outputs={"report": report, "state_updates": updates})
+
+
+@step("improve.ledger", "deterministic", "read",
+      "Summarize the per-workflow ledgers for the run report.")
+async def _ledger_summary(ctx: StepContext) -> StepResult:
+    from ...improve.ledger import Ledger
+
+    ledger = Ledger(_ledger_dir(ctx))
+    lines = []
+    for workflow in ledger.workflows():
+        wl = ledger.load(workflow)
+        open_ = [p for p in wl.proposals if p.state not in ("landed", "closed")]
+        last = wl.cycles[-1] if wl.cycles else {}
+        lines.append(f"{workflow}: tier {wl.tier}, {len(wl.cycles)} cycle(s), last units {last.get('units', 0)}, "
+                     f"{len(open_)} live proposal(s)" + (f", HOLD: {wl.hold}" if wl.hold else ""))
+    summary = "\n".join(lines) or "no workflows in the ledger yet"
+    return StepResult(True, summary=summary[:400], outputs={"ledger_summary": summary})

src/infermatrix_copilot/improve/cli.py

@@ -12,6 +12,11 @@
     improve compare-index   read-only: rebuild a throwaway copy and diff it
                             against the live index (safe with live writers)
     improve verify-index    read-only: id sets and schema-2 columns vs the records
+    improve cycle           [--since EPOCH] [--until EPOCH] [--force] [--ledger-dir DIR]
+                            run one weekly cycle now (Tier 1 lints, baselines,
+                            ledger, report; dry-run — proposals stay local)
+    improve ledger          [--workflow W] [--ledger-dir DIR]   show the ledgers
+    improve lints           the Tier 1 lint catalogue (ids, versions, origins)
 
 The writer-quiescence gate (design §7.1). Pause flags alone do not stop an
 intake or sweep that is already running, so the gate requires evidence that
@@ -94,7 +99,8 @@ def main(argv: list[str] | None = None) -> int:
     parser = argparse.ArgumentParser(prog="infermatrix-copilot improve",
                                      description="operate the meta-improvement engine's trace index")
     sub = parser.add_subparsers(dest="command", required=True)
-    for name in ("migrate-index", "rebuild-index", "rollback-index", "compare-index", "verify-index"):
+    for name in ("migrate-index", "rebuild-index", "rollback-index", "compare-index", "verify-index",
+                 "cycle", "ledger", "lints"):
         p = sub.add_parser(name)
         p.add_argument("--trace-root", default=None, help="trace store root (TRACE_STORE_ROOT / KB_STATE_DIR/traces)")
         if name in ("migrate-index", "rebuild-index", "rollback-index"):
@@ -104,7 +110,22 @@ def main(argv: list[str] | None = None) -> int:
             p.add_argument("--no-backup", action="store_true")
         if name == "rebuild-index":
             p.add_argument("--to-schema", type=int, default=None, help="0 (pin downgrade) or the current schema")
+        if name in ("cycle", "ledger"):
+            p.add_argument("--ledger-dir", default=None)
+        if name == "cycle":
+            p.add_argument("--since", type=float, default=None)
+            p.add_argument("--until", type=float, default=None)
+            p.add_argument("--force", action="store_true", help="run even when improve_enabled is off (operator)")
+        if name == "ledger":
+            p.add_argument("--workflow", default=None)
     args = parser.parse_args(argv)
+    if args.command == "lints":
+        from .lints import catalogue
+
+        print(json.dumps(catalogue(), ensure_ascii=False, indent=1))
+        return 0
+    if args.command in ("cycle", "ledger"):
+        return _cycle_commands(args)
     store = _store(args)
     try:
         if args.command == "migrate-index":
@@ -129,5 +150,43 @@ def main(argv: list[str] | None = None) -> int:
     return 0 if report.get("ok", True) else 1
 
 
+
+def _ledger_dir(args) -> Path:
+    from ..config import Settings
+    from .cycle import ledger_dir_for
+
+    return Path(args.ledger_dir).expanduser() if args.ledger_dir else ledger_dir_for(Settings())
+
+
+def _cycle_commands(args) -> int:
+    from ..config import Settings
+    from .cycle import CycleRefused, run_cycle
+    from .ledger import Ledger
+
+    ledger_dir = _ledger_dir(args)
+    if args.command == "ledger":
+        ledger = Ledger(ledger_dir)
+        names = [args.workflow] if args.workflow else ledger.workflows()
+        out = {}
+        for name in names:
+            wl = ledger.load(name)
+            out[name] = {"tier": wl.tier, "declared": wl.declared, "hold": wl.hold, "cycles": wl.cycles[-4:],
+                         "proposals": [{"id": p.id, "state": p.state, "lint": p.lint, "stage": p.stage,
+                                        "claim": p.claim, "issue": p.issue} for p in wl.proposals]}
+        print(json.dumps(out, ensure_ascii=False, indent=1, default=str))
+        return 0
+    store = _store(args)
+    try:
+        report = run_cycle(store, Settings(), ledger_dir, since=args.since, until=args.until,
+                           dry_run=True, force=args.force)
+    except CycleRefused as exc:
+        print(json.dumps({"refused": str(exc)}), file=sys.stderr)
+        return 1
+    print(json.dumps({k: report[k] for k in ("at", "since", "until", "units", "proposals_opened", "seconds")},
+                     ensure_ascii=False, indent=1, default=str))
+    print(f"report: {ledger_dir / 'reports'}", file=sys.stderr)
+    return 0
+
+
 if __name__ == "__main__":  # pragma: no cover
     sys.exit(main())