Merge pull request #439 from InferMatrixLab/fix/rebase-resume-rerun-local-budget

rebase: CI-parity run level, local timeout multiplier, --rerun-from resume

Author: tzhouam

adapters/vllm_omni/manifest.yaml

@@ -331,6 +331,10 @@ rebase:
     # The wrapper refuses a TEST_TIMEOUT_SEC above this cap; it must not be
     # lower than the planned budget above or every required command is refused.
     module_max_timeout_sec: 21600
+    # Ready's CPU "Simple" steps budget 20 minutes for CI agents. On the shared
+    # rebase host the same serial commands passed 53-88% of their items with
+    # zero failures before that budget expired (run-20261028-152707-a60091).
+    local_timeout_multiplier: 3
     # Phase-2 module agents run on the CUDA wheel/runtime above. Broad pytest
     # paths must retain CUDA-capable and platform-neutral items while leaving
     # NPU/ROCm/XPU/MUSA-only items to their native CI lanes.

doc/architecture/SPEC/app/core.md

@@ -1,6 +1,6 @@
 # app/core.py —— 规范
 
-<!-- verified-against: 2026-10-28 -->
+<!-- verified-against: 2026-10-29 -->
 
 `Copilot` is the headless workflow application. It resolves a TaskSpec,
 delegates reviewer verdict policy to `review.plan_gate`, confirms when needed,
@@ -40,6 +40,10 @@ run（`_contained_run_dir` 拒绝越界路径）；`--repo/--playbook/--task-par
 发布 launcher 因此能把 resume 绑定到不可变的 campaign，而不会误入更新的
 无关任务。损坏的 task.json 同样 BLOCKED 而非崩溃。
 
+`rerun_from`（`--rerun-from <step id>`，2026-10-29）是唯一会改变 resume 行为的
+参数：原样传给 `_execute`，由 executor 让该 step 及其后各 step 重新执行；保存的
+任务身份仍不改。
+
 ## 被信号终止的预约 run（2026-10-28）
 
 `_execute_reserved_locked` 的终态写入（`0 → done`、`BLOCKED_EXIT → blocked`、

doc/architecture/SPEC/app/workflow_execution.md

@@ -1,6 +1,6 @@
 # app/workflow_execution.py — workflow execution boundary
 
-<!-- verified-against: 2026-10-28 -->
+<!-- verified-against: 2026-10-29 -->
 
 `WorkflowExecution` runs an already selected playbook with the caller's
 current LLM dependency. It rechecks the repository context before starting,
@@ -23,6 +23,9 @@ shutdown request exists, otherwise it propagates as before). Finalizers have
 already run inside `run_guarded` by then: GPU locks are released and the
 upstream scratch is kept, so the run is resumable with `--resume`.
 
+A non-empty `rerun_from` is seeded as `state["rerun_from"]`; the executor
+re-executes that step and every later one.
+
 A losing run lock leaves active run artifacts untouched. A failed knowledge
 lock releases an acquired run lock. A caller-provided lock stays owned by the
 caller. Finalizers and locks run on every executor outcome, including failure.

doc/architecture/SPEC/cli.md

@@ -1,6 +1,6 @@
 # cli/ —— 规范
 
-<!-- verified-against: 2026-10-27 -->
+<!-- verified-against: 2026-10-29 -->
 
 `LOC ~1420（6 个文件） · 接口 + 编排门面 · refactor-status: ok`
 
@@ -114,3 +114,4 @@ RFC 子命令原样转发与共享入口：`test_rfc_integration.py`；服务契
 
 `--resume` 现在接受可选 run id；`--repo`、`--playbook`、`--task-param` 与
 `--resume` 同用时是对保存任务的断言（见 `app/core`）。
+`--rerun-from STEP_ID`（2026-10-29）只能与 `--resume` 同用，否则 argparse 报错。

doc/architecture/SPEC/engine/executor.md

@@ -1,6 +1,6 @@
 # engine/executor.py —— 规范
 
-<!-- verified-against: 2026-10-04 -->
+<!-- verified-against: 2026-10-29 -->
 
 `LOC ~293 · 引擎底座（那个循环） · refactor-status: ok`
 
@@ -34,6 +34,9 @@ helper：`_eval_when`、`_merge`。
   step 自己的 params **之下**，所以 `--task-param limit=5` 不会被静默丢弃。
   **playbook 自己的 params 在合并中获胜** —— 其中好几个是安全攸关的
   （`force_push`、`pre_push`），因此被撰写下来的不变量**不得**被命令行覆盖。
+- **`state["rerun_from"]`**（操作者 `--resume --rerun-from <step id>`）：开跑前从
+  progress.json 删掉该 step 及其后所有 step 的检查点并落盘，记 `rerun_from` trace；
+  未知 step id → BLOCKED，检查点不动。之前的 step 照常重放。
 - step 的 span 会记录 playbook 的 step id 和 foreach 条目键：同一个 spec 可能在一个
   playbook 里跑两次、还会再扇出，所以**光靠 `step` 无法标识这份工作**。
 

doc/architecture/SPEC/engine/steps/rebase_v3.md

@@ -1,6 +1,6 @@
 # engine/steps/rebase_v3.py —— 规范
 
-<!-- verified-against: 2026-10-28 -->
+<!-- verified-against: 2026-10-29 -->
 
 `LOC ~2204 · step 库（v3 rebase 装配层） · refactor-status: oversized`
 
@@ -134,6 +134,18 @@ harness 会话通过 MCP tool bridge 拿到同一套工具面，plan gate 在派
   baseline commit 的 wheel，让"omni main 对上次对齐的 wheel 是否也失败"可问；
   任何失败返回 None，保留保守的回归判定。
 
+## 操作者重跑与本地超时预算（2026-10-29）
+
+- `--resume --rerun-from wave1`：`_v3_module_rebase` 短路条件不变（done/skipped），
+  所以 failed 模块重跑、done 模块直接返回；`_v3_wave_gate` 重新裁决 wave 2。
+- `_v3_test_loop` 在 `state["rerun_from"]` 非空时以 `retry_infra=True` 调
+  `run_test_loop`：记录过的 timeout/watchdog infra 再跑一次，已通过的 job 不重跑。
+  普通 resume 下 infra 仍然粘滞。
+- `rebase.testing.local_timeout_multiplier`（默认 1，须为 >= 1 的数，否则 BLOCKED）
+  只放大本地 `TestJob.timeout_sec`；构建出的 manifest 保留 CI 的
+  `timeout_in_minutes`（shell golden 不变）。vllm-omni 取 3：Ready 的 20 分钟 CPU
+  Simple step 在共享 rebase 主机上通过 53–88% 用例、零失败时就被杀。
+
 ## 信号中断的模块：BLOCKED 收口、resume 重跑（2026-10-28）
 
 `_v3_module_rebase` 在 `rebase_module` 返回 `status == "interrupted"`

doc/architecture/SPEC/rebase_engine.md

@@ -1,6 +1,6 @@
 # rebase_engine/ —— 规范
 
-<!-- verified-against: 2026-10-28 -->
+<!-- verified-against: 2026-10-29 -->
 
 `LOC ~7500（26 个模块） · repo-rebase-v3 的原生 rebase 引擎 · refactor-status: ok`
 
@@ -193,7 +193,9 @@ run-20261024 里这些链在 wave 1 完成后又占着 run 的 GPU 锁约 7 小
   `host_env={CUDA_VISIBLE_DEVICES, HF_HOME}`（见 `tool_bridge`）。
 - `module_pytest.py`：`rebase.testing.module_max_timeout_sec` 是 `TEST_TIMEOUT_SEC`
   的上限（vllm-omni 与计划预算同为 21600），`module_test_platform` 过滤非本平台
-  的 pytest 项。
+  的 pytest 项。未显式给 `-m` 的模块跑同时带上与 Buildkite 相同的 run level：
+  默认 `--run-level core_model`，标记为 `(<platform> or 平台中性) and core_model`。
+  调用方已写 `--run-level` 时，标记改用那一档，不再另加 core_model。
 - `wheel.py`：`release_venv_holders`（仅 venv 结构存在时、排除调用者祖先链）、
   EBUSY/ETXTBSY 重试、baseline rev-parse 前置校验、分支重置前 `git clean -ffd`。
 

src/infermatrix_copilot/app/core.py

@@ -264,14 +264,16 @@ def run_playbook(self, name: str, *, params: dict | None = None,
     def _execute(self, playbook, spec: TaskSpec, run_dir: Path, *,
                  resolution_mode: str = "resume", tier: str = "?",
                  resuming: bool = False, held_lock: RunLock | None = None,
-                 planned_context: RepositoryContext | None = None) -> int:
+                 planned_context: RepositoryContext | None = None,
+                 rerun_from: str = "") -> int:
         """Compatibility entry point; application keeps the last-run summary."""
         self.last_run_dir = run_dir
         self.last_blocked_reason = ""
         result = self.workflow_execution.run(
             playbook, spec, run_dir, resolution_mode=resolution_mode,
             tier=tier, resuming=resuming, held_lock=held_lock,
             planned_context=planned_context, llm=self.llm,
+            rerun_from=rerun_from,
         )
         self.last_blocked_reason = result.blocked_reason
         return result.exit_code
@@ -294,12 +296,14 @@ def run_queue(self, specs: list[TaskSpec], *, assume_yes: bool = False,
         return 0
 
     def resume_last(self, run_id: str = "", *, repo: str = "",
-                    playbook: str = "", params: dict | None = None) -> int:
+                    playbook: str = "", params: dict | None = None,
+                    rerun_from: str = "") -> int:
         """Resume a selected run, or the latest run when no id is supplied.
 
         Target arguments are assertions against the saved task, never edits to
         it. Release launchers can therefore bind resume to an immutable campaign
-        without accidentally entering a newer, unrelated task.
+        without accidentally entering a newer, unrelated task. `rerun_from`
+        re-executes that playbook step and everything after it.
         """
         if run_id:
             try:
@@ -348,7 +352,8 @@ def resume_last(self, run_id: str = "", *, repo: str = "",
                 print(style("✋ cannot resume: ", "red", "bold") + str(exc))
                 return BLOCKED_EXIT
             print(f"↻ resuming {run_dir.name}: {spec.describe()}")
-            return self._execute(saved_playbook, spec, run_dir, resuming=True)
+            return self._execute(saved_playbook, spec, run_dir, resuming=True,
+                                 rerun_from=rerun_from)
         print("no resumable run found")
         return 1
 

src/infermatrix_copilot/app/workflow_execution.py

@@ -44,6 +44,7 @@ def run(
         resolution_mode: str = "resume", tier: str = "?",
         resuming: bool = False, held_lock: RunLock | None = None,
         planned_context: RepositoryContext | None = None,
+        rerun_from: str = "",
     ) -> ExecutionResult:
         """Run a resolved `playbook` to completion in `run_dir`: init tracing +
         notifier, seed the shared state (repo path, push policy, protected
@@ -119,6 +120,8 @@ def run(
                 "protected_branches": list(repo_context.protected_branches),
                 "resuming": resuming,
             }
+            if rerun_from:
+                state["rerun_from"] = rerun_from
             if repo_context.high_risk_modules:
                 state["high_risk_modules"] = list(repo_context.high_risk_modules)
             executor = Executor(self.registry, self.settings, run_dir=run_dir,

src/infermatrix_copilot/cli/entry.py

@@ -120,6 +120,10 @@ def main(argv: list[str] | None = None) -> int:
                         metavar="RUN_ID",
                         help="resume RUN_ID, or the most recent run when omitted; "
                              "repo/playbook/task params assert its saved identity")
+    parser.add_argument("--rerun-from", metavar="STEP_ID",
+                        help="with --resume: execute this playbook step and "
+                             "every later step again instead of replaying "
+                             "their checkpoints")
     parser.add_argument("--playbook",
                         help="run a specific playbook by name "
                              "(incl. planner-invisible candidates); "
@@ -215,7 +219,10 @@ def main(argv: list[str] | None = None) -> int:
     if args.resume is not None:
         return copilot.resume_last(args.resume, repo=args.repo,
                                    playbook=args.playbook or "",
-                                   params=parse_task_params(args.task_param))
+                                   params=parse_task_params(args.task_param),
+                                   rerun_from=args.rerun_from or "")
+    if args.rerun_from:
+        parser.error("--rerun-from requires --resume")
     if args.playbook:
         params = parse_task_params(args.task_param)
         return copilot.run_playbook(args.playbook, params=params,