Repository navigation
fix(benchmark): add recoverable FrontierCS-2 compatibility v2 patch - #7958
Conversation
Signed-off-by: song <liusongstep@gmail.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2d695f6770
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Merge current main while preserving the frozen v1 archive. Write and fsync payloads before atomic publication. Replace streamed uploads through a sibling temporary file so read-only artifacts can be updated without broader authority. Validation: 17 opt-in checks passed; one real-Docker case was skipped because Docker is unavailable. Real interrupted-write and local upload regressions fail on the old source and pass after repair. Fresh pinned-base patch application, source-byte parity, frozen v1 identity, Ruff and public-boundary checks passed. Signed-off-by: song <liusongstep@gmail.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max
当前完整提案 a87863c9e529079fae0b01957360c019784f56ee 代码评审为 APPROVE,未发现新的阻塞代码缺陷。实际 base 为 ec5c9702c79cbc68ab994b18031e9069b0935b59。旧 2d695f6 的部分 payload/只读覆盖缺陷已独立确认修复;旧失败和真实 Docker 未验证范围继续保留,批准不清除质量及合并门。
动机 / Motivation
显式使用单机兼容源码的实验操作者,需要在桥接重启或本地回执失败后继续原提交。
v1 无法恢复内存状态,重复 final 也可能在已接受后写本地回执失败;v2 保留原 capability、deadline、机会和最终选择,让同一提交可以继续恢复。
当前独立验证覆盖了真实 HTTP 重启/故障恢复、幂等 final,以及 OS 中断后的完整重试;实际发出的本地上传程序也能反复替换只读目标并保留 mode 和当前 UID。
本 PR 只修显式研究工作区,不安装产品能力、不运行模型或裁判、不回填历史时间或工作区,也不宣称历史分数和长期效果已经资格化。
真实 Docker namespace/solver UID、外部 controller 与完整沙箱仍须单独资格化,源码恢复正确不等于历史试验被修复或出现净收益。
改动思路 / Approach
保留冻结 v1,另立显式 v2 patch 是合适的版本边界;恢复权威留在既有 Trial,native worker 的不确定结果仍交原 controller 核对。当前 PR 交付显式兼容源码中的提交/回执恢复和异步运输,不增加产品控制面 owner,也不自动重评分。
不修复会保留内存状态丢失、重复 final 回执错误和阻塞 event loop;只改客户端不能恢复服务端承诺。当前复用既有 HTTP、队列和 Docker adapter,将 admission 与幂等响应持久化后才排队,inspection 写失败不撤销已提交的权威。运行中 worker 不确定时保留实际成本与已消耗机会,阻止新的 train,要求原 controller 核对后继续。
两项后续修复都留在本地 IO owner:tar 在同目录临时文件完整写入、设权限和 fsync 后原子发布;streamed file 也原子替换 sibling,避免截断只读旧目标。无需引入产品插件框架或第二套预算/receipt authority。这里用文字说明已验证的分支,不把流程图或状态字段当真实采用。
具体改动 / Changes
独立接受依据是 benchmark/frontiercs2-single-host-versions.md,spec_revision f3aee4586d87b678581a22d23b61fba7e49d4672,以及修改前 archive placement;修复后的 README 不作为自身唯一 oracle。条款锚点:
- frozen-version:implemented。从固定提交物化五文件,check/apply 当前 patch;原五文件身份检查通过,历史分数与归档不重写。
- separate-repair:implemented,有界资格。当前源代码和客户端/HTTP/文件恢复可用;没有将本地文件程序测试称作真实 Docker namespace 资格。
- external-authority:implemented。controller、native grader、隔离与实际采用保留外部责任;不确定 worker 不自动重放或返还机会。
- honest-qualification:implemented。保留历史缺失文件/时间、power-loss 和并发 owner 限制、当前 Docker skip 及质量失败。
完整 base→head 四文件 +964/-0:说明 +131、执行补丁 +373、显式验证 +460。相对旧审查,主要新增两处同目录原子文件发布和两项有缺陷敏感性的回归;主干新历史属于实际 base,不计为本 PR 的运行改动。
关键代码讲解
DockerEnvironment._run使用真正异步子进程 IO,超时/取消回收 Docker client。它不保证容器内后代停止;hard-stop 仍归 controller。run_final保留远端已接受的原选择与接受时间,原子写本地 receipt;本地失败显式 pending recovery,重复 final 可修复缓存而不新增远端选择。Trial._restore/persist验证原 arm、私有权限、payload identity 与完整字节;恢复 token/deadline/计数,inspection 与权威分离,运行中 worker 转为 reconciliation required。Trial.request在 durable commit 后排队;_request使用临时文件发布 payload,已存在的目标仍核验字节和 symlink。完整 orphan 可以复用,不完整临时文件不能毒化最终路径。verify_v2.py用真实 loopback HTTP/客户端、OS 文件大小限制中断 writer,以及实际发出的上传 Python 程序检查行为;Docker fixture 另保留真实 namespace/UID 条件,未用本地替代来伪造它。
对主干的风险 / Risks
独立当前 suite 为 17 PASS / 1 SKIP。覆盖重复 final、receipt IO 失败、原接受时间、重启后的 token/deadline/cost、train-only feedback、错误 capability/purpose、机会/时限、queued handoff、运行中 worker reconciliation 和实际子进程异步/超时。没有执行模型、任务、native grader 或历史 candidate。
两项最强反例重新执行了当前测试与旧源码的对照:旧补丁在真实 writer 被 OS 限制中断后,重启再上传会 HTTP 409;旧上传程序反复写 0444/0555 目标会 PermissionError。两个旧案例均真实失败,当前完整 suite 均通过;期望失败控制保留,不计作“基线全绿”。已注册 payload 篡改拒绝、完整 orphan、commit 失败回滚和 checkpoint/final 分离也仍通过。
真实 Docker 项明确 SKIP:本机 daemon 存储返回 IO 错误,未拉取镜像、修改 daemon 或启动容器。只读覆盖测试执行实际生成程序并检查真实文件内容/mode/当前 UID;路径可写前提与容器运输被适配,不能认证容器 namespace、solver UID 或沙箱。该限制保留;当前文件发布不变量由独立实际 IO 与旧缺陷控制覆盖。
完整产品、包装、CI 树相对当前 base 未变;成功的完整 capability catalog 对照逐字相同。未显式应用 patch 时,产品无新增配置、指令、准入或费用。Ruff、冻结五文件 identity、patch check/apply、编译/双语/差异和 public-boundary 检查 PASS。
当前标准 premerge 5 direct / 19 selected,1 lifecycle FAIL,加 benchmark 人工 hold;raw ratchet advisory FAIL 也保留。另通过仓库原生 canary runner 在不可变 base 和当前 head 执行相同 lifecycle check,重现相同 KeyError: evidence 完整失败块,只替换 checkout 前缀后逐字一致;所有产品失败路径未改。早先直接调用遇到默认 runtime 路由冲突,是不同环境观测,不替换这份原生匹配对照。独立当前行为证据足以批准代码,required premerge、Docker skip 和严格 CQR 仍按失败保留,不降级必检。未查询或等待远端 CI。
语义与 CI 对齐
private recovery v2、accepted_at、pending recovery 仅属于这个实验兼容实现;无通用 quota/Todo/scheduler 新词汇或执行权。恢复记录保存原事实,inspection 是派生视图;恢复后的机会/成本与原 worker 对账是实际强制边界,不能称为可忽略建议。冻结 v1 与显式 v2 的兼容分离有真实历史消费者,不为减少版本数而删除归档,也不把 additive receipt 字段当采用或收益证明。
我的整体评价 / Verdict
APPROVE 当前完整代码,作为 justified_increment。 long_horizon 与 user_experience 在源码恢复边界 improved:原提交可以重试,不重复发放机会或强迫操作者重建试验;真实 Docker、controller/sandbox 和历史收益仍明确未资格化。没有提出新的确认轮次、安装或试验授权。
保留冻结 v1,另立显式 v2 patch 是合适的版本边界;恢复权威留在既有 Trial,native worker 的不确定结果仍交原 controller 核对。当前 PR 交付显式兼容源码中的提交/回执恢复和异步运输,不增加产品控制面 owner,也不自动重评分。
有界重构已应用:沿既有 JSON/Trial/transport owner 复用原子写入和异步进程 helper,tar 的特殊发布需求留在同一局部边界;更大的 controller 重构不是本 PR 前提。已检查 CLI、HTTP、scheduler 投影与脚本消费者,没有渲染页面变更,视觉评审不适用。视觉未验证范围:none。严格 CQR/premerge、benchmark 人工门与 exact-head readiness 仍分别判断;未合并、未 dismiss,不追认旧头资格。
English verdict: APPROVE — a87863c. The complete opt-in repair has 17 passing tests and one explicit real-Docker skip. Current regressions reproduce both old failures and pass after the atomic-publication fixes. Native matched base/head evidence attributes the unchanged lifecycle failure; premerge, strict CQR and benchmark holds remain. Local emitted-program tests do not qualify Docker namespace/UID, controller safety or historical benefit.
本轮修复 · 2026-11-03
提交:
a87863c9e529079fae0b01957360c019784f56ee;已合入 mainec5c9702c79cbc68ab994b18031e9069b0935b59,冻结 v1 字节保持不变。两项 review finding 均已复现并修复:
0444/0555文件保留模式与 owner,且不需要打开旧只读文件写入。当前 opt-in 检查为 17 PASS / 1 SKIP。跳过项是本环境不可用的真实 Docker;本地实际上传程序检查不代替容器 namespace/UID 验证。两个新增回归在旧实现分别复现 HTTP 409 与 PermissionError,修复后通过。重新从精确 v1 应用补丁、逐字节源码比对、冻结五文件 identity、Ruff、RFC 与 public-boundary 检查通过。此次不调用模型、任务或评分器,也不补写历史结果。
原提案与此前验证记录
下方保留原设计与历史验证;本轮新 head 的验证范围以上方记录为准。
The single-host compatibility source preserved by #7954 cannot recover its in-memory trial authority after restart, repeated final submission can fail while rewriting an already accepted receipt, and its asynchronous Docker interface blocks the event loop. The published pilot also lacks final-acceptance timestamps and complete workspace captures.
This PR adds an opt-in v2 patch against the exact v1 commit
f3aee4586d87b678581a22d23b61fba7e49d4672, plus application instructions and synthetic regression checks. It preserves the frozen archive and historical score attribution. The patch persists private recovery state before queue handoff, retains the original capability/deadline/opportunity counts and final selection, makes final receipt recovery idempotent, and records the original acceptance time. Interrupted native workers require controller reconciliation rather than automatic regrading. Docker subprocess IO is asynchronous; writable tmpfs uploads stream through unprivileged container exec, addressing a separately reproduced case where a copy command succeeds without visible files.Validation:
loopx/; it does not cover this research patch, whose added vocabulary is local compatibility state.KeyError: evidence, reproduced on clean maine263c3a18512316223eff895def6e8dbd46a2a6d; the separate oversizedrender_canonical_todo_sectionsadvisory has byte-identical stdout on that baseline. The full gate remains failed, not waived, and benchmark-sensitive maintainer review remains required.Scope and remaining gaps: this is a research-workspace patch, with no LoopX product imports, installed capability, active benchmark CI entry, model/task/scorer run, or scoring change. The original external controller still owns its single-writer lock, isolation, worker/result reconciliation, final dispatch and workspace capture. Historical missing files/timestamps are not backfilled. Runtime recovery credentials remain private. No score, treatment-adoption or long-horizon benefit is claimed.
Bounded refactoring: shared asynchronous subprocess handling and atomic recovery writes remain with their existing transport/trial owners. Larger controller and native-grader changes are outside this repair.