Repository navigation
benchmark: split the legacy run/evaluate chain into B, H, and Apps legacy (B01) - #422
Conversation
|
@amy-why-3459 @zengchuang-hw Please review B01 at |
Dong1017
added
area:harness
|
Review of Checked, no change needed
Needs a change or an answer
|
zengchuang-hw
left a comment
zengchuang-hw
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS for the whole stacked PR yet; the focused B01 review found no new blocking implementation defect.
Reviewed head f0dd3ec3ac8a154f9ff47ecdcad5d69fcf12f5c5 against base bf9aeeb528b5354e53b8f9e9476d3c5f853e7a74. This is my first review of #422, responding to #422 (comment). Acting reviewer: OpenAI dot for zengchuang-hw, under the owner's direct request to publish the completed review. This is a non-approving COMMENT, not Request changes or merge authorization.
B01 findings
- The gate registers only the three exact root-to-Apps forwarder pairs; adjacent targets and another importer remain rejected. Existing forwarders keep overall isolation false, and Apps cannot loop back through them.
- The D008 ownership rows match the code: Benchmark owns sampling order/timing/artifact hashes, while the callback owns transport; Apps supplies repository_identity to the required Harness snapshot port. Sampling adds no H/infra import.
- The frozen MoE entry supplies
metric=read_metric; that reader already applies the positive-finite metric validator. Harness retains execution facts rather than semantic metric rules. - The legacy run is composed in Apps with four required H ports and B-owned rules/labels. The focused Linux checks preserve exit codes 0/1/2, incomplete precedence and golden evaluation.json bytes. H-owned JSON writer byte equality and the integrated H02 execution_spec closure assertion pass.
Before whole-PR approval
The current main-target diff still includes the unmerged #397 Apps dependency, as the PR explicitly states. A focused B01 result must not approve or land that dependency indirectly. Resolve the dependency through its separate independent acceptance, or isolate B01 on an accepted base, then re-review the resulting head/base integration. This review neither advances nor approves #397.
Independently executed
Exact-head source was materialized in an isolated workspace and verified against the Git blob SHAs.
- Linux / Python 3.12:
test_legacy_run_v0_frozen.py,test_legacy_run_split.py,test_legacy_run_forwarder_gate.py,test_target_module_import_gate.py,test_moe_benchmark.py: 99 passed, 1 warning, 10.84 s. test_image_modules.pyplustest_harness_execute.py::test_h02_6_no_device_model_or_network_fallback: 6 passed, 4 skipped, 1 warning. Skips are missing optionalskimage.metrics.- The 12 frozen CLI scenarios passed on Linux, including byte comparisons. The MoE fake-runtime/process checks are not a real NPU benchmark.
- Remote evidence for this exact head: seven completed-success check runs and a successful knowledge-model-policy status (Not applicable). This is distinct from my local focused checks.
Not independently executed: full suite, wheel/sdist/install validation, Windows, or real NPU/service measurements. No campaign PASS claimed. Initial local materialization/PYTHONPATH setup errors were corrected before the exact-source final runs and are not attributed to this PR.
Optional documentation follow-up: the PR body's statement that CI has not run is now stale; distinguish the current remote successful checks from the original Windows-local evidence.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base bf9aeeb528b5354e53b8f9e9476d3c5f853e7a74 and head f0dd3ec3ac8a154f9ff47ecdcad5d69fcf12f5c5 against current origin/main fe16b29388de29708d30ea388b589355540f3172. Governing Issue #414 (B01). Independent reviewer; not the PR author.
Blockers (must clear before re-review)
-
Head is not mergeable onto current main; three-dot is still the #397 stack.
mergeable_state=dirty. Mergingorigin/maininto this head conflicts intests/test_harness_execute.pyatNEW_H_CLOSURE(lines ~39–45: formatting/execution_specmembership vs main after #397). Main already contains #397 (8150ee04) and #400 (fe16b293). Three-dot vs merge-basebf9aeebis still 63 files / +2891/−118, includingapps/api.py,apps/archive.py,apps/result.py,docs/apps-output.md, andtests/test_apps_*— even though those blobs already match main. #414 / Amy / zengchuang all require a B01-only integration after #397 lands. Expected: merge or rebase onto currentmain, resolve the conflict, drop the stack-only comment attests/test_harness_execute.pyL31–35, push a new head whose three-dot is B01-scoped, then re-request review. -
Public Harness signature break vs D018/D022 is unanswered.
harness/scenarios/external.pyL18–28 replacesmetric_reader=withmetric=and raisesTypeErrorwhenmetric_keyis set withoutmetric.harness/runner.pyL20–23 removesexecute/snapshotdefaults and requiresassess/report.docs/contracts/function-owners.mdL25 still says existing external public composed entries keep original semantics and paths per D018/D022. Compat wrappers exist underapps/legacy/run.py, but the H entry points themselves changed. Expected: cite an accepted decision that these H paths are outside that promise, or keep compatible H signatures / H-side forwarders until R01; record the choice in the PR body. -
D008 ownership still marked unapproved while the move already landed.
docs/contracts/function-owners.mdL22–23 still sayowner pending formal reviewforcollect_samples→ B sampling and runnersnapshot→ Apps legacy. #414 required owner rows before the move; merging would publish stale Proposed text. Expected: record formal D008 acceptance (architecture owner or delegated acceptance on the Issue/PR) and update the rows so they no longer say pending, or do not merge the ownership claim until accepted.
Verified on this head (not enough for PASS)
- Ready CI on
f0dd3ec: declarations, full-check 3.10/3.11, installed-package 3.10/3.11, pr-references, lint-and-smoke — all success. - Local Linux exact-head: focused B01 set 100 passed in 9.05s (
test_legacy_run_v0_frozen,test_legacy_run_split,test_legacy_run_forwarder_gate,test_target_module_import_gate,test_moe_benchmark,test_h02_6_no_device_model_or_network_fallback). first_wave_report():isolation_completeremains false; blockers includelegacy run forwarders: 3as required by #414.- #397 Apps blobs on this head match current main; residual issue is integration/scope presentation, not divergent Apps source.
Notes (non-blocking)
- Add a one-line pin that
benchmark.sampling._sha256equalsinfra.artifacts.sha256(writer already hastest_harness_writer_bytes_equal_the_infra_writer; local Linux bytes currently match). - PR body still claims CI has not run / Linux CI pending; Ready CI is green on this head — refresh before the next review.
- Prior peer feedback: amy-why-3459 comment and zengchuang-hw COMMENT both withheld whole-PR approval for the same stack reason; this review agrees and does not clear #397 through this PR.
No merge.
|
#397 已随 Amy 提到的旧 Harness 扩展入口 也请一并处理。旧 |
|
Incremental review of Resolved
Still open
|
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base fe16b29388de29708d30ea388b589355540f3172 and head f6371b5c6adae56173bb2a1ab4d09a9457ad2d62; merge-base fe16b293. Current main has advanced to f7b640f2; a local git merge-tree against it is clean. This is an incremental review: I agree with the earlier incremental review in issuecomment-6850804160 and do not repeat the items it records as resolved.
Blockers
docs/contracts/function-owners.md:22-23still says "owner pending formal review" for both D008 rows, while the moves are already in this diff. #414 requires the owner to be recorded beforecollect_samplesmoves. Record the acceptance (or an explicitly accepted exception) and cite it in those rows; no code change is needed. This is the only remaining blocker.
Issue completion
#414 is partially complete: the split, frozen old-run behavior, exit codes and byte-identical evaluation.json are met; the D008 acceptance is still missing, so the issue requirements should not be closed yet.
Check scope
Read #414/#371 and the existing reviews/comments, the head diff, ownership and gate files; ran the focused Linux/Python 3.12 set (102 passed in the review, 51 passed in the issue check). Did not run the full suite, Windows, wheel/sdist, three-host or NPU/service checks.
| | `harness/scenarios/external` `read_metric`/`validate_metric` | B semantic validation; `execute_command` and retention → H | no whole-file ownership; offline assess executes nothing | | ||
| | `campaign_records` `STAGES`/claim capture/reconciliation | H records; dashboard/report presentation → Apps | first remove the `campaign_records → dashboard_data` reverse dependency; #333 is Proposed and no claim is made that a unified event envelope is accepted | | ||
| | `catalog.repository_pack`/`REPO_TYPES` | D020 neutral raw-data technical leaf; target interpretation stays H | K does not expose the pack reader publicly; `infra` changes only its same-leaf import; seven K bodies/root entries/tests move together | | ||
| | `harness/scenarios/media` `collect_samples` (**added with the old-run split; owner pending formal review**) | B sampling: seed ordering, timing records and artifact hashes | request transport stays with the caller's callback; B hashes artifacts itself and imports no H or `infra` module | |
There was a problem hiding this comment.
|
Incremental review of What changed
Still open
Checks at Once item 1 is citable (and 2–3 are fixed), I'll re-review the new head and expect to approve if nothing else changes. |
|
@zengchuang-hw @amy-why-3459 @david6666666 refreshed head: |
|
@Dong1017 I re-checked
Once 1–3 are in a new head, I'll re-review. |
|
Incremental review of My items from 6872720512 / 6872896898
New in this increment (question)
Checks at Once the body is updated and |
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base fe16b29388de29708d30ea388b589355540f3172 and head db924ea165fbf69a8240745ee2eed1cb3c044eea.
Incremental re-review after head moved past cursor[bot] CR on f6371b5c… (ac69ce49 + db924ea1). Independent reviewer; not the PR author. Governing Issue #414.
Resolved since prior CR
- D008 rows no longer say “owner pending formal review”.
docs/contracts/d008-wave2-ownership-decision.mdis a citable adoption record; it defines RSI-master as the architecture-coordination role delegated by Dong1017, links #414/#413, and separates architecture adoption from PR Approve / merge. function-owners.mdlinks the two B01 rows to that record, clarifies Proposed baseline vs adopted entries, and separates the authority paragraph from the table (blank line fixed).- Extra blank line before
test_harness_writer_rejects_nan…removed. - Local focused B01 set on this head: 101 passed (
legacy_run_v0_frozen,legacy_run_split, forwarder/import gates,moe_benchmark,test_h02_6…). Readydeclarations/lint-and-smoke/pr-references/installed-packageare success;full-check3.10/3.11 were still in progress at review time on this docs-only increment.
Blockers (must clear before Approve)
- PR body is still stale relative to this head. It still says D008 is “proposed / pending formal review”, cites head
f6371b5c, and claims the rows were deliberately not rewritten (e.g. body L7, L12, L19, L51–58, L97). That contradicts the adopted decision record and the currentfunction-owners.mdrows. Expected: refresh the PR body to cited008-wave2-ownership-decision.md, drop the pending wording, and name the current head before re-requesting review.
Notes
- Architecture adoption here is not a substitute for independent implementation Approve; this CR withholds Approve only for the stale body. Prior stack / signature / sha256 items remain resolved as previously recorded.
- Confirm
full-check3.10/3.11 reach success ondb924ea1…before merge.
No merge.
|
Posted by an agent on behalf of @Dong1017 (incremental review only; points already raised by Amy and cursor[bot] are not repeated). Incremental note on head The new Suggest either moving L25-45 to #425 (or a separate decision PR), or stating the cross-issue adoption explicitly in this body and recording acceptance in #414/#413 before merge. (Amy raised this as a question; on re-check I think it should be treated as blocking.) |
History-only rewrite for PR 422. This commit has a single parent (current main) and does not rebase or replay unsquashed #397 / H02 review-stack commits. Method: merge-tree(current main, db924ea); not the raw db924ea tree Official head at preparation: db924ea Current main at preparation: 57f8e8a No file-content change relative to that method's resulting tree. Local candidate only; no push. Refs #414 Agent-assisted-by: Grok Bot (decision record and comment fixes) Agent-assisted-by: Claude Code / claude-fable-5-2 [effort=xhigh] (prior architecture review) Agent-assisted-by: Claude Code / claude-fable-5-2 [effort=xhigh] (architecture review and draft) Agent-assisted-by: Grok Bot (architecture decision integration) Agent-assisted-by: Grok Bot Agent-assisted-by: Codex (author) Agent-assisted-by: Claude Code / claude-fable-5-2 effort=xhigh (author) Signed-off-by: GUOGUO <xwdong1998@163.com>
db924ea to
d5f6e02
Compare
|
Re-checked Resolved since 6873299097
Still open
Checks on I did not run tests this round. |
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 57f8e8a4915b38201d9fa58d4d05acbe3c4199ee and head d5f6e02bc48ae22d2fbb2d9ce718265e5395708c. Governing Issue #414 (B01). Independent reviewer via cursor[bot]; not the PR author.
Prior sole blocker (CR 5902400218 on db924ea) — resolved
- PR body no longer claims D008 pending / cites wrong head. It cites
docs/contracts/d008-wave2-ownership-decision.mdand current headd5f6e02…. function-owners.mdL22–23 link the adopted decision record (no “owner pending formal review”).- History rewrite: single commit on parent
57f8e8a…; three-dot vs that base is 52 files, +1335/−112, B01-only (no #397 Apps tree).
Verified on this head
- Spot-check:
apps/legacy/run.pycomposition,harness/atomic_json.py,benchmark/sampling.pymove, rootevaluation/execution/reportingforwarders remain; sha256 pin test present;first_wave_report()still reportslegacy run forwarders: 3andisolation_complete=False. - Local Linux focused set: 101 passed in 9.05s (
test_legacy_run_v0_frozen,test_legacy_run_split, forwarder/import gates,test_moe_benchmark,test_h02_6…). - Ready CI on
d5f6e02: lint-and-smoke, declarations, pr-references, installed-package 3.10/3.11, full-check 3.10/3.11, dashboard-browser, knowledge-model-policy — all success. - Earlier stack / H-signature disclosure / D008 adoption / sha256 pin remain resolved as previously recorded.
Notes (non-blocking)
- Body’s timed “full-check … in_progress” read-back is superseded by green Ready CI on this head.
origin/mainhas advanced past the PR base (e.g. #426); GitHub stillmergeable=trueand a local merge-tree write succeeded.
No campaign PASS claimed. No merge by this review.
Ordinary merge of main at acc881f, which moved the packaged Knowledge corpus to knowledge/data and the repository pack descriptions to harness/data. No rebase, cherry-pick or history rewrite. The merge had no textual conflict. Git merged docs/architecture.md, docs/module-layout.md and src/inferrsi/cli.py automatically; each keeps the K02 text and paths together with the old-run split changes. Agent-assisted-by: Claude Code / claude-fable-5-2 [effort=xhigh] (merge integration) Signed-off-by: GUOGUO <xwdong1998@163.com>
Ordinary merge of accepted main 37220b1. Preserve the original PR ancestry and canonical D008 record, matching function-owner headers, and package payload bytes. Keep the deferred execution/run suffix AST unchanged and import-isolation regression passing. Verification: changed CLI Ruff E/F/I, diff check, byte/AST invariants and 1 focused import-isolation test passed on this exact staged tree. Prior broader evidence remains tied to its recorded tree. Assisted-by: Grok Bot (deterministic integration and validation; no coding-model call). Signed-off-by: GUOGUO <xwdong1998@163.com>
Local candidate only: bounded Windows regression validation remains incomplete. Signed-off-by: GUOGUO <xwdong1998@163.com>
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 8c321005d50c2ecef61bf20132409d88474a6450 and head e427c42c535ddb49fce0d6943b44044e03f11cf2. Governing Issue #414 (B01). Independent reviewer via cursor[bot]; not the PR author.
Prior blockers re-checked on this head — resolved
- Stack / #397: three-dot vs current
mainis B01-only (51 files, +1301/−110); noapps/api|archive|result/ Apps-output stack residue.mergeable=true; local merge-tree clean. #397 already onmain. - H signatures: CHANGELOG + PR body disclose
metric=/ required ports; compat remains inapps.legacy.runand rootexecution.py. In-tree callers updated; H defaults that would recreate H→B/Apps/infra edges were not restored. - D008 adoption:
docs/contracts/d008-wave2-ownership-decision.mdis citable;function-owners.mdrows link adopted decisions; no “owner pending formal review”. - Stale body (prior CR on db924ea): body cites current head
e427c42…, the decision record, and B01 scope. Timed “full-check in_progress” read-back is superseded by green Ready CI on this exact head (non-blocking).
Verified on this head
- Ready CI on
e427c42: lint-and-smoke, declarations, pr-references, installed-package 3.10/3.11, full-check 3.10/3.11, dashboard-browser, knowledge-model-policy (N/A success) — all success. - Local Linux focused set: 102 passed, 4 subtests (
test_legacy_run_v0_frozen,test_legacy_run_splitincl. sha256/writer pins, forwarder/import gates,test_moe_benchmark,test_h02_6…). first_wave_report():isolation_complete=False; blockers includelegacy run forwarders: 3; noabsent: benchmark/.- Spot-check: Apps legacy composition, H
atomic_json, Bsampling(no H/infra imports), deleted H evaluation/reporting/media shells, root forwarders retained for R01.
No campaign PASS claimed. No merge by this review.
|
Incremental review: How I checked the merges: I compared the net diff against the base before ( Resolved since my last comment (6876763192)
Still open
Non-blocking
|
|
@amy-why-3459 @zengchuang-hw — scope clarification for |
|
Re-check on the same head Resolved: the cross-issue adoption is now on record on both governing issues: #414 6887366262 and #413 6887365381. Both adopt the same five-entry D008 record at this head. So the substance of my blocking item 1 is resolved, and the D008 sections can stay in #422. Still open: the PR body contradicts this.
Once the body is updated, I'll approve if nothing else changes. |
amy-why-3459
left a comment
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
Same head e427c42c535ddb49fce0d6943b44044e03f11cf2. No new commits since my code review in 6887178900. The only change since my last comment (6887410296) is the PR body edit at 01:40Z.
What I checked:
- The body now says it carries the shared five-entry D008 metadata for #414/#413, links the adoption comments on #414 (6887366262) and #413 (6887365381), and keeps the OBS implementation in #425. L15 and L116 no longer contradict
d008-wave2-ownership-decision.md, so the blocking item is resolved. - The non-blocking atomic_json wording is fixed: the body now says it already landed with #437 and is unchanged here.
- Reviews on this head: cursor[bot] approved (5908339081). No unresolved CHANGES_REQUESTED or other blocking review on
e427c42c.
Merge current main9ede1835 (#422) into OBS. Preserve both changelog sections and all five D008 owner rows. Adapt workflow test imports to B01 Apps legacy composition and OBS Apps views. No #428 layout decision is made. Local merge preserved before exact-new-head Linux validation; prior5ccb2b7 test results do not validate this head. Windows isolated controls remain inconclusive. Agent-assisted-by: Grok Bot (deterministic merge resolution; no new coding-model request) Signed-off-by: GUOGUO <xwdong1998@163.com>
benchmark: split the legacy run/evaluate chain into B, H, and Apps legacy (B01)
Governing Issue and delivery slice
Refs #414
Cross-issue architecture adoption: This PR carries B01 implementation plus the shared five-entry D008 decision metadata for #414 and #413. The delegated architecture adoption is recorded on Issue #414 and Issue #413. The OBS ownership sections remain in the joint record; OBS implementation remains in PR #425, not this PR. These records document architecture adoption, not independent implementation acceptance or reviewer APPROVE; both issues retain their evidence requirements.
PR-DIVISION-FINAL.md: move scoring rules from the legacy run/evaluate chain into Benchmark, keep execution facts in Harness with injected ports, compose the oldruninapps/legacy/, reuse the H-owned atomic JSON writer (already landed with PR skill-eval: split the four-file family by function owner and drop the residual (S01) #437; unchanged here), which is byte-equivalent toinfra.artifacts, and movecollect_samplesintobenchmark/sampling.pywith its D008 owner adopted in the RSI-master decision record. Reuse the existing feat: share RSI optimization management across project and benchmark tasks #338benchmark/package; do not rebuild it. Leave the three root modules as exact forwarders for R01.overall isolation_completestays false. Architecture adoption does not imply independent PR approval or gate-policy approval.inferrsi.benchmark.legacy_run_v0:evaluate/_summary,RunRules,trial_metric, and the B-owned conclusion labels.inferrsi.apps.legacy: composition of H execution, B scoring, andwrite_report;evaluation.pyadapts the oldRunSpecinto B rules.inferrsi.harness.atomic_json, already landed with PR skill-eval: split the four-file family by function owner and drop the residual (S01) #437; unchanged here and byte-equivalent toinfra.artifacts.inferrsi.benchmark.sampling:collect_samples(adopted architecture decision; independent PR review remains separate).harness/evaluation.py,harness/reporting.py, andharness/scenarios/media.pyare removed with no forwarding shells. Rootevaluation.py/execution.py/reporting.pyremain exact forwarders toapps.legacyuntil R01.cli.pyonly defersfrom .execution import run_experimentinto therunbranch so ordinary optimization does not import Benchmark (honors the feat: share RSI optimization management across project and benchmark tasks #338 import rule). This is not a CLI migration.prepare/assessproduction implementations,model/validatorsmoves, retiring the root forwarders (R01), CLI migration, OBS implementation (the shared five-entry D008 metadata for task: B01 — 旧 run/evaluate 整链迁入最终归属 #414 and task: OBS — 解散 observability 过渡包并归位记录与展示 #413, including OBS / task: OBS — 解散 observability 过渡包并归位记录与展示 #413 ownership entries ind008-wave2-ownership-decision.md, is included here and recorded on Issue #414 and Issue #413; OBS implementation remains in observability: dissolve the transition package into H records, Apps views, and K (OBS) #425), or settingoverall isolation_completetrue.The #371 design was accepted by the user in issue comment
6841315306. Design acceptance is not implementation approval.@amy-why-3459 @zengchuang-hw: this update is for re-review of the B01 implementation plus shared D008 architecture metadata in the three-dot diff against current
main8c321005d50c2ecef61bf20132409d88474a6450at PR heade427c42c535ddb49fce0d6943b44044e03f11cf2. Both D008 choices are adopted and linked to the decision record; this is architecture adoption, not reviewer APPROVE. The public Harness signature question is recorded below with the D018/D022/FINAL quotations, line numbers, and in-tree call-site evidence; H defaults that would create an H→B edge were not restored. None of these are self-approved.Current head (ordinary merge, read back 2026-11-15 07:42 Asia/Shanghai after the remote update):
e427c42c535ddb49fce0d6943b44044e03f11cf2. Parents are previous B01 candidate0c056b930ac1900298dfa4c71285b57837b3d9cfand currentmain8c321005d50c2ecef61bf20132409d88474a6450(includes #437 engine/sandbox). Tree7f4986ff18e313e364ed95815317c1e1199ebdd9. Current main is an ancestor. Unique three-dot diff versus current main: 51 files, +1301 / −110. Commit count versus main is 4. AuthorGUOGUO <xwdong1998@163.com>with the existing DCO and Agent-assisted-by trailers. This is an ordinary merge that preserves B01 legacy-run boundaries, not a force-with-lease rewrite of the current tip and not a merge of the PR into main. Previous official remote tipd5f6e02bc48ae22d2fbb2d9ce718265e5395708cremains an ancestor of this head. The current tree is not identical tod5f6e02/db924ea.Hosted CI that completed on earlier heads
db924ea165fbf69a8240745ee2eed1cb3c044eeaord5f6e02bc48ae22d2fbb2d9ce718265e5395708cran on those earlier trees and must not be treated as tests of current heade427c42. The decision-record headings## Benchmark samplingand## Snapshot portremain in the current tree (blob links above now point ate427c42c535ddb49fce0d6943b44044e03f11cf2). The shared D008 record is unchanged and still also cites the OBS entries; that cross-issue adoption question remains a reviewer item and is not claimed resolved by this body update.A new hosted run started after the 2026-11-15 07:42 Asia/Shanghai ordinary push of this merge onto current main. First read-back 2026-11-15 07:46 Asia/Shanghai:
lint-and-smoke(job),declarations(job),pr-references(job),installed-package3.10/3.11 (jobs, 3.11), andknowledge-model-policy(success, Not applicable, run) already success;full-check3.10 (job),full-check3.11 (job), anddashboard-browser(job) were still in_progress and are not reported as passed. GitHubmergeable=true;mergeable_state=blocked(reviews and required checks outstanding).auto_mergeis null. No amy-why-3459 or zengchuang-hw Approve exists on this head.Local Linux evidence on the exact current head, from an exact
git archiveofe427c42/ tree7f4986ffon the shared Debian 13.7 / Python 3.13.5 host (completed 2026-11-15 07:15 Asia/Shanghai, before this push): 89 passed + 56 subtests, 0 failed/errors/skips, covering the four exact codex/workswarm export and knowledge-reads nodes plus the B01-impacted legacy-run / forwarder / gate modules. Native 45-second timeouts and integrity checks were not relaxed. This is candidate-host evidence, not hosted CI and not a full-suite pass. Prior Windows timeout records are retained and not re-characterized.#397 and the
maintip this branch already merged (not a pending prerequisite)#397 is already squash-merged to
main. Do not treat it as an unmerged prerequisite of this PR, and do not describe this PR as a vehicle that would land #397.bf9aeeb528b5354e53b8f9e9476d3c5f853e7a74main.8150ee048af6503070fa71768fbf64dffcc1a77ebf9aeeb, 2026-11-13 17:05:44 Asia/Shanghai (UTC+8). The unsquashed review-branch SHAs (f9a2336,d8d86387,0f4c22d, …) are not ancestors of this squash.fe16b29388de29708d30ea388b589355540f3172mainat the1db7d6b1integration. #400, "Use user acceptance criteria, with ceiling-analysis fallback (#400)", parent8150ee04, 2026-11-13 17:56:06 Asia/Shanghai (UTC+8). This was the tip merged into this branch at1db7d6b1, not8150ee04alone. It is not the currentmaintip.57f8e8a4915b38201d9fa58d4d05acbe3c4199eemainas of the 2026-11-14 19:36 Asia/Shanghai history rewrite, still the squash-merge of #428 (committer date 2026-11-14 14:07:08 Asia/Shanghai). It was the single parent of then-headd5f6e02bc48ae22d2fbb2d9ce718265e5395708c. Currentmainis now8c321005d50c2ecef61bf20132409d88474a6450and is a parent of current heade427c42.This branch was started earlier by stacking a local merge of
mainat K01d7de7f3plus the then-head of #397 (f9a2336), then cherry-picking later #397 docsd8d86387, and adding a test-onlyNEW_H_CLOSURErow forexecution_specwhen that helper was still missing frommain. Those unsquashed commits remain in this branch's history, but they are historical. After the ordinary merge of then-currentmain(1db7d6b1, parentsf0dd3ec3+fe16b293), the three-dot diff versus that tip no longer includes the #397 Apps tree (apps/api.py,apps/archive.py,apps/result.py,docs/apps-output.md,harness/execution_spec.py,run_store.py,tests/test_apps_*.py). That exclusion still holds versus currentmain57f8e8a. Merging this PR would land the B01 split, not #397 and not #400.The #397 squash also added
inferrsi.harness.execution_specto main's H02NEW_H_CLOSURE. The previous "gap not cleared on main" / "execution_spec is not on main" comments are obsolete and have been removed. The only conflict on the main merge wastests/test_harness_execute.py: both sides already listed the same closure set; this branch kept that set, took main's wrapping, and replaced the stale comment. No #397 source was edited.At the
1db7d6b1/f6371b5cintegration,git diffversus then-currentmain(fe16b293) was recorded as 51 files, +1281 / −111. At previous official headdb924ea, and again atd5f6e02versus then-main57f8e8a, the unique three-dot B01 diff was 52 files, +1335 / −112. Current heade427c42c535ddb49fce0d6943b44044e03f11cf2versus currentmain8c321005d50c2ecef61bf20132409d88474a6450is 51 files, +1301 / −110. This is not a claim that the current tree equals the rawdb924eaord5f6e02trees; later ordinary merges integrated subsequent main, including #437. Git recorded the B01 moves asharness/evaluation.py→benchmark/legacy_run_v0.py,harness/scenarios/media.py→benchmark/sampling.py, andharness/reporting.py→apps/legacy/reporting.py.This addresses the whole-PR blocker in zengchuang-hw's COMMENT review on head
f0dd3ec3(review5888255798, 2026-11-13 16:39:42 Asia/Shanghai (UTC+8)) and amy-why-3459's item 1 in issue comment6848125700(16:36:32 Asia/Shanghai (UTC+8)): merge acceptedmain(with #397) so the three-dot diff contains only B01, and drop the stack-specific closure comment.Commits (B01 work and integration)
The PR branch now contains four unique commits versus current main, listed at the end of the table. Earlier SHAs below remain provenance;
d5f6e02and the later ordinary merges are ancestors of the current tip. The unsquashed #397 / H02 review-stack commits were not replayed.85f4521333c451358ace7c233d34e1ada191f68cmainat K01d7de7f3and then-#397f9a2336. Local base only.4380add202fb3a8feeff7b5d60c11e1f17a53ab5Agent-assisted-by: Claude Code / claude-fable-5-2.77144a6741cfaaee4072b22bd186fa3d9778ad07main/ H02 squashbf9aeeb(before #397 landed).execute_trial/execute_commandstill acceptenvironment.0788ffa6540a9c2fdbda83b8084cc0f87d376c76-xof then-#397 docsd8d86387.f0dd3ec3ac8a154f9ff47ecdcad5d69fcf12f5c5NEW_H_CLOSUREaddition, made whenexecution_specwas not yet onmain. This was the previously reviewed and CI-green head.1db7d6b10b4d7eb159874b68c44962514df7fd38main(fe16b293, includes #397 squash8150ee04and #400). Conflict only intests/test_harness_execute.py, resolved as described above. No reset, rebase, or dropped B01 work.f6371b5c6adae56173bb2a1ab4d09a9457ad2d62test_sampling_sha256_matches_the_infra_hasherpinsbenchmark.sampling._sha256toinfra.artifacts.sha256. Historical; no longer the branch tip.ac69ce491b4966f20fc492bfaee89d0d3db98d2bdb924ea165fbf69a8240745ee2eed1cb3c044eead5f6e02bc48ae22d2fbb2d9ce718265e5395708c57f8e8a; tree268da5969fd28bb535dc4bd8e0225e7ac3637108). Remains an ancestor of the current tip.a663fcc71c0c5f6b897bf0adc014d7eee265ada40c056b930ac1900298dfa4c71285b57837b3d9cfe427c42c535ddb49fce0d6943b44044e03f11cf2OPEN,head.shathis SHA,commits=4,changed_files=51,+1301 / −110,mergeable=true). Parents0c056b93and current main8c321005d50c2ecef61bf20132409d88474a6450; tree7f4986ff18e313e364ed95815317c1e1199ebdd9. Author GUOGUO;Signed-off-by: GUOGUO <xwdong1998@163.com>; Refs #414.Review items (citable D008 decisions and Markdown/nit fixes recorded)
These are listed for independent reviewers. Previous official head
db924ea165fbf69a8240745ee2eed1cb3c044eeaadded the citable record, which remains in current heade427c42c535ddb49fce0d6943b44044e03f11cf2, correct per-row/header status, a blank line after the ownership table, and the test whitespace nit. No Harness defaults or gate-policy approval are added.D008 owner rows now cite the decision record and its per-row anchors. RSI-master means the architecture-coordination role delegated by requesting maintainer Dong1017 (xinwen dong), not an additional GitHub reviewer/approver or a claim of legal repository ownership.
collect_samplesbelongs to B; transport remains caller-injected.(repo, exclude) -> dictcontract, Apps legacy binds it, infra/targets implements it; H has no default.The five record decisions are adopted under delegation; other baseline table entries are not blanket-accepted. Independent implementation review remains separate. Authority text is now separated from the table by a blank line.
Public Harness signatures vs D018/D022 (amy-why-3459 item 2; cursor[bot] review
5889339860blocker 2). This PR does change the Harness ports relative to pre-B01main(bf9aeeb). That is not left as an uncited "in-scope decision"; the contract text, the old call surface, and the in-tree compatibility path are recorded here.Contract text (exact, with line numbers on this head unless noted):
docs/contracts/function-owners.mdL25: "Existing external public composed entries keep their original semantics and paths per D018/D022, and an owner change cannot be used to delete an old promise." The same file L15 already says ofharness/runner: "old run unchanged; migrating oldevaluatedecouples the runner first; new H scoring cannot be retained by attaching alegacylabel." L19 assignsread_metric/validate_metricto B andexecute_command/ retention to H, "no whole-file ownership."docs/contracts/current-design.mdL138–L140: "old entry points, arguments,Path, stdout, stderr, and old run 0/1/2 stay byte-for-byte compatible. New Apps naming/envelope/exit mapping is fixed by D022, but K01/H01 do not implement a CLI."docs/contracts/isolation-errors-and-cli.mdL113 / L117–L119: "Path-by-path retention of old entries"; "K01 / H01 do not implement the new CLI, and they do not touch the consoleinferrsi=inferrsi.cli:main,__main__,arguments.parser/Path/ defaults, old root / harness evaluation, oldrun0 / 1 / 2 / stdout / stderr,plan/--dry-run, existing grade behavior, or plugin entries. Oldrunjudgment still gives incomplete priority over failed, and the arrangement of reasons does not change meaning. Oldevaluatemigration follows D018-B4:B.legacy_run_v0keeps the rules, andApps.legacy.evaluationassembles. First the runner detaches from the internalevaluate, then the oldevaluation/harness.evaluationcompatibility edges are reviewed leaf by leaf. New H must not score through an old shell." D023's preface at L115 says it "does not change the compatibility commitments in section 1."### B4heading in that 118-line file).docs/contracts/gate-policy-runstore-and-first-prs.mdL52–L56: "Blegacy_run_v0.pykeeps the only oldevaluaterules; Appslegacy/evaluation.pyadapts old RunSpec/args to B-local DTOs and returns the original dict"; "new H delivers only fixed execution facts, and evaluation is composed by the EvaluationPort that Apps injects; the old run composition path keeps the original call behavior in Apps legacy." "Re-exporting oldharness.evaluationis then a compatibility endpoint, not an H core dependency"; "This is not exemption by labelling business codelegacy."PR-DIVISION-FINAL.mdB01 L74–L75 (accepted 2026-11-12 11:35–11:36 Asia/Shanghai): "规则进 B，执行事实留 H，组合进 Apps legacy"; "runner不再内嵌判分与写报告，改由 Apps legacy 组合；scenarios/external的执行留 H、metric 校验进 B"; "旧 harness 路径不留会形成 H→B/Apps 的壳；根evaluation.py/execution.py/reporting.py可暂留精确转发，到 R01 再删。" Acceptance at L77 includes "旧run退出码 0/1/2、incomplete 优先、同一夹具下evaluation.json字节不变." B01 explicitly does not retire those root forwarders or migrate the CLI (L78). FINAL's only other D022 mention (A-entry L124) is that A-entry will not deliver D022's new evaluate subcommands or a first-wave DeprecationWarning; that is not a B01 deliverable.Old call surface at H02
bf9aeeb(the pre-B01 public H / root APIs):harness/scenarios/external.pyL12–L20:execute_trial(..., metric_key, *, metric_reader: Callable[..., float] = read_metric, environment=None). The default importedread_metricandvalidate_metricfrominferrsi.model.protocols.metrics(L8) and appliedvalidate_metric(metric_key, metric_reader(...))(L30). That is an H→model/B metric-rule edge.harness/runner.pyL19–L25:run_experiment(spec, environment, *, execute=execute_trial, snapshot=repository_identity)— defaults importedexecute_trialfrom H andrepository_identityfrominfra.targets(L12, L16); the body also importedevaluateandwrite_reportfromharness.evaluation/harness.reporting(L14–L15), which is the inlined scoring/report path B01 must detach.execution.pyL18–L19 was alreadydef run_experiment(spec: RunSpec, environment: dict) -> Path:and forwarded to H withexecute=/snapshot=filled in. CLIrunimported this root entry (cli.pystill does, L498–L500).docs/architecture.mdL251–L264 onbf9aeebimportedrun_experiment/execute_trialfrominferrsi.harness.runnerandinferrsi.harness.scenarios.externaland usedexecute=partial(execute_trial, metric_reader=read_engine_metric).bf9aeeb:tests/test_dashboard_workflow.py,tests/test_flare_adapter.py,tests/test_module_boundaries.py,tests/test_runtime.py(run_experiment as run_with_adaptersplus Hexecute_trial),tests/test_trial_checks.py(from inferrsi.harness.runner import run_experimentandrun_experiment(spec, {"backend": "cpu"})with no ports), plus docs as above.optimization_session.pyalready called Hexecute_trialwithmetric_key=None(execution facts only).What this head keeps, and what it does not restore:
metric=on Hexecute_trialis optional (Noneby default,external.pyL18).TypeErroris raised only whenmetric_key is not None and metric is None(L27–L28). H no longer importsread_metric/validate_metric. Restoring a defaultmetric_reader=read_metricor defaultexecute=execute_trial/assess=evaluate/report=write_report/snapshot=repository_identityon the H callables would re-introduce the H→model / H→B / H→Apps / H→infra-targets edges that FINAL L75 forbids ("旧 harness 路径不留会形成 H→B/Apps 的壳") and that D018-B3 L56 and D022 L119 require to be composed by Apps, not kept as an H core default. Those defaults were not restored.apps.legacy.run(L19–L57):execute_trialstill acceptsmetric_reader=(defaultread_metric) and wraps H withmetric=partial(trial_metric, reader=metric_reader);run_experimentstill defaultsexecute=execute_trialandsnapshot=repository_identityand suppliesassess/report. Rootexecution.pyL18–L19 is unchanged in signature vsbf9aeeband now forwards to that Apps composition.docs/architecture.mdL257–L269 retargets the compose example tofrom inferrsi.apps.legacy.run import execute_trial, run_experimentand keepsmetric_reader=. CHANGELOG L15–L21 discloses the H port change and points callers atapps.legacy.run.In-tree call-site verification on this head (no extra code fix required):
cli.pyL498–L500 →inferrsi.execution.run_experiment(spec, environment);tests/test_runtime.pyL19 / L104 / L576 same root entry;tests/test_trial_checks.pyL11 / L105 / L114,tests/test_flare_adapter.pyL10 / L303 / L310,tests/test_dashboard_workflow.pyL8 / L29–L31, andtests/test_module_boundaries.pyL13 / L62–L65 all useinferrsi.apps.legacy.run(run_experiment(spec, env)orpartial(execute_trial, metric_reader=...)).execute_trialcallers that remain on the H module are execution-fact paths, not the old metric_reader composition:harness/optimization_session.pyL43 and L89–L90 passmetric_key=None;tests/architecture_real_ports.pyL82–L88 andtests/test_moe_benchmark.pyL420–L421 do the same;tests/test_legacy_run_split.pyasserts the newTypeErrorand themetric=path. The MoE task (benchmark/tasks/moe-forward-npu/benchmark.pyL258–L261) receives an injected executor and itself passesmetric=read_metricwith"metrics.suite_latency_ms", which is also whatdocs/moe-forward-benchmark.mdL50–L52 describes. Session injection of the bare H executor atoptimization_session.pyL234 is therefore still valid for that task.from inferrsi.harness.scenarios.external import execute_trialand passesmetric_reader=or a non-Nonemetric_keywithoutmetric=, orfrom inferrsi.harness.runner import run_experimentand callsrun_experiment(spec, env)without the four ports, will getTypeError. No such remaining in-tree production, test, or documented compose path was found on this head. The supported replacement isinferrsi.apps.legacy.run(or the unchanged rootinferrsi.execution.run_experiment(spec, environment)). This is not a request to restore H defaults.Independent verification and test evidence
All times below are Asia/Shanghai (UTC+8). Machine-wide cap of 2 pytest workers was respected.
f0dd3ec3. Remote CI on that head: all listed checks passed (declarations,full-check3.10 and 3.11,installed-package3.10 and 3.11,lint-and-smoke,pr-references, andknowledge-model-policy= Not applicable). The earlier PR-body line that "CI lanes have not run" / "POSIX / Linux CI pending" is obsolete.evaluation.jsonis tested, not only inferred.tests/test_legacy_run_v0_frozen.py::test_old_run_keeps_exit_code_and_evaluation_bytescompares CLIevaluation.jsonbytes to the frozen fixtures, normalizing\r\nto\n. Those tests are part of thefull-checklanes that passed onf0dd3ec3. zengchuang-hw independently reported the 12 frozen CLI scenarios passing on Linux, including byte comparisons, plus 99 passed in the focused B01 modules, in review5888255798.0788ffa, 15:43:16–15:56:05,-n 2, then a test-only closure fix and exact-node retest 16:03:10–16:04:11 (1 passed). That work predates Publish standardized Apps run output #397 landing onmain.f6371b5c, after then-currentmain(fe16b293) was merged. Waiter PID 42268 started the focused set at 18:32:01 once no other pytest was live, and finished at 18:37:33, exit 0. Command (Python 3.13.5, pytest 9.0.2,-n 2) on exact then-headf6371b5c:tests/test_legacy_run_v0_frozen.py,tests/test_legacy_run_split.py(includes the sha256 pin),tests/test_legacy_run_forwarder_gate.py,tests/test_target_module_import_gate.py,tests/test_moe_benchmark.py,tests/test_harness_execute.py::test_h02_6_no_device_model_or_network_fallback, andtests/test_harness_run_store.py::test_h01_6_new_h_closure_reaches_no_legacy_k_b_or_apps_module. Result: 100 passed, 2 skipped in 301.39s (0:05:01). Log:wave2-prep/b01-focused-after-main397-t248.log. SSH push 18:37:33–18:37:43,PUSH_EXIT=0. Result filewave2-prep/b01-wait-recheck-push-t248.result.txt:tests_exit=0 push_exit=0 local=f6371b5c… remote=f6371b5c…. At that time,git ls-remote upstream refs/heads/impl/b01-run-evaluateandgh pr view 422(headRefOid,mergeable=MERGEABLE;BLOCKEDwas the outstandingCHANGES_REQUESTEDreview, not a merge conflict) confirmedf6371b5c. That SHA is no longer the branch tip. No full-suite repeat locally. Hosted CI ondb924eaand the new run ond5f6e02are described under Current head above; thedb924eaReady run is not evidence for the new merge-tree.f6371b5c(read back 18:38:40 Asia/Shanghai (UTC+8), immediately after the push):declarations(16s),lint-and-smoke(14s),pr-references(7s), andinstalled-package3.10/3.11 already success;knowledge-model-policyis Not applicable;full-check3.10 and 3.11 were still pending. That pending status applied only tof6371b5cat that read-back and does not describe previous official headdb924ea, laterd5f6e02, or current heade427c42(see Current head above). Ready CI on the previous reviewed headf0dd3ec3was already fully green, including the frozenevaluation.jsonbyte comparison.Out of scope / not claimed
metric_reader=or defaulted Hrun_experiment. H defaults that would create an H→B edge were not restored.