Use this procedure before declaring a user-facing feature complete. Unit, architecture, specification, lint and format checks establish implementation correctness within their tested scope; fresh-agent acceptance establishes whether an agent can achieve the user's declared task through the shipped CLI and skill. Both are required for a feature verdict. This document is developer orchestration guidance, not an additional operator skill.
Keep acceptance a small, manual check of the declared task on the user's clips. Do not build or maintain a formal evaluation harness, scoring framework or reporting platform for it. A clear brief, fresh operator, saved CLI evidence and the clip report template are sufficient; the regression cases below guide applicable development changes rather than prescribing extra work for every custom clip.
The development agent owns scope, setup, evidence review and the final verdict. Fresh operator agents know only their task brief, the candidate executable, the shipped skills/audio-cli/SKILL.md and its references, public CLI help/results, and their own artifacts. Do not give them this document, repository source, developer instructions, tests, implementation plans, issue/PR discussions, historical results or another operator's conversation. Do not fork the development conversation. Keep the shipped skill outside .agents/skills/; do not install it into the development workspace to conduct acceptance.
Record the candidate commit, executable path/version, skill path, fixture identity, full-source or range scope, required capabilities, required deliverables, and the checks that decide each requirement. Bind the executable to the candidate: retain the build/install command and artifact digest, or verify that the editable installation resolves to that exact checkout and record any uncommitted diff. A matching version string or executable filename is insufficient. Hash the skill entry point and every shipped reference/support file; retain that exact snapshot with its candidate origin. Freeze the candidate during testing and verify identity again afterward. Distinguish a user requirement from an optional diagnostic. An added timing/export check must have its own verdict; its failure must not be reported as a total ASR or diarization failure. Conversely, a required capability cannot become optional after it fails. Declare any accepted fixture-specific limits with the user's prior decision and exact artifact/configuration identity.
When the user authorizes recovery, declare its eligible failure conditions, permitted parameter or stack changes, attempt limit, preserved scope/capabilities, new output destinations and success checks before execution. Give each attempt its own capability and delivery verdict; the initial failure remains recorded even if a later authorized attempt succeeds. A recovery branch does not authorize unrelated repairs, silently reduced requests or internal model substitution. Historical tests without that authorization retain their original stopping rule and verdicts.
Use the full matrix below for changes to transcription, enhancement, shared command behavior, package readiness or the shipped skill. Add a task-specific case for the new feature and for any newly fixed acceptance defect; repeated media cases alone do not prove that a new command or flag works. For a narrower change, record which cases are affected and why the others are not required. Documentation-only changes need link/content verification rather than another model run, unless the changed document is operator guidance whose usability is the feature under test. This scope decision is recorded before execution.
- Inventory the main checkout, candidate worktree, active work and existing evidence before cleanup. Remove only completed-task disposable files; preserve private source media, accepted baselines and unrelated work. Do not overwrite an earlier test directory or remove evidence merely to make the workspace look clean.
- Run the implementation checks appropriate to the change and
uv run --extra dev pre-commit run --all-files. Record skips and failures with their scope. Resolve the actual current test path from the test layout; do not treat an unexecuted check as passed. - Create
tests/artifacts/acceptance/<date>-<candidate>/with a short scope note and a directory for each tested clip. Research runs may instead usemodel_tests/benchmark_runs/. Record original input SHA256 and byte size before and after execution. Keep raw private recordings, transcripts and logs untracked; publish a compact development result undermodel_tests/benchmark/results/YYYY-MM-DD-<topic>.json. - Retain each CLI invocation's exact executable/arguments, working directory, exit status and raw stdout/stderr using the available tool/terminal records and saved files. Finish each command and inspect its result before dependent work, including report writes. Retain available non-audio error evidence; disclose missing records or timestamps when relevant instead of reconstructing them. Do not create custom recorders, trace auditors or a run framework for this procedure, and do not infer success from filenames.
- Start separate agents with fresh contexts and an external working directory outside repository instruction discovery, separate from repository evidence storage. Supply absolute paths to the candidate, skill snapshot and output directory; do not ask operators to infer the candidate from a shell's current
audio. Do not launch the operator from a repository artifact directory whose ancestors contain developer instructions. If the agent system cannot isolate the context, report the acceptance setup blocked rather than label an inherited development context fresh. - Run heavy inspect/enhance/transcribe/model-provision operations sequentially to avoid memory contention. If an existing shared execution lock is used, hold it for the CLI child's lifetime. Do not turn task acceptance into a performance benchmark.
The fixture inputs are optional private files under tests/artifacts/media/. If a fixture is missing, mark its case blocked and request the source through the normal user workflow. Do not download a substitute, silently use a shorter clip or declare a skipped fixture passed. Package readiness and provisioning must use the CLI's explicit doctor/packages lifecycle; no manual environments, weights, lock edits or provider setup. Provisioning explicitly required by a ready-state plan is setup, not permission to retry an unsuccessful model run.
Give each operator the following brief with placeholders resolved. Include only its matrix row and task-specific requirements, not the development gate or other cases' results.
Task: <user-facing operation and required deliverables>.
Candidate executable: <absolute path>.
Shipped skill: <absolute path>/skills/audio-cli/SKILL.md.
Original input: <absolute path>. Evidence output: <new absolute case directory>.
Operator cwd: <new external absolute directory with no repository instructions>.
Required scope/capabilities: <full source or explicit range, stack, wants, profile>.
Optional diagnostics: <explicitly optional checks, if any>.
Authorized recovery: <none, or eligible trigger, allowed changes, attempt limit and delivery checks>.
Learn usage only from this brief, that skill/references and public CLI help/results. Do not inspect repository implementation, developer docs, tests, issues, old results or other agents.
All audio/model processing, inspection, transcription, enhancement, export and provisioning must use the supplied CLI. Non-audio filesystem, command logging and JSON inspection tools are allowed. Do not invoke FFmpeg, an audio library, a provider SDK or another processor directly.
Preserve the original and any canonical JSON. When transcription is required, transcribe original media before enhancement; do not add ASR to an enhancement-only task. No filler/editorial cuts, manual text cleanup, invented timing, unplanned backend substitution or reduced request. An authorized changed attempt uses new canonical, diagnostic and export destinations.
Save every CLI command's arguments, raw streams and exit status, all canonical results, native readable exports and available diagnostic logs. Complete and inspect each command before dependent work; do not batch a metadata write with model submission.
On the first unexpected failure, unsatisfactory required capability or required-output refusal, preserve evidence and stop the dependent workflow unless the observed condition matches the explicitly authorized recovery branch above. Execute only that branch's permitted change, preserve the failed attempt and inspect the new result before proceeding. Outside that branch, do not retry, repair, drop flags, switch stacks or use an alternative. Report the exact blocker and leave unattempted steps explicit. You may finish non-audio inspection/reporting of already saved evidence. Do not make further CLI calls after an unplanned failure or exhausted recovery. Independently predeclared cases may continue in their own agents.
Report required delivery checks, optional observations and operator/protocol/reporting errors separately, with actual artifacts and evidence. A reporting error does not relabel already validated outputs. Save Markdown or JSON directly with a file-writing tool such as apply_patch; do not construct a script or nest report prose inside executable strings. Do not score accuracy or listening quality without a reference or human judgment.
Deliberate negative tests are an exception to the unexpected-failure stop rule only when declared in advance: a specified refusal is their expected success condition. Do not retrofit that label onto an unexpected refusal. Likewise, a valid partial result or abstention can satisfy a test of honest partial-result handling while failing a task that requires complete timing. Keep the two verdicts separate.
An optional numerical miss or unmeasured diagnostic is an observation, not a failed required delivery. LRA, residual-noise numbers and similar preferences do not gate a custom clip unless the user made their attainment a requirement before execution. An actual unexpected command failure still invokes the stopping rule, even during an optional check. Optional findings do not automatically create a fine-tuning task or backlog item.
For the user-selected clipping policy, add a fresh operator case on the full original Qwen D2 request below. Preserve its default 0.501 ms attempt, then allow one explicit --alignment-max-overrun-ms 20 attempt only after a recorded boundary rejection within that cap. The operator must identify the originating segment/unit, original and unit bounds, measured overruns and effective limit from public results, then plan and run the changed request with the same required capabilities and new destinations. Require retained raw diagnostics, auditable original/applied bounds for every recorded correction, actual requested outcomes and native timed Markdown/SRT/VTT when their prerequisites pass. If no eligible rejection appears, report that the recovery branch was not exercised; do not invent a failure.
Separately predeclare an alternate-stack case when that choice is authorized. A fresh operator may receive the default failed canonical and its public diagnostic bundle as named inputs, without the earlier operator's report or development discussion. It discovers and plans a supported alternative with the original source, scope and required capabilities, then makes one new run and evaluates that independent result. Preserve both canonical records; do not splice another model's timing, wording or anonymous speaker labels into the first. Any missing provisioning uses the public package lifecycle and stays within the declared setup scope.
Controlled checks must cover invalid or inapplicable limits, default compatibility, overruns on each side of the selected cap, rejected malformed/reversed words, word ordering and text reconciliation. A higher limit must not bypass these checks or silently affect a stack without ForcedAligner. Confirm that the new plan records the chosen policy, corrections expose what changed, and exports validate the resulting canonical rather than infer timing from the correction ledger. Recovery delivery and acoustic accuracy have separate verdicts; a successfully clipped endpoint is not a reference measurement. An unexpected backend failure outside the declared branch still stops the workflow.
| Case | Original fixture and configuration | Required delivery and checks |
|---|---|---|
| A: multispeaker native stacks | test-sample-multispeaker.m4a; full original source; FireRed with word_timestamps,segment_timestamps; separate VibeVoice run with diarization,segment_timestamps; no forced language or speaker-count hint |
Save both canonical JSON results. Check each requested capability's actual outcome, segment/word timing where supplied, speaker/turn output for VibeVoice, and abstentions. Produce native TXT and timed Markdown for each. Present both canonical and readable transcripts for accuracy judgment. The two stacks are independently requested variants, not fallbacks. |
| B: English demo enhancement | demo-video-audio-to-improve.mp4; audio signal only; full original source; product-demo, explicit rnnoise |
Original inspect → one dry run → one MP4 render → delivered inspect → native saved-report summary/comparison. Deliver playable MP4 and before/predicted/delivered measurements. Check applied treatment, peak/duration validation, unresolved targets and nested evidence limits. No ASR, visual analysis or filler cuts are required. |
| C: mixed-language transcription and enhancement | autio-test-sample.m4a; full original source; FireRed with word_timestamps,segment_timestamps; then product-demo, explicit rnnoise |
Save original canonical JSON and native TXT/timed Markdown first. Then original inspect, dry run, one M4A render, delivered inspect and saved-report comparison. Deliver canonical/readable transcription and playable enhanced audio with the same checks as B. |
| D1: Qwen plain baseline | test-sample-multispeaker.m4a; full original source; qwen-1.7b, floors only; no forced language or speaker-count hint |
Save canonical JSON and native untimed TXT/Markdown. Validate request completion, preserved text and absence semantics. Do not require speaker or timing fields that were not requested. |
| D2: Qwen diarized and timed | Same original source; separately predeclared qwen-1.7b with diarization,overlapped_speech,word_timestamps; no forced language or speaker-count hint |
Save canonical JSON and native readable exports. Check actual speaker-labeled text, turns, overlap output, capability outcomes and every abstention. Required timing delivery is a separate subgoal: native timed Markdown, then SRT/VTT if prior requirements pass. Stop before export if canonical inspection already establishes an unsatisfactory required capability; a saved export refusal from an earlier run is evidence, not permission to retry it. |
The dependency sequence matters. In the default D2 case, incomplete required word timing stops subsequent CLI steps even if ASR and diarization produced useful results. A separately authorized recovery case may follow only its predeclared branch; the failed attempt's exports remain blocked. If the task only requires untimed speaker attribution, declare timing diagnostic rather than required before execution; its outcome then has no authority to fail that narrower user requirement. Missing attribution in a scope is reported even when all emitted text has a speaker label. Anonymous speaker IDs are not names, and model-to-model label equality establishes no shared identity.
Read the canonical artifact, not an agent summary, to verify what a backend supplied. Inspect provenance.outcomes, abstentions, complete, recorded range/coverage, emitted segments, words and turns. A capability declaration or successful plan is not a produced capability. complete: true is not a guarantee of verbatim text, exhaustive speech coverage or complete timing. Distinguish absent words from a supplied empty array; do not turn missing measurements into zeroes or nulls.
For timing, verify the recorded bounds against the source timeline and the documented export requirements. Keep zero-duration, missing, rejected and explicitly corrected timing visible. Do not supply missing timing from source length, a diarizer turn, processing-unit bounds or another model's words. An authorized clipping run adjusts an existing returned estimate under its recorded limit; verify its original/applied bounds and preserve the unmodified raw response. A native exporter can detect missing timing from one saved result without a reference transcript. It cannot establish whether recognized words, speaker labels or boundaries are correct simply because they are structurally valid.
For enhancement, distinguish processing applied from effectiveness measured. Inspect actual encoded-output metrics separately from predictions and intermediate stages, plus nested evidence limits even when unresolved is empty. Compare fixed original regions separately from fresh output detection and its speech reference. Apply only declared thresholds and explicitly accepted fixture-specific limits; do not invent a quality score. A decoded-duration check does not prove exact content alignment or A/V sync. Recognition accuracy, dialect/filler recall, speaker correctness and perceptual improvement need user judgment or an appropriate annotated/clean reference.
| Requirement status | Meaning and effect on the feature gate |
|---|---|
| PASS | The requested behavior and its declared evidence criteria are met; required outputs exist and validate. A negative test passes only with its predeclared refusal. |
| FAIL | The candidate was exercised and a required behavior or deliverable was not achieved. Record the responsible component; correct refusal behavior may pass its safety check while the requested delivery fails. |
| BLOCKED | A prerequisite or required judgment is unavailable, or the stop rule leaves a dependent step unattempted. Name the blocker; do not imply that the unattempted command failed. |
| NOT REQUIRED | Scope excluded this check before execution. Preserve optional observations but do not let them silently redefine the feature. |
The delivery verdict is PASS only when every predeclared required delivery check passes, FAIL when any such check fails, and otherwise BLOCKED when a required check or judgment is unavailable. Optional diagnostics keep their own observations and never enter that aggregate. Record operator, protocol and reporting outcomes separately; do not use a single unqualified overall verdict for different scopes. If an error invalidates required evidence, mark that delivery check BLOCKED (or FAIL if a violation was demonstrated). A later report-writing error can coexist with delivery PASS and reporting FAIL; it does not make successful ASR, diarization or exports fail.
The development feature gate additionally requires its applicable implementation checks and trustworthy evidence for the declared task. If report generation or a protocol behavior is itself the feature, it is a required check for that feature and cannot be waived. For authorized recovery, retain each attempt's verdict; later success does not pass a separately required default-policy case. Preserve raw historical operator verdicts and known limits while explaining their exact scope. A narrow feature may pass while another recorded requirement remains failed or blocked; disclose both without expanding the task retrospectively.
Prior human acceptance can be reused only for the unchanged scope it actually covered and with identity evidence. Byte-identical rendered media can retain that exact render's recorded listening decision. Exact canonical segment/word/turn equality can retain a previously recorded judgment of those fields; run-specific metadata need not match. Neither equality nor another model's output creates ground truth. New or changed Qwen text and attribution still need review. Do not infer a general waiver from prior acceptance of the demo's recorded loudness-range limit.
The development agent reviews required artifacts/refusals, source identity and operator conduct, then uses the clip report template to record scoped delivery checks, optional observations and operator/reporting issues. Link the actual outputs and blocker evidence; state required human judgments and relevant record limitations. Save the report directly as Markdown or JSON data with a file-writing tool such as apply_patch; do not generate a reporting program or embed report text in Python/JavaScript/shell strings. Validate that the saved file opens or parses, then inspect its statuses and links. If writing fails, preserve the available error, finish only non-audio reporting and keep already established delivery results unchanged. Add the compact tracked development result without uploading private media or full transcripts.
Triage feedback against the current source/help and existing issues. Separate confirmed defects, demonstrated usability gaps, optional ideas, already shipped behavior and out-of-scope work. Open focused issues with evidence, acceptance criteria and ownership boundaries only for genuine gaps. The testing operators do not patch the CLI. Fixes happen in development, then affected cases are repeated by new fresh-context operators against a new candidate; retain the original failure record. Obtain independent evidence review and resolve material findings before declaring the feature passed. Permission to test or submit a PR is not permission to merge it.
The 6 September full-source result records four fresh agents, candidate 00873235aed5c1ac17e53baaa2a1c687eaa5e26b, 83 CLI calls and unchanged sources. Native FireRed/VibeVoice results and both enhanced renders matched the earlier artifacts. The earlier acceptance record is functional-delivery evidence and still says PENDING_HUMAN_REVIEW; it does not itself establish human acceptance. The user's subsequent confirmation that they had audited the outcomes and approved closure is recorded in #40 and #41, including the preserved render limits. Reuse requires that subsequent decision and exact artifact scope, not the earlier pending record alone. These are historical evidence, not permanent pass expectations.
On the added Qwen variant, ASR and diarization produced results while word_timestamps abstained for seg_27. Its text and speaker survived, but neither segment nor word bounds existed, so timed Markdown refused it. That proves a timing-delivery failure using Qwen's own canonical result; it does not prove a recognition error or diarization failure. The example's report called delivery partial. Under the explicit gate above, the required timing subgoal is FAIL and unattempted subtitle steps are BLOCKED. The implementation of honest abstention/refusal can still pass a separately scoped safety test. Do not make this known gap an expected passing output of the full timing-delivery requirement.
The follow-up diagnosis and feedback triage explains the exact host detection, the missing evidence needed to identify the alignment cause, and the disposition of each agent suggestion.