Completed draft validation harness

.plans/hosted-deployment-validation-harness-implementation-plan-2026-07-27.md

@@ -1,6 +1,6 @@
 # Hosted Deployment Validation Harness Implementation Plan
 
-Status: Phase 3 complete
+Status: Phase 5 complete
 
 Date: 2026-07-27
 
@@ -217,14 +217,14 @@ Exit criteria:
 
 ## Phase 4 — Dev/stable isolation
 
-- [ ] Require separate explicitly configured dev and stable test accounts.
-- [ ] Create a uniquely marked development project and verify it is absent
+- [x] Require separate explicitly configured dev and stable test accounts.
+- [x] Create a uniquely marked development project and verify it is absent
   from stable.
-- [ ] Create a uniquely marked stable project and verify it is absent from dev.
-- [ ] Delete both records and verify absence after deletion.
-- [ ] Record only project markers, ownership-independent IDs if safe, and
+- [x] Create a uniquely marked stable project and verify it is absent from dev.
+- [x] Delete both records and verify absence after deletion.
+- [x] Record only project markers, ownership-independent IDs if safe, and
   boolean presence results.
-- [ ] Add tests preventing accidental use of the same account or same API
+- [x] Add tests preventing accidental use of the same account or same API
   origin for both sides of the isolation check.
 
 Exit criteria:
@@ -234,16 +234,16 @@ Exit criteria:
 
 ## Phase 5 — Optional OAuth and bounded operations
 
-- [ ] Add an opt-in OAuth preflight that checks the configured callback host
+- [x] Add an opt-in OAuth preflight that checks the configured callback host
   and expected dev/stable redirect configuration without completing a real
   provider authorization.
-- [ ] Require a human-provided/manual OAuth checkpoint for any live login.
-- [ ] Integrate hosted evidence with the existing redaction, state, event,
+- [x] Require a human-provided/manual OAuth checkpoint for any live login.
+- [x] Integrate hosted evidence with the existing redaction, state, event,
   metrics, and summary contracts.
-- [ ] Enforce a separate hosted timeout, browser count, mutation count, and
+- [x] Enforce a separate hosted timeout, browser count, mutation count, and
   cleanup budget.
-- [ ] Add `--dry-run`, `--no-agent`, and explicit hosted-scope validation.
-- [ ] Ensure the Codex repair path cannot invoke hosted mutation commands.
+- [x] Add `--dry-run`, `--no-agent`, and explicit hosted-scope validation.
+- [x] Ensure the Codex repair path cannot invoke hosted mutation commands.
 
 Exit criteria:
 

docs/troubleshooting/repair-harness.md

@@ -11,6 +11,8 @@ change GitHub secrets, or modify hosted databases.
 - Install dependencies with `npm install`.
 - Authenticate the GitHub CLI with `gh auth status` when collecting from GitHub.
 - Have the repository's required browsers installed for E2E reproduction.
+- For hosted validation, have Chromium installed and use only disposable,
+  explicitly approved test accounts and records.
 
 ## Collect a failed PR job
 
@@ -85,6 +87,21 @@ inflation, secret/config changes, and excessive file scope by default.
 Use `--dry-run` to prepare packets without invoking an agent. Use `--no-agent`
 or omit `--agent=codex` when you want to ensure no agent can run.
 
+## E2E timing diagnostics
+
+The timing diagnostic reads an existing Playwright JSON report. It does not
+change Playwright workers, retries, timeouts, or test source:
+
+```bash
+npm run repair:ci -- e2e-timing \
+  --report playwright-report.json
+```
+
+Individual tests at or below 7 seconds are normal, tests above 7 seconds and
+through 10 seconds are reported as warnings, and tests above 10 seconds fail
+the diagnostic. The run directory contains redacted timing evidence grouped
+by project and test file.
+
 ## Measure completed runs
 
 Aggregate redacted outcomes from local runs with:
@@ -108,12 +125,132 @@ commands against the workflow files:
 npm run repair:ci
 ```
 
-The first supported E2E scope is PR Chromium. Unit Node, unit jsdom,
-Storybook, and build scopes are also cataloged. Main-manifest, nightly matrix,
-deployment, and hosted lifecycle scopes remain explicitly unsupported.
+The first supported E2E scope is PR Chromium. Unit Node, unit jsdom, Storybook,
+and build scopes are also cataloged. Hosted validation is a separate, explicit
+workflow described below; it is never added to the local Codex repair path.
 
 ## Clean handoff
 
 Review `summary.md`, `evidence.json`, `events.jsonl`, and the verification
 result before accepting a patch. The harness never publishes changes; a human
 must review, commit, and push any repair separately.
+
+## Hosted deployment validation
+
+Hosted validation checks the configured GitHub Pages and Railway origins. It is
+read-only by default and never deploys, promotes, changes secrets, alters
+GitHub configuration, or repairs a remote failure. Every hosted command
+requires the explicit scope flag and disables agent use:
+
+```bash
+--scope hosted --no-agent
+```
+
+### Configuration
+
+Pass a JSON file to `--config`. The file uses the validated `HOSTED_*` keys;
+URLs must be HTTPS and both API hosts must be listed in the allowlist:
+
+```json
+{
+  "HOSTED_DEV_FRONTEND_URL": "https://pages.example/dev/",
+  "HOSTED_DEV_API_URL": "https://dev-api.example",
+  "HOSTED_STABLE_FRONTEND_URL": "https://pages.example/stable/",
+  "HOSTED_STABLE_API_URL": "https://stable-api.example",
+  "HOSTED_ALLOWED_API_HOSTS": ["dev-api.example", "stable-api.example"],
+  "HOSTED_TEST_ACCOUNT_PROVIDER": "manual",
+  "HOSTED_EXPECTED_DEV_CHANNEL": "dev",
+  "HOSTED_EXPECTED_STABLE_CHANNEL": "stable"
+}
+```
+
+`HOSTED_ALLOW_MUTATIONS` defaults to false. Hosted limits are separate from
+local repair limits: `HOSTED_TIMEOUT_MS` defaults to 15 seconds,
+`HOSTED_MAX_BROWSERS` to 1, `HOSTED_MAX_MUTATIONS` to 2, and
+`HOSTED_MAX_CLEANUP_ATTEMPTS` to 2. Keep these bounds low and run-specific.
+
+### Read-only probes and browser smoke
+
+Deployment probes check health, channel, and optional commit metadata without
+credentials:
+
+```bash
+npm run repair:ci -- hosted-probe \
+  --config hosted.json --scope hosted --no-agent
+```
+
+The real-origin browser smoke opens the configured Pages URL and verifies API
+routing, reachability, and unauthenticated state:
+
+```bash
+npm run repair:ci -- hosted-browser \
+  --config hosted.json --scope hosted --no-agent
+```
+
+Use `--dry-run` with either command to validate configuration without making a
+network request or launching a browser.
+
+### Disposable lifecycle and isolation checks
+
+Mutation commands require both `HOSTED_ALLOW_MUTATIONS=true` in the config and
+the explicit `--allow-hosted-mutations` flag. Credentials are passed in
+memory-only CLI arguments and are never written to the run directory:
+
+```bash
+npm run repair:ci -- hosted-mutate \
+  --config hosted.json --scope hosted --no-agent \
+  --allow-hosted-mutations \
+  --email disposable@example.test --password '<password>'
+```
+
+The two-sided isolation check requires separate disposable accounts and
+creates one uniquely marked project in each environment:
+
+```bash
+npm run repair:ci -- hosted-isolation \
+  --config hosted.json --scope hosted --no-agent \
+  --allow-hosted-mutations \
+  --dev-email dev-disposable@example.test \
+  --dev-password '<dev-password>' \
+  --stable-email stable-disposable@example.test \
+  --stable-password '<stable-password>'
+```
+
+The runner checks both directions of visibility, deletes both projects, and
+confirms absence afterward. If deletion cannot be confirmed, the result is
+`cleanup-required`; do not start another mutation run until the remote record
+has been handled by the designated operator.
+
+### OAuth preflight
+
+OAuth preflight is opt-in and checks only configured callback/redirect metadata;
+it does not authorize a provider or store OAuth credentials. Enable it with
+`HOSTED_ALLOW_OAUTH=true`, `HOSTED_OAUTH_CALLBACK_HOST`,
+`HOSTED_EXPECTED_DEV_OAUTH_REDIRECT`, and
+`HOSTED_EXPECTED_STABLE_OAUTH_REDIRECT`:
+
+```bash
+npm run repair:ci -- hosted-oauth-preflight \
+  --config hosted.json --scope hosted --no-agent \
+  --allow-hosted-oauth
+```
+
+Any future live provider login requires a human-provided manual checkpoint;
+the harness must not automate provider authorization.
+
+### Hosted run artifacts and failure handling
+
+Hosted runs are written beneath `.repair-harness/runs/<run-id>/` and include:
+
+- `hosted-config.json`: redacted validated configuration;
+- `hosted-evidence.json`: approved response or isolation fields only;
+- `hosted-events.jsonl`: normalized hosted step outcomes;
+- `hosted-metrics.json`: bounded status and cleanup metrics;
+- `state.json`, `events.jsonl`, and `hosted-summary.md`: the standard run
+  state/event/summary contracts and the recommended next diagnostic.
+
+Passwords, session cookies, authorization headers, CSRF values, OAuth codes,
+database URLs, and unrestricted response bodies must not enter these files.
+A hosted failure produces evidence and a next diagnostic step; it never invokes
+Codex repair or performs an automatic remote repair. The local `repair` command
+rejects hosted scope and hosted reproduction commands by design.

scripts/repair-harness/cli.ts

@@ -10,8 +10,10 @@ import { appendEvent, readState, resolveResumeDirectory, writeState } from './st
 import type { EvidenceEnvelope } from './types';
 import { aggregateRepairOutcomes, formatRepairMetrics, readRepairOutcomes, writeRepairMetrics } from './metrics';
 import { loadHostedConfig } from './hosted/config';
-import { runHostedBrowser, runHostedMutationCommand, runHostedProbe } from './hosted/run';
+import { runHostedBrowser, runHostedIsolationCommand, runHostedMutationCommand, runHostedProbe } from './hosted/run';
 import { runE2ETiming } from './e2eTimingRun';
+import { runHostedOAuthPreflight } from './hosted/oauth';
+import { assertHostedScope, assertRepairCannotUseHostedScope } from './scope';
 
 const repositoryRoot = path.resolve(new URL('../..', import.meta.url).pathname);
 const args = process.argv.slice(2);
@@ -23,6 +25,11 @@ const value = (name: string): string | undefined => {
 };
 
 const has = (name: string): boolean => args.includes(name);
+const prepareHosted = (command: string): boolean => {
+  assertHostedScope(command, value('--scope'));
+  if (value('--agent')) throw new Error('Hosted validation cannot invoke an agent; use --no-agent.');
+  return has('--dry-run');
+};
 
 async function main(): Promise<void> {
 if (args[0] === 'e2e-timing') {
@@ -38,9 +45,12 @@ if (args[0] === 'e2e-timing') {
   }
 } else if (args[0] === 'hosted-probe') {
   try {
+    const dryRun = prepareHosted('hosted-probe');
     const configPath = value('--config');
     if (!configPath) throw new Error('Expected --config <path> for hosted-probe.');
-    const result = await runHostedProbe({ repo: value('--repo') ?? repositoryRoot, config: loadHostedConfig(path.resolve(configPath)), runId: value('--run-id') });
+    const config = loadHostedConfig(path.resolve(configPath));
+    if (dryRun) { console.log('Hosted probe dry-run: configuration validated; no network request will run.'); return; }
+    const result = await runHostedProbe({ repo: value('--repo') ?? repositoryRoot, config, runId: value('--run-id') });
     console.log(`Hosted probe ${result.status} in ${result.runDirectory}`);
     if (result.status !== 'passed') process.exitCode = 1;
   } catch (error) {
@@ -49,9 +59,12 @@ if (args[0] === 'e2e-timing') {
   }
 } else if (args[0] === 'hosted-browser') {
   try {
+    const dryRun = prepareHosted('hosted-browser');
     const configPath = value('--config');
     if (!configPath) throw new Error('Expected --config <path> for hosted-browser.');
-    const result = await runHostedBrowser({ repo: value('--repo') ?? repositoryRoot, config: loadHostedConfig(path.resolve(configPath)), runId: value('--run-id') });
+    const config = loadHostedConfig(path.resolve(configPath));
+    if (dryRun) { console.log('Hosted browser dry-run: configuration validated; no browser or network request will run.'); return; }
+    const result = await runHostedBrowser({ repo: value('--repo') ?? repositoryRoot, config, runId: value('--run-id') });
     console.log(`Hosted browser ${result.status} in ${result.runDirectory}`);
     if (result.status !== 'passed') process.exitCode = 1;
   } catch (error) {
@@ -60,14 +73,17 @@ if (args[0] === 'e2e-timing') {
   }
 } else if (args[0] === 'hosted-mutate') {
   try {
+    const dryRun = prepareHosted('hosted-mutate');
     const configPath = value('--config');
     const email = value('--email');
     const password = value('--password');
     if (!configPath) throw new Error('Expected --config <path> for hosted-mutate.');
     if (!email || !password) throw new Error('Hosted mutation requires --email and --password; credentials are used in memory only.');
+    const config = loadHostedConfig(path.resolve(configPath));
+    if (dryRun) { console.log('Hosted mutation dry-run: configuration and credentials presence validated; no browser or mutation will run.'); return; }
     const result = await runHostedMutationCommand({
       repo: value('--repo') ?? repositoryRoot,
-      config: loadHostedConfig(path.resolve(configPath)),
+      config,
       runId: value('--run-id'),
       account: { email, password, inviteToken: value('--invite-token') },
       signup: has('--signup'),
@@ -79,6 +95,43 @@ if (args[0] === 'e2e-timing') {
     console.error(error instanceof Error ? error.message : String(error));
     process.exitCode = 1;
   }
+} else if (args[0] === 'hosted-isolation') {
+  try {
+    const dryRun = prepareHosted('hosted-isolation');
+    const configPath = value('--config');
+    const devEmail = value('--dev-email');
+    const devPassword = value('--dev-password');
+    const stableEmail = value('--stable-email');
+    const stablePassword = value('--stable-password');
+    if (!configPath) throw new Error('Expected --config <path> for hosted-isolation.');
+    if (!devEmail || !devPassword || !stableEmail || !stablePassword) throw new Error('Hosted isolation requires separate --dev-email/--dev-password and --stable-email/--stable-password credentials; credentials are used in memory only.');
+    const config = loadHostedConfig(path.resolve(configPath));
+    if (dryRun) { console.log('Hosted isolation dry-run: configuration and credential presence validated; no browser or mutation will run.'); return; }
+    const result = await runHostedIsolationCommand({
+      repo: value('--repo') ?? repositoryRoot,
+      config,
+      runId: value('--run-id'),
+      accounts: { dev: { email: devEmail, password: devPassword }, stable: { email: stableEmail, password: stablePassword } },
+      explicitFlag: has('--allow-hosted-mutations'),
+    });
+    console.log(`Hosted isolation ${result.status} in ${result.runDirectory}`);
+    if (result.status !== 'passed') process.exitCode = 1;
+  } catch (error) {
+    console.error(error instanceof Error ? error.message : String(error));
+    process.exitCode = 1;
+  }
+} else if (args[0] === 'hosted-oauth-preflight') {
+  try {
+    const dryRun = prepareHosted('hosted-oauth-preflight');
+    const configPath = value('--config');
+    if (!configPath) throw new Error('Expected --config <path> for hosted-oauth-preflight.');
+    const config = loadHostedConfig(path.resolve(configPath));
+    const result = runHostedOAuthPreflight(config, has('--allow-hosted-oauth'));
+    console.log(`Hosted OAuth preflight ${result.status}${dryRun ? ' (dry-run)' : ''}.`);
+  } catch (error) {
+    console.error(error instanceof Error ? error.message : String(error));
+    process.exitCode = 1;
+  }
 } else if (args[0] === 'metrics') {
   try {
     const repo = value('--repo') ?? repositoryRoot;
@@ -138,6 +191,7 @@ if (args[0] === 'e2e-timing') {
   }
 } else if (args[0] === 'repair') {
   try {
+    assertRepairCannotUseHostedScope(value('--scope'));
     if (has('--no-agent') || value('--agent') !== 'codex') throw new Error('Repair requires explicit --agent=codex; no agent is enabled by default.');
     const repo = value('--repo') ?? repositoryRoot;
     const runDirectory = value('--run-dir') ?? (value('--resume') ? resolveResumeDirectory(repo, value('--resume')!) : undefined);

scripts/repair-harness/hosted/accounts.ts

@@ -0,0 +1,15 @@
+import type { HostedAccount } from './browser';
+
+export interface HostedIsolationAccounts {
+  dev: HostedAccount;
+  stable: HostedAccount;
+}
+
+/** Credentials are intentionally accepted only in memory and are never serialized. */
+export function assertSeparateHostedAccounts(accounts: HostedIsolationAccounts): void {
+  if (!accounts.dev.email.trim() || !accounts.dev.password) throw new Error('A development hosted test account is required.');
+  if (!accounts.stable.email.trim() || !accounts.stable.password) throw new Error('A stable hosted test account is required.');
+  if (accounts.dev.email.trim().toLowerCase() === accounts.stable.email.trim().toLowerCase()) {
+    throw new Error('Development and stable hosted test accounts must be distinct.');
+  }
+}

scripts/repair-harness/hosted/bounds.ts

@@ -0,0 +1,23 @@
+import type { HostedConfig } from './config';
+
+export interface HostedOperationBudget {
+  browserCount: number;
+  mutationCount: number;
+  cleanupAttempts: number;
+  timeoutMs: number;
+}
+
+export interface HostedOperationUsage extends HostedOperationBudget {
+  elapsedMs: number;
+}
+
+export function hostedOperationBudget(config: HostedConfig): HostedOperationBudget {
+  return { browserCount: config.maxBrowsers, mutationCount: config.maxMutations, cleanupAttempts: config.maxCleanupAttempts, timeoutMs: config.timeoutMs };
+}
+
+export function assertHostedBounds(budget: HostedOperationBudget, usage: HostedOperationUsage): void {
+  if (usage.browserCount > budget.browserCount) throw new Error('Hosted browser budget exceeded.');
+  if (usage.mutationCount > budget.mutationCount) throw new Error('Hosted mutation budget exceeded.');
+  if (usage.cleanupAttempts > budget.cleanupAttempts) throw new Error('Hosted cleanup budget exceeded.');
+  if (usage.elapsedMs >= budget.timeoutMs) throw new Error('Hosted timeout budget exceeded.');
+}