Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/memory/cli/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@ description: "CLI orchestration, report writing, HTML generation, and the host p
| [host-preflight](host-preflight.md) | hostPreflight.ts — the host environment checks the CLI runs before a session, and the platform-branched command-on-PATH probe behind them: `resolveCommandPath` locates with `where` on win32 (first output line) and `which` elsewhere, through defaulted process-boundary overrides that make both branches test-reachable without stubbing globals. |
| [report-writer](report-writer.md) | ReportWriter (packages/cli/src/reportWriter.ts) — the run directory it emits (runner.log, input/, tests/<id>/, summary.json, run.json), the secret-redaction contract every write path crosses, device-log copy-then-redact, first-failure precedence, and the emitted-JSON key-omission contract that makes an absent optional field part of the schema |
| [session-runner](session-runner.md) | sessionRunner.ts — prepareTestSession/executeTestOnSession/runGoal: the per-call ExecutionSessionState every phase records its acquisition into, the single finally that releases whatever is still held, the recording and device-log capture lifecycles, the guarded (never `!`-asserted) install-override preconditions and model-owned platform derivation, and the CLI's two sanctioned dependency seams (TestSessionDeps and testRunnerDependencies) |
| [test-compiler](test-compiler.md) | compileTestObjective (packages/cli/src/testCompiler.ts) and the secret-value exposure chain it anchors: only `${variables.*}` is substituted, so the compile-time guarantee covers the compiled objective alone — once typed, a resolved secret reaches the provider through the captured hierarchy and screenshots. Prompt text is redacted per input at the AIAgent seam when runtime bindings are wired; screenshots, app-rendered transformations of a value, and INFO planner output are not. |
| [test-compiler](test-compiler.md) | compileTestObjective (packages/cli/src/testCompiler.ts) and the secret-value exposure chain it anchors: only `${variables.*}` is substituted, so the compile-time guarantee covers the compiled objective alone — once typed, a resolved secret reaches the provider through the captured hierarchy and screenshots. Prompt text is redacted per input at the AIAgent seam when runtime bindings are wired; screenshots, app-rendered values, sub-3-char secret values, and INFO planner output are not. |
8 changes: 5 additions & 3 deletions docs/memory/cli/report-writer.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,13 @@ Four public emit orchestrators — `writeRunInputs`, `writeTestRecord`, `writeTe

## Secret Redaction

Every value that reaches a file crosses `redactResolvedValue(value, bindings)` (exported from `@finalrun/common`, defined in its `repoPlaceholders.ts`), which rewrites a resolved secret value back to its `${secrets.<key>}` placeholder: runner-log lines, a test record's `message` and `analysis`, step JSON (action text and url, analysis, thought, trace and timing detail strings), the synthetic failure message, and the copied device log.
Every value that reaches a file crosses `redactResolvedValue(value, bindings)` (exported from `@finalrun/common`, defined in its `repoPlaceholders.ts` — [/common/repo-placeholders.md](/common/repo-placeholders.md)), which rewrites a resolved secret value back to its `${secrets.<key>}` placeholder: runner-log lines, a test record's `message` and `analysis`, step JSON (action text and url, analysis, thought, trace and timing detail strings), the synthetic failure message, and the copied device log.

The environment snapshots carry reference forms only — `env.snapshot.yaml` mirrors the *config* `secrets` map (the `${ENV_VAR}` placeholder form) and `env.json` carries `secretReferences` (key plus env-var name). No resolved secret value is written anywhere beneath the run directory, and `reportWriter.test.ts` holds that as a whole-`runDir` sweep over every emitted file after a full lifecycle rather than as per-site assertions.
The environment snapshots carry reference forms only — `env.snapshot.yaml` mirrors the *config* `secrets` map (the `${ENV_VAR}` placeholder form) and `env.json` carries `secretReferences` (key plus env-var name). No **redactable** secret value is written anywhere beneath the run directory, and `reportWriter.test.ts` holds that as a whole-`runDir` sweep over every emitted file after a full lifecycle rather than as per-site assertions.

The contract is bounded in two ways worth naming, because the sweep's breadth invites reading it as global. It covers **write paths only** — the LLM prompt path is a separate seam, redacted per input inside `AIAgent` when runtime bindings are wired, and stdout INFO output crosses neither ([/cli/test-compiler.md](/cli/test-compiler.md)). And the sweep is a **byte-level** check on emitted files, so `tests/<testId>/screenshots/NNN.jpg` passes it while still showing a typed secret in its pixels: the report carries secret-bearing images even though it carries no secret strings.
"Redactable" is the operative word, and the sweep cannot show more than it: its fixture secret is a 17-character value, so what the sweep proves is bounded by the shared function's own bounds. A secret value shorter than `MIN_REDACTABLE_SECRET_LENGTH` (3) is never redacted, so a 1–2 character secret value **does** reach files beneath the run directory raw — an accepted residual, since substituting such a value would rewrite every incidental occurrence of that character sequence in a log line or step record instead. Conversely, an authored `${secrets.KEY}` token already present in report text — test steps like `Enter ${secrets.email}` carry them routinely — is consumed whole by the redaction alternation's last branch and emitted verbatim, never rewritten into a nested token.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Qualify the authored-token pass-through claim.

Line 24 promises verbatim output for every authored token. docs/memory/common/repo-placeholders.md Lines 88-92 document an exact-token secret-value case where the value branch rewrites the token to another placeholder. Restrict “emitted verbatim” to cases where no eligible value branch matches at the token start.

Proposed wording
-... is consumed whole by the redaction alternation's last branch and emitted verbatim, never rewritten into a nested token.
+... is passed through verbatim when no eligible value branch matches at the token start; an exact-token secret value may be rewritten placeholder-to-placeholder, but never into a nested token.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"Redactable" is the operative word, and the sweep cannot show more than it: its fixture secret is a 17-character value, so what the sweep proves is bounded by the shared function's own bounds. A secret value shorter than `MIN_REDACTABLE_SECRET_LENGTH` (3) is never redacted, so a 1–2 character secret value **does** reach files beneath the run directory raw — an accepted residual, since substituting such a value would rewrite every incidental occurrence of that character sequence in a log line or step record instead. Conversely, an authored `${secrets.KEY}` token already present in report text — test steps like `Enter ${secrets.email}` carry them routinely — is consumed whole by the redaction alternation's last branch and emitted verbatim, never rewritten into a nested token.
"Redactable" is the operative word, and the sweep cannot show more than it: its fixture secret is a 17-character value, so what the sweep proves is bounded by the shared function's own bounds. A secret value shorter than `MIN_REDACTABLE_SECRET_LENGTH` (3) is never redacted, so a 1–2 character secret value **does** reach files beneath the run directory raw — an accepted residual, since substituting such a value would rewrite every incidental occurrence of that character sequence in a log line or step record instead. Conversely, an authored `${secrets.KEY}` token already present in report text — test steps like `Enter ${secrets.email}` carry them routinely — is passed through verbatim when no eligible value branch matches at the token start; an exact-token secret value may be rewritten placeholder-to-placeholder, but never into a nested token.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/memory/cli/report-writer.md` at line 24, Update the authored
`${secrets.KEY}` token statement in the report-writer documentation to qualify
verbatim pass-through: it applies only when no eligible secret-value branch
matches at the token’s start. Preserve the existing explanation that the
authored-token alternation branch consumes and emits the token unchanged in
cases where that value-branch condition is not met.


The contract is bounded in two further ways worth naming, because the sweep's breadth invites reading it as global. It covers **write paths only** — the LLM prompt path is a separate seam, redacted per input inside `AIAgent` when runtime bindings are wired, and stdout INFO output crosses neither ([/cli/test-compiler.md](/cli/test-compiler.md)). And the sweep is a **byte-level** check on emitted files, so `tests/<testId>/screenshots/NNN.jpg` passes it while still showing a typed secret in its pixels: the report carries secret-bearing images even though it carries no secret strings.
Comment on lines +20 to +26

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

Limit the guarantee to textual values.

Line 20 says every value that reaches a file crosses redactResolvedValue, but Line 26 says binary screenshot bytes are outside this path. Use “textual value” for the guarantee so readers do not interpret the whole run directory as redacted.

Proposed wording
-Every value that reaches a file crosses `redactResolvedValue(value, bindings)` ...
+Every textual value that reaches a file crosses `redactResolvedValue(value, bindings)` ...

-The sweep is a byte-level check on emitted files ...
+This is a string-redaction guarantee. The sweep is a byte-level check on emitted files ...
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Every value that reaches a file crosses `redactResolvedValue(value, bindings)` (exported from `@finalrun/common`, defined in its `repoPlaceholders.ts`[/common/repo-placeholders.md](/common/repo-placeholders.md)), which rewrites a resolved secret value back to its `${secrets.<key>}` placeholder: runner-log lines, a test record's `message` and `analysis`, step JSON (action text and url, analysis, thought, trace and timing detail strings), the synthetic failure message, and the copied device log.
The environment snapshots carry reference forms only — `env.snapshot.yaml` mirrors the *config* `secrets` map (the `${ENV_VAR}` placeholder form) and `env.json` carries `secretReferences` (key plus env-var name). No resolved secret value is written anywhere beneath the run directory, and `reportWriter.test.ts` holds that as a whole-`runDir` sweep over every emitted file after a full lifecycle rather than as per-site assertions.
The environment snapshots carry reference forms only — `env.snapshot.yaml` mirrors the *config* `secrets` map (the `${ENV_VAR}` placeholder form) and `env.json` carries `secretReferences` (key plus env-var name). No **redactable** secret value is written anywhere beneath the run directory, and `reportWriter.test.ts` holds that as a whole-`runDir` sweep over every emitted file after a full lifecycle rather than as per-site assertions.
The contract is bounded in two ways worth naming, because the sweep's breadth invites reading it as global. It covers **write paths only** — the LLM prompt path is a separate seam, redacted per input inside `AIAgent` when runtime bindings are wired, and stdout INFO output crosses neither ([/cli/test-compiler.md](/cli/test-compiler.md)). And the sweep is a **byte-level** check on emitted files, so `tests/<testId>/screenshots/NNN.jpg` passes it while still showing a typed secret in its pixels: the report carries secret-bearing images even though it carries no secret strings.
"Redactable" is the operative word, and the sweep cannot show more than it: its fixture secret is a 17-character value, so what the sweep proves is bounded by the shared function's own bounds. A secret value shorter than `MIN_REDACTABLE_SECRET_LENGTH` (3) is never redacted, so a 1–2 character secret value **does** reach files beneath the run directory raw — an accepted residual, since substituting such a value would rewrite every incidental occurrence of that character sequence in a log line or step record instead. Conversely, an authored `${secrets.KEY}` token already present in report text — test steps like `Enter ${secrets.email}` carry them routinely — is consumed whole by the redaction alternation's last branch and emitted verbatim, never rewritten into a nested token.
The contract is bounded in two further ways worth naming, because the sweep's breadth invites reading it as global. It covers **write paths only** — the LLM prompt path is a separate seam, redacted per input inside `AIAgent` when runtime bindings are wired, and stdout INFO output crosses neither ([/cli/test-compiler.md](/cli/test-compiler.md)). And the sweep is a **byte-level** check on emitted files, so `tests/<testId>/screenshots/NNN.jpg` passes it while still showing a typed secret in its pixels: the report carries secret-bearing images even though it carries no secret strings.
Every textual value that reaches a file crosses `redactResolvedValue(value, bindings)` (exported from `@finalrun/common`, defined in its `repoPlaceholders.ts`[/common/repo-placeholders.md](/common/repo-placeholders.md)), which rewrites a resolved secret value back to its `${secrets.<key>}` placeholder: runner-log lines, a test record's `message` and `analysis`, step JSON (action text and url, analysis, thought, trace and timing detail strings), the synthetic failure message, and the copied device log.
The environment snapshots carry reference forms only — `env.snapshot.yaml` mirrors the *config* `secrets` map (the `${ENV_VAR}` placeholder form) and `env.json` carries `secretReferences` (key plus env-var name). No **redactable** secret value is written anywhere beneath the run directory, and `reportWriter.test.ts` holds that as a whole-`runDir` sweep over every emitted file after a full lifecycle rather than as per-site assertions.
"Redactable" is the operative word, and the sweep cannot show more than it: its fixture secret is a 17-character value, so what the sweep proves is bounded by the shared function's own bounds. A secret value shorter than `MIN_REDACTABLE_SECRET_LENGTH` (3) is never redacted, so a 1–2 character secret value **does** reach files beneath the run directory raw — an accepted residual, since substituting such a value would rewrite every incidental occurrence of that character sequence in a log line or step record instead. Conversely, an authored `${secrets.KEY}` token already present in report text — test steps like `Enter ${secrets.email}` carry them routinely — is consumed whole by the redaction alternation's last branch and emitted verbatim, never rewritten into a nested token.
The contract is bounded in two further ways worth naming, because the sweep's breadth invites reading it as global. It covers **write paths only** — the LLM prompt path is a separate seam, redacted per input inside `AIAgent` when runtime bindings are wired, and stdout INFO output crosses neither ([/cli/test-compiler.md](/cli/test-compiler.md)). This is a string-redaction guarantee. The sweep is a **byte-level** check on emitted files, so `tests/<testId>/screenshots/NNN.jpg` passes it while still showing a typed secret in its pixels: the report carries secret-bearing images even though it carries no secret strings.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/memory/cli/report-writer.md` around lines 20 - 26, Update the guarantee
in the opening paragraph to say that every textual value reaching a file crosses
redactResolvedValue, rather than claiming every value is redacted. Preserve the
existing explanation that binary screenshot bytes are outside this textual
redaction path and do not broaden the guarantee beyond textual file contents.


## Device Log Artifact Flow

Expand Down
35 changes: 29 additions & 6 deletions docs/memory/cli/test-compiler.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
type: memory
description: "compileTestObjective (packages/cli/src/testCompiler.ts) and the secret-value exposure chain it anchors: only `${variables.*}` is substituted, so the compile-time guarantee covers the compiled objective alone — once typed, a resolved secret reaches the provider through the captured hierarchy and screenshots. Prompt text is redacted per input at the AIAgent seam when runtime bindings are wired; screenshots, app-rendered transformations of a value, and INFO planner output are not."
description: "compileTestObjective (packages/cli/src/testCompiler.ts) and the secret-value exposure chain it anchors: only `${variables.*}` is substituted, so the compile-time guarantee covers the compiled objective alone — once typed, a resolved secret reaches the provider through the captured hierarchy and screenshots. Prompt text is redacted per input at the AIAgent seam when runtime bindings are wired; screenshots, app-rendered values, sub-3-char secret values, and INFO planner output are not."
---
# Test Objective Compilation (cli)

Expand Down Expand Up @@ -60,9 +60,12 @@ hierarchy and screenshot status above (the span detail logs the unresolved `rawD
`AIAgent` (`packages/goal-executor/src/ai/AIAgent.ts`) accepts optional `bindings: RuntimeBindings`,
wired from `createSessionExecutor` (`packages/cli/src/sessionRunner.ts`) as
`config.runtimeBindings`. When bindings are present, `_buildPlannerPrompt` and
`_buildGrounderPrompt` MUST replace exact occurrences of resolved secret values with their
`${secrets.KEY}` placeholder — via `redactResolvedValue` from `@finalrun/common`, whose
longest-value-first alternation is reused rather than reimplemented — applying it:
`_buildGrounderPrompt` MUST replace occurrences of resolved secret values with their
`${secrets.KEY}` placeholder — via `redactResolvedValue` from `@finalrun/common`, whose matching
semantics are reused rather than reimplemented, and whose bounds the prompt path therefore inherits
in full: unanchored longest-value-first matching, an existing placeholder token consumed whole rather
than rewritten, and no redaction of a value shorter than `MIN_REDACTABLE_SECRET_LENGTH` (3)
([/common/repo-placeholders.md](/common/repo-placeholders.md)). Redaction applies:

- to every free-text input individually (`testObjective`, `act`, `history`, each `remember` entry,
`preContext`, `appKnowledge`);
Expand All @@ -77,8 +80,10 @@ only serialized copies are rewritten — so index-based grounding (`flattenedHie
→ tap point) still resolves against the real on-screen values. Absent bindings, every path is a
strict no-op and prompt assembly is byte-identical.

Seven tests in `packages/goal-executor/src/ai/test/AIAgent.test.ts` pin this, including one for each
failure mode a pass over assembled text would introduce (see Design Decisions).
The "Prompt-path secret redaction" block in `packages/goal-executor/src/ai/test/AIAgent.test.ts` pins
this behaviour case by case, including one test for each failure mode a pass over assembled text would
introduce (see Design Decisions) and one for each way unanchored value matching can corrupt prompt
text rather than protect it.

#### Scenario: the typed field is redacted and still locatable
- **GIVEN** bindings `{ secrets: { PASSWORD: 'hunter2-secret' } }` and a captured hierarchy whose
Expand All @@ -94,6 +99,19 @@ failure mode a pass over assembled text would introduce (see Design Decisions).
- **THEN** the emitted `ui_elements` still parses as JSON, `bounds` is `[0, 0, 1080, 240]`, and only
the field whose string value equals the secret becomes `${secrets.PIN}`

#### Scenario: the objective's own placeholder token survives its value's redaction
- **GIVEN** bindings `{ secrets: { PASSWORD: 'PASSWORD' } }` and a `testObjective` reading
`Type ${secrets.PASSWORD} into the field, then type PASSWORD again`
- **WHEN** the planner prompt is built
- **THEN** the prompt carries `Type ${secrets.PASSWORD} into the field, then type ${secrets.PASSWORD}
again` and never a nested `${secrets.${secrets.PASSWORD}}` token

#### Scenario: a secret value too short to redact leaves the prompt untouched
- **GIVEN** bindings `{ secrets: { TOKEN: 's' } }` and any planner request
- **WHEN** the planner prompt is built
- **THEN** the assembled text is byte-identical to a bindings-less agent's assembly of the same
request — the value reaches the provider raw rather than mangling every `s` in the prompt

### Requirement: Residual exposure is stated, not closed
Prompts, provider-side logs and report artifacts MUST still be treated as secret-bearing. What
remains exposed after prompt-text redaction:
Expand All @@ -103,6 +121,11 @@ remains exposed after prompt-text redaction:
field-region detection; dropping them would blind the planner.
- **App-rendered transformations** — a value the app truncates, reformats or partially masks is no
longer an exact match and passes through.
- **Secret values shorter than three characters** — never redacted anywhere, so they reach the
provider (and the report, and spans) raw. The trade is deliberate: substituting a 1–2 character
value rewrites every incidental occurrence of that character sequence and turns the prompt into
garbage, which corrupts model input rather than protecting anything
([/common/repo-placeholders.md](/common/repo-placeholders.md)).
- **Structural strings** — exact matching also rewrites an element's `id`, `class` or `contentDesc`
when its content equals a secret value. That degrades the text the grounder reads; it cannot
corrupt the emitted JSON, because `index` and `bounds` are non-string fields.
Expand Down
3 changes: 2 additions & 1 deletion docs/memory/common/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: "`packages/common` — the base of the dependency graph, imported by every other package: the UI-hierarchy parse contract that planner and grounder prompts are built from (two deliberately non-equivalent parse paths, presence-based alias resolution, two bounds forms, unvalidated field casts), and the `CliEnv` environment-loading contract (dotenv/OS precedence, falsy-value traps, and where model and reasoning-level validation lives)"
description: "`packages/common` — the base of the dependency graph, imported by every other package: the UI-hierarchy parse contract planner and grounder prompts are built from (two non-equivalent parse paths, presence-based alias resolution, two bounds forms, unvalidated casts), the `${variables.*}`/`${secrets.*}` placeholder contract in both directions (with the bounds and residuals of secret redaction), and the `CliEnv` environment-loading contract (dotenv/OS precedence, falsy-value traps)"
---
# Common Documentation

Expand All @@ -9,3 +9,4 @@ description: "`packages/common` — the base of the dependency graph, imported b
|------|-------------|
| [env](env.md) | `CliEnv` environment loading (`packages/common/src/env.ts`): `load` layers `.env.<envName>` → plain `.env` (fill-only) → OS env (highest precedence), `includeDotEnv` opts out only on a literal `false`, and `getRequired`'s falsy check makes an empty string as missing as an absent key. Model and reasoning-level validation live in `constants.ts` beside their level lists, while `env.ts`'s re-export block is a deliberate backward-compat shim for the one import path that still routes through it. |
| [hierarchy](hierarchy.md) | UI-hierarchy parse contract (`common/src/models/Hierarchy.ts`, consumed by four goal-executor files): `fromJsonString` dispatches array→flat / object→tree, and the paths are deliberately not equivalent — only the flat path shortens `:id/` ids, infers `isImage` from the class, takes `identifier`; alias resolution is `??`-presence via `_pick`/`orDefault`, never truthiness, so `false` and `''` survive; bounds are a 4-array or a left/top/right/bottom object; reads are unvalidated casts. |
| [repo-placeholders](repo-placeholders.md) | `repoPlaceholders.ts` (`packages/common`) — both directions of `${variables.*}`/`${secrets.*}` handling, and the `redactResolvedValue` contract every prompt, report and span write crosses: unanchored substring matching sorted longest-value-first (the substring-leak guard), an existing placeholder token consumed whole by the alternation's LAST branch so a value occurrence still wins there, and no redaction below `MIN_REDACTABLE_SECRET_LENGTH` (3) — with the residuals that leaves. |
Loading
Loading