Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
456ec8a
fix(macos): match known Chrome AX title suffixes
SLYILL Aug 25, 2026
3659ed7
Add strict hosted runner execution adapter
abrichr Aug 26, 2026
c08ceb9
fix: type hosted runner release bindings
abrichr Aug 26, 2026
0f24113
feat: complete hosted runner target state
abrichr Aug 26, 2026
a110660
test: preserve boolean parameter digest
abrichr Aug 26, 2026
ea465f6
rescue: atomic frame observation and actuation lease (recovered WIP)
abrichr Aug 26, 2026
2f71b94
feat: retain sealed native source geometry
abrichr Aug 26, 2026
7be6359
fix: require complete native geometry provenance
abrichr Aug 26, 2026
04102b6
fix(record): complete native capture geometry integration
abrichr Aug 27, 2026
aa9e4c3
docs: align Flow truth with current contracts
abrichr Aug 26, 2026
bc5f54d
docs: describe resume appliance wiring
abrichr Aug 26, 2026
13ece5a
chore: refresh public artifact hashes
abrichr Aug 26, 2026
51addbf
docs: correct verification provenance marker
abrichr Aug 26, 2026
3103287
feat(cli): clarify rejected-write verification fixture
abrichr Aug 26, 2026
c6eec41
docs(onboarding): gate the first real workflow
abrichr Aug 26, 2026
65bd41f
docs(onboarding): make the real first workflow reviewable
abrichr Aug 27, 2026
b646836
fix(onboarding): gate replay before first actuation
abrichr Aug 27, 2026
c7382c2
docs(validation): name the rejected transaction outcome
abrichr Aug 27, 2026
bd8c62f
fix(tutorial): require exact rejected-write evidence
abrichr Aug 27, 2026
0bf1ab0
feat(hosted): sign complete terminal outcomes
abrichr Aug 27, 2026
6cfcd03
fix(runtime): bind navigation to stable coordinate mapping
abrichr Aug 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 10 additions & 12 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,12 @@ backend owns a structured layer (a browser DOM, a native UIA/AX tree) the
resolution ladder's TOP rung re-finds the recorded target as an ELEMENT and
acts on it deterministically (`StructuralActionBackend`, see Resolution ladder).
Structure is preferred where present; the visual ladder is the fallback floor
for pixel-only substrates (RDP/Citrix/canvas). Backend evidence varies and is
stated honestly: the **shipped, end-to-end-exercised** backend is
Playwright-driven (headless-capable, CI-friendly, permission-free) and is the
only path proven against a real third-party app. Beyond it, a `WindowsBackend`
(UI Automation over the WindowsAgentArena server) is **proven structurally on a
local Windows-on-ARM VM** (record → compile → replay, DB-judged), and a FreeRDP
`RDPBackend` plus a Citrix/remote-display pixel-only backend exist as
**spikes, not validated integrations** — their live behavior is unmeasured to
the degree disclosed in `docs/backends/RDP.md` and `docs/desktop/CITRIX_PIXEL.md`.
They are adapters onto the one protocol, not rewrites.
for pixel-only substrates (RDP/Citrix/canvas). Playwright, Windows UIA, macOS
AX, Linux AT-SPI, RDP, and Citrix/pixel-window adapters implement the same
backend contract. Their evidence is bound to exact fixtures, counted tasks,
environments, or deployment qualifications in `docs/PRODUCT_STATUS.md` and
`docs/VERIFICATION.md`. A backend name alone does not show that an application
or environment is qualified.

## Core contracts (additive-only; do not change without updating this doc)

Expand Down Expand Up @@ -371,8 +367,10 @@ referral → New Encounter → click "Triage" → click Note field → type note
`Recorder` so frames/events are captured (before frame, act, wait settle,
after frame).

`PlaywrightBackend(page)` implements `Backend` (chromium, fixed viewport
1280x800, deviceScaleFactor=1). `Recorder(backend, out_dir)` wraps a backend
`PlaywrightBackend(page)` implements `Backend` for Chromium. The MockMed
fixture starts at 1280x800 with `deviceScaleFactor=1`. An attached recording can
start a new exact geometry epoch after a stable viewport or device-scale
change. `Recorder(backend, out_dir)` wraps a backend
with the same action methods plus `type_text(text, param=None)` and
`finish() -> recording dir`.

Expand Down
56 changes: 36 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@
[Discussions](https://github.com/OpenAdaptAI/openadapt-flow/discussions) ·
[Contributing](CONTRIBUTING.md)

**openadapt-flow is the OpenAdapt engine: a governed demonstration compiler.**
Record a task once, compile it to a deterministic program, and replay that
program deterministically with zero model calls on the healthy path. Instead of
silently doing the wrong thing when an interface drifts, it re-resolves from the
evidence the demonstration retained, or it **halts** for a human or an AI, gated
by an identity check and independent effect verification. It runs entirely on
your machine; nothing egresses unless you opt in.
**openadapt-flow is the demonstration compiler and governed runtime behind
OpenAdapt.** It compiles a demonstrated GUI workflow into a deterministic,
locally executable program. Healthy runs make no model calls. When an interface
drifts, Flow re-resolves from retained evidence. A person or configured model
can propose a repair. Identity, effect, and policy checks still apply, and Flow
halts when verification fails. It runs on your machine and doesn't send data
anywhere unless you opt in.

It targets repeated workflows across every interface an operator touches:
browser pages, native Windows / macOS / Linux desktops, and remote-display
Expand Down Expand Up @@ -48,14 +48,14 @@ rounds against the wrong-target check.

## Try it

The canonical first run uses the [OpenAdapt](https://github.com/OpenAdaptAI/openadapt)
launcher, which handles Python versions, virtual environments, and shell quoting
for you:
The optional product check uses the
[OpenAdapt](https://github.com/OpenAdaptAI/openadapt) launcher, which handles
Python versions, virtual environments, and shell quoting for you:

```bash
curl -fsSL https://raw.githubusercontent.com/OpenAdaptAI/openadapt-flow/main/scripts/install.sh | sh

openadapt quickstart # the whole loop, VERIFIED
openadapt quickstart # optional product check, VERIFIED
```

Prefer plain pip? Two commands (quote the brackets; on Windows `cmd.exe` use
Expand All @@ -64,7 +64,7 @@ double quotes: `pip install "openadapt[browser]"`):
```bash
pip install 'openadapt[browser]'

openadapt quickstart # the whole loop, VERIFIED
openadapt quickstart # optional product check, VERIFIED
```

**Requirements:** Python 3.10–3.12 (3.13+ is not yet supported; the installer
Expand All @@ -76,9 +76,7 @@ name:
```bash
pip install 'openadapt-flow[browser]'

openadapt-flow tutorial # same loop as `openadapt quickstart`

openadapt-flow tutorial --break-it # then watch it catch a lie
openadapt-flow tutorial # optional product check
```

`tutorial` records a demonstration against the bundled MockMed fixture, mines its
Expand All @@ -87,10 +85,8 @@ policy, and verifies the write by reading the system of record out of band — a
path the app never calls, so the screen cannot influence it. It ends `VERIFIED`
with zero model calls.

`--break-it` reruns the **same certified bundle** against a backend that lies:
the server rejects the write *after* the app paints its success banner, so every
on-screen check passes while nothing lands. The independent read refutes the
mined contract and the engine **HALTS** at the consequential step.
Your first real workflow starts with a small read-only task and test data:
[record it in your own app](#record-your-own-app-on-any-substrate).

Full walkthrough, including `--guided` and the hand-driven
`demo-record` / `compile` / `lint` / `certify` / `replay` stages:
Expand Down Expand Up @@ -167,6 +163,9 @@ Artifacts: [baseline run report](docs/showcase/baseline-run/REPORT.md) and

## Record your own app, on any substrate

Start with one small read-only task and test data. Write down the result you
expect. Record the task, compile it, then inspect and lint the bundle:

Six substrates run on the same `Backend` protocol and the same governed runtime,
selected with `--backend web | windows | macos | linux | rdp | citrix` on
`record`, `replay`, and `run`. The browser is one surface among six, not a
Expand All @@ -177,9 +176,26 @@ compiled bundle is bound to the exact surface it was recorded on.
```bash
openadapt-flow record --backend web --url https://your.app --out rec
openadapt-flow compile rec --out bundle --name my-task
openadapt-flow replay bundle --backend web --url https://your.app
openadapt-flow visualize bundle -o graph.html
openadapt-flow lint bundle
```

Confirm that the bundle contains only the read-only task you selected. If lint
reports a state-changing, unknown, consequential, or irreversible action, stop
and [qualify it](https://openadapt.ai/qualify) before Flow first actuates it.

Then replay the read-only workflow with the browser visible:

```bash
openadapt-flow replay bundle --backend web --url https://your.app --headed \
--run-dir first-run
```

Review `first-run/REPORT.md`. Confirm that the recorded steps and the final
result match what you expected before you expand the task.

Qualify the exact app and environment before unattended use.

- Install matrix, exact commands for all six substrates, the counted evidence
behind each, and the two remote execution modes:
[backends and surface support](docs/SURFACES.md).
Expand Down
18 changes: 10 additions & 8 deletions claims.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,9 @@ claims:
# ------------------------------------------------------------------ web
- id: web-supported
claim: >-
Web (browser) workflows are supported today: record a GUI workflow once,
then replay it deterministically and locally.
The Playwright browser path records a GUI workflow and replays it
deterministically and locally. Required CI exercises the recorder,
compiler, and replay contract.
surfaces: [README.md, website, docs]
tier: supported
evidence:
Expand Down Expand Up @@ -199,17 +200,18 @@ claims:
reflected evidence from Python at the settled boundary instead.
caveats:
- >-
"Supported" is scoped to the reference headless-browser backend in this
registry. Desktop and remote-display workflows use the separately scoped
acceptance and code-qualified claims below.
`supported` names the registry's required-CI evidence tier. It is not a
product state or a general application claim. Production requires active
release admissions for all seven product targets. A Production run also
needs an active workflow admission for the exact sealed bundle.
- >-
The full record->compile->replay browser suite runs in the required
e2e-browser PR gate and repeats in the weekly compatibility matrix.
- >-
Existing-session attachment is Chromium-only and loopback-only. It
requires a dedicated browser process started with remote debugging.
It does not claim support for the Capture Chrome extension prototype
or direct extension replay.
requires a dedicated browser process with remote debugging. The
Playwright-native browser path owns this claim. Capture extension code
is outside its evidence scope.

# -------------------------------------------------- deterministic $0 replay
- id: deterministic-zero-model-replay
Expand Down
6 changes: 3 additions & 3 deletions docs/LIMITS.md
Original file line number Diff line number Diff line change
Expand Up @@ -314,12 +314,12 @@ halt does not prove earlier actions were harmless.

## Interaction and environment limits

The recorded-surface evidence is strongest when the target remains inside one
captured browser surface and the demonstration exposes observable outcomes.
Qualification is strongest when the demonstration retains exact observations
and exposes independently verifiable outcomes.

| Condition | Current boundary |
| --- | --- |
| Zoom, DPI, font, layout, or viewport changes | Structural evidence can survive some reflow and visual rungs can survive some movement, but support is workflow-specific. Large rescale or reflow can halt. |
| Zoom, DPI, font, layout, viewport, or monitor changes | Browser attach recording accepts a stable viewport or device-scale change between actions, starts a new exact geometry epoch, and refuses an action that overlaps the transition. Native offline conversion can normalize a stable source-window move, resize, monitor change, or scale change into one fixed output viewport when every timeline row carries valid mapping metadata. It refuses a malformed timeline or a changed output viewport. RDP and Citrix live backends can rebaseline geometry between actions and refuse a change during the exact-frame lease. Other rescale or reflow behavior remains workflow-specific. |
| Native select menus, file choosers, permission prompts, and secure desktops | OS or browser chrome may not appear in the captured surface and may be unrecordable or undrivable. Prefer an application-level or keyboard/API path. |
| New tabs and windows | Opening a tab can be observed structurally where the backend supports it; interaction inside additional windows and multi-window coordination are not a general supported path. |
| Drag and drop or gesture-heavy controls | Not a general supported primitive. Use a structured/API alternative or validate a purpose-built workflow. |
Expand Down
Loading
Loading