Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
dec8963
feat: add OpenPhone AI Home shell
adamcohenhillel Jul 18, 2026
06fa9ce
feat: project durable agent runs on AI Home
adamcohenhillel Jul 18, 2026
3722c22
feat: add validated adaptive surfaces
adamcohenhillel Jul 18, 2026
e0e86dd
feat: integrate runtime-generated surfaces
adamcohenhillel Jul 18, 2026
e28845b
feat: resume background jobs after review
adamcohenhillel Jul 19, 2026
334a6ed
feat: move compact island into SystemUI
adamcohenhillel Jul 19, 2026
078f0b1
fix: preserve legacy island fallback on upgrade
adamcohenhillel Jul 19, 2026
b1ad50a
fix: defer AI Home until user unlock
adamcohenhillel Jul 19, 2026
fb82bda
fix: support GMS downloader on macOS bash
adamcohenhillel Jul 19, 2026
727a5c1
fix: keep Launcher3 home resolvable
adamcohenhillel Jul 19, 2026
258fa61
fix: defer AI Home state until user unlock
adamcohenhillel Jul 19, 2026
877dbf0
fix: defer assistant service until user unlock
adamcohenhillel Jul 19, 2026
a7665b8
fix: restore compact island design
adamcohenhillel Jul 19, 2026
27f17b3
feat: make AI Home immersive
adamcohenhillel Jul 20, 2026
59ba9f6
refactor: isolate app and OS execution
adamcohenhillel Jul 21, 2026
e9f09ce
demo: route silent speech into AI Home
adamcohenhillel Jul 21, 2026
563ea26
demo: add silent speech bridge command
adamcohenhillel Jul 21, 2026
d6cbdbf
demo: capture silent speech from AI Home
adamcohenhillel Jul 21, 2026
093dcee
demo: configure on-device decoder securely
adamcohenhillel Jul 21, 2026
d13e5a4
demo: use real camera preview surface
adamcohenhillel Jul 21, 2026
397816e
fix: stabilize camera surface lifecycle
adamcohenhillel Jul 21, 2026
9248e6e
demo: match iOS silent speech composer
adamcohenhillel Jul 21, 2026
9eb9c1c
fix: preserve silent speech capture quality
adamcohenhillel Jul 21, 2026
f4496ef
fix: use supported Compose sizing modifier
adamcohenhillel Jul 21, 2026
d792dff
fix: focus AI Home keyboard composer
adamcohenhillel Jul 21, 2026
89afaaf
feat: authenticate OpenPhone silent speech
adamcohenhillel Jul 21, 2026
9bd3fc0
fix: let actionable requests enter agent loop
adamcohenhillel Jul 21, 2026
4330085
fix: keep agent routing model driven
adamcohenhillel Jul 21, 2026
aa08741
fix: keep silent speech capture responsive
adamcohenhillel Jul 21, 2026
3b46a97
fix: rebuild prepared camera preview session
adamcohenhillel Jul 21, 2026
b19109d
fix: keep prepared preview surface alive
adamcohenhillel Jul 21, 2026
0263487
fix: allow silent speech camera prewarm
adamcohenhillel Jul 21, 2026
b6c29f6
fix: restore AI Home after agent failure
adamcohenhillel Jul 21, 2026
125f714
feat: show agent stop control in input dock
adamcohenhillel Jul 22, 2026
7558611
fix: publish active agent state after worker starts
adamcohenhillel Jul 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 46 additions & 12 deletions docs/AGENT_RUNTIME_V1.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ session.
- Persist background agent work as first-class jobs.
- Wake jobs from alarms, boot, package replacement, and explicit tool calls.
- Keep delivery, retries, diagnostics, and failure state with the job.
- Preserve the reviewed autonomy posture: background jobs do not silently
perform state-changing actions.
- Preserve the reviewed autonomy posture: background jobs pause before
state-changing actions and resume only from an exact Android-owned review.
- Keep model routing in the registry and schemas instead of Java keyword
routing.

Expand Down Expand Up @@ -45,18 +45,35 @@ An `agent_job` records durable background work.
"mode": "notification",
"notification_text": "Release notes changed"
},
"status": "active",
"status": "queued",
"created_at": 1781480000000,
"updated_at": 1781480000000,
"next_run_at": 1781483055000,
"running_at": 0,
"last_run_at": 0,
"last_result": "",
"failure_count": 0,
"failure_alert_at": 0
"failure_alert_at": 0,
"phase": "queued",
"progress_text": "Queued",
"progress_current": 0,
"progress_total": 0,
"checkpoint_json": {},
"pending_confirmation_id": "",
"pending_tool_request_json": {},
"last_surface_id": "",
"resume_token": "",
"last_event_at": 1781480000000,
"unread_result": false,
"paused_at": 0
}
```

Lifecycle states are `queued`, `running`, `waiting`, `awaiting_review`,
`paused`, `completed`, `failed`, and `stopped`. Readers continue accepting the
legacy `active` value during migration. Runtime dispatch may also persist
`dispatched` while an external runtime owns the next step.

Job types:

- `agent_turn`: run a bounded model/tool loop against a prompt.
Expand All @@ -77,7 +94,8 @@ model/tool trace and must not be dumped directly into the notification surface.

The assistant schedules a single alarm for the next active job. On wake it:

1. repairs stale `running` jobs,
1. expires pending reviews and repairs stale `running` or review-resolution
jobs,
2. marks due jobs `running`,
3. starts bounded background runners,
4. persists completion/failure,
Expand All @@ -89,13 +107,29 @@ service startup.

## Safety

Background jobs may call observe/read tools and terminal tools. State-changing
tools return `background.confirmation_required` instead of executing. The
foreground reviewed flow remains the only path for actions that mutate device,
account, communication, or external state.

This is intentionally conservative. The follow-up design can add reviewed
background approvals with a notification action and a resumable job checkpoint.
Background jobs may call observe/read tools and terminal tools directly. A
state-changing tool is never executed by the background model loop. Instead,
OpenPhone:

1. bounds and sanitizes the exact proposed tool arguments;
2. persists an `awaiting_review` checkpoint and 15-minute confirmation;
3. shows the exact action in an Android notification and AI Home run detail;
4. atomically claims the first Approve or Deny tap;
5. verifies the tool, parameter digest, runtime, phone session, job, expiry,
idempotency key, checkpoint, and resume token;
6. executes only the persisted request after approval, or records a structured
denial/timeout result;
7. resumes the job with that bound result instead of regenerating the action.

Provider secrets, raw authorization, screenshots, data URLs, large encoded
blobs, and unbounded context are rejected from checkpoints. Audit events store
digests and lifecycle metadata, not raw parameters. If the process dies while
an approved action is resolving, stale-run repair marks the outcome unknown
and does not replay it. A denied action can safely resume after restart.

The wire shapes are defined by `schemas/agent-job.schema.json` and
`schemas/background-confirmation.schema.json`; recorded valid, tampered, and
secret-bearing fixtures run in `./scripts/check.sh`.

## OpenClaw/Hermes Mapping

Expand Down
90 changes: 90 additions & 0 deletions docs/APP_OS_BOUNDARY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# App and OS Boundary

OpenPhone has one product experience with two possible phone capability
profiles:

- **OpenPhone App** uses Android public APIs and user-granted roles and
permissions. It is the future Play-distributable profile.
- **OpenPhone OS** adds a privileged execution adapter, framework services,
SystemUI surfaces, keyguard configuration, and system policy enforcement.

Most product logic should be shared. The OS layer should remain a small,
auditable authority rather than becoming a second assistant implementation.

## Dependency Direction

```text
AI Home / generated surfaces / model and runtime adapters
|
v
PhoneToolGateway
/ \
v v
public Android app OpenPhone OS adapter
implementation (framework Binder client)
|
v
system_server / SystemUI
```

Code above `PhoneToolGateway` must not import `android.openphone` hidden APIs,
construct `FrameworkToolExecutor`, or decide that a privileged action is
authorized. It may request a registered tool, show review UI, and interpret a
structured result.

The gateway reports a stable profile and whether each tool is supported. A
smaller implementation must deny an unsupported tool explicitly; it must not
silently fall back to accessibility automation or a lower-risk action.

## Source Ownership

| Layer | Current source | Responsibility |
| --- | --- | --- |
| Portable product | `assistant/actions`, `model`, `orchestrator`, `runtime`, `surface`, and Compose UI | Conversation, model adapters, tool requests, adaptive UI, runtime sessions, and user review presentation |
| Phone boundary | `assistant/platform/PhoneToolGateway.java` | Stable app-to-phone execution contract and capability-profile seam |
| OpenPhone OS adapter | `assistant/platform/OpenPhoneOsToolGateway.java` and `assistant/agent/FrameworkToolExecutor.java` | Translate registered tools into the hidden OpenPhone framework manager and Android integrations |
| OS authority | `patches/frameworks_base`, Settings patches, SystemUI patches, and SELinux policy | Screen/input authority, secure confirmation, durable audit, island rendering, keyguard-aware behavior, and privileged state |

The current privileged APK is still the only build artifact. This boundary is
the first extraction step, not a claim that the APK can already be uploaded to
Google Play.

## Capability Placement

The public app profile can own AI Home as a user-selected launcher, immersive
window UI, voice and text interaction, generated surfaces, model/runtime
adapters, local history, public Android intents, and scheduled work that fits
normal Android limits.

The OS profile remains authoritative for:

- SystemUI-owned island and keyguard-safe rendering;
- non-secure lock-screen defaults and global hardware gestures;
- silent screen context and capture;
- cross-app input and task control;
- secure settings and other signature/privileged operations;
- tamper-resistant confirmation, policy, and audit storage;
- stronger direct-boot and always-available execution guarantees.

Public Android roles or APIs can cover some phone, SMS, notification, calendar,
contact, and launcher features. Their availability must be represented as
capabilities of the active gateway, never inferred from the package name.

## Extraction Sequence

Keep this work in reviewable follow-up changes:

1. Route runtime and adaptive-surface execution through `PhoneToolGateway`.
2. Split `FrameworkToolExecutor` into public-Android tools and OS-only tools,
with both implementations preserving the same result contract.
3. Move settings and durable stores behind app-owned configuration/storage
interfaces instead of reading arbitrary `Settings.Secure` keys in portable
packages.
4. Add separate app manifests and build targets: a public-SDK Play profile and
the current platform-signed `system_ext` profile.
5. Add a machine-readable capability matrix and run the same runtime/surface
contract suite against both profiles.

Until those steps are complete, changes must keep the existing OpenPhone OS
behavior green and must not weaken framework confirmation or audit paths to
make the app profile easier to implement.
67 changes: 61 additions & 6 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,23 @@ Device
The current repo implements the first OpenPhone product layer:

- `vendor/openphone` product config.
- `OpenPhoneAssistant` privileged app with a chat-style user surface, text and
voice task entry, stateful mic/send/stop composer action, advanced model and
developer controls, task grants, screen context, trace/audit export, OTA
preview controls, and audit controls.
- `OpenPhoneAssistant` privileged app with a dedicated Android Home activity,
a calm voice-first AI Home surface, explicit conventional-launcher App Space
handoff, text and voice task entry, stateful mic/send/stop controls, advanced
model and developer controls, task grants, screen context, trace/audit
export, OTA preview controls, and audit controls. AI Home is an immersive
normal application window rather than an opaque overlay: it hides Android
status/navigation chrome while foreground and allows transient system bars
by edge swipe, while Android retains ownership of secure keyguard,
notifications, IME, recents, emergency UI, and system dialogs. OpenPhone
disables only the non-secure swipe keyguard by default; configuring a
pattern, PIN, or password restores Android's credential gate.
- A `PhoneToolGateway` boundary keeps model, runtime, and adaptive-surface
packages independent from hidden `android.openphone` APIs. The current
`OpenPhoneOsToolGateway` binds that portable contract to the framework-backed
executor; a future public-SDK implementation can expose a smaller supported
tool set without forking the product runtime. See
[APP_OS_BOUNDARY.md](APP_OS_BOUNDARY.md).
- Initial capability and policy config files. `scripts/check.sh` validates that
the assistant fallback `PolicyEngine` covers every capability in
`openphone_capabilities.json` with the same risk class.
Expand All @@ -66,9 +79,51 @@ The current repo implements the first OpenPhone product layer:
- Agent Runtime V1 background jobs. The first assistant-side implementation
persists background agent turns, schedules them with AlarmManager, wakes them
at boot/package replacement/service startup, delivers notification results,
and blocks state-changing background tools until a foreground reviewed
approval flow exists. The runtime contract is documented in
and pauses state-changing tools behind exact, expiring Android-owned review.
Approval or denial produces a bound checkpoint result; resume cannot
regenerate or substitute the authorized request, duplicate taps cannot
execute twice, and interrupted approval execution fails safe without replay.
The runtime contract is documented in
[AGENT_RUNTIME_V1.md](AGENT_RUNTIME_V1.md).
- A unified agent-run projection reads jobs, watchers, commitments, and
foreground execution sessions without creating a second execution store.
AI Home uses it for stable activity bubbles and run detail, while the
SystemUI island reads the same projection for compact recent/live status.
Approval-needed bubbles expose exact review detail and Approve/Deny actions;
queued work can be paused/resumed, and the compact island routes review back
to AI Home.
Inspection and dismissal state is local presentation metadata; source stores
remain authoritative for execution and stop operations.
- The compact OpenPhone island is owned by SystemUI. The assistant publishes a
bounded, privacy-minimized state projection through the OpenPhone agent
Binder service; `system_server` validates it, assigns revision/timestamps,
retains the latest snapshot across assistant UI recreation, and sends
one-way change events to SystemUI. SystemUI renders only a fixed-size
`TYPE_STATUS_BAR_SUB_PANEL` window with non-modal touch behavior, redacts
personal/review state on keyguard, routes detail and approval to AI Home, and
degrades to a generic bounded status when the publisher becomes stale. Its
compact geometry, black capsule, centered typography, and terse state glyphs
preserve the established assistant-island presentation while moving window
ownership into SystemUI. The
assistant's full-screen `TYPE_SYSTEM_ERROR` layer remains only for
non-touchable pointer/glow visualization during active device control; the
old assistant island is a property-gated compatibility renderer.
- Adaptive Surface V1 is a phone-owned, revisioned UI document contract.
Runtimes can request only a bounded semantic component registry; the
assistant validates identity, ownership, expiry, depth/node/text/image
limits, local artifact provenance, registered tool bindings, tool parameter
shapes, accessibility labels, and mutation disclosure before persistence or
rendering. Surface actions resolve against the current revision and then
enter the existing runtime tool bridge, so policy, grants, confirmation,
idempotency, and framework audit remain authoritative. The Compose renderer
has no WebView, HTML, remote image URL, arbitrary color, absolute-position,
or runtime-code path.
- Built-in calendar search and message/notification result paths produce
deterministic Adaptive Surface documents from trusted phone tool results.
Tool artifacts retain session/runtime/capability/sensitivity/expiry
provenance separately from the layout, while the surface repository survives
activity recreation and fails closed on stale, malformed, expired, or
cross-session content.
- Runtime Agent Protocol for external agent runtimes. This is an OpenPhone AI
layer boundary, not a separate product stack: Android owns sessions, screen
context, tool execution, confirmations, and audit, while adapters map remote
Expand Down
2 changes: 2 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ it locally with:

- [Architecture](ARCHITECTURE.md) — system layers, OS services, agent
runtime, and current implementation boundaries.
- [App and OS Boundary](APP_OS_BOUNDARY.md) — shared product layers, the
phone-tool gateway, and the Play-app versus privileged-OS split.
- [Capabilities](CAPABILITIES.md) — named capabilities, risk levels, and
policy configuration.
- [Agent Runtime](AGENT_RUNTIME_V1.md) — background job model, scheduling,
Expand Down
Loading
Loading