Why
Browser and desktop improvements need a fixed set of tasks with explicit success checks and repeatable starting conditions. The existing evaluation runner, job records, and replay tools provide a starting point.
Scope
Create a small suite covering browser forms, date selection and search, plus desktop document editing and file dialogs. Define the required operations for each task and record unsupported operations explicitly. Establish the baseline on tasks the current agent can execute.
Reuse the existing evaluation runner and artifact formats. Record completion rate, total time, decision latency, model cost, and failure reason. Record hardware and model configuration for local inference.
Acceptance criteria
- Every task defines its initial conditions, reset procedure, required operations, and an independent success check.
- Repeated runs use the same task definitions and budgets across models. All attempts count toward completion rate, including failures, timeouts, and unsupported tasks.
- Results distinguish model inference, environment execution, and total task time. Local compute cost assumptions are stated.
- Saved records and replays are sufficient to inspect failures and reproduce the result table.
- The documented command runs the initial suite and produces a baseline report. Desktop results identify the operating system.
Starting points
evals/README.md, s1a/agents/desktop.py, and docs/browser-front.md.
This suite supplies the baseline for Cua-S1 4B 0.2 evaluation, desktop operation extensions, and bounded recovery.
Why
Browser and desktop improvements need a fixed set of tasks with explicit success checks and repeatable starting conditions. The existing evaluation runner, job records, and replay tools provide a starting point.
Scope
Create a small suite covering browser forms, date selection and search, plus desktop document editing and file dialogs. Define the required operations for each task and record unsupported operations explicitly. Establish the baseline on tasks the current agent can execute.
Reuse the existing evaluation runner and artifact formats. Record completion rate, total time, decision latency, model cost, and failure reason. Record hardware and model configuration for local inference.
Acceptance criteria
Starting points
evals/README.md, s1a/agents/desktop.py, and docs/browser-front.md.
This suite supplies the baseline for Cua-S1 4B 0.2 evaluation, desktop operation extensions, and bounded recovery.