Skip to content

[Eval] Add repeatable browser and desktop task suites #13

Description

@Yunaik

Why

Browser and desktop improvements need a fixed set of tasks with explicit success checks and repeatable starting conditions. The existing evaluation runner, job records, and replay tools provide a starting point.

Scope

Create a small suite covering browser forms, date selection and search, plus desktop document editing and file dialogs. Define the required operations for each task and record unsupported operations explicitly. Establish the baseline on tasks the current agent can execute.

Reuse the existing evaluation runner and artifact formats. Record completion rate, total time, decision latency, model cost, and failure reason. Record hardware and model configuration for local inference.

Acceptance criteria

  1. Every task defines its initial conditions, reset procedure, required operations, and an independent success check.
  2. Repeated runs use the same task definitions and budgets across models. All attempts count toward completion rate, including failures, timeouts, and unsupported tasks.
  3. Results distinguish model inference, environment execution, and total task time. Local compute cost assumptions are stated.
  4. Saved records and replays are sufficient to inspect failures and reproduce the result table.
  5. The documented command runs the initial suite and produces a baseline report. Desktop results identify the operating system.

Starting points

evals/README.md, s1a/agents/desktop.py, and docs/browser-front.md.

This suite supplies the baseline for Cua-S1 4B 0.2 evaluation, desktop operation extensions, and bounded recovery.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions