Skip to content

feat: per-task model and effort selection for workflows#124

Merged
sasicodes merged 8 commits into
mainfrom
feat/workflow-model-scoring
Jul 6, 2026
Merged

feat: per-task model and effort selection for workflows#124
sasicodes merged 8 commits into
mainfrom
feat/workflow-model-scoring

Conversation

@sasicodes

@sasicodes sasicodes commented Jul 6, 2026

Copy link
Copy Markdown
Owner

What changed

Sub-agents in a workflow now pick their own model and effort per task, instead of all inheriting the parent session's single model/effort. The goal is cost efficiency: match each task to the cheapest model and lowest effort that can do it well.

Model scores (models.ts)

  • Every allowed model carries { affordability, intelligence, taste } (0-10, higher is better; affordability higher = cheaper), plus a modelScore(provider, id) lookup.

run_workflow tool

  • Each task is now { prompt, model, effort }, all mandatory (Valibot-validated).
  • The tool description is built dynamically and lists every model that has credentials configured, across all providers, with its scores and supported effort levels.
  • Cross-provider: one workflow can mix OpenAI and Anthropic sub-agents.
  • Guidance tells the agent not to default to the highest scores (wastes tokens) and to match each task to the minimum capability it needs.

Runtime

  • Each sub-agent resolves its own model from the task's key and clamps its own effort, instead of a single shared model/effort.

Testing

  • pnpm run lint, pnpm run format, pnpm --filter @start/desktop typecheck - clean
  • pnpm --filter @start/desktop vitest run - 803 tests passing (added workflow-catalog tests, updated subagent tests)

Greptile Summary

This PR upgrades the run_workflow tool so each task carries its own model and effort instead of inheriting the parent session's single selection. A new models.ts score table (affordability, intelligence, taste) drives a dynamically built tool description that guides the agent toward cost-appropriate model selection per task.

  • Per-task model resolution: SubagentTaskInput now requires model (a key string) and effort; runSubagents calls resolveModel(task.model) individually per task and passes the result alongside a clampThinkingLevel call, replacing the former shared model/thinkingLevel pair.
  • Dynamic tool description: workflowToolDescription is exposed as a get description() getter on the tool object so the model menu re-evaluates on each access rather than being frozen at session init; workflowModels() filters the picker list to only models that have a static score entry.
  • Schema tightened: Valibot taskSchema now validates a strict object (prompt + model + effort), and the JSON schema marks all three fields as required; previously loose inputs (plain strings, prompt-only objects) are silently dropped.

Confidence Score: 5/5

Safe to merge; changes are self-contained to the workflow tool and its supporting utilities, with no impact on the main session path.

The core runtime and tool plumbing is clean: per-task model resolution, effort clamping, and the dynamic description getter all behave correctly. The only finding is an unused provider field on WorkflowModelOption, which has no runtime impact. Tests cover the new schema, catalog rendering, and runtime concurrency behaviour.

No files require special attention.

Important Files Changed

Filename Overview
packages/desktop/src/main/models.ts Adds ModelScore interface and inline scores to each AllowedModel entry; exposes a modelScore(id) lookup map. No logic issues.
packages/desktop/src/main/subagents/types.ts Adds ResolvedModel, SubagentTaskInput (model+effort), and WorkflowModelOption types. The provider field on WorkflowModelOption is populated but never consumed anywhere in this PR.
packages/desktop/src/main/subagents/utils/catalog.ts New file: builds the dynamic run_workflow tool description from scored model options. Logic is clean and well-tested.
packages/desktop/src/main/subagents/utils/input.ts Replaces permissive string/object union schema with a strict Valibot object requiring prompt, model, and effort; invalid tasks are silently dropped (consistent with prior behavior for other invalids).
packages/desktop/src/main/providers/tools/subagents.ts Replaces static description string with a get description() getter, removes shared model/thinkingLevel in favour of per-task resolveModel, and updates the tool JSON schema to require model and effort fields.
packages/desktop/src/main/subagents/runtime.ts Refactors from a single shared model/thinkingLevel to per-task resolveModel + clampThinkingLevel. AgentJob wrapper keeps task alongside agent activity cleanly.
packages/desktop/src/main/chat.ts Adds workflowModels() (scores-filtered picker models) and threads resolveModel/availableModels into subagent base context; resolveModel falls back to pickModel when the key is unresolvable (previously flagged).
packages/desktop/src/main/types.ts Narrows effortLevels from EffortLevel[] to readonly tuple with as const satisfies, enabling Valibot picklist usage.
packages/desktop/tests/units/tools/workflow-catalog.test.ts New test file covering workflowModelMenu and workflowToolDescription; tests the empty-provider fallback, score guidance text, and per-model score line format.
packages/desktop/tests/units/subagents-runtime.test.ts Updated to use resolveModel callback and full SubagentTaskInput shape; tests remain faithful to runtime behaviour.
packages/desktop/tests/units/tools/subagents.test.ts Updated normalisation tests to require model+effort; explicitly asserts that plain string arrays and model-less tasks are now dropped.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Agent as Parent Agent
    participant Tool as run_workflow tool
    participant Catalog as workflowToolDescription
    participant Runtime as runSubagents
    participant Resolver as resolveModel(key)
    participant Sub as Sub-Agent Session

    Agent->>Tool: read description (get accessor)
    Tool->>Catalog: workflowToolDescription(availableModels())
    Catalog-->>Agent: model menu with scores

    Agent->>Tool: "execute({tasks: [{prompt, model, effort}, ...]})"
    Tool->>Runtime: "runSubagents({tasks, resolveModel, ...})"

    loop each AgentJob (max 4 concurrent)
        Runtime->>Resolver: resolveModel(task.model)
        alt model found
            Resolver-->>Runtime: ResolvedModel
        else not found
            Resolver-->>Runtime: "pickModel() fallback | null"
        end
        Runtime->>Sub: "createAgentSession({model, thinkingLevel: clampThinkingLevel(model, task.effort)})"
        Sub-->>Runtime: summary
    end

    Runtime-->>Tool: SubagentRunResult
    Tool-->>Agent: tool result text
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Agent as Parent Agent
    participant Tool as run_workflow tool
    participant Catalog as workflowToolDescription
    participant Runtime as runSubagents
    participant Resolver as resolveModel(key)
    participant Sub as Sub-Agent Session

    Agent->>Tool: read description (get accessor)
    Tool->>Catalog: workflowToolDescription(availableModels())
    Catalog-->>Agent: model menu with scores

    Agent->>Tool: "execute({tasks: [{prompt, model, effort}, ...]})"
    Tool->>Runtime: "runSubagents({tasks, resolveModel, ...})"

    loop each AgentJob (max 4 concurrent)
        Runtime->>Resolver: resolveModel(task.model)
        alt model found
            Resolver-->>Runtime: ResolvedModel
        else not found
            Resolver-->>Runtime: "pickModel() fallback | null"
        end
        Runtime->>Sub: "createAgentSession({model, thinkingLevel: clampThinkingLevel(model, task.effort)})"
        Sub-->>Runtime: summary
    end

    Runtime-->>Tool: SubagentRunResult
    Tool-->>Agent: tool result text
Loading

Reviews (2): Last reviewed commit: "fix: keep workflow model menu fresh as p..." | Re-trigger Greptile

Context used:

  • Context used - AGENTS.md (source)

Comment thread packages/desktop/src/main/chat.ts
Comment thread packages/desktop/src/main/providers/tools/subagents.ts Outdated
@sasicodes sasicodes merged commit 98a68b4 into main Jul 6, 2026
4 checks passed
@sasicodes sasicodes deleted the feat/workflow-model-scoring branch July 6, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant