Skip to content

[Story]: Evaluate multi-agent frameworks to support the team developing SprintStart #241

Description

@adoLime

User Story

As the AI team, we want to evaluate whether a multi-agent framework (e.g. crewAI, LangGraph, AutoGen) can support us in developing SprintStart, so that we can deliver the project efficiently by end of October.

Context & Motivation

  • Type: Spike / Evaluation (produces a decision + ADR, optionally a minimal PoC — not a shipping feature).
  • Parent epic: 01 EPIC - Platform Architecture & Developer Experience #50 (01 EPIC – Platform Architecture & Developer Experience)
  • Teams: team:ai
  • Background: In Sprints 5-7 several team members have exam preparation and reduced project capacity. We want to check whether a multi-agent framework can lift repetitive dev work off the team (PR reviews, test scaffolding, bug diagnosis, doc generation) so the remaining time is used more efficiently.
  • Enabler: Niklas provided each team member with a personal OpenRouter API key (500$/person) usable also for coding agents and framework experiments (no team key exists).
  • Builds on: OpenCode / Cursor / VS Code integrations with OpenRouter (see OpenRouter docs). No existing in-house agent pipeline is reused — this is a dev-side evaluation, separate from any production agent work.
  • Related: NFR HP Define dev team split (Backend, Retrieval/AI, Frontend) #3 (local LLMs in production — NOT a constraint here, since this is dev tooling, not shipped product).

Candidate Use Cases (the AI team, in consultation with the rest of the team, selects which to validate in the spike)

The AI team, in consultation with the rest of the team, selects which of the following use cases to validate during the spike. More use cases = larger spike + higher token budget.

  1. Auto-PR-Review — a GitHub Action that runs an agent on every pull request and posts code-review feedback before a human reviews. Directly unburdens the team's reviewers.
  2. Test generator — an agent that reads a new feature/issue and proposes unit tests. Removes raw typing work.
  3. Bug diagnostician — an agent that analyses error logs + code and proposes root cause + fix. Shortens debugging.
  4. Doc generator — an agent that generates comments / README / API docs from code. Removes writing work.

Acceptance Criteria

  • For each selected use case, a minimal PoC is built (on top of the chosen framework) that demonstrates it on a real SprintStart task.
  • The framework(s) used are documented (at minimum 1, optionally a comparison of 2 candidates) with: maturity, licence, OpenRouter/model coupling, ease of setup.
  • An ADR is written (in sprintstart-ai, following the repo's ADR conventions) capturing the recommendation: adopt / pilot / do-not-adopt, with rationale, PoC findings, and migration/roll-out path if "adopt/pilot".
  • Token budget stop: the spike stops when the configured budget is reached — 50$ for 1-2 use cases, 100$ for all 4. OpenRouter usage is monitored per key.
  • The evaluation is presented to the team (short demo/walkthrough of the PoC + ADR).
  • If a CI-deployed agent is proposed (e.g. Auto-PR-Review as a GitHub Action), the ADR names the token-allocation model and open questions (see "Open questions" below).

Sub-Tasks (by team)

  • AI
    • Select 1-4 use cases from the candidate list above (in consultation with the rest of the team).
    • Shortlist + compare candidate framework(s).
    • Build a minimal PoC per selected use case, wired to OpenRouter via the personal key.
    • Monitor token spend against the budget stop.
    • Write the ADR with recommendation + roll-out path.
    • Team walkthrough / demo.

Open Questions (to resolve after the spike, before a CI agent rolls out)

Since there is no dedicated team key, decide how a continuously-running CI agent (e.g. Auto-PR-Review in GitHub Actions) is funded. Candidate options:

  • (a) Single sponsor key — one person donates their key (fair if the agent primarily helps that person).
  • (b) Round-robin over team keys — rotate requests across all team members' keys. Requires a custom middleware (OpenRouter has no native multi-key rotation); each member must contribute their key, which is a trust/comfort question.
  • (c) Dedicated team key — ask Niklas for a separate key for shared agents (separate from the 500$/person personal keys).
  • (d) Per-feature owner pays — each agent funded by the key of the person owning that feature area.
  • (e) Self-serve only — no CI agent; everyone runs agents locally on their own key.
  • (f) Hybrid — lightweight agent on a sponsor key, heavy iterative loops only local.

Decision deferred until the spike proves a CI agent is worth running.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requeststoryteam:aiAI / Retrieval team (sprintstart-ai, Python)

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions