Measure Magpie skill economics against LLMAO's three models

[potiuk]

What

Run the mode-economics harness against LLMAO's three models and record both tokens and wall-clock time per skill, so the published economics stop being single-vendor.

Depends on #1260 confirming the gateway is usable — in particular whether tool use works, since every Magpie skill is tool-driven.

Why

docs/mode-economics.md exists because affordability is a mission commitment, not a nice-to-have. Right now every figure on it comes from one vendor's models. A second, ASF-hosted backend turns "what does this cost" from a vendor-specific number into a comparison, and it is the evidence the page's own Long-term: the ASF inference endpoint section already promises:

As pilot adopters accumulate real usage data, this page will be updated with observed ranges rather than theoretical estimates, so the endpoint sizing argument rests on evidence.

It also runs in the other direction. ASF Tooling have capacity-planned from synthetic load and a few hundred recorded requests. Magpie's skills are a real maintainer workload, and Tooling asked for exactly this.

Models

id	context	reasoning on by default
gemma4-26b	131,072	no — the recommended default
qwen3.8-27b	131,072	yes
qwen3-8b	40,960	yes

A reasoning-on model emits nothing while it thinks, and agent clients abandon the stream and retry — so the box generates twice for one unread response. Expect that to distort naive wall-clock numbers on two of the three, and measure it rather than working around it.

Workloads

Two tiers, and the second is the one that matters:

1. The existing harness — the bounded replay already used for the published figures, for a like-for-like comparison.
2. Real Apache Airflow work — issue-triage and the security-family skills over actual Airflow issues. Airflow is a genuine, large maintainer workload rather than a constructed one, which is what makes the numbers worth anything to Tooling's capacity planning. Contrived prompts would tell them very little.

Done when

• Tokens and wall-clock recorded per skill, per model, for both tiers.
• Failure modes recorded too — refusals, timeouts, abandoned streams, context-window overruns on qwen3-8b's 40,960.
• Results written up (see the ASF-scoped section issue).
• Findings shared back with ASF Tooling, who want the real-load data.

Constraint — this is a shared, unmetered resource

There is no budgeting, quota, access control or fairness mechanism on the gateway. The $100 in the UI is a placeholder and spend_usd reports 0.00 for self-hosted models. Capacity consumed is capacity another community cannot use, and Tooling expect only two people to be exercising it at all.

So this run must be deliberately paced rather than opportunistic:

• rate-limit it, and do not run the three models concurrently;
• agree a window with Tooling before a large pass, rather than discovering the ceiling by hitting it;
• gemma4-26b saturates at ~4 concurrent requests and qwen3.8-27b at 20+, so concurrency is not a free variable.

Treat "we did not degrade the service for anyone else" as an acceptance criterion, not etiquette.