feat(pi): orchestrator runTask — per-task pi sessions with in-process queue tools by gewenyu99 · Pull Request #853 · PostHog/wizard

gewenyu99 · 2026-07-10T01:29:35Z

Implements runTask on the pi harness so the orchestrator runs model-agnostic through pi — pairs with PostHog/context-mill#181 (the flow-scoped agents + step-skills it consumes).

Testing

check out this branch and the context-mill PR branch (experiment/orchestrator)
serve context-mill locally: PORT=8770 npm run dev in the context-mill checkout
point REMOTE_SKILLS_BASE_URL in src/lib/constants.ts at http://localhost:8770 (revert after, it breaks the health-check unit tests while edited)
copy a fixture app somewhere disposable, e.g. wizard-workbench express-todo, without node_modules

run from the wizard checkout:

SNAP_OUT=/tmp/snaps APP_DIR=<app copy> POSTHOG_KEY_FILE=<personal api key file> PROJECT_ID=<project id> \
WIZARD_CI_FLAG_OVERRIDES='{"wizard-orchestrator":"true","wizard-use-pi-harness":"true"}' \
npx tsx scripts/tui-snapshots.no-jest.ts

expect in /tmp/posthog-wizard.log: harness=pi (flag) and sequence=orchestrator (flag), then the queue draining
expect at the end: a 7/8 or 8/8 outro, posthog-setup-report.md in the app, a real dashboard in the project

Full TUI run — 24 frames (7/8 done, 1 reasoned skip, 0 failed, 8m01s, dashboard 1825791)

01-intro — intro — orchestrator + pi flags resolved

02-auth — auth

03-run — run start

04-run — seed planning

05-run — queue seeded, install running

06-run — install

07-run — install done

08-run — init running

09-run — init

10-run — init done

11-run — parallel branch starts

12-run — identify + capture + error-tracking in parallel

13-run — instrumentation

14-run — identify skipped with reason

15-run — instrumentation done

16-run — build — real npm install

17-run — build verifying

18-run — dashboard task

19-run — dashboard insights via posthog_exec

20-run — report writing from handoffs

21-outro — outro

22-outro — outro — 7/8 steps, report on disk

23-mcp — mcp screen

24-slack-connect — slack connect

🤖 Generated with Claude Code

… queue tools, capability clamp Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ting, framework-variant step-skill resolution Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…or-pi-runtask # Conflicts: # CHANGELOG.md # package.json

…or-pi-runtask # Conflicts: # src/lib/agent/runner/harness/pi/index.ts # src/lib/agent/runner/harness/pi/mcp.ts

Orchestrator runTask agents take effort from the capabilities table (luna low, terra medium) rather than the run-wide wizard-pi-effort flag, which now steers only the linear single-agent run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… block `.env.example`/`.sample`/`.template`/`.dist` carry no secrets and are meant to be committed, so the integration can write one documenting the required keys; YARA still scans the content. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… the log Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Agents name a pi (gpt) and an sdk (anthropic) model+effort in frontmatter, not one model; the orchestrator picks the column by the resolved harness so an all-pi and an all-sdk run are both drivable from context-mill. Per-agent frontmatter effort overrides the model table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gewenyu99 · 2026-07-15T13:33:42Z

Orchestrator benchmark — gpt-5.6 (pi) vs claude (sdk) on real prod apps

Ran the orchestrator on greenfield production apps from docs/benchmarking.md, two sets, no linear control: all-pi (gpt-5.6 — terra/medium for judgment, luna/low for mechanical) and all-sdk (claude — sonnet for judgment, haiku for mechanical). n=1 per cell. The diff is the result; each was judged against the pr-evaluator rubric. 10-minute cap.

Headline

The sdk/claude set failed outright — claude-sonnet was so slow it timed out (>13 min) without finishing a single app. gpt-5.6 completes the same work in ~5 min. That gap is the whole point.

pi (gpt-5.6) results

app	stack	files	result	wall	cost	rubric
Outline	React + Koa / TS	2552	✅ 8/8	296s	$1.54	~24/26
Firefly III	Laravel / PHP	2452	✅ 8 done, 1 build-conflict	~5min*	$3.07	~24/26
healthchecks	Django	1273	✅ integrates (see caveat)	314s	$2.28	~23/26
Maybe	Rails	1644	❌ zero diff	—	—	—

* Firefly per-task durations were corrupted by a clock rollover mid-run; cost + file count are real, wall-time isn't.

Quality (rubric, 0 CRITICAL across all successes)

Outline — 10+ real product events (document_created/published/deleted, search_performed, workspace_invites_sent, reaction_added), identify(user.id) + reset() in AuthStore, captureException in ErrorBoundary, .env.example documented. Client-only on a full-stack app (no posthog-node); no reverse proxy.
Firefly III — genuinely idiomatic Laravel: a PostHogService class + config/posthog.php, captureException in the exception Handler, identify(user->id) on auth, 10+ finance events (bill_created, budget_created, transactions_exported, onboarding_completed).
healthchecks — Django server-side integration (middleware/context, exception capture), server events with distinctId + flush.

Across every success: correct real distinct_id (never email/UUID), no PII in capture() properties, framework-idiomatic init + error tracking, and .env.example written for the new keys.

Failures / findings

Maybe (Rails) → zero diff — real bug. The detected framework is rails but the step-skill variant is ruby-on-rails; resolveSkillVariantId builds posthog-integration-install-rails, which never matches, so the tasks run skill-less and write nothing. Same class as the old django/integration-django mismatch. Silent zero-diff — worth failing loudly + a name alias.
sdk/claude too slow. sonnet at the judgment tasks timed out at the cap on the first app. Not viable at this latency.
capture is the bottleneck + high-variance. It reads the whole codebase to pick events: 52s (next-js) → 87s (Outline) → 425s (healthchecks re-run, which then blew the 10-min cap before report). It also dominates cost.
Notebook step needs an MCP write scope the test key lacks — the report still lands; the notebook mirror is skipped.

Method & caveats

n=1 per cell, so single samples — capture variance alone (102s vs 425s on the same app) means times should be read as ranges, not points. Build/typecheck was not toolchain-verified on Ruby/PHP (composer/bundler absent → build reports a conflict and moves on, by design). Model map: terra (medium) = seed, identify, capture, build, dashboard; luna (low) = install, init, error-tracking, report.

…-router, tanstack-router) Three Integration enums have a context-mill variant id that differs from the enum value (rails→ruby-on-rails, react-router→react-react-router, tanstack-router→ react-tanstack-router), so the orchestrator resolved nothing and ran the tasks skill-less — a silent zero-diff. An alias map closes the gap; every enum now resolves to a variant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gewenyu99 · 2026-07-15T13:57:12Z

Follow-up: Rails variant-parity bug fixed

Fixed the framework/variant parity gap from the benchmark. Ground-truthing resolveSkillVariantId against the real menu, exactly three Integration enums resolved to nothing (silent zero-diff) because their context-mill variant id differs from the enum value:

rails → ruby-on-rails
react-router → react-react-router-*
tanstack-router → react-tanstack-router-*

Closed with an alias map in resolveSkillVariantId (both step-skill and reference resolution flow through it) + a test that every enum resolves. Re-ran Maybe (Rails): 0 files → 19 files, 8/8 done — idiomatic posthog-ruby/posthog-rails, PostHog.capture(distinct_id: Current.user…), identify with a real distinct_id helper.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…red variants, shared session machinery (#907)

sarahxsanders

cool with merging behind a flag and testing, nothing screaming at me from the code

sarahxsanders · 2026-07-15T17:53:51Z

+// `.env.dist`) carry no secrets and are meant to be committed — they document the
+// keys other developers must set. The fence lets them through so the integration
+// can write one; YARA still scans the content for any real secret.
+function isEnvExampleFile(basename: string): boolean {


just want to make sure: this will impact all harnesses and sequences as designed - if that is intended, ignore

Yeah. This is fine. Just fixes an age 'ol bug with the way we block .env.example edits

sarahxsanders · 2026-07-15T17:56:33Z

    );
  }

+  // Preflight every task's mini-skills. A missing variant means the task runs


this will silent degrade in production but does it crash in dev? maybe make it throw unconditional or fail gracefully with an error

I'm just gonna make this always throw and tell them to contact us

…909)

…wlist Renames the orchestrator content flow to integration-v2 via an additive ProgramConfig.agentFlow (program id unchanged); routes skill-menu, agent-menu, and agent-prompt fetches through a shared fetch-retry helper; and allows composer/bundle/gem in the bash command allowlist. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…or-pi-runtask # Conflicts: # src/lib/__tests__/wizard-can-use-tool.test.ts # src/lib/agent/agent-interface.ts

Adds a pi rule to install Python deps into a venv (not the externally-managed system interpreter), and restores the .env.example/.sample/.template/.dist carve-out the fence refactor dropped so the agent can write those secret-free documentation files instead of being denied. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… empty Hoists the empty-key guard to the wizard's universal commandments (applies to every SDK): an empty/placeholder token silently disables analytics. Reads from env/config, and where a build has no env (iOS/Android release/archive) embeds the real public token. Per-SDK copies removed from context-mill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

releaser-wizard Bot and others added 11 commits July 8, 2026 16:30

chore(main): release 2.39.0

2046202

feat(pi): orchestrator runTask — per-task pi sessions with in-process…

50bb273

… queue tools, capability clamp Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(pi): explicit MCP direct-tool list, assistant-only message accoun…

3d65d52

…ting, framework-variant step-skill resolution Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(pi): carry the snake_case event-naming note into task mode

34f5268

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into experiment/orchestrat…

a2a3368

…or-pi-runtask # Conflicts: # CHANGELOG.md # package.json

Merge remote-tracking branch 'origin/main' into experiment/orchestrat…

d69765a

…or-pi-runtask # Conflicts: # src/lib/agent/runner/harness/pi/index.ts # src/lib/agent/runner/harness/pi/mcp.ts

feat(pi): log per-task token usage so a run's cost is observable from…

d84aaa3

… the log Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(pi): tag per-task usage log with task type and duration

cf22c95

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gewenyu99 mentioned this pull request Jul 15, 2026

feat(agents): add agents content type, built and served alongside skills PostHog/context-mill#181

Merged

gewenyu99 and others added 2 commits July 15, 2026 10:06

chore: prettier format

1c49cea

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

chore: drop unnecessary type assertion (eslint)

d09b590

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gewenyu99 mentioned this pull request Jul 15, 2026

fix(pi): review fixes — typed effort, live model fallback, menu-declared variants, shared session machinery #907

Merged

fix(pi): review fixes — typed effort, live model fallback, menu-decla…

de41ed5

…red variants, shared session machinery (#907)

gewenyu99 marked this pull request as ready for review July 15, 2026 17:43

gewenyu99 requested a review from a team as a code owner July 15, 2026 17:43

Merge branch 'main' into experiment/orchestrator-pi-runtask

5009784

sarahxsanders approved these changes Jul 15, 2026

View reviewed changes

gewenyu99 mentioned this pull request Jul 15, 2026

feat(orchestrator): fail the run properly on a missing skill variant #909

Merged

gewenyu99 and others added 6 commits July 15, 2026 14:38

feat(orchestrator): fail the run properly on a missing skill variant (#…

970b0c9

…909)

Merge branch 'main' into experiment/orchestrator-pi-runtask

f5380c0

Merge remote-tracking branch 'origin/main' into experiment/orchestrat…

ac95aed

…or-pi-runtask # Conflicts: # src/lib/__tests__/wizard-can-use-tool.test.ts # src/lib/agent/agent-interface.ts

style: prettier formatting for the fence + commandments changes

19ffab0

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gewenyu99 merged commit da2706e into main Jul 15, 2026
16 checks passed

gewenyu99 deleted the experiment/orchestrator-pi-runtask branch July 15, 2026 23:18

releaser-wizard Bot mentioned this pull request Jul 15, 2026

chore(main): release 2.45.0 #910

Merged

gewenyu99 mentioned this pull request Jul 16, 2026

EPIC: Task-queue orchestrator runner (experimental orchestrator variant) #633

Closed

Navigation Menu

Uh oh!

feat(pi): orchestrator runTask — per-task pi sessions with in-process queue tools - #853

feat(pi): orchestrator runTask — per-task pi sessions with in-process queue tools#853
gewenyu99 merged 23 commits into
mainfrom
experiment/orchestrator-pi-runtask

gewenyu99 commented Jul 10, 2026 •

edited

Loading

Uh oh!

gewenyu99 commented Jul 15, 2026

Uh oh!

gewenyu99 commented Jul 15, 2026

Uh oh!

sarahxsanders left a comment

Uh oh!

sarahxsanders Jul 15, 2026

Uh oh!

gewenyu99 Jul 15, 2026

Uh oh!

sarahxsanders Jul 15, 2026

Uh oh!

gewenyu99 Jul 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

gewenyu99 commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

gewenyu99 commented Jul 15, 2026

Orchestrator benchmark — gpt-5.6 (pi) vs claude (sdk) on real prod apps

Headline

pi (gpt-5.6) results

Quality (rubric, 0 CRITICAL across all successes)

Failures / findings

Method & caveats

Uh oh!

gewenyu99 commented Jul 15, 2026

Follow-up: Rails variant-parity bug fixed

Uh oh!

sarahxsanders left a comment

Choose a reason for hiding this comment

Uh oh!

sarahxsanders Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

gewenyu99 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

sarahxsanders Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

gewenyu99 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gewenyu99 commented Jul 10, 2026 •

edited

Loading