Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,15 @@ command names.** Old names mostly no longer exist — only some are kept as alia
| `wizard audit session-replay` | session replay setup |
| `wizard audit web-analytics` | web analytics setup (**wizard-native**, not a skill) |

### Feature flags

`wizard feature-flags` is a **flat skill command** (same shape as
`revenue-analytics` / `mcp-analytics`). It runs the `feature-flags-setup`
context-mill skill: Next.js App Router 15.3+, extends an existing PostHog
install (aborts if not initialized), server `evaluateFlags()` + client
bootstrap, skip-first, optional 0% boolean flag + additive UI path after
confirm. Distinct from `wizard audit feature-flags`, which is read-only.

### Commands vs. skills (the `audit [skill]` gotcha)

A skill and a command are the **same machinery** — a context-mill skill becomes a
Expand All @@ -86,7 +95,7 @@ confuse it with the top-level `wizard skill` command.
- **Registration:** [`bin.ts`](bin.ts) — the `.use()` chain wires each command.
- **Command shape:** [`src/commands/command.ts`](src/commands/command.ts) — the
`Command` interface every command implements.
- **Flat native commands** (e.g. `revenue-analytics`, `upload-source-maps`) are
- **Flat native commands** (e.g. `feature-flags`, `revenue-analytics`, `upload-source-maps`) are
built with `nativeCommandFactory`
([`src/commands/factories/native-command-factory.ts`](src/commands/factories/native-command-factory.ts)).
- **Family commands** (e.g. `audit`) resolve subcommands at runtime against the
Expand Down
21 changes: 19 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,23 @@ new audits appear without a wizard release (`web-analytics` is wizard-native).
> (`wizard audit --help` still labels the positional `[skill]` — read it as "pick
> a subcommand.")

### Feature flags

Add the cheap feature-flags path to an existing PostHog install on Next.js App
Router 15.3+: evaluate once per request on the server, bootstrap those values
into the client, and disable `/flags` polling in CI. After one confirm,
optionally create one boolean flag at **0% rollout** and gate one additive UI
path. Skip is the default — no new flag and no UI change. Production users keep
current behavior until someone raises rollout in PostHog.

```bash
npx @posthog/wizard feature-flags
```

Requires an existing PostHog install (`npx @posthog/wizard`). Next.js App Router
15.3+ only. Distinct from `wizard audit feature-flags`, which audits existing
flag usage and cost and does not install anything.

### Revenue Analytics

Wire up an existing PostHog + Stripe project for revenue analytics:
Expand Down Expand Up @@ -165,8 +182,8 @@ route review to their owning team instead.
| `src/lib/programs/web-analytics-doctor/` | `@PostHog/team-web-analytics` |

Ownership is by directory. Programs not listed above
(`agent-skill`, `audit`, `events-audit`, `mcp`, `migration`, `posthog-doctor`,
`shared`, `slack`) fall through the default and are owned by
(`agent-skill`, `audit`, `events-audit`, `feature-flags`, `mcp`, `migration`,
`posthog-doctor`, `shared`, `slack`) fall through the default and are owned by
`team-wizard-docs`. Today CODEOWNERS only auto-requests review — approval is
not a merge gate.

Expand Down
2 changes: 2 additions & 0 deletions bin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ import { Wizard } from './src/wizard';
import { basicIntegrationCommand } from './src/commands/basic-integration';
import { mcpCommand } from './src/commands/mcp';
import { mcpAnalyticsCommand } from './src/commands/mcp-analytics';
import { featureFlagsCommand } from './src/commands/feature-flags';
import { replayVisionCommand } from './src/commands/replay-vision';
import { aiObservabilityCommand } from './src/commands/ai-observability';
import { metricsCommand } from './src/commands/metrics';
Expand Down Expand Up @@ -88,6 +89,7 @@ function resolveInstallDir(): string {
Wizard.use(basicIntegrationCommand)
.use(mcpCommand)
.use(mcpAnalyticsCommand)
.use(featureFlagsCommand)
.use(replayVisionCommand)
.use(aiObservabilityCommand)
.use(metricsCommand)
Expand Down
12 changes: 12 additions & 0 deletions src/__tests__/programs-cli.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import type { MockedFunction } from 'vitest';
import { auditCommand } from '../commands/audit';
import { migrateCommand } from '../commands/migrate';
import { mcpAnalyticsCommand } from '../commands/mcp-analytics';
import { featureFlagsCommand } from '../commands/feature-flags';
import { replayVisionCommand } from '../commands/replay-vision';
import { revenueCommand } from '../commands/revenue';
import { warehouseCommand } from '../commands/warehouse';
Expand Down Expand Up @@ -86,6 +87,11 @@ describe('top-level command shapes', () => {
expect(mcpAnalyticsCommand.children).toBeUndefined();
});

test('feature-flags is a flat skill command', () => {
expect(featureFlagsCommand.name).toBe('feature-flags');
expect(featureFlagsCommand.children).toBeUndefined();
});

test('replay-vision is a flat skill command', () => {
expect(replayVisionCommand.name).toBe('replay-vision');
expect(replayVisionCommand.children).toBeUndefined();
Expand Down Expand Up @@ -190,6 +196,12 @@ describe('flat skill commands', () => {
expect(config.skillId).toBe('mcp-analytics');
});

test('feature-flags dispatches with feature-flags-setup skillId', () => {
featureFlagsCommand.handler!(makeArgv({ debug: true }));
const [config] = mockRunWizard.mock.calls[0] as [{ skillId?: string }];
expect(config.skillId).toBe('feature-flags-setup');
});

test('replay-vision dispatches with replay-vision-setup skillId', () => {
replayVisionCommand.handler!(makeArgv({ debug: true }));
const [config] = mockRunWizard.mock.calls[0] as [{ skillId?: string }];
Expand Down
14 changes: 14 additions & 0 deletions src/commands/feature-flags.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import { featureFlagsConfig } from '@lib/programs/feature-flags/index';

import type { Command } from './command';
import { nativeCommandFactory } from './factories/native-command-factory';

/**
* `wizard feature-flags` — flat skill command.
*
* Distinct from `wizard audit feature-flags` (read-only cost/correctness
* audit) and from `wizard migrate` (come from another vendor). Stays flat
* while install-and-instrument is the only action.
*/
export const featureFlagsCommand: Command =
nativeCommandFactory(featureFlagsConfig);
19 changes: 19 additions & 0 deletions src/lib/agent/runner/__tests__/switchboard.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ describe('switchboard PROGRAM_BINDINGS', () => {
if (program === 'error-tracking-upload-source-maps') continue; // pinned below
if (program === 'metrics') continue; // pinned below
if (program === 'replay-vision') continue; // pinned below
if (program === 'feature-flags') continue; // pinned below
expect(resolveBinding({ program, flags: {} })).toEqual(DEFAULT_RESOLVED);
}
});
Expand Down Expand Up @@ -91,6 +92,17 @@ describe('switchboard PROGRAM_BINDINGS', () => {
},
trace: { harness: 'binding', model: 'binding', sequence: 'binding' },
},
{
name: 'binds feature-flags to pi + terra medium',
ctx: { program: 'feature-flags', flags: {} },
binding: {
sequence: Sequence.linear,
harness: Harness.pi,
model: GPT5_6_TERRA_MODEL,
thinkingLevel: 'medium',
},
trace: { harness: 'binding', model: 'binding', sequence: 'binding' },
},
{
name: 'binds metrics to the orchestrator on pi; stage models come from the flow frontmatter',
ctx: { program: 'metrics', flags: {} },
Expand Down Expand Up @@ -232,6 +244,13 @@ describe('switchboard composed clamp', () => {
model: GPT5_6_SOL_MODEL,
thinkingLevel: 'medium',
}
: program === 'feature-flags'
? {
...DEFAULT_RESOLVED,
harness: Harness.pi,
model: GPT5_6_TERRA_MODEL,
thinkingLevel: 'medium',
}
: program === 'metrics'
? { ...DEFAULT_RESOLVED, harness: Harness.pi }
: DEFAULT_RESOLVED,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,14 @@ describe('isolation — everything on at once', () => {
model: GPT5_6_SOL_MODEL,
thinkingLevel: 'medium',
});
} else if (program === 'feature-flags') {
// Pi + terra medium from its OWN binding, not the flag.
expect(resolved).toEqual({
sequence: Sequence.linear,
harness: Harness.pi,
model: GPT5_6_TERRA_MODEL,
thinkingLevel: 'medium',
});
} else if (program === 'replay-vision') {
// Orchestrator from its OWN binding, not the flag — the
// wizard-orchestrator experiment does not cover this program, so it
Expand Down
7 changes: 7 additions & 0 deletions src/lib/agent/runner/switchboard/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import {
DEFAULT_AGENT_MODEL,
GPT5_6_SOL_MODEL,
GPT5_6_TERRA_MODEL,
SONNET_5_MODEL,
Harness,
Sequence,
Expand Down Expand Up @@ -141,6 +142,12 @@ export const PROGRAM_BINDINGS: Partial<Record<ProgramId, ProgramBinding>> = {
'mcp-remove': DEFAULT_BINDING,
'mcp-tutorial': DEFAULT_BINDING,
'mcp-analytics': DEFAULT_BINDING,
'feature-flags': {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd love to see evals if you have any for why this model! it's okay if it's just personal preference too :)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually Tried Vincent’s Slack hint out of curiosity. I don't have any real evals I just wanted to see another model appear in the logs 🤣 I’ll revert to default Sonnet unless you’d rather keep the experiment.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need! was just curious :)

sequence: Sequence.linear,
harness: Harness.pi,
model: GPT5_6_TERRA_MODEL,
thinkingLevel: 'medium',
},
// Orchestrator on pi. The binding routes only; every stage's model and
// effort are pinned context-mill side in the flow's frontmatter
// (`model_pi`/`effort_pi`: terra seed, sol tasks, luna report).
Expand Down
27 changes: 27 additions & 0 deletions src/lib/oauth/__tests__/program-scopes.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,26 @@ describe('posthog-integration scopes', () => {
});
});

describe('feature-flags scopes', () => {
it('can list and create flags after the user confirms a gate target', () => {
const scopes = getOAuthScopesForProgram('feature-flags');
expect(scopes).toContain('feature_flag:read');
expect(scopes).toContain('feature_flag:write');
});

it('does not request person-property targeting scopes', () => {
expect(getOAuthScopesForProgram('feature-flags')).not.toContain(
'property_definition:read',
);
});

it('does not strip the base completion scopes', () => {
expect(getOAuthScopesForProgram('feature-flags')).toEqual(
expect.arrayContaining([...getOAuthScopesForProgram(null)]),
);
});
});

/**
* Run 69afc6f8 requested only the base set, so the PostHog MCP served a
* catalog without the scanner tools: every scanner task took its "tool
Expand Down Expand Up @@ -60,6 +80,13 @@ describe('provisioning scopes', () => {
expect(scopes).toContain('project:read');
});

it('layers feature-flags write scopes on the provisioning base', () => {
const scopes = getProvisioningScopesForProgram('feature-flags');
expect(scopes).toContain('feature_flag:write');
expect(scopes).toContain('feature_flag:read');
expect(scopes).not.toContain('property_definition:read');
});

it('keeps other programs on the unmodified base', () => {
expect(getProvisioningScopesForProgram(null)).not.toContain(
'replay_scanner:write',
Expand Down
20 changes: 19 additions & 1 deletion src/lib/oauth/program-scopes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@
* Current additions: `McpTutorial` layers read-only on every product
* surface (feature flags, experiments, surveys, replays, errors, web
* analytics, LLM analytics, cohorts, persons) plus read/write on
* annotations; `AgentSkill` adds feature-flag read/write; the default
* annotations; `AgentSkill` adds feature-flag read/write plus
* `property_definition:read`; `feature-flags` adds only flag read/write; the default
* `PostHogIntegration` run and the standalone `slack` flow add
* `integration:read` for the Connect-Slack step. Persistence writes (dashboard:write,
* insight:write, notebook:write, query:read) come for free from the
Expand Down Expand Up @@ -122,6 +123,22 @@ export const AGENT_SKILL_SCOPE_ADDITIONS = [
'property_definition:read',
] as const;

/**
* Extra scopes `wizard feature-flags` needs on top of `WIZARD_OAUTH_SCOPES`.
*
* After the user confirms a gate target the skill creates one boolean flag
* at 0% rollout. Without `feature_flag:write` the MCP catalog hides
* create-feature-flag and the run cannot finish. `:write` does not imply
* `:read`, so listing existing keys still needs `feature_flag:read`.
*
* Narrower than `AGENT_SKILL_SCOPE_ADDITIONS`: this install does not build
* person-property targeting, so it does not request `property_definition:read`.
*/
export const FEATURE_FLAGS_SCOPE_ADDITIONS = [
'feature_flag:read',
'feature_flag:write',
] as const;

/**
* Extra scopes the self-driving program needs on top of
* `WIZARD_OAUTH_SCOPES`. All consumed by the PostHog MCP tools the
Expand Down Expand Up @@ -268,6 +285,7 @@ const PROGRAM_SCOPE_ADDITIONS: Partial<Record<ProgramId, readonly string[]>> = {
// ever changes, this line will fail to type-check.
'mcp-tutorial': MCP_TUTORIAL_SCOPE_ADDITIONS,
'agent-skill': AGENT_SKILL_SCOPE_ADDITIONS,
'feature-flags': FEATURE_FLAGS_SCOPE_ADDITIONS,
'self-driving': SELF_DRIVING_SCOPE_ADDITIONS,
'warehouse-source': WAREHOUSE_SOURCE_SCOPE_ADDITIONS,
// The integration run carries the Slack outro step, and — when detection
Expand Down
Loading