Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions context/commandments.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -334,4 +334,5 @@ commandments:
- "AI Observability carve-out: this skill instruments LLM calls and is not product-analytics coverage. Do NOT add posthog.capture() events for user actions, captureException() error handlers, or a reverse proxy unless the user explicitly asks for them"
- "AI Observability carve-out: only the wrapper-client and manual-capture install paths construct a PostHog client. The OTel and framework-hook paths have no client at all, so any rule in this file about the Posthog()/PostHog() constructor, exception autocapture, atexit/shutdown registration or flushing simply does not apply — never invent a client just to satisfy one"
- "AI Observability carve-out: the $ai_* payload properties ($ai_input, $ai_output_choices, and the rest) intentionally carry user-generated prompt and completion text, so this file's PII rules do NOT apply to them. Those rules still govern every other property you set"
- "AI Observability carve-out: the Go OpenTelemetry path uses the nested github.com/posthog/posthog-go/otel module, not the core github.com/posthog/posthog-go SDK. This file's Go install and import line does not apply, and neither do its posthog.NewWithConfig client rules: the bridge registers a span processor and builds no PostHog client"
- "AI Observability carve-out: read the PostHog key and host exactly as the variant's install doc reads them. A direct os.environ[\"POSTHOG_API_KEY\"] / process.env lookup already fails loudly and idiomatically when unset, which satisfies this file's missing-configuration rule — do NOT add a separate presence check, guard branch, or custom raise around a bootstrap that is only a few lines long"
8 changes: 8 additions & 0 deletions context/skills/ai-observability/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -571,6 +571,14 @@ variants:
- https://posthog.com/docs/ai-observability/installation/opentelemetry.md
- https://posthog.com/docs/libraries/node.md

- id: opentelemetry-go
framework: opentelemetry
display_name: OpenTelemetry (Go)
tags: [opentelemetry, go, ai_observability]
docs_urls:
- https://posthog.com/docs/ai-observability/installation/opentelemetry.md
- https://posthog.com/docs/libraries/go.md

- id: claude-code
framework: claude-code
display_name: Claude Code
Expand Down
4 changes: 2 additions & 2 deletions context/skills/ai-observability/description.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Wire up PostHog's AI Observability so calls made through {display_name} land in

This skill instruments the LLM calls the project *already makes*. It does **not** install the vendor SDK for you.

Check the project's manifest for an LLM package. The catalog is far wider than the obvious providers — 68 variants covering agent frameworks (`openai-agents`, `claude-agent-sdk`, LangGraph, CrewAI, Mastra, …) and OpenAI-compatible gateways (Groq, OpenRouter, Together, Ollama, …), which an app reaches through the `openai` package plus a `baseURL` override. `1-begin.md` carries the ordered decision rules; follow them rather than matching on the first familiar package name. If no LLM SDK is present, switch to the `manual-capture` variant — it posts `$ai_generation` events directly and works standalone.
Check the project's manifest for an LLM package. The catalog is far wider than the obvious providers — 69 variants covering agent frameworks (`openai-agents`, `claude-agent-sdk`, LangGraph, CrewAI, Mastra, …) and OpenAI-compatible gateways (Groq, OpenRouter, Together, Ollama, …), which an app reaches through the `openai` package plus a `baseURL` override. `1-begin.md` carries the ordered decision rules; follow them rather than matching on the first familiar package name. If no LLM SDK is present, switch to the `manual-capture` variant — it posts `$ai_generation` events directly and works standalone.

Everything else this skill needs — PostHog credentials, instrumentation packages, env vars — the skill installs and configures itself. It does **not** require a pre-existing `posthog.init(...)`. If one is already there, reuse its env-var names in `3-instrument.md`; if not, that step sets fresh values via `set_env_values`.

Expand All @@ -28,7 +28,7 @@ The linked install page carries the exact code blocks for this variant's languag
## Key principles

- **Environment variables.** Read `<ph_project_token>` and `<ph_client_api_host>` from env, using the framework's env-var convention. Never hardcode either value.
- **The SDK wrapper is the default, not OpenTelemetry.** OTel makes the session tree awkward to build and maintain, so provider and gateway variants use PostHog's drop-in wrapper client. Reserve OTel for the `opentelemetry-*` variants and LlamaIndex, and never swap a framework's own tracing hook for an instrumentor.
- **The SDK wrapper is the default, not OpenTelemetry.** OTel makes the session tree awkward to build and maintain, so provider and gateway variants use PostHog's drop-in wrapper client. Reserve OTel for the `opentelemetry-*` variants and LlamaIndex, and never swap a framework's own tracing hook for an instrumentor. Go is the exception: it has no wrapper SDK, so model calls made from Go use the `opentelemetry-go` bridge.
- **Minimal changes.** The wrapper swaps a client constructor and adds parameters to existing calls. Don't restructure the app, and don't wrap the setup in an init function or module globals.
- **Match the docs.** Package names and wrapper imports change between AIO releases. The install page for this variant is the source of truth.
- **Cardinality is what gets graded.** One `$ai_session_id` per conversation, one `posthog_trace_id` per turn, shared by every call in it. An id minted per call is worse than none — it looks instrumented and groups nothing.
Expand Down
20 changes: 14 additions & 6 deletions context/skills/ai-observability/references/1-begin.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,19 @@ Pick the variant, then read the code. Do not edit anything in this step.

## Pick the variant

This skill ships 68 variants. Call `load_skill_menu` with `category: "ai-observability"`. That list is the source of truth.
This skill ships 69 variants. Call `load_skill_menu` with `category: "ai-observability"`. That list is the source of truth.

Apply these rules in order. The first match wins. Frameworks wrap providers, and gateways look like OpenAI, so the order matters.

### 1. A framework wins over the provider under it
### 1. Model calls made from Go take OpenTelemetry

Go has no PostHog wrapper SDK, so a project whose model calls are in Go takes `opentelemetry-go`, whatever SDK or framework makes them. The bridge is `github.com/posthog/posthog-go/otel`, and the install doc carries the code.

A `go.mod` on its own is not the signal. If the repo also has a `package.json`, `pyproject.toml`, or `requirements.txt`, find the call sites before you route. A Go service beside a Node or Python LLM app takes the rules below. If it stays unclear, use `wizard_ask`.

The bridge needs Go 1.25 or newer. An older toolchain fails with `module requires go >= 1.25.0`. Say so in the report rather than raising the project's Go version yourself.

### 2. A framework wins over the provider under it

| Package in the manifest | Variant |
|---|---|
Expand All @@ -36,23 +44,23 @@ Apply these rules in order. The first match wins. Frameworks wrap providers, and

Instrument the framework, not the provider below it. A provider instrumentor keeps the model calls and loses the agent, tool, and handoff structure.

### 2. An `openai` client with a base URL override means a gateway
### 3. An `openai` client with a base URL override means a gateway

Most OpenAI-compatible providers ship no SDK. Apps call them with the `openai` package aimed at another host. Check the client constructor and `OPENAI_BASE_URL`. Common hosts are `api.groq.com`, `openrouter.ai`, `api.together.xyz`, and `localhost:11434`.

Pick the variant that names the provider. The install shape matches plain OpenAI, but the provider name does not. Step 3 explains why that matters.

### 3. A plain provider SDK maps to that provider
### 4. A plain provider SDK maps to that provider

`openai`, `anthropic`, `@anthropic-ai/sdk`, `google-genai`, `@google/genai`, `mistralai`, and `cohere` each have a variant. `boto3` with `bedrock-runtime` maps to `aws-bedrock`.

### 4. Anything else
### 5. Anything else

- Several candidates and no framework: prefer the higher-level one. If it stays unclear, use `wizard_ask` with the candidates as options.
- The app already emits its own OTel spans: `opentelemetry-{python,node}`.
- No LLM SDK at all: `manual-capture`.

Language follows the manifest. A `package.json` means Node. A `pyproject.toml` or `requirements.txt` means Python. Framework variants have no language suffix.
Language follows the manifest. A `package.json` means Node. A `pyproject.toml` or `requirements.txt` means Python. Go call sites went to `opentelemetry-go` under rule 1. Framework variants have no language suffix.

Report the variant and the reason in a `[STATUS]` line, then call `install_skill` with the full id.

Expand Down
6 changes: 5 additions & 1 deletion context/skills/ai-observability/references/2-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,13 @@ Portkey also needs `portkey-ai`.

AWS Bedrock has no wrapper client. It instruments the AWS SDK through OpenTelemetry, so its package list differs from every other provider.

`opentelemetry-go` declares `github.com/posthog/posthog-go/otel` and the OpenTelemetry SDK modules in `go.mod`. There are no Go instrumentation libraries for provider SDKs; step 3 hand-authors the `gen_ai.*` spans instead.

Go is the exception to the no-package-manager and no-lockfile rules on this page. A `go.mod` require line with no matching `go.sum` entry fails the build with `missing go.sum entry for module`, so let the tool write both files: `go get github.com/posthog/posthog-go/otel go.opentelemetry.io/otel go.opentelemetry.io/otel/sdk`, then `go mod tidy`. Do not hand-edit `go.mod` or `go.sum`.

## Do not

- Do not run `npm install` or `pip install`.
- Do not edit the lockfile.
- Do not edit the lockfile. On Go, `go get` and `go mod tidy` write `go.mod` and `go.sum` for you.
- Do not upgrade the vendor SDK.
- Do not add OpenTelemetry to a wrapper variant.
16 changes: 15 additions & 1 deletion context/skills/ai-observability/references/3-instrument.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,20 @@ Route the token and host through env vars with `set_env_values`. Reuse the names

Agent frameworks use their own tracing hook in place of a wrapper. Take it from the install doc. Do not substitute an OTel instrumentor.

### The OpenTelemetry path

On the `opentelemetry-*` variants, which is where Go call sites land, there is no wrapper to swap and no PostHog client to build. Register the PostHog span processor from the install doc on the tracer provider the app already owns. Flush before exit with `ForceFlush` or `Shutdown`, or the buffered spans are lost.

The processor forwards a span only when its **name** or one of its **span attribute keys** starts with `gen_ai.`, `llm.`, `ai.`, or `traceloop.`. It drops every other span silently, with no error. Two things follow:

- Start one span per turn so the calls in it share a trace, and make that turn span pass the filter. Name it in the `gen_ai.` or `ai.` namespace, or give it a `gen_ai.*` attribute. A turn span named `handle turn` never reaches PostHog, and the generations under it arrive with no root.
- `$ai_session_id` begins with `$`, so it does not satisfy the filter on its own. The filter reads span attributes only, never resource attributes. Set `$ai_session_id` on every span in the conversation, and carry `posthog.distinct_id` the way the install doc shows.

Cardinality is the same as the wrapper path: one `$ai_session_id` per conversation, one trace per turn. The OTel trace carries the grouping, so the per-call parameters in the next section do not exist here.

## Attach identity to every call

Three per-call parameters carry the tree. Node uses camelCase.
On the wrapper path, three per-call parameters carry the tree. Node uses camelCase. The OpenTelemetry path has none of them; the section above covers it.

| Parameter | Holds | Cardinality |
|---|---|---|
Expand Down Expand Up @@ -54,6 +65,8 @@ The wrapper records the model call. It never sees the tool dispatch loop, so not

If the app registers tools, capture each run as an `$ai_span` event with `posthog.capture()`. Give it the turn's `$ai_trace_id` so the span joins the trace. The install doc lists the span properties.

On the OpenTelemetry path there is no PostHog client to call. Record the tool run as a child span of the turn span instead, with `gen_ai.*` attributes so it passes the AI span filter. A plain span named after the tool is dropped.

Put the capture next to the existing dispatch. Do not restructure the tool loop.

Agent frameworks and the Vercel AI SDK emit tool spans on their own. Add nothing on those variants.
Expand All @@ -64,6 +77,7 @@ An app that registers no tools has no spans. That is a complete result, not a ga

- Do not restructure the app. This step swaps a constructor and adds arguments to calls.
- Do not omit `posthog_trace_id` and expect the calls to group.
- Do not leave a turn span that fails the AI span filter on the OpenTelemetry path.
- Do not mint a session id per call or per turn.
- Do not leave a gateway reporting `$ai_provider` as `openai`.
- Do not add spans when the app registers no tools.
Expand Down
1 change: 1 addition & 0 deletions context/skills/ai-observability/references/4-verify.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ Before you hand over, run the import the code depends on, such as `python3 -c "f
| `$ai_provider` says `openai` on a gateway | the per-call override is missing |
| Anonymous person | `posthog_distinct_id` is missing |
| No tool spans | the `$ai_span` captures are missing or carry another trace id |
| A trace with no root, on the OTel path | the turn span fails the bridge's AI filter, so the processor drops it |
| Nothing arrives | the code still calls the vendor client, the key or host is unset, or a short script exits before the flush |

## Do not
Expand Down
Loading