chore: sync skills (agent-skills-v0.880.0, context-mill@v1.51.0)

.claude-plugin/marketplace.json

@@ -12,7 +12,7 @@
       "displayName": "PostHog",
       "source": "./",
       "description": "Access PostHog analytics, feature flags, experiments, error tracking, and insights directly from your AI coding tool. Optionally capture Claude Code sessions to PostHog LLM Analytics.",
-      "version": "1.1.62",
+      "version": "1.1.63",
       "author": {
         "name": "PostHog",
         "email": "hey@posthog.com",

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "posthog",
   "description": "Access PostHog analytics, feature flags, experiments, error tracking, and insights directly from your AI coding tool. Optionally capture Claude Code sessions to PostHog LLM Analytics.",
-  "version": "1.1.62",
+  "version": "1.1.63",
   "author": {
     "name": "PostHog",
     "email": "hey@posthog.com",

.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "posthog",
-  "version": "1.0.60",
+  "version": "1.0.61",
   "description": "Access PostHog analytics, feature flags, experiments, error tracking, and insights directly from Codex",
   "author": {
     "name": "PostHog",

.cursor-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "posthog",
   "displayName": "PostHog",
-  "version": "1.1.56",
+  "version": "1.1.57",
   "description": "Access PostHog analytics, feature flags, experiments, error tracking, and insights directly from Cursor",
   "author": {
     "name": "PostHog",

gemini-extension.json

@@ -1,6 +1,6 @@
 {
   "name": "posthog",
-  "version": "1.0.58",
+  "version": "1.0.59",
   "description": "Access PostHog analytics, feature flags, experiments, error tracking, and insights directly from Gemini CLI",
   "mcpServers": {
     "posthog": {

skills/creating-online-evaluations/SKILL.md

@@ -2,15 +2,15 @@
 name: creating-online-evaluations
 description: >
   Author continuously-running online evaluations in PostHog AI observability, grounded in real failure
-  modes you've identified. Use when the user wants evaluations that automatically score new generations
-  or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y",
+  modes you've identified. Use when the user wants evaluations that automatically score new generations or
+  whole traces going forward — "create an eval to catch X", "continuously check that responses do Y",
   "turn these failures into evals". Covers letting the explored data decide how many evals to create,
-  proposing that set in plain language and asking the user which ones they want, choosing the target and
-  eval type (hog / llm_judge / sentiment), configuring a provider, model, and usable provider key for an
-  llm_judge eval, scoping which generations trigger it via conditions, creating disabled, verifying scope,
-  and enabling. Falls back to proposing a sentiment eval when no failure mode is worth catching.
-  Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first.
-  To debug or manage evaluations that already exist, use exploring-llm-evaluations.
+  proposing that set for the user to pick, choosing the target and eval type (hog / llm_judge /
+  sentiment), configuring a provider and model for an llm_judge eval (a provider key gates enabling, not
+  creation), scoping which generations trigger it via conditions, creating disabled, verifying scope, and
+  enabling. Proposes a sentiment eval when no failure mode is worth catching. Finding and ranking the
+  failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage
+  evaluations that already exist, use exploring-llm-evaluations.
 ---
 
 # Creating online evaluations
@@ -129,11 +129,11 @@ set into production, which is noise and (for a judge) cost the user didn't agree
 
 ### 2.1 — Choose the eval type
 
-| Use…        | When the criterion is…                                                                                                                |
-| ----------- | ------------------------------------------------------------------------------------------------------------------------------------- |
-| `hog`       | Structural / rule-based (JSON parses, length, regex, tool-call shape). Cheap, deterministic, **no provider key needed.**              |
-| `llm_judge` | Subjective / fuzzy (tone, factuality, on-topic). Costs an LLM call per run; needs a provider, model, and usable provider key.         |
-| `sentiment` | You want sentiment labels on user messages, not a pass/fail (unless very specifically asked for, usually not relevant to this skill). |
+| Use…        | When the criterion is…                                                                                                                                                            |
+| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `hog`       | Structural / rule-based (JSON parses, length, regex, tool-call shape). Cheap, deterministic, **no provider key needed.**                                                          |
+| `llm_judge` | Subjective / fuzzy (tone, factuality, on-topic). Costs an LLM call per run; needs a provider and model. A usable provider key is only needed to enable it, not to create a draft. |
+| `sentiment` | You want sentiment labels on user messages, not a pass/fail (unless very specifically asked for, usually not relevant to this skill).                                             |
 
 Reach for `hog` first, escalate to `llm_judge` if there is no deterministic way to check for what we want to check.
 
@@ -210,12 +210,14 @@ when exact structure matters.
 
 ### 2.3 — Configure the LLM judge
 
-An `llm_judge` evaluation requires a valid `provider` and `model`. It also needs a usable provider key
-when it runs. `provider_key_id` controls whether the evaluation pins one specific key:
+An `llm_judge` evaluation requires only a valid `provider` and `model` to create as a draft. A usable
+provider key is needed to enable it, not to save it, so no `ok` key is a reason to keep the eval disabled,
+not a reason to stop. `provider_key_id` controls whether the evaluation pins one specific key:
 
 - Set `provider_key_id` to the UUID of an `ok`-state key for the same provider to pin it.
-- Set `provider_key_id` to `null` to use the team's active provider key. The active key must be in the
-  `ok` state and use the same provider as `model_configuration.provider`.
+- Set `provider_key_id` to `null` to use the team's active provider key. This is always a valid saved
+  value. To enable the eval, the active key must be in the `ok` state and use the same provider as
+  `model_configuration.provider`.
 
 Hog and sentiment evaluations skip this step.
 
@@ -229,10 +231,12 @@ Confirm the provider and model with `llma-evaluation-judge-models`.
 Call it with no arguments to see the whole catalog at once.
 Providers PostHog funds no models for come back empty unless you pass `key_id` for one of the team's keys; the response's `providers` list flags which ones those are.
 Prefer pinning the chosen key so a later team-wide active-key change does not change how the evaluation runs.
-Leave `provider_key_id` as `null` only after `llma-evaluation-config-get` confirms the active key is usable and its provider matches.
+Leave `provider_key_id` as `null` to run on the team's active key; before enabling, confirm with
+`llma-evaluation-config-get` that the active key is usable and its provider matches.
 
-If there is no usable key, you may still create a disabled draft for the user to review. Do not spot-run or
-enable it. Ask the user to add or validate a key in the UI before continuing.
+No usable key does not block creation. Create the disabled draft with a valid `provider` and `model` for
+the user to review. Do not spot-run or enable it, and ask the user to add or validate a key in the UI so it
+can be enabled later.
 
 ### 2.4 — Create it disabled
 
@@ -261,8 +265,9 @@ posthog:llma-evaluation-create
 ```
 
 For `llm_judge`, swap `evaluation_config` to `{ "prompt": "…" }` and add
-`"model_configuration": { "provider": "openai", "model": "gpt-5-mini", "provider_key_id": "<uuid of an ok-state key from llma-provider-key-list>" }`.
-Use `null` only when the active team key is `ok` and uses the same provider. Full field reference:
+`"model_configuration": { "provider": "openai", "model": "gpt-5-mini", "provider_key_id": null }`.
+Pin `provider_key_id` to the UUID of an `ok`-state key from `llma-provider-key-list` to run on one specific
+key; `null` runs on the team's active key. Either value saves a draft. Full field reference:
 [references/evaluation-payload.md](references/evaluation-payload.md).
 
 ### 2.5 — Verify the scope before enabling
@@ -283,7 +288,8 @@ For generation targets, `count()` is the run volume. For trace targets, count di
 `$ai_trace_id` values because matching generations from the same trace schedule only one run.
 
 If volume is high, set `rollout_percentage` below 100 to sample. Spot-check the evaluator with
-`llma-evaluation-test-hog` (hog) or `llma-evaluation-run` against one generation (llm_judge).
+`llma-evaluation-test-hog` (hog) or `llma-evaluation-run` against one generation (llm_judge; skip the
+spot-run for a judge with no usable key, per 2.3).
 Both tools currently use generation samples; for a trace target they can check shared source or prompt behavior,
 but they do not reproduce the complete settled trace. Review the first live trace results before increasing rollout.
 
@@ -354,8 +360,9 @@ creating so the user can review and toggle it in the UI.
   criterion genuinely can't be coded.
 - **Always create disabled, verify scope, then enable.** An eval firing on the wrong events is worse than
   none — noise, and (for llm_judge) cost.
-- **Configure llm_judge credentials before running.** A judge needs a valid provider and model plus a usable
-  provider key. `provider_key_id` may be `null` only when the matching active team key can be used.
+- **A judge draft needs only a provider and model.** A usable provider key is needed to enable it, not to
+  save it, so no `ok` key means keep it disabled — not stop. `provider_key_id` may be `null` to run on the
+  team's active key.
 - **`bytecode` is server-written** for hog evals — never pass it; send only `evaluation_config.source`.
 - For cluster-scoped evals, identify the cluster with `exploring-llm-clusters`, then translate its event
   filter into `conditions`.

skills/creating-online-evaluations/references/evaluation-payload.md

@@ -152,11 +152,12 @@ Empty object: `{}`.
 | ----------------- | -------- | ------------------------------------------------------------------------------ |
 | `provider`        | yes      | One of `openai`, `anthropic`, `gemini`, `openrouter`, `fireworks`, `azure_openai`, `together_ai`. |
 | `model`           | yes      | Model id, e.g. `gpt-5-mini`. Validate against `llma-evaluation-judge-models`.  |
-| `provider_key_id` | no       | UUID of an `ok`-state key for the same provider. `null` uses the matching active team key. |
+| `provider_key_id` | no       | UUID of an `ok`-state key for the same provider, or `null` to use the active team key. |
 
-`provider` and `model` are required. Pin `provider_key_id` to run on one specific key. Leave it `null` only
-when `llma-evaluation-config-get` shows an `ok`-state active key for the same provider. A disabled draft may
-be saved without a usable key, but it cannot be tested or enabled until a key can be resolved.
+`provider` and `model` are required. Pin `provider_key_id` to run on one specific key, or set it to `null`
+to run on the team's active key — both save a valid draft. To enable an unpinned eval, the active key must
+be `ok` and use the same provider (`llma-evaluation-config-get` shows it). A disabled draft may be saved
+without a usable key, but it cannot be tested or enabled until a key can be resolved.
 
 ## `conditions`
 
@@ -250,13 +251,14 @@ and `item.output` when exact structure matters.
   "evaluation_config": { "prompt": "Return true if the assistant's reply stays on the user's topic, false if it changes subject. Return N/A if the user did not ask a question." },
   "output_type": "boolean",
   "output_config": { "allows_na": true },
-  "model_configuration": { "provider": "openai", "model": "gpt-5-mini", "provider_key_id": "<ok-state key uuid from llma-provider-key-list>" },
+  "model_configuration": { "provider": "openai", "model": "gpt-5-mini", "provider_key_id": null },
   "target": "generation",
   "target_config": {},
   "conditions": [{ "id": "default", "rollout_percentage": 100, "properties": [] }],
   "enabled": false
 }
 ```
 
-Set `provider_key_id` to `null` only when the team's active key is in the `ok` state and its provider is
-`openai`.
+`provider_key_id: null` runs on the team's active key and always saves a valid draft. Pin it to an
+`ok`-state key uuid from `llma-provider-key-list` to run on one specific key. To enable an unpinned eval,
+the team's active key must be `ok` and its provider `openai`.