Skip to content

Add opt-in anonymous PostHog telemetry - #705

Open
time-attack wants to merge 1 commit into
yc-software:mainfrom
time-attack:posthog-telemetry-event-plan
Open

Add opt-in anonymous PostHog telemetry#705
time-attack wants to merge 1 commit into
yc-software:mainfrom
time-attack:posthog-telemetry-event-plan

Conversation

@time-attack

@time-attack time-attack commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

What

Opt-in, anonymous product telemetry via PostHog, so we can learn which features matter without collecting anything sensitive. Off by default; a no-op unless the deployment sets POSTHOG_API_KEY and an org admin turns on the new Usage telemetry toggle.

Admin toggle

Privacy design

  • Events carry only enums, counts, durations, and the build SHA, tied to a random per-deployment UUID stored in Postgres (telemetry_instance). No message content, file/skill names, user identifiers, channel/session ids, error messages, or org name — verified against raw stored PostHog payloads, not just by reading the code.
  • Consent is durable and re-checked on every flush, so opting out reaches every process (API + workers) within ~30s and drops anything still buffered. TELEMETRY_DISABLED=true is a hard kill switch that can only disable, never enable — the admin toggle is the single way on.
  • Events set $geoip_disable and $process_person_profile: false; docs note that the receiving project must enable PostHog's "Discard client IP data" (sender can't force it).
  • docs/telemetry.md lists every event and property; only explicit allowlists emit.

How

No new dependency — a ~260-line module (src/insights/telemetry.ts) POSTs to PostHog's batch API with fetch, flushing via createSweeper every 30s and (bounded to 2s) on shutdown. Transient failures requeue with exponential backoff; permanent 4xx drops the batch. Most events come from existing sinks rather than new call sites:

  • decorator over AuditLog.record maps an allowlisted subset of audit actions (skills, crons, webhooks, deploys, connectors, keychain, files, memory, projects, grants) plus an explicit admin-resource list → config_updated
  • wrappers over MetricsSink/ErrorLogturn_completed / memory_auto_captured / error_recorded
  • the harness router resolver → harness_invoked (harness/model adoption)
  • one shared trigger runner in wiring → trigger_fired for cron/webhook/monitor/keychain-ask/secret-drop
  • daily instance_heartbeat deduped through the durable instance row so multiple processes and redeploys don't inflate it

The org toggle follows the existing org-flag pattern in config-store (incl. a durable getter, mirroring getInteractiveFastModeDurable) and the standard ADMIN_RESOURCES boolean + admin-panel card. Also deletes a stale duplicate org-key list in flushScope found during review.

Testing

  • test/telemetry.test.ts (11 tests): payload shape + privacy flags, no-PII assertion against a poisoned audit event, opt-out drops buffered events, durable confirm overrides a stale cache, backoff + permanent-drop, durable daily heartbeat, bounded shutdown flush.
  • Typecheck, eslint, oxlint, prettier clean; admin/config-store/wiring suites pass.
  • Verified end-to-end on a live dev instance against PostHog Cloud: real chat turns, cron creation, the admin toggle flow (off by default → enable → events flow; the enabling config_updated itself is correctly dropped because consent wasn't yet active), and raw stored payload inspection.

Review: 8 independent fresh-context review passes (correctness, reuse, conventions, efficiency, simplification, altitude, removed-behavior, cross-file tracing); all confirmed findings addressed in this diff.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Telemetry is a no-op without POSTHOG_API_KEY, and the org-wide admin
toggle (default off) is the only way to turn it on; TELEMETRY_DISABLED is
a hard kill switch that can only disable. The toggle is re-checked
durably on every flush, so opting out reaches every process within one
flush cycle and drops anything still buffered. Events carry only enums,
counts, durations, and the build SHA, tied to a random per-deployment
UUID stored in Postgres; no content, names, or user identifiers ever
leave the instance. Events request no GeoIP enrichment and no person
profiles; docs/telemetry.md lists every event and property, and only
explicit allowlists emit.

Most events derive from existing sinks: a decorator over the audit log
maps allowlisted actions, wrappers over the metrics sink and error log
cover turn outcomes, the harness resolver reports harness/model usage,
and one shared trigger runner covers cron, webhook, monitor, and
keychain fires. A sweeper flushes batches every 30s and (bounded to 2s)
on shutdown; transient failures requeue with exponential backoff,
permanent rejections drop the batch, and the daily heartbeat dedupes
through the durable instance row so deploys and multiple processes don't
inflate it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant