Skip to content

Improve analytics identity and failure diagnostics - #1733

Draft
r3dbars wants to merge 6 commits into
mainfrom
codex/analytics-observability
Draft

Improve analytics identity and failure diagnostics#1733
r3dbars wants to merge 6 commits into
mainfrom
codex/analytics-observability

Conversation

@r3dbars

@r3dbars r3dbars commented Sep 10, 2026

Copy link
Copy Markdown
Owner

Why

Sentry reported failures with zero impacted users, capture degradation obscured hard errors, and PostHog health snapshots frequently lacked outcome metadata. This change makes usage and failures traceable across both vendors and the existing support-diagnostics flow.

Product Impact

  • Affects: dictation and meeting analytics, failure reporting, and support diagnostics.
  • Lane: activation, dictation reliability, meeting reliability.
  • Analytics collection uses the existing Usage stats preference. Settings retains its existing reporting controls and gains clearer disclosure in the existing info popover.

What changed

  1. Preserve the existing anonymous analytics UUID; use it for Sentry user.id and PostHog distinct_id, with opted-in install traits.
  2. Keep completed-but-degraded capture in health analytics and local warnings. Hard failures retain Sentry visibility.
  3. Add consistent failure codes, stages, operation correlation, release, coarse device classes, permission state, and trigger. Preserve meeting identity through stopping and background transcription, and mirror canonical reliability failures across both vendors.
  4. Expand existing Copy diagnostics with install identity and recent failure taxonomy. Exclude raw logs and reliability-packet text from support copies.
  5. Add bucket-only daily digests with stable event UUIDs, calendar-aware rollover, partial-quit disclosure, bounded priority retries, date-only dedup receipts, and opt-out purging.

The local metadata ledger supports daily digests and support diagnostics. This PR does not add a customer-facing usage or health dashboard.

The QA bench also references the checked-in 1.1.59 release fixture instead of the removed 1.1.58 fixture.

How I checked it

  • Latest panel-removal change: bash build.sh --no-open; app build and performance budget passed, with launch to interactive at 882.0 ms against a 3000 ms budget.
  • Latest panel-removal change: bash run-tests.sh: 13,509 passed, zero failures.
  • Latest panel-removal change: independent review PASS for e1312cf..9783455; no dangling panel or preview references.
  • Build-source validation, analytics taxonomy normalization, and diff checks.
  • Full analytics candidate at e1312cf, before removing the panel: full QA passed 15/15 checks; dependency rebuild, deterministic E2E, integration smoke, synthetic pasteback/audio, and strict security/release-health checks passed. Security score 100/100. Core suite: 1,092 tests, 13 opt-in skips, zero failures. Current-artifact validation used warnings-only mode.
  • Full analytics diff independently reviewed against origin/main: PASS after seven findings were fixed, with no rejected findings.

Regression coverage includes identity migration, encoded person traits, shared taxonomy, content exclusion, opt-out with preference observation, unexpected no-audio failure deduplication, local midnight/DST, same-day re-opt-in, offline count/byte pressure, and UUID/timestamp preservation across HTTP retries.

Privacy and limits

  • No transcript/dictation/meeting content, audio bytes, titles, speaker names, emails, content paths, or raw logs are added to telemetry.
  • Outgoing usage totals and durations are bucketed. Analytics-off blocks events/person traits/digests and clears the ledger and pending delivery file. At most 14 date-only receipts prevent duplicate same-day digests. Crash reporting retains its separate preference.
  • GeoIP enrichment is disabled on captures. Server-side IP retention remains a separate PostHog project-setting verification.
  • A normal quit can send a partial day once. Later same-day activity remains in lifecycle events and the local ledger. Retries are bounded at 14 days for digests and 24 hours for ordinary events.
  • The first observed day on an upgraded install is not its historical installation date.
  • Vendor inspection was read-only. Local checks do not establish live candidate ingestion or physical audio behavior. Follow the candidate verification checklist to match a controlled failure in both vendors and inspect new-build health fields.

Risk Review

  • Privacy, bounded storage, opt-out/reentrancy, diagnostic copy, and telemetry migration reviewed.
  • No release/version/appcast change; draft for human review. No merge or deployment performed.
  • Metadata contract and verification workpad.

Agent handoff

COORD_DONE: GREEN | https://github.com/r3dbars/transcripted/pull/1733 | analytics and support diagnostics; no customer usage panel | cleanup: none | human review and candidate ingestion verification remain | local build/tests and independent review passed | lanes used: Codex=implementation plus independent read-only reviewer; Claude/Local/Windows=skipped for bounded scope | review draft PR; do not merge automatically

@r3dbars r3dbars changed the title Improve analytics identity, failure taxonomy, and usage visibility Improve analytics identity and failure diagnostics Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant