Skip to content

Commit 461862f

Browse files
alexkromanclaude
andauthored
Add anonymous usage telemetry collection (#66)
* Add anonymous, opt-out usage telemetry (Datadog logs intake) Model after the Supabase CLI: ship a write-only Datadog client token (pub…, embeddable by design — deliberately committed and allowlisted in gitleaks; never an API key) and send one allow-listed event per command run to the Datadog logs intake. - aai_cli/telemetry.py: consent (AAI_TELEMETRY_DISABLED / DO_NOT_TRACK / persisted choice), event build (command path, outcome class, exit code, duration, version/OS/CI — never args, paths, or account data), and a detached flusher subprocess so commands never wait on telemetry; all send-side failures are swallowed. - context.run_command wraps each command body in telemetry.track(), so CLIError outcomes keep their machine-readable error_type. - config.py: random anonymous device_id + telemetry_enabled persisted in config.toml. - aai telemetry status/enable/disable: the user-facing consent surface, documented in README ("Telemetry") with the full opt-out story. - Test suite blanks the shipped token via an autouse fixture so no test spawns a real flusher; telemetry paths are covered by dedicated unit, command, and run_command-integration tests. https://claude.ai/code/session_017kPHAMnWAUs6XHVGk4FyWE * Fix order-dependent flake in yt-dlp section tests _fake_ytdlp replaces sys.modules["yt_dlp"] with a non-package namespace, so the --download-sections path's `from yt_dlp.utils import …` only resolved when an earlier test had already imported the real module — passing or failing with pytest-randomly's seed. Cache the real yt_dlp.utils first so the import is deterministic. https://claude.ai/code/session_017kPHAMnWAUs6XHVGk4FyWE * Make the telemetry flusher an explicit `aai telemetry flush` subcommand Address the Aikido review findings on the dispatch path: spawning a dynamically-constructed `python -c` snippet is needlessly opaque. The detached flusher is now the CLI's own hidden `telemetry flush` command (the same shape the Vercel CLI uses) — an explicit, reviewable entry point. The child env sets AAI_TELEMETRY_DISABLED=1 so a flush can never spawn another flusher, and a test pins that `flush` stays hidden from help. https://claude.ai/code/session_017kPHAMnWAUs6XHVGk4FyWE --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent b8861ca commit 461862f

15 files changed

Lines changed: 921 additions & 3 deletions

.gitleaks.toml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,11 @@ regexTarget = "match"
1515
regexes = [
1616
'''sk_abcdef1234''',
1717
'''sk_zzzzzz9999''',
18+
# The shipped telemetry credential (aai_cli/telemetry.py SHIPPED_CLIENT_TOKEN) is a
19+
# Datadog *client token*: write-only and designed to be embedded in client apps, so
20+
# committing it is deliberate (the Supabase-CLI model). Only this exact value is
21+
# allowlisted -- any other real-looking token still fails the gate.
22+
'''pub0d633113b9f7d22faff215fefaf30b43''',
1823
]
1924
# `gitleaks dir` scans the working tree regardless of .gitignore, so high-entropy values
2025
# in a developer's gitignored `.claude/settings.local.json` (a personal Claude Code file

.importlinter

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ source_modules =
2626
aai_cli.render
2727
aai_cli.stdio
2828
aai_cli.streaming
29+
aai_cli.telemetry
2930
aai_cli.theme
3031
aai_cli.transcribe_render
3132
aai_cli.ws
@@ -51,6 +52,7 @@ modules =
5152
aai_cli.commands.setup
5253
aai_cli.commands.share
5354
aai_cli.commands.stream
55+
aai_cli.commands.telemetry
5456
aai_cli.commands.transcribe
5557
aai_cli.commands.transcripts
5658

@@ -64,5 +66,6 @@ source_modules =
6466
aai_cli.environments
6567
aai_cli.errors
6668
aai_cli.llm
69+
aai_cli.telemetry
6770
forbidden_modules =
6871
rich

AGENTS.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ A Typer CLI. `aai_cli/main.py` builds the `app`, registers each command sub-app,
153153

154154
### Command layer
155155

156-
Each file in `aai_cli/commands/` is a Typer sub-app (`transcribe`, `stream`, `agent`, `speak`, `llm`, `transcripts`, `login` (login/logout/whoami), `doctor`, `init`, `dev`, `share`, `deploy`, `setup`, `onboard`, `account` (balance/usage/limits), `keys`, `sessions`, `audit`). Command bodies run through `context.run_command(ctx, fn, json=...)`, which maps any `CLIError` to clean stderr output + the error's exit code. Commands never print tracebacks for expected failures.
156+
Each file in `aai_cli/commands/` is a Typer sub-app (`transcribe`, `stream`, `agent`, `speak`, `llm`, `transcripts`, `login` (login/logout/whoami), `doctor`, `init`, `dev`, `share`, `deploy`, `setup`, `onboard`, `account` (balance/usage/limits), `keys`, `sessions`, `audit`, `telemetry` (status/enable/disable)). Command bodies run through `context.run_command(ctx, fn, json=...)`, which maps any `CLIError` to clean stderr output + the error's exit code. Commands never print tracebacks for expected failures.
157157

158158
### Cross-cutting state (resolution order matters)
159159

@@ -171,6 +171,7 @@ Each file in `aai_cli/commands/` is a Typer sub-app (`transcribe`, `stream`, `ag
171171
- **`code_gen/`** — backs `--show-code` on `transcribe`/`stream`/`agent`: builds a ready-to-run Python SDK script from exactly the flags passed (no API key needed; generated code reads `ASSEMBLYAI_API_KEY`).
172172
- **`auth/`** — browser-assisted `assembly login` via AMS + **Stytch B2B OAuth discovery** (`discovery.py`, `flow.py`, `loopback.py`, `ams.py`). Not Stytch Connected Apps.
173173
- **`init/`** — scaffolds a self-contained FastAPI + HTML starter (`audio-transcription`/`live-captions`/`voice-agent` templates), optionally installs deps and opens the browser; writes the key to a git-ignored `.env`.
174+
- **`telemetry.py`** — anonymous, opt-out usage telemetry (Supabase-CLI model): `context.run_command` wraps each command body in `telemetry.track(ctx.command_path)`, which dispatches one allow-listed event (command path, outcome/exit code, duration, version/OS — never args, paths, or account data) to the Datadog logs intake via a **detached flusher subprocess** (the hidden `assembly telemetry flush`), so commands never wait on telemetry. `SHIPPED_CLIENT_TOKEN` is a committed write-only Datadog *client* token (`pub…`, embeddable by design — never an API key; `AAI_TELEMETRY_CLIENT_TOKEN` overrides). The test suite blanks it via an autouse conftest fixture so no test ever spawns a real flusher. Opt-out: `AAI_TELEMETRY_DISABLED=1` / `DO_NOT_TRACK=1` / `assembly telemetry disable` (persisted as `telemetry_enabled` in config.toml, alongside the random `device_id`). Send-side failures are swallowed (`OSError`/`CLIError`) — telemetry must never break a command.
174175
- **`commands/setup.py`**`assembly setup install/status/remove` wires a coding agent up to AssemblyAI by installing three artifacts: the `assemblyai-docs` docs MCP (via `claude mcp add`), the AssemblyAI skill (via `npx skills add`), and the bundled `aai-cli` skill (copied out of the wheel, no network). Missing `claude`/`npx` is reported and skipped, not an error.
175176

176177
## Conventions

README.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -248,6 +248,19 @@ op run -- assembly transcribe call.mp3 # …or wrap the whole
248248

249249
In CI, set `ASSEMBLYAI_API_KEY` as a masked secret. `assembly logout` purges the keyring entry; `assembly whoami` / `assembly doctor` confirm the active source without printing the key.
250250

251+
## Telemetry
252+
253+
`assembly` collects **anonymous** usage telemetry to help improve the CLI: the command name (never its arguments), outcome class and exit code, duration, CLI version, OS, Python version, whether it ran in CI, and a random install id. It never collects arguments, file paths or contents, transcripts, API keys, or account data — and delivery runs in a detached background process, so it never slows a command down.
254+
255+
Opt out any time, persistently or per-environment:
256+
257+
```sh
258+
assembly telemetry disable # persisted on this machine (assembly telemetry status to inspect)
259+
export AAI_TELEMETRY_DISABLED=1 # env kill-switch; the cross-tool DO_NOT_TRACK=1 also works
260+
```
261+
262+
The ingestion credential in the source is a Datadog **client token** — the write-only, embeddable credential class (it can submit events, read nothing). No account secret ships with the CLI.
263+
251264
## Account Self-Service
252265

253266
These commands use your browser login session (run `assembly login`), not your API key:

aai_cli/commands/telemetry.py

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
"""`assembly telemetry` — inspect or change anonymous usage telemetry.
2+
3+
The collection itself lives in ``aai_cli/telemetry.py``; this command is the
4+
user-facing consent surface: see what would be sent and turn it off (or back
5+
on) persistently. The env kill-switches (``AAI_TELEMETRY_DISABLED=1``,
6+
``DO_NOT_TRACK=1``) always win over the persisted choice.
7+
"""
8+
9+
from __future__ import annotations
10+
11+
import typer
12+
13+
from aai_cli import config, options, output, telemetry
14+
from aai_cli.context import AppState, run_command
15+
from aai_cli.help_text import examples_epilog
16+
17+
app = typer.Typer(help="Anonymous usage telemetry: status, enable, disable.")
18+
19+
20+
def _consent_label() -> str:
21+
return "granted" if telemetry.consent_granted() else "denied"
22+
23+
24+
@app.command(
25+
epilog=examples_epilog(
26+
[
27+
("Show whether telemetry is active", "assembly telemetry status"),
28+
("As JSON for scripting", "assembly telemetry status --json"),
29+
]
30+
)
31+
)
32+
def status(
33+
ctx: typer.Context,
34+
json_out: bool = options.json_option(),
35+
) -> None:
36+
"""Show whether anonymous usage telemetry is active, and why."""
37+
38+
def body(_state: AppState, json_mode: bool) -> None:
39+
data: dict[str, object] = {
40+
"enabled": telemetry.is_enabled(),
41+
"consent": _consent_label(),
42+
"token_configured": bool(telemetry.client_token()),
43+
}
44+
45+
def render(d: dict[str, object]) -> object:
46+
state_line = (
47+
output.success("Telemetry is enabled.")
48+
if d["enabled"]
49+
else output.muted("Telemetry is disabled.")
50+
)
51+
detail = output.muted(
52+
f"Consent: {d['consent']}. Intake token configured: "
53+
f"{'yes' if d['token_configured'] else 'no'}."
54+
)
55+
hint = output.hint(
56+
"Opt out any time: 'assembly telemetry disable' or AAI_TELEMETRY_DISABLED=1."
57+
)
58+
return output.stack(state_line, detail, hint)
59+
60+
output.emit(data, render, json_mode=json_mode)
61+
62+
run_command(ctx, body, json=json_out)
63+
64+
65+
@app.command(
66+
epilog=examples_epilog([("Re-enable telemetry", "assembly telemetry enable")]),
67+
)
68+
def enable(
69+
ctx: typer.Context,
70+
json_out: bool = options.json_option(),
71+
) -> None:
72+
"""Re-enable anonymous usage telemetry for this machine."""
73+
74+
def body(_state: AppState, json_mode: bool) -> None:
75+
config.set_telemetry_enabled(enabled=True)
76+
output.emit(
77+
{"telemetry_enabled": True},
78+
lambda _d: output.success("Telemetry enabled."),
79+
json_mode=json_mode,
80+
)
81+
82+
run_command(ctx, body, json=json_out)
83+
84+
85+
@app.command(
86+
hidden=True,
87+
epilog=examples_epilog(
88+
[
89+
(
90+
"Internal plumbing, spawned by the CLI itself",
91+
"assembly telemetry flush '<payload-json>'",
92+
)
93+
]
94+
),
95+
)
96+
def flush(
97+
payload: str = typer.Argument(..., help="Serialized telemetry payload (internal)."),
98+
) -> None:
99+
"""Deliver one serialized telemetry event to the intake (internal).
100+
101+
This is the detached flusher `telemetry.dispatch` spawns so user commands never
102+
wait on the network — an explicit, reviewable entry point rather than inline
103+
code. Hidden from help; runs with stdio discarded, so it neither needs nor
104+
produces output.
105+
"""
106+
telemetry.flush_payload(payload)
107+
108+
109+
@app.command(
110+
epilog=examples_epilog([("Opt out of telemetry", "assembly telemetry disable")]),
111+
)
112+
def disable(
113+
ctx: typer.Context,
114+
json_out: bool = options.json_option(),
115+
) -> None:
116+
"""Opt out of anonymous usage telemetry for this machine."""
117+
118+
def body(_state: AppState, json_mode: bool) -> None:
119+
config.set_telemetry_enabled(enabled=False)
120+
output.emit(
121+
{"telemetry_enabled": False},
122+
lambda _d: output.success("Telemetry disabled."),
123+
json_mode=json_mode,
124+
)
125+
126+
run_command(ctx, body, json=json_out)

aai_cli/config.py

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
import re
66
import tempfile
77
import tomllib
8+
import uuid
89
from pathlib import Path
910

1011
import keyring
@@ -44,6 +45,11 @@ class Config(BaseModel):
4445

4546
active_profile: str | None = None
4647
profiles: dict[str, Profile] = Field(default_factory=dict)
48+
# Telemetry state (see telemetry.py): a random anonymous install id, and the
49+
# persisted opt-out. None means "never chosen", which the opt-out model reads
50+
# as enabled — distinct from an explicit False written by `assembly telemetry disable`.
51+
device_id: str | None = None
52+
telemetry_enabled: bool | None = None
4753

4854

4955
class StoredSession(BaseModel):
@@ -348,6 +354,29 @@ def persist_login(
348354
_dump(prior_cfg)
349355

350356

357+
def get_device_id() -> str:
358+
"""A stable anonymous install id for telemetry: a random UUID minted locally on
359+
first use and persisted in config.toml. Carries nothing derivable from the
360+
machine or account."""
361+
cfg = _load()
362+
if cfg.device_id is None:
363+
cfg.device_id = str(uuid.uuid4())
364+
_dump(cfg)
365+
return cfg.device_id
366+
367+
368+
def get_telemetry_enabled() -> bool | None:
369+
"""The persisted telemetry choice: True/False if the user ran
370+
`aai telemetry enable/disable`, None if they never chose."""
371+
return _load().telemetry_enabled
372+
373+
374+
def set_telemetry_enabled(*, enabled: bool) -> None:
375+
cfg = _load()
376+
cfg.telemetry_enabled = enabled
377+
_dump(cfg)
378+
379+
351380
def resolve_api_key(*, profile: str | None = None, api_key_flag: str | None = None) -> str:
352381
if api_key_flag is not None:
353382
if not api_key_flag:

aai_cli/context.py

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
import keyring.errors
1010
import typer
1111

12-
from aai_cli import config, environments, output
12+
from aai_cli import config, environments, output, telemetry
1313
from aai_cli.auth import run_login_flow
1414
from aai_cli.environments import Environment
1515
from aai_cli.errors import APIError, CLIError, NotAuthenticated
@@ -188,7 +188,10 @@ def run_command(
188188
state: AppState = ctx.obj
189189
json_mode = output.resolve_json(explicit=json)
190190
try:
191-
fn(state, json_mode)
191+
# Inside the try so telemetry sees the raw CLIError (and its error_type)
192+
# before it's folded into a typer.Exit below.
193+
with telemetry.track(ctx.command_path):
194+
fn(state, json_mode)
192195
except NotAuthenticated as err:
193196
if not auto_login or not _should_auto_login(err):
194197
output.emit_error(err, json_mode=json_mode)

aai_cli/main.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@
3636
share,
3737
speak,
3838
stream,
39+
telemetry,
3940
transcribe,
4041
transcripts,
4142
)
@@ -66,6 +67,7 @@
6667
# Setup & Tools — get set up & maintain
6768
"doctor",
6869
"setup",
70+
"telemetry",
6971
# History — browse past work
7072
"transcripts",
7173
"sessions",
@@ -326,6 +328,7 @@ def main(
326328
app.add_typer(deploy.app)
327329
app.add_typer(onboard.app)
328330
app.add_typer(setup.app, name="setup", rich_help_panel=help_panels.SETUP)
331+
app.add_typer(telemetry.app, name="telemetry", rich_help_panel=help_panels.SETUP)
329332
app.add_typer(keys.app, name="keys", rich_help_panel=help_panels.ACCOUNT)
330333

331334

0 commit comments

Comments
 (0)