diff --git a/README.md b/README.md index ebab56b..ca4873d 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,19 @@ Or with Task: task check ``` +## Logs and troubleshooting + +Capture includes timestamped progress, grouped warnings, and actionable errors. +Use `--verbose` for individual asset details or save sanitized diagnostics: + +```bash +uv run mindcap --debug --log-file "$HOME/Downloads/mindcap-debug.jsonl" \ + capture suno "" +``` + +See [terminal output and diagnostics](docs/diagnostics.md) for quiet/JSON output, +exit codes, and the diagnostic log format. + ## Protect the Artifact Cache Mindcap defaults to the Ego Hygiene repository root at: @@ -127,6 +140,20 @@ Capture a workspace archive using a placeholder workspace UUID: uv run mindcap capture suno "00000000-0000-0000-0000-000000000000" ``` +If audio downloads return HTTP 403, see the +[Suno audio troubleshooting guide](docs/suno-audio.md). Metadata authentication +and permission to download media are separate checks. The capture summary +reports downloaded/expected primary audio and marks advertised asset failures +as partial; archive integrity PASS checks the files that were actually saved. + +Inspect an existing capture without contacting Suno or printing its URLs: + +```bash +uv run mindcap inspect suno \ + ".cache/mindcap/workspaces/suno/suno-00000000-0000-0000-0000-000000000000/v1" \ + --asset-diagnostics +``` + ## Vault Ingestion and Verification Mindcap can ingest finalized Suno workspace bundles into an immutable vault made of sealed SQLite catalog generations and ZIP64 pack files. Example: diff --git a/docs/cli-architecture.md b/docs/cli-architecture.md index e3091a9..e925d59 100644 --- a/docs/cli-architecture.md +++ b/docs/cli-architecture.md @@ -70,19 +70,24 @@ envelope = strategy.capture(request) ``` mindcap_cli/ -└── app.py Typer application and all command registrations +├── app.py Typer application and command registrations +├── runtime.py Invocation boundaries and diagnostic option resolution +├── logging.py Safe terminal errors and optional JSONL diagnostics +└── _compat.py Installed Typer command-engine compatibility ``` -The current phase places all command implementations in `app.py`. A future -phase may split them into `commands/` submodules. +`DiagnosticTyper` assigns the same invocation boundary to every registered +command. Command implementations remain in `app.py`. --- ## Progress rendering The CLI constructs a `CaptureProgressReporter` backed by a Rich `Console`. -Provider strategies accept a reporter so they can emit phase labels and -progress bars during capture. +Provider strategies accept a reporter so they can emit timestamped phase +labels, progress bars, warning groups, and attempt counts during capture. +Diagnostic output goes to stderr; the final summary has a separate stdout +console. Redirected output uses bounded plain start/end events. Progress rendering is a CLI concern. The library emits structured `MindcapEvent` objects through the `EventSink` protocol instead. @@ -95,14 +100,14 @@ Progress rendering is a CLI concern. The library emits structured |------------------------------|-----------| | Success | 0 | | `MindcapError`, `OSError` | 1 | -| Partial batch failure | 2 | +| Partial capture/batch or usage error | 2 | | Interrupted (SIGINT) | 130 | --- ## JSON output -Pass `--json` to any command to receive machine-readable output on stdout: +Commands with `--json` support emit machine-readable output on stdout: ```bash mindcap capture chatgpt "..." --json @@ -116,7 +121,9 @@ result = application.capture(...) console.print_json(data={"status": result.status, ...}) ``` -Progress is suppressed or redirected to stderr when `--json` is active. +Progress stays on stderr, including export imports, interrupted sync runs, and +their resume instructions. Operational failures emit a sanitized error envelope +on stderr when the command's JSON option is active. --- @@ -124,7 +131,15 @@ Progress is suppressed or redirected to stderr when `--json` is active. - **stdout**: final result (path, JSON, or human-readable summary) - **stderr**: progress, warnings, errors (via Rich `Console`) -- When `--quiet` is active, only the final path or JSON is printed +- When `--quiet` is active on capture/sync, stdout is the final path or JSON; + warnings and errors remain visible on stderr + +Global `--verbose`, `--debug`, `--quiet`, and `--log-file` options apply before +the command name. Capture/sync retain their existing local verbosity options. +Handlers configure only the `mindcap` namespace, restore its previous state +after invocation, and never enable HTTP request logging. Debug errors include +sanitized frame locations without source lines or locals. See +[diagnostics](diagnostics.md) for the log format and redaction limits. --- diff --git a/docs/diagnostics.md b/docs/diagnostics.md new file mode 100644 index 0000000..92090d7 --- /dev/null +++ b/docs/diagnostics.md @@ -0,0 +1,91 @@ +# Terminal output and diagnostics + +Mindcap displays timestamped, colored severity labels on an interactive +terminal. Phase progress, warnings, retries, and errors go to **stderr**; the +final result or JSON goes to **stdout**. Redirected output uses plain text with +bounded phase start/end lines instead of animated progress. Rich respects +`NO_COLOR=1` for terminals that should not use color. + +## Everyday capture + +The normal command already includes progress and warning summaries: + +```bash +uv run mindcap capture suno "" +``` + +Repeated asset failures are collapsed into a representative warning and a +grouped count. `--verbose` retains individual asset warnings and any per-item +detail supplied by the adapter. A finished progress phase counts **attempts**; +the final capture summary reports assets actually downloaded, capture +completeness, and saved-file integrity separately. + +```bash +uv run mindcap capture suno "" --verbose +uv run mindcap capture suno "" --quiet +uv run mindcap capture suno "" --json +``` + +Quiet mode retains warnings and errors. Capture prints only its archive path; +`--json` takes precedence for the final result, while quiet still suppresses +progress. Source adapters determine which phase and per-item details exist. + +## Investigate a failure + +Errors show a stable code, a sanitized description, and a next step. Debug mode +adds diagnostic events and exception frame locations, without dumping source +lines or local variables. Authentication, browser locks, invalid input, archive +verification, and filesystem failures have distinct codes. + +Global diagnostic options go **before** the command name. Existing capture and +sync `--verbose`, `--debug`, and `--quiet` options still work after the command. + +```bash +uv run mindcap --debug capture suno "" +uv run mindcap --verbose doctor suno +uv run mindcap --debug --log-file "$HOME/Downloads/mindcap-debug.jsonl" \ + capture suno "" +``` + +The optional log contains newline-delimited JSON records with UTC timestamps, +severity, logger, and sanitized messages. It appends to a private file with +owner-only permissions, in an existing folder. Log setup errors stop the +command before capture begins; a write failure during capture emits a warning +and lets capture continue. No file is created unless `--log-file` is supplied. +Debug and quiet control the file's verbosity too; quiet takes precedence. + +Diagnostics strip common credential fields, cookies, authorization headers, +JWTs, URL credentials, URL queries/fragments, and terminal control sequences. +Request/response bodies and browser storage state are never diagnostic input. +Logs can still contain source identifiers, titles, hosts, and local paths; +inspect a log before sharing it. + +## Results and exit codes + +| Exit | Meaning | +|---|---| +| `0` | Command succeeded. | +| `1` | Operation failed, including an unexpected error or unavailable log file. | +| `2` | Partial capture/batch result, or command usage error. | +| `130` | Interrupted. Sync prints its checkpoint/resume instructions. | + +A partial single-source capture still prints its saved archive or JSON result +before exiting `2`. Archive integrity can pass while required assets are +missing. Inspect the warnings before retrying; a media `403` alone does not +establish that a login cookie is missing. See [Suno audio](suno-audio.md). + +Commands that support `--json` emit their success result on stdout. A failure +emits a structured error envelope on stderr, with `code`, `type`, `message`, +and `hint`; debug mode can add sanitized frame locations. Stderr may also +contain preceding phase diagnostics, so consume stdout for result data and use +the exit status to determine success. Argument conversion and missing-argument +errors also honor a recognized JSON option; unrecognized-option errors may +use a text diagnostic. + +## Library callers + +Importing Mindcap does not configure the root logger or HTTP client loggers. +CLI handlers are installed only for one command and removed on exit. Embedding +applications can configure their own `mindcap` logging namespace. The progress +reporter remains silent when no console is attached; structured application +events remain a separate library contract. diff --git a/docs/suno-audio.md b/docs/suno-audio.md new file mode 100644 index 0000000..0d673d3 --- /dev/null +++ b/docs/suno-audio.md @@ -0,0 +1,88 @@ +# Troubleshoot Suno audio downloads + +If project metadata, lyrics and artwork download but audio returns HTTP 403, +the metadata login worked. The media endpoint denied its own requests. This +does not establish that the stored Clerk cookie is missing or expired. + +## Start with the existing archive + +Run the offline diagnostic on the exact version directory printed by capture: + +```bash +uv run mindcap inspect suno \ + ".cache/mindcap/workspaces/suno/suno-00000000-0000-0000-0000-000000000000/v2" \ + --asset-diagnostics +``` + +Successful output is JSON containing counts and fixed labels. It excludes +titles, clip/project IDs, hosts, full URLs, query values and authentication +material. It verifies the saved bundle and makes no network requests. This +also works with captures made before the audio-download fix. + +| Field | Meaning | +| --- | --- | +| `clips_with_archived_primary_audio` | Clip records listing a saved primary audio asset. Integrity verification separately checks the saved files. | +| `audio_url_kinds.forbidden_endpoint` | Suno supplied an `/api/forbidden` URL instead of a downloadable file. Refreshing a cookie alone does not establish download eligibility. | +| `audio_url_kinds.suno_cdn` | Primary URLs point to recognized Suno media hosts. A 403 may involve download authorization, an expired link, or CDN restrictions. | +| `audio_url_kinds.suno_api` | Primary URLs point to a recognized Suno API host. | +| `audio_url_kinds.other_http` | Primary URLs point elsewhere; no host values are printed. | +| `audio_url_kinds.missing` / `invalid` | Audio URLs were absent or malformed in the saved clip metadata. | +| `audio_urls_with_signature_keys` | URLs contain recognized signature parameter names. Their values are never printed; this count does not establish validity or expiry. | + +You can share this successful diagnostic JSON when investigating a failure. +Keep the original metadata JSON, capture archive, cookies, browser network +exports and signed URLs private. + +## Check one song in Suno + +Open the same song in Suno and use its **More Actions → Download** menu. +If the site's own download is unavailable, inspect the account's download +eligibility or contact Suno support. Playback working does not establish that +file downloads are allowed. + +Suno announced download-policy changes effective September 3, 2026, including +plan-based download allowances. See the current +[official FAQ](https://help.suno.com/en/articles/13614785) and +[download instructions](https://help.suno.com/en/articles/2409921). Mindcap's +experimental adapter predates those changes. It currently consumes the media +URLs supplied in clip metadata; it does not implement a separate download +authorization/allowance flow. Do not infer account limits from a 403 alone. + +If the site's download succeeds, the diagnostic counts help distinguish an +outdated adapter URL from a failed media request. The client now sends +credential-free requests to external media hosts and preserves any supplied +signed query string. It keeps API authentication scoped to the configured API +origin and removes it when an asset redirect leaves that origin. This fixes +credential scoping; it does not prove that a particular live 403 is resolved. + +## Authentication and retry + +For metadata-login failures, run: + +```bash +uv run mindcap doctor suno +``` + +This checks stored authentication and authenticated API reachability, not media +download eligibility. Refresh the Clerk cookie through the existing +`mindcap auth suno --cookie-stdin` flow only when authentication needs renewal. +Never paste credentials into an issue or chat. + +After updating Mindcap, capture the workspace again. A new capture reads fresh +provider metadata and preserves earlier versions: + +```bash +uv run mindcap capture suno "00000000-0000-0000-0000-000000000000" +``` + +Progress bars count attempts. The final **Audio** row reports downloaded versus +advertised primary audio. Any failed advertised asset makes the capture +partial, even if every clip's metadata was saved. **Archive integrity PASS** +means the saved files passed verification; it does not mean all audio exists. +Do not use that result alone as evidence that the music is fully backed up. + +The primary audio format is whatever Suno supplies. The existing `--audio-format` +option changes its saved extension; it does not transcode audio or negotiate a +WAV export. `--request-wav` and `--include-stems` currently do not implement Suno +export requests. Use the site's supported export controls when those formats +are needed. diff --git a/src/mindcap/core/diagnostics.py b/src/mindcap/core/diagnostics.py new file mode 100644 index 0000000..58c9e8e --- /dev/null +++ b/src/mindcap/core/diagnostics.py @@ -0,0 +1,159 @@ +"""Small, source-agnostic helpers for safe human-readable diagnostics. + +Diagnostics are descriptions, never a serialization format for request bodies, +browser state, or provider payloads. This last line of defence removes common +credentials and terminal controls from descriptions before they are displayed +or written to a log. Callers must still avoid logging private payloads. +""" + +from __future__ import annotations + +import re +import unicodedata +from urllib.parse import urlsplit, urlunsplit + +_REDACTED = "[REDACTED]" +_ANSI = re.compile( + r"\x1b\][^\x07\x1b]*(?:\x07|\x1b\\|$)" + r"|\x1b(?:\[[0-?]*[ -/]*[@-~]|[@-_])" +) +_URL = re.compile(r"\b[a-z][a-z\d+.-]*://[^\s<>\"']+", re.IGNORECASE) +_SECRET_KEY = ( + r"(?:authorization|proxy[-_]authorization|auth|(?:set[-_])?cookies?|" + r"(?:x[-_])?api[-_]?key|(?:access|refresh|id)[-_]?token|" + r"(?:client[-_])?secret|password|passwd|session(?:[-_](?:id|token))?|" + r"__session|__client(?:_uat)?|__clerk_db_jwt|" + r"(?:c|x)srf(?:[-_]token)?|storage[-_]state|token)" +) +_SECRET_ASSIGNMENT = re.compile( + rf"(?[\"']?{_SECRET_KEY}[\"']?\s*[:=]\s*)" + r"(?P[\"'])(?:\\.|(?!\2).)*?\2", + re.IGNORECASE, +) +_HEADER_SECRET = re.compile( + rf"^(?P\s*{_SECRET_KEY}\s*:\s*)(?!\[REDACTED\])\S.*$", + re.IGNORECASE | re.MULTILINE, +) +_BARE_SECRET = re.compile( + rf"(?[\"']?{_SECRET_KEY}[\"']?\s*[:=]\s*)" + r"(?!\[REDACTED\])[^\s,;}\]\"']+", + re.IGNORECASE, +) +_INLINE_CREDENTIAL = re.compile( + r"\b(?PBearer|Basic)\s+(?!\[REDACTED\])[^\s,;\"'<>]+", + re.IGNORECASE, +) +_SECRET_OPTION = re.compile( + rf"(?P--{_SECRET_KEY}\s+)" + r"(?:\"(?:\\.|[^\"\\])*\"|'(?:\\.|[^'\\])*'|[^\s]+)", + re.IGNORECASE, +) +_INLINE_HEADER = re.compile( + r"(?P\b(?:authorization|proxy[-_]authorization|" + r"(?:set[-_])?cookies?)\s*[:=]\s*)" + r"(?![\"']|\[REDACTED\])[^\r\n]+", + re.IGNORECASE, +) +_JWT = re.compile( + r"(? str: + """Omit cookie/storage-state objects without serializing their members.""" + cursor = 0 + pieces: list[str] = [] + for match in _SECRET_ASSIGNMENT.finditer(text): + if match.end() < cursor or text[match.end() : match.end() + 1] not in "{[": + continue + if text[match.end() :].startswith(_REDACTED): + continue + start = match.end() + depth = 0 + quote = "" + escaped = False + end = len(text) + for index in range(start, len(text)): + character = text[index] + if escaped: + escaped = False + elif quote: + if character == "\\": + escaped = True + elif character == quote: + quote = "" + elif character in "\"'": + quote = character + elif character in "{[": + depth += 1 + elif character in "}]": + depth -= 1 + if depth == 0: + end = index + 1 + break + pieces.extend((text[cursor:start], _REDACTED)) + cursor = end + return "".join((*pieces, text[cursor:])) + + +def _redact_url(match: re.Match[str]) -> str: + value = match.group() + # Keep prose punctuation outside the URL. URL details are intentionally + # reduced: all query and fragment values may contain private/signed data. + suffix = "" + while value and value[-1] in ".,;)}]": + if value.endswith(_REDACTED): + break + suffix = value[-1] + suffix + value = value[:-1] + try: + parts = urlsplit(value) + host = parts.hostname + if not host: + return _REDACTED + suffix + authority = f"[{host}]" if ":" in host else host + if parts.port is not None: + authority += f":{parts.port}" + return ( + urlunsplit( + ( + parts.scheme, + authority, + parts.path, + _REDACTED if parts.query else "", + _REDACTED if parts.fragment else "", + ) + ) + + suffix + ) + except ValueError: + return _REDACTED + suffix + + +def redact_text(value: str) -> str: + """Return a literal, single-line diagnostic with common secrets removed. + + URL hosts, paths, and HTTP status descriptions remain useful, but URL + credentials, queries, fragments, cookies, tokens, and passwords do not. + ANSI escapes, control characters, and bidi controls cannot alter the + surrounding terminal output. This function is intentionally idempotent. + """ + text = _ANSI.sub("", value) + text = _redact_structured_secrets(text) + text = _QUOTED_SECRET.sub(lambda m: m["key"] + _REDACTED, text) + text = _HEADER_SECRET.sub(lambda m: m["key"] + _REDACTED, text) + text = _INLINE_CREDENTIAL.sub(lambda m: m["kind"] + " " + _REDACTED, text) + text = _SECRET_OPTION.sub(lambda m: m["key"] + _REDACTED, text) + text = _INLINE_HEADER.sub(lambda m: m["key"] + _REDACTED, text) + text = _URL.sub(_redact_url, text) + text = _BARE_SECRET.sub(lambda m: m["key"] + _REDACTED, text) + text = _JWT.sub(_REDACTED, text) + text = "".join( + " " if unicodedata.category(character).startswith("C") else character + for character in text + ) + return " ".join(text.split()) diff --git a/src/mindcap/core/progress.py b/src/mindcap/core/progress.py index c131553..dddf255 100644 --- a/src/mindcap/core/progress.py +++ b/src/mindcap/core/progress.py @@ -8,10 +8,14 @@ from __future__ import annotations +import logging +import re import time +from collections import Counter from collections.abc import Iterator from contextlib import contextmanager from dataclasses import dataclass, field +from datetime import datetime from rich.console import Console from rich.progress import ( @@ -25,6 +29,13 @@ TimeRemainingColumn, ) from rich.table import Table +from rich.text import Text + +from mindcap.core.diagnostics import redact_text + +_LOGGER = logging.getLogger("mindcap.progress") +_LOGGER.addHandler(logging.NullHandler()) +_UUID = re.compile(r"\b[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}\b", re.IGNORECASE) @dataclass @@ -34,6 +45,7 @@ class CaptureStats: clips_discovered: int = 0 clips_archived: int = 0 audio_downloaded: int = 0 + audio_expected: int | None = None artwork_downloaded: int = 0 videos_downloaded: int = 0 bytes_downloaded: int = 0 @@ -41,6 +53,7 @@ class CaptureStats: bytes_written: int = 0 elapsed_seconds: float = 0.0 verification_passed: bool = True + capture_complete: bool = True warnings: list[str] = field(default_factory=list) @property @@ -82,7 +95,10 @@ class CaptureProgressReporter: debug: Emit low-level diagnostic messages (never exposes secrets). quiet: - Suppress all non-error output, including the final summary. + Suppress informational output and the final summary, retaining warnings. + summary_console: + Optional destination for the final result table. Diagnostic events + always use *console*, allowing a CLI to keep stdout machine-readable. """ def __init__( @@ -92,89 +108,186 @@ def __init__( verbose: bool = False, debug: bool = False, quiet: bool = False, + summary_console: Console | None = None, ) -> None: self._console = console + self._summary_console = summary_console or console self.verbose = verbose self.debug = debug self.quiet = quiet self._start: float = time.monotonic() - - # ------------------------------------------------------------------ - # Phase / label helpers - # ------------------------------------------------------------------ + self._warnings: dict[str, int] = {} + self._observed_warnings: Counter[str] = Counter() + self._reported_groups: dict[str, int] = {} + self._warning_count = 0 + + def _emit(self, message: str, level: int = logging.INFO) -> None: + """Write a literal message once, with a matching event for file logging.""" + if self._console is None: + return + safe = redact_text(message) + _LOGGER.log(level, "%s", safe, extra={"progress_event": True}) + if self.quiet and level < logging.WARNING: + return + style = { + logging.DEBUG: "dim", + logging.INFO: "bright_cyan", + logging.WARNING: "bright_yellow", + logging.ERROR: "bold bright_red", + }.get(level, "bright_cyan") + label = logging.getLevelName(level) + line = Text() + line.append(datetime.now().strftime("%H:%M:%S"), style="dim") + line.append(f" {label:<7} ", style=style) + line.append(safe) + self._console.print(line, highlight=False) def phase(self, message: str) -> None: """Emit a top-level phase label.""" - if self._console and not self.quiet: - self._console.print(message) + self._emit(message) def detail(self, message: str) -> None: """Emit a verbose-only detail line.""" - if self._console and self.verbose and not self.quiet: - self._console.print(f" [dim]{message}[/dim]") + if self.verbose: + self._emit(message) def debug_line(self, message: str) -> None: - """Emit a debug-only line (must never contain secrets).""" - if self._console and self.debug and not self.quiet: - self._console.print(f" [dim yellow]{message}[/dim yellow]") + """Emit a debug-only line with credential redaction.""" + if self.debug: + self._emit(message, logging.DEBUG) def warn(self, message: str) -> None: - """Always emit a warning, even in quiet mode.""" - if self._console: - self._console.print(f"[bold yellow]Warning:[/bold yellow] {message}") + """Emit warnings, collapsing repeated asset failures in normal output. - # ------------------------------------------------------------------ - # Spinner context manager - # ------------------------------------------------------------------ + Every occurrence remains available to an attached file logger. The + summary reports grouped counts, while verbose output shows each item. + Identifiers are normalized generically, without provider assumptions. + """ + if self._console is None: + return + safe = redact_text(message) + key = _UUID.sub("", safe) + self._warnings[key] = self._warnings.get(key, 0) + 1 + self._observed_warnings[safe] += 1 + self._warning_count += 1 + if self.verbose or self._warnings[key] == 1: + self._emit(safe, logging.WARNING) + else: + _LOGGER.warning("%s", safe, extra={"progress_event": True}) - @contextmanager - def spinner(self, label: str) -> Iterator[None]: - """Display a spinner while the block executes. + def begin_capture(self) -> None: + """Start warning reconciliation for another capture on this reporter. + + Live warnings and returned envelope warnings deduplicate within one + capture. Aggregate groups and occurrence counts span the whole run. + """ + self._observed_warnings.clear() + + def report_warnings(self, messages: list[str]) -> None: + """Report an envelope's warnings without repeating provider events. - Falls back to a plain phase line when the console is not available or - is not a TTY (e.g., captured test output). + Providers may emit warnings as they occur or only return them at the + end. Reconciliation preserves occurrences and makes both paths behave + identically, including quiet and structured-output CLI invocations. """ - if self._console and not self.quiet and self._console.is_terminal: - with Progress( - SpinnerColumn(), - TextColumn("[progress.description]{task.description}"), - console=self._console, - transient=True, - ) as prog: - prog.add_task(label) + expected = Counter(redact_text(message) for message in messages) + for message, count in expected.items(): + missing = count - self._observed_warnings[message] + for _ in range(max(0, missing)): + self.warn(message) + if not self.verbose: + for warning, count in self._warnings.items(): + if count > 1 and self._reported_groups.get(warning) != count: + self._emit(f"{count}x {warning}", logging.WARNING) + self._reported_groups[warning] = count + + def _phase_finish( + self, + label: str, + started: float, + *, + interrupted: bool, + attempted: int | None = None, + total: int | None = None, + ) -> None: + elapsed = time.monotonic() - started + state = "interrupted" if interrupted else "finished" + count = "" + if attempted is not None: + count = f"; attempted {attempted} / {total}" + self._emit( + f"{label}: {state}{count}; elapsed {elapsed:.1f}s", + logging.ERROR if interrupted else logging.INFO, + ) + + @contextmanager + def spinner(self, label: str) -> Iterator[None]: + """Show transient TTY activity and bounded start/end diagnostic events.""" + started = time.monotonic() + self.phase(label) + try: + if self._console and not self.quiet and self._console.is_terminal: + with Progress( + SpinnerColumn(), + TextColumn("{task.description}", markup=False), + console=self._console, + transient=True, + ) as prog: + prog.add_task(redact_text(label)) + yield + else: yield + except BaseException: + self._phase_finish(label, started, interrupted=True) + raise else: - self.phase(label) - yield - - # ------------------------------------------------------------------ - # Progress-bar context manager - # ------------------------------------------------------------------ + self._phase_finish(label, started, interrupted=False) @contextmanager def progress_bar(self, label: str, total: int) -> Iterator[_ProgressHandle]: - """Display a Rich progress bar for *total* items. + """Show attempts, including a bounded completion event outside a TTY. - Yields a :class:`_ProgressHandle` with an ``advance()`` method. + An advance records an attempted item; asset download success is tracked + by capture statistics. Leaving this context never implies all assets + succeeded, and exceptions always produce an interrupted event. """ - if self._console and not self.quiet and self._console.is_terminal: - with Progress( - TextColumn("[progress.description]{task.description}"), - BarColumn(), - MofNCompleteColumn(), - TaskProgressColumn(), - TimeRemainingColumn(), - TimeElapsedColumn(), - console=self._console, - transient=False, - ) as prog: - task_id = prog.add_task(label, total=total) - handle = _ProgressHandle(prog, task_id) + started = time.monotonic() + self.phase(f"{label} (0 / {total})") + handle = _ProgressHandle(None, None) + try: + if self._console and not self.quiet and self._console.is_terminal: + with Progress( + TextColumn("{task.description}", markup=False), + BarColumn(), + MofNCompleteColumn(), + TaskProgressColumn(), + TimeRemainingColumn(), + TimeElapsedColumn(), + console=self._console, + transient=True, + ) as prog: + task_id = prog.add_task(redact_text(label), total=total) + handle = _ProgressHandle(prog, task_id) + yield handle + else: yield handle + except BaseException: + self._phase_finish( + label, + started, + interrupted=True, + attempted=handle.attempted, + total=total, + ) + raise else: - self.phase(f"{label} (0 / {total})") - handle = _ProgressHandle(None, None) - yield handle + self._phase_finish( + label, + started, + interrupted=False, + attempted=handle.attempted, + total=total, + ) # ------------------------------------------------------------------ # Summary @@ -188,16 +301,20 @@ def summary( archive_path: str | None = None, ) -> None: """Render the final capture summary table.""" + self.report_warnings(stats.warnings) if self.quiet or not self._console: return stats.elapsed_seconds = time.monotonic() - self._start table = Table(show_header=False, box=None, padding=(0, 2)) table.add_column(style="bold") - table.add_column() + table.add_column(overflow="fold") if project_name: - table.add_row("Project", project_name) + table.add_row("Project", Text(redact_text(project_name))) table.add_row("Clips", str(stats.clips_archived)) - table.add_row("Audio", str(stats.audio_downloaded)) + audio_label = str(stats.audio_downloaded) + if stats.audio_expected is not None: + audio_label = f"{stats.audio_downloaded} / {stats.audio_expected}" + table.add_row("Audio", audio_label) table.add_row("Artwork", str(stats.artwork_downloaded)) table.add_row("Videos", str(stats.videos_downloaded)) table.add_row("Elapsed", CaptureStats._fmt_elapsed(stats.elapsed_seconds)) @@ -205,19 +322,43 @@ def summary( table.add_row("Downloaded", CaptureStats._fmt_bytes(stats.bytes_downloaded)) if stats.throughput_mbps > 0: table.add_row("Throughput", f"{stats.throughput_mbps:.1f} MB/s") - verification_label = ( - "[green]PASS[/green]" if stats.verification_passed else "[red]FAIL[/red]" + verification_label = Text( + "PASS" if stats.verification_passed else "FAIL", + style="bright_green" if stats.verification_passed else "bright_red", + ) + table.add_row( + "Capture completeness", + Text( + "Complete" if stats.capture_complete else "Partial", + style="bright_green" if stats.capture_complete else "bright_yellow", + ), ) - table.add_row("Verification", verification_label) + table.add_row("Archive integrity", verification_label) if archive_path: - table.add_row("Archive", archive_path) - self._console.print() - self._console.rule("[bold]Capture Complete[/bold]") - self._console.print(table) - if stats.warnings: - self._console.print( - f"[yellow]{len(stats.warnings)} warning(s) recorded.[/yellow]" - ) + table.add_row("Archive", Text(redact_text(archive_path), overflow="fold")) + summary_console = self._summary_console + assert summary_console is not None + summary_console.print() + capture_label = ( + "Capture Complete" if stats.capture_complete else "Capture Partial" + ) + summary_console.rule( + Text(capture_label, style="bold"), + style="bright_green" if stats.capture_complete else "bright_yellow", + ) + _LOGGER.info( + "%s; clips=%s; audio=%s; archive_integrity=%s; archive=%s", + capture_label, + stats.clips_archived, + audio_label, + "PASS" if stats.verification_passed else "FAIL", + redact_text(archive_path or "not recorded"), + extra={"progress_event": True}, + ) + summary_console.print(table) + warning_count = max(self._warning_count, len(stats.warnings)) + if warning_count: + self._emit(f"{warning_count} warning(s) recorded.", logging.WARNING) class _ProgressHandle: @@ -226,7 +367,9 @@ class _ProgressHandle: def __init__(self, prog: Progress | None, task_id: object) -> None: self._prog = prog self._task_id = task_id + self.attempted = 0 def advance(self, n: int = 1) -> None: + self.attempted += n if self._prog is not None and self._task_id is not None: self._prog.advance(self._task_id, n) # type: ignore[arg-type] diff --git a/src/mindcap/plugins/suno/archive/diagnostics.py b/src/mindcap/plugins/suno/archive/diagnostics.py new file mode 100644 index 0000000..02f020a --- /dev/null +++ b/src/mindcap/plugins/suno/archive/diagnostics.py @@ -0,0 +1,143 @@ +"""Offline, allowlisted diagnostics for audio links in an existing archive.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any +from urllib.parse import parse_qsl, urlsplit + +from mindcap.core.errors import VerificationError +from mindcap.plugins.suno.archive.verifier import verify_workspace_bundle + +_API_HOSTS = {"studio-api-prod.suno.com", "studio-api.prod.suno.com"} +_CDN_HOSTS = {"cdn1.suno.ai", "cdn2.suno.ai", "cdn.suno.ai", "cdn-o.suno.com"} +_SIGNATURE_KEYS = {"signature", "key-pair-id", "policy", "x-amz-signature"} + + +def _audio_url(clip: dict[str, Any]) -> object: + provider = clip.get("provider_metadata") + if not isinstance(provider, dict): + provider = {} + metadata = provider.get("metadata") + if not isinstance(metadata, dict): + metadata = {} + return ( + clip.get("audio_url") or provider.get("audio_url") or metadata.get("audio_url") + ) + + +def suno_asset_diagnostics(bundle_path: Path) -> dict[str, Any]: + """Inspect saved URL shapes without network calls or exposing URL values. + + This deliberately returns counts and fixed labels only: no raw metadata, + titles, identifiers, hosts, signed links, cookies, or exception messages. + It works with archives created before asset-count reporting was added. + """ + bundle_path = bundle_path.expanduser().resolve() + try: + manifest = json.loads( + (bundle_path / "manifest.json").read_text(encoding="utf-8") + ) + checksums = json.loads( + (bundle_path / "checksums.json").read_text(encoding="utf-8") + ) + if manifest.get("workspace_metadata_path") != "workspace/metadata.json": + raise ValueError + if ( + sum( + isinstance(entry, dict) + and entry.get("path") == "workspace/metadata.json" + for entry in checksums["files"] + ) + != 1 + ): + raise ValueError + verify_workspace_bundle(bundle_path) + metadata = json.loads( + (bundle_path / "workspace" / "metadata.json").read_text(encoding="utf-8") + ) + if not isinstance(metadata, dict) or not isinstance( + metadata.get("clips"), list + ): + raise ValueError + clips = metadata["clips"] + if any(not isinstance(clip, dict) for clip in clips): + raise ValueError + except ( + VerificationError, + OSError, + ValueError, + KeyError, + TypeError, + AttributeError, + ) as error: + raise VerificationError( + "Suno archive audio metadata could not be verified." + ) from error + + counts = { + "missing": 0, + "invalid": 0, + "forbidden_endpoint": 0, + "suno_api": 0, + "suno_cdn": 0, + "other_http": 0, + } + query_urls = 0 + signed_query_urls = 0 + archived_audio = 0 + for clip in clips: + assets = clip.get("archived_assets") + if isinstance(assets, list) and any( + isinstance(asset, dict) + and asset.get("asset_type") == "audio" + and asset.get("capture_status", "downloaded") in ("downloaded", "verified") + for asset in assets + ): + archived_audio += 1 + url = _audio_url(clip) + if not url: + counts["missing"] += 1 + continue + try: + if not isinstance(url, str): + raise ValueError + parsed = urlsplit(url) + if ( + parsed.scheme not in {"http", "https"} + or not parsed.hostname + or parsed.username is not None + or parsed.password is not None + ): + raise ValueError + _ = ( + parsed.port + ) # Validate malformed/out-of-range ports without emitting them. + host = parsed.hostname.lower() + query_keys = { + key.lower() for key, _ in parse_qsl(parsed.query, max_num_fields=200) + } + except ValueError: + counts["invalid"] += 1 + continue + query_urls += bool(parsed.query) + signed_query_urls += bool(query_keys & _SIGNATURE_KEYS) + if host in _API_HOSTS and parsed.path.rstrip("/") == "/api/forbidden": + counts["forbidden_endpoint"] += 1 + elif host in _API_HOSTS: + counts["suno_api"] += 1 + elif host in _CDN_HOSTS: + counts["suno_cdn"] += 1 + else: + counts["other_http"] += 1 + + return { + "schema": "mindcap.suno-asset-diagnostics/v0.1", + "offline": True, + "clip_count": len(clips), + "clips_with_archived_primary_audio": archived_audio, + "audio_url_kinds": counts, + "audio_urls_with_query": query_urls, + "audio_urls_with_signature_keys": signed_query_urls, + } diff --git a/src/mindcap/plugins/suno/archive/inspector.py b/src/mindcap/plugins/suno/archive/inspector.py index 557de13..a82638d 100644 --- a/src/mindcap/plugins/suno/archive/inspector.py +++ b/src/mindcap/plugins/suno/archive/inspector.py @@ -110,6 +110,12 @@ def inspect_suno_archive(bundle_path: Path, console: Console) -> None: table.add_row("Capture version", str(manifest.get("capture_version", "?"))) table.add_row("Captured at", str(manifest.get("captured_at", "?"))) table.add_row("Clips", str(clip_count)) + completeness = manifest.get("capture_completeness") + if completeness not in ("complete", "partial"): + completeness = ( + "partial" if manifest.get("warnings") else "unknown (legacy archive)" + ) + table.add_row("Capture completeness", completeness) table.add_row("Audio", str(audio_count)) table.add_row("Artwork", str(artwork_count)) table.add_row("Videos", str(video_count)) diff --git a/src/mindcap/plugins/suno/archive/manifest.py b/src/mindcap/plugins/suno/archive/manifest.py index a890d56..09ffc2f 100644 --- a/src/mindcap/plugins/suno/archive/manifest.py +++ b/src/mindcap/plugins/suno/archive/manifest.py @@ -26,6 +26,21 @@ def build_manifest( "raw_unit_count": raw_unit_count, "asset_count": asset_count, "clip_count": len(normalized.get("clips") or []), + "capture_complete": normalized.get("capture_complete"), + "capture_completeness": normalized.get("capture_completeness"), + **{ + key: normalized[key] + for key in ( + "expected_clip_count", + "audio_count", + "audio_expected_count", + "audio_failed_count", + "asset_expected_count", + "asset_downloaded_count", + "asset_failed_count", + ) + if key in normalized + }, "warnings": normalized.get("warnings") or [], "readme_path": "README.md", "checksums_path": "checksums.json", diff --git a/src/mindcap/plugins/suno/archive/service.py b/src/mindcap/plugins/suno/archive/service.py index b0684da..322d5a5 100644 --- a/src/mindcap/plugins/suno/archive/service.py +++ b/src/mindcap/plugins/suno/archive/service.py @@ -135,18 +135,21 @@ def _asset_candidates( {}, ) ) - if options.get("include_video", True): - video_url = clip.get("video_url") or metadata.get("video_url") - if isinstance(video_url, str) and video_url: - candidates.append( - ( - "video", - video_url, - "video/mp4", - f"clips/{clip_id}/video/original.mp4", - {}, - ) + video_url = clip.get("video_url") or metadata.get("video_url") + if ( + options.get("include_video", True) + and isinstance(video_url, str) + and video_url + ): + candidates.append( + ( + "video", + video_url, + "video/mp4", + f"clips/{clip_id}/video/original.mp4", + {}, ) + ) # Additional media_urls variants beyond primary audio/video seen_urls = {audio_url, video_url} for media_entry in _object_list(clip.get("media_urls")): @@ -409,18 +412,18 @@ def capture(self, request: CaptureRequest) -> CaptureEnvelope: # ── Completeness validation ─────────────────────────────────────── actual_count = len(clips_by_id) - capture_complete: bool + capture_complete = len(assets) == len(all_asset_jobs) if expected_clip_count is not None and actual_count < expected_clip_count: warnings.append( f"Incomplete capture: expected {expected_clip_count} clips, " f"archived {actual_count}." ) capture_complete = False - else: - capture_complete = True stats.clips_archived = actual_count stats.audio_downloaded = audio_count + stats.audio_expected = len(audio_jobs) + stats.capture_complete = capture_complete stats.artwork_downloaded = artwork_count stats.videos_downloaded = video_count @@ -444,6 +447,11 @@ def capture(self, request: CaptureRequest) -> CaptureEnvelope: "api_origin": client.api_origin, "project_title": project_title, "audio_count": audio_count, + "audio_expected_count": len(audio_jobs), + "audio_failed_count": len(audio_jobs) - audio_count, + "asset_expected_count": len(all_asset_jobs), + "asset_downloaded_count": len(assets), + "asset_failed_count": len(all_asset_jobs) - len(assets), "artwork_count": artwork_count, "video_count": video_count, }, diff --git a/src/mindcap/plugins/suno/archive/storage.py b/src/mindcap/plugins/suno/archive/storage.py index cd4fca7..a1d5737 100644 --- a/src/mindcap/plugins/suno/archive/storage.py +++ b/src/mindcap/plugins/suno/archive/storage.py @@ -117,13 +117,32 @@ def write_text(relative_path: str, payload: str) -> None: json.dumps(normalized, indent=2, sort_keys=True, ensure_ascii=False), ) write_text("README.md", transcript) - report_json = { + report_json: dict[str, Any] = { "schema": "mindcap.suno-capture-report/v0.1", "source_id": source_id, "capture_version": version, - "status": "complete_with_warnings" - if normalized.get("warnings") - else "complete", + "status": ( + "partial" + if normalized.get("capture_complete") is False + else "complete_with_warnings" + if normalized.get("warnings") + else "complete" + ), + "capture_complete": normalized.get("capture_complete"), + "capture_completeness": normalized.get("capture_completeness"), + **{ + key: normalized[key] + for key in ( + "expected_clip_count", + "audio_count", + "audio_expected_count", + "audio_failed_count", + "asset_expected_count", + "asset_downloaded_count", + "asset_failed_count", + ) + if key in normalized + }, "workspace_id": normalized["workspace_id"], "clip_count": len(normalized.get("clips") or []), "asset_count": len(envelope.assets), @@ -137,10 +156,24 @@ def write_text(relative_path: str, payload: str) -> None: f"- Capture version: {version}", f"- Workspace ID: `{normalized['workspace_id']}`", f"- Status: {report_json['status']}", + f"- Capture completeness: {report_json['capture_completeness']}", f"- Clips archived: {report_json['clip_count']}", f"- Assets archived: {report_json['asset_count']}", - "", ] + if "audio_expected_count" in report_json: + report_markdown.append( + f"- Primary audio archived: {report_json['audio_count']} / " + f"{report_json['audio_expected_count']} " + f"({report_json['audio_failed_count']} failed)" + ) + if "asset_expected_count" in report_json: + report_markdown.append( + f"- Advertised assets archived: " + f"{report_json['asset_downloaded_count']} / " + f"{report_json['asset_expected_count']} " + f"({report_json['asset_failed_count']} failed)" + ) + report_markdown.append("") warnings = report_json.get("warnings") or [] if warnings: report_markdown.extend(["## Warnings", ""]) diff --git a/src/mindcap/plugins/suno/client.py b/src/mindcap/plugins/suno/client.py index 08c5c50..751ba42 100644 --- a/src/mindcap/plugins/suno/client.py +++ b/src/mindcap/plugins/suno/client.py @@ -2,7 +2,10 @@ import base64 import json +import shutil import tempfile +from collections.abc import Iterator +from contextlib import contextmanager from dataclasses import dataclass from datetime import UTC, datetime from pathlib import Path @@ -71,6 +74,12 @@ def __init__( timeout=httpx.Timeout(20.0, connect=10.0, read=20.0, write=20.0), transport=transport, ) + # Media hosts must never inherit the API client's authentication or cookies. + self._media_client = httpx.Client( + follow_redirects=False, + timeout=httpx.Timeout(20.0, connect=10.0, read=20.0, write=20.0), + transport=transport, + ) def _browser_token(self) -> str: payload = json.dumps({"timestamp": int(datetime.now(UTC).timestamp() * 1000)}) @@ -429,27 +438,100 @@ def get_aligned_lyrics( category="aligned-lyrics", ) + def _same_origin(self, left: httpx.URL, right: httpx.URL) -> bool: + return (left.scheme, left.host, left.port) == ( + right.scheme, + right.host, + right.port, + ) + + @contextmanager + def _stream_asset(self, url: str) -> Iterator[httpx.Response]: + try: + target = self._client.base_url.join(url) + except httpx.InvalidURL: + raise CaptureFailedError("Suno asset URL is invalid.") from None + api_url = httpx.URL(self.api_origin) + can_authenticate = self._same_origin(target, api_url) + for _ in range(21): + if ( + target.scheme not in {"http", "https"} + or not target.host + or target.userinfo + ): + raise CaptureFailedError("Suno asset URL is not a supported HTTP URL.") + headers = {"accept": "*/*", "user-agent": "mindcap/0.1.0"} + if can_authenticate and self._same_origin(target, api_url): + self.ensure_jwt() + headers.update(self._headers()) + # Construct requests directly: Client.build_request() would merge the + # media client's cookie jar, including cookies set by previous hosts. + request = httpx.Request("GET", target, headers=headers) + response = self._media_client.send(request, stream=True) + try: + if response.is_redirect: + location = response.headers.get("location") + if not location: + raise CaptureFailedError( + "Suno asset redirect has no download location." + ) + try: + next_target = target.join(location) + except httpx.InvalidURL: + raise CaptureFailedError( + "Suno asset redirect URL is invalid." + ) from None + if not self._same_origin(target, next_target): + can_authenticate = False + target = next_target + continue + yield response + return + finally: + response.close() + raise CaptureFailedError("Suno asset exceeded the download redirect limit.") + def download_to_file(self, url: str) -> tuple[Path, dict[str, Any]]: suffix = Path(redact_signed_url(url) or url).suffix or ".bin" temp_dir = Path(tempfile.mkdtemp(prefix="mindcap-suno-", dir="/tmp")) destination = temp_dir / f"asset{suffix}" part_path = destination.with_suffix(destination.suffix + ".part") - with self._client.stream("GET", url, headers=self._headers()) as response: - if response.status_code >= 400: + try: + try: + with self._stream_asset(url) as response: + if response.status_code == 403: + raise CaptureFailedError( + "Failed to download Suno asset (403). The asset host " + "denied access; check whether Download works for this " + "clip in Suno. Provider download permissions or an " + "expired asset URL may require a fresh capture." + ) + if response.status_code >= 400: + raise CaptureFailedError( + f"Failed to download Suno asset ({response.status_code})." + ) + with part_path.open("wb") as handle: + total = 0 + for chunk in response.iter_bytes(): + handle.write(chunk) + total += len(chunk) + if total == 0: + raise CaptureFailedError("Suno asset download was empty.") + media_type = response.headers.get( + "content-type", "application/octet-stream" + ) + except httpx.HTTPError: raise CaptureFailedError( - f"Failed to download Suno asset ({response.status_code})." - ) - with part_path.open("wb") as handle: - total = 0 - for chunk in response.iter_bytes(): - handle.write(chunk) - total += len(chunk) - part_path.replace(destination) - return destination, { - "byte_size": destination.stat().st_size, - "media_type": response.headers.get( - "content-type", "application/octet-stream" - ), - "downloaded_at": datetime.now(UTC).isoformat(), - "source_url": redact_signed_url(url), - } + "Suno asset transfer failed. Retry the capture; no complete " + "asset was saved." + ) from None + part_path.replace(destination) + return destination, { + "byte_size": destination.stat().st_size, + "media_type": media_type, + "downloaded_at": datetime.now(UTC).isoformat(), + "source_url": redact_signed_url(url), + } + except BaseException: + shutil.rmtree(temp_dir, ignore_errors=True) + raise diff --git a/src/mindcap/plugins/suno/normalizer.py b/src/mindcap/plugins/suno/normalizer.py index 185621b..1e60d01 100644 --- a/src/mindcap/plugins/suno/normalizer.py +++ b/src/mindcap/plugins/suno/normalizer.py @@ -274,6 +274,9 @@ def normalize_suno( ) ) + capture_complete = bool( + envelope.safe_metadata.get("capture_complete", not envelope.warnings) + ) return { "schema": "mindcap.suno-workspace/v0.3", "provider": "suno", @@ -295,7 +298,21 @@ def normalize_suno( "is_public": workspace.get("is_public"), "is_trashed": workspace.get("is_trashed"), }, - "capture_completeness": "partial" if envelope.warnings else "complete", + "capture_complete": capture_complete, + "capture_completeness": "complete" if capture_complete else "partial", + **{ + key: envelope.safe_metadata[key] + for key in ( + "expected_clip_count", + "audio_count", + "audio_expected_count", + "audio_failed_count", + "asset_expected_count", + "asset_downloaded_count", + "asset_failed_count", + ) + if key in envelope.safe_metadata + }, "pagination_evidence": [ unit.safe_metadata for unit in envelope.response_units diff --git a/src/mindcap/sync/runner.py b/src/mindcap/sync/runner.py index 68a1f09..d99ab4a 100644 --- a/src/mindcap/sync/runner.py +++ b/src/mindcap/sync/runner.py @@ -26,10 +26,12 @@ import signal import time +from contextlib import suppress from datetime import UTC, datetime from pathlib import Path from typing import Any +from mindcap.core.diagnostics import redact_text from mindcap.core.errors import AuthenticationRequiredError, MindcapError from mindcap.core.models import CaptureRequest from mindcap.core.progress import CaptureProgressReporter @@ -142,6 +144,7 @@ def run( BatchRunState Final run state after completion or interruption. """ + self._interrupted = False effective_run_id = run_id or generate_run_id(config.provider) storage = RunStorage(artifact_root, config.provider, effective_run_id) storage.ensure_dir() @@ -153,13 +156,13 @@ def run( else: state = create_initial_state(effective_run_id, config, artifact_root) - self._install_signal_handlers(state, storage) - storage.save_state(state) - storage.append_run_event( - {"event": "run-started", "run_id": effective_run_id} - ) + previous_handlers = self._install_signal_handlers(state, storage) try: + storage.save_state(state) + storage.append_run_event( + {"event": "run-started", "run_id": effective_run_id} + ) state = self._run_body( state=state, config=config, @@ -171,21 +174,30 @@ def run( except _InterruptedError: pass # Handled in signal handler; state already updated. except (MindcapError, OSError, Exception) as error: + safe_error = redact_text(str(error)) state.status = RunStatus.FAILED - state.errors.append(str(error)) + state.completed_at = datetime.now(UTC) + state.errors.append(safe_error) + self._reporter.warn(f"Sync failed: {safe_error}") storage.save_state(state) - storage.append_run_event({"event": "run-failed", "error": str(error)}) + storage.save_report(state) + storage.append_run_event({"event": "run-failed", "error": safe_error}) finally: - if state.status not in { - RunStatus.COMPLETE, - RunStatus.COMPLETE_WITH_FAILURES, - RunStatus.FAILED, - }: - if self._interrupted: - state.status = RunStatus.INTERRUPTED - state.completed_at = datetime.now(UTC) - storage.save_state(state) - storage.save_report(state) + try: + if state.status not in { + RunStatus.COMPLETE, + RunStatus.COMPLETE_WITH_FAILURES, + RunStatus.FAILED, + }: + if self._interrupted: + state.status = RunStatus.INTERRUPTED + state.completed_at = datetime.now(UTC) + storage.save_state(state) + storage.save_report(state) + finally: + for signum, previous_handler in previous_handlers.items(): + with suppress(OSError, ValueError): + signal.signal(signum, previous_handler) return state @@ -209,6 +221,7 @@ def _run_body( or not state.discovery_result.discovery_complete ) if needs_discovery: + self._reporter.phase("Discovering sources...") state.status = RunStatus.DISCOVERING storage.save_state(state) collection_request = CollectionRequest( @@ -245,6 +258,7 @@ def _run_body( raise _InterruptedError() # ---- Planning -------------------------------------------------- + self._reporter.phase("Planning captures...") plugin = build_registry().get(config.provider) verify_fn = _make_verify_fn(plugin) @@ -273,9 +287,9 @@ def _run_body( # ---- Execution ------------------------------------------------- strategy_name = plugin.default_strategy() - for item in items: + for position, item in enumerate(items, start=1): if self._interrupted: - item.transition(ItemStatus.INTERRUPTED) + # Unstarted items remain planned and can be captured on resume. break action = item.action @@ -293,6 +307,10 @@ def _run_body( item.transition(ItemStatus.BLOCKED) continue + self._reporter.phase( + f"Capturing source {position} / {len(items)}: " + f"{redact_text(item.descriptor.canonical_identifier)}" + ) self._capture_item( item=item, config=config, @@ -306,7 +324,9 @@ def _run_body( # ---- Finalize -------------------------------------------------- counts = state.counts() - if counts["failed"] > 0: + if self._interrupted: + state.status = RunStatus.INTERRUPTED + elif counts["failed"] > 0: state.status = RunStatus.COMPLETE_WITH_FAILURES else: state.status = RunStatus.COMPLETE @@ -334,6 +354,7 @@ def _capture_item( ) -> None: """Execute one item, retrying on retryable failures.""" descriptor = item.descriptor + safe_identifier = redact_text(descriptor.canonical_identifier) action = item.action next_status = ( @@ -345,6 +366,10 @@ def _capture_item( while True: attempt_num = item.retry_count + 1 + self._reporter.detail( + f"Source {safe_identifier}: attempt {attempt_num} " + f"of {item.max_retries + 1}" + ) started = datetime.now(UTC) storage.append_item_event( { @@ -373,7 +398,9 @@ def _capture_item( options=config.options, ) strategy_obj = plugin.strategy(strategy_name, reporter=self._reporter) + self._reporter.begin_capture() envelope = strategy_obj.capture(request) + self._reporter.report_warnings(envelope.warnings) normalized = plugin.normalize(envelope, descriptor.canonical_identifier) transcript = plugin.render(normalized) stored = plugin.storage().persist( @@ -386,14 +413,41 @@ def _capture_item( item.attempts.append(attempt) item.archive_path = str(stored.path) item.archive_version = stored.version + attempt.warnings = [redact_text(w) for w in envelope.warnings] + item.warnings.extend(attempt.warnings) + + if envelope.safe_metadata.get("capture_complete") is False: + partial_error = ( + "Capture is partial: required assets missing; " + "saved archive retained. Inspect warnings before retrying." + ) + attempt.status = ItemStatus.FAILED_TERMINAL + attempt.error = partial_error + item.errors.append(partial_error) + item.transition(ItemStatus.FAILED_TERMINAL) + self._reporter.warn( + f"Source {safe_identifier} failed: {partial_error}" + ) + storage.append_item_event( + { + "event": "item-failed", + "id": descriptor.canonical_identifier, + "terminal": True, + "status": item.status.value, + "archive_path": str(stored.path), + "error": partial_error, + } + ) + return if stored.status == "unchanged": item.transition(ItemStatus.UNCHANGED) - elif item.warnings or envelope.warnings: + elif item.warnings: item.transition(ItemStatus.COMPLETE_WITH_WARNINGS) else: item.transition(ItemStatus.COMPLETE) + self._reporter.detail(f"Source {safe_identifier}: {item.status.value}") storage.append_item_event( { "event": "item-complete", @@ -405,25 +459,32 @@ def _capture_item( return except AuthenticationRequiredError as error: + safe_error = redact_text(str(error)) attempt.finished_at = datetime.now(UTC) attempt.status = ItemStatus.FAILED_TERMINAL - attempt.error = str(error) + attempt.error = safe_error item.attempts.append(attempt) - item.errors.append(f"Authentication expired: {error}") + item.errors.append(f"Authentication expired: {safe_error}") item.transition(ItemStatus.FAILED_TERMINAL) + self._reporter.warn( + f"Source {safe_identifier} failed: {safe_error}. " + "Authenticate again before retrying this source." + ) storage.append_item_event( { "event": "item-failed", "id": descriptor.canonical_identifier, "terminal": True, - "error": str(error), + "error": safe_error, } ) return except (MindcapError, OSError, Exception) as error: + safe_error = redact_text(str(error)) attempt.finished_at = datetime.now(UTC) - item.errors.append(str(error)) + attempt.error = safe_error + item.errors.append(safe_error) if _is_retryable_error(error) and item.retry_count < item.max_retries: attempt.status = ItemStatus.FAILED_RETRYABLE @@ -432,27 +493,36 @@ def _capture_item( item.transition(ItemStatus.FAILED_RETRYABLE) item.transition(ItemStatus.CAPTURING) backoff = min(2**item.retry_count, 30) - if not self._interrupted: - time.sleep(backoff) + self._reporter.warn( + f"Source {safe_identifier}: attempt {attempt_num} failed: " + f"{safe_error}. Retrying in {backoff}s " + f"(retry {item.retry_count} of {item.max_retries})." + ) storage.append_item_event( { "event": "item-retry", "id": descriptor.canonical_identifier, "attempt": attempt_num, - "error": str(error), + "error": safe_error, } ) + if not self._interrupted: + time.sleep(backoff) continue attempt.status = ItemStatus.FAILED_TERMINAL item.attempts.append(attempt) item.transition(ItemStatus.FAILED_TERMINAL) + self._reporter.warn( + f"Source {safe_identifier} failed after {attempt_num} " + f"attempt(s): {safe_error}." + ) storage.append_item_event( { "event": "item-failed", "id": descriptor.canonical_identifier, "terminal": True, - "error": str(error), + "error": safe_error, } ) return @@ -463,8 +533,8 @@ def _capture_item( def _install_signal_handlers( self, state: BatchRunState, storage: RunStorage - ) -> None: - """Install SIGINT and SIGTERM handlers for graceful interruption.""" + ) -> dict[int, Any]: + """Install interruption handlers and return the handlers to restore.""" def _handle(signum: int, frame: object) -> None: self._interrupted = True @@ -477,12 +547,16 @@ def _handle(signum: int, frame: object) -> None: except Exception: pass - try: - signal.signal(signal.SIGINT, _handle) - signal.signal(signal.SIGTERM, _handle) - except (OSError, ValueError): - # Signal handling is not available in all environments. - pass + previous_handlers: dict[int, Any] = {} + for signum in (signal.SIGINT, signal.SIGTERM): + try: + previous_handler = signal.getsignal(signum) + signal.signal(signum, _handle) + previous_handlers[signum] = previous_handler + except (OSError, ValueError): + # Signal handling is not available in all environments. + pass + return previous_handlers # --------------------------------------------------------------------------- diff --git a/src/mindcap/vault/catalog.py b/src/mindcap/vault/catalog.py index 2b928de..37ec178 100644 --- a/src/mindcap/vault/catalog.py +++ b/src/mindcap/vault/catalog.py @@ -104,7 +104,6 @@ ) - def connect_database(path: Path) -> sqlite3.Connection: conn = sqlite3.connect(path) conn.row_factory = sqlite3.Row @@ -113,7 +112,6 @@ def connect_database(path: Path) -> sqlite3.Connection: return conn - def apply_migrations(conn: sqlite3.Connection, metadata: VaultMetadata) -> None: conn.execute( """ @@ -146,7 +144,6 @@ def apply_migrations(conn: sqlite3.Connection, metadata: VaultMetadata) -> None: conn.commit() - def prepare_staged_database( vault_metadata: VaultMetadata, latest_catalog: Path | None, @@ -164,7 +161,6 @@ def prepare_staged_database( return (workdir, staged_db) - def load_imported_units(conn: sqlite3.Connection) -> set[tuple[str, str, str]]: rows = conn.execute( "SELECT provider, source_id, capture_version FROM archive_units" @@ -175,7 +171,6 @@ def load_imported_units(conn: sqlite3.Connection) -> set[tuple[str, str, str]]: } - def load_object_locations(conn: sqlite3.Connection) -> dict[str, tuple[str, str, int]]: rows = conn.execute( "SELECT object_sha256, pack_id, member_name, byte_size FROM object_locations" @@ -190,7 +185,6 @@ def load_object_locations(conn: sqlite3.Connection) -> dict[str, tuple[str, str, } - def next_generation(conn: sqlite3.Connection) -> int: row = conn.execute( "SELECT COALESCE(MAX(generation), 0) AS generation FROM catalog_generations" @@ -199,7 +193,6 @@ def next_generation(conn: sqlite3.Connection) -> int: return int(row["generation"]) + 1 - def insert_generation( conn: sqlite3.Connection, generation: int, @@ -215,7 +208,6 @@ def insert_generation( ) - def insert_ingestion_run( conn: sqlite3.Connection, *, @@ -251,7 +243,6 @@ def insert_ingestion_run( ) - def record_pack_seal(conn: sqlite3.Connection, seal: PackSeal) -> None: conn.execute( """ @@ -291,7 +282,6 @@ def record_pack_seal(conn: sqlite3.Connection, seal: PackSeal) -> None: ) - def insert_archive( conn: sqlite3.Connection, archive: PlannedArchive, @@ -348,7 +338,6 @@ def insert_archive( ) - def insert_provider_record( conn: sqlite3.Connection, archive_unit_id: int, @@ -384,7 +373,6 @@ def insert_provider_record( ) - def validate_database(conn: sqlite3.Connection) -> None: integrity = conn.execute("PRAGMA integrity_check").fetchone() if integrity is None or integrity[0] != "ok": @@ -394,7 +382,6 @@ def validate_database(conn: sqlite3.Connection) -> None: raise CatalogIntegrityError(f"Catalog foreign-key check failed: {violations}") - def summarize(conn: sqlite3.Connection) -> dict[str, Any]: queries = { "archive_units": "SELECT COUNT(*) FROM archive_units", @@ -430,7 +417,6 @@ def summarize(conn: sqlite3.Connection) -> dict[str, Any]: return values - def _json_dumps(value: dict[str, Any]) -> str: import json diff --git a/src/mindcap_cli/_compat.py b/src/mindcap_cli/_compat.py new file mode 100644 index 0000000..8daf82a --- /dev/null +++ b/src/mindcap_cli/_compat.py @@ -0,0 +1,10 @@ +"""Use the command engine owned by the installed Typer version.""" + +try: + from typer import _click as click + from typer._click.globals import get_current_context +except ImportError: # Typer versions before the command engine was vendored. + import click # type: ignore[no-redef,import-not-found,unused-ignore] + from click import get_current_context # type: ignore[no-redef] + +__all__ = ["click", "get_current_context"] diff --git a/src/mindcap_cli/app.py b/src/mindcap_cli/app.py index 44a7e2e..b4f3848 100644 --- a/src/mindcap_cli/app.py +++ b/src/mindcap_cli/app.py @@ -1,6 +1,7 @@ from __future__ import annotations import json +import logging import platform import subprocess import sys @@ -22,6 +23,7 @@ distrokid_profile_dir, find_repository_root, ) +from mindcap.core.diagnostics import redact_text from mindcap.core.errors import MindcapError from mindcap.core.models import CaptureEnvelope, CaptureRequest, RawResponseUnit from mindcap.core.progress import CaptureProgressReporter, CaptureStats @@ -45,21 +47,25 @@ from mindcap.plugins.suno.doctor import doctor_suno as run_suno_doctor from mindcap.registry import build_registry from mindcap.storage import verify_bundle +from mindcap_cli.runtime import DiagnosticTyper, diagnostic_options, fail -app = typer.Typer( +app = DiagnosticTyper( name="mindcap", + pretty_exceptions_enable=False, no_args_is_help=True, help="Capture and normalize sources for knowledge extraction.", ) -auth_app = typer.Typer(no_args_is_help=True, help="Manage source authentication.") -plugins_app = typer.Typer(no_args_is_help=True, help="Inspect source plugins.") -doctor_app = typer.Typer(no_args_is_help=True, help="Inspect safe browser diagnostics.") -inspect_app = typer.Typer(no_args_is_help=True, help="Inspect captured archives.") -sync_app = typer.Typer( +auth_app = DiagnosticTyper(no_args_is_help=True, help="Manage source authentication.") +plugins_app = DiagnosticTyper(no_args_is_help=True, help="Inspect source plugins.") +doctor_app = DiagnosticTyper( + no_args_is_help=True, help="Inspect safe browser diagnostics." +) +inspect_app = DiagnosticTyper(no_args_is_help=True, help="Inspect captured archives.") +sync_app = DiagnosticTyper( no_args_is_help=True, help="Provider-wide cache-aware synchronization.", ) -vault_app = typer.Typer( +vault_app = DiagnosticTyper( no_args_is_help=True, help="Ingest, inspect, verify, and restore immutable archival vaults.", ) @@ -70,6 +76,7 @@ app.add_typer(sync_app, name="sync") app.add_typer(vault_app, name="vault") console = Console() +logger = logging.getLogger("mindcap.cli") def _print_version() -> None: @@ -95,13 +102,27 @@ def main( callback=_version_callback, ), ] = False, + verbose: Annotated[ + bool, typer.Option("--verbose", help="Show per-item diagnostic details.") + ] = False, + debug: Annotated[ + bool, typer.Option("--debug", help="Show sanitized debug diagnostics.") + ] = False, + quiet: Annotated[ + bool, typer.Option("--quiet", help="Suppress informational diagnostics.") + ] = False, + log_file: Annotated[ + Path | None, + typer.Option( + "--log-file", help="Append sanitized JSONL diagnostics to a file." + ), + ] = None, ) -> None: - """Configure eager top-level options for the Mindcap CLI.""" + """Configure global diagnostic options; put these before the command name.""" def _fail(error: Exception) -> None: - console.print(f"[bold red]Mindcap error:[/bold red] {error}") - raise typer.Exit(code=1) from error + fail(error) def _fmt_bytes_human(value: int) -> str: @@ -128,11 +149,13 @@ def _capture( options: dict[str, Any], identifier_override: str | None = None, ) -> None: - verbose: bool = bool(options.get("verbose")) - debug: bool = bool(options.get("debug")) - quiet: bool = bool(options.get("quiet")) and not bool(options.get("json")) + flags = diagnostic_options() + verbose = bool(options.get("verbose")) or flags["verbose"] + debug = bool(options.get("debug")) or flags["debug"] + quiet = bool(options.get("quiet")) or flags["quiet"] reporter = CaptureProgressReporter( - console=console, + console=Console(stderr=True), + summary_console=console, verbose=verbose, debug=debug, quiet=quiet, @@ -164,10 +187,23 @@ def _capture( reporter.phase("Verifying archive...") stored = plugin.storage().persist(request, envelope, normalized, transcript) elapsed = time.monotonic() - start_time + reporter.report_warnings(envelope.warnings) + logger.info( + "Archive saved: integrity=%s capture_complete=%s elapsed=%.2fs path=%s", + stored.status, + envelope.safe_metadata.get("capture_complete", not envelope.warnings), + elapsed, + stored.path, + extra={"progress_event": True}, + ) if options.get("json"): console.print_json( data={ "status": stored.status, + "capture_complete": envelope.safe_metadata.get( + "capture_complete", not envelope.warnings + ), + "capture_completeness": normalized.get("capture_completeness"), "source_id": stored.source_id, "version": stored.version, "path": str(stored.path), @@ -180,11 +216,15 @@ def _capture( stats = CaptureStats( clips_archived=int(safe_meta.get("clip_count") or 0), audio_downloaded=int(safe_meta.get("audio_count") or 0), + audio_expected=safe_meta.get("audio_expected_count"), artwork_downloaded=int(safe_meta.get("artwork_count") or 0), videos_downloaded=int(safe_meta.get("video_count") or 0), bytes_downloaded=int(safe_meta.get("bytes_downloaded") or 0), elapsed_seconds=elapsed, verification_passed=stored.status in ("complete", "unchanged"), + capture_complete=safe_meta.get( + "capture_complete", not envelope.warnings + ), warnings=list(envelope.warnings), ) reporter.summary( @@ -194,6 +234,12 @@ def _capture( else None, archive_path=str(stored.path), ) + if not envelope.safe_metadata.get("capture_complete", not envelope.warnings): + logger.warning( + "Capture is partial. Saved files remain available; inspect the " + "archive warnings before retrying. Exit code: 2." + ) + raise typer.Exit(code=2) except (MindcapError, OSError, ValueError, json.JSONDecodeError) as error: _fail(error) @@ -213,8 +259,9 @@ def _capture_export( from mindcap.plugins.chatgpt.plugin import ChatGPTPlugin from mindcap.plugins.chatgpt.strategies.export import ExportCaptureStrategy - quiet: bool = bool(options.get("quiet")) and not bool(options.get("json")) - verbose: bool = bool(options.get("verbose")) + flags = diagnostic_options() + quiet: bool = bool(options.get("quiet")) or flags["quiet"] + verbose: bool = bool(options.get("verbose")) or flags["verbose"] artifact_root = (output or default_artifact_root()).resolve() start_time = time.monotonic() @@ -227,15 +274,20 @@ def _capture_export( _fail(error) return + reporter = CaptureProgressReporter( + console=Console(stderr=True), + verbose=verbose, + debug=bool(options.get("debug")) or flags["debug"], + quiet=quiet, + ) if not quiet: - console.print( - f"[bold]Inspecting export:[/bold] {source}\n" + reporter.phase( + f"Inspecting export: {source}\n" f" Conversation files: {len(discovery.conversation_files)}\n" f" Metadata files: {len(discovery.metadata_files)}\n" f" Unknown files: {len(discovery.unknown_files)}" ) - for w in discovery.warnings: - console.print(f" [yellow]⚠ {w}[/yellow]") + reporter.report_warnings(discovery.warnings) import_id = f"import-{uuid.uuid4().hex[:16]}" import_root = artifact_root / "imports" / "chatgpt" / import_id @@ -249,7 +301,7 @@ def _capture_export( total_discovered = 0 if not quiet: - console.print("\n[bold]Parsing conversations...[/bold]") + reporter.phase("Parsing conversations...") try: for record in export_strategy.iter_conversations( @@ -306,17 +358,19 @@ def _capture_export( else: imported.append(entry) if verbose: - console.print(f" [{stored.status}] {conv_id} → {stored.path.name}") + reporter.detail(f"{stored.status}: {conv_id} → {stored.path.name}") except (MindcapError, OSError, ValueError, json.JSONDecodeError) as exc: failed.append( { "conversation_id": conv_id, "status": "failed", - "error": str(exc), + "error": redact_text(str(exc)), "source_file": record.source_file, } ) - all_warnings.append(f"Failed to import conversation {conv_id}: {exc}") + warning = redact_text(f"Failed to import conversation {conv_id}: {exc}") + all_warnings.append(warning) + reporter.warn(warning) except ( UnsupportedExportError, UnsupportedConversationSchemaError, @@ -392,7 +446,7 @@ def _capture_export( console.print(table) if failed: - raise typer.Exit(code=1) + raise typer.Exit(code=2) @auth_app.command("chatgpt") @@ -879,12 +933,24 @@ def doctor_google_drive_command( @inspect_app.command("suno") def inspect_suno( archive: Annotated[Path, typer.Argument(help="Suno workspace bundle directory.")], + asset_diagnostics: Annotated[ + bool, + typer.Option( + "--asset-diagnostics", + help="Print offline audio-link counts as JSON without URLs or identifiers.", + ), + ] = False, ) -> None: """Inspect a captured Suno workspace archive.""" from mindcap.plugins.suno.archive.inspector import inspect_suno_archive try: - inspect_suno_archive(archive, console) + if asset_diagnostics: + from mindcap.plugins.suno.archive.diagnostics import suno_asset_diagnostics + + console.print_json(data=suno_asset_diagnostics(archive)) + else: + inspect_suno_archive(archive, console) except (MindcapError, OSError) as error: _fail(error) @@ -1041,9 +1107,9 @@ def _conv_or_norm(key: str) -> str: console.print(table) if verbose: for w in graph_integrity.get("warnings") or []: - console.print(f" [yellow]⚠ Graph: {w}[/yellow]") + logger.warning("Graph: %s", w) for w in normalized.get("attachment_warnings") or []: - console.print(f" [yellow]⚠ {w}[/yellow]") + logger.warning("%s", w) return _fail( @@ -1120,7 +1186,7 @@ def vault_create( _fail( ValueError( "Pass --name when creating a Google Drive-backed vault " - "(for example: --name \"suno.mindcap-vault\")." + '(for example: --name "suno.mindcap-vault").' ) ) return @@ -1226,7 +1292,7 @@ def vault_ingest( console.print(table) if summary.warning: console.print() - console.print(f"[yellow]Warning:[/yellow] {summary.warning}") + logger.warning("%s", summary.warning) @vault_app.command("inspect") @@ -1323,8 +1389,8 @@ def vault_verify( console.print(table) console.print() console.print("[bold green]PASS[/bold green]") - console.print( - "[yellow]Warning:[/yellow] Successful filesystem verification does " + logger.warning( + "Successful filesystem verification does " "not confirm that a Google Drive client has finished remote " "synchronization." ) @@ -1414,9 +1480,10 @@ def _run_sync( from mindcap.sync.runner import SyncRunner, build_sync_result artifact_root = (output or default_artifact_root()).resolve() - _quiet = quiet and not json_output + _quiet = quiet reporter = CaptureProgressReporter( - console=console, + console=Console(stderr=True), + summary_console=console, verbose=verbose, debug=debug, quiet=_quiet, @@ -1477,16 +1544,15 @@ def _run_sync( ) if len(candidates) == 0: if not _quiet: - console.print( - "[yellow]No compatible unfinished run found; " - "starting a new run.[/yellow]" + reporter.warn( + "No compatible unfinished run found; starting a new run." ) elif len(candidates) == 1: storage = candidates[0] prior_state = storage.load_state() effective_run_id = storage.run_id if not _quiet: - console.print(f"[bold]Resuming run:[/bold] {effective_run_id}") + reporter.phase(f"Resuming run: {effective_run_id}") else: ids = ", ".join(s.run_id for s in candidates[:5]) _fail( @@ -1549,8 +1615,8 @@ def _run_sync( console.print(str(run_dir)) if state.status == RunStatus.INTERRUPTED: - console.print( - f"\n[bold yellow]Capture interrupted safely.[/bold yellow]\n" + reporter.warn( + f"Capture interrupted safely.\n" f"\nRun ID:\n {state.run_id}\n" f"\nResume:\n" f" mindcap sync {provider} --resume --run-id {state.run_id}" diff --git a/src/mindcap_cli/logging.py b/src/mindcap_cli/logging.py new file mode 100644 index 0000000..69ab2a6 --- /dev/null +++ b/src/mindcap_cli/logging.py @@ -0,0 +1,360 @@ +"""Invocation-scoped, credential-safe CLI diagnostics. + +Human diagnostics and JSON error envelopes go to stderr. Optional diagnostic +files contain JSON Lines and never Rich markup, terminal escapes, source lines, +locals, or arbitrary LogRecord extras. Only the ``mindcap`` namespace is owned +here; HTTP libraries and the application's root logger are left alone. +""" + +from __future__ import annotations + +import json +import logging +import os +import stat +import sys +from collections.abc import Iterator +from contextlib import contextmanager, suppress +from datetime import UTC, datetime +from pathlib import Path +from types import TracebackType +from typing import Any, TextIO + +from rich.console import Console +from rich.panel import Panel +from rich.text import Text +from typer import Abort + +from mindcap.core.diagnostics import redact_text +from mindcap.core.errors import ( + AuthenticationRequiredError, + CaptureFailedError, + InvalidSourceError, + MindcapError, + NormalizationError, + ProfileLockedError, + StableChromeNotFoundError, + VerificationError, +) +from mindcap_cli._compat import click + +_STYLES = { + logging.DEBUG: "dim cyan", + logging.INFO: "bright_cyan", + logging.WARNING: "bold bright_yellow", + logging.ERROR: "bold bright_red", + logging.CRITICAL: "bold bright_red", +} + + +class DiagnosticLogError(MindcapError): + """The requested local diagnostic file cannot be opened safely.""" + + +def _console() -> Console: + return Console(stderr=True, markup=False, highlight=False) + + +def _frame_locations(traceback: TracebackType | None) -> list[str]: + locations = [] + while traceback is not None: + code = traceback.tb_frame.f_code + locations.append( + redact_text( + f"{Path(code.co_filename).name}:{traceback.tb_lineno} in {code.co_name}" + ) + ) + traceback = traceback.tb_next + return locations + + +def _message(record: logging.LogRecord) -> str: + try: + return redact_text(record.getMessage()) + except Exception: + # Logging must not leak raw args through logging's default handleError. + return "Diagnostic message could not be formatted." + + +class _TerminalHandler(logging.Handler): + def __init__(self) -> None: + super().__init__() + self.console = _console() + + def emit(self, record: logging.LogRecord) -> None: + if getattr(record, "progress_event", False): + return + timestamp = datetime.fromtimestamp(record.created).strftime("%H:%M:%S") + label = redact_text(record.levelname) + text = Text(f"{timestamp} ", style="dim") + text.append(f"{label:<7} ", style=_STYLES.get(record.levelno, "")) + text.append(_message(record), style="default") + # A closed stderr should not trigger logging's unsanitized traceback. + with suppress(OSError): + self.console.print(text, soft_wrap=True) + + +class _JsonFileHandler(logging.Handler): + def __init__(self, stream: TextIO) -> None: + super().__init__() + self.stream = stream + self.failed = False + + def emit(self, record: logging.LogRecord) -> None: + payload: dict[str, Any] = { + "timestamp": datetime.fromtimestamp(record.created, UTC).isoformat( + timespec="milliseconds" + ), + "level": redact_text(record.levelname), + "logger": redact_text(record.name), + "message": _message(record), + } + event = getattr(record, "event", None) + if isinstance(event, str): + payload["event"] = redact_text(event) + error = getattr(record, "diagnostic_error", None) + if isinstance(error, dict): + payload["error"] = { + key: redact_text(value) + for key, value in error.items() + if key in {"code", "type", "message", "hint"} and isinstance(value, str) + } + frames = getattr(record, "diagnostic_frames", None) + if isinstance(frames, list): + payload["frames"] = [ + redact_text(frame) for frame in frames if isinstance(frame, str) + ] + try: + self.stream.write(json.dumps(payload, ensure_ascii=False) + "\n") + self.stream.flush() + except (OSError, ValueError): + if not self.failed: + self.failed = True + _console().print( + Text( + "WARNING Diagnostic log could not be written; " + "terminal diagnostics remain available.", + style="bold yellow", + ) + ) + + def close(self) -> None: + try: + self.stream.close() + except OSError: + pass + finally: + super().close() + + +def _open_log_file(path: Path) -> TextIO: + """Append to a private regular file, refusing symlinks and hard links.""" + destination = Path(path).expanduser() + descriptor: int | None = None + try: + if destination.is_symlink(): + raise DiagnosticLogError( + "Diagnostic log must be a regular file, not a symbolic link." + ) + flags = os.O_WRONLY | os.O_CREAT | os.O_APPEND + flags |= getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0) + descriptor = os.open(destination, flags, 0o600) + opened = os.fstat(descriptor) + current = destination.lstat() + if ( + not stat.S_ISREG(opened.st_mode) + or not stat.S_ISREG(current.st_mode) + or (opened.st_dev, opened.st_ino) != (current.st_dev, current.st_ino) + or opened.st_nlink != 1 + ): + raise DiagnosticLogError( + "Diagnostic log must be a regular file without links." + ) + os.fchmod(descriptor, 0o600) + stream = os.fdopen(descriptor, "a", encoding="utf-8", buffering=1) + descriptor = None + return stream + except DiagnosticLogError: + raise + except OSError as error: + raise DiagnosticLogError( + "Cannot open diagnostic log. Choose a writable file in an existing folder." + ) from error + finally: + if descriptor is not None: + os.close(descriptor) + + +@contextmanager +def diagnostic_session( + *, + verbose: bool = False, + debug: bool = False, + quiet: bool = False, + log_file: Path | None = None, +) -> Iterator[None]: + """Install only Mindcap handlers and restore them after each invocation. + + Verbose enables detailed progress in the caller. Debug additionally enables + DEBUG records. Quiet suppresses INFO progress but preserves warnings/errors. + File diagnostics follow the same level and include reporter-owned events. + """ + del verbose # Progress detail belongs to the progress reporter. + namespace = logging.getLogger("mindcap") + previous = ( + list(namespace.handlers), + namespace.level, + namespace.propagate, + namespace.disabled, + ) + handlers: list[logging.Handler] = [_TerminalHandler()] + try: + if log_file is not None: + handlers.append(_JsonFileHandler(_open_log_file(log_file))) + namespace.handlers = handlers + namespace.setLevel( + logging.WARNING if quiet else logging.DEBUG if debug else logging.INFO + ) + namespace.propagate = False + namespace.disabled = False + yield + finally: + namespace.handlers = previous[0] + namespace.setLevel(previous[1]) + namespace.propagate = previous[2] + namespace.disabled = previous[3] + for handler in handlers: + handler.close() + + +def _error_details(error: Exception) -> dict[str, str]: + cases: tuple[tuple[type[Exception], str, str], ...] = ( + ( + Abort, + "interrupted", + "Saved archives remain available; retry when ready.", + ), + ( + click.ClickException, + "invalid_usage", + "Run this command with --help to check arguments and options.", + ), + ( + DiagnosticLogError, + "diagnostic_log_unavailable", + "Choose a writable regular file in an existing local folder.", + ), + ( + AuthenticationRequiredError, + "authentication_required", + "Run the matching mindcap auth command, then retry.", + ), + ( + ProfileLockedError, + "browser_profile_locked", + "Close the dedicated capture browser, then retry.", + ), + ( + StableChromeNotFoundError, + "chrome_not_found", + "Install Google Chrome, then run mindcap doctor for this source.", + ), + ( + VerificationError, + "verification_failed", + "Keep the archive and inspect it before retrying the capture.", + ), + ( + InvalidSourceError, + "invalid_source", + "Check the source identifier and selected plugin.", + ), + ( + CaptureFailedError, + "capture_failed", + "Check source access with mindcap doctor; use --verbose for phase details.", + ), + ( + NormalizationError, + "normalization_failed", + "Keep the original capture and check its supported input format.", + ), + ( + PermissionError, + "permission_denied", + "Check permissions or choose a writable local destination.", + ), + (FileNotFoundError, "path_not_found", "Check that the input path exists."), + ( + FileExistsError, + "path_already_exists", + "Choose a new destination or use the command's documented resume option.", + ), + (ValueError, "invalid_input", "Check the command arguments and input format."), + (OSError, "io_error", "Check local files, available space, and permissions."), + ( + MindcapError, + "mindcap_error", + "Use --verbose for phase details and check the source with mindcap doctor.", + ), + ) + code = "unexpected_error" + hint = "Retry with --debug for sanitized frame locations and share the diagnostics." + for exception_type, candidate_code, candidate_hint in cases: + if isinstance(error, exception_type): + code, hint = candidate_code, candidate_hint + break + return { + "code": code, + "type": redact_text(type(error).__name__), + "message": ( + "Interrupted. Saved archives remain available; " + "no successful capture is implied." + if isinstance(error, Abort) + else redact_text(str(error)) or "The operation could not be completed." + ), + "hint": hint, + } + + +def render_error( + error: Exception, *, json_output: bool = False, debug: bool = False +) -> None: + """Emit a safe error on stderr, with no raw traceback or private locals.""" + details = _error_details(error) + frames = _frame_locations(error.__traceback__) if debug else [] + if any( + isinstance(handler, (_TerminalHandler, _JsonFileHandler)) + for handler in logging.getLogger("mindcap").handlers + ): + logging.getLogger("mindcap.cli").error( + "%s: %s", + details["code"], + details["message"], + extra={ + "progress_event": True, + "event": "command_failed", + "diagnostic_error": details, + "diagnostic_frames": frames, + }, + ) + if json_output: + envelope: dict[str, Any] = {"error": details} + if frames: + envelope["frames"] = frames + print(json.dumps(envelope, ensure_ascii=False), file=sys.stderr) + return + body = Text(details["message"]) + body.append("\n\nNext step: ", style="bold") + body.append(details["hint"]) + if frames: + body.append(f"\n\n{details['type']}\n", style="dim") + body.append("\n".join(frames), style="dim") + _console().print( + Panel( + body, + title=Text(f"Mindcap error · {details['code']}", style="bold bright_red"), + border_style="bright_red", + expand=False, + ) + ) diff --git a/src/mindcap_cli/runtime.py b/src/mindcap_cli/runtime.py new file mode 100644 index 0000000..eda7d0f --- /dev/null +++ b/src/mindcap_cli/runtime.py @@ -0,0 +1,141 @@ +"""Command boundaries: consistent diagnostics without changing library callers.""" + +from __future__ import annotations + +import logging +import time +from collections.abc import Callable +from pathlib import Path +from typing import Any, NoReturn, TypeVar + +import typer +from typer.core import TyperCommand, TyperGroup + +from mindcap_cli._compat import click, get_current_context +from mindcap_cli.logging import DiagnosticLogError, diagnostic_session, render_error + +logger = logging.getLogger("mindcap.cli") +CommandFunction = TypeVar("CommandFunction", bound=Callable[..., Any]) + + +def diagnostic_options(ctx: click.Context | None = None) -> dict[str, Any]: + """Combine global diagnostics with backwards-compatible command options.""" + ctx = ctx or get_current_context(silent=True) + options: dict[str, Any] = { + "verbose": False, + "debug": False, + "quiet": False, + "log_file": None, + "json_output": False, + } + contexts = [] + while ctx is not None: + contexts.append(ctx) + ctx = ctx.parent + for current in reversed(contexts): + for key in ("verbose", "debug", "quiet", "json_output"): + options[key] = options[key] or bool(current.params.get(key)) + if current.params.get("asset_diagnostics"): + options["json_output"] = True + if current.meta.get("mindcap_json_requested"): + options["json_output"] = True + if current.params.get("log_file") is not None: + options["log_file"] = Path(current.params["log_file"]) + return options + + +def fail(error: Exception) -> NoReturn: + options = diagnostic_options() + render_error(error, json_output=options["json_output"], debug=options["debug"]) + raise typer.Exit(1) from error + + +class DiagnosticCommand(TyperCommand): + """Keep all operational failures inside the CLI's safe output boundary.""" + + def parse_args(self, ctx: click.Context, args: list[str]) -> list[str]: + # The token parser does no conversion/callbacks. Remember a requested + # JSON mode even when another option fails conversion before --json. + try: + raw, _, _ = self.make_parser(ctx).parse_args(args=list(args)) + ctx.meta["mindcap_json_requested"] = bool( + raw.get("json_output") or raw.get("asset_diagnostics") + ) + return super().parse_args(ctx, args) + except click.ClickException as error: + _usage_error(error, ctx) + + def invoke(self, ctx: click.Context) -> Any: + options = diagnostic_options(ctx) + for flag in ("verbose", "debug", "quiet"): + if flag in ctx.params: + ctx.params[flag] = options[flag] + started = time.monotonic() + try: + with diagnostic_session( + verbose=options["verbose"], + debug=options["debug"], + quiet=options["quiet"], + log_file=options["log_file"], + ): + logger.debug("Starting %s", ctx.command_path) + try: + result = super().invoke(ctx) + except (KeyboardInterrupt, typer.Abort) as error: + render_error(typer.Abort(), json_output=options["json_output"]) + raise typer.Exit(130) from error + except (typer.Exit, click.ClickException): + raise + except Exception as error: + fail(error) + else: + logger.info( + "Finished %s in %.2fs", + ctx.command_path, + time.monotonic() - started, + extra={"progress_event": True, "event": "command_completed"}, + ) + return result + except (OSError, DiagnosticLogError) as error: + # Includes failure to open the requested diagnostic file. Never let + # setup failures fall through to Typer's traceback renderer. + fail(error) + + +class DiagnosticGroup(TyperGroup): + """Redact nested usage errors and keep their normal usage exit status.""" + + def parse_args(self, ctx: click.Context, args: list[str]) -> list[str]: + try: + return super().parse_args(ctx, args) + except click.ClickException as error: + if isinstance(error, click.exceptions.NoArgsIsHelpError): + raise + _usage_error(error, ctx) + + def invoke(self, ctx: click.Context) -> Any: + try: + return super().invoke(ctx) + except click.ClickException as error: + _usage_error(error, getattr(error, "ctx", None) or ctx) + + +def _usage_error(error: click.ClickException, ctx: click.Context) -> NoReturn: + options = diagnostic_options(ctx) + render_error(error, json_output=options["json_output"], debug=options["debug"]) + raise typer.Exit(error.exit_code) from error + + +class DiagnosticTyper(typer.Typer): + """Give current and future registered commands the same error boundary.""" + + def __init__(self, *args: Any, **kwargs: Any) -> None: + kwargs.setdefault("cls", DiagnosticGroup) + kwargs.setdefault("pretty_exceptions_enable", False) + super().__init__(*args, **kwargs) + + def command( + self, *args: Any, **kwargs: Any + ) -> Callable[[CommandFunction], CommandFunction]: + kwargs.setdefault("cls", DiagnosticCommand) + return super().command(*args, **kwargs) diff --git a/tests/test_cli_diagnostics.py b/tests/test_cli_diagnostics.py new file mode 100644 index 0000000..d378a1d --- /dev/null +++ b/tests/test_cli_diagnostics.py @@ -0,0 +1,172 @@ +"""Real command boundaries, output channels, and failure exit statuses.""" + +from __future__ import annotations + +import json +from datetime import UTC, datetime +from pathlib import Path +from unittest.mock import Mock, patch + +import pytest +import typer +from typer.testing import CliRunner + +from mindcap.core.errors import AuthenticationRequiredError +from mindcap.sync.models import BatchRunConfig, BatchRunState, RunStatus +from mindcap_cli.app import app + + +@pytest.mark.parametrize("json_output", [False, True]) +def test_expected_error_is_actionable_and_never_on_stdout(json_output: bool) -> None: + args = ["capture", "suno", "not-a-workspace"] + if json_output: + args.append("--json") + result = CliRunner().invoke(app, args) + assert result.exit_code == 1 + assert result.stdout == "" + if json_output: + assert json.loads(result.stderr)["error"]["code"] == "invalid_source" + else: + assert "Next step:" in result.stderr + assert "Traceback" not in result.stderr + + +@pytest.mark.parametrize("debug", [False, True]) +def test_unexpected_error_is_caught_without_dumping_locals(debug: bool) -> None: + args = ["--debug"] if debug else [] + with patch( + "mindcap_cli.app.verify_bundle", + side_effect=RuntimeError( + "Failed https://user:private@cdn.test/audio?Signature=secret " + "Authorization: Bearer hidden" + ), + ): + result = CliRunner().invoke(app, [*args, "verify", "/synthetic/archive"]) + assert result.exit_code == 1 + assert "unexpected_error" in result.stderr + assert "Next step:" in result.stderr + for secret in ("private", "secret", "hidden", "Traceback (most recent"): + assert secret not in result.output + assert (" in verify" in result.stderr) == debug + + +def test_auth_error_uses_same_boundary_as_capture() -> None: + with patch( + "mindcap_cli.app.authenticate_distrokid", + side_effect=AuthenticationRequiredError("Session expired"), + ): + result = CliRunner().invoke(app, ["auth", "distrokid"]) + assert result.exit_code == 1 + assert result.stdout == "" + assert "authentication_required" in result.stderr + assert "mindcap auth" in result.stderr + + +@pytest.mark.parametrize("error", [KeyboardInterrupt(), typer.Abort()]) +def test_interruption_is_clean_and_nonzero(error: BaseException) -> None: + with patch("mindcap_cli.app.verify_bundle", side_effect=error): + result = CliRunner().invoke(app, ["verify", "/synthetic/archive"]) + assert result.exit_code == 130 + assert result.stdout == "" + assert "interrupted" in result.stderr + assert "Traceback" not in result.output + + +def test_global_debug_file_applies_to_nested_command(tmp_path: Path) -> None: + destination = tmp_path / "diagnostics.jsonl" + result = CliRunner().invoke( + app, ["--debug", "--log-file", str(destination), "plugins", "list"] + ) + assert result.exit_code == 0, result.output + records = [json.loads(line) for line in destination.read_text().splitlines()] + assert any(record["level"] == "DEBUG" for record in records) + assert any(record.get("event") == "command_completed" for record in records) + assert "Source plugin" in result.stdout + assert "Starting" in result.stderr + + +def test_log_setup_failure_prevents_command_and_has_no_traceback( + tmp_path: Path, +) -> None: + with patch("mindcap_cli.app.verify_bundle") as verify: + result = CliRunner().invoke( + app, + ["--log-file", str(tmp_path / "missing" / "log"), "verify", "/archive"], + ) + assert result.exit_code == 1 + assert "diagnostic_log_unavailable" in result.stderr + assert result.stdout == "" + verify.assert_not_called() + + +def test_global_quiet_reaches_capture_but_preserves_warning(tmp_path: Path) -> None: + plugin = Mock() + plugin.default_strategy.return_value = "api" + plugin.canonicalize.return_value = ("unit-1", None) + plugin.strategy.return_value.capture.return_value = Mock( + safe_metadata={"capture_complete": False}, + warnings=["A required asset could not be downloaded (403)."], + ) + plugin.normalize.return_value = {} + plugin.storage.return_value.persist.return_value = Mock( + status="complete", source_id="unit-1", version=1, path=tmp_path / "v1" + ) + with patch("mindcap_cli.app.build_registry") as registry: + registry.return_value.get.return_value = plugin + result = CliRunner().invoke(app, ["--quiet", "capture", "suno", "unit-1"]) + assert result.exit_code == 2, result.output + assert result.stdout.strip() == str(tmp_path / "v1") + assert "403" in result.stderr + assert "Authenticating" not in result.output + assert "INFO" not in result.stderr + + +def test_sync_json_stdout_excludes_progress_and_resume_advice(tmp_path: Path) -> None: + now = datetime.now(UTC) + config = BatchRunConfig(provider="suno", collection_identifier="suno-account") + state = BatchRunState( + run_id="synthetic-run", + provider="suno", + collection_identifier="suno-account", + config=config, + config_fingerprint=config.fingerprint(), + started_at=now, + updated_at=now, + status=RunStatus.INTERRUPTED, + ) + with patch("mindcap.sync.runner.SyncRunner.run", return_value=state): + result = CliRunner().invoke( + app, ["sync", "suno", "--json", "--output", str(tmp_path)] + ) + assert result.exit_code == 130, result.output + assert json.loads(result.stdout)["run_id"] == "synthetic-run" + assert "Authenticating" in result.stderr + assert "mindcap sync suno --resume" in result.stderr + + +def test_nested_usage_error_keeps_usage_exit_code() -> None: + result = CliRunner().invoke(app, ["inspect", "suno"]) + assert result.exit_code == 2 + assert result.stdout == "" + assert "invalid_usage" in result.stderr + assert "--help" in result.stderr + + +@pytest.mark.parametrize("json_first", [False, True]) +def test_json_usage_error_is_structured_even_before_json_is_converted( + json_first: bool, +) -> None: + args = ["--max-items", "token=private-value"] + args = ["--json", *args] if json_first else [*args, "--json"] + result = CliRunner().invoke(app, ["sync", "suno", *args]) + assert result.exit_code == 2 + assert result.stdout == "" + assert json.loads(result.stderr)["error"]["code"] == "invalid_usage" + assert "private-value" not in result.output + + +def test_top_level_usage_error_uses_safe_boundary() -> None: + result = CliRunner().invoke(app, ["--token=private-value"]) + assert result.exit_code == 2 + assert "invalid_usage" in result.stderr + assert "private-value" not in result.output diff --git a/tests/test_cli_logging.py b/tests/test_cli_logging.py new file mode 100644 index 0000000..35ae80b --- /dev/null +++ b/tests/test_cli_logging.py @@ -0,0 +1,303 @@ +from __future__ import annotations + +import io +import json +import logging +import os +import stat +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from mindcap.core.errors import AuthenticationRequiredError +from mindcap_cli.logging import ( + DiagnosticLogError, + diagnostic_session, + render_error, +) + + +def test_terminal_diagnostics_are_literal_redacted_stderr( + capsys: pytest.CaptureFixture[str], +) -> None: + with diagnostic_session(): + logging.getLogger("mindcap.plugins.suno").warning( + "[bold]HTTP 403[/bold] %s Authorization: Bearer credential-secret", + "https://user:password-secret@cdn.example/audio.mp3?token=signed-secret", + ) + captured = capsys.readouterr() + assert captured.out == "" + assert "WARNING" in captured.err + assert "[bold]HTTP 403[/bold]" in captured.err + assert "cdn.example/audio.mp3" in captured.err + for secret in ("credential-secret", "password-secret", "signed-secret"): + assert secret not in captured.err + + +@pytest.mark.parametrize( + ("options", "expected", "absent"), + [ + ({}, ("info event", "warning event"), ("debug event",)), + ({"verbose": True}, ("info event", "warning event"), ("debug event",)), + ({"debug": True}, ("debug event", "info event", "warning event"), ()), + ({"quiet": True}, ("warning event",), ("debug event", "info event")), + ], +) +def test_session_levels( + options: dict[str, bool], + expected: tuple[str, ...], + absent: tuple[str, ...], + capsys: pytest.CaptureFixture[str], +) -> None: + with diagnostic_session( + verbose=options.get("verbose", False), + debug=options.get("debug", False), + quiet=options.get("quiet", False), + ): + logger = logging.getLogger("mindcap.cli") + logger.debug("debug event") + logger.info("info event") + logger.warning("warning event") + output = capsys.readouterr().err + assert all(message in output for message in expected) + assert all(message not in output for message in absent) + + +def test_progress_events_reach_log_file_without_repeating_terminal( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + path = tmp_path / "capture.jsonl" + with diagnostic_session(log_file=path): + logging.getLogger("mindcap.capture").warning( + "asset warning", + extra={"progress_event": True, "event": "capture_warning"}, + ) + assert capsys.readouterr().err == "" + record = json.loads(path.read_text()) + assert record["message"] == "asset warning" + assert record["event"] == "capture_warning" + + +def test_jsonl_logs_append_with_private_permissions_and_no_ansi(tmp_path: Path) -> None: + path = tmp_path / "capture.jsonl" + path.write_text('{"previous": true}\n') + path.chmod(0o644) + for _ in range(2): + with diagnostic_session(log_file=path): + logging.getLogger("mindcap.cli").info( + "\x1b[31mGET https://cdn.example/a?signature=signed-secret#private " + "Cookie: session=cookie-secret; other=other-secret" + ) + raw = path.read_text() + records = [json.loads(line) for line in raw.splitlines()] + assert len(records) == 3 + assert records[0] == {"previous": True} + assert records[1]["level"] == "INFO" + assert records[1]["logger"] == "mindcap.cli" + assert records[1]["timestamp"].endswith("+00:00") + assert "\x1b" not in raw + for secret in ("signed-secret", "private", "cookie-secret", "other-secret"): + assert secret not in raw + assert stat.S_IMODE(path.stat().st_mode) == 0o600 + + +def test_symlink_log_refused_without_touching_target(tmp_path: Path) -> None: + target = tmp_path / "target.txt" + target.write_text("preserve me") + link = tmp_path / "capture.jsonl" + link.symlink_to(target) + with ( + pytest.raises(DiagnosticLogError, match="symbolic link"), + diagnostic_session(log_file=link), + ): + pytest.fail("Session must refuse a symbolic link") + assert target.read_text() == "preserve me" + + +def test_hardlinked_log_refused_without_touching_target(tmp_path: Path) -> None: + target = tmp_path / "target.txt" + target.write_text("preserve me") + link = tmp_path / "capture.jsonl" + os.link(target, link) + with ( + pytest.raises(DiagnosticLogError, match="without links"), + diagnostic_session(log_file=link), + ): + pytest.fail("Session must refuse a hard link") + assert target.read_text() == "preserve me" + + +@pytest.mark.parametrize("destination", ["missing/capture.jsonl", "."]) +def test_bad_log_destination_fails_concisely(tmp_path: Path, destination: str) -> None: + with ( + pytest.raises(DiagnosticLogError, match="Cannot open diagnostic log"), + diagnostic_session(log_file=tmp_path / destination), + ): + pytest.fail("Session must refuse an invalid destination") + + +def test_fifo_log_destination_fails_without_waiting_for_reader(tmp_path: Path) -> None: + destination = tmp_path / "capture.fifo" + os.mkfifo(destination) + with pytest.raises(DiagnosticLogError), diagnostic_session(log_file=destination): + pytest.fail("Session must refuse a FIFO") + + +@pytest.mark.parametrize("fail", [False, True]) +def test_session_restores_logging_and_closes_files( + monkeypatch: pytest.MonkeyPatch, fail: bool +) -> None: + namespace = logging.getLogger("mindcap") + snapshot = ( + list(namespace.handlers), + namespace.level, + namespace.propagate, + namespace.disabled, + ) + stream = io.StringIO() + monkeypatch.setattr("mindcap_cli.logging._open_log_file", lambda path: stream) + try: + with diagnostic_session(debug=True, log_file=Path("unused.jsonl")): + logging.getLogger("mindcap.capture").debug("within session") + if fail: + raise ValueError("expected test failure") + except ValueError: + assert fail + assert stream.closed + assert ( + list(namespace.handlers), + namespace.level, + namespace.propagate, + namespace.disabled, + ) == snapshot + + +def test_session_leaves_root_and_http_loggers_untouched(tmp_path: Path) -> None: + root = logging.getLogger() + http = logging.getLogger("httpx") + snapshot = ( + list(root.handlers), + root.level, + list(http.handlers), + http.level, + http.propagate, + http.disabled, + ) + path = tmp_path / "capture.jsonl" + with diagnostic_session(debug=True, log_file=path): + logging.getLogger("mindcap.capture").info("safe event") + assert ( + list(root.handlers), + root.level, + list(http.handlers), + http.level, + http.propagate, + http.disabled, + ) == snapshot + assert len(path.read_text().splitlines()) == 1 + + +def test_json_error_uses_stderr_stable_code_and_sanitization( + capsys: pytest.CaptureFixture[str], +) -> None: + render_error( + AuthenticationRequiredError("Cookie: session=private-cookie"), json_output=True + ) + captured = capsys.readouterr() + assert captured.out == "" + payload = json.loads(captured.err) + assert payload["error"]["code"] == "authentication_required" + assert payload["error"]["type"] == "AuthenticationRequiredError" + assert "mindcap auth" in payload["error"]["hint"] + assert "private-cookie" not in captured.err + assert "\x1b" not in captured.err + assert "frames" not in payload + + +def test_debug_error_has_only_sanitized_frame_locations( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + def fail_with_private_local() -> None: + private_capture_body = "private-conversation-body" + assert private_capture_body + raise RuntimeError("Failed https://cdn.example/a?token=signed-secret") + + path = tmp_path / "debug.jsonl" + with diagnostic_session(debug=True, log_file=path): + try: + fail_with_private_local() + except RuntimeError as error: + render_error(error, json_output=True, debug=True) + stderr = capsys.readouterr().err + payload = json.loads(stderr) + assert payload["error"]["type"] == "RuntimeError" + assert any("in fail_with_private_local" in frame for frame in payload["frames"]) + assert all(frame.startswith("test_cli_logging.py:") for frame in payload["frames"]) + for output in (stderr, path.read_text()): + assert "signed-secret" not in output + assert "private-conversation-body" not in output + assert "private_capture_body" not in output + assert "raise RuntimeError" not in output + assert "Traceback" not in output + + +def test_pretty_error_logged_once_without_markup_interpretation( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + path = tmp_path / "error.jsonl" + with diagnostic_session(log_file=path): + render_error(ValueError("[bold]bad input[/bold]")) + stderr = capsys.readouterr().err + assert stderr.count("[bold]bad input[/bold]") == 1 + assert "Mindcap error" in stderr + assert "invalid_input" in stderr + assert "Next step" in stderr + records = [json.loads(line) for line in path.read_text().splitlines()] + assert len(records) == 1 + assert records[0]["error"]["code"] == "invalid_input" + + +def test_bad_log_message_does_not_emit_logging_internal_traceback( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + path = tmp_path / "error.jsonl" + with diagnostic_session(log_file=path): + logging.getLogger("mindcap.cli").error("bad %s %s", "secret-argument") + output = capsys.readouterr().err + path.read_text() + assert "could not be formatted" in output + assert "secret-argument" not in output + assert "Traceback" not in output + assert "Logging error" not in output + + +def test_repeated_cli_invocations_restore_handlers(tmp_path: Path) -> None: + from mindcap_cli.app import app + + namespace = logging.getLogger("mindcap") + snapshot = (list(namespace.handlers), namespace.level, namespace.propagate) + runner = CliRunner() + path = tmp_path / "commands.jsonl" + for _ in range(2): + result = runner.invoke(app, ["--log-file", str(path), "plugins", "list"]) + assert result.exit_code == 0, result.output + current = (list(namespace.handlers), namespace.level, namespace.propagate) + assert current == snapshot + assert path.exists() + + +def test_interrupted_and_usage_errors_have_actionable_codes( + capsys: pytest.CaptureFixture[str], +) -> None: + import typer + + from mindcap_cli._compat import click + + render_error(typer.Abort(), json_output=True) + interrupted = json.loads(capsys.readouterr().err)["error"] + assert interrupted["code"] == "interrupted" + assert "Saved archives remain available" in interrupted["message"] + render_error(click.exceptions.UsageError("Invalid option"), json_output=True) + usage = json.loads(capsys.readouterr().err)["error"] + assert usage["code"] == "invalid_usage" + assert "--help" in usage["hint"] diff --git a/tests/test_diagnostics.py b/tests/test_diagnostics.py new file mode 100644 index 0000000..64b36d6 --- /dev/null +++ b/tests/test_diagnostics.py @@ -0,0 +1,129 @@ +"""Credential and terminal-injection coverage for shared diagnostics.""" + +from __future__ import annotations + +import pytest + +from mindcap.core.diagnostics import redact_text + + +def test_named_session_cookies_are_redacted_outside_cookie_header() -> None: + text = redact_text("__client=private-value __clerk_db_jwt=another-value") + assert "private-value" not in text + assert "another-value" not in text + + +@pytest.mark.parametrize( + "value,secrets", + [ + ( + "HTTP 403 https://alice:password@cdn.example/song.mp3" + "?signature=signed-secret&expires=42#private-fragment", + ["alice", "password", "signed-secret", "private-fragment", "expires"], + ), + ( + "Authorization: Bearer auth-secret\n" + "Cookie: first=cookie-secret; __session=session-secret\n" + "HTTP 403 at cdn.example", + ["auth-secret", "cookie-secret", "session-secret"], + ), + ( + "headers={'Authorization': 'Bearer auth-secret', " + "'Cookie': 'first=cookie-secret; second=another-cookie', " + "'X-API-Key': 'key-secret'} HTTP 403", + ["auth-secret", "cookie-secret", "another-cookie", "key-secret"], + ), + ( + 'password="password with spaces" token=access-secret ' + "session_id=session-secret refresh_token=refresh-secret " + "client_secret=client-secret HTTP 403", + [ + "password with spaces", + "access-secret", + "session-secret", + "refresh-secret", + "client-secret", + ], + ), + ( + "Bearer bearer-secret HTTP 403 Basic dXNlcjpwYXNz", + ["bearer-secret", "dXNlcjpwYXNz"], + ), + ( + "HTTP 403 token eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.signature_value", + ["eyJhbGciOiJIUzI1NiJ9", "eyJzdWIiOiIxIn0", "signature_value"], + ), + ( + 'HTTP 403 command --password "private password" --api-key private-key', + ["private password", "private-key"], + ), + ], +) +def test_credentials_are_redacted_but_status_remains( + value: str, secrets: list[str] +) -> None: + safe = redact_text(value) + for secret in secrets: + assert secret not in safe + assert "403" in safe + assert "[REDACTED]" in safe + assert redact_text(safe) == safe + + +def test_url_redaction_preserves_public_host_path_port_and_punctuation() -> None: + safe = redact_text( + "Fetch (https://user:secret@cdn.example:8443/song.mp3?t=secret)." + ) + assert safe == "Fetch (https://cdn.example:8443/song.mp3?[REDACTED])." + assert redact_text(safe) == safe + + +def test_ipv6_and_invalid_urls_are_safe() -> None: + assert redact_text("https://user:secret@[::1]:80/audio?key=value") == ( + "https://[::1]:80/audio?[REDACTED]" + ) + assert "private-secret" not in redact_text( + "https://user:private-secret@[invalid/audio" + ) + + +def test_terminal_controls_and_hyperlinks_cannot_inject_events() -> None: + safe = redact_text( + "\x1b[31mHTTP 403\x1b[0m\r\nFORGED\x00\u202e" + "\x1b]8;;https://private-secret\x07label\x1b]8;;\x07" + ) + assert safe == "HTTP 403 FORGED label" + assert "private-secret" not in safe + + +def test_useful_plain_messages_and_literal_markup_are_preserved() -> None: + value = "Asset download failed (audio): HTTP 403 [red]provider text[/red]" + assert redact_text(value) == value + + +@pytest.mark.parametrize( + "value", + [ + 'HTTP 403 cookies={"name": "private", "value": "secret", "nested": [1, 2]}', + "HTTP 403 'cookie': {'name': 'private', 'value': 'secret'}", + 'HTTP 403 storage_state={"cookies": [{"value": "secret"}]}', + 'HTTP 403 auth={"username": "private", "password": "secret"}', + 'HTTP 403 cookies=[{"value": "secret"}]', + 'HTTP 403 cookies={"value": "secret"', + ], +) +def test_nested_or_malformed_auth_payloads_are_omitted(value: str) -> None: + safe = redact_text(value) + assert "secret" not in safe + assert "private" not in safe + assert "403" in safe + assert redact_text(safe) == safe + + +def test_inline_digest_auth_omits_entire_header_payload() -> None: + safe = redact_text( + 'HTTP 403 Authorization: Digest username="private", nonce=secret' + ) + assert "private" not in safe + assert "secret" not in safe + assert "403" in safe diff --git a/tests/test_progress.py b/tests/test_progress.py index 6ce91e1..a4da6c8 100644 --- a/tests/test_progress.py +++ b/tests/test_progress.py @@ -5,6 +5,7 @@ import time from io import StringIO +import pytest from rich.console import Console from mindcap.core.progress import CaptureProgressReporter, CaptureStats @@ -163,3 +164,193 @@ def test_summary_quiet_suppresses(self) -> None: stats = CaptureStats() reporter.summary(stats) assert buf.getvalue() == "" + + def test_partial_capture_distinguishes_integrity_from_completeness(self) -> None: + buf = StringIO() + reporter = CaptureProgressReporter( + console=Console(file=buf, highlight=False, markup=True, width=100) + ) + reporter.summary( + CaptureStats( + clips_archived=36, + audio_downloaded=0, + audio_expected=36, + capture_complete=False, + verification_passed=True, + ) + ) + + output = buf.getvalue() + assert "Capture Partial" in output + assert "Capture Complete" not in output + assert "0 / 36" in output + assert "Capture completeness Partial" in " ".join(output.split()) + assert "Archive integrity PASS" in " ".join(output.split()) + + def test_complete_capture_summary_keeps_default_stats_compatible(self) -> None: + buf = StringIO() + reporter = CaptureProgressReporter( + console=Console(file=buf, highlight=False, markup=True, width=100) + ) + reporter.summary(CaptureStats(audio_downloaded=1)) + + output = buf.getvalue() + assert "Capture Complete" in output + assert "Capture completeness Complete" in " ".join(output.split()) + assert "Archive integrity PASS" in " ".join(output.split()) + assert " / " not in output + + +def test_non_tty_progress_reports_attempted_count_and_elapsed() -> None: + output = StringIO() + reporter = CaptureProgressReporter(console=Console(file=output, width=140)) + with reporter.progress_bar("Audio", 36) as bar: + bar.advance(5) + lines = output.getvalue().splitlines() + assert len(lines) == 2 + assert "Audio (0 / 36)" in lines[0] + assert "Audio: finished; attempted 5 / 36; elapsed" in lines[1] + assert "completed" not in output.getvalue().lower() + + +@pytest.mark.parametrize("error", [RuntimeError("failed"), KeyboardInterrupt()]) +def test_interrupted_progress_does_not_report_success(error: BaseException) -> None: + output = StringIO() + reporter = CaptureProgressReporter(console=Console(file=output, width=140)) + with pytest.raises(type(error)), reporter.progress_bar("Audio", 36) as bar: + bar.advance(2) + raise error + assert "interrupted; attempted 2 / 36; elapsed" in output.getvalue() + assert "finished" not in output.getvalue() + assert "ERROR" in output.getvalue() + + +def test_interrupted_spinner_does_not_report_success() -> None: + output = StringIO() + reporter = CaptureProgressReporter(console=Console(file=output, width=140)) + with pytest.raises(RuntimeError), reporter.spinner("Authenticating"): + raise RuntimeError("failed") + assert "Authenticating: interrupted; elapsed" in output.getvalue() + assert "finished" not in output.getvalue() + + +def test_repeated_asset_warnings_group_and_preserve_every_event( + caplog: pytest.LogCaptureFixture, +) -> None: + output = StringIO() + reporter = CaptureProgressReporter(console=Console(file=output, width=180)) + warnings = [ + f"Asset download failed for {index:08x}-1234-5678-9012-123456789012 " + "(audio): Failed to download asset (403)." + for index in range(36) + ] + with caplog.at_level("INFO", logger="mindcap.progress"): + for warning in warnings: + reporter.warn(warning) + reporter.report_warnings(warnings) + reporter.summary(CaptureStats(capture_complete=False, warnings=warnings)) + text = output.getvalue() + assert "36x Asset download failed for (audio)" in text + assert "36 warning(s) recorded." in text + assert text.count("Asset download failed") == 2 + warning_records = [ + record + for record in caplog.records + if record.message.startswith("Asset download failed") + ] + assert len(warning_records) == 36 + assert all(getattr(record, "progress_event", False) for record in caplog.records) + + +def test_warning_groups_keep_different_statuses_and_verbose_details() -> None: + output = StringIO() + reporter = CaptureProgressReporter( + console=Console(file=output, width=160), verbose=True + ) + reporter.warn("Asset failure 00000001-1234-5678-9012-123456789012 HTTP 403") + reporter.warn("Asset failure 00000002-1234-5678-9012-123456789012 HTTP 403") + reporter.warn("Asset failure 00000003-1234-5678-9012-123456789012 HTTP 429") + reporter.summary(CaptureStats()) + assert output.getvalue().count("Asset failure") == 3 + assert "3 warning(s) recorded." in output.getvalue() + assert "x Asset failure" not in output.getvalue() + + +def test_warning_reconciliation_supports_providers_returning_only_warnings() -> None: + output = StringIO() + reporter = CaptureProgressReporter( + console=Console(file=output, width=160), quiet=True + ) + reporter.warn("Same warning") + reporter.report_warnings(["Same warning", "Same warning", "Another warning"]) + reporter.report_warnings(["Same warning", "Same warning", "Another warning"]) + assert reporter._warning_count == 3 + assert output.getvalue().count("WARNING Same warning") == 1 + assert output.getvalue().count("2x Same warning") == 1 + assert "Another warning" in output.getvalue() + + +def test_warnings_with_no_console_remain_completely_silent( + caplog: pytest.LogCaptureFixture, capsys: pytest.CaptureFixture[str] +) -> None: + reporter = CaptureProgressReporter() + with caplog.at_level("DEBUG", logger="mindcap.progress"): + reporter.warn("A warning") + reporter.phase("A phase") + reporter.report_warnings(["A warning", "A warning"]) + reporter.summary(CaptureStats(warnings=["A warning"])) + assert not caplog.records + assert capsys.readouterr() == ("", "") + + +def test_summary_keeps_long_archive_path_at_narrow_width() -> None: + output = StringIO() + archive = "/Users/example/mindcap/.cache/workspaces/suno/synthetic-workspace/v2" + reporter = CaptureProgressReporter(console=Console(file=output, width=60)) + reporter.summary(CaptureStats(), archive_path=archive) + assert archive in "".join(output.getvalue().split()) + assert "…" not in output.getvalue() + + +def test_warning_reconciliation_is_per_capture_but_groups_span_run() -> None: + output = StringIO() + reporter = CaptureProgressReporter(console=Console(file=output, width=100)) + for _ in range(2): + reporter.begin_capture() + reporter.report_warnings(["Required attachment unavailable."]) + reporter.report_warnings(["Required attachment unavailable."]) + assert reporter._warning_count == 2 + assert "2x Required attachment unavailable." in output.getvalue() + + +def test_external_markup_and_credentials_are_literal_and_redacted( + caplog: pytest.LogCaptureFixture, +) -> None: + output = StringIO() + reporter = CaptureProgressReporter(console=Console(file=output, width=160)) + malicious = ( + "[bold red]literal[/bold red] https://user:secret@cdn.example/audio?t=token" + ) + with caplog.at_level("INFO", logger="mindcap.progress"): + reporter.phase(malicious) + reporter.summary(CaptureStats(), project_name=malicious, archive_path=malicious) + assert "[bold red]literal[/bold red]" in output.getvalue() + assert "cdn.example/audio" in output.getvalue() + assert "user:secret" not in output.getvalue() + assert "t=token" not in output.getvalue() + assert all("user:secret" not in record.message for record in caplog.records) + assert all("t=token" not in record.message for record in caplog.records) + + +def test_summary_can_be_routed_separately_from_progress() -> None: + diagnostics, result = StringIO(), StringIO() + reporter = CaptureProgressReporter( + console=Console(file=diagnostics, width=140), + summary_console=Console(file=result, width=140), + ) + reporter.phase("Downloading audio") + reporter.summary(CaptureStats(), project_name="Test project") + assert "Downloading audio" in diagnostics.getvalue() + assert "Capture Complete" not in diagnostics.getvalue() + assert "Capture Complete" in result.getvalue() + assert "Downloading audio" not in result.getvalue() diff --git a/tests/test_suno_client.py b/tests/test_suno_client.py index f634f90..98db6a3 100644 --- a/tests/test_suno_client.py +++ b/tests/test_suno_client.py @@ -2,12 +2,16 @@ import base64 import json +import shutil +from collections.abc import Iterator from datetime import UTC, datetime, timedelta +from pathlib import Path from typing import Any import httpx import pytest +from mindcap.core.errors import CaptureFailedError from mindcap.plugins.suno.auth import SunoAuthState from mindcap.plugins.suno.client import SunoClient from mindcap.plugins.suno.errors import SunoApiError @@ -103,3 +107,170 @@ def handler(request: httpx.Request) -> httpx.Response: "/api/project/workspace-123/", ] assert "/api/workspaces/workspace-123/" in attempted_paths + + +def test_media_download_omits_api_credentials_and_cookie_jars() -> None: + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + forbidden = { + "authorization", + "cookie", + "x-browser-token", + "x-device-id", + "origin", + "referer", + } + if forbidden & set(request.headers): + return httpx.Response(403, content=b"not allowed") + return httpx.Response( + 200, headers={"content-type": "audio/mpeg"}, content=b"audio" + ) + + client = SunoClient(state=_state(), transport=httpx.MockTransport(handler)) + client._client.cookies.set("api-cookie", "secret", domain="suno.com") + client._media_client.cookies.set("media-cookie", "secret", domain="suno.com") + url = "https://cdn.suno.com/audio.mp3?signature=private-signature&expires=100" + path, metadata = client.download_to_file(url) + try: + assert path.read_bytes() == b"audio" + assert str(requests[0].url) == url + assert metadata["source_url"] == "https://cdn.suno.com/audio.mp3" + assert metadata["byte_size"] == 5 + assert metadata["media_type"] == "audio/mpeg" + assert "private-signature" not in str(metadata) + finally: + shutil.rmtree(path.parent) + + +@pytest.mark.parametrize( + "asset_url", + [ + "https://studio-api-prod.suno.com/asset.mp3", + "/asset.mp3", + ], +) +def test_api_origin_asset_keeps_api_authentication(asset_url: str) -> None: + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response(200, content=b"audio") + + state = _state() + client = SunoClient(state=state, transport=httpx.MockTransport(handler)) + path, _ = client.download_to_file(asset_url) + try: + assert requests[0].headers["authorization"] == f"Bearer {state.jwt}" + assert requests[0].headers["cookie"] == state.cookie_header + assert "x-browser-token" in requests[0].headers + finally: + shutil.rmtree(path.parent) + + +@pytest.mark.parametrize( + "asset_url", + [ + "https://studio-api-prod.suno.com:444/asset.mp3", + "http://studio-api-prod.suno.com/asset.mp3", + "https://studio-api-prod.suno.com.evil.test/asset.mp3", + ], +) +def test_similar_asset_origin_never_gets_api_authentication(asset_url: str) -> None: + def handler(request: httpx.Request) -> httpx.Response: + assert "authorization" not in request.headers + assert "cookie" not in request.headers + assert "x-browser-token" not in request.headers + return httpx.Response(200, content=b"audio") + + client = SunoClient(state=_state(), transport=httpx.MockTransport(handler)) + path, _ = client.download_to_file(asset_url) + shutil.rmtree(path.parent) + + +def test_media_redirect_drops_auth_and_never_restores_it() -> None: + requests: list[httpx.Request] = [] + urls = [ + "https://studio-api-prod.suno.com/asset.mp3", + "https://cdn.suno.com/asset.mp3?signature=secret", + "https://studio-api-prod.suno.com/final.mp3", + ] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + if len(requests) < 3: + return httpx.Response( + 302, + headers={ + "location": urls[len(requests)], + "set-cookie": "redirect-cookie=secret; Domain=.suno.com; Path=/", + }, + ) + return httpx.Response(200, content=b"audio") + + client = SunoClient(state=_state(), transport=httpx.MockTransport(handler)) + path, _ = client.download_to_file(urls[0]) + try: + assert [str(request.url) for request in requests] == urls + assert "authorization" in requests[0].headers + for request in requests[1:]: + assert "authorization" not in request.headers + assert "cookie" not in request.headers + assert "x-browser-token" not in request.headers + assert "x-device-id" not in request.headers + finally: + shutil.rmtree(path.parent) + + +def test_media_403_is_actionable_and_removes_temporary_directory( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + temporary = tmp_path / "download" + temporary.mkdir() + monkeypatch.setattr( + "mindcap.plugins.suno.client.tempfile.mkdtemp", lambda **kwargs: str(temporary) + ) + + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(403, content=b"body containing private-provider-data") + + client = SunoClient(state=_state(), transport=httpx.MockTransport(handler)) + with pytest.raises(CaptureFailedError) as error: + client.download_to_file("https://cdn.suno.com/audio.mp3?signature=private") + message = str(error.value) + assert "403" in message + assert "Download works" in message + assert "private" not in message + assert "cdn.suno.com" not in message + assert not temporary.exists() + + +@pytest.mark.parametrize("failure", ["empty", "stream", "redirect-loop"]) +def test_failed_media_transfer_removes_partial_files( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, failure: str +) -> None: + temporary = tmp_path / "download" + temporary.mkdir() + monkeypatch.setattr( + "mindcap.plugins.suno.client.tempfile.mkdtemp", lambda **kwargs: str(temporary) + ) + + class BrokenStream(httpx.SyncByteStream): + def __iter__(self) -> Iterator[bytes]: + yield b"partial audio" + raise httpx.ReadError("private-provider-data") + + def handler(request: httpx.Request) -> httpx.Response: + if failure == "stream": + return httpx.Response(200, stream=BrokenStream()) + if failure == "redirect-loop": + return httpx.Response(302, headers={"location": "/audio.mp3"}) + return httpx.Response(200, content=b"") + + client = SunoClient(state=_state(), transport=httpx.MockTransport(handler)) + with pytest.raises(CaptureFailedError) as error: + client.download_to_file("https://cdn.suno.com/audio.mp3?signature=secret") + assert "private-provider-data" not in str(error.value) + assert "signature=secret" not in str(error.value) + assert not temporary.exists() diff --git a/tests/test_suno_inspect.py b/tests/test_suno_inspect.py index 770d973..1bea93b 100644 --- a/tests/test_suno_inspect.py +++ b/tests/test_suno_inspect.py @@ -10,6 +10,8 @@ from typer.testing import CliRunner from mindcap.core.errors import VerificationError +from mindcap.core.hashing import sha256_bytes +from mindcap.plugins.suno.archive.diagnostics import suno_asset_diagnostics from mindcap.plugins.suno.archive.inspector import inspect_suno_archive from mindcap_cli.app import app @@ -88,6 +90,198 @@ def _make_minimal_bundle(root: Path) -> Path: return bundle +def _replace_metadata(bundle: Path, metadata: object) -> None: + relative = "workspace/metadata.json" + path = bundle / relative + path.write_text(json.dumps(metadata), encoding="utf-8") + checksums_path = bundle / "checksums.json" + checksums = json.loads(checksums_path.read_text(encoding="utf-8")) + for entry in checksums["files"]: + if entry["path"] == relative: + entry["sha256"] = sha256_bytes(path.read_bytes()) + entry["byte_size"] = path.stat().st_size + checksums_path.write_text(json.dumps(checksums), encoding="utf-8") + + +def test_offline_audio_diagnostics_only_emits_counts(tmp_path: Path) -> None: + bundle = _make_minimal_bundle(tmp_path) + marker = "private-credential-marker" + _replace_metadata( + bundle, + { + "title": marker, + "provider_metadata": {"cookie": marker}, + "clips": [ + { + "clip_id": marker, + "audio_url": f"https://cdn1.suno.ai/{marker}.mp3?Signature={marker}", + "archived_assets": [ + {"asset_type": "audio", "capture_status": "verified"} + ], + }, + { + "audio_url": f"https://studio-api.prod.suno.com/api/forbidden?token={marker}" + }, + {"audio_url": "https://studio-api-prod.suno.com/api/forbidden/"}, + {"audio_url": "https://studio-api-prod.suno.com/api/download"}, + {"audio_url": f"https://{marker}.example/audio.mp3"}, + {"audio_url": "https://[invalid"}, + {}, + ], + }, + ) + with ( + patch("httpx.Client.request", side_effect=AssertionError("network")), + patch( + "mindcap.plugins.suno.client.load_suno_auth_state", + side_effect=AssertionError("auth"), + ), + ): + result = CliRunner().invoke( + app, ["inspect", "suno", str(bundle), "--asset-diagnostics"] + ) + assert result.exit_code == 0, result.output + data = json.loads(result.stdout) + assert data == { + "schema": "mindcap.suno-asset-diagnostics/v0.1", + "offline": True, + "clip_count": 7, + "clips_with_archived_primary_audio": 1, + "audio_url_kinds": { + "missing": 1, + "invalid": 1, + "forbidden_endpoint": 2, + "suno_api": 1, + "suno_cdn": 1, + "other_http": 1, + }, + "audio_urls_with_query": 2, + "audio_urls_with_signature_keys": 1, + } + assert marker not in result.output + assert str(bundle) not in result.output + assert "https://" not in result.output + + +def test_audio_diagnostics_handles_legacy_metadata_fallback(tmp_path: Path) -> None: + bundle = _make_minimal_bundle(tmp_path) + _replace_metadata( + bundle, + { + "clips": [ + { + "provider_metadata": { + "metadata": {"audio_url": "https://cdn1.suno.ai/track.mp3"} + } + }, + {"provider_metadata": None}, + {"audio_url": 5}, + ] + }, + ) + data = suno_asset_diagnostics(bundle) + assert data["audio_url_kinds"]["suno_cdn"] == 1 + assert data["audio_url_kinds"]["missing"] == 1 + assert data["audio_url_kinds"]["invalid"] == 1 + + +@pytest.mark.parametrize("metadata", [None, [], {"clips": {}}, {"clips": [None]}]) +def test_audio_diagnostics_rejects_invalid_metadata_without_echo( + tmp_path: Path, metadata: object +) -> None: + bundle = _make_minimal_bundle(tmp_path) + _replace_metadata(bundle, metadata) + result = CliRunner().invoke( + app, ["inspect", "suno", str(bundle), "--asset-diagnostics"] + ) + assert result.exit_code == 1 + assert "could not be verified" in result.output + assert str(bundle) not in result.output + + +def test_audio_diagnostics_verifies_saved_files(tmp_path: Path) -> None: + bundle = _make_minimal_bundle(tmp_path) + _replace_metadata(bundle, {"clips": []}) + (bundle / "clips/clip-alpha/audio/original.mp3").write_bytes(b"tampered") + with pytest.raises(VerificationError, match="could not be verified"): + suno_asset_diagnostics(bundle) + + +def test_audio_diagnostics_handles_malformed_checksum_document(tmp_path: Path) -> None: + bundle = _make_minimal_bundle(tmp_path) + (bundle / "checksums.json").write_text("[]", encoding="utf-8") + result = CliRunner().invoke( + app, ["inspect", "suno", str(bundle), "--asset-diagnostics"] + ) + assert result.exit_code == 1 + assert "could not be verified" in result.output + + +@pytest.mark.parametrize("substitute", [False, True]) +def test_audio_diagnostics_requires_checked_canonical_metadata( + tmp_path: Path, substitute: bool +) -> None: + bundle = _make_minimal_bundle(tmp_path) + _replace_metadata(bundle, {"clips": []}) + if substitute: + manifest_path = bundle / "manifest.json" + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest["workspace_metadata_path"] = "reports/capture-report.json" + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + else: + checksums_path = bundle / "checksums.json" + checksums = json.loads(checksums_path.read_text(encoding="utf-8")) + checksums["files"] = [ + entry + for entry in checksums["files"] + if entry["path"] != "workspace/metadata.json" + ] + checksums_path.write_text(json.dumps(checksums), encoding="utf-8") + with pytest.raises(VerificationError, match="could not be verified"): + suno_asset_diagnostics(bundle) + + +def test_audio_diagnostics_validates_urls_and_malformed_asset_status( + tmp_path: Path, +) -> None: + bundle = _make_minimal_bundle(tmp_path) + _replace_metadata( + bundle, + { + "clips": [ + { + "audio_url": "https://cdn1.suno.ai:bad/audio.mp3", + "archived_assets": [{"asset_type": "audio", "capture_status": {}}], + }, + {"audio_url": "https://user:secret@cdn1.suno.ai/audio.mp3"}, + {"audio_url": "https://cdn1.suno.ai:65536/audio.mp3"}, + ] + }, + ) + data = suno_asset_diagnostics(bundle) + assert data["clips_with_archived_primary_audio"] == 0 + assert data["audio_url_kinds"]["invalid"] == 3 + + +def test_legacy_inspection_does_not_claim_capture_complete(tmp_path: Path) -> None: + bundle = _make_minimal_bundle(tmp_path) + result = CliRunner().invoke(app, ["inspect", "suno", str(bundle)]) + assert result.exit_code == 0 + assert "unknown (legacy archive)" in result.stdout + + +def test_inspection_reports_saved_partial_completeness(tmp_path: Path) -> None: + bundle = _make_minimal_bundle(tmp_path) + manifest_path = bundle / "manifest.json" + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest["capture_completeness"] = "partial" + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + result = CliRunner().invoke(app, ["inspect", "suno", str(bundle)]) + assert result.exit_code == 0 + assert "Capture completeness" in result.stdout + assert "partial" in result.stdout + + # --------------------------------------------------------------------------- # inspect_suno_archive unit tests # --------------------------------------------------------------------------- diff --git a/tests/test_suno_pipeline.py b/tests/test_suno_pipeline.py index 8701794..5d41bcd 100644 --- a/tests/test_suno_pipeline.py +++ b/tests/test_suno_pipeline.py @@ -5,6 +5,7 @@ from pathlib import Path from unittest.mock import patch +import pytest from typer.testing import CliRunner from mindcap.core.models import ( @@ -141,6 +142,97 @@ def test_plugins_list_includes_suno() -> None: assert "api" in result.stdout +def test_partial_media_counts_survive_verified_archive_persistence( + tmp_path: Path, +) -> None: + plugin = SunoPlugin() + envelope = _make_envelope(tmp_path) + envelope.assets = [ + asset for asset in envelope.assets if asset.asset_type != "audio" + ] + counts = { + "audio_count": 0, + "audio_expected_count": 1, + "audio_failed_count": 1, + "asset_expected_count": 2, + "asset_downloaded_count": 1, + "asset_failed_count": 1, + } + envelope.safe_metadata.update({"capture_complete": False, **counts}) + envelope.warnings.append( + "Asset download failed for clip-alpha (audio): " + "Failed to download Suno asset (403)." + ) + normalized = plugin.normalize(envelope, WORKSPACE_ID) + stored = plugin.storage().persist( + _request(tmp_path), envelope, normalized, plugin.render(normalized) + ) + + # A valid archive of a partial capture passes integrity verification. + assert stored.status == "complete" + plugin.storage().verify(stored.path) + for relative_path in ( + "workspace/metadata.json", + "manifest.json", + "reports/capture-report.json", + ): + artifact = json.loads((stored.path / relative_path).read_text(encoding="utf-8")) + assert artifact["capture_complete"] is False + assert artifact["capture_completeness"] == "partial" + assert {key: artifact[key] for key in counts} == counts + report = json.loads( + (stored.path / "reports/capture-report.json").read_text(encoding="utf-8") + ) + assert report["status"] == "partial" + report_markdown = (stored.path / "reports/capture-report.md").read_text( + encoding="utf-8" + ) + assert "Capture completeness: partial" in report_markdown + assert "Primary audio archived: 0 / 1 (1 failed)" in report_markdown + assert "Advertised assets archived: 1 / 2 (1 failed)" in report_markdown + + +@pytest.mark.parametrize("json_output", [False, True]) +def test_cli_reports_partial_capture_with_valid_archive( + tmp_path: Path, json_output: bool +) -> None: + envelope = _make_envelope(tmp_path) + envelope.assets = [ + asset for asset in envelope.assets if asset.asset_type != "audio" + ] + envelope.safe_metadata.update( + { + "clip_count": 2, + "audio_count": 0, + "audio_expected_count": 1, + "audio_failed_count": 1, + "capture_complete": False, + } + ) + envelope.warnings = ["Primary audio download failed (403)."] + args = ["capture", "suno", WORKSPACE_ID, "--output", str(tmp_path / "archive")] + if json_output: + args.append("--json") + with patch( + "mindcap.plugins.suno.archive.service.SunoWorkspaceCaptureService.capture", + return_value=envelope, + ): + result = CliRunner().invoke(app, args) + assert result.exit_code == 2, result.output + if json_output: + data = json.loads(result.stdout) + assert data["status"] == "complete" + assert data["capture_complete"] is False + assert data["capture_completeness"] == "partial" + assert "Authenticating" in result.stderr + else: + output = " ".join(result.stdout.split()) + assert "Capture Partial" in output + assert "Audio 0 / 1" in output + assert "Capture completeness Partial" in output + assert "Archive integrity PASS" in output + + def test_capture_defaults_to_plugin_strategy_for_suno(tmp_path: Path) -> None: runner = CliRunner() envelope = _make_envelope(tmp_path) diff --git a/tests/test_suno_production_schema.py b/tests/test_suno_production_schema.py index 64a1f3b..21b30b8 100644 --- a/tests/test_suno_production_schema.py +++ b/tests/test_suno_production_schema.py @@ -12,6 +12,7 @@ from typing import Any import httpx +import pytest from mindcap.config import suno_api_origin from mindcap.core.models import CaptureEnvelope, RawResponseUnit @@ -672,7 +673,10 @@ def handler(request: httpx.Request) -> httpx.Response: assert any("Incomplete capture" in w for w in envelope.warnings) -def test_complete_capture_has_no_incomplete_warning(tmp_path: Path) -> None: +@pytest.mark.parametrize("asset_status", [403, 404]) +def test_all_failed_assets_make_capture_partial( + tmp_path: Path, asset_status: int +) -> None: from mindcap.core.models import CaptureRequest from mindcap.plugins.suno.archive.downloader import SunoAssetDownloader @@ -687,6 +691,8 @@ def handler(request: httpx.Request) -> httpx.Response: headers={"content-type": "application/json"}, content=json.dumps(project).encode(), ) + if request.url.host == "cdn.example.test": + return httpx.Response(asset_status, request=request, content=b"denied") return httpx.Response(404, request=request, content=b"{}") client = SunoClient(state=_state(), transport=httpx.MockTransport(handler)) @@ -704,10 +710,116 @@ def handler(request: httpx.Request) -> httpx.Response: ) envelope = service.capture(request) - assert envelope.safe_metadata["capture_complete"] is True + assert envelope.safe_metadata["capture_complete"] is False + assert envelope.safe_metadata["audio_count"] == 0 + assert envelope.safe_metadata["audio_expected_count"] == 3 + assert envelope.safe_metadata["audio_failed_count"] == 3 + assert envelope.safe_metadata["asset_expected_count"] == 9 + assert envelope.safe_metadata["asset_downloaded_count"] == 0 + assert envelope.safe_metadata["asset_failed_count"] == 9 + assert len(envelope.warnings) == 9 + # All clip metadata was returned; incompleteness is in the media. assert not any("Incomplete capture" in w for w in envelope.warnings) +@pytest.mark.parametrize( + "failed_asset", + [None, "audio.mp3", "cover.jpg", "video.mp4", "variant.m4a"], +) +def test_capture_completeness_counts_all_advertised_asset_types( + tmp_path: Path, failed_asset: str | None +) -> None: + from mindcap.core.models import CaptureRequest + + project = { + "id": PROJECT_ID, + "clip_count": 1, + "clips": [ + { + "id": "clip-1", + "audio_url": "https://cdn.example.test/audio.mp3", + "image_url": "https://cdn.example.test/cover.jpg", + "video_url": "https://cdn.example.test/video.mp4", + "media_urls": [ + { + "url": "https://cdn.example.test/variant.m4a", + "encoding": "m4a", + "content_type": "audio/mp4", + } + ], + } + ], + } + + def handler(request: httpx.Request) -> httpx.Response: + if request.url.host == "cdn.example.test": + status = 403 if request.url.path == f"/{failed_asset}" else 200 + return httpx.Response(status, request=request, content=b"synthetic media") + if request.url.path == f"/api/project/{PROJECT_ID}": + return httpx.Response(200, request=request, json=project) + return httpx.Response(404, request=request, json={}) + + client = SunoClient(state=_state(), transport=httpx.MockTransport(handler)) + service = SunoWorkspaceCaptureService(client=client) + envelope = service.capture( + CaptureRequest( + source_type="suno", + source=PROJECT_ID, + provider="suno", + canonical_identifier=PROJECT_ID, + canonical_url=None, + strategy="api", + artifact_root=tmp_path, + ) + ) + + metadata = envelope.safe_metadata + assert metadata["capture_complete"] is (failed_asset is None) + assert metadata["audio_expected_count"] == 1 + assert metadata["audio_count"] == (0 if failed_asset == "audio.mp3" else 1) + assert metadata["audio_failed_count"] == (1 if failed_asset == "audio.mp3" else 0) + assert metadata["asset_expected_count"] == 4 + assert metadata["asset_downloaded_count"] == (4 if failed_asset is None else 3) + assert metadata["asset_failed_count"] == (0 if failed_asset is None else 1) + assert len(envelope.warnings) == (0 if failed_asset is None else 1) + for asset in envelope.assets: + asset.temporary_path.unlink() + asset.temporary_path.parent.rmdir() + + +def test_capture_with_no_advertised_assets_is_complete(tmp_path: Path) -> None: + from mindcap.core.models import CaptureRequest + + project = {"id": PROJECT_ID, "clip_count": 1, "clips": [{"id": "clip-1"}]} + + def handler(request: httpx.Request) -> httpx.Response: + if request.url.path == f"/api/project/{PROJECT_ID}": + return httpx.Response(200, request=request, json=project) + return httpx.Response(404, request=request, json={}) + + client = SunoClient(state=_state(), transport=httpx.MockTransport(handler)) + envelope = SunoWorkspaceCaptureService(client=client).capture( + CaptureRequest( + source_type="suno", + source=PROJECT_ID, + provider="suno", + canonical_identifier=PROJECT_ID, + canonical_url=None, + strategy="api", + artifact_root=tmp_path, + options={"include_video": False}, + ) + ) + + assert envelope.safe_metadata["capture_complete"] is True + assert envelope.safe_metadata["audio_expected_count"] == 0 + assert envelope.safe_metadata["audio_failed_count"] == 0 + assert envelope.safe_metadata["asset_expected_count"] == 0 + assert envelope.safe_metadata["asset_downloaded_count"] == 0 + assert envelope.safe_metadata["asset_failed_count"] == 0 + assert not envelope.warnings + + # --------------------------------------------------------------------------- # Client: production endpoint is tried first # --------------------------------------------------------------------------- diff --git a/tests/test_sync_runner.py b/tests/test_sync_runner.py index 174adc1..3f0e3dc 100644 --- a/tests/test_sync_runner.py +++ b/tests/test_sync_runner.py @@ -2,18 +2,26 @@ from __future__ import annotations +import json +import logging +import signal from collections.abc import Iterable from datetime import UTC, datetime +from io import StringIO from pathlib import Path from typing import Any from unittest.mock import MagicMock, patch import pytest +from rich.console import Console +from mindcap.core.errors import AuthenticationRequiredError, CaptureFailedError +from mindcap.core.progress import CaptureProgressReporter from mindcap.sync.models import ( BatchRunConfig, CollectionRequest, DiscoveryResult, + ItemStatus, RunStatus, SourceDescriptor, ) @@ -316,6 +324,472 @@ def _capture(request: Any) -> CaptureEnvelope: assert run_exit_code(state) == EXIT_PARTIAL_FAILURE +class TestSafeDiagnostics: + def test_transient_failures_report_backoff_and_preserve_attempt_history( + self, tmp_path: Path + ) -> None: + plugin = _mock_plugin() + capture = plugin.strategy.return_value.capture.side_effect + calls = 0 + observations: list[str] = [] + reporter = MagicMock(spec=CaptureProgressReporter) + reporter.warn.side_effect = lambda message: observations.append(message) + + def fail_twice(request: Any) -> Any: + nonlocal calls + calls += 1 + if calls <= 2: + raise CaptureFailedError( + "request timed out for https://media.example/audio?token=hidden" + ) + return capture(request) + + plugin.strategy.return_value.capture.side_effect = fail_twice + + def record_sleep(seconds: int) -> None: + observations.append(f"sleep {seconds}") + + with ( + patch("mindcap.sync.runner.build_registry") as registry, + patch("mindcap.sync.runner.time.sleep", side_effect=record_sleep) as sleep, + ): + registry.return_value.get.return_value = plugin + state = SyncRunner( + FakeDiscovery([_descriptor("ws-retry")]), + "workspaces/suno", + reporter=reporter, + ).run(config=_config(force=True), artifact_root=tmp_path) + + assert state.status == RunStatus.COMPLETE + assert calls == 3 + assert [call.args[0] for call in sleep.call_args_list] == [2, 4] + assert "attempt 1 failed" in observations[0] + assert "Retrying in 2s (retry 1 of 3)" in observations[0] + assert observations[1] == "sleep 2" + assert "attempt 2 failed" in observations[2] + assert observations[3] == "sleep 4" + assert any( + "attempt 3 of 4" in call.args[0] for call in reporter.detail.call_args_list + ) + attempts = state.items[0].attempts + assert [attempt.status for attempt in attempts] == [ + ItemStatus.FAILED_RETRYABLE, + ItemStatus.FAILED_RETRYABLE, + ItemStatus.COMPLETE, + ] + assert all(attempt.error for attempt in attempts[:2]) + run_dir = tmp_path / "runs" / "suno" / state.run_id + saved = (run_dir / "run.json").read_text() + events = (run_dir / "items.jsonl").read_text() + assert "hidden" not in saved + events + " ".join(observations) + assert "media.example/audio" in saved + assert "[REDACTED]" in events + + @pytest.mark.parametrize( + "error_type, message, expected_attempts", + [ + (CaptureFailedError, "permission denied", 1), + (AuthenticationRequiredError, "session expired", 1), + (CaptureFailedError, "request timed out", 4), + ], + ) + def test_final_failures_are_visible_and_redacted( + self, + tmp_path: Path, + error_type: type[Exception], + message: str, + expected_attempts: int, + ) -> None: + output = StringIO() + reporter = CaptureProgressReporter( + Console(file=output, width=160, color_system=None), verbose=True + ) + error = error_type( + f"{message}: https://media.example/audio?signature=signed-secret\n" + "Authorization: Bearer bearer-secret\n" + "Cookie: session=cookie-secret" + ) + plugin = _mock_plugin(capture_raises=error) + with ( + patch("mindcap.sync.runner.build_registry") as registry, + patch("mindcap.sync.runner.time.sleep"), + ): + registry.return_value.get.return_value = plugin + state = SyncRunner( + FakeDiscovery([_descriptor("ws-failed")]), + "workspaces/suno", + reporter=reporter, + ).run(config=_config(force=True), artifact_root=tmp_path) + + assert state.status == RunStatus.COMPLETE_WITH_FAILURES + item = state.items[0] + assert len(item.attempts) == expected_attempts + assert item.status == ItemStatus.FAILED_TERMINAL + assert all(attempt.error for attempt in item.attempts) + assert "ws-failed" in output.getvalue() + assert "failed" in output.getvalue() + if error_type is AuthenticationRequiredError: + assert "Authenticate again" in " ".join(output.getvalue().split()) + else: + assert f"after {expected_attempts} attempt(s)" in output.getvalue() + run_dir = tmp_path / "runs" / "suno" / state.run_id + diagnostics = output.getvalue() + "".join( + path.read_text() + for path in run_dir.iterdir() + if path.suffix in {".json", ".jsonl", ".md"} + ) + for secret in ("signed-secret", "bearer-secret", "cookie-secret"): + assert secret not in diagnostics + assert "[REDACTED]" in diagnostics + events = [ + json.loads(line) + for line in (run_dir / "items.jsonl").read_text().splitlines() + ] + assert events[-1]["event"] == "item-failed" + assert events[-1]["terminal"] is True + + def test_run_failure_writes_safe_final_report_and_displays_reason( + self, tmp_path: Path + ) -> None: + discovery = FakeDiscovery([]) + discovery.discover = MagicMock( # type: ignore[method-assign] + side_effect=CaptureFailedError( + "discovery failed: https://api.example/items?token=query-secret\n" + "Authorization: Bearer header-secret" + ) + ) + output = StringIO() + reporter = CaptureProgressReporter( + Console(file=output, width=160, color_system=None) + ) + state = SyncRunner(discovery, "workspaces/suno", reporter=reporter).run( + config=_config(), artifact_root=tmp_path + ) + + assert state.status == RunStatus.FAILED + assert state.completed_at is not None + assert "Sync failed" in output.getvalue() + run_dir = tmp_path / "runs" / "suno" / state.run_id + report = json.loads((run_dir / "report.json").read_text()) + assert report["status"] == "failed" + assert report["completed_at"] is not None + assert report["errors"] == state.errors + diagnostics = output.getvalue() + "".join( + path.read_text() + for path in run_dir.iterdir() + if path.suffix in {".json", ".jsonl", ".md"} + ) + assert "query-secret" not in diagnostics + assert "header-secret" not in diagnostics + assert "api.example/items" in diagnostics + + @pytest.mark.parametrize("capture_complete", [None, True, 0, "false"]) + def test_advisory_envelope_warnings_are_visible_and_preserved( + self, tmp_path: Path, capture_complete: object + ) -> None: + from mindcap.core.models import CaptureEnvelope + + plugin = _mock_plugin() + + def capture_with_warning(request: Any) -> CaptureEnvelope: + return CaptureEnvelope( + provider=request.provider, + source_type=request.source_type, + canonical_identifier=request.canonical_identifier, + canonical_url=request.canonical_url, + captured_at=datetime.now(UTC), + strategy="api", + response_units=[], + warnings=[ + "Audio download failed (403): " + "https://media.example/audio?signature=warning-secret" + ], + safe_metadata={"capture_complete": capture_complete}, + ) + + plugin.strategy.return_value.capture.side_effect = capture_with_warning + output = StringIO() + reporter = CaptureProgressReporter( + Console(file=output, width=160, color_system=None) + ) + with patch("mindcap.sync.runner.build_registry") as registry: + registry.return_value.get.return_value = plugin + state = SyncRunner( + FakeDiscovery([_descriptor("ws-advisory")]), + "workspaces/suno", + reporter=reporter, + ).run(config=_config(force=True), artifact_root=tmp_path) + + item = state.items[0] + assert state.status == RunStatus.COMPLETE + assert item.status == ItemStatus.COMPLETE_WITH_WARNINGS + assert item.attempts[0].status == ItemStatus.COMPLETE + assert len(item.warnings) == 1 + assert item.attempts[0].warnings == item.warnings + assert "Audio download failed (403)" in item.warnings[0] + assert "warning-secret" not in item.warnings[0] + assert "Audio download failed (403)" in output.getvalue() + assert "warning-secret" not in output.getvalue() + run_dir = tmp_path / "runs" / "suno" / state.run_id + saved_item = json.loads((run_dir / "run.json").read_text())["items"][0] + assert saved_item["warnings"] == item.warnings + assert saved_item["attempts"][0]["warnings"] == item.warnings + + def test_matching_envelope_warnings_from_two_sources_count_both_occurrences( + self, tmp_path: Path, caplog: pytest.LogCaptureFixture + ) -> None: + from mindcap.core.models import CaptureEnvelope + + warning = "Optional cover artwork was unavailable." + plugin = _mock_plugin() + capture = plugin.strategy.return_value.capture.side_effect + + def capture_with_advisory(request: Any) -> CaptureEnvelope: + envelope = capture(request) + assert isinstance(envelope, CaptureEnvelope) + envelope.warnings.append(warning) + return envelope + + plugin.strategy.return_value.capture.side_effect = capture_with_advisory + output = StringIO() + reporter = CaptureProgressReporter( + Console(file=output, width=160, color_system=None) + ) + with ( + caplog.at_level(logging.WARNING, logger="mindcap.progress"), + patch("mindcap.sync.runner.build_registry") as registry, + ): + registry.return_value.get.return_value = plugin + state = SyncRunner( + FakeDiscovery([_descriptor("ws-first"), _descriptor("ws-second")]), + "workspaces/suno", + reporter=reporter, + ).run(config=_config(force=True), artifact_root=tmp_path) + + assert state.status == RunStatus.COMPLETE + assert all(item.warnings == [warning] for item in state.items) + assert reporter._warning_count == 2 + assert f"2x {warning}" in output.getvalue() + warning_events = [ + record + for record in caplog.records + if record.name == "mindcap.progress" and record.getMessage() == warning + ] + assert len(warning_events) == 2 + + @pytest.mark.parametrize( + "stored_status, with_console", [("complete", True), ("unchanged", False)] + ) + def test_explicitly_partial_archive_fails_without_retry_and_retains_archive( + self, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + stored_status: str, + with_console: bool, + ) -> None: + from mindcap.core.models import CaptureEnvelope + from mindcap.sync.models import EXIT_PARTIAL_FAILURE, run_exit_code + + plugin = _mock_plugin() + capture = plugin.strategy.return_value.capture.side_effect + + def capture_partial(request: Any) -> CaptureEnvelope: + envelope = capture(request) + assert isinstance(envelope, CaptureEnvelope) + envelope.safe_metadata["capture_complete"] = False + envelope.warnings.append("Required audio download failed (403).") + return envelope + + plugin.strategy.return_value.capture.side_effect = capture_partial + plugin.storage.return_value.persist.return_value.status = stored_status + output = StringIO() + reporter = CaptureProgressReporter( + Console(file=output, width=160, color_system=None) if with_console else None + ) + with ( + patch("mindcap.sync.runner.build_registry") as registry, + patch("mindcap.sync.runner.time.sleep") as sleep, + ): + registry.return_value.get.return_value = plugin + state = SyncRunner( + FakeDiscovery([_descriptor("ws-partial")]), + "workspaces/suno", + reporter=reporter, + ).run(config=_config(force=True), artifact_root=tmp_path) + + assert state.status == RunStatus.COMPLETE_WITH_FAILURES + assert run_exit_code(state) == EXIT_PARTIAL_FAILURE + item = state.items[0] + assert item.status == ItemStatus.FAILED_TERMINAL + assert item.archive_path == "/tmp/bundle" + assert item.archive_version == 1 + assert len(item.attempts) == 1 + attempt = item.attempts[0] + assert attempt.status == ItemStatus.FAILED_TERMINAL + assert attempt.archive_path == item.archive_path + assert ( + attempt.warnings + == item.warnings + == ["Required audio download failed (403)."] + ) + assert attempt.error == item.errors[0] + assert "Capture is partial" in item.errors[0] + assert "saved archive retained" in item.errors[0] + assert plugin.strategy.return_value.capture.call_count == 1 + sleep.assert_not_called() + run_dir = tmp_path / "runs" / "suno" / state.run_id + report = json.loads((run_dir / "report.json").read_text()) + assert report["status"] == "complete-with-failures" + assert report["counts"]["failed"] == 1 + assert report["counts"]["completed"] == 0 + saved_item = json.loads((run_dir / "run.json").read_text())["items"][0] + assert saved_item["status"] == "failed-terminal" + assert saved_item["attempts"][0]["status"] == "failed-terminal" + events = [ + json.loads(line) + for line in (run_dir / "items.jsonl").read_text().splitlines() + ] + assert events[-1]["event"] == "item-failed" + assert events[-1]["archive_path"] == item.archive_path + assert events[-1]["error"] == attempt.error + if with_console: + assert "Required audio download failed (403)" in output.getvalue() + assert "Capture is partial" in output.getvalue() + else: + assert output.getvalue() == "" + assert capsys.readouterr() == ("", "") + + +class TestSignalLifecycle: + @pytest.mark.parametrize("item_count", [1, 2]) + def test_signal_during_capture_preserves_interrupted_run_status( + self, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + item_count: int, + ) -> None: + from mindcap.core.models import CaptureEnvelope + from mindcap.sync.models import EXIT_INTERRUPTED, run_exit_code + + previous: dict[int, Any] = { + signal.SIGINT: object(), + signal.SIGTERM: object(), + } + active = previous.copy() + monkeypatch.setattr(signal, "getsignal", lambda signum: active[signum]) + monkeypatch.setattr( + signal, + "signal", + lambda signum, handler: active.__setitem__(signum, handler), + ) + plugin = _mock_plugin() + capture = plugin.strategy.return_value.capture.side_effect + + def interrupt_during_capture(request: Any) -> CaptureEnvelope: + active[signal.SIGINT](signal.SIGINT, None) + envelope = capture(request) + assert isinstance(envelope, CaptureEnvelope) + return envelope + + plugin.strategy.return_value.capture.side_effect = interrupt_during_capture + with patch("mindcap.sync.runner.build_registry") as registry: + registry.return_value.get.return_value = plugin + state = SyncRunner( + FakeDiscovery([_descriptor(f"ws-{i}") for i in range(item_count)]), + "workspaces/suno", + ).run(config=_config(force=True), artifact_root=tmp_path) + + assert state.status == RunStatus.INTERRUPTED + assert state.interruption_reason == f"signal-{signal.SIGINT}" + assert run_exit_code(state) == EXIT_INTERRUPTED + assert plugin.strategy.return_value.capture.call_count == 1 + assert state.items[0].status == ItemStatus.COMPLETE + if item_count > 1: + assert state.items[1].status == ItemStatus.PLANNED + assert state.items[1].attempts == [] + run_dir = tmp_path / "runs" / "suno" / state.run_id + report = json.loads((run_dir / "report.json").read_text()) + assert report["status"] == "interrupted" + assert json.loads((run_dir / "run.json").read_text())["status"] == "interrupted" + assert active == previous + + @pytest.mark.parametrize("fail_discovery", [False, True]) + def test_previous_signal_handlers_are_restored( + self, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + fail_discovery: bool, + ) -> None: + previous = {signal.SIGINT: object(), signal.SIGTERM: object()} + active = previous.copy() + monkeypatch.setattr(signal, "getsignal", lambda signum: active[signum]) + monkeypatch.setattr( + signal, + "signal", + lambda signum, handler: active.__setitem__(signum, handler), + ) + discovery = FakeDiscovery([]) + if fail_discovery: + discovery.discover = MagicMock( # type: ignore[method-assign] + side_effect=CaptureFailedError("discovery rejected") + ) + with patch("mindcap.sync.runner.build_registry") as registry: + registry.return_value.get.return_value = _mock_plugin() + state = SyncRunner(discovery, "workspaces/suno").run( + config=_config(), artifact_root=tmp_path + ) + + assert state.status == ( + RunStatus.FAILED if fail_discovery else RunStatus.COMPLETE + ) + assert active == previous + + def test_runner_can_be_reused_after_interruption( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + previous: dict[int, Any] = { + signal.SIGINT: object(), + signal.SIGTERM: object(), + } + active = previous.copy() + monkeypatch.setattr(signal, "getsignal", lambda signum: active[signum]) + monkeypatch.setattr( + signal, + "signal", + lambda signum, handler: active.__setitem__(signum, handler), + ) + discovery = FakeDiscovery([_descriptor("ws-001")]) + discover = discovery.discover + interrupted_once = False + + def interrupt_first_discovery( + request: CollectionRequest, reporter: Any + ) -> Iterable[SourceDescriptor]: + nonlocal interrupted_once + descriptors = discover(request, reporter) + if not interrupted_once: + interrupted_once = True + active[signal.SIGINT](signal.SIGINT, None) + return descriptors + + monkeypatch.setattr(discovery, "discover", interrupt_first_discovery) + runner = SyncRunner(discovery, "workspaces/suno") + with patch("mindcap.sync.runner.build_registry") as registry: + registry.return_value.get.return_value = _mock_plugin() + interrupted = runner.run( + config=_config(force=True), artifact_root=tmp_path, run_id="first" + ) + assert interrupted.status == RunStatus.INTERRUPTED + assert active == previous + completed = runner.run( + config=_config(force=True), artifact_root=tmp_path, run_id="second" + ) + + assert completed.status == RunStatus.COMPLETE + assert completed.counts()["complete"] == 1 + assert active == previous + + # --------------------------------------------------------------------------- # Retryable error classification # ---------------------------------------------------------------------------