Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,19 @@ Or with Task:
task check
```

## Logs and troubleshooting

Capture includes timestamped progress, grouped warnings, and actionable errors.
Use `--verbose` for individual asset details or save sanitized diagnostics:

```bash
uv run mindcap --debug --log-file "$HOME/Downloads/mindcap-debug.jsonl" \
capture suno "<workspace-id>"
```

See [terminal output and diagnostics](docs/diagnostics.md) for quiet/JSON output,
exit codes, and the diagnostic log format.

## Protect the Artifact Cache

Mindcap defaults to the Ego Hygiene repository root at:
Expand Down Expand Up @@ -127,6 +140,20 @@ Capture a workspace archive using a placeholder workspace UUID:
uv run mindcap capture suno "00000000-0000-0000-0000-000000000000"
```

If audio downloads return HTTP 403, see the
[Suno audio troubleshooting guide](docs/suno-audio.md). Metadata authentication
and permission to download media are separate checks. The capture summary
reports downloaded/expected primary audio and marks advertised asset failures
as partial; archive integrity PASS checks the files that were actually saved.

Inspect an existing capture without contacting Suno or printing its URLs:

```bash
uv run mindcap inspect suno \
".cache/mindcap/workspaces/suno/suno-00000000-0000-0000-0000-000000000000/v1" \
--asset-diagnostics
```

## Vault Ingestion and Verification

Mindcap can ingest finalized Suno workspace bundles into an immutable vault made of sealed SQLite catalog generations and ZIP64 pack files. Example:
Expand Down
33 changes: 24 additions & 9 deletions docs/cli-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,19 +70,24 @@ envelope = strategy.capture(request)

```
mindcap_cli/
└── app.py Typer application and all command registrations
├── app.py Typer application and command registrations
├── runtime.py Invocation boundaries and diagnostic option resolution
├── logging.py Safe terminal errors and optional JSONL diagnostics
└── _compat.py Installed Typer command-engine compatibility
```

The current phase places all command implementations in `app.py`. A future
phase may split them into `commands/` submodules.
`DiagnosticTyper` assigns the same invocation boundary to every registered
command. Command implementations remain in `app.py`.

---

## Progress rendering

The CLI constructs a `CaptureProgressReporter` backed by a Rich `Console`.
Provider strategies accept a reporter so they can emit phase labels and
progress bars during capture.
Provider strategies accept a reporter so they can emit timestamped phase
labels, progress bars, warning groups, and attempt counts during capture.
Diagnostic output goes to stderr; the final summary has a separate stdout
console. Redirected output uses bounded plain start/end events.

Progress rendering is a CLI concern. The library emits structured
`MindcapEvent` objects through the `EventSink` protocol instead.
Expand All @@ -95,14 +100,14 @@ Progress rendering is a CLI concern. The library emits structured
|------------------------------|-----------|
| Success | 0 |
| `MindcapError`, `OSError` | 1 |
| Partial batch failure | 2 |
| Partial capture/batch or usage error | 2 |
| Interrupted (SIGINT) | 130 |

---

## JSON output

Pass `--json` to any command to receive machine-readable output on stdout:
Commands with `--json` support emit machine-readable output on stdout:

```bash
mindcap capture chatgpt "..." --json
Expand All @@ -116,15 +121,25 @@ result = application.capture(...)
console.print_json(data={"status": result.status, ...})
```

Progress is suppressed or redirected to stderr when `--json` is active.
Progress stays on stderr, including export imports, interrupted sync runs, and
their resume instructions. Operational failures emit a sanitized error envelope
on stderr when the command's JSON option is active.

---

## stdout vs stderr rules

- **stdout**: final result (path, JSON, or human-readable summary)
- **stderr**: progress, warnings, errors (via Rich `Console`)
- When `--quiet` is active, only the final path or JSON is printed
- When `--quiet` is active on capture/sync, stdout is the final path or JSON;
warnings and errors remain visible on stderr

Global `--verbose`, `--debug`, `--quiet`, and `--log-file` options apply before
the command name. Capture/sync retain their existing local verbosity options.
Handlers configure only the `mindcap` namespace, restore its previous state
after invocation, and never enable HTTP request logging. Debug errors include
sanitized frame locations without source lines or locals. See
[diagnostics](diagnostics.md) for the log format and redaction limits.

---

Expand Down
91 changes: 91 additions & 0 deletions docs/diagnostics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Terminal output and diagnostics

Mindcap displays timestamped, colored severity labels on an interactive
terminal. Phase progress, warnings, retries, and errors go to **stderr**; the
final result or JSON goes to **stdout**. Redirected output uses plain text with
bounded phase start/end lines instead of animated progress. Rich respects
`NO_COLOR=1` for terminals that should not use color.

## Everyday capture

The normal command already includes progress and warning summaries:

```bash
uv run mindcap capture suno "<workspace-id>"
```

Repeated asset failures are collapsed into a representative warning and a
grouped count. `--verbose` retains individual asset warnings and any per-item
detail supplied by the adapter. A finished progress phase counts **attempts**;
the final capture summary reports assets actually downloaded, capture
completeness, and saved-file integrity separately.

```bash
uv run mindcap capture suno "<workspace-id>" --verbose
uv run mindcap capture suno "<workspace-id>" --quiet
uv run mindcap capture suno "<workspace-id>" --json
```

Quiet mode retains warnings and errors. Capture prints only its archive path;
`--json` takes precedence for the final result, while quiet still suppresses
progress. Source adapters determine which phase and per-item details exist.

## Investigate a failure

Errors show a stable code, a sanitized description, and a next step. Debug mode
adds diagnostic events and exception frame locations, without dumping source
lines or local variables. Authentication, browser locks, invalid input, archive
verification, and filesystem failures have distinct codes.

Global diagnostic options go **before** the command name. Existing capture and
sync `--verbose`, `--debug`, and `--quiet` options still work after the command.

```bash
uv run mindcap --debug capture suno "<workspace-id>"
uv run mindcap --verbose doctor suno
uv run mindcap --debug --log-file "$HOME/Downloads/mindcap-debug.jsonl" \
capture suno "<workspace-id>"
```

The optional log contains newline-delimited JSON records with UTC timestamps,
severity, logger, and sanitized messages. It appends to a private file with
owner-only permissions, in an existing folder. Log setup errors stop the
command before capture begins; a write failure during capture emits a warning
and lets capture continue. No file is created unless `--log-file` is supplied.
Debug and quiet control the file's verbosity too; quiet takes precedence.

Diagnostics strip common credential fields, cookies, authorization headers,
JWTs, URL credentials, URL queries/fragments, and terminal control sequences.
Request/response bodies and browser storage state are never diagnostic input.
Logs can still contain source identifiers, titles, hosts, and local paths;
inspect a log before sharing it.

## Results and exit codes

| Exit | Meaning |
|---|---|
| `0` | Command succeeded. |
| `1` | Operation failed, including an unexpected error or unavailable log file. |
| `2` | Partial capture/batch result, or command usage error. |
| `130` | Interrupted. Sync prints its checkpoint/resume instructions. |

A partial single-source capture still prints its saved archive or JSON result
before exiting `2`. Archive integrity can pass while required assets are
missing. Inspect the warnings before retrying; a media `403` alone does not
establish that a login cookie is missing. See [Suno audio](suno-audio.md).

Commands that support `--json` emit their success result on stdout. A failure
emits a structured error envelope on stderr, with `code`, `type`, `message`,
and `hint`; debug mode can add sanitized frame locations. Stderr may also
contain preceding phase diagnostics, so consume stdout for result data and use
the exit status to determine success. Argument conversion and missing-argument
errors also honor a recognized JSON option; unrecognized-option errors may
use a text diagnostic.

## Library callers

Importing Mindcap does not configure the root logger or HTTP client loggers.
CLI handlers are installed only for one command and removed on exit. Embedding
applications can configure their own `mindcap` logging namespace. The progress
reporter remains silent when no console is attached; structured application
events remain a separate library contract.
88 changes: 88 additions & 0 deletions docs/suno-audio.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Troubleshoot Suno audio downloads

If project metadata, lyrics and artwork download but audio returns HTTP 403,
the metadata login worked. The media endpoint denied its own requests. This
does not establish that the stored Clerk cookie is missing or expired.

## Start with the existing archive

Run the offline diagnostic on the exact version directory printed by capture:

```bash
uv run mindcap inspect suno \
".cache/mindcap/workspaces/suno/suno-00000000-0000-0000-0000-000000000000/v2" \
--asset-diagnostics
```

Successful output is JSON containing counts and fixed labels. It excludes
titles, clip/project IDs, hosts, full URLs, query values and authentication
material. It verifies the saved bundle and makes no network requests. This
also works with captures made before the audio-download fix.

| Field | Meaning |
| --- | --- |
| `clips_with_archived_primary_audio` | Clip records listing a saved primary audio asset. Integrity verification separately checks the saved files. |
| `audio_url_kinds.forbidden_endpoint` | Suno supplied an `/api/forbidden` URL instead of a downloadable file. Refreshing a cookie alone does not establish download eligibility. |
| `audio_url_kinds.suno_cdn` | Primary URLs point to recognized Suno media hosts. A 403 may involve download authorization, an expired link, or CDN restrictions. |
| `audio_url_kinds.suno_api` | Primary URLs point to a recognized Suno API host. |
| `audio_url_kinds.other_http` | Primary URLs point elsewhere; no host values are printed. |
| `audio_url_kinds.missing` / `invalid` | Audio URLs were absent or malformed in the saved clip metadata. |
| `audio_urls_with_signature_keys` | URLs contain recognized signature parameter names. Their values are never printed; this count does not establish validity or expiry. |

You can share this successful diagnostic JSON when investigating a failure.
Keep the original metadata JSON, capture archive, cookies, browser network
exports and signed URLs private.

## Check one song in Suno

Open the same song in Suno and use its **More Actions → Download** menu.
If the site's own download is unavailable, inspect the account's download
eligibility or contact Suno support. Playback working does not establish that
file downloads are allowed.

Suno announced download-policy changes effective September 3, 2026, including
plan-based download allowances. See the current
[official FAQ](https://help.suno.com/en/articles/13614785) and
[download instructions](https://help.suno.com/en/articles/2409921). Mindcap's
experimental adapter predates those changes. It currently consumes the media
URLs supplied in clip metadata; it does not implement a separate download
authorization/allowance flow. Do not infer account limits from a 403 alone.

If the site's download succeeds, the diagnostic counts help distinguish an
outdated adapter URL from a failed media request. The client now sends
credential-free requests to external media hosts and preserves any supplied
signed query string. It keeps API authentication scoped to the configured API
origin and removes it when an asset redirect leaves that origin. This fixes
credential scoping; it does not prove that a particular live 403 is resolved.

## Authentication and retry

For metadata-login failures, run:

```bash
uv run mindcap doctor suno
```

This checks stored authentication and authenticated API reachability, not media
download eligibility. Refresh the Clerk cookie through the existing
`mindcap auth suno --cookie-stdin` flow only when authentication needs renewal.
Never paste credentials into an issue or chat.

After updating Mindcap, capture the workspace again. A new capture reads fresh
provider metadata and preserves earlier versions:

```bash
uv run mindcap capture suno "00000000-0000-0000-0000-000000000000"
```

Progress bars count attempts. The final **Audio** row reports downloaded versus
advertised primary audio. Any failed advertised asset makes the capture
partial, even if every clip's metadata was saved. **Archive integrity PASS**
means the saved files passed verification; it does not mean all audio exists.
Do not use that result alone as evidence that the music is fully backed up.

The primary audio format is whatever Suno supplies. The existing `--audio-format`
option changes its saved extension; it does not transcode audio or negotiate a
WAV export. `--request-wav` and `--include-stems` currently do not implement Suno
export requests. Use the site's supported export controls when those formats
are needed.
Loading
Loading