Set Memory

A macOS app for DJs with two co-headline surfaces:

1. Set Memory - a searchable, deduped record of every track and session you have ever played across every CDJ-export USB. Plug a drive in, the desktop or CDJ-export library is read automatically, tracks collapse via a normalised (title, artist + version-aware) key so the same file on two USBs counts once with plays summed.
2. Playlist Lab - ~25 generators that mine that record for ready-to-deploy rekordbox playlists. One click writes any of them into your Mac rekordbox library (backed up first).

All processing on-machine. No cloud, no API keys, no telemetry.

What Playlist Lab proposes

The headline output is a tonight's-set recipe: 60 / 90 / 120 minute setlists composed of opener → build → peak → cooldown → closer slots, each slot filled from real history of where you've actually placed tracks in past sets. Other generator families:

• Memory - Forgotten Favourites (tiered by depth + recency), Rested Heroes (revival-leverage scored), Heavy Rotation Right Now, Personal Canon (recency-weighted top 50).
• Fresh - Try These (recent buys, unplayed), Buy-and-Bury (verdict-time for tracks added but never played).
• Pairings - Markov transition chains ("From X you usually go..."), Cliques (groups of tracks that keep showing up together), Cold Cliques (clusters dormant 60+ days), Seed + Companions, Similar Sessions (Jaccard retrieval), Playlist Co-occurrence (mined from your manual rekordbox playlists).
• Arcs - Key Chains (Camelot wheel walks), BPM ramps, Energy Arc presets (Warm-up / Peak-time / Sunset Closer), Half-time / Double-time bridges.
• Roles - Opener Pool, Peak-Slot Workhorses, Closer Pool, plus First-4 / Last-5-of-the-night history.
• Extras - Anniversaries ("Your set, 1 year ago today"), This-month-last-year, Starred-but-cold, Genre + Color packs, Loved-at-home-never-played-out, Neglected playlists.

Empty / partial / error states are first-class: fresh install shows a "plug in a USB" welcome; ingest pre-flights rekordbox.app and disables Create-in-rekordbox with an explanation when it's locked.

GUI

scripts/install.sh builds and installs SetMemory.app at /Applications/SetMemory.app. The app reads state.db directly (plain SQLite; the SQLCipher boundary stays in Python), auto-refreshes on every ingest write, and pauses background work when it isn't the active app so idle CPU is near zero.

• Sidebar - three groups: Playlist Lab (Suggestions), Set Memory (Forgotten / Recent / Never / Together / Sessions / Search), Library (Distribution / USB Drives / Possibly Deleted).
• Filter chips at the top of Playlist Lab - Tonight / Memory / Fresh / Pairings / Arcs / Roles / Extras - filter the ~70 cards per refresh into navigable cohorts.
• Track rows at pro-tool density (4-5px padding), with BPM tempo ramp colour + Camelot key chip + monospaced-digit tabular figures.
• Live sync progress in the toolbar button ("Reading USB", "Saving sessions", "Deduping", "Indexing", "Done").
• Inline expansion on row click - no separate inspector pane.

Top toolbar: hand-drawn brand glyph + wordmark, USB picker (live mount/unmount), search field (diacritic-folded across title + artist + genre + label + comment), Sync button (greyed until a USB is mounted), Settings (via native Settings scene; ⌘,). Bottom status bar collapses all stats into a single mid-dot-separated line.

⌘R - Sync now · ⌘⇧R - Refresh from state.db · ⌘1..⌘9 jump to sidebar items in display order · ⌘F focus search · Escape clears.

CLI

~/miniconda3/bin/python ~/Downloads/set-memory/set_memory.py query forgotten
~/miniconda3/bin/python ~/Downloads/set-memory/set_memory.py query together
~/miniconda3/bin/python ~/Downloads/set-memory/set_memory.py query search --search "marlon"
~/miniconda3/bin/python ~/Downloads/set-memory/set_memory.py query sessions --since 2026-01-01

# Playlist Lab
~/miniconda3/bin/python ~/Downloads/set-memory/set_memory.py suggestions | jq
~/miniconda3/bin/python ~/Downloads/set-memory/set_memory.py create-playlist "Tonight" --suggestion full_set_90
~/miniconda3/bin/python ~/Downloads/set-memory/set_memory.py dismiss-suggestion forgotten_pack
~/miniconda3/bin/python ~/Downloads/set-memory/set_memory.py undismiss-suggestion --all

---

What's new in 0.9.0

Set Memory has two co-headline features now: the original "what did you play and when" memory surface, and a brand-new generator that turns your history into ready-to-deploy rekordbox playlists.

• Playlist Lab. ~70 deterministic playlist suggestions per refresh, drawing on signals the old build ignored entirely. New families:
  • Markov transition chains - "From X you usually go..." walks the real back-to-back transition graph you've built across every session. Half a setlist on tap.
  • Position-role pools - Your Openers / Peak-time Anchors / Closers by average position in past sets; First-4 and Last-5 history packs.
  • Personal Canon (270-day half-life) + Heavy Rotation Right Now (60-day half-life) - what's actually live this season.
  • Rested Heroes - revival-leverage scored by plays × log(days cold).
  • Cold Cliques - dormant 60+ days, distinct from the live ones.
  • Anniversary Sets - "Your set, 1 year ago today" with the actual tracklist.
  • Energy Arcs - Warm-up / Peak-time / Sunset Closer presets shaped along BPM curves.
  • Half-time / Double-time Bridges - 70↔140 matched at the same key.
  • Session Similarity - Jaccard over past sessions surfaces "more like your 2026-04-12 set, here are the tracks you haven't reused yet".
  • Editorial co-occurrence from your own rekordbox playlists - the cleanest taste signal you produce, finally read.
  • Mined fields: rekordbox star Rating, free-text Comment, Genre, Label, ColorName, DJPlayCount, ReleaseYear, Length - all available via state.db v5 after the next sync.
• Performance pass. App pauses every background query while it's not the front window. PRAGMA data_version short-circuits reload when nothing has been written to state.db. Heavy sections (distribution, played-together, sessions, deleted) are now lazy-loaded - the slice fetches when you click the sidebar item. Suggester subprocess throttled to one launch per 12s during ingest write bursts.
• Reframed sidebar. Two co-equal top groups - Playlist Lab and Set Memory - reflect the dual-headline framing. Library hygiene (USB drives, distribution, possibly-deleted) gets its own third group.
• Prep Audit removed. It duplicated rekordbox's own analyse-on-load flow and produced churn rather than signal.
• Critical fixes shipping in this build:
  • Suggester no longer TypeErrors on launch (variants kwargs).
  • GUI no longer flashes blank↔normal during ingest (SQLITE_BUSY + silent-empty-array regression now handled with busy_timeout + a single do/catch over the whole reload).
  • Dismiss-and-replace works as you'd expect - dismissing a suggestion surfaces a new one from the deep pool of family variants.

What's new in 0.4.0

• CDJ-export USBs (.pdb) now ingestable. The original release only read the legacy PIONEER/Master/master.db (SQLCipher) layout. Modern rekordbox exports to PIONEER/rekordbox/export.pdb (Pioneer's reverse-engineered DeviceSQL format), and the existing pyrekordbox couldn't touch it. New pure-Python pdb_reader.py (no extra pip deps) parses the .pdb directly; discovery accepts either layout and dispatches to the right ingest path automatically.
• GUI overhaul against the AI-design-tells ban list. Custom wordmark, dense rekordbox-inspired row layout, inline-expand on selection (no separate inspector pane), bottom status bar with all stats (no sidebar footer), collapsible sidebar groups (Library / Patterns / Maintenance), split-role accents (cyan = selection, amber = action, coral = danger), tempo-coloured BPM column, Camelot-wheel-coloured key chips, deliberately tight spacing.
• USB picker. Multiple CDJ USBs mounted? Sync targets one or all, selectable from the top bar.
• App auto-surfaces on USB mount. When launchd fires the ingest, Set Memory opens in the background dock so results are one click away.

What's new in 0.3.0

• Native SwiftUI GUI. Set Memory now ships with /Applications/SetMemory.app: three-pane window with sidebar surfaces, live search, table views per analysis, distribution charts (BPM histogram, Camelot key bars, monthly sparkline), a track-detail inspector with copy-to-clipboard, a Sync Now toolbar button (live mounted-USB indicator), and an inline Settings sheet for every threshold. App reads state.db directly via plain SQLite; auto- refreshes whenever the launchd agent writes a new digest.
• scripts/install.sh now builds + installs the GUI alongside the launchd agent.

What's new in 0.2.0

• Never-played actually works. The original release had a structural bug: the tracks table only got populated from session appearances, so the "never played" query (filtering for zero appearances against that table) was always empty by construction. Set Memory now syncs the full djmdContent library on every mount, then layers play counts on top.
• Notifications actually fire. Single-quoted AppleScript strings don't compile; every notification has silently failed since release with syntax error -2741 (visible only in stderr.log). Fixed; click-through added via terminal-notifier when present.
• BPM correctly normalised from rekordbox's value * 100 integer encoding, with zero treated as "not analysed."
• Prep audit flags library tracks missing BPM, key, or hot cues.
• Played-together pairs from co-appearance across sessions.
• Recently-added-unplayed as a distinct buy-regret signal.
• Distribution stats across BPM buckets and Camelot keys.
• Per-USB tracking in a new usb_drives table.
• Possibly-deleted tracks surfaced by staleness.
• Sparkline + headline in the digest.
• query CLI subcommand for all of the above without remounting.
• Schema migrations with schema_version in meta; upgrades a v1 state.db in place without data loss.

What this is NOT

• Not audio analysis. Set Memory reads structured rows from rekordbox's djmdHistory and djmdSongHistory tables. It does not process audio signals, waveforms, or any binary media. "Played" means "appeared in a session log row," not "was heard by the system."
• Not a real-time DJ critic. Runs once per USB mount. Does not touch a live session in progress.
• Not a replacement for rekordbox. Set Memory is a read-only companion that accumulates a local history log (state.db) across syncs.

---

How it works

1. You plug in any rekordbox USB drive.
2. macOS fires the launchd agent (com.mord58562.setmemory).
3. Set Memory scans every mounted volume for PIONEER/Master/master.db. Non-rekordbox volumes are ignored silently; multiple DJ drives mounted at once are all processed in the same run.
4. For each rekordbox USB, it opens master.db (decrypted via SQLCipher + pyrekordbox) and reads djmdHistory.
5. New sessions (not yet in state.db) are ingested by content fingerprint, so re-mounting the same drive or two USBs mirroring each other never double-count.
6. Forgotten-favourites and never-played lists are computed across the union of every session ever recorded, regardless of which drive it came from.
7. digest.md is written atomically. A macOS notification fires.
8. Total expected runtime: under 10 seconds for a typical sync.

---

Install

Prerequisites: Homebrew, miniconda3 at ~/miniconda3/.

# Step 1: Install system SQLCipher library
brew install sqlcipher

# Step 2: Install pyrekordbox into miniconda3 (pulls sqlcipher3-wheels
# in as a dependency; no separate adapter build step needed)
~/miniconda3/bin/pip install pyrekordbox==0.4.4

# Step 3: Run the installer (verifies the toolchain, builds the
# launchd plist, smoke-tests the pipeline)
bash ~/Downloads/set-memory/scripts/install.sh

After install, the launchd agent is live. Mount any rekordbox USB to trigger the first real sync.

---

Uninstall

bash ~/Downloads/set-memory/scripts/uninstall.sh

This removes the launchd agent only. state.db, digest.md, and logs/ are preserved. Delete them manually if you want a clean slate:

rm ~/Downloads/set-memory/state.db
rm ~/Downloads/set-memory/digest.md
rm -rf ~/Downloads/set-memory/logs/

---

Configuration

There is no per-drive setup. Set Memory discovers any mounted volume that has a rekordbox library on it (PIONEER/Master/master.db + a PIONEER/ rekordbox/ export tree) and ingests sessions from each. Plug in a new USB and it just works; reformat or rename a drive and nothing breaks.

config.json holds only the analysis thresholds, written next to the script on first run. Changes take effect on the next mount, no restart needed.

{
  "forgotten_min_appearances": 5,
  "forgotten_days_since_last": 90,
  "forgotten_limit": 10,
  "never_played_min_days_since_add": 30,
  "never_played_limit": 10,
  "state_db_path": "state.db",
  "digest_path": "digest.md",
  "append_to_jury_digest": false
}

First-run tuning note (Q4 from DESIGN): forgotten_min_appearances: 5 means a track needs to appear in at least 5 recorded sessions to qualify as a "favourite." If your sessions are infrequent (one per week or less), the forgotten list may be empty for the first few months while the history log builds up. In that case, lower the threshold to 3 in config.json - it costs nothing to change and you can raise it again later once the log has depth.

---

Running tests

Unit tests use a synthetic plain-SQLite fixture (no SQLCipher, no USB needed):

~/miniconda3/bin/pytest ~/Downloads/set-memory/tests/ -v

Smoke test (requires USB mounted and pyrekordbox installed):

RUN_SMOKE=1 ~/miniconda3/bin/pytest ~/Downloads/set-memory/tests/test_smoke.py -v

---

Troubleshooting

"Schema incompatibility" notification

Symptom: Notification body is "Schema error - check logs."

Cause: The rekordbox version on your XDJ changed and the djmdHistory, djmdSongHistory, or djmdContent table structure no longer matches what Set Memory expects.

Fix: Check logs/stderr.log for the specific missing column or table name. Open a GitHub issue with the error message if this happens after a rekordbox update.

"USB locked - retry on next mount"

Cause: The USB was disconnected from the XDJ before it finished writing the WAL file. Set Memory retried once and still could not get a clean snapshot.

Fix: Eject the USB cleanly from the XDJ (hold eject button until the disk icon disappears), then re-mount on Mac.

state.db is corrupt

Set Memory detects a corrupt state.db on open and recreates it automatically. The new file starts fresh (no history). Re-mounting the USB will re-ingest all djmdHistory sessions from scratch.

Logs

~/Downloads/set-memory/logs/stdout.log   - standard output from the launchd run
~/Downloads/set-memory/logs/stderr.log   - errors and warnings

USB volume label changed

Not a problem. Set Memory walks /Volumes/* and only cares whether each volume has a rekordbox library on it. Renaming or reformatting a drive doesn't change behaviour.

---

Data notes

• djmdContent.DateCreated is the file creation date, not the date you added the track to rekordbox. A track ripped in 2022 but added to your library this week will show 2022. The never-played threshold is therefore an approximation. This is a known limitation documented in the DESIGN (D5).

• DJPlayCount in rekordbox has uncertain hardware-increment semantics (per RECON). Set Memory ignores it entirely. "Played" means "appeared in at least one djmdSongHistory row," which corresponds to a confirmed session log entry.

---

File layout

~/Downloads/set-memory/
  set_memory.py          - entry point (--on-mount flag)
  ingest.py              - reads USB djmd* tables -> state.db
  analyse.py             - forgotten + never-played computation (pure SQL/Python)
  digest.py              - writes digest.md
  notify.py              - macOS notification via osascript
  config.py              - loads/validates config.json
  config.json            - user-editable thresholds
  state.db               - accumulated session log (never leaves machine)
  digest.md              - latest digest (overwritten on each sync)
  launchd/
    com.mord58562.setmemory.plist
  scripts/
    install.sh
    uninstall.sh
  tests/
    conftest.py
    test_ingest.py
    test_analyse.py
    test_digest.py
    test_notify.py
    test_smoke.py
    fixtures/
      synthetic_master.db     - built at test-time; plain SQLite, no encryption
      config_default.json
  logs/
    stdout.log
    stderr.log