Skip to content

feat(synthesize): observation -> learning synthesis CLI - #106

Merged
raphasouthall merged 1 commit into
mainfrom
fix/36-synthesize
Aug 22, 2026
Merged

feat(synthesize): observation -> learning synthesis CLI#106
raphasouthall merged 1 commit into
mainfrom
fix/36-synthesize

Conversation

@raphasouthall

Copy link
Copy Markdown
Owner

Part of #36 — the remaining part 1 (parts 2/3/5 shipped via #85; tool-layer TTL default stays deliberately skipped as a data-safety landmine, see the 2026-07-06 scoping).

What

  • neurostack synthesize (new src/neurostack/synthesize.py): consolidates aged observation heaps into single learning memories.
    • Candidates: alive, non-superseded observations older than --min-age-days (default 7).
    • Clustering: greedy anchor grouping on stored embeddings (pairwise cosine, --threshold 0.35, anchor + --min-siblings 3). The issue's FTS5-sibling wording predates Bug: memory embeddings silently dropped on failure; no backfill path #29's full embedding coverage; stored-embedding clustering is cheaper (no embedder round-trips) and stronger. Candidates without embeddings are counted and reported, never silently dropped.
    • One LLM synthesis per cluster (same llm_url path as Consolidation replay: nightly promotion of episodic memories into semantic notes #96 consolidation); originals tagged superseded_by:<learning_id> — tagged, never deleted or archived.
    • Safety: dry-run default, --run opt-in, --cap 5 clusters per pass, per-cluster failure isolation. NOT wired into session_end auto_harvest (deliberate — manual/timer-driven for now).
    • Report includes active observation:learning ratio before/after.
  • Harvest: auto-captured observations now expire in 720h (30d) — harvest save path only, agent-written memories unaffected. The tool-layer TTL default from the issue would silently expire durable handoff context memories, so it stays skipped.
  • memory-management skill: promote-observations guidance + the new command.

Gate

ruff clean; 768 tests pass (18 new: candidate selection incl. NULL-tags regression, clustering/min-siblings/missing-embedding accounting, dry-run/cap, supersede tagging, per-cluster failure isolation, the #36 acceptance ratio scenario 3.5:1 -> <=2:1, harvest per-type TTL e2e).

Live verify

Dry run against the local DB exercises the CLI end-to-end (0 clusters — live data is on LXC 122; a live dry run follows the deploy).

- neurostack synthesize: greedy cosine clustering of aged observations
  (min-age 7d, anchor + >=3 siblings, threshold 0.35), one LLM-consolidated
  learning per cluster, originals tagged superseded_by:<id> (never deleted).
  Dry-run default, --run opt-in, cap 5 clusters per pass, per-cluster
  failure isolation. Report carries observation:learning ratio before/after.
- harvest: auto-captured observations get a 720h TTL (harvest save path
  only; agent-written memories keep caller-chosen TTL).
- memory-management skill: promote-observations guidance.
@raphasouthall
raphasouthall merged commit 03200c9 into main Aug 22, 2026
5 checks passed
@raphasouthall
raphasouthall deleted the fix/36-synthesize branch August 22, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant