Skip to content

End-to-end test and validate memory system #245

Description

@geier

Context

After deploying Graphiti (#240), building the watcher (#241), configuring MCP (#242), and optionally setting up the Kubeflow service (#243) and custom entity types (#244), the full system needs end-to-end validation to ensure everything works together reliably.

Goal

Validate the complete memory pipeline: session → watcher → Graphiti ingestion → MCP retrieval, across the key scenarios.

Test Scenarios

1. Basic Ingestion Flow

  • Start a new OpenCode session
  • Have a conversation that includes: an architectural decision, a code preference, an error + fix
  • End the session
  • Verify the watcher detects session completion and ingests the transcript
  • Verify Graphiti extracts entities and facts from the transcript
  • Query Graphiti to confirm knowledge is searchable

2. Cross-Session Retrieval

  • Start a new session in the same project
  • Use search_memory_facts to recall the decision from the previous session
  • Use search_nodes to find the error solution
  • Verify results are relevant and correctly attributed

3. Multi-Project Scoping

  • Run sessions in two different project directories
  • Verify each project's knowledge is scoped to its group_id
  • Verify cross-project queries work when specifying multiple group_ids

4. Multi-Machine Simulation

  • Run the watcher from two different directories pointing to the same Graphiti instance
  • Verify both sessions are ingested without conflicts
  • Verify knowledge from both is searchable

5. Temporal Knowledge

  • In session 1: "We're using React for the frontend"
  • In session 2 (later): "We switched from React to SolidJS"
  • Verify Graphiti correctly invalidates the old fact and tracks the change
  • Verify search_memory_facts returns the current state (SolidJS), not the old one

6. Watcher Resilience

  • Kill the watcher while a session is active
  • Restart the watcher
  • Verify it catches up on the missed session
  • Verify no duplicate ingestion occurs

7. Deduplication

  • Complete a session, verify it's ingested
  • Restart the watcher, verify it does NOT re-ingest the same session
  • Add a new message to the session, verify the delta is ingested

Acceptance Criteria

  • All 7 test scenarios pass
  • Memory retrieval returns relevant results (not random/unrelated)
  • Temporal facts are correctly tracked and invalidated
  • No duplicate ingestion on watcher restart
  • No data loss on watcher crash/restart
  • System works across multiple projects and (simulated) machines
  • Document any issues found and file follow-up issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions