Prove what your AI forgot.

English · 简体中文

🧪 Try it in 60 seconds · 🔍 See the proof · 🧭 Open the live matrix

MemoryProof is an open-source memory assurance toolkit for AI agents. It tests a question that a successful DELETE response cannot answer:

Can the information still be observed through any path the backend exposes?

It creates isolated synthetic canaries, runs deterministic before/after probes, checks raw and derived boundaries, and emits an offline evidence bundle that developers can review, verify, and run in CI.

Agent memory is rarely one table. A single fact can be copied into a raw record, summary, embedding, graph node, cache, block, or working context. A delete endpoint may remove one representation while another still answers a search query.

MemoryProof makes that gap concrete with a controlled experiment:

The goal is not to assign a vendor a simplistic score. The goal is to make what was checked, what passed, and what cannot be observed explicit.

flowchart LR
    A["Scenario + synthetic canaries"] --> B["Rust runner"]
    B --> C["Versioned NDJSON adapter"]
    C --> D["Mem0 · Letta · Zep"]
    B --> E["Before probes"]
    D --> F["Erase / isolate"]
    F --> G["Settle + after probes"]
    E --> H["Evidence bundle"]
    G --> H
    H --> I["HTML · JUnit · CI · Matrix"]

1. Seed a canary that is unique, synthetic, and safe to send to a test namespace.
2. Prove the precondition: the target canary is observable before deletion.
3. Erase or isolate only resources owned by this run.
4. Wait for asynchronous writes to settle; a timeout becomes UNKNOWN, not a false pass.
5. Probe raw, derived, Agent, and control paths with deterministic rules.
6. Seal the evidence with sorted SHA-256 checksums and a bundle hash.

The repository includes a deliberately leaky backend. It deletes the raw item but leaves a derived artifact. MemoryProof must fail at the derived boundary:

status: FAIL
profile: erasure.derived
probe: target-derived-after
reason: the unique canary is still observable in a derived artifact

This is the core experience: an ambiguous memory concern becomes a named, reproducible, reviewable regression.

Requirements: Rust stable and Python 3.11+.

git clone https://github.com/Hughhhhcoder/MemoryProof.git
cd MemoryProof

cargo run -- adapters list
cargo run -- run examples/reference-clean.yml

The clean reference scenario exits 0 and writes a bundle under .memoryproof/runs/. Verify and open it offline:

cargo run -- verify .memoryproof/runs/<run-id>
open .memoryproof/runs/<run-id>/report.html       # macOS
# xdg-open .memoryproof/runs/<run-id>/report.html # Linux

Now run the intentionally leaky backend:

cargo run -- run examples/reference-leaky.yml

It exits 1: the raw item disappears, but a derived artifact remains observable. That failure is expected and demonstrates the check is working.

Download a platform binary from Releases, or run the GHCR image. Each binary archive includes the dependency-free Python adapter modules; if you keep them in a different location, set MEMORYPROOF_ADAPTER_ROOT to that archive's python/ directory.

docker run --rm -v "$PWD":/workspace \
  ghcr.io/hughhhhcoder/memoryproof:1 \
  run /workspace/examples/reference-clean.yml \
  --output /workspace/.memoryproof/runs

For compatibility with the original v0.1 project, the forgetproof binary name and forgetproof Python import path remain available during the migration window.

MemoryProof is an umbrella with two suites:

The human-facing legacy names FP-Object, FP-Scope, FP-Derived, and FP-Agent are accepted when loading v0.1 scenarios. New scenarios use the stable names above.

Every assertion is one of:

There is no aggregate score. A capability boundary should be readable, not averaged away.

These boundaries are part of the product, not a footnote. Reports explicitly separate proved, not observed, and out of scope.

Remote access is opt-in. Credentials are read from environment variables and are never written into scenarios or evidence:

MEM0_BASE_URL=https://... MEM0_API_KEY=... \
  cargo run -- run examples/mem0.yml --allow-network

Adapters communicate with the Rust runner through the versioned memoryproof.adapter/v1 NDJSON protocol. Third-party adapters can implement the same contract without linking to the Rust binary.

Each run emits an offline-readable package:

manifest.json       format, run, adapter mode, protocol, and declared files
scenario.lock.json  redacted, frozen scenario snapshot
scenario.lock.yml   the same frozen snapshot in the requested YAML form
events.ndjson       ordered method journal with safe request IDs
results.json        machine-readable assertions and profile states
report.html         single-file bilingual report
junit.xml           CI-native test report
checksums.sha256    per-file SHA-256 integrity list
bundle.hash         hash of the checksum manifest

Use memoryproof doctor --allow-network only when a provider endpoint is intentionally reachable, and use memoryproof expand ... --allow-network only when probe generation is configured to call a non-loopback OpenAI-compatible endpoint. The generated probes are frozen before a run; the model never makes the final pass/fail decision.

By default, payloads are reduced to hashes, lengths, types, and safe structural summaries. --allow-network authorizes a remote backend; it does not disable redaction.

The repository ships a Docker-based action that uploads the evidence bundle even when the test fails:

name: Memory assurance

on:
  pull_request:

jobs:
  memoryproof:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: Hughhhhcoder/MemoryProof@v1
        with:
          scenario: examples/reference-clean.yml

Use a leaky or vendor-specific scenario in a separate job when you want the PR to fail on a known regression. The action exposes bundle-path, status, and exit-code outputs and writes a short job summary.

The stable scenario API is memoryproof.dev/v1. A scenario contains:

• an adapter and non-sensitive configuration references;
• isolated target and control subjects;
• synthetic target/control fixtures;
• settle policy for asynchronous backends;
• an erase intent such as object_delete or subject_erase;
• deterministic probes and selected profiles;
• a privacy policy for redacted evidence.

The adapter protocol methods are hello, capabilities, prepare, ingest, settle, probe, erase, inspect, agent_query, cleanup, and close. stdout is reserved for protocol frames; adapter logs go to stderr. A capability that is not implemented must become SKIP or UNKNOWN, never a fabricated pass.

The checked-in *-adapter-contract evidence uses deterministic local mocks. For a real Mem0, Letta, or Zep deployment, use the opt-in credentialed provider workflow. It reads API keys from GitHub Secrets, requires --allow-network, uploads a reviewed artifact, and never commits remote evidence automatically.

cargo fmt --all
CARGO_NET_OFFLINE=true CARGO_TARGET_DIR=/tmp/memoryproof-target \
  cargo clippy --workspace --all-targets --all-features -- -D warnings
CARGO_NET_OFFLINE=true CARGO_TARGET_DIR=/tmp/memoryproof-target \
  cargo test --workspace -- --test-threads=2
PYTHONPATH=python python3 -m unittest discover -s python/tests -v

Please read CONTRIBUTING.md, SECURITY.md, and their 中文版本 before opening a pull request. MemoryProof is Apache-2.0 licensed and follows the DCO sign-off workflow.

• 🌐 Live Memory Assurance Matrix
• 🧪 Public conformance evidence
• ☁️ Credentialed provider conformance · 中文说明
• 📐 Scenario schema
• 🏗️ Architecture and trust boundaries · 中文架构说明
• 🧭 中文说明
• 🤝 Contributing · 中文贡献指南
• 🛡️ Security policy · 中文安全策略
• 📜 Changelog
• 📦 Releases
• 🐳 Container packages
• 🚀 Release operations · 中文发布说明

MemoryProof is the new umbrella name for the project formerly published as ForgetProof. The v0.1 evidence format and compatibility binary remain readable, while new scenarios and releases use memoryproof.dev/v1 and the memoryproof command. If you have an old GitHub Action reference, update Hughhhhcoder/ForgetProof@v0.1.0 to Hughhhhcoder/MemoryProof@v1; GitHub does not redirect Action references after a repository rename.