Skip to content

OpenMind v2 Phase 2: Workspace Asset, Revision, Segment & Evidence foundation #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HelloThisWorld merged 1 commit into from
Jul 19, 2026
Merged

OpenMind v2 Phase 2: Workspace Asset, Revision, Segment & Evidence foundation #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 71 additions & 6 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,17 @@ jobs:
- name: Migration tests
run: python scripts/run_acceptance.py --only verify_migrations

- name: CLI tests
run: python scripts/run_acceptance.py --only verify_cli
- name: Content-store tests
run: python scripts/run_acceptance.py --only verify_content_store

- name: Asset model tests
run: python scripts/run_acceptance.py --only verify_asset_model

- name: CLI tests (base + asset commands)
run: python scripts/run_acceptance.py --only verify_cli --only verify_asset_cli

- name: Asset adapter tests
run: python scripts/run_acceptance.py --only verify_asset_adapters

- name: MCP smoke test
run: |
Expand All @@ -112,12 +121,17 @@ jobs:
from openmind import mcp_server
from openmind.runtime import get_runtime

expected = {"search", "route", "dispatch", "get_glossary",
"find_similar_cases", "save_case", "get_doc",
"propose_fix", "apply_fix"}
# The nine core tools are a STABLE external contract; Phase 2 adds
# read-only Asset tools ALONGSIDE them, never in place of one.
core = {"search", "route", "dispatch", "get_glossary",
"find_similar_cases", "save_case", "get_doc",
"propose_fix", "apply_fix"}
asset = {"list_assets", "get_asset", "get_asset_revisions", "get_evidence"}
server = mcp_server.create_mcp_server(get_runtime())
names = {t.name for t in asyncio.run(server.list_tools())}
assert names == expected, f"MCP tool drift: {names ^ expected}"
assert core <= names, f"core MCP tool missing: {core - names}"
assert asset <= names, f"asset MCP tool missing: {asset - names}"
assert names == core | asset, f"unexpected MCP tools: {names ^ (core | asset)}"
assert server.name == "open-mind", server.name
print("MCP smoke OK:", len(names), "tools")
PY
Expand Down Expand Up @@ -213,6 +227,57 @@ jobs:
- name: Artifact export contract
run: python tests/verify_artifacts.py

- name: Migration to v0003 + content-store byte round-trip
shell: bash
run: |
python - <<'PY'
import os, tempfile
os.environ["OPENMIND_DATA_DIR"] = tempfile.mkdtemp()
os.environ["OPENMIND_MACHINE_DIR"] = tempfile.mkdtemp()
from openmind import db, content_store as cs
db.init_db()
assert db.migration_status()["version"] == 3, db.migration_status()
# content-addressed blob byte round-trip + reuse + SHA-256 identity
import hashlib
data = b"phase2 content \x00\x01\x02 bytes"
h = cs.put("p_smoke0000ci", data)
assert h == hashlib.sha256(data).hexdigest()
assert cs.put("p_smoke0000ci", data) == h # reuse
assert cs.get("p_smoke0000ci", h) == data # round-trip
print("migration v0003 + content-store round-trip OK:", h[:12])
PY

- name: CLI asset help + fixture ingest + asset list + evidence read
shell: bash
run: |
export OPENMIND_DATA_DIR="$(mktemp -d)"
export OPENMIND_MACHINE_DIR="$(mktemp -d)"
python -m openmind.cli asset --help > /dev/null
python -m openmind.cli asset list --help > /dev/null
WS=$(python -m openmind.cli init --name smoke --path ./fixtures/sample-repo \
--ingest --wait --json | python -c "import json,sys; print(json.load(sys.stdin)['workspace_id'])")
# Small fixture ingest -> asset list -> resolve a revision/segment/evidence
# id chain -> read historical evidence back from the immutable snapshot.
OM_WS="$WS" python - <<'PY'
import json, os, subprocess, sys
ws = os.environ["OM_WS"]
def cli(*args):
return json.loads(subprocess.check_output(
[sys.executable, "-m", "openmind.cli", *args]))
data = cli("asset", "list", "--workspace", ws, "--json")
assert data["ok"] and data["total"] >= 1, data
aid = data["assets"][0]["id"]
show = cli("asset", "show", "--workspace", ws, "--asset", aid, "--json")
rid = show["asset"]["current_revision"]["id"]
segs = cli("asset", "segments", "--workspace", ws, "--revision", rid, "--json")
eid = segs["segments"][0]["evidence_id"]
ev = cli("asset", "evidence", "--workspace", ws, "--evidence", eid, "--json")
assert ev["snapshot"]["status"] == "available", ev
assert not ev["locator"]["file"].startswith("/"), ev
print("asset CLI smoke OK:", data["total"], "assets, evidence snapshot",
ev["snapshot"]["status"])
PY

- name: Skill bridge startup
shell: bash
run: |
Expand Down
80 changes: 69 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ Point Open Mind at a local repository and it builds persisted artifacts:
| Artifact | Implemented behavior |
|---|---|
| **Source-traceable knowledge index** | Stores repo-relative paths, content hashes, source locations, file metadata, and search chunks. |
| **Canonical Asset model** (v2 Phase 2) | Every indexed file is an **Asset**; every observed version an immutable **Revision**; each revision divided into deterministic **Segments**, each with source-locatable **Evidence**. Historical content is snapshotted in an immutable SHA-256 content store, so evidence for an old revision stays readable after the file changes. Unchanged re-ingestion creates no revision and never re-embeds; removed files are marked removed without erasing history. |
| **Verbatim glossary** | Extracts terms/acronyms from definition tables, definition lines, acronym expansions, and code comments; definitions are copied verbatim and carry `source_file`, `line_number`, and `content_hash`. The interactive learn path scans code/config sources; the `.openmind` export walk additionally scans README/docs/GLOSSARY files (primary definition sources). |
| **Structure and graph map** | Builds a module tree, per-file definition index, import/dependency graph, entry points, and a name-based call/usage graph. Ambiguous call edges are flagged instead of guessed. |
| **Exact-token + hybrid search** | Bare identifiers use token-boundary matching; natural-language queries use vector + lexical retrieval. |
Expand All @@ -98,6 +99,52 @@ that such agents can call.

---

## Canonical Asset Model (v2 Phase 2)

OpenMind v2 introduces a canonical content-identity model beneath the retrieval
index. The vector store becomes a *projection*; the durable source of truth is:

```text
Workspace the project (id p_*; the REST API still says /projects)
└── Asset one logical engineering object — a source/config file
└── Revision an immutable observation of that file's exact bytes
├── Segment a stable structural unit (Java type/method/constructor,
│ or a deterministic line-range) — verbatim or derived
└── Evidence a source-locatable citation, recoverable from the snapshot
```

- **Workspace vs Asset vs Revision.** A *Workspace* is the existing project. An
*Asset* is one file, identified by its normalized workspace-relative path (no
absolute path ever enters the portable database). A *Revision* is one observed
version of that file's bytes; the first observation is sequence 1, a change
creates the next sequence and supersedes the previous, and a revert (A → B → A)
is a new revision that reuses the old content blob.
- **Content snapshots.** Each revision's exact bytes are stored in an immutable,
content-addressed blob store (`data/<workspace>/objects/…`, keyed by SHA-256).
The database stores only the hash. This is why Evidence for a historical
revision stays readable after the source file changes on disk — and why vector
chunks are a *projection*, not the canonical store.
- **Unchanged ingestion reuses revisions.** Re-ingesting an unchanged file
creates no new revision and does not re-embed it. A Phase 1 workspace backfills
Assets on its first Phase 2 ingest *without* re-embedding unchanged files
(their existing Chroma chunks are reused).
- **Removed files preserve history.** Deleting a source file marks its Asset
`removed` and drops its live retrieval chunks, but keeps every revision,
segment, evidence and content blob. Only workspace *terminate* / *delete* wipe
Asset history.
- **Inspect it** through the CLI (`openmind asset list|show|revisions|segments|
evidence|add`), additive read-only REST endpoints under `/projects/{id}/…`,
and read-only MCP tools (`list_assets`, `get_asset`, `get_asset_revisions`,
`get_evidence`) usable straight from Claude Code.

Deliberately **not** in this phase (still later v2 work): PDF/DOCX/XLSX parsing,
requirement and business-rule extraction, Claim/Relation tables, Knowledge-Graph
edges and requirement-to-code traceability. Nothing here claims document
knowledge or traceability exists yet. Full design:
[docs/v2/phase-2-asset-model.md](docs/v2/phase-2-asset-model.md).

---

## Built For AI Agent Workflows

Open Mind is designed as infrastructure for practical AI agents and tool
Expand Down Expand Up @@ -286,6 +333,11 @@ python -m openmind.cli ingest --workspace <id> --wait
# what does it know?
python -m openmind.cli status --workspace <id>

# inspect the canonical Asset model: files -> revisions -> segments -> evidence
python -m openmind.cli asset list --workspace <id> --type source-code --json
python -m openmind.cli asset revisions --workspace <id> --asset <asset-id> --json
python -m openmind.cli asset evidence --workspace <id> --evidence <evidence-id> --json

# expose the knowledge layer to an editor or agent over MCP
python -m openmind.cli mcp serve

Expand Down Expand Up @@ -559,8 +611,10 @@ openmind/
domain/ typed application errors + the types crossing service calls
ports/ the three narrow boundaries services depend on
services/ use-case orchestration shared by CLI, MCP and FastAPI
(workspace, ingest, job, export, health + the container)
migrations/ versioned, checksummed SQLite schema migrations
(workspace, ingest, job, asset, export, health + the container)
content_store.py immutable SHA-256 content-addressed blob store (revision snapshots)
segmentation.py deterministic Segment + Evidence drafts (shares boundaries with rag)
migrations/ versioned, checksummed SQLite schema migrations (v0003 = Asset model)
walker.py selection-aware walk, .gitignore handling, hashing
detect.py manifest/language detection and stack cues
langspec.py declarative language registry
Expand Down Expand Up @@ -710,15 +764,19 @@ against the neutral fixture repos in `fixtures/`.

The following are not claimed as complete in the current build:

**v2 enterprise knowledge layer** — none of this is implemented. Phase 1
(shipped) is the tool-first runtime foundation described in
[docs/v2/phase-1-core-foundation.md](docs/v2/phase-1-core-foundation.md); it
deliberately creates extension points for the items below rather than building
them:

- enterprise Asset, Revision, Claim and Relation models;
- the engineering Knowledge Graph;
- PDF, DOCX and XLSX parsing; COBOL and JCL support;
**v2 enterprise knowledge layer.** Two foundation phases have shipped:
Phase 1 is the tool-first runtime
([docs/v2/phase-1-core-foundation.md](docs/v2/phase-1-core-foundation.md)), and
Phase 2 is the canonical **Asset / Revision / Segment / Evidence** model plus the
immutable content store
([docs/v2/phase-2-asset-model.md](docs/v2/phase-2-asset-model.md)). The following
later-phase items are **not** implemented; the foundation creates extension
points for them rather than building them:

- Claim and Relation models;
- the engineering Knowledge Graph and requirement-to-code traceability;
- PDF, DOCX and XLSX parsing; OCR; COBOL and JCL support; requirement and
business-rule extraction;
- cloud model providers (OpenAI, Anthropic, Bedrock, Azure, Vertex) — the
runtime remains local-first with no cloud credentials;
- requirement-to-code traceability, conflict detection and branch overlays;
Expand Down
60 changes: 57 additions & 3 deletions docs/cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,57 @@ running FastAPI server.
A count of `unknown` (`null` in JSON) means that store could not be read — it is
deliberately distinct from `0`, which means "read successfully, and empty".

### `asset`

Inspect the canonical **Asset model** (OpenMind v2 Phase 2): every indexed
file is an **Asset**, every observed version of it an immutable **Revision**,
each revision divided into **Segments**, each with source-locatable **Evidence**.
See [docs/v2/phase-2-asset-model.md](v2/phase-2-asset-model.md) for the model.

All subcommands take `--workspace` and support `--json` (exactly one object on
stdout). Lists are bounded and report a total; content is never printed unbounded.

```bash
# list assets (bounded; filter by type/state)
python -m openmind.cli asset list --workspace p_... --type source-code \
--state active --limit 100 --json

# one asset + its current-revision summary
python -m openmind.cli asset show --workspace p_... --asset a_... --json

# an asset's revision history (newest first)
python -m openmind.cli asset revisions --workspace p_... --asset a_... --json

# a revision's segments (each carries an evidence_id for the next step)
python -m openmind.cli asset segments --workspace p_... --revision r_... \
--limit 100 --json

# one evidence citation: locator, snapshot + current-source status,
# and bounded content recovered from the immutable snapshot
python -m openmind.cli asset evidence --workspace p_... --evidence e_... \
--max-chars 4000 --json

# ingest a single existing file that lives under a registered source root
python -m openmind.cli asset add --workspace p_... --path ./src/File.java \
--wait --json
```

`asset add` accepts **one existing file** under an already-registered source
root; a directory is rejected (exit `2`) with a pointer to `openmind add`. An
unsupported format is registered as an `unsupported` Asset — recorded honestly,
never falsely reported as parsed — and not ingested. Without `--wait` it returns
the job id.

`asset evidence` reports both a **snapshot** status (`available` / `corrupt` /
`missing`, recovered from the immutable content blob) and a **current source**
status (`matches` / `changed` / `missing`), so historical evidence stays readable
after the source file changes. Cross-workspace access to any asset/revision/
segment/evidence id is a typed not-found (exit `1`), never a leak.

`status` additionally reports asset counts (`assets_total` / `assets_active` /
`assets_removed`, `revisions`, `segments`, `evidence`) — additive; every prior
`status` key is unchanged.

### `export`

```bash
Expand All @@ -200,9 +251,12 @@ python -m openmind.cli mcp serve
```

Runs the MCP stdio server — the *same* implementation as
`python -m openmind.mcp_server`, not a second copy of the tools. The nine tools
(`search`, `route`, `dispatch`, `get_glossary`, `find_similar_cases`,
`save_case`, `get_doc`, `propose_fix`, `apply_fix`) are unchanged.
`python -m openmind.mcp_server`, not a second copy of the tools. The nine core
tools (`search`, `route`, `dispatch`, `get_glossary`, `find_similar_cases`,
`save_case`, `get_doc`, `propose_fix`, `apply_fix`) are unchanged. Phase 2 adds
four **read-only** Asset tools alongside them (`list_assets`, `get_asset`,
`get_asset_revisions`, `get_evidence`); merely serving MCP never starts the
ingestion worker.

stdout is the MCP transport, so all CLI chatter goes to stderr on this command.

Expand Down
23 changes: 19 additions & 4 deletions docs/database-migrations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,18 +75,33 @@ runner switches the connection to explicit-transaction mode
| --- | --- | --- |
| 1 | `baseline` | the schema as it stood before migrations existed: `projects`, `jobs`, `model_config`, `file_index`, `kv`, `ask_history` + its index |
| 2 | `paths_sidecar` | moves legacy in-DB project paths into the machine-local sidecar and blanks `projects.paths_json` |
| 3 | `asset_model` | the canonical Asset model: `assets`, `asset_revisions`, `segments`, `evidence` + their indexes. Additive — creates only new tables (all `IF NOT EXISTS`), touches no existing row |

`v0003` adds the OpenMind v2 canonical content-identity model. `assets`
references `projects(id)` and the whole subtree cascades on `ON DELETE CASCADE`,
so removing a project drops its assets, revisions, segments and evidence in one
statement (with the connection's `PRAGMA foreign_keys=ON`). `(asset_id,
content_hash)` is deliberately **not** unique so an A → B → A revert is
representable. Content bytes live in an immutable content-addressed blob store
(`data/<workspace>/objects/…`), never in the database — the DB stores only the
SHA-256 hash. See [docs/v2/phase-2-asset-model.md](v2/phase-2-asset-model.md).

### Upgrading an existing database

Nothing to do — open OpenMind and it migrates itself. Concretely:

```text
empty database -> v0001 creates every table -> ledger records 1, 2
legacy database -> v0001 statements are all no-ops -> ledger records 1, 2
(data untouched)
current database -> nothing to apply -> no writes
empty database -> v0001..v0003 create every table -> ledger records 1, 2, 3
legacy database -> v0001 statements are all no-ops, -> ledger records 1, 2, 3
v0002/v0003 apply additively (existing data untouched)
current database -> nothing to apply -> no writes
```

A Phase 1 database (already at v0002) upgrades to v0003 with no data loss:
`v0003` only *creates* the new Asset tables. The Asset rows for existing files
are then backfilled on the next ingestion — without re-embedding unchanged
files, reusing their existing Chroma chunks.

A legacy database is **baselined, not recreated**. `v0001` is written entirely
with `CREATE TABLE IF NOT EXISTS`, so against a database that already has those
tables every statement is a no-op and the runner simply records version 1.
Expand Down
Loading
Loading