feat(core): add deterministic wiki projector - #1381
Conversation
Signed-off-by: phernandez <paul@basicmachines.co>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d02f1066ea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if direct_notes: | ||
| body.extend(["## Notes", ""]) | ||
| body.extend( | ||
| f"- [[{_without_markdown_suffix(note.path)}|{note.title}]]" for note in direct_notes |
There was a problem hiding this comment.
When a note has a valid multiline title such as Bad]]\n- relates_to [[evil, interpolating it directly closes the generated link and creates an additional semantic relation when index.md is parsed; project names and change titles have the same problem in the other rendering paths. Escape or reject Markdown delimiters and line breaks before constructing these canonical generated notes so user metadata cannot change their structure.
AGENTS.md reference: AGENTS.md:L156-L160
Useful? React with 👍 / 👎.
| direct_notes = sorted( | ||
| (note for note in notes if _parent_scope(note.path) == scope), | ||
| key=lambda note: (note.title.casefold(), note.path.casefold()), |
There was a problem hiding this comment.
When a case-sensitive project contains Foo.md and foo.md with titles that also compare equally under casefold(), both sort keys are identical and Python preserves whichever snapshot order the adapter supplied. Local and Cloud adapters can therefore produce different index.md bytes and checksums for the same project state; include the original title and path as deterministic tie-breakers.
Useful? React with 👍 / 👎.
| parsed = PurePosixPath(candidate) | ||
| if parsed.is_absolute() or any(part in {"", ".", ".."} for part in parsed.parts): | ||
| raise ValueError(f"Wiki path must be project-relative and normalized: {path}") |
There was a problem hiding this comment.
For an input such as C:/outside.md, PurePosixPath reports a relative path with no rejected components, so the planner accepts it and can emit C:/index.md. On Windows, joining that result to a project root switches to the specified drive, meaning an adapter that trusts this contract can fail or write outside the project; validate with PureWindowsPath.drive/is_absolute() rather than relying only on POSIX semantics.
Useful? React with 👍 / 👎.
1 participant
Why
Basic Memory Cloud Spec 88 requires local Basic Memory and Cloud to project the same accepted project state into byte-equivalent OKF Wiki documents. That deterministic contract belongs in Core so runtime adapters can share one source of truth without adding an agent dependency, queue assumption, or second filesystem.
This is the first Core layer in the Spec 88 stack. It deliberately stops at pure planning; project-partition persistence and Cloud scheduling/ledger integration follow in stacked PRs.
Related Cloud issue: https://github.com/basicmachines-co/basic-memory-cloud/issues/1863
What Changed
index.mdandlog.mddocuments.Implementation Details
src/basic_memory/indexing/wiki_projector.pyis pure and performs no I/O. Local and Cloud adapters supply an accepted snapshot and execute the returned canonical write plan through their existing mutation paths.index.mdandlog.mdchanges from the Wiki Projector are excluded from subsequent projection work and logs.generated.atderives from the accepted source boundary, so replaying identical input at the same watermark produces identical bytes.Testing
Automated
BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -p pytest_mock --no-cov -q tests/indexing/test_wiki_projector.py: 10 passed.uv run ruff format --check src/basic_memory/indexing/wiki_projector.py tests/indexing/test_wiki_projector.py: passed.uv run ruff check src/basic_memory/indexing/wiki_projector.py tests/indexing/test_wiki_projector.py: passed.uv run ty check src/basic_memory/indexing/wiki_projector.py tests/indexing/test_wiki_projector.py: passed.uv run ty check src tests test-int: passed after installing the repository-declared Milvus extra.Manual
tests/fixtures/wiki_projector/basic_projection.json.Risks / Follow-ups
WikiProjectionWritevalues through canonical checksum-protected accepted-note commands; direct storage writes are not compatible with this contract.