Skip to content

[OMEGA-326] Chroma knowledge indexing fails under Landlock during compaction #295

Description

@PAY2109

Describe the bug
OmegaClaw fails to finish indexing Markdown knowledge priors when running under its default Landlock filesystem policy.

Indexing succeeds for several files, then ChromaDB fails during compaction:

chromadb.errors.InternalError: Error in compaction: Failed to apply logs to the metadata segment

Two conditions contribute to the failure:

  1. The writable Landlock permissions do not include AccessFs.REFER, which Chroma compaction requires for filesystem reparent/rename operations.
  2. lib_chromadb initializes Chroma with ./chroma_db, while src/rag.py defaults to /PeTTa/chroma_db. ChromaDB 1.5.9 uses the literal persistence path as its shared-system identifier, so these paths create separate Chroma systems even though they resolve to the same directory. The RAG system is then initialized after Landlock has been applied.

To Reproduce
Steps to reproduce the behavior:

  1. Add enough Markdown files to OmegaClaw-Core/knowledge-priors to trigger Chroma compaction. The reproduced case used 33 files totalling approximately 330 KB.

  2. Start OmegaClaw with:

    • Local embeddings
    • The default profile/policy.yaml
    • Persistent Chroma memory at /PeTTa/chroma_db
  3. Follow the container logs:

    docker logs --follow omegaclaw 2>&1 |
      grep --line-buffered -E 'Embedding type selected|\.md: (indexed|unchanged)|Knowledge(:| init failed)'
  4. Observe that several files are indexed before knowledge initialization fails.

Example output:

Embedding type selected is Local
  file1.md: indexed 20 chunks
  file2.md: indexed 3 chunks
  file3.md: indexed 14 chunks
  ...
Knowledge init failed: Error in compaction: Failed to apply logs to the metadata segment

The underlying traceback ends at collection.upsert():

Traceback (most recent call last):
  File "/PeTTa/repos/OmegaClaw-Core/src/rag.py", line 276, in init_knowledge
    collection.upsert(
  File "/usr/local/lib/python3.11/dist-packages/chromadb/api/models/Collection.py", line 530, in upsert
    self._client._upsert(
  File "/usr/local/lib/python3.11/dist-packages/chromadb/api/rust.py", line 541, in _upsert
    return self.bindings.upsert(
chromadb.errors.InternalError: Error in compaction: Failed to apply logs to the metadata segment

Expected behavior
OmegaClaw should index all available knowledge files and report successful completion:

Knowledge: 33 files (0 unchanged, 33 re-indexed)

Knowledge indexing should work with the default filesystem security policy and persistent memory configuration.

Actual behavior
Knowledge initialization stops partway through indexing with:

Knowledge init failed: Error in compaction: Failed to apply logs to the metadata segment

The Chroma database remains only partially populated. Restarting the container may index additional files, but compaction fails again before initialization completes.

Desktop (please complete the following information):

  • OS: N/A — reproduced in an Ubuntu Linux Docker deployment
  • Browser: N/A
  • Version: N/A

Smartphone (please complete the following information):

  • Device: N/A
  • OS: N/A
  • Browser: N/A
  • Version: N/A

Additional context

  • OmegaClaw version: v0.1.15
  • ChromaDB version: 1.5.9
  • py-landlock version: 0.1.1
  • Communication channel: WebSocket
  • Embedding provider: Local
  • Model provider: ASICloud

The Markdown files themselves are not the cause. Indexing the same 33 files into a fresh Chroma database without the Landlock policy succeeds.

The following two changes were tested together and resolved the issue:

  1. Add AccessFs.REFER to FileSystemPolicy.READ_WRITE_DIR_ACCESS:

    READ_WRITE_DIR_ACCESS = (
        AccessFs.READ_FILE
        | AccessFs.READ_DIR
        | AccessFs.WRITE_FILE
        | AccessFs.TRUNCATE
        | AccessFs.MAKE_REG
        | AccessFs.MAKE_DIR
        | AccessFs.MAKE_SYM
        | AccessFs.REMOVE_FILE
        | AccessFs.REMOVE_DIR
        | AccessFs.MAKE_FIFO
        | AccessFs.MAKE_SOCK
        | AccessFs.REFER
    )
  2. Make src/rag.py use the same default persistence-path identifier as lib_chromadb:

    DB_PATH = os.environ.get("CHROMA_DB_PATH", "./chroma_db")

After applying both changes, all 33 files indexed successfully, knowledge retrieval returned the expected content, and the container remained running with zero restarts.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingin-jiraThe issue has been accepted for fixing

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions