feat(ingest): add T3 thread SQLite source - #634
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
Warning Review limit reached
Next review available in: 24 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (14)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 21b9a1b. Configure here.
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| uri = f"file:{self.state_db_path}?mode=ro&immutable=0" | ||
| conn = sqlite3.connect(uri, uri=True, timeout=1.0, isolation_level=None) |
There was a problem hiding this comment.
🟠 High ingest/t3.py:161
_connect interpolates self.state_db_path directly into a SQLite URI string, so a database filename containing ?, #, or a % sequence is misinterpreted as URI query/fragment/escape syntax. SQLite then fails to open the intended file and raises OperationalError (or opens the wrong path) even though is_file() passed. Consider percent-encoding the path before appending the ?mode=ro&immutable=0 query string.
- uri = f"file:{self.state_db_path}?mode=ro&immutable=0"
+ from urllib.parse import quote
+
+ uri = f"file:{quote(str(self.state_db_path))}?mode=ro&immutable=0"🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/brainlayer/ingest/t3.py around lines 161-162:
`_connect` interpolates `self.state_db_path` directly into a SQLite URI string, so a database filename containing `?`, `#`, or a `%` sequence is misinterpreted as URI query/fragment/escape syntax. SQLite then fails to open the intended file and raises `OperationalError` (or opens the wrong path) even though `is_file()` passed. Consider percent-encoding the path before appending the `?mode=ro&immutable=0` query string.
| return payload if isinstance(payload, dict) else {} | ||
|
|
||
|
|
||
| def _t3_health_issue(payload: dict[str, Any]) -> HealthIssue | None: |
There was a problem hiding this comment.
🟡 Medium brainlayer/health_check.py:779
_t3_health_issue returns None (healthy) whenever the T3 health snapshot is absent, unreadable, or stale, because it only checks payload.get("alerting"). If the T3 ingestion service never starts or stops after writing a successful snapshot, the health check never reports a T3 issue, so a persistent ingestion failure is silently missed. Consider validating that the snapshot exists and is fresh (and/or checking the launchd label) before treating a missing payload as healthy.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/brainlayer/health_check.py around line 779:
`_t3_health_issue` returns `None` (healthy) whenever the T3 health snapshot is absent, unreadable, or stale, because it only checks `payload.get("alerting")`. If the T3 ingestion service never starts or stops after writing a successful snapshot, the health check never reports a T3 issue, so a persistent ingestion failure is silently missed. Consider validating that the snapshot exists and is fresh (and/or checking the launchd label) before treating a missing payload as healthy.
| conn = sqlite3.connect(uri, uri=True, timeout=1.0, isolation_level=None) | ||
| conn.execute("PRAGMA query_only=ON") | ||
| conn.execute("PRAGMA busy_timeout=1000") |
There was a problem hiding this comment.
🟡 Medium ingest/t3.py:162
_connect sets isolation_level=None, so the three SELECTs in _read_snapshot each run in autocommit mode rather than inside a single read transaction. Because T3 is a live database, a concurrent update committed between queries produces an inconsistent snapshot — e.g. a message whose thread_id was deleted after thread_rows was read is still loaded into messages_by_thread but then silently dropped because its thread is absent, while ingestion reports success. Start an explicit read transaction (BEGIN … COMMIT) around schema validation and all snapshot queries so the three reads observe a consistent database state.
| conn = sqlite3.connect(uri, uri=True, timeout=1.0, isolation_level=None) | |
| conn.execute("PRAGMA query_only=ON") | |
| conn.execute("PRAGMA busy_timeout=1000") | |
| uri = f"file:{self.state_db_path}?mode=ro&immutable=0" | |
| conn = sqlite3.connect(uri, uri=True, timeout=1.0) | |
| conn.execute("PRAGMA query_only=ON") | |
| conn.execute("PRAGMA busy_timeout=1000") | |
| conn.execute("BEGIN") | |
| return conn |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/brainlayer/ingest/t3.py around lines 162-164:
`_connect` sets `isolation_level=None`, so the three `SELECT`s in `_read_snapshot` each run in autocommit mode rather than inside a single read transaction. Because T3 is a live database, a concurrent update committed between queries produces an inconsistent snapshot — e.g. a message whose `thread_id` was deleted after `thread_rows` was read is still loaded into `messages_by_thread` but then silently dropped because its thread is absent, while ingestion reports success. Start an explicit read transaction (`BEGIN` … `COMMIT`) around schema validation and all snapshot queries so the three reads observe a consistent database state.
|
VERDICT: APPROVE |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 21b9a1b5ef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "allow_duplicate": True, | ||
| "message_id": message.message_id, | ||
| "project": thread.project_name, | ||
| "provenance_class": T3_PROVENANCE_CLASS, |
There was a problem hiding this comment.
When a T3 user message reaches auto-supersede before enrichment, this nonempty value prevents _chunk_to_claim() from deriving its trust class (provenance_autosupersede.py:295-299). Because Claim.rank maps unknown classes such as t3-thread to 0 (provenance.py:220-222), a direct user statement can lose to an agent paraphrase; retain t3 in source metadata and store the derived trust-lattice class here instead.
AGENTS.md reference: AGENTS.md:L5-L8
Useful? React with 👍 / 👎.
| result.chunks_indexed = _index_chunks( | ||
| chunks, | ||
| source_file=str(Path(state_db_path).expanduser()), | ||
| project=None, | ||
| db_path=destination, |
There was a problem hiding this comment.
When embedding or destination upsert raises, including when another process owns the single writer slot, read_threads() has already written alerting=false and execution never reaches the final health update. The periodic health check therefore continues reporting T3 as healthy even though this run wrote no chunks; catch indexing failures, persist an alerting ingestion-health snapshot, and then re-raise.
AGENTS.md reference: AGENTS.md:L5-L8
Useful? React with 👍 / 👎.
| if chunk.get("allow_duplicate"): | ||
| duplicate = None | ||
| dedupe_fields = compute_dedupe_fields(chunk["content"], created_at) |
There was a problem hiding this comment.
On a daily rerun where message text is unchanged but its project title, provider runtime, or mirror linkage changed, the deterministic T3 ID matches the existing row. This branch bypasses find_duplicate, but merge_existing_chunk_seen() subsequently returns early after updating only seen/tag-related fields (dedupe.py:622-680), so the new project and metadata never reach the ON CONFLICT update; route same-ID T3 rows through a metadata-refreshing upsert.
AGENTS.md reference: AGENTS.md:L5-L8
Useful? React with 👍 / 👎.
| failures=0 | ||
| enrichment_ok=0 | ||
| if ! install_many index drain watch hotlane-brainbar; then | ||
| if ! install_many index t3-ingest drain watch hotlane-brainbar; then |
There was a problem hiding this comment.
On hosts that use BrainLayer without T3 Code, install.sh all now unconditionally installs this RunAtLoad service even though ~/.t3/userdata/state.sqlite does not exist. The reader treats that absence as a fatal failure and writes an alerting snapshot consumed as a critical global health issue, so an otherwise valid installation becomes permanently unhealthy; condition this target on source availability or an explicit enable flag.
AGENTS.md reference: AGENTS.md:L5-L8
Useful? React with 👍 / 👎.
| message_rows = conn.execute( | ||
| """ |
There was a problem hiding this comment.
While the live T3 database is being updated, these independent autocommit queries can observe different commits. For example, a thread and message committed after thread_rows is materialized can appear only in message_rows, after which the message is silently discarded because no corresponding thread is iterated; use one short read transaction or a joined single-statement read so each ingestion run is internally consistent.
AGENTS.md reference: AGENTS.md:L5-L8
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_c247269d-14f5-45fa-bbbf-562dbcc30af2) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a19f7fab12
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| from ..paths import DEFAULT_DB_PATH | ||
|
|
||
| destination = DEFAULT_DB_PATH | ||
| result.chunks_indexed = _index_chunks( |
There was a problem hiding this comment.
When an edited T3 message produces fewer chunks than a previous ingestion—or a message/thread disappears—this call only upserts IDs present in the latest snapshot. Previously stored source='t3' rows and vectors for the surplus IDs are never archived or deleted, so retrieval can continue returning obsolete text alongside the updated message; reconcile stored T3 IDs against the completed snapshot within the writer path.
AGENTS.md reference: AGENTS.md:L6-L8
Useful? React with 👍 / 👎.
| if not payload.get("alerting"): | ||
| return None |
There was a problem hiding this comment.
If the scheduled agent fails before invoking Python or stops after one successful run, _load_json supplies either {} or an arbitrarily old alerting=false snapshot, and this branch suppresses the only T3 health issue indefinitely; run_health_check also does not inspect the T3 launchd label. Enabled T3 ingestion can therefore remain offline while global health reports it healthy, so validate snapshot presence and updated_at freshness or check service liveness.
AGENTS.md reference: AGENTS.md:L6-L8
Useful? React with 👍 / 👎.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_242770c2-db19-46d1-893c-2d33d6d9efa0) |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 34a04d5648
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| <key>Hour</key> | ||
| <integer>3</integer> | ||
| <key>Minute</key> | ||
| <integer>45</integer> |
There was a problem hiding this comment.
When the nightly index is delayed or configured beyond its 1,800-second default, this 03:45 job overlaps the index agent scheduled at 03:15; the checked writer wrapper uses a nonblocking pidfile lock and raises WriterInUseError rather than waiting, so the sole daily T3 run can be skipped until the following day. Schedule it with a safe gap or add writer-aware retry/serialization instead of starting at the index deadline boundary.
AGENTS.md reference: AGENTS.md:L28-L31
Useful? React with 👍 / 👎.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_1210cef9-d42b-436d-9979-362aec9f912b) |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1755c2b887
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if chunk.get("allow_duplicate"): | ||
| duplicate = None | ||
| dedupe_fields = compute_dedupe_fields(chunk["content"], created_at) |
There was a problem hiding this comment.
When a T3 message is edited without changing its stable chunk ID and the edit only changes casing, punctuation, or stopwords, this opt-out still falls through to merge_existing_chunk_seen(), whose normalized comparison treats the old and new content as identical. The subsequent early continue leaves both the stored text and embedding stale despite a successful ingestion; same-ID T3 rows should compare or refresh literal content before taking the seen-only path.
AGENTS.md reference: AGENTS.md:L5-L8
Useful? React with 👍 / 👎.
1 participant
What broke and how it was found
The T3 SQLite state database was never ingested as a first-class source. The new reader treats its thread projections as t3-thread chunks while keeping the live T3 database read-only.
Evidence
Review
This branch was pair-reviewed by a separate agent under the taste gate: smallest correct solution, behavioural tests, and no speculative machinery.
Provenance-column warning
chunks.provenance_class currently serves both as a trust ladder and a source class, with independent writers and no precedence rule. This PR is correct, but enrichment can overwrite its source tag until that collision is resolved. See design collision.
Enrichment and drain are currently disabled with launchctl to stop that decay; re-enable deliberately only after the column question is settled.
Note
Medium Risk
Touches chunk upsert/dedup and provenance persistence on the canonical DB plus a new scheduled ingestion path; schema drift is guarded but enrichment may still collide on
provenance_classper the PR note.Overview
Adds first-class T3 thread ingestion from T3’s live read-only
state.sqliteinto BrainLayer ast3-threadchunks.A new
brainlayer ingest-t3command and daily launchd job (com.brainlayer.t3-ingest, 03:45, run-at-load) drive the adapter.T3Readervalidates a minimal projection schema, snapshots threads/messages, joinsprojection_projectsfor human project titles (not raw UUIDs), and attaches provider runtime fields (t3_provider_name,t3_provider_session_id,t3_mirrored). Failures writet3-health.jsonand raiset3_schema_driftalarms.health-checkgains--t3-health-path, embeds the snapshot in results, and surfaces criticalt3_ingest_unhealthywhenalertingis true.pauseincludes the T3 launchd label.Indexing/upsert paths are extended for SQLite sources:
index_chunks_to_sqliteskips JSONL timestamp sniffing on.sqlite/.db, honors metadata for ids/timestamps/project/provenance, andVectorStorerespectsallow_duplicate(mirrored T3 content) and persistsprovenance_classwhen present. The deadread_t3_threadsexport is removed; tests and installer wiring cover the new surface.Reviewed by Cursor Bugbot for commit 1755c2b. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add T3 thread SQLite source ingestion pipeline to BrainLayer
T3Readerthat connects to a T3 SQLite DB, validates schema, reads threads/messages/projects, and emits per-message chunks with stable IDs, provenance metadata, and anallow_duplicateflag for mirrored content.ingest-t3CLI subcommand that orchestrates the read, filters to user/assistant roles, writes chunks to the index, and prints ingestion metrics; supports--dry-run.VectorStore.upsert_chunksin vector_store.py to bypass deduplication whenallow_duplicate=Trueand persistprovenance_classwhen the schema supports it..sqlite/.dbsources and instead use per-chunk metadata for identity and timestamps.ingest-t3daily at 03:45, and wires T3 health snapshot alerting into thehealth-checkcommand as a critical issue.allow_duplicate=Trueare inserted as separate rows regardless of content hash, which is intentional for mirrored T3 messages but departs from the existing deduplication contract.Macroscope summarized 1755c2b.