diff --git a/ROADMAP.md b/ROADMAP.md index b64658a..15a7ede 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -45,7 +45,7 @@ standards register names the rows that ask for them. | 025 | Encryption at rest, and the key-custody seam | 2 Tools | M | 010, 024 | planned | | 026 | Store-and-forward receipts for disconnected operation | 2 Tools | L | 024 | planned | | 030 | Persona (SOUL) + always-on memory tier | 3 Memory | M | 012 | planned | -| 031 | Session search (SQLite FTS5) | 3 Memory | S | 010 | planned | +| 031 | Session search (SQLite FTS5) | 3 Memory | S | 010 | done | | 032 | Embeddings + semantic memory + hybrid retrieval | 3 Memory | L | 031 | planned | | 033 | Project context: AGENTS.md | 3 Memory | S | 030, 022 | planned | | 034 | Export, import, restore | 3 Memory | S | 030, 031 | planned | diff --git a/config/config.exs b/config/config.exs index 6938876..c7bc28b 100644 --- a/config/config.exs +++ b/config/config.exs @@ -71,12 +71,16 @@ config :trinity, :tools, Trinity.Tools.FS.Grep, Trinity.Tools.Web.Fetch, Trinity.Tools.Web.Search, + # Slice 031: full-text search over past messages. + Trinity.Tools.SessionSearch, Trinity.Tools.Shell.Run ], toolsets: %{ fs: ["fs_read", "fs_write", "fs_edit", "fs_list", "fs_glob", "fs_grep"], web: ["web_fetch", "web_search"], - shell: ["shell"] + shell: ["shell"], + # Slice 031: search over past conversations. + memory: ["session_search"] } # Slice 022: the filesystem roots beside the data directory (always a root) and the session's diff --git a/config/test.exs b/config/test.exs index 22d94ba..5444eef 100644 --- a/config/test.exs +++ b/config/test.exs @@ -25,13 +25,17 @@ config :trinity, :tools, Trinity.Tools.FS.Grep, Trinity.Tools.Web.Fetch, Trinity.Tools.Web.Search, + # Slice 031: full-text search over past messages. + Trinity.Tools.SessionSearch, Trinity.Tools.Shell.Run ], toolsets: %{ core: ["echo", "sleep", "crash", "big", "write_note"], fs: ["fs_read", "fs_write", "fs_edit", "fs_list", "fs_glob", "fs_grep"], web: ["web_fetch", "web_search"], - shell: ["shell"] + shell: ["shell"], + # Slice 031: search over past conversations. + memory: ["session_search"] }, timeout_ms: 2_000 diff --git a/coverage.tsv b/coverage.tsv index 52998d3..4a5b6a2 100644 --- a/coverage.tsv +++ b/coverage.tsv @@ -11,3 +11,4 @@ slice_id percent sha date 023 75.39 5989b73 2026-09-20 003 75.39 39518c2 2026-09-20 024 76.55 f977b84 2026-09-20 +031 76.96 866dc3f 2026-09-20 diff --git a/docs/01-architecture.md b/docs/01-architecture.md index 7fa9961..478281e 100644 --- a/docs/01-architecture.md +++ b/docs/01-architecture.md @@ -74,7 +74,7 @@ without anything failing. |---|---|---| | `Trinity.Sessions` | Session process, turn loop, message log | LLM, Tools, **Effects**, Permissions, Memory, Skills, Repo, PubSub | | `Trinity.LLM` | Provider behaviour, req_llm adapter, model registry, streaming, usage | Repo (usage), Telemetry | -| `Trinity.Tools` | Tool behaviour, registry, execution runtime, core tools, and (as built at 024) the compile-time effect catalog `Trinity.Tools.Catalog`, because the registry reads it and Effects depends on Tools | Permissions, Sandbox, Repo | +| `Trinity.Tools` | Tool behaviour, registry, execution runtime, core tools, and (as built at 024) the compile-time effect catalog `Trinity.Tools.Catalog`, because the registry reads it and Effects depends on Tools | Permissions, Sandbox, Repo, **Memory** (as built at 031: `session_search` reads the index; Memory never depends on Tools) | | `Trinity.Permissions` | Policy, tier/1 (name-only), fingerprint-bound approvals, override adjudication | Repo, PubSub | | `Trinity.Effects` | The membrane; the runner in force (`Effects.Runner`, the executor `Tools.Runner` takes as a function); decision and query receipts; the boot receipt | **Tools**, Permissions, Authority, Receipts, Repo | | `Trinity.Authority` | Behaviour; `Local` implementation (the one caller of `execute/2` for effectful tools); selection at boot; `Staged` | Receipts, Repo | diff --git a/docs/05-data-model.md b/docs/05-data-model.md index af5405b..c31b4f9 100644 --- a/docs/05-data-model.md +++ b/docs/05-data-model.md @@ -48,6 +48,13 @@ turn ended that way); the role, seq and session never change. ### messages_fts (Slice 031): SQLite `fts5(content, session_id UNINDEXED, message_id UNINDEXED)`; on Postgres a `tsvector` generated column on `messages`. +As built: SQLite `messages_fts` with `tokenize = 'porter unicode61'`, `rowid` equal to the message row's, kept +by three triggers (`messages_fts_ai`, `_ad`, `_au` on `content`) and backfilled by the migration; rebuildable by +`mix trinity.search.reindex`. Postgres: `messages.content_tsv` generated as `to_tsvector('english', content)` with +the GIN index `messages_content_tsv_idx`; nothing to rebuild. Stemming is suffix-based on both: "running" meets +"runs" at `run` and never "ran". `Trinity.Memory.Search.messages/2` binds the query as a parameter and quotes +every term for FTS5, so operators are text. + ### memories (Slice 030/032) | column | type | notes | |---|---|---| diff --git a/lib/mix/tasks/trinity.search.reindex.ex b/lib/mix/tasks/trinity.search.reindex.ex new file mode 100644 index 0000000..9b2b2ab --- /dev/null +++ b/lib/mix/tasks/trinity.search.reindex.ex @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Mix.Tasks.Trinity.Search.Reindex do + @shortdoc "Rebuilds the message search index from the messages table" + + @moduledoc """ + Slice 031. On SQLite, empties `messages_fts` and refills it from `messages`, then optimises; + on Postgres the index is a generated column and there is nothing to rebuild, which the task + says. Prints the time. + + mix trinity.search.reindex + """ + use Boundary, classify_to: Trinity + use Mix.Task + + @impl Mix.Task + def run(_argv) do + Mix.Task.run("app.start") + {us, {:ok, what}} = :timer.tc(fn -> Trinity.Memory.Search.reindex() end) + + case what do + :rebuilt -> + Mix.shell().info("reindexed messages_fts in #{div(us, 1000)} ms") + + :generated_column -> + Mix.shell().info("nothing to rebuild: content_tsv is a generated column") + end + end +end diff --git a/lib/trinity.ex b/lib/trinity.ex index 0005393..8052023 100644 --- a/lib/trinity.ex +++ b/lib/trinity.ex @@ -28,6 +28,7 @@ defmodule Trinity do LLM, Memory, Memory.Tokens, + Memory.Search, Tools, Permissions, Permissions.Approval, diff --git a/lib/trinity/memory.ex b/lib/trinity/memory.ex index 46f4299..257305f 100644 --- a/lib/trinity/memory.ex +++ b/lib/trinity/memory.ex @@ -6,5 +6,5 @@ defmodule Trinity.Memory do to 032 add the tiers, the search and the semantic recall. It depends on the LLM and on the core, never on Sessions: the Session calls it and writes what it returns. """ - use Boundary, deps: [Trinity, Trinity.LLM], exports: [Tokens, Compactor] + use Boundary, deps: [Trinity, Trinity.LLM], exports: [Tokens, Compactor, Search] end diff --git a/lib/trinity/memory/search.ex b/lib/trinity/memory/search.ex new file mode 100644 index 0000000..3976260 --- /dev/null +++ b/lib/trinity/memory/search.ex @@ -0,0 +1,165 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.Search do + @moduledoc """ + Full-text search over every message (slice 031). SQLite: the `messages_fts` FTS5 table, + porter stemming, `bm25()` order, `snippet()`; Postgres: the generated `content_tsv` column, + `plainto_tsquery('english')`, `ts_rank`, `ts_headline`. Same shape of hit either way, so the + tool and the page do not know which database they run on. + + The query text is bound as a parameter and never spliced into SQL. On SQLite every term is + additionally quoted for FTS5, so a user's `"`, `*`, `-` or `OR` is text to find, not an + operator; the search is "all these words", stemmed, in any order. Stemming is a suffix + operation: "running" and "runs" meet at "run", and "ran" does not (NOTES.md, fact 2). + + Schemaless queries on purpose: Memory depends on the core and LLM, never on Sessions + (docs/01), so the tables are named here and the ids are cast through `Trinity.UUID`. + """ + + import Ecto.Query + + alias Trinity.Repo + + @default_limit 20 + @max_limit 100 + + # The adapter is fixed at compile time (config/config.exs, slice 010), so the two databases' + # query shapes are compiled in, not chosen at run time; the postgres job proves the other. + @adapter Application.compile_env(:trinity, :db_adapter, Ecto.Adapters.SQLite3) + + @type hit :: %{ + message_id: String.t(), + session_id: String.t(), + session_title: String.t() | nil, + seq: non_neg_integer(), + role: String.t(), + snippet: String.t(), + inserted_at: DateTime.t(), + rank: float() + } + + @doc """ + Ranked hits for `query`. Options: `limit:` (#{@default_limit}, at most #{@max_limit}), `role:` + (a message role), `persona_id:`, `since:` and `until:` (`DateTime`, on the message's + `inserted_at`). An empty or all-punctuation query is no hits. + """ + @spec messages(String.t(), keyword()) :: [hit()] + def messages(query, opts \\ []) when is_binary(query) do + case terms(query) do + [] -> [] + terms -> terms |> build(opts) |> Repo.all() |> Enum.map(&to_hit/1) + end + end + + @doc "The words of a query, punctuation dropped; the population FTS5 and tsquery both receive." + @spec terms(String.t()) :: [String.t()] + def terms(query) do + query + |> String.split(~r/[^\p{L}\p{N}_']+/u, trim: true) + |> Enum.map(&String.replace(&1, "'", "")) + |> Enum.reject(&(&1 == "")) + |> Enum.take(32) + end + + @doc "Rebuilds the index from `messages`; on Postgres the column is generated and this reports so." + @spec reindex() :: {:ok, :rebuilt | :generated_column} + if @adapter == Ecto.Adapters.SQLite3 do + def reindex do + Repo.transaction(fn -> + Repo.query!("DELETE FROM messages_fts") + + Repo.query!( + "INSERT INTO messages_fts(rowid, content, session_id, message_id) SELECT rowid, content, session_id, id FROM messages" + ) + + Repo.query!("INSERT INTO messages_fts(messages_fts) VALUES('optimize')") + end) + + {:ok, :rebuilt} + end + else + def reindex, do: {:ok, :generated_column} + end + + defp build(terms, opts) do + limit = opts |> Keyword.get(:limit, @default_limit) |> min(@max_limit) |> max(1) + + base() + |> match(terms) + |> filter(:role, opts[:role]) + |> filter(:persona_id, opts[:persona_id]) + |> filter(:since, opts[:since]) + |> filter(:until, opts[:until]) + |> limit(^limit) + end + + defp base do + from(m in "messages", + join: s in "sessions", + on: s.id == m.session_id, + select: %{ + message_id: type(m.id, Trinity.UUID), + session_id: type(m.session_id, Trinity.UUID), + session_title: s.title, + seq: m.seq, + role: m.role, + inserted_at: type(m.inserted_at, :utc_datetime_usec), + persona_id: type(s.persona_id, Trinity.UUID) + } + ) + end + + # SQLite: every term quoted for FTS5 (a double quote inside is doubled), joined by spaces, + # which FTS5 reads as AND; the whole string is one bound parameter. + if @adapter == Ecto.Adapters.SQLite3 do + defp match(query, terms) do + needle = Enum.map_join(terms, " ", &("\"" <> String.replace(&1, "\"", "\"\"") <> "\"")) + + from([m, s] in query, + join: f in "messages_fts", + on: f.rowid == m.rowid, + where: fragment("messages_fts MATCH ?", ^needle), + order_by: fragment("bm25(messages_fts)"), + select_merge: %{ + snippet: fragment("snippet(messages_fts, 0, '[', ']', '…', 12)"), + rank: fragment("bm25(messages_fts)") + } + ) + end + else + defp match(query, terms) do + needle = Enum.join(terms, " ") + + from([m, s] in query, + where: fragment("? @@ plainto_tsquery('english', ?)", m.content_tsv, ^needle), + order_by: [ + desc: fragment("ts_rank(?, plainto_tsquery('english', ?))", m.content_tsv, ^needle) + ], + select_merge: %{ + snippet: + fragment( + "ts_headline('english', ?, plainto_tsquery('english', ?), 'StartSel=[, StopSel=], MaxWords=12, MinWords=6')", + m.content, + ^needle + ), + rank: fragment("ts_rank(?, plainto_tsquery('english', ?))", m.content_tsv, ^needle) + } + ) + end + end + + defp filter(query, _key, nil), do: query + defp filter(query, :role, role), do: from([m, s] in query, where: m.role == ^role) + + defp filter(query, :persona_id, id), + do: from([m, s] in query, where: s.persona_id == type(^id, Trinity.UUID)) + + defp filter(query, :since, at), do: from([m, s] in query, where: m.inserted_at >= ^at) + defp filter(query, :until, at), do: from([m, s] in query, where: m.inserted_at <= ^at) + + defp to_hit(row) do + row + |> Map.delete(:persona_id) + |> Map.update!(:rank, &(&1 * 1.0)) + end +end diff --git a/lib/trinity/tools.ex b/lib/trinity/tools.ex index d4b56de..da6514a 100644 --- a/lib/trinity/tools.ex +++ b/lib/trinity/tools.ex @@ -10,8 +10,9 @@ defmodule Trinity.Tools do comparison of a session's declared surfaces with the calls its turns made (`surface_diff/1`, docs/07: a non-empty diff is a finding). """ + # Slice 031: Memory, for the core tools that read it (session_search; 032's recall follows). use Boundary, - deps: [Trinity, Trinity.Permissions], + deps: [Trinity, Trinity.Permissions, Trinity.Memory], exports: [Tool, Context, Result, Registry, Runner, Schema, Catalog] alias Trinity.Sessions.Message diff --git a/lib/trinity/tools/session_search.ex b/lib/trinity/tools/session_search.ex new file mode 100644 index 0000000..d647f10 --- /dev/null +++ b/lib/trinity/tools/session_search.ex @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Tools.SessionSearch do + @moduledoc """ + `session_search`: full-text search over every past message (slice 031, `Trinity.Memory.Search`). + A read: risk `:read`, effect `:none`, so it runs without asking and leaves a query receipt. + The hits are the user's own history and come back as text the model reads; they are + wrapped as untrusted all the same, because a past message may itself have carried + untrusted content (docs/07: a tool result re-entering the prompt is data, not command). + """ + @behaviour Trinity.Tools.Tool + + alias Trinity.Memory.Search + alias Trinity.Tools.{Context, Untrusted} + + @default_limit 10 + @max_limit 50 + + @impl true + def name, do: "session_search" + + @impl true + def description, + do: + "Searches every past conversation for the words given (stemmed, any order). Returns at most `limit` hits, newest-ranked first, each as `session · when · role: …snippet…` with the session id, so a decision or a fact from an earlier session can be found and quoted." + + @impl true + def schema, + do: %{ + "type" => "object", + "properties" => %{ + "query" => %{"type" => "string", "description" => "The words to find"}, + "limit" => %{ + "type" => "integer", + "minimum" => 1, + "maximum" => @max_limit, + "description" => "At most this many hits (default #{@default_limit})" + } + }, + "required" => ["query"], + "additionalProperties" => false + } + + @impl true + def risk, do: :read + + @impl true + def effect, do: :none + + @impl true + def execute(%{"query" => query} = args, %Context{}) do + limit = args |> Map.get("limit", @default_limit) |> min(@max_limit) + hits = Search.messages(query, limit: limit) + + text = + case hits do + [] -> "No message matches #{inspect(query)}." + hits -> Enum.map_join(hits, "\n", &line/1) + end + + meta = %{"query" => query, "hits" => length(hits), "limit" => limit} + {:ok, Untrusted.result(text, tool: name(), source_ref: "search:" <> query, meta: meta)} + end + + defp line(hit) do + title = hit.session_title || "untitled" + when_ = Calendar.strftime(hit.inserted_at, "%Y-%m-%d %H:%M") + "#{title} (#{hit.session_id}) · #{when_} · #{hit.role}: #{hit.snippet}" + end +end diff --git a/lib/trinity_web/live/search_live.ex b/lib/trinity_web/live/search_live.ex new file mode 100644 index 0000000..fc046fb --- /dev/null +++ b/lib/trinity_web/live/search_live.ex @@ -0,0 +1,117 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule TrinityWeb.SearchLive do + @moduledoc """ + `/search` (slice 031): a query box and the ranked hits from `Trinity.Memory.Search`, each + linking into its session at the message (`/s/:id#message-`, the chat's stream dom id). + The query lives in the URL (`?q=`), so a search is a link. The page reads; it writes nothing. + """ + use TrinityWeb, :live_view + + alias Trinity.Memory.Search + + @limit 50 + + @impl true + def mount(_params, _session, socket), + do: {:ok, assign(socket, page_title: gettext("Search"), q: "", hits: [])} + + @impl true + def handle_params(params, _uri, socket) do + q = params |> Map.get("q", "") |> String.trim() + hits = if q == "", do: [], else: Search.messages(q, limit: @limit) + {:noreply, assign(socket, q: q, hits: hits)} + end + + @impl true + def handle_event("search", %{"q" => q}, socket), + do: {:noreply, push_patch(socket, to: ~p"/search?#{[q: q]}")} + + def handle_event("new_session", _params, socket), + do: {:noreply, TrinityWeb.SessionLive.Index.new_session(socket)} + + def handle_event("cancel", _params, socket), do: {:noreply, socket} + + @impl true + def render(assigns) do + ~H""" + + <:bar> + {gettext("Search")} + + + + """ + end + + # The snippet's [brackets] mark the matches, rendered as ; everything else is text + # the template escapes. + attr :text, :string, required: true + + defp snippet(assigns) do + pieces = + assigns.text + |> String.split(~r/\[|\]/, include_captures: true) + |> Enum.reduce({[], false}, fn + "[", {acc, _} -> {acc, true} + "]", {acc, _} -> {acc, false} + piece, {acc, marked} -> {[{piece, marked} | acc], marked} + end) + |> elem(0) + |> Enum.reverse() + + assigns = assign(assigns, pieces: pieces) + + ~H""" + {piece}{piece} + """ + end +end diff --git a/lib/trinity_web/live/session_live/index.ex b/lib/trinity_web/live/session_live/index.ex index aeb0b58..1fd4f2b 100644 --- a/lib/trinity_web/live/session_live/index.ex +++ b/lib/trinity_web/live/session_live/index.ex @@ -29,6 +29,9 @@ defmodule TrinityWeb.SessionLive.Index do <:bar> {gettext("Sessions")} + <.link id="search-link" navigate={~p"/search"} class="text-meta opacity-70 hover:opacity-100"> + {gettext("search")} +
{gettext("receipts")} + <.link id="search-link" navigate={~p"/search"} class="text-meta opacity-70 hover:opacity-100"> + {gettext("search")} +
sqlite_up() + Ecto.Adapters.Postgres -> postgres_up() + end + end + + def down do + case repo().__adapter__() do + Ecto.Adapters.SQLite3 -> sqlite_down() + Ecto.Adapters.Postgres -> postgres_down() + end + end + + defp sqlite_up do + execute(""" + CREATE VIRTUAL TABLE messages_fts USING fts5( + content, + session_id UNINDEXED, + message_id UNINDEXED, + tokenize = 'porter unicode61' + ) + """) + + execute(""" + CREATE TRIGGER messages_fts_ai AFTER INSERT ON messages BEGIN + INSERT INTO messages_fts(rowid, content, session_id, message_id) + VALUES (new.rowid, new.content, new.session_id, new.id); + END + """) + + execute(""" + CREATE TRIGGER messages_fts_ad AFTER DELETE ON messages BEGIN + DELETE FROM messages_fts WHERE rowid = old.rowid; + END + """) + + execute(""" + CREATE TRIGGER messages_fts_au AFTER UPDATE OF content ON messages BEGIN + DELETE FROM messages_fts WHERE rowid = old.rowid; + INSERT INTO messages_fts(rowid, content, session_id, message_id) + VALUES (new.rowid, new.content, new.session_id, new.id); + END + """) + + execute(""" + INSERT INTO messages_fts(rowid, content, session_id, message_id) + SELECT rowid, content, session_id, id FROM messages + """) + end + + defp sqlite_down do + execute("DROP TRIGGER IF EXISTS messages_fts_ai") + execute("DROP TRIGGER IF EXISTS messages_fts_ad") + execute("DROP TRIGGER IF EXISTS messages_fts_au") + execute("DROP TABLE IF EXISTS messages_fts") + end + + defp postgres_up do + execute(""" + ALTER TABLE messages ADD COLUMN content_tsv tsvector + GENERATED ALWAYS AS (to_tsvector('english', coalesce(content, ''))) STORED + """) + + execute("CREATE INDEX messages_content_tsv_idx ON messages USING GIN (content_tsv)") + end + + defp postgres_down do + execute("DROP INDEX IF EXISTS messages_content_tsv_idx") + execute("ALTER TABLE messages DROP COLUMN IF EXISTS content_tsv") + end +end diff --git a/slices/031-session-search-fts/NOTES.md b/slices/031-session-search-fts/NOTES.md new file mode 100644 index 0000000..a5fcf2e --- /dev/null +++ b/slices/031-session-search-fts/NOTES.md @@ -0,0 +1,83 @@ +# Slice 031: NOTES + +## Two facts measured 2026-09-20 before any code + +**FTS5 is in the bundled SQLite.** exqlite 0.40.0 (the lock) carries SQLite 3.53.4 with `ENABLE_FTS3`, +`ENABLE_FTS4` and `ENABLE_FTS5` in `PRAGMA compile_options`; a `fts5(content, tokenize = 'porter unicode61')` +table creates, `snippet()` and `bm25()` answer. Whether the same holds in the Burrito binary is slice 100's +question, as SLICE.md's risk line says; this is the development and gate build. + +**Porter does not reach "ran".** In that table, `MATCH 'running'` returns "running late" and "it runs" (porter +stems both to `run`) and not "we ran the tests": an irregular past tense is not a suffix, and no stemmer maps it. +`MATCH 'ran'` finds "ran" alone. AC2 says "running" finds "ran"/"runs": the test proves "runs" and asserts that +"ran" is not found, stating the tokenizer's limit rather than claiming it away. On Postgres, `to_tsvector('english')` +(Snowball) stems the same way, and the test documents it the same way. Deviation (a), stated before code. + +## G1 plan, 2026-09-20 + +Tree at `d9ee12d` on `main` (024 approved, M2 reached); branch `slice/031-session-search-fts`; ROADMAP row 031 +to `in_progress` in this commit. Each line names its test. + +1. Migration `create_messages_fts`: on SQLite (branching on `repo().__adapter__()`), the virtual table + `messages_fts(content, session_id UNINDEXED, message_id UNINDEXED, tokenize = 'porter unicode61')` and the + three triggers (insert, update of `content`, delete) on `messages`, plus a backfill of the rows present; on + Postgres, a generated `content_tsv tsvector` column on `messages` (`to_tsvector('english', coalesce(content, + ''))`) with a GIN index. Test on both adapters (the postgres job): a message inserted through the Store is + found at once (AC1); an updated content is re-found; a deleted one is gone. +2. `Trinity.Memory.Search.messages(query, opts)`: SQLite through `MATCH` with `bm25()` order and `snippet()`; + Postgres through `plainto_tsquery` with `ts_rank` and `ts_headline`; hits as `%{message_id, session_id, + session_title, seq, role, snippet, inserted_at, rank}`; options `limit:` (20, capped at 100), `role:`, + `persona_id:`, `since:`, `until:`. The query text is bound as a parameter and never spliced; FTS5 syntax + characters are quoted so a user's `"` or `*` is text, not an operator. Tests: stemming (AC2, both halves), + filters, the empty query, an injection-shaped query. +3. `mix trinity.search.reindex` (SQLite: `INSERT INTO messages_fts(messages_fts) VALUES('rebuild')` after a + delete-all and a refill from `messages`; Postgres: a no-op that says the column is generated) and + `Trinity.Memory.Search.reindex/0` behind it. Test AC3: 10,000 generated messages, the incremental index's hit + counts for ten queries equal the rebuilt index's; the time printed. +4. Tool `Trinity.Tools.SessionSearch` (`session_search`, risk `:read`, effect `:none`, schema `query` and + `limit`), registered in `config :trinity, :tools` for every environment; the result is the hits as text the + model reads (`session`, `when`, `snippet`), capped by `limit`. Test: the tool through the runner returns at + most `limit` hits with snippets and is receipted as a read. +5. `/search` LiveView: a query box, results with the session title, the time, the role and the snippet, each + linking to `/s/:id#message-` (the chat's stream dom ids are `message-`; SLICE.md's `#m-` is + read as that anchor, deviation (b)); the chat's bar and the index page link to it. LiveView test. +6. docs/05 synced (the table as built), docs/01's Memory row mentions Search. + +Manual verification queue (two items, for the owner at G4): +- **AC4**: the tool returns at most `limit` hits with snippets, and the agent answers "what did we decide about + X last week" using it. I record a GIF under `proof/` from a real-provider run through + `scripts/dev_chat_on_test_registry.sh`; the owner watches it. +- **AC5**: the search page renders results and deep-links. A screenshot under `proof/` from the same run; the + owner opens the page. + +Deviations stated before any code: (a) AC2's "ran" is asserted as not found, with the reason (above); (b) the +deep link anchors on the chat's existing dom id `message-` rather than a new `m-`; (c) reindex on +Postgres is a documented no-op, because a generated column cannot be stale. + +## Findings, 2026-09-20 + +1. **The type checker knows the adapter.** `Trinity.Repo.__adapter__()` is a compile-time constant, so a + `case` over it is a "clause will never match" warning under `--warnings-as-errors`; the two databases' + query shapes are compiled in with `if @adapter == ...` around the definitions, which is also the truth: the + adapter is chosen at compile time (slice 010). The test's count helper reads the adapter from the application + environment at run time for the same reason. +2. **Schemaless selects do not cast.** `inserted_at` came back as text on SQLite and `Calendar.strftime/2` + refused it; the select casts it with `type/2`, as the ids are cast through `Trinity.UUID`. +3. **Tools now depends on Memory.** The `session_search` tool reads `Trinity.Memory.Search`; the boundary edge + is new and one-way (Memory never depends on Tools), and docs/01's row says so. +4. **`~s(...)` cannot hold an unbalanced parenthesis**; the operator-shaped test query uses `~s|...|`. +5. **Ranking on short rows**: bm25 put this week's question above last week's answer for "launch date" on the + screenshot; both are hits, and the page says which session each is. Ranking is not a criterion of this slice. +6. **The real-provider run** for AC4 went through `nvidia:nemotron` (the session's model, set at seed) with the + owner's `.env`, on the dev database, the dev server on port 4031 through a wrapper script; the model called + `session_search` unprompted beyond the question naming it, and answered from the hit. Killing the server by + `pkill -f phx.server` took the shell with it (exit 144), as the 013 notes warned; kill by pid. +7. **AC3's time**: 32 ms and 46 ms for 10,000 rows on SQLite here (two runs; `optimize` included); 0 ms on + Postgres because there is nothing to rebuild (a generated column), which the task and the test both say. + +## Follow-ups +- 032 (semantic search) fuses with this index; the `rank` field is bm25's on SQLite and `ts_rank` on Postgres, + not comparable across adapters, and 032 should normalise before fusing. +- The search page shows a scope of hits wholesale; a filter row (role, persona, dates) is a small addition when + the persona slice (030) gives it something to filter by. +- The receipts page's "open the covered range" follow-up from 024 can reuse the deep link shape (`#message-`). diff --git a/slices/031-session-search-fts/PROOF.md b/slices/031-session-search-fts/PROOF.md new file mode 100644 index 0000000..d77eb01 --- /dev/null +++ b/slices/031-session-search-fts/PROOF.md @@ -0,0 +1,122 @@ +# Proof for slice 031: Session search (FTS5) + +Agent: Trinity · Coding Agent · Date: 2026-09-20 · Branch: slice/031-session-search-fts · Final commit: (the commit carrying this file; named in the closing correction) + +## Summary +Full-text search over every message: an FTS5 table with porter stemming kept by triggers on SQLite, a generated +`tsvector` column with a GIN index on Postgres, one `Trinity.Memory.Search.messages/2` over both with the query +bound as a parameter and FTS5 terms quoted, a reindex task, the `session_search` core read tool, and the `/search` +page whose hits deep-link into the session at the message. AC2's "ran" half is asserted as not found, because +no stemmer maps an irregular past tense (NOTES.md, fact 2). The real-provider run for AC4 went through +`nvidia:nemotron`, which called the tool and answered from last week's session; the GIF and the screenshots are +in `proof/`. + +## Gate +``` +$ mix gate (this machine, OTP 28.5.0.5, Elixir 1.20.4, under a 32 GiB cgroup, tree 866dc3f) +1450 mods/funs, found no issues. +... SCAN COMPLETE ... +No retired or security advisory packages found +No vulnerabilities found. +Result: 326 passed, 17 excluded +trinity.coverage: 024 76.55% vs 003 75.39%: OK +plan_check: PASS +exit=0 +``` +CI, run 35545182679 on the tree at 866dc3f: `gate` success (326 passed, 17 excluded), `postgres` success +(318 passed, 25 excluded; the search tests included, `AC3: reindex of 10000 messages: generated_column, 0 ms`), +`fips-tag` and `fips` success. + +## Tests +``` +$ mix test --cover (tree 866dc3f) +Result: 326 passed, 17 excluded +| 100.00% | Trinity.Memory.Search | +| 100.00% | Trinity.Tools.SessionSearch | +| 94.87% | TrinityWeb.SearchLive | +| 76.96% | Total | +``` +`coverage.tsv` row: `031 76.96 866dc3f 2026-09-20` (from 76.55 at 024). + +The slice's nine tests (`--trace`): +``` +* test a query in the URL renders its hits, marked, linking to the session at the message [L#20] + * test a query in the URL renders its hits, marked, linking to the session at the message (82.9ms) [L#20] + * test submitting the form patches the URL and searches; nothing matching says so (5.8ms) [L#33] + * test the index and the chat link to the search page (18.2ms) [L#41] + * test registered as a core read tool with the catalog untouched (1.6ms) [L#30] + * test returns at most limit hits with snippets, names the session, and runs as a read with a query receipt (14.1ms) [L#38] + * test AC1: a message is searchable the moment it is inserted; updated content is re-found; a deleted one is gone (2.5ms) [L#26] + + * test AC3: reindex over 10,000 messages yields the same hit counts as the incremental index, and the time is printed (176.0ms) [L#99] + * test AC2: stemming: 'running' finds 'runs' (a suffix) and not 'ran' (irregular, no stemmer maps it) (1.8ms) [L#56] + * test filters: role, persona, since and until; limit capped; empty and operator-shaped queries (2.7ms) [L#71] +Result: 9 passed +``` + +## Acceptance criteria evidence + +### AC1 [auto]: inserting a message makes it searchable immediately (trigger) on SQLite and Postgres (tests) +`AC1: a message is searchable the moment it is inserted; updated content is re-found; a deleted one is gone` +(test/trinity/memory/search_test.exs), through `Trinity.Sessions.append_message/2` with no reindex between the +insert and the search; the same file on the postgres job (run 35545182679), where the generated column does +the same work. + +### AC2 [auto]: query with stemming: "running" finds "ran"/"runs" via porter on SQLite; documented behaviour on Postgres (test) +`AC2: stemming: 'running' finds 'runs' (a suffix) and not 'ran' (irregular, no stemmer maps it)`: measured +before code (NOTES.md, fact 2) and asserted as measured on both adapters: `running` returns the "runs" and +"running" rows and not "ran"; `ran` returns "ran". The criterion's "ran" is a claim porter (and Snowball on +Postgres) cannot meet, stated rather than claimed away: deviation (a). + +### AC3 [auto]: `reindex` on a DB with 10k messages completes and yields identical hit counts to the incremental index (test with generated data; time recorded) +`AC3: reindex over 10,000 messages yields the same hit counts as the incremental index, and the time is printed`: +10,000 generated rows through `insert_all` (so the triggers index them), ten queries counted over the index +before and after `Search.reindex/0`, equal. Time printed by the test: `rebuilt, 32 ms` and `46 ms` on two runs +here; `generated_column, 0 ms` on the postgres job (nothing to rebuild). `mix trinity.search.reindex` prints the +same line. + +### AC4 [manual]: tool returns ≤ `limit` hits with snippets; agent can answer "what did we decide about X last week" using it (manual GIF) +Automatic half: `returns at most limit hits with snippets, names the session, and runs as a read with a query +receipt` (test/trinity/tools/session_search_test.exs): `limit: 3` gives three lines, each naming last week's +session and carrying `[friday]`; `hits` and `limit` in the meta; the decision and query receipts; the no-hit +text; a limit past the schema refused. Manual half, recorded for the owner: `proof/ac4-session-search.gif` (64 +frames at four a second, 827 KB) and `proof/ac4-answer.png`: the dev server on `nvidia:nemotron` with the +owner's keys, a session seeded a week earlier with "We decided: the launch date is October 21st", a new session +asked "What did we decide about the launch date last week? Use session_search."; the model called +`session_search` (the tool row, `ok`) and answered "Last week the team agreed to set the Trinity 1.0 release +launch date for October 21st." The owner watches the GIF. + +### AC5 [manual]: search page renders results and deep-links (screenshot) +Automatic half (test/trinity_web/live/search_live_test.exs): `a query in the URL renders its hits, marked, +linking to the session at the message` (`decided`, `href="/s/#message-"`), `submitting the +form patches the URL and searches; nothing matching says so`, `the index and the chat link to the search page`. +Manual half: `proof/ac5-search.png` (`/search?q=launch date`: five hits across two sessions, the words marked, +each hit naming its session, time, role and seq) and `proof/ac5-deeplink.png` (the "Launch planning" hit +followed: the older session opens at the message). The owner opens the page. + +## Manual verification for the reviewer +- AC4: open `proof/ac4-session-search.gif`; or run `set -a; . ./.env; set +a; PORT=4031 mix phx.server` and ask a + new session what was decided about the launch date last week. +- AC5: open `proof/ac5-search.png` and `proof/ac5-deeplink.png`; or open `/search?q=launch date` on the same server. + +## Deviations from SLICE.md +(a) AC2's "ran" is asserted as not found, with the reason; (b) the deep link anchors on the chat's dom id +`message-` rather than a new `m-`; (c) reindex on Postgres is a documented no-op. All three stated in +NOTES.md before code. Found during the build: the Tools boundary gains Memory (NOTES finding 3, docs/01 as built). + +## Versions touched +`VERSIONS.md` updated: no. No dependency changed; FTS5 is in the locked exqlite's SQLite (NOTES.md, fact 1). + +## Git +``` +$ git log --oneline main..HEAD +866dc3f feat(s031): session_search in a memory toolset; the registry test lists it +cd3d952 feat(s031): the /search page with deep links into sessions; docs/05 as built +b8e6704 feat(s031): the session_search tool, a core read; Tools may depend on Memory +24dfa2a feat(s031): the FTS5 table and triggers (tsvector on Postgres), Trinity.Memory.Search, the reindex task +602d6ee docs(s031): G1 plan with FTS5 and the porter limit measured, and the slice opens +``` + +## Closing correction, 2026-09-20 +Supersedes the header's "Final commit" placeholder: the closing commit is `d642688` (`feat(s031): complete +slice 031 (session search, FTS5)`), and this correction rides on the commit after it. diff --git a/slices/031-session-search-fts/proof/ac4-answer.png b/slices/031-session-search-fts/proof/ac4-answer.png new file mode 100644 index 0000000..66d68e7 Binary files /dev/null and b/slices/031-session-search-fts/proof/ac4-answer.png differ diff --git a/slices/031-session-search-fts/proof/ac4-session-search.gif b/slices/031-session-search-fts/proof/ac4-session-search.gif new file mode 100644 index 0000000..b43f4ed Binary files /dev/null and b/slices/031-session-search-fts/proof/ac4-session-search.gif differ diff --git a/slices/031-session-search-fts/proof/ac5-deeplink.png b/slices/031-session-search-fts/proof/ac5-deeplink.png new file mode 100644 index 0000000..bc37254 Binary files /dev/null and b/slices/031-session-search-fts/proof/ac5-deeplink.png differ diff --git a/slices/031-session-search-fts/proof/ac5-search.png b/slices/031-session-search-fts/proof/ac5-search.png new file mode 100644 index 0000000..9f0e2a8 Binary files /dev/null and b/slices/031-session-search-fts/proof/ac5-search.png differ diff --git a/test/trinity/memory/search_test.exs b/test/trinity/memory/search_test.exs new file mode 100644 index 0000000..772087f --- /dev/null +++ b/test/trinity/memory/search_test.exs @@ -0,0 +1,161 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Memory.SearchTest do + @moduledoc """ + Slice 031 AC1 to AC3 on whichever adapter the suite compiled with (the postgres job runs the + same file). AC2's two halves are asserted as measured: stemming reaches "runs", not "ran". + """ + use Trinity.DataCase, async: false + + alias Trinity.Factory + alias Trinity.Memory.Search + alias Trinity.Sessions + + setup do + session = Factory.session!(%{title: "Planning the release"}) + {:ok, session: session} + end + + defp say(session, content, attrs \\ %{}) do + {:ok, m} = + Sessions.append_message(session.id, Map.merge(%{role: "user", content: content}, attrs)) + + m + end + + test "AC1: a message is searchable the moment it is inserted; updated content is re-found; a deleted one is gone", + %{session: session} do + assert Search.messages("zebra") == [] + m = say(session, "we saw a zebra at the station") + + assert [ + %{ + message_id: id, + session_title: "Planning the release", + role: "user", + snippet: snippet, + seq: 1 + } + ] = + Search.messages("zebra") + + assert id == m.id + assert snippet =~ "[zebra]" + + Trinity.Repo.update_all(from(x in "messages", where: x.id == type(^m.id, Trinity.UUID)), + set: [content: "the giraffe instead"] + ) + + assert Search.messages("zebra") == [] + assert [%{message_id: ^id}] = Search.messages("giraffe") + + Trinity.Repo.delete_all(from(x in "messages", where: x.id == type(^m.id, Trinity.UUID))) + assert Search.messages("giraffe") == [] + end + + test "AC2: stemming: 'running' finds 'runs' (a suffix) and not 'ran' (irregular, no stemmer maps it)", + %{session: session} do + say(session, "we ran the tests yesterday") + say(session, "the job runs nightly") + say(session, "running late again") + + found = Search.messages("running") |> Enum.map(& &1.snippet) + assert length(found) == 2 + assert Enum.any?(found, &(&1 =~ "[runs]")) + assert Enum.any?(found, &(&1 =~ "[running]")) + refute Enum.any?(found, &(&1 =~ "ran")) + assert [%{snippet: s}] = Search.messages("ran") + assert s =~ "[ran]" + end + + test "filters: role, persona, since and until; limit capped; empty and operator-shaped queries", + %{session: session} do + other = Factory.session!(%{title: "Other persona"}) + say(session, "decision: ship on friday") + say(session, "decision noted", %{role: "assistant"}) + say(other, "decision: postpone") + + assert length(Search.messages("decision")) == 3 + assert [%{role: "assistant"}] = Search.messages("decision", role: "assistant") + assert length(Search.messages("decision", persona_id: session.persona_id)) == 2 + assert Search.messages("decision", until: ~U[2000-01-01 00:00:00Z]) == [] + assert length(Search.messages("decision", since: ~U[2000-01-01 00:00:00Z])) == 3 + assert length(Search.messages("decision", limit: 1)) == 1 + assert Search.messages("") == [] + assert Search.messages(" ,,, ") == [] + # Operators and quotes are text to find, never syntax: no error, no hits for the junk. + assert Search.messages(~s|decision" OR * -x NEAR(a b)|) |> is_list() + assert Search.messages(~s|"decision"|) |> length() == 3 + + assert Search.terms(~s|it's "quoted" and-hyphenated|) == [ + "its", + "quoted", + "and", + "hyphenated" + ] + end + + @tag timeout: 300_000 + test "AC3: reindex over 10,000 messages yields the same hit counts as the incremental index, and the time is printed", + %{session: session} do + words = + ~w(alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon) + + now = DateTime.utc_now() + + rows = + for i <- 1..10_000 do + %{ + id: Trinity.UUID.generate(), + session_id: session.id, + seq: i, + role: "user", + content: + "message #{i} #{Enum.at(words, rem(i, 20))} #{Enum.at(words, rem(i * 7, 20))} #{Enum.at(words, rem(i * 13, 20))}", + parts: %{}, + provider_meta: %{}, + inserted_at: now, + updated_at: now + } + end + + rows + |> Enum.chunk_every(1_000) + |> Enum.each(fn chunk -> Trinity.Repo.insert_all(Trinity.Sessions.Message, chunk) end) + + queries = Enum.take(words, 10) + before = Map.new(queries, &{&1, length(Search.messages(&1, limit: 100))}) + assert Enum.all?(before, fn {_, n} -> n == 100 end) + counts_before = Map.new(queries, &{&1, count(&1)}) + + {us, {:ok, what}} = :timer.tc(&Search.reindex/0) + IO.puts("AC3: reindex of #{length(rows)} messages: #{what}, #{div(us, 1000)} ms") + + counts_after = Map.new(queries, &{&1, count(&1)}) + assert counts_after == counts_before + assert Enum.all?(counts_before, fn {_, n} -> n >= 500 end) + end + + # A count over the index without the limit, for the AC3 comparison. The adapter is read from + # the application environment at run time here so the other branch is not a type warning. + defp count(term) do + case Application.get_env(:trinity, :db_adapter) do + Ecto.Adapters.SQLite3 -> + %{rows: [[n]]} = + Trinity.Repo.query!("SELECT count(*) FROM messages_fts WHERE messages_fts MATCH ?", [ + "\"#{term}\"" + ]) + + n + + _ -> + %{rows: [[n]]} = + Trinity.Repo.query!( + "SELECT count(*) FROM messages WHERE content_tsv @@ plainto_tsquery('english', $1)", + [term] + ) + + n + end + end +end diff --git a/test/trinity/tools/registry_test.exs b/test/trinity/tools/registry_test.exs index 8ffcf0b..edfe8d2 100644 --- a/test/trinity/tools/registry_test.exs +++ b/test/trinity/tools/registry_test.exs @@ -30,6 +30,7 @@ defmodule Trinity.Tools.RegistryTest do "fs_list", "fs_read", "fs_write", + "session_search", "shell", "sleep", "web_fetch", @@ -65,6 +66,7 @@ defmodule Trinity.Tools.RegistryTest do "fs_list", "fs_read", "fs_write", + "session_search", "shell", "sleep", "web_fetch", diff --git a/test/trinity/tools/session_search_test.exs b/test/trinity/tools/session_search_test.exs new file mode 100644 index 0000000..451c4ad --- /dev/null +++ b/test/trinity/tools/session_search_test.exs @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Tools.SessionSearchTest do + @moduledoc "Slice 031 AC4's automatic half: the tool through the runner in force, at most limit hits, snippets, a read's receipts." + use Trinity.DataCase, async: false + + alias Trinity.{Effects, Factory, Receipts, Sessions} + alias Trinity.Tools.Context + + setup do + old = Factory.session!(%{title: "Last week's planning"}) + + for i <- 1..5, + do: + {:ok, _} = + Sessions.append_message(old.id, %{ + role: "user", + content: "we decided to ship the #{i}th widget on friday" + }) + + {:ok, _} = + Sessions.append_message(old.id, %{role: "assistant", content: "noted: friday it is"}) + + here = Factory.session!(%{title: "Now"}) + scope = Receipts.session_scope(here.id) + on_exit(fn -> Receipts.stop_writer(scope) end) + {:ok, here: here, old: old, scope: scope} + end + + test "registered as a core read tool with the catalog untouched" do + assert {:ok, %{kind: :core, risk: :read, effect: :none}} = + Trinity.Tools.lookup("session_search") + + assert Trinity.Permissions.tier("session_search") == :read + refute "session_search" in Trinity.Tools.Catalog.names() + end + + test "returns at most limit hits with snippets, names the session, and runs as a read with a query receipt", + %{here: here, old: old, scope: scope} do + ctx = %Context{session_id: here.id, caller: here.id} + call = %{id: "c1", name: "session_search", args: %{"query" => "decided friday", "limit" => 3}} + assert {:ok, %{content: text, meta: meta}, _} = Effects.Runner.run(call, ctx) + lines = String.split(text, "\n") + assert length(lines) == 3 + assert Enum.all?(lines, &(&1 =~ "Last week's planning (#{old.id})" and &1 =~ "[friday]")) + assert meta["hits"] == 3 and meta["limit"] == 3 + assert Enum.map(Receipts.list(scope), & &1.kind) == ["decision", "query"] + + assert {:ok, %{content: "No message matches \"zzz\"."}, _} = + Effects.Runner.run( + %{id: "c2", name: "session_search", args: %{"query" => "zzz"}}, + ctx + ) + + assert {:error, {:invalid_args, _}, _} = + Effects.Runner.run( + %{id: "c3", name: "session_search", args: %{"query" => "x", "limit" => 500}}, + ctx + ) + end +end diff --git a/test/trinity_web/live/search_live_test.exs b/test/trinity_web/live/search_live_test.exs new file mode 100644 index 0000000..af43372 --- /dev/null +++ b/test/trinity_web/live/search_live_test.exs @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule TrinityWeb.SearchLiveTest do + @moduledoc "Slice 031 AC5's automatic half: the page renders hits with marks and deep-links into the session at the message." + use TrinityWeb.ConnCase, async: false + + import Phoenix.LiveViewTest + + alias Trinity.{Factory, Sessions} + + setup do + session = Factory.session!(%{title: "Release planning"}) + + {:ok, m} = + Sessions.append_message(session.id, %{role: "user", content: "we decided to ship on friday"}) + + {:ok, session: session, message: m} + end + + test "a query in the URL renders its hits, marked, linking to the session at the message", %{ + conn: conn, + session: session, + message: m + } do + {:ok, view, html} = live(conn, ~p"/search?q=decided") + assert html =~ "1 hits for decided" + assert has_element?(view, "#hit-#{m.id}") + assert html =~ "decided" + assert html =~ "Release planning" + assert html =~ ~s(href="/s/#{session.id}#message-#{m.id}") + end + + test "submitting the form patches the URL and searches; nothing matching says so", %{conn: conn} do + {:ok, view, _} = live(conn, ~p"/search") + html = view |> form("#search-form", q: "zebra") |> render_submit() + assert_patch(view, "/search?q=zebra") + assert html =~ "Nothing matches zebra." + refute has_element?(view, "#hits") + end + + test "the index and the chat link to the search page", %{conn: conn, session: session} do + {:ok, view, _} = live(conn, ~p"/") + assert has_element?(view, "#search-link") + {:ok, view, _} = live(conn, ~p"/s/#{session.id}") + assert has_element?(view, "#search-link") + end +end