Build in web search to help augment the brain...

[darron]

Sometimes there are things that exist but maybe aren't in my brain.

It would be good if we can build in web search to find and potentially add them to the brain if they aren't found.

Brave/Exa or maybe others would fit the bill - this would be optional.

[darron]

Initial draft thoughts on this:

# External Search RAG Plan

Status: proposal
Date: 2026-06-16
Issue: https://github.com/darron/dbrain/issues/40

## Summary

`dbrain` should add optional external search RAG for research and chat, but it
must not blur the boundary between the local brain and the open web.

The right first shape is:

1. Build the normal local `research_pack.v1` from SQLite, rendered notes, tags,
   transcripts, OCR, summaries, topic graphs, and related evidence.
2. Judge whether local evidence is enough for the question.
3. If the user explicitly enabled external search, run one bounded external
   retrieval pass through a provider plugin such as Brave or Exa.
4. Normalize provider results into evidence rows with explicit external-search
   lane provenance.
5. Synthesize from local and external evidence together, while making citations
   identify which claims came from `dbrain` and which came from the web.
6. Save the trace so bad answers can be reduced to local retrieval, external
   retrieval, merge, truncation, and citation decisions.

Adding external results to the permanent corpus is useful, but it should be a
later, explicit promotion workflow. Search results are noisy, duplicate-prone,
and often ephemeral. Auto-importing them during chat would turn ordinary
research into a silent web bookmarking system, which is not the right default.

## Strong Position

External RAG is worth doing, but not because the web should become the new
source of truth. It is worth doing because the current harness can correctly say
"your brain does not contain enough evidence," and the next useful action is
often "look outside, cite that outside evidence, and let me decide whether it
belongs in my brain."

The failure mode to avoid is an answer that looks like a normal dbrain answer
but is actually grounded in uncached provider snippets or a provider-generated
answer. That would poison the most important invariant in the project: local
evidence and model synthesis are separate. External search should extend the
retrieval set for one run, not silently create memory.

Confidence: high.

## Current State

The existing research stack already has the right seams:

- `internal/brainresearch` builds typed research packs.
- `internal/researchrun` runs a bounded, traced research/chat flow.
- `internal/ask` is the lower-level local evidence retrieval layer.
- `internal/retrieval` owns evidence DTOs, content sections, chunks, retrieval
  lanes, and retrieval signals.
- `retrieval.RetrievalLane` already has `Name`, `Status`, `Reason`,
  `Provider`, and `Store` fields, which are enough to represent an external
  search lane without inventing a second evidence model.
- `internal/researchtrace` persists JSON and Markdown traces under
  `data/research-runs`.
- `dbrain eval research` can compare trace behavior and should become the
  quality gate for external RAG.

The existing `docs/research-harness.md` plan already defers local semantic
retrieval until the lexical harness is traceable and eval-backed. External
search is different from semantic retrieval: it retrieves outside the corpus
rather than finding semantically similar local passages. It should still follow
the same discipline: explicit lane provenance, bounded calls, traces, and evals.

## Goals

- Let research and chat answer questions where the local brain is incomplete.
- Keep local evidence visibly distinct from external web evidence.
- Make external search provider-specific but runner-agnostic.
- Support Brave first and Exa second through a common provider contract.
- Keep all ordinary flows local-only unless the user enables external search.
- Preserve citation verification: every cited key must exist in the final pack,
  whether it is local or external.
- Save enough trace data to debug provider queries, returned URLs, ranking,
  truncation, and answer citations.
- Leave manual import/promotion as a later phase.

## Non-Goals

- Do not make external search the default for all research.
- Do not add external provider answers as authoritative evidence.
- Do not auto-import every external result into `sources`.
- Do not mutate upstream services.
- Do not add a separate daemon, browser extension, or sidecar service.
- Do not replace SQLite FTS, exact tags, related evidence, or local future
  semantic retrieval.
- Do not let external evidence hide that dbrain had weak or no local evidence.
- Do not require hosted inference for private local research. Search providers
  may be hosted; synthesis can still use the configured local model.

## Product Behavior

### Default Mode: Local Only

Normal research and chat should remain local-only.

```text
local retrieval -> local pack -> synthesis -> citation verification

This preserves the current behavior and avoids sending private questions to a
search provider unless the user opts in.

Manual External Mode

The user can enable external search for a single run:

local retrieval -> local judge -> external retrieval -> merged pack -> synthesis

The UI should show this as a clear mode, not a hidden behavior. Suggested web
control:

External web: Off | If local evidence is weak | Always for this turn
Provider: Brave | Exa | Brave then Exa
Freshness: Any time | 24h | 7d | 31d | 1y | Custom

Suggested CLI flags:

dbrain research "question" --runner --external-search=on-weak --external-provider=brave
dbrain research "question" --runner --external-search=always --external-provider=brave --external-freshness=pw
dbrain research "question" --retrieval-only --external-search=always --json

The first implementation can expose fewer flags, but the internal model should
be capable of these controls.

Current/Freshness Questions

Questions that obviously ask for recent state, such as "latest", "today",
"current", "recent", "what changed", or "as of now", should still require user
permission before external search. Once enabled, the runner should be willing to
use external results even when local evidence exists, because a local-first
archive can be stale by design.

The answer should make the boundary explicit:

• "Your local brain has evidence up to X."
• "External search found newer material from Y."
• "I used external evidence for the current-state claims."

Research Output Labels

Research output must clearly mark items that came from outside the dbrain
corpus. This is not cosmetic; it is part of the evidence contract.

Any external result shown in CLI, JSON, web Chat, Harness Lab, trace Markdown,
or public/share output should carry an explicit label such as:

Outside dbrain corpus / Brave / ephemeral
Outside dbrain corpus / Exa / ephemeral

Minimum displayed fields for each external item:

• provider name
• canonical URL
• title or provider title
• snippet/highlight/text role
• whether the row is ephemeral or promoted
• external source key, if citations need it

Minimum pack/output counters:

local_evidence_count
external_evidence_count
external_by_provider

For human text output, use blunt language:

Local dbrain evidence: 4 rows
Outside dbrain corpus: 3 rows from Brave

For JSON/API output, external rows should not be hidden in local evidence. Use a
separate external_evidence field or a clearly equivalent typed field. If a
client still renders one combined list, each external row must still include
kind=external, source_type=external_web, and an external_search retrieval
lane with store=ephemeral.

For synthesized answers, citations to external results should remain source-key
citation tokens, but the structured citation metadata should make the external
origin obvious:

{
  "source_key": "ext:brave:...",
  "origin": "outside_dbrain_corpus",
  "provider": "brave",
  "url": "https://example.com/article"
}

If an external result duplicates an existing local source, the research output
should prefer the local src:* citation and show a smaller signal such as
"also found by Brave." It should not show the duplicated row as outside-corpus
evidence unless the external provider supplied materially different text that
is actually used.

Later Promotion Mode

External results may later be promoted into dbrain:

external evidence -> user selects URLs -> linkadd/sourceenrich -> normal src:* rows

This should be a deliberate action such as:

dbrain research promote --trace data/research-runs/<run-id> --source-key ext:brave:<hash>

or a web button:

Add selected external result to dbrain

Promotion should reuse internal/linkadd and internal/sourceenrich so the
result becomes a normal src:* source with raw extracted text, extraction
status, summaries, note projection, and later search coverage. The initial RAG
feature should not write to the corpus.

Key Architecture Decision

External search should be a retrieval lane in the research harness, not a
second answer engine.

Desired dependency direction:

web Chat / CLI research / MCP research
        |
        v
internal/researchrun
        |
        v
internal/brainresearch
        |
        +--> local retrieval: internal/ask + internal/store
        |
        +--> external retrieval: internal/websearch providers

Do not put Brave or Exa calls in the web handler. Do not make MCP own external
search behavior. The runner should compose local and external retrieval because
the runner already owns judging weak evidence, progress events, synthesis,
verification, and trace persistence.

Package Shape

Add a provider-neutral package:

internal/websearch
internal/websearch/brave
internal/websearch/exa

If the implementation needs separate merge policy, add:

internal/externalrag

but keep it small. The runner should remain the orchestration owner.

Suggested provider contract:

package websearch

type Provider interface {
    Name() string
    Capabilities() Capabilities
    Search(ctx context.Context, req Request) (Response, error)
}

type Capabilities struct {
    Search        bool
    Contents      bool
    Freshness     bool
    DomainFilters bool
    Highlights    bool
    FullText      bool
    LocalContext  bool
}

type Request struct {
    Query           string
    Count           int
    Freshness       string
    IncludeDomains  []string
    ExcludeDomains  []string
    SearchLanguage  string
    Country         string
    MaxContentChars int
    MaxTokens       int
    Timeout         time.Duration
}

type Response struct {
    Provider  string
    RequestID string
    Query     string
    Results   []Result
    Warnings  []string
    RawMeta   map[string]any
}

type Result struct {
    ID          string
    URL         string
    CanonicalURL string
    Title       string
    Domain      string
    Snippet     string
    Highlights  []string
    Text        string
    Summary     string
    Author      string
    PublishedAt string
    CrawledAt   string
    Score       float64
    Rank        int
    Raw         json.RawMessage
}

Provider packages register themselves through a Go registry. That is the right
meaning of "plugin" for v1: compiled provider adapters behind a common
interface, enabled by config. Dynamic Go plugins or arbitrary shell plugins
would be the wrong default for a Homebrew-friendly single binary.

A later http_provider plugin can support custom provider endpoints without
adding a separate process to dbrain itself.

Evidence Model

External results should normalize into retrieval.EvidenceDocument while
keeping their external identity obvious.

Recommended fields:

source_key:     ext:<provider>:<urlhash>
kind:           external
source_type:    external_web
url:            canonical external URL
note_path:      empty
evidence_role:  external_snippet | external_text | external_summary
retrieval.lanes:
  - name: external_search
    status: used
    provider: brave | exa
    store: ephemeral
content_sections:
  - role: external_snippet
  - role: external_text, when provider returned extracted text

The source key must be stable enough for citation verification within a pack,
but it does not need to be a durable local corpus key. Use a deterministic hash
of provider plus canonical URL. If two providers return the same URL, merge
provider lanes onto one external evidence row.

Add a separate pack field rather than hiding external evidence in the local
evidence list:

ExternalEvidence []ask.Evidence `json:"external_evidence,omitempty"`

Synthesis can receive local evidence, exact-tag evidence, topic brief, and
external evidence in distinct sections. Citation verification should accept any
source key present in local Evidence, ExactTagEvidence, or
ExternalEvidence.

Coverage should add external counters:

ExternalEvidenceCount int      `json:"external_evidence_count,omitempty"`
ExternalByProvider    []Bucket `json:"external_by_provider,omitempty"`
ExternalFreshness     []Bucket `json:"external_freshness,omitempty"`
ExternalRecallNote    string   `json:"external_recall_note,omitempty"`

The UI can then say "3 local rows, 4 external rows" instead of pretending every
row came from the same store.

Duplicate Handling

Before adding an external result to the pack, resolve its canonical URL against
local sources.

Cases:

• URL already exists locally with usable extract/summary: prefer the local
  src:* evidence row and add an external_search retrieval signal saying the
  provider also found it.
• URL exists locally but extraction is missing or stale: show the local row and
  include a warning that promotion/repair would be needed for durable local
  coverage.
• URL does not exist locally: create an ext:* ephemeral evidence row.

This is crucial. If Brave finds a page already in dbrain, the answer should cite
the local source key, not an unnecessary external duplicate.

Ranking And Merge Policy

The merge policy should be conservative:

• local exact source-key hits outrank everything
• local exact user-tag evidence remains visibly local
• local lexical evidence with good concept coverage outranks generic external
  snippets
• external evidence can outrank local evidence for explicitly current/freshness
  questions
• primary sources, official docs, and original reporting should get source
  quality boosts
• duplicated URLs should collapse before ranking
• provider answer text should not be ranked as evidence in v1

External evidence rows should carry retrieval signals such as:

external_provider_rank
external_query_match
external_freshness
external_primary_source
external_duplicate_local_source

The system should explain why an external row appeared, especially when local
evidence was weak.

Search Policy

Add a typed external search policy:

type ExternalSearchMode string

const (
    ExternalSearchNever  ExternalSearchMode = "never"
    ExternalSearchOnWeak ExternalSearchMode = "on_weak_local_evidence"
    ExternalSearchAlways ExternalSearchMode = "always"
)

Runner behavior:

1. Build the local pack.
2. Judge local evidence with the existing researchrun.Judge.
3. If external mode is never, continue locally.
4. If mode is on_weak_local_evidence, call external search only when the
   judge verdict is no_evidence or weak_evidence.
5. If mode is always, call external search after local retrieval.
6. Merge local and external evidence.
7. Re-run citation prep and synthesis on the merged pack.

Keep the first version bounded:

• at most one external retrieval pass
• one or two provider queries derived from existing query variants
• 5 external URLs in the pack by default
• 10 external URLs maximum for normal chat
• 8,000 to 16,000 external-context tokens maximum
• stage timeout around 20 to 30 seconds

Provider failures should not collapse local research. If local evidence exists,
the answer can continue with a warning. If external search was required and no
local evidence exists, stop with a clear no-evidence/provider-failed state.

Brave Provider First

Brave should be the first provider.

Reason: Brave's current docs separate ordinary Web Search from LLM Context.
Their Web Search docs describe human-facing web results and point agent/chatbot
use cases toward LLM Context. The LLM Context docs describe pre-extracted
content, token budgets, URL limits, freshness filtering, relevance thresholds,
and source metadata for RAG/agent use.

Implementation target:

GET/POST https://api.search.brave.com/res/v1/llm/context
Header: X-Subscription-Token

Use LLM Context for the first RAG implementation:

• map grounding.generic[] snippets into external evidence rows
• map sources[url] metadata into title/domain/date fields
• use maximum_number_of_urls, maximum_number_of_tokens, and
  maximum_number_of_tokens_per_url as strict budget controls
• use freshness for recency-aware questions
• set context_threshold_mode=strict when precision matters more than recall

Keep Brave Web Search as a fallback or result-card provider:

GET https://api.search.brave.com/res/v1/web/search
extra_snippets=true

Do not use Brave Answers in v1. The point is to feed dbrain's own synthesis and
citation verifier, not outsource the answer.

Exa Provider Second

Exa should be the second provider after Brave is working.

Implementation target:

POST https://api.exa.ai/search
Header: x-api-key
Body: { "query": "...", "contents": { "highlights": true } }

Exa search can return result metadata plus highlights/text. Exa contents can
fetch text, summaries, and metadata for selected URLs:

POST https://api.exa.ai/contents
Body: { "urls": [...], "text": true, "highlights": true }

Use Exa as an alternate retrieval provider, especially for semantic web search
and richer content extraction.

Do not use Exa /answer in v1. It returns an LLM answer plus citations, which
is useful as a comparison tool but wrong as the primary evidence source. If
used later, treat it as a diagnostic or provider summary, never as authoritative
evidence.

Configuration

Extend config.yaml.sample with an opt-in external RAG section:

external_search:
  enabled: false # DBRAIN_EXTERNAL_SEARCH_ENABLED
  default_mode: "never" # DBRAIN_EXTERNAL_SEARCH_MODE; never, on_weak_local_evidence, always
  providers: ["brave"] # DBRAIN_EXTERNAL_SEARCH_PROVIDERS
  freshness: "" # DBRAIN_EXTERNAL_SEARCH_FRESHNESS; pd, pw, pm, py, or YYYY-MM-DDtoYYYY-MM-DD
  count: 5 # DBRAIN_EXTERNAL_SEARCH_COUNT
  max_tokens: 8192 # DBRAIN_EXTERNAL_SEARCH_MAX_TOKENS
  timeout: "30s" # DBRAIN_EXTERNAL_SEARCH_TIMEOUT
  brave:
    api_key: "" # BRAVE_SEARCH_API_KEY, DBRAIN_BRAVE_SEARCH_API_KEY; secret ref supported
    base_url: "https://api.search.brave.com/res/v1"
  exa:
    api_key: "" # EXA_API_KEY, DBRAIN_EXA_API_KEY; secret ref supported
    base_url: "https://api.exa.ai"

Secrets should use the existing runtimeenv.FirstNonEmptySecret path so config
can hold direct values, env:, op://, or keychain:// refs.

Preflight should check external provider credentials only when external search
is enabled or explicitly requested for that command. Tests should unset provider
keys unless they are testing preflight.

CLI, Web, And MCP

CLI

Add flags to dbrain research:

--external-search never|on-weak|always
--external-provider brave
--external-provider exa
--external-freshness pd|pw|pm|py|YYYY-MM-DDtoYYYY-MM-DD
--external-count 5

--retrieval-only --json should show local and external evidence separately so
provider behavior can be inspected without synthesis.

Web Chat

Add a compact control near the chat composer:

External web: Off / If local is weak / Always
Provider: Brave
Freshness: Any / 24h / 7d / 31d / 1y

Evidence cards should show external rows separately or badge them clearly:

External / Brave / ephemeral

Do not put external snippets into public chat shares unless the shared answer
already cites them. Shared pages should show external URLs as source URLs, not
local note paths.

MCP

Keep dbrain_research_pack local-only by default.

Add explicit optional arguments later:

{
  "allow_external_search": true,
  "external_search_mode": "on_weak_local_evidence",
  "external_providers": ["brave"]
}

MCP clients must not trigger outbound web searches accidentally. The MCP tool
description should say when it may call external providers and what provider
was used.

Trace Shape

Extend research traces with external events:

external_search_skipped
external_query_built
external_provider_request
external_provider_response
external_result_deduped
external_evidence_selected
external_merge_completed

Trace JSON should include:

• external search mode
• provider names
• query text sent to each provider
• freshness/window parameters
• result URLs and titles
• selected external evidence source keys
• provider errors and rate-limit warnings
• merge decisions, especially local duplicate replacement

Do not store API keys. Do not store raw provider payloads in ordinary traces if
they are huge or may contain irrelevant page content; store bounded normalized
payloads and, if needed, a redacted sidecar for debugging.

Eval And Test Plan

No CI test should call Brave or Exa.

Add fake providers:

internal/websearch/fake

Required tests:

• external search is off by default
• on_weak_local_evidence does not call providers when local evidence is enough
• weak/no local evidence triggers the fake provider when enabled
• provider failure preserves local-only answers when local evidence is enough
• external source keys pass citation verification when present in the final pack
• hallucinated external source keys fail citation verification
• external rows carry external_search lane provenance with provider and
  store=ephemeral
• external duplicates of local URLs collapse to local src:* evidence
• traces include external query/result/merge events without secrets
• web Chat renders external rows without mobile overflow
• public shares do not expose trace-only provider internals
• provider credentials are checked only when external search is selected

Add local eval support:

evals/local/external-rag/*.json

The eval format should allow fixture-backed external provider responses so
important cases can be replayed without live network access.

Useful eval assertions:

• expected local source keys
• expected external source keys
• expected external URLs
• expected provider names
• expected external evidence count
• forbidden provider calls
• expected stop reason when provider fails
• expected answer status when local and external evidence are both missing

Implementation Phases

Phase 0: Proposal

• Add this document.
• Link issue Build in web search to help augment the brain... #40 in the planning doc.
• No behavior changes.

Phase 1: Provider Contract And Fake Provider

• Add internal/websearch types and registry.
• Add fake provider for tests.
• Add config/env parsing for external search, but keep it disabled.
• Add trace event types for external search.
• Add no-op external policy plumbing to researchrun.Options.

Acceptance:

• local-only research behavior is unchanged
• tests prove external search is off by default
• traces can represent skipped external search

Phase 2: Brave Ephemeral Evidence

• Implement Brave LLM Context provider.
• Add CLI flags for dbrain research --runner.
• Normalize Brave snippets to ExternalEvidence.
• Merge duplicate local URLs.
• Extend synthesis input and citation verification.
• Add retrieval-only JSON output for external rows.

Acceptance:

• one CLI command can answer from local plus Brave external evidence
• Brave results are not written to SQLite
• citation verification accepts only source keys present in the merged pack
• provider failure is visible but does not destroy local-only research

Phase 3: Web Chat Experiment

• Add web controls for external search mode/provider/freshness.
• Stream external progress events.
• Badge external evidence in Chat and Harness Lab as outside the dbrain corpus.
• Save external retrieval events in traces.

Acceptance:

• web Chat can run local-only, on-weak, and always-external turns
• research output says which rows are outside the dbrain corpus
• Harness Lab can compare traces with external evidence
• public share pages remain clean and do not leak trace internals

Phase 4: Exa Provider

• Implement Exa search with highlights.
• Optionally call Exa contents for selected URLs under the same budget.
• Keep Exa Answer out of the v1 evidence path.
• Add provider comparison evals using fixtures.

Acceptance:

• --external-provider=exa works behind the same interface
• Brave and Exa evidence rows normalize to the same pack shape
• duplicate URLs merge across providers

Phase 5: Manual Promotion

• Add trace-based promotion for selected ext:* rows.
• Reuse internal/linkadd and internal/sourceenrich.
• Show promoted result as a normal src:* local source after enrichment.
• Optionally link the trace/run that led to promotion.

Acceptance:

• user can promote one selected external URL into dbrain
• promotion never happens automatically during ordinary chat
• promoted source preserves raw extracted text separately from summaries

Open Questions

• Should ExternalEvidence be a separate top-level pack field, or should all
  evidence live in one list with stronger lane labels? Separate is clearer, but
  a single list is simpler for older clients.
• Should external search be allowed from MCP at all, or should MCP get a
  separate dbrain_external_research_pack tool to avoid surprise network calls?
• Should Brave Web Search with extra_snippets=true be the first minimal
  implementation, or should the implementation go directly to LLM Context?
  Current evidence points to LLM Context because it is built for RAG.
• How much external text should traces retain by default?
• Should freshness-aware questions automatically suggest external search in the
  UI even when external search is off?
• Should promoted external results keep a relationship to the trace that found
  them?

References

• Issue Build in web search to help augment the brain... #40: Build in web search to help augment the brain... #40
• Current research harness: docs/research-harness.md
• Web research/chat proposal: docs/web-brain-research.md
• Brave Web Search docs:
  https://api-dashboard.search.brave.com/app/documentation/web-search/get-started
• Brave LLM Context docs:
  https://api-dashboard.search.brave.com/documentation/services/llm-context
• Exa Search docs: https://exa.ai/docs/reference/search
• Exa Contents docs: https://exa.ai/docs/reference/get-contents
• Exa Answer docs: https://exa.ai/docs/reference/answer