Replace local SLM embeddings with scope.models.embed()#3
Open
kriszyp wants to merge 8 commits into
Open
Conversation
Removes the bundled bge-small-en-v1.5 model and the harper-fabric-embeddings + node-llama-cpp dependency chain. Embeddings now run through Harper's new scope.models API (harper#510), which dispatches to whatever embedding backend the host has configured (Ollama on GPU hosts, Anthropic via OpenAI gateway, etc.) — zero model lifecycle to manage in the app. - lib/modelCapture.js: tiny Plugin API hook that stashes Scope on globalThis so Resource classes can reach scope.models. Wired up via `extensionModule:` in config.yaml. - lib/embeddings.js: now a one-line wrapper around scope.models.embed(). - package.json: bumps engines.harperdb to ^5.1, drops the SLM-only deps. - scripts/download-model.js and models/ removed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the @anthropic-ai/sdk + @anthropic-ai/vertex-sdk direct calls with a single scope.models.generate() call. Routes to whatever backend the host has configured for models.generative.default — vLLM on Fabric GPU hosts, Anthropic/OpenAI/Ollama elsewhere. Removes: - @anthropic-ai/sdk + @anthropic-ai/vertex-sdk deps - lib/config.js (LLM_PROVIDER / ANTHROPIC_API_KEY / VERTEX_*) - Anthropic web search tool + pause_turn handling - Per-token cost calculation (local model = no $) - Multi-block response stitching (scope.models returns plain content) Embeddings already used scope.models.embed() (prior commit on this branch). The Stats table now tracks cacheHits only; totalSaved is held at zero. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
UI was breaking on `meta.cost.saved` undefined. Bring back the
cost block but interpret it as a hypothetical — what each generation
WOULD have cost on Claude Sonnet 4.5 ($3/1M input, $15/1M output).
Local GPU compute is effectively $0; the dashboard tracks what we're
saving by self-hosting + the semantic cache.
- Estimated cost computed from token counts vLLM returns
- Stored on each assistant Message (same Float field as before)
- Cache hits credit the original message's cost to Stats.totalSaved
- meta.cost { input, output, total, saved } shape matches what
Chat.js's buildMeta() reads, so no UI changes needed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
scope.models.generate() only returns { content, finishReason } today —
the backend's usage info isn't surfaced to the caller. Approximate
input/output tokens from character counts so the estimated-Claude-cost
comparator on the Chat dashboard shows non-zero values.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs were stacking up:
1. The HNSW search returned matches that satisfy the threshold but didn't
guarantee distance-ascending iteration order, so the loop's first
match wasn't necessarily the closest. Collect candidates, compute
cosine distance explicitly, sort, then pick the closest valid one.
2. The threshold (0.12 cosine distance ≈ 0.88 similarity) was loose
enough that distinct queries about the same topic would collide
("describe the moon landing" vs "tell me about apollo 11"). Tighten
to 0.05 (≈ 0.95 similarity), which still catches near-paraphrases
and re-wordings while filtering out merely-related questions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous .find(role==='assistant') walked past any subsequent user messages and grabbed the next assistant reply even if it was many turns later in the conversation. With a long conversation like: user: Is soccer fun? assistant: ...soccer is fun... user: Is soccer fun (cache hit, no new assistant stored) user: Is soccer fun?? (cache hit, no new assistant stored) user: what is 2 plus 3 assistant: 2 plus 3 is 5 matching the second "Is soccer fun" returned "2 plus 3 is 5" — the next assistant message in chronological order, but completely unrelated. Only accept the IMMEDIATELY-following message as the reply. If the next message is another user message, skip this candidate and try the next. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Now that the immediate-next-message fix is in, the threshold can be relaxed to catch more rewordings without serving wrong answers. The 0.05 setting was paranoia about false positives that turned out to come from the slice/find bug. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kriszyp
force-pushed
the
kris/scope-models-embed
branch
from
July 23, 2026 01:26
95dab14 to
f5eed7b
Compare
kriszyp
marked this pull request as ready for review
July 23, 2026 13:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the bundled `bge-small-en-v1.5` model and the `harper-fabric-embeddings` + `node-llama-cpp` dependency chain. Embeddings now run through Harper's new `scope.models` API (harper#510) — on Fabric GPU hosts this lands at the shared Ollama process the host-manager spins up (host-manager#86).
Changes
Status
Test plan
Generated by Claude Sonnet 4.6.