refactor: KB server microservice + library manager microservice + LAMB integration#356
Draft
NoveliaYuki wants to merge 86 commits into
Draft
refactor: KB server microservice + library manager microservice + LAMB integration#356NoveliaYuki wants to merge 86 commits into
NoveliaYuki wants to merge 86 commits into
Conversation
Standalone knowledge-base microservice on port 9092, coexisting with
the stable server on 9090 (ADR-2). Pure computation service: LAMB owns
access control and delivers content + permalinks in each request; the
KB server never calls the Library Manager (ADR-1, ADR-6).
- Three plugin families with ENABLE/DISABLE env gating:
* Vector DB: chromadb (default), qdrant (optional)
* Chunking: simple, hierarchical (parent-child), by_page, by_section
* Embedding: openai, ollama, local (optional, sentence-transformers)
- Per-org filesystem isolation at data/storage/{org}/{collection}/ (ADR-9)
- Async SQLite-backed ingestion queue with crash recovery, semaphore
concurrency, and in-memory-only embedding credentials (ADR-4)
- Locked store setup after creation (ADR-3); only name/description mutable
- Permalinks attached to every chunk for RAG citation propagation
- 13 endpoints (collections CRUD, add-content, delete-by-source, query,
job status, health/backends/strategies/vendors listings)
- Bearer-auth only (timing-safe compare); startup refuses without token
- Non-root Dockerfile, healthcheck, WAL-mode SQLite, single-instance lock
- 50 pytest tests, 84% coverage, ruff clean
- README documents setup, API, config, security, and all 11 ADRs
LAMB backend integration, lamb-cli kb commands, and the Svelte UI are
deferred to a follow-up issue.
Splits the existing flat tests/ into tiered structure with per-tier fixtures and a combined coverage gate. Brings backend/ from 84% to 99% line+branch coverage across 572 tests: - 333 unit tests: direct module tests for database, config, plugins (chunking, embedding, vector DB), services, schemas. Real SQLite, real ChromaDB, real local-mode Qdrant, FakeEmbedding for speed. - 178 integration tests: ASGI in-process via httpx + worker lifecycle for routers, async worker concurrency/timeout/recovery, main lifespan, request logging middleware. - 61 e2e tests: real uvicorn subprocess + docker stack (Ollama for embeddings, Qdrant container) + real HTTP. Covers full pipeline matrix, multi-tenant isolation, SIGKILL crash recovery, 20-job concurrency back-pressure, auth boundary edge cases, and the full HTTP error-code matrix. Source fix: qdrant_backend.py now uses query_points() instead of the removed search() method (qdrant-client>=1.12 API). Infrastructure: docker-compose.test.yml (Qdrant + Ollama), scripts/run_tests.sh per-tier runner with --cov-fail-under=95 gate, mutmut config, vcrpy dev dep, pytest tier markers with auto-tagging. Latent bugs surfaced (documented as regression guards): - Latin-1 token bytes cause 500 not 401 (hmac.compare_digest TypeError) - extra_metadata dict[str, Any] accepts None/nested but ChromaDB rejects - collection.chunk_count race under concurrent ingestion (RMW counter) - chunking_params silently ignores unknown keys
Adds three manual mutation-testing scripts that apply curated, high-
impact mutations to backend/services/ingestion_service.py and
backend/tasks/worker.py, run the test subset for each, and revert via
git checkout.
Result: 32/32 mutations killed (100% on the curated set).
Mutations exercised:
- Negated null/empty guards
- Swapped success/failure status assignments
- Inverted batch-size comparisons (5→4, 5→6, 5→1)
- Counter direction reversals (+= → -=)
- > vs >= boundary off-by-ones
- max → min clamp inversions
- _dispatched.add → discard (worker dedup)
- Poll-interval and max-attempts constant inflation
mutmut>=3's coverage-based test selection couldn't link mutants to
our class-based test layout ("no test case for any mutant"); the
manual scripts are the working alternative. Mutmut config retained
in pyproject.toml for future tooling versions.
Bearer tokens with non-ASCII characters caused hmac.compare_digest to raise TypeError, surfaced by FastAPI as 500 instead of 401. Wrap the comparison in try/except so non-ASCII tokens fall through the equality branch and the caller gets a clean 401.
extra_metadata was typed dict[str, Any], accepting None values and nested dicts that ChromaDB later rejects deep in the pipeline as a 500. Narrow the type to dict[str, str | int | float | bool] and add a field validator so bad values are rejected at request boundary as 422 with a clear error message.
Each chunking strategy reads only the keys it recognises, silently dropping unknown keys (typos, cross-strategy keys). Misconfiguration produced no signal at the API boundary. Add validate_chunking_params helper that checks the param dict against the strategy's get_parameters() allow-list. Wire it into: - collection-create endpoint (returns 422 with bad keys named) - ingestion dispatcher (defense-in-depth for direct DB writes) Test-only chunkers (sleep_chunker, fail_on_nth_chunker) declare get_parameters() so they remain compatible with the standard payload.
Counter update was an ORM read-modify-write against the SQLite row. Under concurrent ingestion jobs against the same collection, two workers could both read N, both compute N+k, both write N+k — losing contributions. Vectors stayed correct; only the displayed chunk_count drifted. Replace RMW with atomic SQL UPDATE so SQLite serializes the increment. Apply the same pattern to deletion via CASE expressions clamping at 0. The 20-job concurrency e2e test now asserts strict equality with the sum of per-job chunks_created, removing the deliberately-weak '> 0' guard. Also strengthen the Ollama healthcheck in the e2e fixture: /api/tags returns 200 before nomic-embed-text finishes pulling, so tests racing the pull saw 404s. Wait for the model to appear in /api/tags before yielding the stack. Update mutation script: target the new atomic UPDATE patterns and use .venv/bin/pytest so mutations are actually exercised.
The extended/final mutation scripts called bare `pytest`, which is not on PATH — every "kill" was exit 127 (command not found), masking real survivors as killed. Wire them through .venv/bin/pytest like the main script. Also update one EXT pattern that no longer matches after the atomic UPDATE refactor (delete_vectors clamp). Real kill rates after fix: - main: 17/17 killed (legitimate exit 1) - extended: 13/13 killed (2 pre-existing pattern-not-found in worker.py) - final: 2/2 killed Total: 32/32, matching the prior claim with honest exit codes.
The test was previously @pytest.mark.skip-decorated because the session-scoped kb_server_process couldn't be restarted without breaking 60 other e2e tests sharing it. Extract the server-spawn block from kb_server_process into reusable _spawn_kb_server / _stop_kb_server helpers. Add a function-scoped kb_server_no_chromadb fixture that runs two server lifecycles against one DATA_DIR: phase 1 with chromadb registered to create the collection, phase 2 with VECTOR_DB_CHROMADB=DISABLE so the persisted collection's backend is unavailable at query time. Test now actually issues a real-HTTP query and asserts the 503 response end-to-end, complementing the integration-tier monkeypatch coverage.
Replace module re-import with direct VectorDBRegistry insertion so unit tests that bound QdrantBackend at collection time keep working when the combined suite runs in a single pytest process.
Tracking branch for sub-PRs landing the new KB server microservice and the LAMB-side integration that consumes it. Squashes/merges into main once feature-complete.
…res)
Adds the LAMB-side surface for the new KB Server (port 9092) as parallel
infrastructure alongside the stable kb_registry / /knowledgebases routes,
which remain untouched.
- Migration 17: knowledge_stores + kb_content_links tables and CRUD.
- auth_context: can_access_knowledge_store / require_knowledge_store_access
mirroring the library helpers.
- org_config_resolver: get_knowledge_store_config (with LAMB_KB_SERVER_V2
env fallback) and get_provider_api_key so the embedding key reuses the
existing org-level providers[vendor].api_key.
- knowledge_store_client.py: async httpx client with org allow-list
validation, per-call lifecycle.
- knowledge_store_router.py mounted at /creator/knowledge-stores: options,
CRUD, share toggle, library-only content ingestion (pulls markdown from
Library Manager, builds permalinks against LAMB /docs proxy, posts to
KB Server), query, and job polling.
- library_router: FR-10 guard — DELETE /libraries/{id}/items/{id} returns
409 when any active kb_content_links row references the item.
- docker-compose-example.yaml: new kb-server service on port 9092 with
healthcheck; backend gets LAMB_KB_SERVER_V2 + token env vars and
kb-server in depends_on.
…eval
Sibling of simple_rag.py — kept entirely separate so the legacy stable KB
Server retrieval path is not modified. Auto-discovered by load_plugins('rag')
and surfaced through /capabilities so the assistant builder dropdown picks
it up without further wiring.
Reads assistant.RAG_collections as Knowledge Store UUIDs, looks up each KS
in LAMB DB to discover its locked embedding vendor, resolves the org's
provider key via the existing org-level providers[vendor].api_key, and
queries each KS in parallel via asyncio.gather. Returns the same
{context, sources, ...} shape as simple_rag so the existing chat-side
citation renderer works unchanged; permalinks on each chunk point at
LAMB's /docs/... proxy (set at ingestion time by the KS router).
Adds the lamb-cli surface for Knowledge Stores. The primary command is ``lamb ks`` (short, mirrors the existing ``lamb kb`` precedent for the stable KB Server) with ``lamb knowledge-store`` registered as a long-form alias resolving to the same Typer app. Existing ``lamb kb`` commands are not modified. Commands: options, create, list, get, update, delete, share, add-content, list-content, remove-content, status, query. Status polling uses exponential backoff (1s -> 2s -> 4s -> 8s -> 16s, capped) capped at a configurable max wait, replacing the flaky 15s hard budget pattern Marc flagged in #336 review item #19.
Covers /creator/knowledge-stores/* CRUD, share toggle, allow-list validation (invalid chunking strategy is rejected), duplicate-name 409, options endpoint shape, and 404 hiding. Mirrors library_api.spec.js structure (test.describe.serial, beforeAll auth via storageState, apiCall helper, afterAll cleanup). The full library -> KS -> ingest -> query -> citations -> FR-10 workflow is intentionally deferred to knowledge_store_e2e_workflow.spec.js in Phase 3 because it needs a working KB Server with embedding credentials. The legacy stable KB Server specs (kb_detail_modals.spec.js, kb_delete_modal.spec.js) are not modified.
Adds the foundational frontend pieces for the new KB Server surface: - knowledgeStoreService.js: axios API client mirroring libraryService.js (authHeaders, errorMessage, getApiUrl, browser-only checks). Covers options, CRUD, share, content add/list/get/remove, query, job polling, plus a waitForLinks() helper using exponential backoff (Marc #336 #19). - KnowledgeStoresList.svelte: tabs (My / Shared), search/sort/pagination, share toggle, delete via shared ConfirmationModal. Mirrors LibrariesList structure so the UX is consistent across the two surfaces. Creation is delegated to the unified wizard launched from the parent page. - KnowledgeStoreDetail.svelte: locked-config display with "cannot be changed" notice, linked content list with status badges + auto-polling for in-flight items, query test box that renders permalink citations, edit name/description, share toggle. Uses $effect on the ksId prop to reload on prop change (Marc #336 #3). - AddContentToKSModal.svelte: library + item picker for adding more content to an existing KS. All ready items pre-selected by default (defaults-everywhere principle). The parent /libraries page wiring + the unified create-knowledge wizard land in subsequent commits.
Adds CreateKnowledgeWizard.svelte (shell) and 10 step components under components/knowledge/wizard/. The wizard is the primary creation entry point for the renamed Knowledge tab. Step flow: - Step 0: Library path — existing dropdown OR new - Step 1-3: New-Library path (details, import config, optional content) - Step 4: Knowledge Store path — existing dropdown OR new - Step 5-6: New-KS path (details, locked-setup config with immutability notice and "Edit defaults" expander) - Step 7: Multi-select items to ingest (all pre-selected) - Step 8: Review & create — performs all DB writes here so the wizard is fully reversible until the user clicks Create - Step 9: Done — quick links to open the resulting KS / Library / start another wizard Defaults-everywhere principle: every form field is pre-populated so a user can click Next on every step and end up with a working Library + Knowledge Store + first ingestion. Auto-suggested names use today's date. Skip rules: existing-Library skips 1-3 and lands on Step 4; existing-KS skips 5-6 and lands on Step 7; Steps 3 and 7 are individually skippable. Back-button respects all skip rules in both directions. All i18n strings have inline default fallbacks so the wizard renders correctly even before locale files are populated; the locale keys are added in the next commit.
Adds knowledgeStores.* and knowledge.* (wizard) trees to all four locale files: en.json, es.json, ca.json, eu.json. 56 knowledgeStores keys + 22 wizard.* keys per locale, all matching the inline defaults already used in the components. This addresses Marc's #336 critical issue #2 pre-emptively: never ship modal/wizard UI text only in en — non-English users see translated text from day one rather than English fallbacks. Translations: - "Knowledge Store" -> "Almacén de Conocimiento" (es) / "Magatzem de Coneixement" (ca) / "Ezagutza-biltegia" (eu) - "Library" reuses the existing project translations - "embedding", "Vector DB", "Markdown", "Top K" kept as-is matching the existing project house style for technical terms in knowledgeBases.* / libraries.* trees - "chunking" / "ingest" translated
Restructures the /libraries route into a single "Knowledge" page with
sub-tabs ("Libraries", "Knowledge Stores") and a primary "Create
Knowledge" button that opens the unified wizard.
URL state machine (back-compat preserved):
/libraries -> section=libraries, view=list
/libraries?view=detail&id=X -> back-compat: libraries detail
/libraries?section=knowledge-stores -> KS list
/libraries?section=knowledge-stores&view=detail&id=X -> KS detail
Adds /knowledge-stores route as a power-user direct entry; redirects to
/libraries?section=knowledge-stores via goto() in onMount with
replaceState so the back button doesn't loop.
After the wizard completes, both list components are re-keyed to force
a refresh and the page navigates to the resulting KS detail (or the
library if no KS was produced).
The legacy /knowledgebases route (stable KB Server) and the global nav
are not touched. Adds two missing i18n keys (knowledgeStores.detailTitle,
knowledgeStores.backButton) to all four locale files.
Eleven UI-level tests covering the unified Knowledge page interactions: default route renders sub-tabs and Create Knowledge button; sub-tab switching updates URL; direct URL deep-links land on the right sub-tab; the /knowledge-stores route redirects to /libraries?section=knowledge-stores; seeded KS appears in the My Knowledge Stores list; Create Knowledge button opens the wizard at Step 0; new-Library wizard path walks Step 0 through Step 4 then aborts cleanly without persisting; existing-Library skip rule jumps from Step 0 directly to Step 4; existing-KS skip rule jumps from Step 4 directly to Step 7; Back button on Step 4 returns to the previous non-skipped step; Esc key closes the wizard. Best-effort beforeAll seeds one KS via the Creator API using values from /options; afterAll deletes it idempotently. Tests guard with test.skip when prerequisites (no libraries, KS create failed) aren't available so the suite stays usable across environments. Existing legacy KB Server specs (kb_detail_modals.spec.js, kb_delete_modal.spec.js) and library_api.spec.js / knowledge_store_api.spec.js are not modified.
The headline test of issue #337. Single test.describe.serial walks the complete user-visible chain end-to-end: create Library -> upload sample.md -> poll item ready -> fetch KS options -> create Knowledge Store -> add library item as content -> poll job until ready -> query "capital of France" -> assert chunk metadata carries a /docs/-prefixed LAMB permalink -> resolve the permalink with bearer auth (200, body matches fixture) -> attempt to delete the library item (409 with the KS in conflict.detail) -> remove KS content -> retry library-item delete (200, FR-10 released). Polling uses exponential backoff (1s -> 16s, capped) with 60s and 90s total budgets — replaces the flaky 15s hard window pattern Marc flagged in #336 review #19. Skip-vs-fail policy: a module-scoped pipelineSkipReason is set when LAMB returns 503 from the KS server (KB Server unreachable) or when the content-link poll terminates in 'failed' (embedding API key missing). Subsequent dependent steps test.skip with the propagated reason rather than failing the run, so the suite stays green in environments without a working OpenAI key. afterAll cleanup is idempotent (swallows 404). Adds the sample.md fixture used by the upload step.
Updates CLAUDE.md to make the three-way distinction between Libraries (import), legacy Knowledge Bases (ingest, port 9090, /knowledgebases), and new Knowledge Stores (ingest, port 9092, /knowledge-stores) explicit in the Terminology section. Adds a Knowledge Stores subsection covering the LAMB integration (tables, client, RAG processor, CLI, frontend route, FR-10 enforcement, Docker Compose env vars). Notes that lamb-kb-server/ is in-tree and editable (vs lamb-kb-server-stable/ which is vendored read-only). Adds 17 ADRs at lamb-kb-server/Documentation/issue_337_lamb_integration_adrs.md capturing the architectural decisions made during the integration: parallel surfaces (KS-1), LAMB-owns-ACL (KS-2), library-only ingestion (KS-3), existing-org-key reuse (KS-4), locked store setup (KS-5), LAMB-proxy permalinks (KS-6), FR-10 placement (KS-7), delete-KB-first (KS-8), provisional create (KS-9), per-call httpx (KS-10), routing order (KS-11), sibling RAG processor (KS-12), CLI naming (KS-13), unified frontend (KS-14), defaults-everywhere wizard (KS-15), wizard skip rules (KS-16), exponential-backoff polling (KS-17).
…back loop
Step components dispatched 'update' events on every $effect run; the wizard's
handleStateUpdate replaced wizardState with a new object, which re-triggered
the step's $effect, dispatching again -- an infinite loop that suspended the
parent's reactivity (so wizardOpen=false from Esc/X/done never re-rendered
{#if wizardOpen}).
Fixes:
- handleStateUpdate skips no-op updates (deep-compares object/array values
via JSON serialization).
- Step0/Step1 wrap their dispatch logic in untrack() so reading wizardState
inside the effect doesn't subscribe.
- Wizard no longer owns isOpen as a $bindable; the parent's {#if wizardOpen}
controls visibility, and the wizard fires onclose() to notify.
- Parent /libraries route attaches a top-level <svelte:window> Esc handler
while the wizard is open; X-button and backdrop-click both go through the
same close path.
KnowledgeStoresList was overwriting the backend's correct is_owner field with a client-side comparison s.owner_user_id === \$user.id, but the user store does not expose an id property (only token/name/email). Every store ended up with is_owner=false, sending all owned stores into the "Shared" tab. Drop the broken remap and use the backend's is_owner directly -- matches LibrariesList's pattern.
When creating a Knowledge Store the caller may omit embedding_endpoint
(common for OpenAI; required for self-hosted Ollama or local). LAMB now
falls back to setups.default.providers.{vendor}.endpoint (or base_url /
api_endpoint) instead of forwarding an empty string to the KB Server.
- OrganizationConfigResolver.get_provider_endpoint(vendor) helper.
- create_knowledge_store resolves the endpoint via the resolver and
persists the resolved value, so add-content/query also pick it up.
ESLint cleanups across the new Knowledge Store components: - Use SvelteSet from svelte/reactivity for selectedIds (Set is not reactive in Svelte 5). - Drop unused 'tick' import in CreateKnowledgeWizard. - Drop unused listContent import in KnowledgeStoreDetail. - Bracket the /docs permalink anchors with eslint-disable for svelte/no-navigation-without-resolve (these are backend API URLs, not SvelteKit routes). - Drop unused errorMessage helper in knowledgeStoreService. - Drop svelte-ignore comments that no longer match a real warning.
…dator ChromaDB 0.5+ eagerly validates the OpenAI embedding function at collection-create time, raising ValueError if no api_key is set. Knowledge Store create only sends the real embedding key per-request on add-content/query. Provide harmless placeholders (OPENAI_API_KEY, EMBEDDINGS_APIKEY) in the kb-server v2 service so collection creation succeeds; per-request keys still override.
§19 documents how the implementation is verified: four Opus subagents in parallel review the backend, frontend, CLI, and test suites; Claude Code orchestrates them and runs the deterministic Playwright + pytest suites. Records the test commands and the success criteria.
The Knowledge Store + Library specs share LAMB DB state and snapshot resource counts to assert no leakage. With the previous default of unbounded workers in dev mode, Playwright spawned multiple workers and the specs interleaved -- one spec's afterAll deletion ran while another spec's beforeAll snapshotted counts, producing flaky "Expected 2, Received 1" assertions. Force workers=1 unconditionally (matches CI). fullyParallel was already false but only governs intra-file parallelism; without workers=1 the specs themselves still ran in parallel.
Run `npm run format` once across the tree to flush format debt that had been hidden by prettier-plugin-svelte crashing on every .svelte file at the prettier@^3.4.2 -> 3.8.x resolution. Entirely mechanical (whitespace, quote style, trailing commas); zero runtime behaviour change. The prettier version pin (3.5.3) that lets this run cleanly lands in the feature commit alongside the actual code changes.
- prettier: format the six files prettier flagged after the initial commits. - prettierignore: exclude static/config.js (runtime-generated copy of config.js.sample, not meant to be linted). - eslint: drop unused imports (`user`), unused locals (`result`), unused catch parameters, and the leftover ``openWizard`` shortcut on /libraries/+page.svelte. Convert ``new Set(...)`` to ``new SvelteSet(...)`` for the polling-side ``pendingItemIds`` so it reuses Svelte's reactive collection. Park the legitimate ``$state(new SvelteSet(...))`` patterns behind ``// eslint-disable-next-line svelte/no-unnecessary-state-wrap`` with a comment explaining the reassignment-needs-$state nuance. - jsdoc: extend the ``WizardState`` typedef with ``selectionInitialized`` so svelte-check no longer flags the new field as unknown.
…nt-libraries-ks feat(#365): frontend refinement for libraries and knowledge stores
The empty-token startup-guard test spawned its subprocess with cwd set to a hardcoded absolute Linux path that only existed on one developer's machine. Anywhere else the subprocess silently failed to start with a working directory error, masking the actual guard behavior under test. Resolve the repo root from the test file's location instead so the test runs portably on macOS, Linux, and CI.
…pty-token-test fix(#367): derive subprocess cwd from __file__ in lifespan test
Adds a read-only viewer for imported library items and inverts the item-delete flow so the 'Are you sure?' prompt only appears when no Knowledge Store references the item.
When the user switches libraries quickly in the Create Knowledge wizard, a slower response for the previous library could overwrite items belonging to the now-selected one. Guard with a request sequence number and add a test that locks the contract in place.
…-content feat(#370): view library content and invert delete-confirm flow
firecrawl-py >= 2.x renamed crawl_url() to crawl() and replaced the params dict with keyword arguments + a ScrapeOptions value object. The returned doc.metadata is now a Pydantic model with attribute access; support both shapes for forward compatibility.
…gration fix(#377): migrate url_import to firecrawl-py 2.x API
- Library→KS inverse query: backend DB method + GET
/creator/libraries/{lib}/knowledge-stores endpoint, new
`lamb library list-knowledge-stores` CLI, LibraryDetail KS panel,
KS detail library filter (with ?library= deep-link).
- View original source file: GET
/creator/libraries/{lib}/items/{item}/original wrapper, new
--view markdown|original flag for `lamb library item-content`,
ItemContentModal "Open original" button, in-tab PDF/image preview
via <embed> wrapper, source-URL open-in-tab for URL/YouTube items.
- /items reload race: auto-retry helper with backoff, error+retry
panel for HTTP failures, "having trouble loading" amber panel
(never "no items yet") when library record says items exist but
response is empty.
- Library Manager: SQLite engine switched to NullPool so requests
cannot inherit stale read-snapshots from pooled connections —
root-cause fix for the intermittent empty-items race.
Bundles previously in-flight changes to kb-server chunking plugins,
library-manager import plugins, knowledge_store router/client, and
the unified knowledge wizard / KS list modal components.
Add translated defaultName key (en/es/ca/eu) for use in library creation defaults. Also enable Vite polling watcher for Colima/SSHFS environments where inotify events are not propagated to the guest VM.
…ured The url_import plugin previously required a firecrawl_key in api_keys and raised an error when none was provided. Now it checks for the key first: if present it uses Firecrawl for multi-page deep crawls; if absent it falls back to a direct MarkItDown HTTP fetch of the single URL. Removes the 'firecrawl' entry from required_keys so the plugin is usable out of the box.
…odal Replace the 'View markdown', 'View original', 'View error' text buttons in the item actions column with a single eye icon button (red-tinted for failed items). Also replace the delete text button with a trash icon. ItemContentModal is redesigned with two tabs: Markdown (rendered content or error message) and Original (source URL with open-in-tab button for url/youtube items, open-file button for file imports). Tabs are only shown when source data is available; the Original tab links back out to the source without proxying.
Remove red styling from the eye button on failed items — the status badge already signals failure, the action icon doesn't need to repeat it. All eye buttons are now consistently neutral gray. In ItemContentModal, classify the error string before rendering: - rate_limit → amber banner with a 'wait and retry' hint - no_subtitles → blue banner with a 'try a different language' hint - other errors → existing red banner with raw message
Permalink proxy (/docs/*):
- Add /docs/* to Caddyfile and Vite dev proxy; was missing so all
Source/Markdown buttons in query results returned 404
- Fix _build_permalinks: original now points to actual source file or
external URL; previously both original and full_markdown were identical
- Replace permalink <a> tags with authenticated fetch+blob-URL buttons
so /docs/* requests carry the Bearer token
Ingestion resilience:
- Re-split oversized chunks with RecursiveCharacterTextSplitter instead
of truncating; prevents context-length errors on large documents without
data loss; configurable via MAX_EMBED_CHARS / RESPLIT_CHUNK_SIZE env vars
KB Server embedding plugins:
- Rewrite Ollama plugin to use ollama SDK directly; chromadb's built-in
OllamaEmbeddingFunction is incompatible with newer ollama SDK versions
- Rewrite OpenAI plugin to use openai v1 SDK directly
- Pin chromadb to <0.6.0 in pyproject.toml
- ChromaDB adapter: always wrap via _Adapter; fixes exception re-wrapping
for SDK classes whose __init__ does not accept a positional message arg
KS list UX:
- Add "Add Content" (+) action button per row (owners only); opens modal
inline without navigating to the detail view
- Fix sharing filters: "Mine" = owned and not shared; "Shared" = is_shared
- Unified modal backdrop to bg-black/30 across list and detail contexts
- Label top-k input as "Results" so its purpose is clear
Library/KS backend integration:
- Expose is_owner flag in library list endpoint
- Add /creator/libraries/{id}/kb-links pre-check endpoint for FR-10 UI
- Expose source_url in library-manager item summary (needed for permalink)
- Fix knowledgeStoreService.listContent to read items key instead of content
- Add getLibraryKbLinks to libraryService
- KS client: built-in plugin fallback data when KB Server is unreachable
Modal click-outside-to-close:
- PublishModal, ChangePasswordModal, UserActionModal, RubricAIGenerationModal,
ItemContentModal: add overlay onclick + stopPropagation on panel
- ItemContentModal: default to original tab; always show original tab
Library list filters:
- Fix shared filter (same is_shared pattern as KS list)
- Fix date filters to use local midnight instead of UTC for today check
After adding content via the list-row + button, a progress modal opens immediately showing live status badges for each ingested item. - Fetches all statuses in parallel on open (no initial 4 s wait); items that are already ready appear so instantly - Polls every 4 s for anything still pending/processing, stops when all items reach ready or failed — same interval as KSDetail - Click-outside is disabled; only Close / View Knowledge Store dismiss it so users cannot accidentally lose the progress view - loadStores() runs on modal close, not on open, so the list stays visible and usable behind the backdrop while processing is in flight - View Knowledge Store button navigates to the KS detail page
The z-50 panel container sat above the z-40 backdrop, intercepting all outside clicks before they could reach the backdrop's onclick handler. Moved handleOverlayClick to the z-50 container and added stopPropagation on the inner panel card so only outside clicks dismiss the modal.
Frontend (Vitest): - vitest-setup-client.js: polyfill localStorage for Node.js v26 compatibility Backend: - test_creator_knowledge_stores_integration.py: add chunking_params=None to update mock assertion KB Server (lamb-kb-server): - tests/conftest.py: force LAMB_API_TOKEN override (was blocked by env var already set in container) - test_embedding_plugins.py: rewrite OpenAI/Ollama tests to patch new client APIs (openai.OpenAI, ollama.Client); add skipif guard for sentence-transformers - test_schemas.py: add chunking_params field to UpdateCollectionRequest dump assertion - test_vector_db_chromadb.py: update _to_chroma_ef test to match always-wrapping adapter behavior Playwright (E2E): - global-setup.js: replace fragile UI login with direct API login; write storageState manually; fixes missing preventDefault on form submit - Set admin role for admin@owi.com in LAMB DB (was 'user', needs 'admin' for admin routes) - fr10_ui.spec.js: update blocked-delete test to match new pre-flight UI (modal hides confirm button when item is KS-referenced; no DELETE ever sent) - knowledge_store_api.spec.js: fix list-content assertion from .content to .items (API response field name) - knowledge_store_ui.spec.js: fix tab-click test to wait for Logout button (app fully hydrated); skip wizard tests whose entry point was removed in #365
…ilters pill, local time format #334
Adds a generic PluginParamFields renderer driven by each plugin's
get_parameters() schema. Wired into the create-knowledge wizard
(StepLibrarySetup, StepLibraryContent, StepKSSetup), the standalone
import and create-KS modals, and the KS detail view. A new import
plugin or chunking strategy now appears in the UI automatically with
its declared inputs (int / float / string / bool / enum), default
values, and min/max validation — no UI change required.
Knowledge Store detail surfaces the current chunking parameters and
gives owners an inline editor. Edits flow through the existing
PUT /creator/knowledge-stores/{ks_id} endpoint and apply only to
future ingestions. The edit affordance is hidden from non-owners and
the API still enforces owner-level access.
Plumbing:
- Service layer: uploadFile / importUrl / importYouTube forward
plugin_params; createKnowledgeStore forwards embedding_params and
vector_db_params; updateKnowledgeStore forwards chunking_params.
- Creator-interface: YoutubeImportRequest accepts plugin_params (the
legacy top-level language field is kept as a fallback);
KnowledgeStoreCreate accepts embedding_params and vector_db_params.
- Library Manager: YouTube router prefers plugin_params['language']
over the deprecated top-level field.
- lamb-kb-server: CreateCollectionRequest accepts embedding_params and
vector_db_params; values are accepted and logged today, plugin-
constructor wiring is a follow-up gated on ORM persistence.
- i18n: plugins.params namespace added to en / es / ca / eu.
- Tests: 7 vitest cases for PluginParamFields + 3 pytest cases for
end-to-end plugin_params passthrough.
NoveliaYuki
changed the title
KnowledgeStoreDetail.svelte: full Phase C consistency-contract pass (Card/Banner/SkeletonCard primitives, inline-edit name+description, locked-warning above grid, toast for async writes, status badges via statusBadgeProps, cache-seeded header for perceived performance). Fixes share-toggle flicker (private->shared->private->shared on click) introduced by the cache-seed pattern: the $effect read 'ks' via the '!ks' guard, so the optimistic toggle reassignment retriggered the effect and fired a racing loadAll() GET that overwrote the optimistic state mid-flight. Wrapping the body in untrack() and pinning deps to ksId/orgId only keeps the toggle purely optimistic until the PUT resolves.
Bundles in-progress work-in-progress across UI primitives, library folders + file tree, content-handler capabilities, plugin matcher, toast store, knowledge-store cache, lamb-kb-server + library-manager backends, and assorted backend tests + CLI updates. .gitignore hardening: catch .env.local at any depth, *.pem/*.key, secrets.json, credentials.json, and **/test-results/. Removes testing/playwright/.env.local from tracking (file kept locally) — the credentials it contained should be rotated; the file's prior contents remain in pushed git history. Adds an environment_data/ tree of .env.example / .env.sample templates collected per service for onboarding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Aggregates the KB-server refactor into a single landing PR for
main. Sub-PRs merge intoprojects/refactor/kbserver-lamb-integration; this PR consolidates the diff and ships when all sub-PRs have merged.Skipping
devas the integration target —devandmainare at the same commit right now, so a single PR is sufficient.Sub-PRs landing on the refactor branch
lamb-kb-server/, port 9092)/creator/knowledge-bases/*endpoints,knowledge_bases+kb_content_linkstables,kb_server_manager_v2proxy client, org-config extensions, deletion guard on referenced library itemslamb-clilamb kbcommands/knowledge-bases-v2UI with citation renderingHeld in draft until the integration sub-PRs above land. Marking ready for review once the refactor branch is feature-complete.
Test plan
lamb-kb-server(scripts/run_tests.sh) + LAMB backend pytest + frontendnpm run checklamb kb create→lamb kb add-content→lamb kb queryend-to-end