diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3927a486..be8310c2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -15,6 +15,9 @@ code-index/ ├── plugins/cix/ # Claude Code plugin (hooks, skills, slash commands, bats tests) ├── skills/ # Canonical sources for cross-cutting skills │ # (mirrored into plugins/cix/skills/ via sync-skills.sh) +├── mac/ # cix.app — bundle template, artwork, build + DMG scripts +│ # (the launcher itself lives in cli/launcher/) +├── site/ # codeindex.app — marketing site (Vite, deployed from main) └── doc/ # Tracked documentation ``` @@ -33,14 +36,20 @@ The repo runs a two-branch model: **Open every PR against `develop`.** Promotion from `develop` to `main` happens as part of the release workflow, not per-feature. -CI is wired in three workflows — `ci-cli.yml`, `ci-server.yml`, -`ci-plugin.yml` — all gated on the same branch set: +CI is wired in four path-filtered workflows — `ci-cli.yml`, +`ci-server.yml`, `ci-plugin.yml`, `ci-site.yml` — all gated on the same +branch set: - Push to `main` or `develop` runs CI. - Pull request targeting `main` or `develop` runs CI. - Push to any other branch does **not** trigger CI directly — CI fires when you open the PR. +`mac/` has no CI job of its own: the app is built only by +`release-mac.yml`, on a `mac/v*` tag. A change under `mac/` is therefore +worth building locally (`mac/scripts/build-app.sh`) before merging — see +[`mac/README.md`](mac/README.md). + There is no required branch-name convention. You can name your feature branch anything (`feat/foo`, `fix/bug-123`, `your-handle/sandbox`, …); CI runs from the PR, not from the branch name. @@ -70,7 +79,7 @@ what. | Tool | Version | Purpose | |------|---------|---------| -| Go | 1.24+ | server + CLI | +| Go | 1.26+ (server) / 1.25+ (CLI) | matches each module's `go` directive | | Docker | 24+ | containerized server | | make | any | build shortcuts | @@ -92,9 +101,13 @@ make build # → server/dist/cix-darwin-arm64/cix-server (or linux-amd64) make bundle # Run server locally (no embeddings) +# Point BOTH vector paths at /tmp: the store imports the legacy chromem +# tree it finds under CIX_CHROMA_PERSIST_DIR on first open, and the +# default is your real ~/.cix/data/chroma. CIX_PORT=21847 CIX_EMBEDDINGS_ENABLED=false \ CIX_SQLITE_PATH=/tmp/cix-dev.db \ - CIX_CHROMA_PERSIST_DIR=/tmp/cix-chroma \ + CIX_VECTORS_DIR=/tmp/cix-dev-vectors \ + CIX_CHROMA_PERSIST_DIR=/tmp/cix-dev-chroma \ ./dist/cix-darwin-arm64/cix-server ``` diff --git a/README.md b/README.md index 29e2d264..9e51ea8d 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,10 @@ Or open `http://localhost:21847/dashboard` in your browser: > pipeline stabilizes, an upgrade can change how code is embedded. A reindex > brings every project onto the new pipeline; within a version, search is > consistent once reindexed. +> +> The 0.13.0 vector-store change is the exception: it moves the vectors you +> already have into SQLite by itself, on first boot, without re-embedding +> anything ([`doc/VECTORSTORE.md`](doc/VECTORSTORE.md)). --- @@ -51,8 +55,9 @@ Grep and fuzzy file search work fine for small projects. At scale they break dow ## What you get -- **`cix-server`** — Go HTTP API with embedded llama.cpp sidecar for embeddings, SQLite for symbols + metadata, chromem-go for vectors, FTS5 BM25 mirror for hybrid ranking. Ships as a single distroless container. -- **Web dashboard** at `/dashboard` — projects, search, users + API keys, runtime sidecar control, drift indicator. Embedded in the server binary. See [`doc/DASHBOARD.md`](doc/DASHBOARD.md). +- **`cix-server`** — Go HTTP API with embedded llama.cpp sidecar for embeddings, SQLite for symbols, metadata **and vectors**, FTS5 BM25 mirror for hybrid ranking. Vectors are read from disk per query rather than held in RAM, so an idle server sits at tens of megabytes regardless of index size ([`doc/VECTORSTORE.md`](doc/VECTORSTORE.md)). Ships as a single distroless container. +- **Web dashboard** at `/dashboard` — projects, search, users + API keys, runtime settings, resource + database maintenance, drift indicator. Embedded in the server binary. See [`doc/DASHBOARD.md`](doc/DASHBOARD.md). +- **macOS menu bar app** — `cix.app`, a drag-to-install launcher that runs and self-updates a local server on Apple Silicon. See [`doc/MACOS_APP.md`](doc/MACOS_APP.md). - **`cix` CLI** — `cix search`/`symbols`/`files`/`workspace …` for terminal + agent use. See [`doc/CLI_REFERENCE.md`](doc/CLI_REFERENCE.md). - **File watcher** — `cix watch` keeps the index fresh as you edit. - **Workspaces** — group multiple repos into one named corpus; cix clones them server-side, indexes them, and runs hybrid BM25 + dense search across the union. GitHub webhooks auto-reindex on `push`. See [`workspaces.md`](workspaces.md). @@ -80,7 +85,7 @@ Grep and fuzzy file search work fine for small projects. At scale they break dow │ Indexing pipeline │ │ ├── tree-sitter/wasm (AST chunking, 30+ langs) (wazero) │ │ ├── embedding provider (local llama.cpp / Voyage / OpenAI) │ -│ ├── chromem-go (cosine similarity vector store) │ +│ ├── SQLite vector store (float32 BLOBs, streamed cosine scan) │ │ └── SQLite FTS5 mirror (BM25) + metadata (modernc/sqlite) │ └────────────┬─────────────────────────────────────┬──────────────┘ │ HTTP │ Unix socket @@ -98,13 +103,24 @@ Pure-Go static binary; CUDA-image variants add a CUDA runtime layer for GPU embe | Mode | Best for | GPU | Prerequisites | |------|----------|-----|---------------| +| **macOS app** | Apple Silicon Macs — the default on a Mac | Metal | macOS 13+, Apple Silicon | | **Docker (CPU)** | any OS, dev / small repos | none | Docker | | **Docker (CUDA)** | NVIDIA GPU servers | CUDA 12.x | Docker + NVIDIA Container Toolkit | -| **Native (macOS)** | Apple Silicon w/ full Metal | Metal | Go 1.25+, Node.js, Xcode CLT | +| **Native from source (macOS)** | hacking on cix itself | Metal | Go 1.25+, Node.js, Xcode CLT | ### 1. Start the server -One command, any mode — the installer detects your platform, asks a few questions (deployment mode, admin email, password, port — every one has a sensible default), and brings the server up: +**On a Mac, install the app.** `cix.app` is a menu bar launcher: download +`cix--arm64.dmg` from the +[releases page](https://github.com/dvcdsys/code-index/releases), drag it to +Applications, and open it. It downloads the server, the CLI and a +Metal-accelerated `llama-server` into `~/.cix/runtime/`, creates your admin +account and an API key, points the `cix` CLI at it, and keeps the server updated +in place — so steps 2 and 3 below are already done and you can go straight to +step 4. macOS blocks an ad-hoc-signed app twice on first launch; clearing that +and everything else the app does is in [`doc/MACOS_APP.md`](doc/MACOS_APP.md). + +**Everywhere else, one command** — the installer detects your platform, asks a few questions (deployment mode, admin email, password, port — every one has a sensible default), and brings the server up: ```bash curl -fsSL https://raw.githubusercontent.com/dvcdsys/code-index/main/install-server.sh | bash @@ -112,7 +128,7 @@ curl -fsSL https://raw.githubusercontent.com/dvcdsys/code-index/main/install-ser (Equivalent from a clone: `git clone https://github.com/dvcdsys/code-index && cd code-index && ./install-server.sh`.) -At the end it prints the dashboard URL and your admin login — and offers to install the `cix` CLI and connect it to the new server, so `cix init` works immediately (steps 2–3 below happen automatically on a fresh install). Re-running after a `git pull` upgrades in place; `--uninstall` removes the server but keeps your data. Native-mode details and manual setup: [`doc/SETUP_MACOS_NATIVE.md`](doc/SETUP_MACOS_NATIVE.md). For shared/team deployment, see [`doc/TEAM_DEPLOYMENT.md`](doc/TEAM_DEPLOYMENT.md). +At the end it prints the dashboard URL and your admin login — and offers to install the `cix` CLI and connect it to the new server, so `cix init` works immediately (steps 2–3 below happen automatically on a fresh install). Re-running after a `git pull` upgrades in place; `--uninstall` removes the server but keeps your data. Building the macOS server from a checkout instead of installing the app: [`doc/SETUP_MACOS_NATIVE.md`](doc/SETUP_MACOS_NATIVE.md). For shared/team deployment, see [`doc/TEAM_DEPLOYMENT.md`](doc/TEAM_DEPLOYMENT.md).
Manual Docker setup (what the installer automates) @@ -243,6 +259,7 @@ Most common environment variables (full surface in [`doc/CONFIG_REFERENCE.md`](d | `CIX_EMBEDDING_MODEL` | `awhiteside/CodeRankEmbed-Q8_0-GGUF` | Local GGUF repo or absolute path. | | `CIX_N_GPU_LAYERS` | `-1` macOS / `0` else / `99` Docker CUDA | `99` = full offload, `0` = CPU. | | `CIX_EMBEDDINGS_ENABLED` | `true` | `false` boots without the llama sidecar. | +| `CIX_VECTORS_DIR` | sibling of the chroma dir, i.e. `/vectors` | Where the vector databases live. Must be on a persistent volume — this is your index. | | `CIX_SECRET_KEY` / `_KEYFILE` | auto-generated keyfile | AES-256-GCM key for `github_tokens` encryption. **Back this up.** | | `CIX_PUBLIC_URL` | — | Public origin for webhook URLs. Trumped by a live Managed Tunnel. | @@ -279,18 +296,25 @@ docker compose down -v # stop AND wipe data + models (destructive) | Doc | Purpose | |---|---| | [`doc/CLI_REFERENCE.md`](doc/CLI_REFERENCE.md) | Full CLI command surface + per-project config (`.cixignore`, `.cixconfig.yaml`) | +| [`doc/CLI_CONFIG.md`](doc/CLI_CONFIG.md) | Everything the CLI lets you configure (servers, defaults, output) | | [`doc/DASHBOARD.md`](doc/DASHBOARD.md) | Dashboard pages, authentication, authorization model, drift indicator | +| [`doc/MACOS_APP.md`](doc/MACOS_APP.md) | The macOS menu bar app — install, first run, updates, uninstall | | [`doc/TEAM_DEPLOYMENT.md`](doc/TEAM_DEPLOYMENT.md) | Self-hosting cix for a team — production / shared-infrastructure deployment for DevOps | | [`doc/TROUBLESHOOTING.md`](doc/TROUBLESHOOTING.md) | Common issues + search-quality tuning (`--min-score`) | | [`workspaces.md`](workspaces.md) | User-facing workspace guide (when to use, agent trust rules, query patterns) | | [`doc/WORKSPACES.md`](doc/WORKSPACES.md) | Operator setup (encryption keys, Cloudflare tunnel, workers, REST API) | | [`doc/SEARCH_ALGORITHM.md`](doc/SEARCH_ALGORITHM.md) | How per-project + hybrid workspace search rank results | +| [`doc/VECTORSTORE.md`](doc/VECTORSTORE.md) | The SQLite vector store — layout, tuning, migration from chromem-go | +| [`doc/DATABASE_MAINTENANCE.md`](doc/DATABASE_MAINTENANCE.md) | Reclaim, compaction, scheduled maintenance | | [`doc/WEBHOOKS.md`](doc/WEBHOOKS.md) | GitHub webhook lifecycle, modes, HMAC validation | +| [`doc/POLLING.md`](doc/POLLING.md) | Git polling sync, for repos where a webhook is not an option | | [`doc/COWORK_MCP.md`](doc/COWORK_MCP.md) | Using cix from Claude Desktop / Cowork over MCP (`cix mcp install`, multi-server) | | [`doc/UPDATES.md`](doc/UPDATES.md) | Release-poll banner + stable vs develop install channels | | [`doc/CONFIG_REFERENCE.md`](doc/CONFIG_REFERENCE.md) | Complete env-var reference | -| [`doc/RELEASES.md`](doc/RELEASES.md) | Cutting CLI + server releases, CVE scans, make targets | -| [`doc/SETUP_MACOS_NATIVE.md`](doc/SETUP_MACOS_NATIVE.md) | Native macOS Metal setup + launchd plist | +| [`doc/RELEASES.md`](doc/RELEASES.md) | Cutting CLI + server + app releases, CVE scans, make targets | +| [`doc/DEPRECATION_POLICY.md`](doc/DEPRECATION_POLICY.md) | How long a removed feature is announced before it goes | +| [`doc/SETUP_MACOS_NATIVE.md`](doc/SETUP_MACOS_NATIVE.md) | Building the macOS server from a checkout (the app is the normal path) | +| [`mac/README.md`](mac/README.md) | How the macOS app and its runtime are built and signed | | [`doc/SECURITY_DEPLOYMENT.md`](doc/SECURITY_DEPLOYMENT.md) | Production hardening | | [`doc/DOCKER_TAGS.md`](doc/DOCKER_TAGS.md) | Docker Hub tag lifecycle | | [`doc/LANGUAGES.md`](doc/LANGUAGES.md) | Supported chunker languages | @@ -328,9 +352,12 @@ projects and teams that make it possible: tree-sitter binding cix's AST chunking first grew from; thank you for the head start. - [chromem-go](https://github.com/philippgille/chromem-go) — the - embedded cosine-similarity vector store. + embedded vector store cix shipped through v0.12.x, and the model its + collection semantics still follow. Now kept only to read a pre-0.13 + index during the one-time import. - [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) — cgo-free - SQLite for project metadata, symbols, and the FTS5/BM25 mirror. + SQLite for project metadata, symbols, the FTS5/BM25 mirror, and (since + v0.13.0) the vectors themselves. - [go-git](https://github.com/go-git/go-git) — server-side repository cloning for workspaces. diff --git a/doc/CONFIG_REFERENCE.md b/doc/CONFIG_REFERENCE.md index 1e2efc13..4bbb1c30 100644 --- a/doc/CONFIG_REFERENCE.md +++ b/doc/CONFIG_REFERENCE.md @@ -31,8 +31,9 @@ the DB. | Variable | Default | Description | |---|---|---| | `CIX_PORT` | `21847` | Listen port (both Docker images bake this in). | +| `CIX_DATA_DIR` | `~/.cix/data` (`/tmp/cix-data` with no `$HOME`) | Base directory the other storage defaults are derived from. Ignored where a path is set explicitly — which the containers do, so it is a native-install variable in practice. | | `CIX_BIND_ADDR` | — | Interface to listen on, as a bare address with no port. Empty means every interface, which is what a container needs; set `127.0.0.1` to make the server reachable only from the machine it runs on. The macOS app writes `127.0.0.1` at first run and exposes a menu toggle. | -| `CIX_SQLITE_PATH` | `/data/sqlite/projects.db` | SQLite path. Suffixed with the model-safe name on open. | +| `CIX_SQLITE_PATH` | `/data/sqlite/projects.db` | System SQLite database — opened literally, with no model suffix appended. (A pre-0.x per-model filename is migrated to this path on first boot.) | | `CIX_CHROMA_PERSIST_DIR` | `/data/chroma` | Legacy chromem-go store. Read on startup for the one-time import into the SQLite vector store, then left untouched as the rollback path. See [VECTORSTORE.md](VECTORSTORE.md). | | `CIX_VECTORS_DIR` | sibling of `CIX_CHROMA_PERSIST_DIR` (`/data/vectors`) | Vector store directory: one SQLite database per embedding namespace. | | `CIX_VECTOR_MMAP_SIZE` | `0` (off) | `PRAGMA mmap_size` for the vector store, in bytes. Roughly 40% lower search latency in exchange for resident memory — mapped database pages count in RSS. | @@ -49,6 +50,8 @@ the DB. | `CIX_LANGUAGES` | all | Comma-separated allow-list of chunker languages. Empty = all baked-in. See [`LANGUAGES.md`](LANGUAGES.md). | | `CIX_EMBED_INCLUDE_PATH` | `true` | Path/language/symbol preamble before each chunk. Toggling requires `cix reindex --full`. | | `CIX_MAX_CHUNK_TOKENS` | `1500` | Max chunk size before falling back to sliding window. Must stay ≤ `CIX_LLAMA_CTX`. | +| `CIX_INDEX_EMBED_BATCH_CHUNKS` | `0` (built-in default) | Chunks per embedding batch during indexing. Also editable at runtime from the dashboard. | +| `CIX_CHUNK_MAX_CONCURRENT` | `0` (built-in default) | Files chunked in parallel. Also editable at runtime from the dashboard. | ## llama-server sidecar @@ -61,6 +64,7 @@ the DB. | `CIX_LLAMA_CTX` | `2048` | `--ctx-size` passed to llama-server. | | `CIX_N_GPU_LAYERS` | `-1` darwin / `0` else / `99` Docker CUDA | `99` offloads all layers; `0` forces CPU. | | `CIX_LLAMA_STARTUP_TIMEOUT` | `60` | Seconds to wait for the sidecar's readiness probe. | +| `CIX_LLAMA_CACHE_RAM` | `0` (disabled) | llama-server's host prompt cache in MiB (`--cache-ram`). Embeddings get no prompt reuse from it, and upstream's 8192 default has OOM-killed this server; `-1` is unlimited. | | `CIX_GGUF_PATH` | auto-resolve | Absolute path to a GGUF file. Empty → cache lookup → HF download. | | `CIX_BOOTSTRAP_GGUF_PATH` | — | Optional. If set, cix imports this `.gguf` into `CIX_GGUF_CACHE_DIR` once (atomic `.partial → rename`) and ignores the env on subsequent boots. Useful for air-gapped or rate-limited environments. | @@ -96,6 +100,39 @@ See [`WORKSPACES.md`](WORKSPACES.md) for the operator guide, [`WEBHOOKS.md`](WEBHOOKS.md) for webhook lifecycle, and [`POLLING.md`](POLLING.md) for the polling alternative. +### Managed tunnels + +A managed tunnel gives a NAT-ed server a public URL for webhook delivery. +The binaries are in both Docker images; on a native install they come from +`PATH` unless a path is given here. + +| Variable | Default | Purpose | +|---|---|---| +| `CIX_TUNNEL_CLOUDFLARE_BIN_PATH` | `cloudflared` (from `PATH`; images set `/cloudflared`) | `cloudflared` executable. | +| `CIX_TUNNEL_CLOUDFLARE_METRICS_ADDR` | `127.0.0.1:21848` | Where `cloudflared` exposes its metrics endpoint, which is how readiness is detected. | +| `CIX_TUNNEL_CLOUDFLARE_STARTUP_TIMEOUT` | `30` | Seconds to wait for the tunnel to come up. | +| `CIX_TUNNEL_NGROK_BIN_PATH` | `ngrok` (from `PATH`) | `ngrok` executable. | +| `CIX_TUNNEL_NGROK_STARTUP_TIMEOUT` | `30` | Seconds to wait for the tunnel to come up. | +| `CIX_TUNNEL_BIN_MANAGED` | `false` | Let the server download and update the tunnel binary itself. | +| `CIX_TUNNEL_BIN_DIR` | `/tunnel-bin` | Where a managed binary is kept. | + +## Database maintenance + +Reclaim and compaction are normally driven from **Server → Resources → +Database**; these variables exist for deployments nobody opens a dashboard +for. A schedule saved in the dashboard overrides the environment, and an +invalid cron expression is refused at startup rather than at the first tick. + +| Variable | Default | Purpose | +|---|---|---| +| `CIX_DB_MAINTENANCE_CRON` | — (no automatic run) | Default schedule for the database tasks, as a crontab expression. | +| `CIX_DB_MAINTENANCE_MIN_FREE_PERCENT` | `25` | How much of the file must be freelist waste before a scheduled run bothers. | +| `CIX_DB_MAINTENANCE_MIN_FREE_BYTES` | `256 MiB` | The same threshold in absolute bytes; both have to be met. | + +Compaction takes the server **read-only and then restarts it** — read +[`DATABASE_MAINTENANCE.md`](DATABASE_MAINTENANCE.md) before scheduling one +on a server other people use. + ## Version-check banner | Variable | Default | Description | @@ -110,12 +147,21 @@ See [`UPDATES.md`](UPDATES.md) for how the banner works end-to-end. | | Native (Apple Silicon) | Docker (CPU) | Docker (CUDA) | |--|---|---|---| -| Image size | n/a | ~21 MB | ~1.0 GB | -| Memory (idle) | ~1 GB | ~1 GB | ~1 GB (system) + ~0.7 GB VRAM | -| Memory (indexing) | up to 2 GB | up to 2 GB | up to 2 GB system + ~0.7 GB VRAM | +| Image size (compressed pull) | n/a | ~80 MB | ~1.1 GB | +| Memory, `cix-server` idle | tens of MB | tens of MB | tens of MB | +| Memory, `llama-server` sidecar | ~0.5–0.7 GB | ~0.5–0.7 GB | ~0.2 GB system + ~0.7 GB VRAM | +| Memory (indexing) | up to ~1.5 GB total | up to ~1.5 GB total | same + ~0.7 GB VRAM | | GPU | Metal | none | NVIDIA CUDA 12.x | | Disk | `~/.cix/data/` (~50–200 MB/project) | same (mounted volume) | same | -| Auto-restart | `launchd` agent, set up by `install-server.sh` (see [`SETUP_MACOS_NATIVE.md`](SETUP_MACOS_NATIVE.md)) | yes | yes | +| Auto-restart | `launchd` agent, installed by **cix.app** (see [`MACOS_APP.md`](MACOS_APP.md)) or by `install-server.sh` for a from-source build | yes | yes | + +Since 0.13.0 the server's own resident memory no longer scales with the +index: vectors are read from SQLite per query instead of being loaded into +the heap at startup, and idle connections are closed after 30 seconds +([`VECTORSTORE.md`](VECTORSTORE.md) measures 19 MB idle on a +312k-document index that cost 2209 MB before). The one setting that +brings index-proportional memory back on purpose is +`CIX_VECTOR_MMAP_SIZE`. What is left at idle is the embedding sidecar. ## Switching embedding models @@ -136,12 +182,15 @@ You can switch in two places: restart. The dashboard's runtime override (if any) wins; the env value becomes the bootstrap default. -ChromaDB and SQLite paths are suffixed by a sanitised form of the -model name (e.g. `projects_awhiteside_coderankembed_q8_0_gguf.db`). -This isolates vector spaces per model — switching back and forth -keeps old indices intact and avoids dim-mismatch errors. -Re-indexing under a model is not free (chunk count × embedding -latency), but you don't lose state. +Vector spaces are isolated per model by **directory**, not by filename: +each embedding namespace — provider kind, model slug, optional variant — +gets its own `///vectors.db` +(`Config.VectorDirFor`). The system SQLite database is shared and not +model-specific. Switching back and forth therefore opens a different +`vectors.db`, keeps old indices intact and makes a dim-mismatch +impossible. Re-indexing under a model is not free (chunk count × +embedding latency), but you don't lose state. See +[`VECTORSTORE.md`](VECTORSTORE.md). ## Related files diff --git a/doc/DASHBOARD.md b/doc/DASHBOARD.md index 3af6ff95..dab6cb6f 100644 --- a/doc/DASHBOARD.md +++ b/doc/DASHBOARD.md @@ -8,16 +8,17 @@ service to run, no nginx config, no separate static-files volume. | Page | Audience | What it does | |------|----------|--------------| | **Home** | everyone | Live status strip (server version, current embedding model, sidecar Ready/Loading), update-available banner when a newer `server/v*` release is published on GitHub, module shortcuts. | -| **Projects** | everyone | List indexed projects with stats (file count, languages, symbols, vector count, sqlite/chroma sizes), per-project **Reindex** button + live indexing indicator, copy reindex commands. Cards turn red with a **Stale model** badge when the runtime embedding model differs from the model the project was indexed with (see [Drift indicator](#drift-indicator)). | +| **Projects** | everyone | List indexed projects with stats (file count, languages, symbols, vector count, SQLite size and vector-store size — the latter is the logical byte count of that project's rows inside the shared `vectors.db`, a floor rather than a file size), per-project **Reindex** button + live indexing indicator, copy reindex commands. Cards turn red with a **Stale model** badge when the runtime embedding model differs from the model the project was indexed with (see [Drift indicator](#drift-indicator)). | | **Workspaces** | everyone | Group multiple repositories into a named workspace and search them as one corpus. The in-dashboard add-repo flow streams clone + index progress live; pick the org/account first, then the repo. Status tracking: `pending` → `cloning` → `indexing` → `indexed` / `failed`. Hybrid BM25 + dense search across the whole group. See [`../workspaces.md`](../workspaces.md). | | **Search** | everyone | Five modes: semantic, symbols, references, definitions, files. Same engine the CLI uses. | | **API Keys** | everyone | Mint long-lived `cix_*` keys (256-bit entropy, GitHub-class), copy them once, revoke at any time. Keys inherit the issuing user's role. | -| **GitHub Tokens** | admin | Store personal access tokens used by external (cloned) projects + workspaces. Tokens are AES-256-GCM encrypted at rest; the plaintext is returned once on creation and never again. Scopes are **derived from GitHub** at storage time (not user-declared), so the dashboard shows the PAT's true capabilities. | +| **GitHub Integration** | admin | Store personal access tokens used by external (cloned) projects + workspaces. Tokens are AES-256-GCM encrypted at rest; the plaintext is returned once on creation and never again. Scopes are **derived from GitHub** at storage time (not user-declared), so the dashboard shows the PAT's true capabilities. | | **Users** | admin | Invite teammates, set role (admin / user), reset password (forces change on next login), disable account. | -| **Groups** | admin | Manage *view-groups* — named user sets used to share projects and workspaces with specific people. Add/remove members, grant shares from the project or workspace detail page. | +| **View Groups** | admin | Manage *view-groups* — named user sets used to share projects and workspaces with specific people. Add/remove members, grant shares from the project or workspace detail page. | | **Managed Tunnels** | admin | Enable a Cloudflare Tunnel or ngrok tunnel to give the server a public origin for GitHub webhook ingress from behind NAT. Configure provider, mode (quick / named), and credentials; agent binary auto-installs on demand; live status + restart + round-trip test. | +| **Login security** | admin | Accounts currently locked out by failed sign-ins, with the ability to clear a lock. | | **Settings** | everyone | Theme, default editor, change own password. | -| **Server** | admin | Runtime config — embedding model, `n_ctx`, `n_gpu_layers`, `n_threads`, batch size, queue concurrency. **Save & Restart** drains in-flight embeddings, restarts the sidecar, polls until ready. Source pill on each field shows whether the live value comes from the DB override, env bootstrap, or the recommended fallback. | +| **Server** | admin | Two tabs. **Runtime settings** — embedding provider and model, `n_ctx`, `n_gpu_layers`, `n_threads`, batch size, queue concurrency; **Save & restart** drains in-flight embeddings, restarts the sidecar and polls until ready, and a source pill on each field shows whether the live value comes from the DB override, env bootstrap, or the recommended fallback. **Resources** — see [Resources & maintenance](#resources--maintenance) below. | ## Authentication @@ -51,10 +52,46 @@ the caller isn't allowed to use, and the CLI surfaces a 404 (not a 403) when probing a resource the caller has no business knowing exists. Full hardening posture: [`SECURITY_DEPLOYMENT.md`](SECURITY_DEPLOYMENT.md). +## Resources & maintenance + +**Server → Resources** is where an admin sees what cix is costing the machine +and gets it back. It holds two cards. + +**Storage & memory** leads with the server process's resident memory as the +operating system reports it — not the Go heap, which since 0.13.0 says almost +nothing about the real footprint (the vector store's pages live outside it). +Below that is disk, attributed by category, behind an explicit **Analyze** +pass because the scan walks the filesystem and the database. **Clean** then +removes what you select: + +| Category | What it is | +|---|---| +| `orphan_collections` | A vector collection with no matching row in `projects`. | +| `orphan_repos` | A cloned checkout under `/` whose project is gone. | +| `stale_namespaces` | A vector-store namespace for a provider/model that is no longer active. | +| `legacy_chromem` | The pre-0.13 chromem gob tree of a namespace already imported into SQLite. | +| `stale_jobs` | Finished rows in the `jobs` queue, which has no retention policy of its own. | +| `unused_models` | A cached GGUF other than the active model. Off by default — re-downloading costs minutes. | + +`legacy_chromem` is the one that cannot be undone. Nothing reads that tree any +more, but it is what a downgrade to a pre-0.13 server would read, so it is +kept until you say otherwise and the dashboard says so before it deletes. +Everything else here is recoverable by reindexing or re-downloading. See +[`VECTORSTORE.md`](VECTORSTORE.md). + +**Database** reports how much of the system SQLite file is freelist waste and +offers **Reclaim now** (bounded, no window, needs incremental auto-vacuum), +**Compact now** (rebuild + restart), the auto-vacuum mode switch, and a cron +schedule for both. A compaction puts the server into a read-only window and +then restarts it, so a banner announces it across the dashboard while it runs. +The whole feature, including what a compaction does to other people's +sessions, is in [`DATABASE_MAINTENANCE.md`](DATABASE_MAINTENANCE.md). + ## Drift indicator -When you change the runtime embedding model (Server → Embedding model → Save & -Restart), every project indexed with the previous model becomes stale — +When you change the runtime embedding model (Server → Runtime settings → +Embedding model → Save & restart), every project indexed with the previous +model becomes stale — vectors are no longer comparable to fresh queries. The dashboard surfaces this with red borders + `Stale model` badges on project cards, and a banner on the project detail page with a copy-to-clipboard `cix reindex --full ` diff --git a/doc/DEPRECATION_POLICY.md b/doc/DEPRECATION_POLICY.md index 994204ae..71b1b109 100644 --- a/doc/DEPRECATION_POLICY.md +++ b/doc/DEPRECATION_POLICY.md @@ -17,6 +17,24 @@ See `doc/DOCKER_TAGS.md` for the current tag inventory. +## chromem-go vector store and `CIX_CHROMA_PERSIST_DIR` + +Deprecated in `server/v0.13.0`, which replaced the chromem-go store with a +SQLite one (`doc/VECTORSTORE.md`). Nothing writes to the chromem tree any +more: + +- `CIX_CHROMA_PERSIST_DIR` is still read, for exactly two things — locating + the legacy gob files for the one-time import, and deriving the default + `CIX_VECTORS_DIR` beside them. Set `CIX_VECTORS_DIR` explicitly and it + stops mattering. +- The `/chroma` tree is kept deliberately: it is what a downgrade to a + pre-0.13 server would read. **Server → Resources → Clean** offers it as the + `legacy_chromem` category, which is the supported way to reclaim it once + you have decided you will not roll back. + +Neither is scheduled for removal yet. When one is, the notice lands here one +minor version ahead, per the rule above. + ## Python backend The Python FastAPI backend (`legacy/python-api/`) was deprecated in diff --git a/doc/DOCKER_TAGS.md b/doc/DOCKER_TAGS.md index 852adcb8..e0c0e62d 100644 --- a/doc/DOCKER_TAGS.md +++ b/doc/DOCKER_TAGS.md @@ -4,13 +4,16 @@ | Tag | Architecture | Base | Size | Notes | |---|---|---|---|---| -| `latest` | linux/amd64 + linux/arm64 | Go CPU (distroless/static) | ~100 MB | Use with `CIX_EMBEDDINGS_ENABLED=false` | -| `` (e.g. `0.6.0`) | linux/amd64 + linux/arm64 | same as `latest` | ~100 MB | Version-pinned CPU image. Immutable. | -| `cu128` | linux/amd64 | distroless/cc-debian13 + CUDA libs | ~1.0 GB | RTX 3090 prod; embeddings via llama-server | -| `-cu128` (e.g. `0.6.0-cu128`) | linux/amd64 | same as `cu128` | ~1.0 GB | Version-pinned CUDA image. Immutable. | -| `develop-cu128` | linux/amd64 | same as `cu128` | ~1.0 GB | Floating pre-release; force-updated on every merge to `develop` that touches `server/`. Not for production. | +| `latest` | linux/amd64 + linux/arm64 | distroless/cc-debian13 + bundled CPU llama.cpp | ~80 MB | Embeddings work out of the box on CPU; `CIX_EMBEDDINGS_ENABLED=false` only if you want none. | +| `` (e.g. `0.6.0`) | linux/amd64 + linux/arm64 | same as `latest` | ~80 MB | Version-pinned CPU image. Immutable. | +| `cu128` | linux/amd64 | distroless/cc-debian13 + CUDA libs | ~1.1 GB | RTX 3090 prod; embeddings via llama-server | +| `-cu128` (e.g. `0.6.0-cu128`) | linux/amd64 | same as `cu128` | ~1.1 GB | Version-pinned CUDA image. Immutable. | +| `develop-cu128` | linux/amd64 | same as `cu128` | ~1.1 GB | Floating pre-release; force-updated on every merge to `develop` that touches `server/`. Not for production. | | `0.2-python-legacy` | linux/amd64 | Python FastAPI | ~5 GB | Frozen; rollback only | +Sizes are the compressed pull as Docker Hub reports it for `linux/amd64` +(v0.13.0: 81 MB CPU, 1077 MB CUDA; the arm64 CPU image is 73 MB). + ## Develop channels `develop` has a matched pair of floating pre-release artifacts: diff --git a/doc/MACOS_APP.md b/doc/MACOS_APP.md index 2181d136..be08cce4 100644 --- a/doc/MACOS_APP.md +++ b/doc/MACOS_APP.md @@ -101,9 +101,9 @@ Everything it runs lives outside it, under your home directory: ``` ~/.cix/runtime/ - 0.12.8/ cix-server cix llama/ runtime.json - 0.12.7/ the version this one replaced, kept for rollback - current -> 0.12.8 + 0.13.0/ cix-server cix llama/ runtime.json + 0.12.9/ the version this one replaced, kept for rollback + current -> 0.13.0 ``` Those are *server* versions — the same ones on Docker Hub. The app has its own, @@ -132,7 +132,7 @@ is what catches a runtime that is not what it claims to be. ## First run The very first launch asks for an email address, downloads the runtime (about -40 MB), then generates a password and an API key, starts the server, and shows +37 MB), then generates a password and an API key, starts the server, and shows you the credentials. You will be asked to change the password when you first sign in. @@ -175,8 +175,8 @@ macOS announces any newly registered background agent. ● Embeddings: llama.cpp (bundled) Port: 21847 Model: awhiteside/Co…bed-Q8_0-GGUF Network: this Mac only ───────────── Model: awhiteside/CodeRankEmbed-Q8_0-GGUF -Stop Server Server 0.12.8 -Open Dashboard Server 0.12.8 (llama b10238) +Stop Server Server 0.13.0 +Open Dashboard Server 0.13.0 (llama b10238) ───────────── Start at Login ✓ Allow Network Access ✓ diff --git a/doc/MIGRATION_FROM_PYTHON.md b/doc/MIGRATION_FROM_PYTHON.md index 6f87725b..9712e768 100644 --- a/doc/MIGRATION_FROM_PYTHON.md +++ b/doc/MIGRATION_FROM_PYTHON.md @@ -32,8 +32,10 @@ See `.env.example` for a complete template. ## Vector store (action required) -The Python server used ChromaDB (DuckDB + parquet). -The Go server uses chromem-go (JSON format). **These are not compatible.** +The Python server used ChromaDB (DuckDB + parquet). The Go server stores +vectors in SQLite (`server/internal/vectorstore/`, since 0.13.0; chromem-go +gob files before that). **None of these layouts are compatible with each +other.** On first boot the Go server automatically detects the old ChromaDB layout (`chroma.sqlite3` in the persist dir) and backs it up: @@ -64,6 +66,9 @@ If you need to go back to the Python server: # Rename it back to /data/chroma to restore the old index. ``` +Restoring that backup only means anything together with rolling the image +back: a 0.13+ server reads `/data/vectors` and would not look at it. + ## Sunset timeline The Python code in `legacy/python-api/` was deleted in `server/v0.4.0` diff --git a/doc/RELEASES.md b/doc/RELEASES.md index afc500a5..1286185b 100644 --- a/doc/RELEASES.md +++ b/doc/RELEASES.md @@ -1,12 +1,20 @@ # Releases -CLI and server ship on independent tag streams so a bugfix on one -doesn't drag the other through a rebuild + retest cycle. +Server, CLI and the macOS app ship on three independent tag streams so a +bugfix on one doesn't drag the others through a rebuild + retest cycle. | Component | Tag pattern | Workflow | Artifact | |---|---|---|---| -| Server (`cix-server`) | `server/v*` (e.g. `server/v0.6.0`) | [`release-server.yml`](../.github/workflows/release-server.yml) | Docker images on Docker Hub: `:latest`, `:`, `:cu128`, `:-cu128` | +| Server (`cix-server`) | `server/v*` (e.g. `server/v0.6.0`) | [`release-server.yml`](../.github/workflows/release-server.yml) | Docker images on Docker Hub: `:latest`, `:`, `:cu128`, `:-cu128` — **plus** `cix-runtime--darwin-arm64.tar.gz` on the GitHub Release | | CLI (`cix`) | `cli/v*` (e.g. `cli/v0.6.0`) | [`release-cli.yml`](../.github/workflows/release-cli.yml) | `cix-{darwin,linux}-{amd64,arm64}.tar.gz` on a GitHub Release | +| macOS app (`cix.app`) | `mac/v*` (e.g. `mac/v0.1.1`) | [`release-mac.yml`](../.github/workflows/release-mac.yml) | `cix--arm64.dmg` + `checksums.txt` on a GitHub Release | + +The app and the server it runs are deliberately not the same release. The +app is ~4 MB and holds one executable; the *runtime* it installs — the +server, the CLI and a Metal `llama-server` — ships from the `server/v*` +tag, so a Mac and a container on the same version run the same server and +a new server reaches a Mac without a new app. See +[`MACOS_APP.md`](MACOS_APP.md) and [`../mac/README.md`](../mac/README.md). Bare `v*` tags are the historical pre-split CLI line — the installer still falls back to them when no `cli/v*` release exists, but no new @@ -75,7 +83,8 @@ takes >30 min on CI, so this is more disciplined than the CLI path: 4. CI (`release-server.yml`) builds CPU multi-arch + CUDA `amd64` images with provenance + SBOM attestations, pushes them to Docker Hub with both pinned (`:0.7.0`, `:0.7.0-cu128`) and floating - (`:latest`, `:cu128`) tags, and creates a GitHub Release. + (`:latest`, `:cu128`) tags, builds the macOS runtime tarball, and + creates a GitHub Release carrying it. 5. **Promote** in production (Portainer, your compose file, etc.) by updating the image tag to `:0.7.0` / `:0.7.0-cu128` and @@ -84,6 +93,48 @@ takes >30 min on CI, so this is more disciplined than the CLI path: CI does not deploy to production. It stops at Docker Hub push by design — promotion is a manual operator step. +Two constraints the `macos-runtime` job adds to a server tag: + +- **A `cli/v*` tag must be reachable from the tagged commit.** The runtime + bundles the `cix` CLI and the job fails rather than ship it stamped + `0.0.0-dev`. In practice this means cutting `server/v*` on `main`. +- **The release is not publishable without it.** `macos-runtime` is a hard + dependency of the release job, because a server release with no runtime + attached is one no Mac can install or update to — `cix.app` reads its + server from exactly these assets. + +## Cutting a macOS app release + +Only when the *app* changes — a server release reaches Macs on its own. + +1. Bump the version wherever the app advertises it, then tag: + + ```bash + git tag mac/v0.1.2 + git push origin mac/v0.1.2 + ``` + +2. CI (`release-mac.yml`) builds and ad-hoc-signs `cix.app` on an arm64 + runner, verifies the bundle (`codesign --verify --strict`, a + `cix-launcher -report`, and a check that `Contents/MacOS` holds exactly + one executable), wraps it in the styled DMG, writes `checksums.txt`, + and publishes a GitHub Release whose body carries the Gatekeeper + instructions. + +The release is created with `make_latest: false` on purpose: the Docker +image is this project's primary deliverable and owns the "latest" pointer. +Mac installs and the in-app updater filter releases by the `mac/` tag +prefix instead. + +Unlike `server/v*`, this stream needs no other tag reachable — nothing in +the app is stamped from the server or the CLI. What is actually installed +on a Mac is recorded in `~/.cix/runtime/current/runtime.json`. + +There is no Apple Developer certificate and therefore no notarization; +signing is ad-hoc, which macOS blocks on first launch by design. The +signing order and the failure modes it avoids are documented in +[`../mac/README.md`](../mac/README.md). + ## Docker Scout workflow (iterate before pushing) For non-tag iterations on the CUDA image (debugging a new layer, @@ -140,9 +191,12 @@ and historical lifecycle. The quick version: ## Related files -- `.github/workflows/release-server.yml` — stable server build/release pipeline +- `.github/workflows/release-server.yml` — stable server build/release pipeline (Docker images + the macOS runtime) - `.github/workflows/release-cli.yml` — stable CLI build/release pipeline +- `.github/workflows/release-mac.yml` — macOS app + DMG - `.github/workflows/prerelease-server.yml` / `prerelease-cli.yml` — develop channels +- [`MACOS_APP.md`](MACOS_APP.md) — what the app does with what this stream ships +- [`../mac/README.md`](../mac/README.md) — the app build pipeline and signing order - [`DOCKER_TAGS.md`](DOCKER_TAGS.md) — Docker Hub tag lifecycle - [`DEPRECATION_POLICY.md`](DEPRECATION_POLICY.md) — when tags / behaviours retire - [`UPDATES.md`](UPDATES.md) — release-poll banner + install channels diff --git a/doc/SEARCH_ALGORITHM.md b/doc/SEARCH_ALGORITHM.md index 71874358..2b041972 100644 --- a/doc/SEARCH_ALGORITHM.md +++ b/doc/SEARCH_ALGORITHM.md @@ -64,9 +64,13 @@ sister SQLite tables: - `chunks_fts` — FTS5 virtual table over `(content, symbol_name, file_path)` — provides BM25 scoring against literal tokens. -Both tables share a rowid and are written inside the indexer's -per-file SQL transaction, so a chunk is either in *both* stores or -*neither*. +Both tables share a rowid and are written inside the indexer's per-file +SQL transaction, together with that file's symbols, references and hash — +so a chunk is in both of *these* tables or neither. The vector store is a +separate database file with its own write (see +[`VECTORSTORE.md`](VECTORSTORE.md)), so it is not part of that +transaction: an interrupted index can leave a file embedded but not +mirrored, which the next indexing pass corrects. Code: `server/internal/chunksfts/chunksfts.go`. Introduced by `f00e3d3`. @@ -139,8 +143,8 @@ Trust rules for an agent consuming the response (`chunks[]` vs ## 4. Symbols / definitions / references / files These bypass the embedding pipeline entirely. They run against -SQLite-backed indexes that the chunker populates in the same per-file -transaction as the vector store: +SQLite-backed indexes in the system database, which the chunker fills in +its own per-file transaction alongside the FTS mirror: - **`cix symbols `** — substring-and-trigram lookup over `symbols` (kind ∈ {function, class, method, type}). Fast (<50 ms on a diff --git a/doc/SETUP_MACOS_NATIVE.md b/doc/SETUP_MACOS_NATIVE.md index 612f19bd..4a3d59fd 100644 --- a/doc/SETUP_MACOS_NATIVE.md +++ b/doc/SETUP_MACOS_NATIVE.md @@ -1,13 +1,26 @@ -# Native macOS setup (Apple Silicon, Metal GPU) +# Native macOS setup from source (Apple Silicon, Metal GPU) + +> [!IMPORTANT] +> **The normal way to run cix on a Mac is the app.** `cix.app` is a +> drag-to-install menu bar launcher that downloads the same +> Metal-accelerated server into `~/.cix/runtime/`, creates your admin +> account, wires up the CLI and keeps itself and the server updated — no +> Go, Node or Xcode toolchain involved. See [`MACOS_APP.md`](MACOS_APP.md). +> +> This document is the **advanced path**: building and running the server +> out of a checkout, which is what you want when you are working on cix +> itself, need a build of an unreleased branch, or want the launchd agent +> under your own control. Docker Desktop on macOS runs containers inside a Linux VM, and the -Metal GPU is **not accessible** from within that VM. For full Metal -acceleration on Apple Silicon you must run cix-server natively. +Metal GPU is **not accessible** from within that VM — so a Mac gets its +Metal acceleration from a natively-built `cix-server` either way. The app +ships one; the rest of this doc builds one. > For Docker (CPU) and Docker (CUDA) deployments, follow README's -> *Quick Start* section instead. This doc is only for native macOS. +> *Quick Start* section instead. -## 1. Install (recommended: the installer) +## 1. Install from source (recommended: the installer) Prerequisites: @@ -167,6 +180,8 @@ The minimum env-var set for a Metal native run: | `CIX_N_GPU_LAYERS` | (leave unset) | macOS defaults to offloading all layers to Metal. `0` forces CPU. | | `CIX_EMBEDDINGS_ENABLED` | `true` | Default. Set `false` to skip the sidecar entirely. | | `CIX_LLAMA_BIN_DIR` | (set by `make run`) | Path to the `llama-server` bundle dir. The dev runner sets it; for `launchd` you set it yourself (see below). | +| `CIX_VECTORS_DIR` | `~/.cix/data/vectors` | Where the vectors actually live — one SQLite database per embedding namespace. See [`VECTORSTORE.md`](VECTORSTORE.md). | +| `CIX_CHROMA_PERSIST_DIR` | `~/.cix/data/chroma` | Only relevant when upgrading a pre-0.13 install: the legacy chromem tree the one-time import reads. Never written to. | The full env-var surface is documented in [`CONFIG_REFERENCE.md`](CONFIG_REFERENCE.md). @@ -206,6 +221,9 @@ and `YOUR_USER` placeholder before loading. CIX_LLAMA_BIN_DIR/ABSOLUTE/PATH/TO/server/dist/cix-darwin-arm64/llama CIX_PORT21847 CIX_SQLITE_PATH/Users/YOUR_USER/.cix/data/sqlite/projects.db + CIX_VECTORS_DIR/Users/YOUR_USER/.cix/data/vectors + CIX_CHROMA_PERSIST_DIR/Users/YOUR_USER/.cix/data/chroma CIX_GGUF_CACHE_DIR/Users/YOUR_USER/.cix/data/models @@ -252,6 +270,7 @@ place instead of being duplicated into `EnvironmentVariables`.) | `make bundle` fails downloading llama-server | Network blocked, or upstream release moved. | Inspect `server/Makefile`'s download URL; report if upstream changed. | | Server starts but `/health` 404s | Wrong port. | `lsof -i :21847` to confirm. Check `CIX_PORT` in `.env`. | | Health check takes minutes on first boot | The embedding model (~150 MB) downloads before serving. | Watch `tail -f ~/.cix/logs/cix-server.err`; it's a one-time cost. | +| Server silent for a while on the first boot after upgrading to 0.13 | The one-time chromem→SQLite vector import runs before the HTTP listener binds (17 s on a 312k-document index; longer on a slow disk). | Watch `tail -f ~/.cix/logs/cix-server.err` for `migrating vector store`; it logs progress and is resumable if interrupted. | | GPU not used (CPU fallback) | `CIX_N_GPU_LAYERS=0` set in `.env`. | Remove it (macOS default offloads all layers) or set `99`. | | "killed: 9" on first llama-server launch | macOS amfid rejected the unsigned binary. | Re-run `make bundle` (or the installer) to refresh the local signature. | | Server starts via terminal but not via `launchd` | Launcher script or `.env` missing / unreadable. | Check `~/.cix/logs/cix-server.err`; re-run `./install-server.sh` to regenerate. | diff --git a/doc/TEAM_DEPLOYMENT.md b/doc/TEAM_DEPLOYMENT.md index 537bcea2..f5438bbf 100644 --- a/doc/TEAM_DEPLOYMENT.md +++ b/doc/TEAM_DEPLOYMENT.md @@ -36,16 +36,19 @@ Everyone on the team connects to *this one server*: CI jobs (CIX_API_URL + key) ├────────► reverse proxy / TLS Claude Code plugin ┘ │ ▼ - cix-server :21847 ──► /data (sqlite+chroma) + cix-server :21847 ──► /data (sqlite+vectors) └► embedding provider ``` Two images, pick one (never merge them): -| Image | Base | Size | Runtime user | Use | +| Image | Base | Download | Runtime user | Use | |---|---|---|---|---| -| `dvcdsys/code-index:latest` | distroless static | ~40 MB | `65532:65532` | CPU-only | -| `dvcdsys/code-index:cu128` | distroless cc + CUDA libs | ~1.0 GB | `1001:1001` | NVIDIA GPU | +| `dvcdsys/code-index:latest` | distroless cc + bundled CPU llama.cpp | ~80 MB | `65532:65532` | CPU-only | +| `dvcdsys/code-index:cu128` | distroless cc + CUDA libs | ~1.1 GB | `1001:1001` | NVIDIA GPU | + +Sizes are the compressed pull, as Docker Hub reports them for `linux/amd64` +(v0.13.0: 81 MB and 1077 MB; the arm64 CPU image is 73 MB). See [`DOCKER_TAGS.md`](DOCKER_TAGS.md) for the full tag lifecycle. @@ -56,8 +59,8 @@ See [`DOCKER_TAGS.md`](DOCKER_TAGS.md) for the full tag lifecycle. - Docker Engine 24+ with Compose v2. - For the CUDA image: an NVIDIA GPU, recent driver, and the **NVIDIA Container Toolkit** installed on the host (`nvidia-ctk`). -- A persistent disk for `/data` (SQLite + chroma vectors grow with the - number and size of indexed repos). +- A persistent disk for `/data` (the SQLite database and the vector store + grow with the number and size of indexed repos). - DNS + TLS termination if the team reaches it over the network (reverse proxy — see §7). - For server-side workspace cloning of private repos: a GitHub PAT @@ -130,8 +133,12 @@ at runtime from **Dashboard → Server** without a restart. The compose files mount two things: - `${HOME}/.cix/data:/data` — operator-managed bind holding **SQLite** - (`/data/sqlite/projects.db`) and **chroma vectors** (`/data/chroma`). Back - this up. + (`/data/sqlite/projects.db`) and the **vector store** (`/data/vectors`, + one `vectors.db` per embedding namespace). Back both up. A server upgraded + from 0.12 or earlier also has `/data/chroma` — the frozen pre-0.13 + chromem-go tree, kept only as a rollback path (see + [`VECTORSTORE.md`](VECTORSTORE.md)); it is never written to and does not + need backing up. - `cix-models:/data/models` — Docker-managed named volume for the GGUF model cache. Downloaded once; survives `docker compose down` (not `down -v`). @@ -223,8 +230,11 @@ Back up, in order of importance: 1. **`/data/sqlite/projects.db`** — users, API keys, projects, symbols, workspaces, runtime config. Use SQLite online backup or stop-copy-start. -2. **`/data/chroma`** — vector store. Recoverable by reindex, but a backup - avoids re-embedding everything. +2. **`/data/vectors`** — the vector store, one SQLite database per embedding + namespace. Recoverable by reindex, but a backup avoids re-embedding + everything. Copy it the way you copy any live SQLite database — online + backup, or stop-copy-start; a plain `cp` of `vectors.db` without its + `-wal` is not a valid backup. 3. **The secret key** (`CIX_SECRET_KEYFILE`, or the auto-generated keyfile under the SQLite parent dir). **Losing it invalidates every stored GitHub PAT** — they'd all have to be re-entered. Back it up *separately* from the @@ -270,12 +280,21 @@ upgrade developer CLIs in lockstep. See [`RELEASES.md`](RELEASES.md). - **Liveness:** the image's own check, `/cix-server -healthcheck` (already wired in both compose files — no `curl` needed), GETs `/health` and exits - 0/1. `start_period` is 120 s to allow the first model download. + 0/1. `start_period` is 600 s: it has to cover the first model download and, + on the first boot after upgrading to 0.13, the one-time chromem→SQLite + vector import (17 s on a 312k-document index, longer on slow disks). - **Readiness probe (external):** `GET /health` on `:21847`. - **Logs:** `docker compose logs -f code-index-api`. Set `CIX_LOG_LEVEL=debug` to diagnose indexing or provider issues. -- **Drift indicator** in the dashboard flags projects whose on-disk code has - diverged from the index. +- **Drift indicator** in the dashboard flags projects indexed with an + embedding model other than the one now running ("Stale model") — their + vectors are no longer comparable to fresh queries, so they need a reindex. +- **During a database compaction** the server is deliberately read-only and + then restarts itself: writes answer `503` with `Retry-After`, `/health` + keeps returning `200` with `"maintenance": true` so a restart policy does + not kill the run, and `GET /maintenance/status` reports progress. See + [`DATABASE_MAINTENANCE.md`](DATABASE_MAINTENANCE.md) before scheduling one + on a shared server. --- diff --git a/doc/TROUBLESHOOTING.md b/doc/TROUBLESHOOTING.md index 46bca16e..bee0444b 100644 --- a/doc/TROUBLESHOOTING.md +++ b/doc/TROUBLESHOOTING.md @@ -31,7 +31,13 @@ cix watch stop && cix watch /path/to/project - Lower the threshold: `cix search "query" --min-score 0.2` (default `0.4`) - `cix list` to verify the project is registered -**Dashboard shows "Stale model" on every project after upgrade** → The runtime model was changed (or its version stamp shifted). Either reindex affected projects (`cix reindex --full` per project) or revert the model change in **Server → Embedding model**. +**Dashboard shows "Stale model" on every project after upgrade** → The runtime model was changed (or its version stamp shifted). Either reindex affected projects (`cix reindex --full` per project) or revert the model change in **Server → Runtime settings → Embedding model**. + +**First boot after upgrading to 0.13 takes minutes and the server answers nothing** → It is importing your existing vectors from the legacy chromem files into the SQLite vector store. That runs before the HTTP listener binds — 17 s on a 312k-document index, longer on a slow disk or a cold cache. It logs progress at warn level (`migrating vector store …`) and is resumable, so an interrupted import picks up where it stopped rather than starting over. Nothing is re-embedded and no reindex is needed. See [`VECTORSTORE.md`](VECTORSTORE.md#migration-from-chromem-go). + +**Every write answers `503` with a `Retry-After`, but reads work** → A database compaction is running. The server is deliberately read-only for the duration and then restarts itself to adopt the compacted file; `/health` keeps returning `200` (with `"maintenance": true`) so a container restart policy does not kill it mid-run. `GET /maintenance/status` and the dashboard banner report progress. See [`DATABASE_MAINTENANCE.md`](DATABASE_MAINTENANCE.md). + +**Memory grew after setting `CIX_VECTOR_MMAP_SIZE`** → That is what it does: it trades resident memory for search latency by letting SQLite map the vector database into the process. A fan-out across a large index can then hold gigabytes rather than tens of megabytes. Unset it to go back, or size it deliberately under a memory ceiling. See [`VECTORSTORE.md`](VECTORSTORE.md#pragmas). **Dashboard banner says an update is available** → A newer `server/v*` release is on GitHub. Click through to the release notes; bump your Docker tag / native build at a convenient time. Disable the poll with `CIX_VERSION_CHECK_ENABLED=false` if you don't want it. See [`UPDATES.md`](UPDATES.md). diff --git a/doc/UPDATES.md b/doc/UPDATES.md index a5e5d456..78e93169 100644 --- a/doc/UPDATES.md +++ b/doc/UPDATES.md @@ -1,9 +1,9 @@ # Keeping cix Up to Date -cix ships in two release streams (server + CLI) and has a built-in -release-poll banner on the dashboard so you know when an upgrade is -available. This doc covers how the banner works, how to opt out, and -how to use the **develop channel** for testing unreleased changes. +cix ships in three release streams — server, CLI, and the macOS app — and +has a built-in release-poll banner on the dashboard so you know when an +upgrade is available. This doc covers how the banner works, how to opt out, +and how to use the **develop channel** for testing unreleased changes. ## 1. Release-poll banner @@ -29,7 +29,15 @@ How it works: `GET /api/v1/admin/version` for the dashboard. The banner is informational only — it links to the release page on -GitHub. cix does not self-update. +GitHub. A Docker or from-source server does not self-update. + +**The macOS app is the exception.** `cix.app` watches two streams of its +own — `server/v*` for the runtime it manages and `mac/v*` for itself — at +startup and at most every 30 minutes after, and updates both: the server by +unpacking beside the running version and moving a symlink (with automatic +rollback if the new one does not come back), the app by a detached helper +that swaps the bundle while it is closed. See +[`MACOS_APP.md`](MACOS_APP.md#check-for-updates). ### Configuration @@ -133,11 +141,19 @@ button (`596748e`); from the CLI it's `cix reindex --full `. Upgrading the **CLI** never requires a reindex — the CLI is a thin HTTP client. +**0.13.0 is not one of these cases.** It moves the vector store from +chromem-go to SQLite, and the server imports the vectors you already have +on its first boot — nothing is re-embedded and no reindex is needed. The +import runs before the listener binds, so that one boot takes longer than +usual (17 s on a 312k-document index). See +[`VECTORSTORE.md`](VECTORSTORE.md#migration-from-chromem-go). + ## 4. Related files - `server/internal/versioncheck/check.go` — release-poll service - `install.sh` / `install-develop.sh` — stable + develop installers -- `.github/workflows/release-server.yml` / `release-cli.yml` — stable build pipelines +- `.github/workflows/release-server.yml` / `release-cli.yml` / `release-mac.yml` — stable build pipelines +- [`MACOS_APP.md`](MACOS_APP.md) — the macOS app's own update mechanism - `.github/workflows/prerelease-server.yml` / `prerelease-cli.yml` — develop build pipelines - [`DOCKER_TAGS.md`](DOCKER_TAGS.md) — Docker tag lifecycle, including `develop-cu128` - [`RELEASES.md`](RELEASES.md) — how to cut a stable release diff --git a/doc/benchmarks.md b/doc/benchmarks.md index c4205c93..e6c8eb5d 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -71,7 +71,7 @@ numbers have not been backfilled. The doc states "Once actual measured deltas." Expected baseline (CodeRankEmbed Q8_0 on RTX 3090): ~0.5–0.7 GB idle -VRAM (weights ~200–250 MB + pre-allocated `n_ctx=8192` context +VRAM (weights ~200–250 MB + pre-allocated `n_ctx=2048` context ~200–400 MB). If you re-run the profiler, update `vram-profiling.md` in place — the @@ -79,6 +79,27 @@ file was always intended as a placeholder. --- +## 4. SQLite vector store vs chromem-go + +**Last measured:** server/v0.13.0, on a real 312,334-document / +47-collection index. +**Where:** [`VECTORSTORE.md`](VECTORSTORE.md#why-it-changed) (headline +table), plus the scan-latency and page-size measurements in the *Search* +and *Pragmas* sections of the same file. + +Headline: resident memory at idle 2209 MB → 19 MB, time from process +start to first answerable query 47 s → ≈1 ms, search latency roughly 4× +higher and far less sensitive to `k`, whole-index import 17 s. + +## 5. Database compaction & auto-vacuum + +**Last measured:** server/v0.13.0. +**Where:** [`DATABASE_MAINTENANCE.md`](DATABASE_MAINTENANCE.md) — the +compaction wall-clock per gigabyte, what the read-only window costs, and +the insert-side price of incremental auto-vacuum. + +--- + ## Raw artefacts The dated grep-vs-cix run also produced raw transcripts and metric diff --git a/install-server.sh b/install-server.sh index fe3ed455..1a159b20 100755 --- a/install-server.sh +++ b/install-server.sh @@ -495,6 +495,17 @@ elif [[ "$HAS_NVIDIA" == true ]]; then MODE_DEFAULT="docker-gpu" fi +if [[ -z "$ARG_MODE" && "$OS" == "Darwin" && "$ARCH" == "arm64" ]]; then + # The app is the usual way to run cix on a Mac; this script builds from a + # checkout, which is the path for hacking on cix itself. Say so once — + # someone who ran the one-liner may simply not know the app exists. Only + # informational: the default below is unchanged. + say "On a Mac, cix.app is the usual install — a menu bar launcher that" + say "downloads and self-updates the server for you, no toolchain needed:" + say " https://github.com/dvcdsys/code-index/releases (cix-*-arm64.dmg)" + say "This script builds the server from this checkout instead." +fi + if [[ -n "$ARG_MODE" ]]; then MODE="$ARG_MODE" else diff --git a/mac/README.md b/mac/README.md index cafa846e..02d41dc7 100644 --- a/mac/README.md +++ b/mac/README.md @@ -24,12 +24,12 @@ separate things: | Asset | Stream | Contents | Size | |---|---|---|---| | `cix--arm64.dmg` | `mac/v*` | the app — `cix-launcher` and its icons, nothing else | ~4 MB | -| `cix-runtime--darwin-arm64.tar.gz` | `server/v*` | `cix-server`, the `cix` CLI, `llama/` | ~35 MB | +| `cix-runtime--darwin-arm64.tar.gz` | `server/v*` | `cix-server`, the `cix` CLI, `llama/` | ~37 MB | The runtime **is** the server, so it carries the server's version and ships from the server's tag — the same `server/vX.Y.Z` and the same workflow run that publishes the Docker images (`release-server.yml`, job `macos-runtime`). A Mac -install on 0.12.8 and a container on 0.12.8 are the same server. That is also +install on 0.13.0 and a container on 0.13.0 are the same server. That is also why llama has no version of its own here: a llama bump is a server release, as it has always been. diff --git a/plugins/cix/skills/cix-workspace/SKILL.md b/plugins/cix/skills/cix-workspace/SKILL.md index 790d1ada..0a06bfc7 100644 --- a/plugins/cix/skills/cix-workspace/SKILL.md +++ b/plugins/cix/skills/cix-workspace/SKILL.md @@ -553,7 +553,7 @@ unrelated repos. **The structural failure:** 1. Pure-dense fan-out cannot tell "no signal" apart from "weak - signal" — chromem always returns the K nearest vectors. + signal" — a vector search always returns the K nearest vectors. 2. Long natural-language queries dilute the few tokens that carry the actual gating signal. 3. Without a sparse-retrieval channel, an acronym or unique @@ -615,9 +615,9 @@ Either: ### `status: "partial_failure"` At least one repo errored out (`failed_repos` array names them). -Common cause: corrupt chromem collection. The remaining repos still -returned results. Surface to the user; don't silently treat as -complete. +Common cause: a missing or corrupt vector collection. The remaining +repos still returned results. Surface to the user; don't silently +treat as complete. ### Top-2 projects are at near-equal candidacy diff --git a/server/dashboard/src/modules/server/sections/ResourcesSection.tsx b/server/dashboard/src/modules/server/sections/ResourcesSection.tsx index a09fc6b1..373d6f48 100644 --- a/server/dashboard/src/modules/server/sections/ResourcesSection.tsx +++ b/server/dashboard/src/modules/server/sections/ResourcesSection.tsx @@ -11,13 +11,13 @@ import { ConfirmCleanDialog } from '../components/ConfirmCleanDialog'; import { ReclaimCategoryList } from '../components/ReclaimCategoryList'; import { useAnalyzeReclaimable, useCleanResources, useResourceUsage } from '../hooks'; -// Resources answers "why is this process holding 4 GB?" and then offers to do -// something about it. +// Resources answers "what is this server costing the machine?" and then offers +// to do something about it. // -// The honest answer to that question is that the vector store is an in-memory -// database — chromem loads every document of every collection at startup and -// never evicts — so the heap figure and the resident document count are the -// two numbers that belong side by side here. +// The headline is resident memory as the operating system reports it, not the +// Go heap: since the vector store moved to SQLite the heap says almost nothing +// about the real footprint, because the pages a query touches are not in it. +// Everything below that is disk, which is where the reclaimable garbage lives. export function ResourcesSection() { const usage = useResourceUsage(); const analyze = useAnalyzeReclaimable(); diff --git a/server/internal/maintenance/maintenance.go b/server/internal/maintenance/maintenance.go index a057e411..f82c1220 100644 --- a/server/internal/maintenance/maintenance.go +++ b/server/internal/maintenance/maintenance.go @@ -1,17 +1,19 @@ // Package maintenance answers two admin questions about a running server: // "what is it using?" and "how much of that is garbage I can drop?". // -// It exists because the vector store is an in-memory database. chromem-go -// eagerly decodes every document of every collection at startup and never -// evicts, so a collection whose project was deleted from SQLite keeps its -// documents resident — for the life of the process — while being reachable by -// nothing. Deleting a project used to leave exactly that: the row went away, -// FK CASCADE took the chunks and symbols, and the vector collection stayed in -// RAM forever. The delete path now cleans up after itself (see -// projects.Artifacts), but every server that ran the old code has a backlog, -// and a cloned checkout or an abandoned provider namespace can still be -// orphaned by a crash mid-delete. This package finds that backlog and removes -// it on request. +// It exists because nothing else reconciles what is on disk against what the +// database still claims. A collection whose project was deleted from SQLite is +// reachable by nothing and pays for itself in disk forever: the row went away, +// FK CASCADE took the chunks and symbols, and the vector collection stayed. +// The delete path now cleans up after itself (see projects.Artifacts), but +// every server that ran the old code has a backlog, and a cloned checkout or +// an abandoned provider namespace can still be orphaned by a crash mid-delete. +// This package finds that backlog and removes it on request. +// +// Under chromem-go — the vector store through v0.12.x — an orphan cost +// resident memory too, because that engine decoded every document of every +// collection at startup and never evicted. Since v0.13.0 the vectors live in +// SQLite and are read per query, so what is being reclaimed here is disk. // // The split of responsibilities is deliberate: this package holds all the // logic and touches the filesystem, the database and the vector store @@ -41,14 +43,14 @@ import ( type CategoryID string const ( - // CatOrphanCollections is the only category that frees RAM as well as - // disk: a chromem collection with no matching row in `projects`. + // CatOrphanCollections is a vector-store collection with no matching row + // in `projects`. CatOrphanCollections CategoryID = "orphan_collections" // CatOrphanRepos is a cloned checkout under / whose project is gone. CatOrphanRepos CategoryID = "orphan_repos" // CatStaleNamespaces is a vector-store namespace directory belonging to a - // provider/model that is no longer active. Disk only — chromem opens the - // active namespace and nothing else, so these were never in RAM. + // provider/model that is no longer active. The server opens the active + // namespace and nothing else, so these cost disk and nothing more. CatStaleNamespaces CategoryID = "stale_namespaces" // CatLegacyChromem is the pre-migration chromem gob tree of a namespace // whose collections are all in the SQLite store. Nothing reads it; it is @@ -156,7 +158,8 @@ type Item struct { // which is a SQL delete. Unexported: it is an implementation detail of // Clean and must never reach the client. path string - // collection is the raw chromem collection name for orphan_collections. + // collection is the raw vector-store collection name for + // orphan_collections. collection string } diff --git a/server/internal/maintenance/usage.go b/server/internal/maintenance/usage.go index 6ee4c798..07a621b0 100644 --- a/server/internal/maintenance/usage.go +++ b/server/internal/maintenance/usage.go @@ -28,10 +28,10 @@ type DiskUsage struct { FSFreeBytes *int64 `json:"fs_free_bytes,omitempty"` } -// VectorStoreUsage summarises the in-memory vector database. Every field here -// is read from chromem's process image, so this costs nothing — which is why -// the usage endpoint can report document counts even when it skips the -// directory walks. +// VectorStoreUsage summarises the vector store. The counts are aggregates over +// the store's own tables rather than a filesystem walk, which is why the usage +// endpoint can still report them when it skips the directory walks — but they +// are a query, not a free read of a process image as they were under chromem. type VectorStoreUsage struct { Collections int `json:"collections"` Documents int64 `json:"documents"` diff --git a/site/public/sitemap.xml b/site/public/sitemap.xml index cad6425f..4cad4224 100644 --- a/site/public/sitemap.xml +++ b/site/public/sitemap.xml @@ -2,10 +2,10 @@ https://codeindex.app/ - 2026-07-19 + 2026-08-15 https://codeindex.app/docs/ - 2026-07-19 + 2026-08-15 diff --git a/site/src/docs/docs.jsx b/site/src/docs/docs.jsx index e2c96b2c..12be3153 100644 --- a/site/src/docs/docs.jsx +++ b/site/src/docs/docs.jsx @@ -94,17 +94,9 @@ export function DocsApp() {
-

Server — one command, any mode

-

The interactive installer detects your platform and offers the right mode — native (macOS Apple Silicon, full Metal GPU), docker (CPU, any OS), or docker-gpu (NVIDIA CUDA). It checks prerequisites, asks a few questions (every one has a sensible default), writes the configuration, and brings the server up:

- {`curl -fsSL https://raw.githubusercontent.com/dvcdsys/code-index/main/install-server.sh | bash - -# equivalent, from a clone: -git clone https://github.com/dvcdsys/code-index && cd code-index -./install-server.sh`} -

At the end it prints the dashboard URL and your admin login (the password is temporary — you change it on first login), and offers to install the cix CLI and connect it to the new server — so cix init works immediately. Re-running after a git pull upgrades in place; --uninstall removes the server but keeps your data. Forgot the admin password later? ./server/scripts/reset-password.sh <email> resets it offline. Details: SETUP_MACOS_NATIVE.md.

- -

macOS — the menu bar app

-

On an Apple Silicon Mac there is a second way in: cix.app, a menu bar app that runs the server for you. It sets up an admin account on first launch, downloads the server itself (about 40 MB, the same build the Docker images are cut from), and gives you start/stop, network access, launch-at-login and a password reset without a terminal. The server keeps running when you quit the app, and both halves update themselves. Download cix.app {MAC_APP_VERSION} — macOS 13 or later, Apple Silicon only (upstream llama.cpp publishes no macOS x86_64 build).

+

macOS — start here

+

On an Apple Silicon Mac the whole thing is an app: cix.app, a menu bar launcher that runs the server for you. The download is about 4 MB because the server is not inside it — on first launch the app sets up an admin account, downloads the server itself (about 37 MB, the same build the Docker images are cut from) into ~/.cix/runtime/, mints an API key and points the cix CLI at it. You get start/stop, network access, launch-at-login and a password reset without a terminal; the server keeps running when you quit the app, and both halves update themselves. Download cix.app {MAC_APP_VERSION} — macOS 13 or later, Apple Silicon only (upstream llama.cpp publishes no macOS x86_64 build).

+

That covers the rest of this section: the installer, the first login and the CLI configuration below have already happened by the time the app finishes, so the next thing you do is index a project. Full detail: MACOS_APP.md.

macOS will block the download, twice: once for the disk image, once for the app inside it. cix is open source and not signed with a paid Apple Developer certificate, so macOS cannot verify it — nothing is wrong with your download. The fastest way through is one command before you open the image, because the app inherits the download mark from the image at the moment you copy it out:

{`xattr -d com.apple.quarantine ~/Downloads/cix-*-arm64.dmg`} @@ -112,6 +104,15 @@ git clone https://github.com/dvcdsys/code-index && cd code-index

To click through it instead: open the .dmg, and when macOS says it "could not verify" the file choose Done — never Move to Bin, which deletes the download. Then go to System Settings → Privacy & Security, scroll to Security, and click Open Anyway next to the blocked file. That button appears only after the failed attempt, so going there first shows nothing. Drag the app across, open it, and repeat for the app itself. On macOS 15 and later the old right-click → Open shortcut works for neither block. You do this once per version.

+

Everywhere else — one command, any mode

+

The interactive installer detects your platform and offers the right mode — docker (CPU, any OS), docker-gpu (NVIDIA CUDA), or native (a from-source build on Apple Silicon, for hacking on cix itself). It checks prerequisites, asks a few questions (every one has a sensible default), writes the configuration, and brings the server up:

+ {`curl -fsSL https://raw.githubusercontent.com/dvcdsys/code-index/main/install-server.sh | bash + +# equivalent, from a clone: +git clone https://github.com/dvcdsys/code-index && cd code-index +./install-server.sh`} +

At the end it prints the dashboard URL and your admin login (the password is temporary — you change it on first login), and offers to install the cix CLI and connect it to the new server — so cix init works immediately. Re-running after a git pull upgrades in place; --uninstall removes the server but keeps your data. Forgot the admin password later? ./server/scripts/reset-password.sh <email> resets it offline. Building the macOS server from a checkout: SETUP_MACOS_NATIVE.md.

+

Manual setup (what the installer automates)

Docker (CPU):

{`git clone https://github.com/dvcdsys/code-index && cd code-index @@ -123,7 +124,7 @@ curl http://localhost:21847/health # → {"status":"ok"}`}

Docker (CUDA / NVIDIA GPU) — requires an NVIDIA driver ≥ 525 (CUDA 12.x) and the NVIDIA Container Toolkit; the CUDA image sets CIX_N_GPU_LAYERS=99 for full GPU offload:

{`docker compose -f docker-compose.cuda.yml pull docker compose -f docker-compose.cuda.yml up -d`} -

Native macOS (Apple Silicon) — Docker on macOS can't reach the Metal GPU; for full Metal offload build and run natively (Go 1.25+, Node.js, Xcode CLT):

+

Native macOS from source (advanced) — this is the path for working on cix itself; to just run it on a Mac, use the app, which installs the same natively-built server without a toolchain. Docker on macOS can't reach the Metal GPU, so a from-source build is how you get full Metal offload out of a checkout (Go 1.25+, Node.js, Xcode CLT):

{`cd server && make bundle # builds server + downloads Metal-enabled llama-server cp ../.env.example ../.env # set the bootstrap admin vars make run`} @@ -265,13 +266,14 @@ cix reindex --full`}
-

All server settings use the CIX_* prefix. This is the curated set — the full reference (~40 variables) lives in doc/CONFIG_REFERENCE.md. Tuning values are also editable at runtime from /dashboard/server; env values are the boot-time seed.

+

All server settings use the CIX_* prefix. This is the curated set — the full reference (~40 variables) lives in doc/CONFIG_REFERENCE.md. Tuning values are also editable at runtime from Server → Runtime settings; env values are the boot-time seed. The neighbouring Server → Resources tab is where an admin sees resident memory and disk by category, reclaims orphaned collections, abandoned embedding namespaces and the pre-0.13 chromem tree, and runs or schedules database maintenance.

Core

Interface to listen on, as a bare address with no port. 127.0.0.1 makes the server reachable only from the machine it runs on — which is what the macOS app writes at first run, behind its Allow Network Access toggle.], ['CIX_AUTH_DISABLED', 'false', 'Dev only. Never set in production.'], ['CIX_LOG_LEVEL', 'info', <>debug|info|warn|error.], ]}/> @@ -279,9 +281,18 @@ cix reindex --full`} /sqlite/projects.db', 'Model-independent system DB.'], - ['CIX_CHROMA_PERSIST_DIR', '/chroma', 'Vector store; namespaced per provider/model.'], + ['CIX_VECTORS_DIR', '/vectors', 'The vector store — one SQLite database per embedding namespace (provider kind + model).'], + ['CIX_VECTOR_MMAP_SIZE', '0 (off)', <>PRAGMA mmap_size in bytes. Buys ~40% lower search latency by trading resident memory for it — mapped pages count in RSS.], + ['CIX_CHROMA_PERSIST_DIR', '/chroma', 'Legacy chromem-go tree. Read once for the import into the SQLite store, then never written — it is the rollback path.'], ['CIX_REPOS_DIR', '/repos', 'Server-side clones of GitHub-backed projects.'], ]}/> +

Database maintenance

+

Normally driven from Server → Resources → Database; these exist for deployments nobody opens a dashboard for. A compaction takes the server read-only and then restarts it — read doc/DATABASE_MAINTENANCE.md before scheduling one on a server other people use. Reclaim, the cheaper of the two, needs no window at all.

+

Embeddings & sidecar

Model for the local llama.cpp provider — HuggingFace repo or absolute .gguf path. Switch providers (Voyage, OpenAI) from the dashboard.], @@ -368,7 +379,13 @@ POST /api/v1/admin/users/{id}/reset-password GET/PUT /api/v1/admin/runtime-config GET /api/v1/admin/models · /login-locks POST …/login-locks/reset GET /api/v1/admin/embedding-providers GET/PUT …/active · POST …/{kind}/test -POST /api/v1/admin/sidecar/restart GET …/sidecar/status`} +POST /api/v1/admin/sidecar/restart GET …/sidecar/status + +GET /api/v1/admin/resources · …/resources/analyze +POST /api/v1/admin/resources/clean +GET /api/v1/admin/database POST …/database/compact · …/database/reclaim +PUT /api/v1/admin/database/auto-vacuum +GET /api/v1/admin/schedules PUT …/schedules/{name}`}
@@ -432,7 +449,7 @@ cix mcp uninstall claude-desktop`}
-

The official Claude Code plugin (v{PLUGIN_VERSION}) ships from the repo's marketplace. It is a client for your self-hosted cix server — install the server first (see Install); without a reachable server the commands and skills have nothing to talk to. It bundles the CLI, eight slash commands, two lazy-loading skills (cix + cix-workspace), a cix-workspace-investigator sub-agent for parallel cross-repo research, and five behavioral hooks. The skills were distilled from dozens of recorded agent sessions on large codebases, scored retrospectively for grep-vs-semantic effectiveness.

+

The official Claude Code plugin (v{PLUGIN_VERSION}) ships from the repo's marketplace. It is a client for your self-hosted cix server — install the server first (see Install); without a reachable server the commands and skills have nothing to talk to. It bundles the CLI, eight slash commands, two lazy-loading skills (cix + cix-workspace), a cix-workspace-investigator sub-agent for parallel cross-repo research, and hooks on five events. The skills were distilled from dozens of recorded agent sessions on large codebases, scored retrospectively for grep-vs-semantic effectiveness.

Install

{`# Run in a terminal — NOT inside a Claude Code session. claude plugin marketplace add dvcdsys/code-index diff --git a/site/src/landing/sections.jsx b/site/src/landing/sections.jsx index f7447ce6..cc959075 100644 --- a/site/src/landing/sections.jsx +++ b/site/src/landing/sections.jsx @@ -1,6 +1,6 @@ import { CLITabs } from './tabs.jsx'; import { WorkspaceDemo } from './workspace-demo.jsx'; -import { GITHUB_URL, PLUGIN_VERSION } from '../shared/versions.js'; +import { GITHUB_URL, MAC_APP_VERSION, PLUGIN_VERSION } from '../shared/versions.js'; import { TeamDiagram } from '../shared/team-diagram.jsx'; const FIT = [ @@ -82,9 +82,9 @@ const FEATURES = [ { glyph: '~', cls: 'moss', title: 'Live file watcher', body: 'Native filesystem events (FSEvents / inotify) with a 5-second debounce. Edit a file — the index follows. SHA-256 hashes mean only changed files re-embed.' }, { glyph: '⊕', cls: '', title: 'Embedded dashboard', - body: 'React SPA baked into the Go binary at /dashboard. Projects, search, users, API keys, runtime sidecar control. No extra service.' }, - { glyph: '⏻', cls: 'alt', title: 'Self-hosted. GPU optional.', - body: 'Single distroless container; your code stays on your network by default. CUDA image for NVIDIA, native Metal on Apple Silicon — or plain CPU: the model is 145 MB on disk, ~0.7 GB VRAM.' }, + body: 'React SPA baked into the Go binary at /dashboard. Projects, workspaces, search, users, API keys, view-groups — plus a Server page for runtime tuning, disk reclaim and database maintenance. No extra service.' }, + { glyph: '⏻', cls: 'alt', title: 'Self-hosted, and small enough to leave running', + body: 'Single distroless container, a menu bar app on Apple Silicon, CUDA image for NVIDIA — your code stays on your network by default. Vectors live in SQLite and are read per query rather than loaded into RAM, so an idle server sits at tens of megabytes whatever the index size. The model adds 145 MB on disk and ~0.7 GB while loaded.' }, ]; export function Features() { @@ -263,7 +263,7 @@ export function Agent() { const QS_STEPS = [ { n: '01', title: 'Run the installer', - desc: <>One interactive installer for every mode — Docker CPU, Docker CUDA (driver ≥ 525 + Container Toolkit), or native Metal on Apple Silicon. It picks the right mode for your machine, brings the server up, and installs + connects the cix CLI for you., + desc: <>One interactive installer for every mode — Docker CPU, Docker CUDA (driver ≥ 525 + Container Toolkit), or a native Metal build on Apple Silicon. It picks the right mode for your machine, brings the server up, and installs + connects the cix CLI for you. On a Mac, download the app instead — it installs the same server into ~/.cix/runtime/, creates your account and connects the CLI, so you land straight on step 04., code: <> $ curl -fsSL https://raw.githubusercontent.com{'\n'} {' '}/dvcdsys/code-index/main/install-server.sh | bash{'\n'} @@ -349,7 +349,7 @@ export function QuickStart() { const FAQS = [ { q: 'Does my code leave my machine?', - a: <>Not by default. The server runs on your hardware (Docker, native macOS, or your own GPU box), and embeddings happen locally via a llama.cpp sidecar — no SaaS endpoint, no telemetry. If you choose to switch the embedding provider to a remote API (Voyage, OpenAI-compatible), chunks go to that provider; that's an explicit admin action, off by default. }, + a: <>Not by default. The server runs on your hardware (Docker, the macOS menu bar app, a native build, or your own GPU box), and embeddings happen locally via a llama.cpp sidecar — no SaaS endpoint, no telemetry. If you choose to switch the embedding provider to a remote API (Voyage, OpenAI-compatible), chunks go to that provider; that's an explicit admin action, off by default. }, { q: "How is this different from Sourcegraph, GitHub code search, or Cursor's indexing?", a: <>Scope. Those give you search inside their surface — a web app, a code host, one editor. cix ships the whole platform as one MIT repo you run yourself: the Go server with an embedded dashboard, the CLI, the file watcher, multi-repo workspaces, a Claude Code plugin (slash commands, skills, hooks), an MCP server for Claude Desktop & Cowork, and team-deployment docs down to TLS, backups and upgrades. It's not an indexer you build a workflow around — it is the workflow, from docker compose up to your agent quoting file:line. }, { q: 'How is this different from indexing frameworks like CocoIndex or LlamaIndex?', diff --git a/skills/cix-workspace/SKILL.md b/skills/cix-workspace/SKILL.md index 790d1ada..0a06bfc7 100644 --- a/skills/cix-workspace/SKILL.md +++ b/skills/cix-workspace/SKILL.md @@ -553,7 +553,7 @@ unrelated repos. **The structural failure:** 1. Pure-dense fan-out cannot tell "no signal" apart from "weak - signal" — chromem always returns the K nearest vectors. + signal" — a vector search always returns the K nearest vectors. 2. Long natural-language queries dilute the few tokens that carry the actual gating signal. 3. Without a sparse-retrieval channel, an acronym or unique @@ -615,9 +615,9 @@ Either: ### `status: "partial_failure"` At least one repo errored out (`failed_repos` array names them). -Common cause: corrupt chromem collection. The remaining repos still -returned results. Surface to the user; don't silently treat as -complete. +Common cause: a missing or corrupt vector collection. The remaining +repos still returned results. Surface to the user; don't silently +treat as complete. ### Top-2 projects are at near-equal candidacy diff --git a/workspaces.md b/workspaces.md index 87df9ebb..993b543e 100644 --- a/workspaces.md +++ b/workspaces.md @@ -566,7 +566,7 @@ query ▼ For every indexed workspace repo, in parallel: │ - ├── dense path: chromem.Search(query_embedding) → top-50 by cosine + ├── dense path: vector store scan (query_embedding) → top-50 by cosine ├── sparse path: SQLite FTS5 BM25 over chunks_fts → top-50 by BM25 │ ▼ @@ -625,8 +625,8 @@ Request-time: ### Why hybrid The pre-hybrid algorithm (pure dense fan-out) had a known failure mode: -chromem always returns the nearest K vectors regardless of how far -"nearest" actually is. A workspace with repos that share zero +a nearest-neighbour search always returns the nearest K vectors +regardless of how far "nearest" actually is. A workspace with repos that share zero vocabulary with the query still surfaced 50 chunks per repo at noise-level cosine. BM25 fixes this: a repo that scores 0 on the literal token side gets caught by the relative project gate even if @@ -895,8 +895,8 @@ re-add it via `POST /git-repos` with the correct branch. `POST /api/v1/projects/{hash}/reindex` on each. **`status: "partial_failure"`** -→ At least one repo's dense search errored (corrupt chromem collection, -disk pressure). Other repos still returned. Check server logs; the +→ At least one repo's dense search errored (missing or corrupt vector +collection, disk pressure). Other repos still returned. Check server logs; the fastest fix is usually a reindex of the failed repo. **Webhook isn't triggering reindex**