diff --git a/.github/workflows/_ci.yml b/.github/workflows/_ci.yml index deb2664..b433cfa 100644 --- a/.github/workflows/_ci.yml +++ b/.github/workflows/_ci.yml @@ -55,7 +55,23 @@ jobs: if: startsWith(matrix.os, 'ubuntu-') run: | sudo apt-get update - sudo apt-get install -y cmake ninja-build git gcc g++ pkg-config libssl-dev ccache patchelf + # NOT `apt-get install cmake`: ubuntu-24.04 ships CMake 3.28.3 while + # the engine requires 3.30+ (engine/CMakeLists.txt:1, README "CMake + # 3.30+"), and CMake refuses to read a project above its own version. + # Every invocation therefore aborted at line 1 — + # `cmake_minimum_required` — so both Ubuntu jobs died before compiling + # a single file: the Rust step because server/build.rs drives cmake, + # and the engine step directly. macOS (Homebrew cmake, 4.x) configured + # and ran the tests without noticing, which is why this only ever + # showed up on Linux. The PyPI wheel pins one version for both Ubuntu + # runners and takes precedence over the distro package in /usr/local. + sudo apt-get install -y ninja-build git gcc g++ pkg-config libssl-dev ccache patchelf + pip install "cmake==3.31.6" + cmake --version + cmake --version | grep -qE "version 3\.(3[0-9]|[4-9])" || { + echo "::error::CMake 3.30+ is required (engine/CMakeLists.txt:1), got: $(cmake --version | head -1)" + exit 1 + } - name: Install deps (macOS) if: matrix.os == 'macos-15' @@ -186,38 +202,35 @@ jobs: # Run every C++ test binary produced under the build dir. # nullglob: unmatched globs expand to nothing instead of a literal. + # + # The set of tests is defined by engine/tests/*.cpp, NOT by a + # hand-maintained list. The previous hard-coded skip list had + # drifted badly: it skipped 26 tests whose recorded reasons + # ("known failing", "requires removed LadybugDB graph feature") + # were no longer true, including the entire per-language + # false-positive precision suite (test_fp_*). Those tests all pass + # on this tree, so skipping them only hid regressions. + # + # Keep skip_tests EMPTY unless a test genuinely cannot run here. + # Never add an entry to silence a failure — fix the test or remove + # it from engine/tests/ with a tracked issue. + skip_tests="" shopt -s nullglob failed=0 for test_bin in engine/build-tests/test_*; do # Only run regular executable files (skip dirs / non-binaries). [ -f "$test_bin" ] && [ -x "$test_bin" ] || continue test_name=$(basename "$test_bin") - # Skip known manual/debug tools that need external args, - # and known-failing tests tracked for future fix. - # This list must match the Makefile's TEST_EXES skip list. - case "$test_name" in - test_bench|test_bench_enhance|test_bench_project|test_pipeline_bench|test_fast_scan_debug|test_verify_aiscope) - echo " SKIP $test_name (requires external args)" - continue - ;; - test_enhance_e2e|test_js_visitor|test_ts_visitor|test_tsx_visitor) - echo " SKIP $test_name (known failure, tracked for future fix)" - continue - ;; - test_rust_e2e|test_qualified_id_ast|test_ts_e2e|test_type_extraction) - echo " SKIP $test_name (requires removed LadybugDB graph feature)" - continue - ;; - test_c_e2e|test_cpp_e2e|test_e2e|test_go_e2e|test_java_e2e|test_js_e2e) - echo " SKIP $test_name (find_def requires removed LadybugDB graph feature)" - continue - ;; - test_fp_c|test_fp_cpp|test_fp_go|test_fp_java|test_fp_js|test_fp_python|test_fp_rust|test_fp_ts) - echo " SKIP $test_name (known failing test, tracked for future fix)" - continue - ;; - test_call_graph_method|test_call_graph_p1|test_membulk_parity|test_query_algorithms) - echo " SKIP $test_name (requires graph_nodes table, migrated to entity/relation)" + # A restored build cache can contain binaries for tests that no + # longer exist in the tree; running those would fail for the + # wrong reason. The source tree is the source of truth. + if [ ! -f "engine/tests/${test_name}.cpp" ]; then + echo " SKIP $test_name (no engine/tests/${test_name}.cpp)" + continue + fi + case " $skip_tests " in + *" $test_name "*) + echo " SKIP $test_name (listed in skip_tests)" continue ;; esac diff --git a/.github/workflows/_lint.yml b/.github/workflows/_lint.yml index 076bbd1..fae6406 100644 --- a/.github/workflows/_lint.yml +++ b/.github/workflows/_lint.yml @@ -17,7 +17,14 @@ jobs: - name: Install system deps run: | sudo apt-get update - sudo apt-get install -y cmake ninja-build + # cmake is not installed from apt: ubuntu-24.04 ships 3.28.3 and the + # engine requires 3.30+ (engine/CMakeLists.txt:1). `cargo clippy` + # builds server/build.rs, which drives cmake over the engine, so the + # old version made this job fail for a reason that had nothing to do + # with linting. Same pinned wheel as _ci.yml. + sudo apt-get install -y ninja-build + pip install "cmake==3.31.6" + cmake --version # Install clang-format via pip — the PyPI wheel provides an # exact version match (21.1.8) with the local dev environment # (Homebrew llvm@21, which is what `which clang-format` resolves diff --git a/.github/workflows/dev.yml b/.github/workflows/dev.yml index 5358ac5..27749b4 100644 --- a/.github/workflows/dev.yml +++ b/.github/workflows/dev.yml @@ -5,7 +5,11 @@ name: Dev CI (Windows Cross-Compile) on: workflow_dispatch: push: - branches: [hhh] + # `dev` is the integration branch this workflow is named after; without it + # the Windows cross-compile only ever ran on `master`, so a regression in + # build.rs cross-compilation (e.g. the target-cpu=native failure fixed in + # d44f0fd) could not be caught before merge. + branches: [dev, master] permissions: contents: read @@ -49,3 +53,77 @@ jobs: run: | file target/x86_64-pc-windows-gnu/release/codescope.exe | grep -i "PE32" echo "✅ Windows cross-compile succeeded" + + # ── TD-5 step 3: run the parallel scheduler on a real Windows host ── + # The cross-compile job above proves the binary *links*; it cannot prove the + # shared-file chunk queue works, because that is a runtime property of two + # Windows processes mapping the same file (`src/scheduler/mapped_file.rs`). + # This job indexes a real project twice — once through the module-level + # scheduler, once through the chunk-level (work-stealing) one — and asserts + # the graph came out non-empty both times. + # + # Three Windows-specific requirements, all handled below: + # * CodeScope needs the GNU ABI (build.rs panics on msvc), so even on a + # Windows host the build target is x86_64-pc-windows-gnu and MinGW's + # gcc/g++ must be on PATH — build.rs looks for plain `gcc`/`g++`. + # * The merge step of the parallel scheduler shells out to the `sqlite3` + # CLI (`scheduler/merge.rs`), which the runner image does not ship. + # * The chunk queue and per-module DBs live in the platform temp directory, + # so no /tmp-like path is assumed. + # + # NOTE: this job has not been executed yet (it needs a Windows runner) and is + # deliberately reachable through workflow_dispatch only, so it cannot redden a + # normal push until its first successful run promotes it. + windows-smoke: + name: Windows smoke (index-parallel) + if: github.event_name == 'workflow_dispatch' + runs-on: windows-latest + timeout-minutes: 40 + steps: + - uses: actions/checkout@v7 # v4.2.2 + + - name: Setup Rust + uses: dtolnay/rust-toolchain@1.95.0 # pinned to match local dev + + - name: Install MinGW (gcc/g++ for the engine) + sqlite3 (merge step) + uses: msys2/setup-msys2@v2 + with: + msystem: MINGW64 + update: false + install: mingw-w64-x86_64-gcc mingw-w64-x86_64-sqlite3 + path-type: inherit + + - name: Add Windows GNU target + run: rustup target add x86_64-pc-windows-gnu + + - name: Verify the toolchain the build depends on + shell: pwsh + run: | + gcc --version + g++ --version + sqlite3 --version + + - name: Build codescope.exe + run: cargo build --release --target x86_64-pc-windows-gnu --bin codescope + + - name: index-parallel indexes a real project (module scheduler) + shell: pwsh + run: | + $exe = "target/x86_64-pc-windows-gnu/release/codescope.exe" + $result = & $exe index-parallel engine/src --workers 2 --parallel 2 2>$null | ConvertFrom-Json + $result | ConvertTo-Json -Depth 3 -Compress | Write-Host + if ($result.ok -ne $true) { throw "index-parallel reported ok=$($result.ok)" } + if ($result.total_nodes -le 0) { throw "no nodes indexed (total_nodes=$($result.total_nodes))" } + Write-Host "✅ module scheduler: $($result.total_nodes) nodes / $($result.total_edges) edges" + + - name: index-parallel indexes a real project (chunk scheduler) + shell: pwsh + run: | + $env:CODESCOPE_CPU_DYNAMIC = "1" + $exe = "target/x86_64-pc-windows-gnu/release/codescope.exe" + $result = & $exe index-parallel engine/src --workers 2 --parallel 2 2>$null | ConvertFrom-Json + $result | ConvertTo-Json -Depth 3 -Compress | Write-Host + if ($result.sched_mode -ne "chunked") { throw "expected the chunked scheduler, got '$($result.sched_mode)'" } + if ($result.complete -ne $true) { throw "chunked run reported complete=$($result.complete)" } + if ($result.total_nodes -le 0) { throw "no nodes indexed (total_nodes=$($result.total_nodes))" } + Write-Host "✅ chunk scheduler: $($result.total_nodes) nodes, $($result.success) chunk workers" diff --git a/.gitignore b/.gitignore index ea4b57a..59a12bb 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,10 @@ .codescope/ .cursh/ .mcp.json +# Agent worktrees. These are full copies of the repository, so indexing them +# doubles every symbol and makes by-name lookup ambiguous (find_callers returns +# candidates instead of callers). They must never be part of the project. +.kilo/ # IDE and Editors @@ -186,7 +190,26 @@ *.db-* *.db -plan/ +# Generated by `make test-savings`: the numbers change on every run, so the file +# is an output rather than a source. It stays on disk after the run. +tests/token_savings_report.md + +# Local planning scratch. `plan/*` rather than `plan/` so that a single +# subdirectory can be exempted below: git will not re-include anything inside an +# excluded DIRECTORY, so the exemption has to start one level up. +plan/* +# …except the rules, which are cited from a lot of places that a fresh clone +# has to be able to follow — CONTRIBUTING links plan/rules/code_rules.md, and +# ~40 source comments point at it and at relation_contract.md by name. +!plan/rules/ +# The scheduler cites this one by section number from five source files +# (chunk_plan.rs "§5", chunk_queue.rs "§4, §6", chunk_queue_ops.rs "§8", …), so +# it has to be in the repo for those references to mean anything. Same two-step +# as above: the directory has to be re-included before a file inside it can be, +# which is why `!plan/next/` comes before `plan/next/*`. +!plan/next/ +plan/next/* +!plan/next/DYNAMIC_SCHED_REDESIGN.md **/build_*/ **/build-*/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 0347d1f..0203f4c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,49 @@ # Changelog +## v0.2.7 (2026-10-09) + +First release since v0.2.6 (`main`, 2026-08-14): **41 commits, 394 files**. Two themes: the architectural work for 0.3 (handle-based ABI, Windows parallel indexing), and silent failure — the cases where a tool answered confidently and wrongly, on the default path, for real projects. + +### 🚀 New Features + +- **Handle-based engine ABI (TD-1, three knives)** (`engine/include/engine.h`, `engine/src/engine_context.*`): `engine_create()` returns an opaque `engine_t`, all 73 stateful entry points take it, and the three process-global singletons are gone — no global object is constructed before `main()`, and a stale call site is a compile error. The still process-wide subsystems (verifier registry, async knowledge builder, index-progress and parse-failure buffers) are recorded in `engine_context.h` rather than implied. Evidence: a 43-tool differential matrix is byte-identical old vs new, and both binaries index one tree to the same call-edge set. +- **`index-parallel` on Windows (TD-5)** (`server/src/scheduler/mapped_file.rs`): the POSIX `open`/`ftruncate`/`mmap` sequence both schedulers open-coded now lives in one mapping layer with a Windows backend (`CreateFileMappingW`/`MapViewOfFile`), and per-run paths come from `std::env::temp_dir()` instead of `/tmp`. A `windows-smoke` job indexes a real project through both schedulers — **it has not run yet**, so Windows stays Beta (README). +- **Graph-shape regression baselines** (`benchmarks/graph_baseline.sh`, `make bench-graph`): six real checkouts, one per language, compared on files, entities, call edges, `resolution_kind`, `resolve_strategy`, `deciding_factor` and unresolved sites; exact rather than tolerance-based because the shape is reproducible. `self` 2274 · `c-redis` 27474 · `go-tinygo` 4313 · `java-spring-petclinic` 47 · `rust-pyo3` 2859 · `ts-codebase-memory-mcp` 8440 call edges. +- **Counterfactual resolver-factor switch** (`CODESCOPE_RESOLVER_DISABLE_FACTOR=`): drops one factor's weight and score at the single accumulation point, so a factor's real effect is measured by A/B instead of argued from the weight table — `relation.reason`'s `decided_by=` names ImportMatch for every scored edge on this repository, while disabling factors shows which ones actually move the graph. No weight is changed until that table says otherwise. +- **`codescope parse-failures` / `reset-failures`** (the `parse_failures` table was write-only), **`get_communities` restored** (synchronous label propagation; deterministic; summary-first — the earlier revision returned ~100K tokens for one query), **`graph_query` honours `LIMIT`** and rejects trailing text instead of dropping it silently. + +### 🐛 Bug Fixes + +**Indexing and symbol names** + +- **A Maven tree indexed nothing**: the engine's test/docs/samples skip collided with Java package components, so `src/main/java/…/samples/petclinic/` was skipped at any depth and `discover_modules` reported no modules. Both entry points now share `applyProjectLanguageContext()`; spring-petclinic goes 0 → 30 files. +- **Java methods were named after their return type** (`JavaVisitor::extractName` accepted `type_identifier` and returned the first match): `public String processUpdateForm(…)` was recorded as an entity called `String` — 27 such entities in petclinic, 4 `Pet`, 3 `LocalDate` — while `void` methods were correct, and those type-named entities then collected edges. Names are read from the grammar's `name` field; petclinic 33 → 47 edges and `find_definition("processUpdateForm")` resolves. +- **The same call shape resolved for Go, Java and Python and was dropped for C++, Rust and TypeScript**: a parameter is a declaration, but only variable declarations fed the visitors' type table, so a call through a parameter carried a receiver with no type and the ambiguity gate abstained. All three now record parameter types; 6 of 6 languages resolve to the owner's method in `test_resolver_language_consistency`. +- **`arity` and `param_count` had no producer**: `param_count` was structurally 0 (no visitor emits a `Parameter` record) and `arity` — the input the resolver's overload step reads — was 0 for all 24604 goagent rows, so every candidate scored as "unknown arity". Both come from the CST declaration now, with a variadic tail or a default reported as unknown rather than a count that would penalise a valid call. +- **TypeScript annotations lost array element types and leaked object-type fragments**: `extractTsTypeAnnotation` unwrapped element and member types as if they were annotations, so `Foo[]` yielded nothing, and it returned raw text for shapes it does not model, so `const tabs: { id: TabId; … }[]` recorded the receiver type of `tabs.map(…)` as `id: TabId`. Element types resolve and anything that is not a plain type name is reported as unknown. +- **Builtin-name filters deleted calls to the file's own functions** (a user `format()`, `valueOf()`, `write()` or `free()` collided with the filter's list and produced no call record at all): each visitor now keeps the names the file declares, and the filter applies only to unqualified calls. +- **C/C++ functions with a qualified return type were invisible** (`std::string`, `char *`, namespace-nested): the name scan matched the return type and dropped the whole definition — on this repository `engine/src/query/**` contributed 3 definitions out of 1972 statements. Names are read from the `declarator` field. +- **Aliased imports resolved to nothing** (`import { pick as pickLib }` bound a local name with no record of the imported symbol) and **a call omitting a default argument lost its edge** (C++ declares the default in the header while the entity's `arity` comes from the definition). + +**Determinism** — one tree, one binary, one graph: the two resolver loader tables are ordered by content (a worker-dependent rowid order decided which interface six call sites on goagent resolved to), and entity ids derive from the record's semantic identity instead of SQLite's scan order. + +**Silent success and silent drops** — a run that indexed nothing no longer reports `ok:true` (0-file and no-symbol runs report `ok:false`/`complete:false`); failed CSR builds, bulk COMMITs, nested transactions, migrations and batch inserts roll back and report instead of committing whatever landed; `index-parallel` installs its database at the resolved path (it only reported a temp path, so the next command queried an empty DB) and runs the post-index pass, so the DB it hands back is searchable; `force_index_files` is no longer filtered at the graph stage (forcing `engine/tests` added 13174 records and zero entities); `createSchema` no longer has a use-after-free; the CMake generator is resolved from the build tree, because a mismatched generator silently kept new translation units out of the archive. + +**MCP surface** — `isError` was serialized as `is_error`, so a spec-compliant client saw no failure flag for any tool result; an MCP-indexed project had no `projects` row (every entity landed under `project_id = 0`, next to a full graph); front-end build output was indexed (`dist/`: 700 minified nodes beside 224 real symbols); array-shaped `node_types`/`edge_types` filters were silently ignored; `project_overview.entry_points` read the legacy table and was always empty; `verify_summary` and `verify_review` scored the same empty input 1.0 and 0.0; ambiguous names returned an empty list with no next step. + +**Paths and tool contracts** — `detect_changes` reported "nothing changed" for the input shape its own schema advertises and for any path spelling other than the stored one; `find_definition` / `find_references` / `find_callers` / `find_callees` missed a file named through a symlink; `shortest_path` and `codescope_trace` advertised no required arguments; a failing engine test could pass the gate (the `CHECK` counter was consulted only by `return checkFailures() ? 1 : 0`, which 8 test files never wrote). + +### 🧹 Cleanup + +- **One JSON implementation** (TD-3): every hand-built envelope migrated to `util::JsonWriter`, escaping converged on `util::jsonEscapeString`, and the duplicate escapers and declarations deleted. +- **~1.5k lines of dead code removed** (`scanner_visitor`, the superseded resolver stack, the zero-call-site scorers, five duplicate copies of one scoring rule), the superseded `Planner`/`VerdictBuilder` chain deleted, and `verify_statement` retired now that `verify_claim` is the single entry point. + +### ✅ Testing + +- Every engine test uses `CHECK()` (929 assertion sites migrated, no live `assert()` remains) and a recorded failure makes the binary exit non-zero — enforced by an exit hook, because 8 files printed a banner and returned 0. +- New suites: `test_resolver_language_consistency` (six languages, one call shape), `test_import_alias_resolution`, `test_function_arity`, `test_receiver_type_canonical`, `test_dispatch_order_determinism`, `test_metrics_persist`, `test_force_index_files`, `test_engine_handles` (instance and ABI boundaries), `test_resolver_language_filter`, `test_gitignore_build_dirs`, plus property-based tests over clamping, exclude-path escaping and MCP payload round-trips. +- `make check` green: 106 engine test binaries, 136 server tests; call-graph accuracy gate TP 36 / FP 0 / FN 0; `make bench-graph` matches all six baselines. + ## v0.2.6 (2026-08-12) Improves full-index (non-fast) performance on large projects, fixes a resolver JOIN defect that both slowed indexing and silently over-matched cross-file references, and moves fuzzy symbol search fully in-memory — resolver 10.2x faster on CodeScope's own index with zero precision loss, plus completed FAST-mode pruning rules and quantified discovery timing. @@ -601,3 +645,9 @@ Open-source release. Closes the gap between the Resolver Pipeline and the query/ - `docs/en/final_benchmark_report.md` + `docs/zh/final_benchmark_report.md`: Comprehensive 5-project comparison table. - `skills/`: Tool usage guides with token consumption tables. - All ASCII art diagrams replaced with mermaid format. + +### 🛠 Build / Tooling + +- **`make clean` verifies that it actually cleaned** (`Makefile`): a clean that silently does not happen is worse than no clean, because the next build reuses the stale tree — and a delete can be blocked by permissions or a sandbox guard while the recipe still prints success. The target now checks that the build directories are gone and fails loudly if they are not. +- **CI now runs the full format check** (`Makefile`): `lint-cpp` checks only files modified in the last hour (a 3-second target for local iteration), so a file mis-formatted yesterday and untouched today passed `make check`. The gate depends on a new `lint-verify` target (`lint-cpp-full` + clippy). + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 04ffea4..0de0bdd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -9,7 +9,7 @@ CodeScope is a **Project Truth Engine**: it transforms source code into verifiab facts, models, and inspectable evidence so AI can validate claims against reality instead of hallucinating. It is built from a C++ core engine (parsing via tree-sitter → unified IR → facts store → resolver → model/verification) and a Rust -MCP server that exposes ~37 tools to AI clients. CodeScope runs locally and indexes +MCP server that exposes 47 tools to AI clients. CodeScope runs locally and indexes source code on the user's machine. ## Development Setup @@ -67,8 +67,8 @@ Follow [plan/rules/code_rules.md](plan/rules/code_rules.md). Key rules: `go_visitor.cpp` / `go_translator.cpp`. 6. Add the new translator/visitor files to `ENGINE_SOURCES` in `engine/CMakeLists.txt`. -7. Add tests (e.g. `engine/tests/test_fp_foo.cpp`) and list them in `TEST_EXES` in - the `Makefile`. +7. Add tests as `engine/tests/test_fp_foo.cpp`. There is no list to update: + `make test-engine` derives its executables from `engine/tests/*.cpp`. ## Adding a New MCP Tool @@ -81,7 +81,7 @@ Follow [plan/rules/code_rules.md](plan/rules/code_rules.md). Key rules: `server/src/tools/mod.rs`. 4. Register it in the `TOOL_HANDLERS` map in the same file. 5. Add a `Tool { name, description, input_schema }` entry to `all_tools()` in - `server/src/tools/mod.rs`. + `server/src/tools/catalog.rs`. A unit test (`test_all_tools_have_registered_handler`) enforces that every tool returned by `all_tools()` has a registered handler — do not skip step 4. @@ -89,13 +89,22 @@ returned by `all_tools()` has a registered handler — do not skip step 4. ## Testing Requirements - `make check` must pass before opening a PR (build + lint + tests). -- All automated tests listed in `TEST_EXES` (Makefile) are run by - `make test-engine`. If you add a new automated test under `engine/tests/`, - **add it to `TEST_EXES`** so it is not silently skipped. +- Every `engine/tests/*.cpp` is built and run by `make test-engine` — the list of + executables is derived from those sources, so a new file is picked up + automatically and cannot silently go unrun. `TEST_EXCLUDES` in the `Makefile` + must stay empty unless a test genuinely cannot run there (it needs external + arguments or a network service); never add an entry to hide a failure. - Automated tests must run without command-line arguments. Tools that need external args (e.g. a source tree to scan) belong in `engine/manual/` (built only with `-DBUILD_MANUAL=ON`), not in `engine/tests/`. - Rust tests: `cd server && cargo nextest run` (or `cargo test`). +- Regression tests name the finding they pin (`2026-09-27 review D1-1`, + `2026-09-18 review #15`) instead of quoting it. Those point-in-time review + ledgers are removed from the tree once their findings are closed or recorded + as deliberate deviations — a stale finding list gets quoted as current. Resolve + an ID through git history: + `git log --diff-filter=D --oneline -- CODE_REVIEW_2026-09-27.md`, then + `git show ^:CODE_REVIEW_2026-09-27.md`. ## Pull Request Process diff --git a/Cargo.lock b/Cargo.lock index ab062b5..5a5733d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2,15 +2,95 @@ # It is not intended for manual editing. version = 4 +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "bit-set" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3" +dependencies = [ + "bit-vec", +] + +[[package]] +name = "bit-vec" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7" + +[[package]] +name = "bitflags" +version = "2.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06" + +[[package]] +name = "cfg-if" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600" + [[package]] name = "codescope" -version = "0.2.6" +version = "0.2.7" dependencies = [ "libc", "once_cell", + "proptest", "serde", "serde_json", "walkdir", + "windows-sys", +] + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "fastrand" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" + +[[package]] +name = "fnv" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "libc", + "r-efi 5.3.0", + "wasip2", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi 6.0.0", ] [[package]] @@ -25,18 +105,42 @@ version = "0.2.186" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + [[package]] name = "memchr" version = "2.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + [[package]] name = "once_cell" version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + [[package]] name = "proc-macro2" version = "1.0.106" @@ -46,6 +150,31 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "proptest" +version = "1.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b45fcc2344c680f5025fe57779faef368840d0bd1f42f216291f0dc4ace4744" +dependencies = [ + "bit-set", + "bit-vec", + "bitflags", + "num-traits", + "rand", + "rand_chacha", + "rand_xorshift", + "regex-syntax", + "rusty-fork", + "tempfile", + "unarray", +] + +[[package]] +name = "quick-error" +version = "1.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0" + [[package]] name = "quote" version = "1.0.46" @@ -55,6 +184,87 @@ dependencies = [ "proc-macro2", ] +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rand" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" +dependencies = [ + "rand_chacha", + "rand_core", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + +[[package]] +name = "rand_xorshift" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "513962919efc330f829edb2535844d1b912b0fbe2ca165d613e4e8788bb05a5a" +dependencies = [ + "rand_core", +] + +[[package]] +name = "regex-syntax" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" + +[[package]] +name = "rustix" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "891efababe418670775f199f0d233d84843c227a0949a883ce15b37c78d6629d" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + +[[package]] +name = "rusty-fork" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc6bf79ff24e648f6da1f8d1f011e9cac26491b619e6b9280f2b47f1774e6ee2" +dependencies = [ + "fnv", + "quick-error", + "tempfile", + "wait-timeout", +] + [[package]] name = "same-file" version = "1.0.6" @@ -118,12 +328,40 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom 0.4.3", + "once_cell", + "rustix", + "windows-sys", +] + +[[package]] +name = "unarray" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94" + [[package]] name = "unicode-ident" version = "1.0.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" +[[package]] +name = "wait-timeout" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ac3b126d3914f9849036f826e054cbabdc8519970b8998ddaf3b5bd3c65f11" +dependencies = [ + "libc", +] + [[package]] name = "walkdir" version = "2.5.0" @@ -134,6 +372,15 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "wasip2" +version = "1.0.4+wasi-0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" +dependencies = [ + "wit-bindgen", +] + [[package]] name = "winapi-util" version = "0.1.11" @@ -158,6 +405,32 @@ dependencies = [ "windows-link", ] +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "zerocopy" +version = "0.8.59" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6df92bf3d9227be3d53173901ddbffac2babc27ae50f397776ffd6dc33f800cb" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.59" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac4f328cf2f05d084e496c3e9c3f33ed0a183656a16e1fcec4d464d8373aec82" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "zmij" version = "1.0.23" diff --git a/Makefile b/Makefile index 0663c0b..d51b350 100644 --- a/Makefile +++ b/Makefile @@ -2,6 +2,7 @@ SHELL := /bin/bash .PHONY: all build build-engine build-server \ test test-engine test-server test-bench test-savings \ accuracy-check \ + bench-check bench-full bench-graph bench-graph-update \ lint lint-cpp lint-rust fmt fmt-cpp fmt-rust check \ clean distclean help @@ -91,9 +92,25 @@ ifneq (,$(findstring Darwin,$(UNAME_S))) ENGINE_CMAKE_FLAGS += -DCMAKE_OSX_SYSROOT=$(shell xcrun --show-sdk-path) endif -# Use Ninja if available for faster builds -BUILD_GENERATOR := $(shell which ninja >/dev/null 2>&1 && echo "Ninja" || echo "Unix Makefiles") -ENGINE_LIB := $(BUILD_DIR)/libastgraph_engine.a +# ─── Build Generator ───────────────────────────────────────────── +# The generator belongs to the BUILD TREE, not to the current shell. CMake +# refuses to reconfigure an existing tree with a different -G ("does not +# match the generator used previously"), and the previous recipe hid that +# failure by piping the configure output through `tail`, whose exit status +# is always 0. The result: once ninja was installed alongside an existing +# Unix-Makefiles tree, every CMakeLists.txt change re-ran a failing +# configure that make believed had succeeded, the stale build system was +# reused, and newly added source files were silently never compiled. +# +# Resolution order: +# 1. BUILD_GENERATOR=... on the command line (explicit) +# 2. the generator recorded in $(BUILD_DIR)/CMakeCache.txt +# 3. Ninja when installed, else Unix Makefiles (fresh tree only) +# `make clean` removes the tree, so the next build re-detects from scratch. +ENGINE_LIB := $(BUILD_DIR)/libastgraph_engine.a +CMAKE_CACHE := $(BUILD_DIR)/CMakeCache.txt +CACHED_GENERATOR := $(shell sed -n 's/^CMAKE_GENERATOR:INTERNAL=//p' "$(CMAKE_CACHE)" 2>/dev/null) +BUILD_GENERATOR ?= $(if $(strip $(CACHED_GENERATOR)),$(CACHED_GENERATOR),$(shell which ninja >/dev/null 2>&1 && echo "Ninja" || echo "Unix Makefiles")) $(BUILD_DIR): @mkdir -p $(BUILD_DIR) @@ -112,9 +129,19 @@ else CMAKE_GEN_FILE := $(BUILD_DIR)/Makefile endif +# When the recorded generator and BUILD_GENERATOR disagree, the tree cannot be +# reconfigured in place. Checked here (not with $(error) at parse time) so that +# `make clean` — the documented remedy — still runs. $(CMAKE_GEN_FILE): $(ENGINE_DIR)/CMakeLists.txt $(wildcard $(ENGINE_DIR)/cmake/*.cmake) | $(BUILD_DIR) + @if [ -n "$(strip $(CACHED_GENERATOR))" ] && [ "$(BUILD_GENERATOR)" != "$(CACHED_GENERATOR)" ]; then \ + printf "$(CROSS) generator mismatch: $(BUILD_DIR) was configured with '$(CACHED_GENERATOR)' but BUILD_GENERATOR is '$(BUILD_GENERATOR)'\n"; \ + printf " Run 'make clean' to reconfigure the tree from scratch.\n"; \ + exit 1; \ + fi @printf "$(CYAN)[engine]$(RESET) Configuring CMake ($(BUILD_GENERATOR))...\n" - @cd $(BUILD_DIR) && cmake -G "$(BUILD_GENERATOR)" $(CURDIR)/$(ENGINE_DIR) $(ENGINE_CMAKE_FLAGS) 2>&1 | tail -3 + @# pipefail (bash): a failing configure must abort the build instead of + @# being masked by `tail`'s exit status (see the note above). + @cd $(BUILD_DIR) && set -o pipefail && cmake -G "$(BUILD_GENERATOR)" $(CURDIR)/$(ENGINE_DIR) $(ENGINE_CMAKE_FLAGS) 2>&1 | tail -5 $(ENGINE_LIB): $(CMAKE_GEN_FILE) @printf "$(CYAN)[engine]$(RESET) Building C++ engine...\n" @@ -136,59 +163,31 @@ test: test-engine test-server @printf "$(CHECK) all tests passed\n" # Automated C++ test executables run by `make test-engine`. -# Every automated test_*.cpp in engine/tests/ MUST be listed here so it -# does not silently go unrun. Excluded tools are NOT automated tests: -# - test_fast_scan, test_fast_scan_debug, test_verify_aiscope: manual -# debug tools needing external args ( ); -# moved to engine/manual/ (built only with -DBUILD_MANUAL=ON). -# - test_bench, test_bench_enhance, test_bench_project, -# test_pipeline_bench, test_bun: benchmarks / manual debug, built -# on demand (see test-bench / bench-check targets). -# Previously-excluded tests (now passing, wired back into TEST_EXES): -# - test_enhance_e2e: was a duplicate engine_free_string(st) in the test -# (double-free SIGABRT) + trace_path gated on callgraph_ready which -# index_project now sets after buildGraph. Both fixed. -# - test_js_visitor, test_ts_visitor, test_tsx_visitor: unit tests for the -# Js/Ts/Tsx translators that dlopen the grammar .so at runtime — they need -# GRAMMARS_DIR set (see test-engine target below). They pass once the -# grammar .so files are on disk under engine/grammars. -# test_evidence_builder, test_project_state, test_domain_rules, -# test_verify_planner, and test_self_bench were previously excluded for -# hardcoded local paths; they now resolve rules/CWD portably, and the -# three that pass are wired into TEST_EXES above. The other two -# (test_verify_planner, test_self_bench) were removed: their assertions -# referenced pre-v0.2.5 verify-pipeline / graph_nodes-table behavior -# that the SQLite-only store no longer fills. -TEST_EXES := \ - test_ir test_graph test_graph_semantic test_graph_call_precision \ - test_semantic_unit \ - test_e2e test_c_e2e test_cpp_e2e test_go_e2e test_rust_e2e \ - test_js_e2e test_ts_e2e test_java_e2e \ - test_fp_c test_fp_cpp test_fp_go test_fp_js test_fp_ts test_fp_python \ - test_fp_rust test_fp_java \ - test_type_extraction test_state_builder_batch test_module_edge \ - test_module_path_column \ - test_model_engine test_claim_parser test_verifier_registry \ - test_fuzzy_resolver test_resolver_fuzzy_cache \ - test_documentation_drift test_capability_drift test_architecture_drift \ - test_query_algorithms test_connected_components_ffi test_trigram_search \ - test_exclude_paths test_index_metrics \ - test_call_graph_p1 test_readme_ingestion test_call_graph_method \ - test_project_id \ - test_enhance_e2e \ - test_js_visitor test_ts_visitor test_tsx_visitor \ - test_semantic_fact_extractor \ - test_accuracy_baseline \ - test_verifier_lifecycle test_verifier_claim_coverage \ - test_verifier_ground_truth \ - test_typed_relation_query \ - test_metrics_readiness \ - test_call_graph_accuracy \ - test_step11_go_smoke \ - test_evidence_builder test_project_state test_domain_rules +# +# The list is DERIVED from engine/tests/*.cpp so a new test can never +# silently go unrun. The previous hand-maintained list had drifted: it +# omitted six tests that were passing and in the tree (test_membulk, +# test_membulk_parity, test_resolve_strategy, test_homonym_filter, +# test_parent_chain, test_self_inspect), and the matching CI list skipped +# 26 more for reasons that were no longer true. CMake already builds every +# engine/tests/*.cpp (file(GLOB), see engine/CMakeLists.txt), so the +# sources are the single source of truth. +# +# TEST_EXCLUDES must stay empty unless a test genuinely cannot run under +# `make test-engine` (for example it needs external arguments or a network +# service). Never add an entry to hide a failure — either fix the test or +# delete it from engine/tests/ with a tracked issue. +TEST_EXCLUDES := +TEST_EXES := $(filter-out $(TEST_EXCLUDES), \ + $(patsubst $(ENGINE_DIR)/tests/%.cpp,%,$(wildcard $(ENGINE_DIR)/tests/*.cpp))) test-engine: $(ENGINE_LIB) @printf "$(CYAN)[test/engine]$(RESET) Building and running C++ tests...\n" + @# code_rules §4 / REVIEW_0.2.7 TEST-3: engine tests use CHECK() from + @# engine/tests/test_check.h, never assert(). CHECK() is always evaluated + @# (NDEBUG cannot compile it out) and a recorded failure makes the test + @# binary exit non-zero, so the build type no longer decides whether a + @# test actually checks anything. @rm -f $(TEST_DB) $(TEST_DB)-wal $(TEST_DB)-shm @rm -f /tmp/test_*.db /tmp/test_*.db-wal /tmp/test_*.db-shm 2>/dev/null || true @rm -f /tmp/codescope_test_*.db /tmp/codescope_test_*.db-wal /tmp/codescope_test_*.db-shm 2>/dev/null || true @@ -303,20 +302,50 @@ else endif @printf "$(CHECK) bench-full complete\n" +# Graph-SHAPE regression: not speed, but the RESULT — how many call edges the +# resolver produces on fixed multi-language checkouts and how they distribute +# across resolution kinds, parse-time strategies and deciding factors. A visitor +# or factor change that moves real edges is invisible to the unit tests, which +# pin hand-built fixtures; this compares against benchmarks/baselines/ and exits +# non-zero on any difference. Relocate the checkouts with +# CODESCOPE_BENCH_PROJECTS; missing ones are skipped. +bench-graph: + @printf "$(CYAN)[bench/graph]$(RESET) Comparing graph shape against baselines...\n" + @bash $(BENCH_DIR)/graph_baseline.sh + +bench-graph-update: + @printf "$(CYAN)[bench/graph]$(RESET) Rewriting graph baselines (intended changes only)...\n" + @bash $(BENCH_DIR)/graph_baseline.sh --update + $(BENCH_BIN): $(ENGINE_LIB) @cmake --build $(BUILD_DIR) -j$(NPROC) 2>&1 | tail -1 # ─── Lint ──────────────────────────────────────────────────────── -LINT_CPP_FILES := $(shell find $(ENGINE_DIR)/src $(ENGINE_DIR)/include -name '*.cpp' -o -name '*.h' | grep -v build) +# Build trees are pruned by PATH. The previous `| grep -v build` dropped any +# path containing "build" anywhere, which silently excluded eight real source +# files — state_builder.{cpp,h}, project_state_builder.{cpp,h}, +# graph_builder.{cpp,h} and evidence_builder.{cpp,h} — from clang-format and +# from `make check`'s lint-verify step. state_builder.cpp had drifted 50 lines +# without a single gate noticing, and the code written there during this work +# passed the check while failing clang-format on its own. +CPP_LINT_PRUNE := \( -path '*/build/*' -o -path '*/build-*/*' \) -prune -o +LINT_CPP_FILES := $(shell find $(ENGINE_DIR)/src $(ENGINE_DIR)/include $(CPP_LINT_PRUNE) \( -name '*.cpp' -o -name '*.h' \) -print) lint: lint-cpp lint-rust @printf "$(CHECK) lint complete\n" +# What CI runs. `lint-cpp` only formats-check the files modified in the last +# hour (a deliberate 3s target for local iteration), so a file that was +# mis-formatted yesterday and not touched today passed `make check`. The full +# check costs well under a second here, so the gate uses it. +lint-verify: lint-cpp-full lint-rust + @printf "$(CHECK) lint (full) complete\n" + # Fast lint-cpp (3s target) - only check recently modified files lint-cpp: $(BUILD_DIR)/compile_commands.json @printf "$(CYAN)[lint/cpp]$(RESET) Running clang-format check...\n" @# Get recently modified files (last 1 hour) for fast lint - @RECENT_FILES=$$(find $(ENGINE_DIR)/src -name '*.cpp' -o -name '*.h' -mmin -60 | grep -v build); \ + @RECENT_FILES=$$(find $(ENGINE_DIR)/src $(CPP_LINT_PRUNE) \( -name '*.cpp' -o -name '*.h' \) -mmin -60 -print); \ if [ -z "$$RECENT_FILES" ]; then \ RECENT_FILES="$(ENGINE_DIR)/src/query/query_engine.cpp $(ENGINE_DIR)/src/engine_lifecycle.cpp"; \ fi; \ @@ -358,7 +387,7 @@ fmt-rust: @printf " $(CHECK) done\n" # ─── Check (CI) ────────────────────────────────────────────────── -check: build lint test-engine test-server +check: build lint-verify test-engine test-server @printf "$(CHECK) check complete\n" # ─── Clean ─────────────────────────────────────────────────────── @@ -368,6 +397,18 @@ check: build lint test-engine test-server clean: @printf "$(CYAN)[clean]$(RESET) Cleaning build artifacts...\n" @rm -rf $(BUILD_DIR) $(ENGINE_DIR)/build-release + @# A clean that did not happen is worse than no clean at all: the next + @# build silently reuses the stale tree. `rm -rf` can be blocked (a + @# sandbox or a checkout guard may refuse bulk deletes) and the recipe + @# would still print success, so verify it and fail loudly instead — + @# that failure was observed in practice and went unnoticed because a + @# piped caller takes the pipeline's exit status, not make's. + @for d in $(BUILD_DIR) $(ENGINE_DIR)/build-release; do \ + if [ -e "$$d" ]; then \ + printf " $(CROSS) clean FAILED: $$d still exists — the delete was blocked (permissions or a sandbox guard). Remove it by hand before rebuilding.\n"; \ + exit 1; \ + fi; \ + done @cd $(SERVER_DIR) && cargo clean 2>&1 | tail -1 @rm -f $(TEST_DB) @printf " $(CHECK) cleaned\n" diff --git a/README.md b/README.md index 4f572be..fba88c8 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ It transforms source code into verifiable facts, understandable models, and inspectable evidence — enabling AI to validate claims against reality instead of hallucinating. -**Version**: v0.2.6 | **License**: Apache 2.0 +**Version**: v0.2.7 | **License**: Apache 2.0 --- @@ -16,7 +16,7 @@ CodeScope is a **Project Truth Engine** that answers one question: Not "what does this code mean", but "does the code actually do what you claim?" -It indexes source code into a structured code graph (call graph + reference graph + module knowledge), then exposes **47 MCP tools** that let AI agents locate symbols, trace call paths, verify claims, detect documentation drift, and analyze architecture — all with **~98.9% token savings** vs reading raw source files. +It indexes source code into a structured code graph (call graph + reference graph + module knowledge), then exposes **46 MCP tools** that let AI agents locate symbols, trace call paths, verify claims, detect documentation drift, and analyze architecture — all with **~98.9% token savings** vs reading raw source files. ### Supported Languages (8) @@ -31,6 +31,57 @@ It indexes source code into a structured code graph (call graph + reference grap | TypeScript | ✅ | ✅ | ✅ | | Java | ✅ | ✅ | ✅ | +**What "Verified" means**: the language has an automated test that asserts a real +semantic record (a named call, method declaration, or interface implementation) +is emitted — not merely that the response JSON contains the expected keys. Those +tests live in `engine/tests/test_ir_edge_coverage.cpp`; before they existed the +per-language E2E harness only checked for the strings `"callers"` / `"callees"` / +`"total_nodes"`, so a translator that emitted **zero** edges still passed. + +**Known limitations** (documented rather than silently dropped): + +- **Rust**: macro invocations (`println!`, `vec!`, user `foo!()`) are recorded as + calls to the macro name. tree-sitter does not expand macros, so code *generated* + by a macro body is not in the graph, and a macro is matched by name only. +- **JavaScript**: `class A extends B` is not recorded as an `InterfaceImpl` + (JavaScript has no `implements` clause); the extends expression is still visited. + TypeScript/TSX `implements` clauses **are** recorded. +- **Inference is best-effort**: receiver types come from local declarations, + composite literals, and `this`/`self`; dynamically typed receivers stay unknown. +- **Pathologically deep ASTs are truncated, not walked in full**: when a file's + AST nests deeper than `kMaxVisitDepth` (250), the deeper subtrees are skipped + and the file is reported once on stderr + (`[module=ir, method=…] AST nesting exceeded kMaxVisitDepth=250`). The cap + exists because the recursive walk runs on the indexer's 512 KB worker stacks; + hand-written code never reaches it, generated code occasionally does. +- **Some extensions are detected but not parsed**: `.kt`/`.kts`, `.rb`, `.scala` + and `.swift` are recognized (they count as candidate files) but no grammar is + vendored for them, so they are skipped and recorded in `parse_failures` as + `language_missing` — listed by `codescope parse-failures`, never silently + dropped, re-attempted on every run, and exempt from the + `CODESCOPE_FAIL_RETRY_MAX` fail-fast skip. Swift's `parser.c` is + ABI-incompatible with the vendored tree-sitter core, which is why its grammar, + visitor and builtin table are absent (see `engine/src/parser/parser.cpp`). +- **A file keeps one identity across the entry points.** The walk-based entry + points store the path exactly as passed, while the single-file ones + (`force_index_files`, the scheduler's `--file-list` retry) receive + `std::fs::canonicalize`d paths. Indexing through a symlinked path (macOS + `/tmp/x` is really `/private/tmp/x`, and the same applies to any symlinked + workspace) and then force-indexing something inside it used to store one file + under two spellings, duplicating its symbols — `find_symbol` answered with + the same symbol twice. The stored spelling is now reused both for + root-relative forms and, as a second pass, for any absolute row whose + ** canonical form equals** the incoming path's (equality, not a guess, so two + files can never be merged), and a relative spelling is recognised when its + prefix resolves to the project root from the working directory the indexer + ran in. Still uncovered, and deliberately left at the old behaviour rather + than guessed: Windows paths (the narrowing pattern uses `/`) and a relative + spelling whose index-time working directory is no longer the current one. + Rows written by an older build keep their extra spelling — re-indexing deletes + entity rows only for the spelling being written, so clear the affected + `parse_failures` rows with `codescope reset-failures` and rebuild the database + (delete `.codescope/codescope.db` and re-index) for duplicated symbols. + ### Tech Stack | Layer | Technology | @@ -39,7 +90,7 @@ It indexes source code into a structured code graph (call graph + reference grap | Indexer | C++23 (Clang 17+), SQLite (WAL mode, FTS5) | | Server | Rust 2024 Edition, MCP Protocol (JSON-RPC 2.0, stdio transport) | | Graph Storage | SQLite (sole graph store, CSR adjacency for sub-ms call-graph queries) | -| Scheduler | Built-in multi-process parallel indexer (chunk-level work-stealing) | +| Scheduler | Built-in multi-process parallel indexer (static proportional allocation by default; opt-in shared chunk queue via `CODESCOPE_CPU_DYNAMIC=1`) | | Build | CMake 3.30+ (C++), Cargo (Rust) | --- @@ -53,7 +104,7 @@ graph TB end subgraph "Rust MCP Server" - MCP["MCP Protocol (JSON-RPC 2.0)
47 tools / stdio transport"] + MCP["MCP Protocol (JSON-RPC 2.0)
46 tools / stdio transport"] DISPATCH["Tool Dispatch
project_id auto-restore"] end @@ -112,11 +163,11 @@ Inspector --------- evidence / finding ```mermaid flowchart LR Q["MCP Client
tool call"] --> Q1["Server receives
project_id auto-restore"] - Q1 --> Q2{"Tool type?"} - Q2 -->|"index_project"| Q3["Spawn worker subprocess
→ memory isolated
→ exits after done"] - Q2 -->|"query tools"| Q4["C++ FFI → SQLite query
graph_nodes, graph_edges
search_index, ..."] + Q1 --> Q2{"Tool type?"} + Q2 -->|"session auto-index"| Q3["At initialize: spawn worker subprocess
→ memory isolated
→ exits after done
(not a callable tool)"] + Q2 -->|"query tools"| Q4["C++ FFI → SQLite query
entity, relation
code_fts, ..."] Q2 -->|"get_communities"| Q5["Load full graph
Label Propagation
→ JSON with max_communities limit"] - Q2 -->|"get_hotspots"| Q6["SQL: COUNT(ge.id) JOIN
graph_edges edge_type=1
ORDER BY caller_count"] + Q2 -->|"get_hotspots"| Q6["SQL: COUNT(r.id) JOIN
relation edge_type=1
ORDER BY caller_count"] Q4 --> R["Result JSON
back to MCP Client"] Q5 --> R Q6 --> R @@ -126,24 +177,37 @@ flowchart LR ```mermaid flowchart LR - subgraph A["Phase A: Fast Scan (ms)"] + subgraph A["Phase A: Index (scan + full tree-sitter parse)"] S1["scan_project"] S2["total_symbols"] S3["module_tree"] S4["entry_points"] + S5["full tree-sitter parse → IR records"] end - subgraph B["Phase B: Background Enhance (async, seconds)"] + subgraph B["Phase B: Graph Finalize (async, seconds)"] E1["enhance_project"] - E2["full tree-sitter"] - E3["call graph"] + E2["buildGraph — NO re-parse"] + E3["call graph / CSR"] E4["FTS index"] E5["semantic_fact (v0.3)"] end - A -->|"trigger"| B + A -->|"trigger (CLI index-parallel path only)"| B ``` +**Where the parsing happens**: the full tree-sitter parse runs in **Phase A**, not +Phase B. Phase B (`enhance_project`) is a lightweight *GraphFinalize* step — +`buildGraph` → `buildFTSFromGraph` → `resolveStagedMetrics` → semantic_fact +extraction → model build — and never re-parses or re-translates +(`engine_queries.cpp`). + +**When Phase B runs automatically**: only the CLI path (`codescope index-parallel`) +triggers it after the merge (`server/src/main.rs`). The MCP session auto-index +(§5, `index_project`) triggers the FTS build only, so call `enhance_project` +explicitly if you need a full graph finalization inside an MCP session. + + --- ## 3. 8-Layer Smart Filtering: Why Only 6,029 of 36,919 Files Are Indexed @@ -170,15 +234,26 @@ Without filtering, CodeScope would waste 84% of its time indexing tests, vendore ### The 8-Layer Filter Cascade ``` -Layer 1: any-depth skip directories (~120 patterns) +Layer 1: any-depth skip directories (~150 patterns) .git, .svn, .hg, node_modules, .venv, target, build, dist, - vendor, __pycache__, .github, deploy, docker, k8s, ... - → Catches VCS, build artifacts, dependencies, CI/CD, infra at ANY depth - -Layer 2: top-only skip directories (depth ≤ 3, Java-safe) - test, tests, docs, examples, samples, scripts, e2e, integration, - assets, static, public, media, i18n, bench, benchmarks, ... - → For Java: protects package namespaces (org/.../samples/petclinic) + vendor, vendored, third_party, bin, bench, benchmarks, + test, tests, e2e, docs, examples, samples, + __pycache__, .github, deploy, docker, k8s, ... + → Catches VCS, build artifacts, dependencies, vendored code and the + test/docs/sample conventions at ANY depth + +Layer 2: top-only skip directories (depth ≤ 3, every language) + scripts, hack, migrations, seeds, integration, + locale, locales, i18n, l10n, assets, static, public, media, external + → These are ordinary business directories outside the web convention + (src/integration/, pkg/scripts/, db/migrations/, app/static/ all + hold first-party code), so skipping them at any depth silently + dropped real symbols — an undetectable false negative. They are + still skipped at the project root and the next two levels, where + the web-frontend convention they target actually lives. The + non-source formats filling those dirs are rejected by Layer 3. + → For Java: the Layer-1 names (test, docs, samples, ...) are also + top-only, protecting package namespaces (org/.../samples/petclinic) Layer 3: file suffix skip (always applied) .md, .txt, .json, .yaml, .toml, .ini, .png, .jpg, .svg, @@ -190,8 +265,8 @@ Layer 4: exact filename skip .env, .env.local, .gitkeep, .gitignore, ... Layer 5: filename/directory prefix skip - File prefixes: ._*, ~$*, #*# - Directory prefixes: build_*, test_*, tmp_* + File prefixes: .env.*, docker-compose.* (plus trailing-`~` editor backups) + Directory prefixes: build_*, cmake-build-*, _build, tools-*, tools_* Layer 6: .gitignore pattern matching Respects every rule in the project's .gitignore @@ -200,10 +275,18 @@ Layer 7: .codescopeignore (user-defined) Additional custom ignore patterns per project Layer 8: file size limit + language detection - • Max file size (default 10 MB, configurable via CODESCOPE_MAX_FILE_SIZE) + • Max file size (default 5 MB = 5242880 bytes, configurable via CODESCOPE_MAX_FILE_SIZE) • Undetectable language files are silently skipped ``` +> **Layer numbers are a classification, not the execution order.** The skip +> verdict is a boolean OR, so only the *result* is observable, but the checks +> actually run in this order (`FilterPolicy::shouldSkipEntry`): path components +> (Layer 1 / 2) **plus Layer 6 `.gitignore` and Layer 7 `.codescopeignore`** → +> bundle suffixes → Layer 4 exact filename + Layer 5 prefixes → Layer 3 suffix → +> the STRICT gate → the user's `CODESCOPE_EXCLUDE_PATHS` patterns last, as an +> override on top of everything else. + ### Real-World Impact | Project | Raw Source Files | After Filtering | Filtered Out | Time Saved | @@ -229,10 +312,13 @@ codescope cli force_index_files '{"paths":["/path/to/test/file.rs"]}' |----------|-------------| | **macOS** | Xcode CLT, cmake, Rust (1.85+) | | **Linux** | build-essential, cmake, Rust (1.85+) | -| **Windows** ⚠️ **Beta** | MinGW-w64 14.0.0+, Rust `x86_64-pc-windows-gnu` target, cmake. Every graph-query tool (shortest_path, get_neighbors, get_callers/callees, graph_query, subgraph, entry_points, trace_path, hotspots, impact_analysis, ...) works via the built-in SQLite graph-query backend (CSR adjacency, sub-millisecond call-graph lookups). | +| **Windows** ⚠️ **Beta** | MinGW-w64 14.0.0+, Rust `x86_64-pc-windows-gnu` target, cmake, and the **`sqlite3` CLI on `PATH`** (the parallel and chunked schedulers merge their worker databases by piping SQL into it). Every graph-query tool (shortest_path, get_neighbors, get_callers/callees, graph_query, subgraph, entry_points, trace_path, hotspots, impact_analysis, ...) works via the built-in SQLite graph-query backend (CSR adjacency, sub-millisecond call-graph lookups). Parallel indexing (`index-parallel`, including the chunk-level scheduler) now works on Windows as well — the shared state both schedulers map has a Windows backend (`CreateFileMappingW`/`MapViewOfFile`) — but stays **Beta**: the `windows-smoke` CI job is its acceptance gate and it is dispatch-only. If the `sqlite3` CLI is missing, use `codescope index` (single-process), which does not merge. | ### Install Pre-built Binary +> New here? [`docs/en/QUICK_START.md`](docs/en/QUICK_START.md) is the 5-minute path — install, index a +> project, connect an MCP client, run the first queries. + ```bash curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/install.sh | bash export PATH="$PATH:$HOME/.codescope/bin" @@ -262,8 +348,8 @@ cargo build --release ### Index and Query ```bash -# Index a project -codescope cli index_project '{"project_path":"/path/to/your/project"}' +# Index a project (spawns an isolated worker subprocess per module) +codescope index-parallel /path/to/your/project # Quick overview codescope cli project_overview '{}' @@ -274,21 +360,59 @@ codescope ### Large Projects -For projects with thousands of files, use the built-in parallel scheduler: +For projects with thousands of files, tune the built-in parallel scheduler: + +```bash +codescope index-parallel /path/to/large/project --workers 8 --parallel 4 +``` + +`--workers` sets the total parse-worker cores (default 8) and `--parallel` the +maximum concurrent module workers (default 4). + +### Maintenance ```bash -codescope index-parallel /path/to/large/project +# Files the indexer could not parse, with the reason and the retry count +codescope parse-failures --limit 50 + +# Clear them, so those files are attempted again on the next run +codescope reset-failures ``` +Both accept `--db ` (or `CODESCOPE_DB_PATH`) and operate on the most +recent project in that database. A file whose only failure is a missing grammar +(e.g. `.swift`) is retried on every run anyway and needs no reset. + --- -## 5. MCP Tools (47 Tools) +## 5. MCP Tools (46 Tools) + +The 46 tools listed here are exactly the ones advertised by `tools/list` +(`server/src/tools/catalog.rs`) and dispatched by name (`TOOL_HANDLERS` in +`server/src/tools/mod.rs`); the two sets are identical — there are no +advertised-but-unhandled or handled-but-unadvertised tools. + +> **Responses are capped at 1 MiB by the stdio transport.** A tool result +> larger than that is replaced by a JSON-RPC error (`-32000`) naming the size +> and suggesting a narrower query or smaller limits — it is never truncated +> silently. Bound large result sets with the tools' own arguments +> (`LIMIT` in a `graph_query` DSL string, `limit`/`node_limit`/`edge_limit`, +> `max_findings`, …). ### Indexing +> **`index_project` is session-only and NOT callable by name.** It is absent +> from `tools/list` and from the tool dispatcher, so +> `tools/call {"name":"index_project"}` returns +> `{"error":"Unknown tool: index_project …"}`. A worker-subprocess index runs +> automatically once per MCP session at `initialize` +> (`server/src/mcp/server.rs`). The CLI/agent equivalent is +> `codescope index-parallel `; `codescope cli index_project` is **not** a +> valid invocation. Call `enhance_project` explicitly if you need a full graph +> finalization inside a session (see §2). + | Tool | Description | Parameters | |------|-------------|------------| -| `index_project` | Index a project directory: parse all source files, build IR, and construct the code graph. | `{"project_path": "string (required)", "language_filter": "string (optional)"}` | | `index_file` | Index a single source file. | `{"file_path": "string (required)"}` | | `force_index_files` | Force-index files/dirs bypassing default skip rules (test/, docs/, node_modules/, .gitignore, etc.). | `{"paths": ["string (required)"], "language_filter": "string (optional)"}` | @@ -321,19 +445,21 @@ codescope index-parallel /path/to/large/project | `find_callers_by_entity` | Find callers of a symbol by its graph entity id. | `{"entity_id": "integer (required)"}` | | `find_callees_by_entity` | Find callees of a symbol by its graph entity id. | `{"entity_id": "integer (required)"}` | | `get_verifier_registry_status` | Inspect the registered verifiers and their health (supported claim types, unsupported list, backend readiness). | `{}` | -| `codescope_trace` | Interactive recursive call exploration (depth + direction) or shortest path. | `{"function_name": "string", "depth": "integer (default 1, max 5)", "direction": "callers|callees|both", "from": "string", "to": "string"}` | +| `codescope_trace` | Interactive recursive call exploration (depth + direction) or shortest path. | `{"function_name": "string (required for exploration mode)", "depth": "integer (default 1, max 5)", "direction": "callers\|callees\|both", "from": "string (shortest-path mode: required with to)", "to": "string (shortest-path mode: required with from)"}` | | `trace_flow` | Recursive execution flow tracing (caller→callee chain). | `{"function_name": "string (required)", "depth": "integer (default 3, max 10)"}` | -| `shortest_path` | Shortest call path between two functions (BFS). | `{"from": "string", "to": "string", "from_id": "integer", "to_id": "integer"}` | +| `shortest_path` | Shortest call path between two functions (BFS). | `{"from": "string (required unless from_id)", "to": "string (required unless to_id)", "from_id": "integer (required unless from)", "to_id": "integer (required unless to)"}` | | `connected_components` | Connected components in the call graph. | `{}` | +| `get_communities` | **New** — detect communities (clusters) in the call graph via deterministic label propagation. Returns `{communities:[{id,label,member_count}], total_communities, returned_communities, inter_community_edges, truncated}`; members are omitted unless `include_members` is true. | `{"max_communities": "integer (default 20, max 500)", "include_members": "boolean (default false)", "max_members": "integer (default 10, max 200)"}` | ### Graph Query | Tool | Description | Parameters | |------|-------------|------------| -| `graph_query` | Cypher-like DSL query: `MATCH (Function:main)-[Calls]->(Method)`. | `{"dsl": "string (required)"}` | +| `graph_query` | Cypher-like DSL query: `MATCH (Function:main)-[Calls]->(Method)`, plus optional trailing clauses: `LIMIT ` (**honoured**; the response then also carries `truncated: true`) and `RETURN ` (accepted, no effect today). Any other trailing text is an error. Use `LIMIT` on large graphs — a broad pattern can exceed the 1 MiB transport cap (see the note above the table). | `{"dsl": "string (required)"}` | | `get_graph` | Retrieve the complete code graph in paginated pages. | `{"node_offset": "integer", "node_limit": "integer (max 50000)", "edge_offset": "integer", "edge_limit": "integer (max 200000)", "node_types": "string", "edge_types": "string"}` | | `get_subgraph` | Fetch a local region centered on a node (1-hop). | `{"node_id": "integer (required)", "radius": "integer", "node_types": "string", "edge_types": "string"}` | | `get_neighbors` | Fetch direct neighbors (callers + callees) of a graph node. | `{"node_id": "integer (required)", "edge_type": "integer (default -1)", "radius": "integer"}` | +| `get_knowledge_graph` | Direct-query a knowledge-layer table. Tables: `entity`, `relation`, `architecture_edge`, `module_edge`, `capability`, `document`, `module_summary` (see §6). | `{"table": "string (required)", "limit": "integer (default 100, max 1000)"}` | ### Search @@ -346,8 +472,8 @@ codescope index-parallel /path/to/large/project | Tool | Description | Parameters | |------|-------------|------------| -| `verify_integrity` | Check README-promised features actually exist in code. | `{}` | -| `verify_claim` | Verify a single claim (capability_exists / contract_holds / architecture_follows). | `{"claim": "string (required)"}` | +| `verify_integrity` | Check README-promised features actually exist in code. | `{"max_findings": "integer (optional, default 200, max 2000)"}` | +| `verify_claim` | Verify a single claim. `claim` is a **JSON object string**, not free text: `{"type": "capability_exists \| contract_holds \| architecture_follows \| function_implements", "subject": "symbol/module/contract", "predicate": "optional detail"}`. An unknown or missing `type` is rejected with `error_code: claim_type_unsupported` instead of being guessed. | `{"claim": "string (required, JSON object)"}` | | `verify_summary` | Parse natural-language summary and verify each claim. | `{"text": "string (required)"}` | | `verify_review` | Verify code review comment claims. | `{"text": "string (required)"}` | | `verify_reality` | Verify a single AI statement against code evidence. | `{"text": "string (required)"}` | @@ -358,9 +484,8 @@ The v0.3 Evidence Pipeline transforms indexed code into verifiable evidence and | Tool | Description | Parameters | |------|-------------|------------| -| `enhance_project` | Run background enhancement: full tree-sitter parse, call graph, FTS, and v0.3 semantic_fact extraction. Prerequisite for `build_evidence` to produce non-empty findings. Complexity metrics and n-gram semantic vectors are built during `index_project` (restored in v0.2.5). | `{}` | +| `enhance_project` | Run background graph finalization: `buildGraph` (call graph + CSR), FTS, staged metrics, model build, and v0.3 semantic_fact extraction. It does **not** re-parse — the full tree-sitter parse happens during indexing (Phase A, §2). Prerequisite for `build_evidence` to produce non-empty findings. Complexity metrics and n-gram semantic vectors are built during `index_project` (restored in v0.2.5); n-gram vectors additionally require `CODESCOPE_INDEX_MODE=deep`. Returns `ok:false` with `failed_step` when a finalization step fails. | `{}` | | `build_evidence` | Build evidence findings by applying the rule set (sync/memory/error/pattern/framework/ffi) to the project's semantic_fact rows. Each rule declares fact needs + a combine mode (Collect / MissingMatch / MissingMatchPerFunction / Count). Returns a JSON array of Evidence objects. Run after `enhance_project` so semantic facts are populated. | `{"category": "string (optional, one of: sync|memory|error|pattern|framework|ffi)"}` | -| `verify_statement` | Verify a natural-language claim against the project's indexed evidence. Pipeline: IntentParser → Planner → EvidenceBuilder → VerdictBuilder. Returns JSON with `verdict` (Supported\|Contradicted\|PartiallyVerified\|Unknown), `confidence`, `requirements[]`, and `evidence[]`. Use this for yes/no questions about code behavior (e.g. "does this project safely handle CString?"). | `{"claim": "string (required)"}` | | `build_project_state` | Build (or rebuild) and persist the project state snapshot. Runs the full v0.3 Evidence Pipeline (evidence aggregation + state queries) and UPSERTs the result into the `project_state` table. Returns the snapshot JSON: overall confidence, capability/architecture/workflow/dead_code scores, per-category issue counts, and `last_updated` timestamp. | `{}` | | `get_project_state` | Get the previously persisted project state snapshot (without rebuilding). Returns the `snapshot_json` string for the project, or a JSON error object if no snapshot exists yet (run `build_project_state` first). Use this for fast reads of the latest health snapshot. | `{}` | @@ -400,7 +525,6 @@ HTTP routes → get_routes Type info → get_type_info Verify claim → verify_claim Enhance project → enhance_project -Verify statement → verify_statement Build evidence → build_evidence Project health → build_project_state Detect drift → detect_documentation_drift @@ -438,6 +562,12 @@ Supported tables: `entity`, `relation`, `architecture_edge`, `module_edge`, `cap All benchmarks measured on **Apple M3 Max (36 GB RAM, 14 cores), macOS, 2026-08-14**, using `target/release/codescope` (v0.2.6) in `worker` (serial full index) mode with the pure-SQLite graph backend (no LadybugDB/Kuzu dependency), `CODESCOPE_INDEX_MODE=normal`. Query latency measured via MCP server mode (engine initialized once, median of 7 runs). Numbers reflect the in-memory fuzzy resolver + ordering fix (edge count 1,249 → 1,189 vs. the pre-fix binary; nodes/files unchanged). Other hardware will produce different results — expect slower performance on less capable machines. +> **Indicative, single-run figures.** These tables are one measurement on one +> machine, not a regression gate: `benchmarks/run_benchmark.sh` re-runs the +> **index-time** rows, but the latencies and the token-savings table below are +> recorded by hand and are not reproduced by any script. Treat them as +> order-of-magnitude indications. + ### Index Time | Project | Language | Files | Nodes | Edges | Index Time | Peak RSS | @@ -546,21 +676,37 @@ Each script calls `codescope cli ''` internally. See `ski | Variable | Default | Description | |----------|---------|-------------| | `CODESCOPE_DB_PATH` | `.codescope/codescope.db` | SQLite database path | -| `CODESCOPE_INDEX_MODE` | `normal` | Index mode: `fast` / `normal` / `strict` | -| `CODESCOPE_EXCLUDE_PATHS` | (unset) | Comma-separated glob patterns to exclude | -| `CODESCOPE_WORKERS` | `min(hw,8)` | Total parse-worker cores for `index-parallel` (`kDefaultParseWorkers=8`) | +| `CODESCOPE_INDEX_MODE` | `normal` | Parse-pipeline mode: `fast` / `normal` / `deep`. `strict` is a separate discovery-only mode — see the notes below. | +| `CODESCOPE_EXCLUDE_PATHS` | (unset) | Comma-separated glob patterns to exclude, matched against the project-relative path and applied **last** (they override the built-in filters). Escape a literal comma inside a pattern as `\,` (e.g. `a\,b/**`); `\\` is a literal backslash. | +| `CODESCOPE_WORKERS` | `min(hw,8)` | Total parse-worker cores (`kDefaultParseWorkers=8`). The in-memory path used for projects of ≤2000 files defaults to **4** instead. | | `CODESCOPE_WORKER_TIMEOUT` | `300` | Worker subprocess timeout in seconds | -| `CODESCOPE_MAX_FILE_SIZE` | (unset) | Max source file size to index in bytes | +| `CODESCOPE_MAX_FILE_SIZE` | `5242880` (5 MB) | Max source file size to index in bytes. Files above it are skipped silently. | +| `CODESCOPE_FAIL_RETRY_MAX` | `1` | Parse failures before a file is skipped entirely on the automatic index paths (min 1); see `parse_failures` / `codescope reset-failures`. Files whose only failure is a missing grammar (`language_missing`, e.g. `.swift`) never count and are retried on every run. `force_index_files` applies **no** such skip: a forced file is always re-attempted. | | `CODESCOPE_MMAP_SIZE` | 256 MB | SQLite `mmap_size` pragma value | | `CODESCOPE_MEM_LIMIT_MB` | `4096` | Dynamic-scheduler memory ceiling | -| `CODESCOPE_DYNAMIC_SCHED` | `auto` | Dynamic CPU scheduling: `1` on, `0` off, unset = auto | -| `CODESCOPE_VERBOSE` | `0` | Set to `1` for verbose logging | +| `CODESCOPE_DYNAMIC_SCHED` | (unset = static) | Opt-in dynamic CPU scheduling. `CODESCOPE_CPU_DYNAMIC` is the canonical name and this is the legacy alias; either one works. `1`/`true`/`on` enables the shared chunk-queue scheduler, `0`/`false`/`off` disables it, unset stays on the **static** proportional allocator. | | `CODESCOPE_LSP` | (unset) | LSP server command for type enhancement | +> **`strict` and `deep`**: `CODESCOPE_INDEX_MODE=strict` only tightens file +> *discovery* (a whitelist gate on detected languages); it does not change the +> parse pipeline. `deep` additionally builds the n-gram semantic vectors that +> `normal` skips. +> +> **`index-parallel` always parses in `fast` mode.** Its per-module and chunk +> workers are launched with `CODESCOPE_INDEX_MODE=fast` hard-coded +> (`server/src/scheduler/worker.rs`, `quarantine.rs`), because the merged DB gets +> its graph pass afterwards. Setting `normal`/`strict`/`deep` therefore has no +> effect on `codescope index-parallel`; use the single-process +> `codescope index` path when you need a non-`fast` pipeline. +> +> **Dynamic scheduling is opt-in, never automatic.** The production dispatcher +> checks the explicit flag only; it does not auto-enable the chunk scheduler for +> large projects (this is deliberate — see `plan/next/DYNAMIC_SCHED_REDESIGN.md`). + --- ## 10. License Apache 2.0 — see [LICENSE](LICENSE). -**CodeScope v0.2.6** — Built with Rust 2024 + C++23 + tree-sitter + SQLite. +**CodeScope v0.2.7** — Built with Rust 2024 + C++23 + tree-sitter + SQLite. diff --git a/README.zh.md b/README.zh.md index a79756d..5248855 100644 --- a/README.zh.md +++ b/README.zh.md @@ -4,7 +4,7 @@ 它将源代码转化为可验证的事实、可理解的模型和可检查的证据 — 让 AI 能够根据现实验证断言，而非凭空编造。 -**版本**: v0.2.6 | **许可证**: Apache 2.0 +**版本**: v0.2.7 | **许可证**: Apache 2.0 --- @@ -16,7 +16,7 @@ CodeScope 是一个 **项目真相引擎（Project Truth Engine）**，回答一 不是"这段代码什么意思"，而是"代码到底有没有实现你声称的功能？" -它把源码索引为结构化的代码图（调用图 + 引用图 + 模块知识），然后暴露 **37 个 MCP 工具**，让 AI 代理可以定位符号、追踪调用路径、验证断言、检测文档漂移、分析架构 — 相比读取原始源文件，平均节省 **~98.9% 的 token 消耗**。 +它把源码索引为结构化的代码图（调用图 + 引用图 + 模块知识），然后暴露 **46 个 MCP 工具**，让 AI 代理可以定位符号、追踪调用路径、验证断言、检测文档漂移、分析架构 — 相比读取原始源文件，平均节省 **~98.9% 的 token 消耗**。 ### 支持的语言（8 种） @@ -31,6 +31,18 @@ CodeScope 是一个 **项目真相引擎（Project Truth Engine）**，回答一 | TypeScript | ✅ | ✅ | ✅ | | Java | ✅ | ✅ | ✅ | +**「已验证」的含义**：该语言有一份自动化测试**断言真实产生了语义记录**（具名调用、方法声明或接口实现），而不只是断言响应 JSON 里出现了预期字段。这些用例位于 `engine/tests/test_ir_edge_coverage.cpp`。此前各语言的 E2E 只检查 JSON 里是否存在 `"callers"` / `"callees"` / `"total_nodes"` 字符串，因此一个**一条边都不发**的转换器也能通过整个测试套件。 + +**已知限制**（如实记录而非静默丢弃）： + +- **Rust**：宏调用（`println!`、`vec!`、用户 `foo!()`）会按宏名记录为调用。tree-sitter 不做宏展开，因此**宏体生成**的代码不在图中，宏也只按名字匹配。 +- **JavaScript**：`class A extends B` 不会记录为 `InterfaceImpl`（JS 没有 `implements` 子句）；extends 的表达式仍会被访问。TypeScript/TSX 的 `implements` 子句**会**记录。 +- **推断是尽力而为**：接收者类型来自局部声明、复合字面量与 `this`/`self`；动态类型的接收者保持未知。 +- **病态深嵌套 AST 会被截断，而不是完整遍历**：当文件的 AST 嵌套超过 `kMaxVisitDepth`（250）时，更深的子树会被跳过，并在 stderr 上按文件报告一次 + （`[module=ir, method=…] AST nesting exceeded kMaxVisitDepth=250`）。该上限存在的原因是递归遍历运行在索引器 512 KB 的 worker 栈上；手写代码不会触及，生成代码偶尔会。 +- **部分扩展名可识别但不解析**：`.kt`/`.kts`、`.rb`、`.scala`、`.swift` 会被识别（计入候选文件），但没有内置语法，因此被跳过并在 `parse_failures` 中记录为 `language_missing` —— 可用 `codescope parse-failures` 查看，绝不静默丢弃，每次运行都会重试，且不计入 `CODESCOPE_FAIL_RETRY_MAX` 的 fail-fast 跳过。Swift 的 `parser.c` 与内置 tree-sitter core 的 ABI 不兼容，因此其语法、visitor 与 builtin 表均不存在（见 `engine/src/parser/parser.cpp`）。 +- **同一文件在各入口之间保持同一身份。** 目录遍历类入口按传入的写法记录路径，而单文件入口（`force_index_files`、调度器的 `--file-list` 重试）拿到的是 `std::fs::canonicalize` 之后的路径。通过符号链接路径索引（macOS 上 `/tmp/x` 实为 `/private/tmp/x`，任何被软链的工作区同理）之后再对其中的文件做 force-index，过去会把同一个文件存成两种写法并**复制其符号** —— `find_symbol` 会把同一个符号返回两次。现在已存写法会被复用：既覆盖「相对于项目根」的各种写法，也在第二遍里接受任何**规范化形式与被查路径相等**的绝对路径行（相等判定，不是猜测，因此绝不会把两个文件合成一个身份）；相对写法只要其前缀能从索引时的当前目录解析到项目根，也会被识别。仍未覆盖、且按既定取向保留旧行为（宁可保持旧行为也不猜）的情形：Windows 路径（缩小范围的模式用 `/`），以及索引时的工作目录已不再是当前目录的相对写法。旧版本写入的行仍会保留多出来的写法 —— 重新索引只按「正在写入的那个写法」删除 entity 行，因此重复的失败行用 `codescope reset-failures` 清掉，重复的符号需要重建数据库（删除 `.codescope/codescope.db` 后重新索引）。 + ### 技术栈 | 层 | 技术 | @@ -39,7 +51,7 @@ CodeScope 是一个 **项目真相引擎（Project Truth Engine）**，回答一 | 索引引擎 | C++23（Clang 17+），SQLite（WAL 模式，FTS5） | | 服务端 | Rust 2024 Edition，MCP 协议（JSON-RPC 2.0，stdio 传输） | | 图存储 | SQLite（唯一图存储，CSR 邻接表实现亚毫秒级调用图查询） | -| 调度器 | 内置多进程并行索引器（chunk 级 work-stealing） | +| 调度器 | 内置多进程并行索引器（默认静态比例分配；`CODESCOPE_CPU_DYNAMIC=1` 可启用共享 chunk 队列） | | 构建 | CMake 3.30+（C++），Cargo（Rust） | --- @@ -53,7 +65,7 @@ graph TB end subgraph "Rust MCP 服务端" - MCP["MCP 协议 (JSON-RPC 2.0)
37 个工具 / stdio 传输"] + MCP["MCP 协议 (JSON-RPC 2.0)
46 个工具 / stdio 传输"] DISPATCH["工具分发
project_id 自动恢复"] end @@ -113,10 +125,10 @@ Inspector --------- evidence / finding flowchart LR Q["MCP 客户端
工具调用"] --> Q1["服务端接收
project_id 自动恢复"] Q1 --> Q2{"工具类型?"} - Q2 -->|"index_project"| Q3["启动 worker 子进程
→ 内存隔离
→ 完成后退出"] - Q2 -->|"查询工具"| Q4["C++ FFI → SQLite 查询
graph_nodes, graph_edges
search_index, ..."] + Q2 -->|"会话自动索引"| Q3["initialize 时：启动 worker 子进程
→ 内存隔离
→ 完成后退出
（不是可按名调用的工具）"] + Q2 -->|"查询工具"| Q4["C++ FFI → SQLite 查询
entity, relation
code_fts, ..."] Q2 -->|"get_communities"| Q5["加载完整图
标签传播
→ JSON (max_communities 限制)"] - Q2 -->|"get_hotspots"| Q6["SQL: COUNT(ge.id) JOIN
graph_edges edge_type=1
ORDER BY caller_count"] + Q2 -->|"get_hotspots"| Q6["SQL: COUNT(r.id) JOIN
relation edge_type=1
ORDER BY caller_count"] Q4 --> R["结果 JSON
返回 MCP 客户端"] Q5 --> R Q6 --> R @@ -126,24 +138,30 @@ flowchart LR ```mermaid flowchart LR - subgraph A["阶段 A: 快速扫描 (毫秒级)"] + subgraph A["阶段 A: 索引（扫描 + 完整 tree-sitter 解析）"] S1["scan_project"] S2["total_symbols"] S3["module_tree"] S4["entry_points"] + S5["完整 tree-sitter 解析 → IR 记录"] end - subgraph B["阶段 B: 后台增强 (异步, 秒级)"] + subgraph B["阶段 B: 图最终化 (异步, 秒级)"] E1["enhance_project"] - E2["完整 tree-sitter"] - E3["调用图"] + E2["buildGraph —— 不重新解析"] + E3["调用图 / CSR"] E4["FTS 索引"] E5["semantic_fact (v0.3)"] end - A -->|"触发"| B + A -->|"触发（仅 CLI index-parallel 路径）"| B ``` +**解析发生在哪里**：完整的 tree-sitter 解析在**阶段 A**完成，不在阶段 B。阶段 B（`enhance_project`）是一个轻量的 *GraphFinalize* 步骤 —— `buildGraph` → `buildFTSFromGraph` → `resolveStagedMetrics` → semantic_fact 抽取 → 模型构建，**从不重新解析或重新转换**（`engine_queries.cpp`）。 + +**阶段 B 何时自动运行**：只有 CLI 路径（`codescope index-parallel`）在合并后触发它（`server/src/main.rs`）。MCP 会话自动索引（§5 的 `index_project`）只触发 FTS 构建；如果你需要在 MCP 会话内完成整图最终化，请显式调用 `enhance_project`。 + + --- ## 3. 8 层智能过滤：为什么 36,919 个文件只索引了 6,029 个 @@ -170,7 +188,7 @@ CodeScope **不会**索引项目中的每一个文件。它通过 **8 层级联 ### 8 层过滤级联 ``` -第 1 层：任意深度跳过目录（约 120 个模式） +第 1 层：任意深度跳过目录（约 150 个模式） .git, .svn, .hg, node_modules, .venv, target, build, dist, vendor, __pycache__, .github, deploy, docker, k8s, ... → 在任何深度捕获 VCS、构建产物、依赖、CI/CD、基础设施 @@ -190,8 +208,8 @@ CodeScope **不会**索引项目中的每一个文件。它通过 **8 层级联 .env, .env.local, .gitkeep, .gitignore, ... 第 5 层：文件名/目录前缀跳过 - 文件前缀：._*, ~$*, #*# - 目录前缀：build_*, test_*, tmp_* + 文件前缀：.env.*, docker-compose.*（以及以 `~` 结尾的编辑器备份文件） + 目录前缀：build_*, cmake-build-*, _build, tools-*, tools_* 第 6 层：.gitignore 模式匹配 尊重项目 .gitignore 中的每一条规则 @@ -200,10 +218,12 @@ CodeScope **不会**索引项目中的每一个文件。它通过 **8 层级联 每个项目可额外添加自定义忽略模式 第 8 层：文件大小限制 + 语言检测 - • 最大文件大小（默认 10 MB，可通过 CODESCOPE_MAX_FILE_SIZE 配置） + • 最大文件大小（默认 5 MB = 5242880 字节，可通过 CODESCOPE_MAX_FILE_SIZE 配置） • 无法检测语言的文件静默跳过 ``` +> **层号是分类，不是执行顺序。** 跳过判定是布尔或，因此只有**结果**可观测，但检查的实际执行顺序是（`FilterPolicy::shouldSkipEntry`）：路径分量（第 1/2 层）**加上第 6 层 `.gitignore` 与第 7 层 `.codescopeignore`** → bundle 后缀 → 第 4 层精确文件名 + 第 5 层前缀 → 第 3 层后缀 → STRICT 门 → 用户 `CODESCOPE_EXCLUDE_PATHS` 最后作为覆盖。 + ### 实际效果 | 项目 | 原始文件数 | 过滤后 | 过滤比例 | 节省时间 | @@ -229,10 +249,13 @@ codescope cli force_index_files '{"paths":["/path/to/test/file.rs"]}' |------|------| | **macOS** | Xcode CLT, cmake, Rust (1.85+) | | **Linux** | build-essential, cmake, Rust (1.85+) | -| **Windows** ⚠️ **Beta** | MinGW-w64 14.0.0+，Rust `x86_64-pc-windows-gnu` 目标，cmake。所有图查询工具均通过内置 SQLite 图查询后端（CSR 邻接表）工作。| +| **Windows** ⚠️ **Beta** | MinGW-w64 14.0.0+，Rust `x86_64-pc-windows-gnu` 目标，cmake，以及 **`PATH` 上的 `sqlite3` CLI**（并行与分块调度器通过把 SQL 管道给它来合并各 worker 的数据库）。所有图查询工具（shortest_path、get_neighbors、get_callers/callees、graph_query、subgraph、entry_points、trace_path、hotspots、impact_analysis 等）均通过内置 SQLite 图查询后端（CSR 邻接表，亚毫秒级调用图查询）工作。并行索引（`index-parallel`，含分块级调度器）现在也能在 Windows 上工作 —— 两个调度器映射的那块共享状态已有 Windows 后端（`CreateFileMappingW`/`MapViewOfFile`）—— 但仍**为 Beta**：`windows-smoke` CI 作业是其验收门禁，且仅支持手动触发。若缺少 `sqlite3` CLI，请改用 `codescope index`（单进程），它不做合并。 | ### 安装预编译二进制 +> 初次使用？[`docs/zh/QUICK_START.md`](docs/zh/QUICK_START.md) 是 5 分钟上手路径 —— 安装、索引一个项目、 +> 连接 MCP 客户端、跑通第一批查询。 + ```bash curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/install.sh | bash export PATH="$PATH:$HOME/.codescope/bin" @@ -262,8 +285,8 @@ cargo build --release ### 索引与查询 ```bash -# 索引一个项目 -codescope cli index_project '{"project_path":"/path/to/your/project"}' +# 索引一个项目（按模块派生子进程，内存相互隔离） +codescope index-parallel /path/to/your/project # 快速概览 codescope cli project_overview '{}' @@ -274,21 +297,45 @@ codescope ### 大型项目 -对于包含数千个文件的项目，使用内置的并行调度器： +对于包含数千个文件的项目，可以调整内置并行调度器的规模： + +```bash +codescope index-parallel /path/to/large/project --workers 8 --parallel 4 +``` + +`--workers` 为解析 worker 核心总数（默认 8），`--parallel` 为并发模块 worker 上限（默认 4）。 + +### 维护 ```bash -codescope index-parallel /path/to/large/project +# 列出解析器无法解析的文件（含原因与重试次数） +codescope parse-failures --limit 50 + +# 清空它们，使这些文件在下次运行中被重新尝试 +codescope reset-failures ``` +两者都接受 `--db `（或环境变量 `CODESCOPE_DB_PATH`），作用于该数据库中最近的项目。仅因缺少语法而失败的文件（如 `.swift`）本来每次运行都会重试，无需 reset。 + --- -## 5. MCP 工具（37 个工具） +## 5. MCP 工具（46 个工具） + +这里列出的 46 个工具正是 `tools/list` 广播的（`server/src/tools/catalog.rs`）与按名分发的（`server/src/tools/mod.rs` 的 `TOOL_HANDLERS`）—— 两个集合完全一致，不存在「广播了但没有处理器」或「有处理器但没广播」的工具。 + +> **stdio 传输对单条响应有 1 MiB 上限。** 超过上限的工具结果会被替换为一条 JSON-RPC 错误（`-32000`），其中给出实际字节数并建议缩小查询范围或调小 limit —— **绝不静默截断**。大结果请用各工具自带的参数收窄（`graph_query` 的 DSL 里写 `LIMIT`，以及 `limit`/`node_limit`/`edge_limit`/`max_findings` 等）。 ### 索引 +> **`index_project` 仅限会话内自动运行，不可按名调用。** 它既不在 `tools/list` 中，也不在工具分发表里，因此 +> `tools/call {"name":"index_project"}` 会返回 +> `{"error":"Unknown tool: index_project …"}`。每个 MCP 会话在 `initialize` 时自动运行一次 worker 子进程索引 +> （`server/src/mcp/server.rs`）。命令行/Agent 等价入口是 `codescope index-parallel `； +> `codescope cli index_project` **不是**有效调用。若需要在会话内完成整图最终化，请显式调用 +> `enhance_project`（见 §2）。 + | 工具 | 用途 | 参数 | |------|------|------| -| `index_project` | 索引整个项目目录：解析所有源文件，构建 IR，构建代码图。 | `{"project_path": "string (必填)", "language_filter": "string (可选)"}` | | `index_file` | 索引单个源文件。 | `{"file_path": "string (必填)"}` | | `force_index_files` | 强制索引文件/目录，跳过默认排除规则（test/, docs/, node_modules/, .gitignore 等）。 | `{"paths": ["string (必填)"], "language_filter": "string (可选)"}` | @@ -318,25 +365,27 @@ codescope index-parallel /path/to/large/project |------|------|------| | `find_callers` | 查找谁调用了某个函数。 | `{"symbol_name": "string (必填)", "file_filter": "string (可选)"}` | | `find_callees` | 查找某个函数调用了什么。 | `{"symbol_name": "string (必填)", "file_filter": "string (可选)"}` | -| `codescope_trace` | 交互式递归调用探索（深度 + 方向）或最短路径。 | `{"function_name": "string", "depth": "integer (默认 1, 最大 5)", "direction": "callers|callees|both", "from": "string", "to": "string"}` | +| `codescope_trace` | 交互式递归调用探索（深度 + 方向）或最短路径。 | `{"function_name": "string（探索模式必填）", "depth": "integer (默认 1, 最大 5)", "direction": "callers\|callees\|both", "from": "string（最短路径模式：需与 to 同时给出）", "to": "string（最短路径模式：需与 from 同时给出）"}` | | `trace_flow` | 递归执行流追踪（caller→callee 链）。 | `{"function_name": "string (必填)", "depth": "integer (默认 3, 最大 10)"}` | -| `shortest_path` | 两个函数之间的最短调用路径（BFS）。 | `{"from": "string", "to": "string", "from_id": "integer", "to_id": "integer"}` | +| `shortest_path` | 两个函数之间的最短调用路径（BFS）。 | `{"from": "string（未给 from_id 时必填）", "to": "string（未给 to_id 时必填）", "from_id": "integer（未给 from 时必填）", "to_id": "integer（未给 to 时必填）"}` | | `connected_components` | 调用图中的连通分量。 | `{}` | +| `get_communities` | **新增** — 通过确定性的标签传播检测调用图中的社区（聚类）。返回 `{communities:[{id,label,member_count}], total_communities, returned_communities, inter_community_edges, truncated}`；除非 `include_members` 为 true，否则不返回成员列表。 | `{"max_communities": "integer（默认 20，上限 500）", "include_members": "boolean（默认 false）", "max_members": "integer（默认 10，上限 200）"}` | ### 图查询 | 工具 | 用途 | 参数 | |------|------|------| -| `graph_query` | Cypher 风格 DSL 查询：`MATCH (Function:main)-[Calls]->(Method)`。 | `{"dsl": "string (必填)"}` | +| `graph_query` | Cypher 风格 DSL 查询：`MATCH (Function:main)-[Calls]->(Method)`，尾部可跟可选子句：`LIMIT `（**生效**，响应会附带 `truncated: true`）与 `RETURN `（接受但当前无效果）。其它尾部文本一律报错。大图上务必用 `LIMIT` —— 宽泛模式可能超过 1 MiB 传输上限（见表格上方的说明）。 | `{"dsl": "string (必填)"}` | | `get_graph` | 分页获取完整代码图。 | `{"node_offset": "integer", "node_limit": "integer (最大 50000)", "edge_offset": "integer", "edge_limit": "integer (最大 200000)", "node_types": "string", "edge_types": "string"}` | | `get_subgraph` | 获取以某节点为中心的局部区域（1 跳）。 | `{"node_id": "integer (必填)", "radius": "integer", "node_types": "string", "edge_types": "string"}` | | `get_neighbors` | 获取图节点的直接邻居（调用者 + 被调用者）。 | `{"node_id": "integer (必填)", "edge_type": "integer (默认 -1)", "radius": "integer"}` | +| `get_knowledge_graph` | 直查知识层表。支持表：`entity`、`relation`、`architecture_edge`、`module_edge`、`capability`、`document`、`module_summary`（见 §6）。 | `{"table": "string (必填)", "limit": "integer (默认 100, 最大 1000)"}` | ### 搜索 | 工具 | 用途 | 参数 | |------|------|------| -| `search` | **推荐** — 统一搜索（基于 FTS5；本 sprint 已下线语义/向量搜索，FTS5 是唯一路径）。 | `{"query": "string (必填)", "limit": "integer (默认 20, 最大 100)"}` | +| `search` | **推荐** — 统一搜索：FTS5 精确/前缀匹配，并在结果不足时由 **n-gram 语义向量检索**补充（v0.2.5 恢复；词法相似度，无需外部模型）。 | `{"query": "string (必填)", "limit": "integer (默认 20, 最大 100)"}` | | `search_code` | [已废弃 — 使用 search] | `{"query": "string (必填)", "limit": "integer"}` | ### 验证 @@ -344,7 +393,7 @@ codescope index-parallel /path/to/large/project | 工具 | 用途 | 参数 | |------|------|------| | `verify_integrity` | 检查 README 中承诺的功能是否确实存在于代码中。 | `{}` | -| `verify_claim` | 验证单个断言（capability_exists / contract_holds / architecture_follows）。 | `{"claim": "string (必填)"}` | +| `verify_claim` | 验证单个断言。`claim` 是 **JSON 对象字符串**，不是自然语言：`{"type": "capability_exists \| contract_holds \| architecture_follows \| function_implements", "subject": "符号/模块/契约", "predicate": "可选补充"}`。缺省或无法识别的 `type` 会返回 `error_code: claim_type_unsupported`，不会被猜测性兜底。 | `{"claim": "string (必填，JSON 对象)"}` | | `verify_summary` | 解析自然语言摘要并验证每个断言。 | `{"text": "string (必填)"}` | | `verify_review` | 验证代码审查评论中的断言。 | `{"text": "string (必填)"}` | | `verify_reality` | 验证 AI 对项目状态的单个陈述。 | `{"text": "string (必填)"}` | @@ -419,6 +468,8 @@ get_knowledge_graph {"table":"capability","limit":10} 所有基准测试在 **Apple M3 Max（36 GB RAM，14 核），macOS，2026-08-14** 上测得，使用 `target/release/codescope`（v0.2.6）`worker`（串行全量索引）模式 + 纯 SQLite 图后端（无 LadybugDB/Kuzu 依赖），`CODESCOPE_INDEX_MODE=normal`。查询延迟通过 MCP server 模式测得（引擎只初始化一次，7 次中位数）。数据反映内存化 fuzzy 解析器 + 排序修复后的状态（边数相对修复前二进制 1,249 → 1,189；节点/文件数不变）。其他硬件会产生不同的结果 — 性能较低的机器上预期会慢一些。 +> **仅供参考的单次测量。** 这些表格是单机一次性测量，不是回归门禁：`benchmarks/run_benchmark.sh` 只能复现**索引时间**部分，下面的查询延迟与 token 节省表为手工记录，没有任何脚本可一键复现。请把它们当作量级参考。 + ### 索引时间 | 项目 | 语言 | 文件数 | 节点数 | 边数 | 索引时间 | 峰值内存 | @@ -529,21 +580,27 @@ cd CodeScope | 变量 | 默认值 | 说明 | |------|--------|------| | `CODESCOPE_DB_PATH` | `.codescope/codescope.db` | SQLite 数据库路径 | -| `CODESCOPE_INDEX_MODE` | `standard` | 索引模式：`fast` / `standard` / `strict` | -| `CODESCOPE_EXCLUDE_PATHS` | （未设置） | 逗号分隔的 glob 排除模式 | -| `CODESCOPE_WORKERS` | `4` | `index-parallel` 的解析 worker 核心数 | +| `CODESCOPE_INDEX_MODE` | `normal` | 解析管线模式：`fast` / `normal` / `deep`。`strict` 是另一类仅影响文件发现的模式 —— 见下方说明。 | +| `CODESCOPE_EXCLUDE_PATHS` | （未设置） | 逗号分隔的 glob 排除模式，按项目相对路径匹配，**最后**应用（覆盖内置过滤）。模式中的字面逗号需转义为 `\,`（如 `a\,b/**`）；`\\` 表示字面反斜杠。 | +| `CODESCOPE_WORKERS` | `min(hw,8)` | 解析 worker 核心总数（`kDefaultParseWorkers=8`）。≤2000 文件走的 in-memory 路径默认 **4**。 | | `CODESCOPE_WORKER_TIMEOUT` | `300` | Worker 子进程超时时间（秒） | -| `CODESCOPE_MAX_FILE_SIZE` | （未设置） | 允许索引的最大源文件大小（字节） | +| `CODESCOPE_MAX_FILE_SIZE` | `5242880`（5 MB） | 允许索引的最大源文件大小（字节）。超限文件被静默跳过。 | +| `CODESCOPE_FAIL_RETRY_MAX` | `1` | 自动索引路径中文件被彻底跳过前允许的解析失败次数（最小 1），见 `parse_failures` / `codescope reset-failures`。仅因**缺少语法**而失败的文件（`language_missing`，例如 `.swift`）不计入，每次运行都会重试。`force_index_files` **不**套用该跳过：被强制索引的文件总会被重新尝试。 | | `CODESCOPE_MMAP_SIZE` | 256 MB | SQLite `mmap_size` 参数值 | | `CODESCOPE_MEM_LIMIT_MB` | `4096` | 动态调度器内存上限（MB） | -| `CODESCOPE_DYNAMIC_SCHED` | `auto` | 动态 CPU 调度：`1` 开启，`0` 关闭，未设置 = 自动 | -| `CODESCOPE_VERBOSE` | `0` | 设为 `1` 开启详细日志 | +| `CODESCOPE_DYNAMIC_SCHED` | （未设置 = 静态） | 可选开启的动态 CPU 调度。规范名是 `CODESCOPE_CPU_DYNAMIC`，本变量是历史别名，两者均可。`1`/`true`/`on` 启用共享 chunk 队列调度器，`0`/`false`/`off` 关闭，未设置则保持**静态**比例分配。 | | `CODESCOPE_LSP` | （未设置） | 类型增强的 LSP 服务端命令 | +> **`strict` 与 `deep`**：`CODESCOPE_INDEX_MODE=strict` 只收紧文件**发现**（对可识别语言的白名单门），不改变解析管线；`deep` 则会额外构建 `normal` 跳过的 n-gram 语义向量。 +> +> **`index-parallel` 始终以 `fast` 模式解析。** 它的每个 module/chunk worker 都硬编码了 `CODESCOPE_INDEX_MODE=fast`（`server/src/scheduler/worker.rs`、`quarantine.rs`），因为合并后的库会再走一遍图构建。因此在 `codescope index-parallel` 下设置 `normal`/`strict`/`deep` **无效**；需要非 `fast` 管线时请使用单进程 `codescope index`。 +> +> **动态调度是 opt-in，永不自动开启。** 生产分发器只检查显式开关，不会因为项目大就自动启用 chunk 调度器（这是有意设计，见 `plan/next/DYNAMIC_SCHED_REDESIGN.md`）。 + --- ## 10. 许可证 Apache 2.0 — 详见 [LICENSE](LICENSE)。 -**CodeScope v0.2.6** — 使用 Rust 2024 + C++23 + tree-sitter + SQLite 构建。 \ No newline at end of file +**CodeScope v0.2.7** — 使用 Rust 2024 + C++23 + tree-sitter + SQLite 构建。 \ No newline at end of file diff --git a/RELEASE.md b/RELEASE.md index 86d47cf..0aa7cfc 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,34 +1,44 @@ -## v0.2.6 (2026-08-14) +# CodeScope v0.2.7 -Speeds up full (non-fast) indexing end-to-end and fixes a resolver JOIN defect that both slowed indexing and silently over-matched cross-file references. Fuzzy symbol search is now fully in-memory (no per-entity SQL), FAST-mode pruning rules are completed, and discovery timing is quantified for the first time — with zero precision loss across every benchmark (accuracy gate stays P/R/F1 = 1.0). +First release since v0.2.6 (`main`, 2026-08-14): **41 commits, 394 files**. Two themes: the architectural work for 0.3 (handle-based ABI, Windows parallel indexing), and silent failure — the cases where a tool answered confidently and wrongly, on the default path, for real projects. -### Performance & Results +### 🚀 New Features -- **Fuzzy search fully in-memory** (`fuzzy_resolver.{h,cpp}`, `pipeline.cpp`): the resolver previously ran up to 3 SQL queries per unresolved ref (case-insensitive + prefix + suffix) and hydrated every fuzzy hit with a per-id SQL lookup. All entities are now loaded once into memory (ASCII-fold exact index + `sqliteLikeMatch`, byte-identical to SQLite LIKE) and hits are copied from an `entity_by_id` map — no SQL in the hot loop. **CodeScope self-index (215 files): `resolver::run` 298ms → 30ms (10.2x), `buildGraph` 332ms → 62ms, index-parallel 686ms → 517ms (-25%)** with identical resolved refs/edges on same-input A/B. -- **Bigger projects gain resolution, not just speed**: the old 500ms fuzzy budget silently dropped queries on large repos; the in-memory path never trips it. **goagent (1,374 files)**: +86 refs resolved, +64 call edges; **rustc (6,029 files)**: fuzzy hits 32 → 918, +283 refs, +223 edges — at lower wall time in every case. -- **Fuzzy prefix/suffix lookup O(N) → O(log N)** via sorted folded-name / reversed-folded-name indexes (`std::lower_bound`); wildcard queries keep the exact SQLite-LIKE path. -- **`buildModuleSummaries` merged its two `relation` LEFT JOINs into one**: rustc 117k-relation × 129k-entity phase drops ~5.95s → **~0.25s (23.8x)**, result-identical. -- **`idx_scope_kind_name(project_id, kind, name)` index**: turns the `scope` full-table scan (129,893 entities × 26,975 scopes per row) into an index seek; the `import.source_scope_id` UPDATE result is now checked (was silently ignored, leaving imports at scope 0). -- **Dead `Literal`/`Variable` rows dropped at the DB write path** (`store_batch.cpp`): `semantic_records` 95,944 → 25,146 rows, SQLite flush ~3.7x faster, full-index time 2.18s → ~1.36s (**-38%**). In-memory GraphBuilder Variable nodes stay intact. -- **Resolver self-join missing the `file_path` term fixed** (`pipeline.cpp`): since `original_id` is per-file, the `global_var_types_`/`global_struct_fields_` preloads cross-matched same-id parents across files (~35x join inflation). Adding `file_path` cut resolver 11.2s → 5.4s and the goagent full index 20.6s → 14.6s (**-29%**) while resolving more refs correctly. -- **More full-index wins**: `buildModuleSummaries` split into two CTEs (17.1s → 174ms on goagent, enhance_project -94%), `idx_sr_proj_file_oid(project_id, file_path, original_id)` composite index (resolver type self-joins 3.3s → 63ms on rustc), parse workers 4 → 8 default (wall-clock -18% on rustc, `CODESCOPE_WORKERS` still overrides), and the per-reference candidate deep-copy eliminated (`resolve_loop` -16%). -- **FAST mode is no longer "NORMAL with a different name"**: `fast_extra_skip_dirs_` (was empty) now skips 11 build/test-artifact dirs (`.output`, `storybook-static`, `__generated__`, `playwright-report`, `test-results`, `allure-results`, `allure-report`, `.sass-cache`, `.scss-cache`, `logs`, `.logs`) plus 4 exact files via `fast_extra_filenames_` (`.eslintcache`, `.stylelintcache`, `.prettiercache`, `tsconfig.tsbuildinfo`) — synthetic A/B drops candidates 4 → 1. Discovery is now timed (`discovery=`, `seen_dirs` counts directories only): rustc 143ms / 4,650 dirs, goagent 25ms / 845 dirs. -- **`get_graph` migrated to canonical tables**: it read the deprecated, empty `graph_nodes`/`graph_edges` (nodes always 0, edges stale) — now pages `entity`+`relation` with the same public JSON schema; `get_graph` matches `get_graph_stats` (`total_nodes:39686, total_edges:4415`). +- **Handle-based engine ABI (TD-1, three knives)** (`engine/include/engine.h`, `engine/src/engine_context.*`): `engine_create()` returns an opaque `engine_t`, all 73 stateful entry points take it, and the three process-global singletons are gone — no global object is constructed before `main()`, and a stale call site is a compile error. The still process-wide subsystems (verifier registry, async knowledge builder, index-progress and parse-failure buffers) are recorded in `engine_context.h` rather than implied. Evidence: a 43-tool differential matrix is byte-identical old vs new, and both binaries index one tree to the same call-edge set. +- **`index-parallel` on Windows (TD-5)** (`server/src/scheduler/mapped_file.rs`): the POSIX `open`/`ftruncate`/`mmap` sequence both schedulers open-coded now lives in one mapping layer with a Windows backend (`CreateFileMappingW`/`MapViewOfFile`), and per-run paths come from `std::env::temp_dir()` instead of `/tmp`. A `windows-smoke` job indexes a real project through both schedulers — **it has not run yet**, so Windows stays Beta (README). +- **Graph-shape regression baselines** (`benchmarks/graph_baseline.sh`, `make bench-graph`): six real checkouts, one per language, compared on files, entities, call edges, `resolution_kind`, `resolve_strategy`, `deciding_factor` and unresolved sites; exact rather than tolerance-based because the shape is reproducible. `self` 2274 · `c-redis` 27474 · `go-tinygo` 4313 · `java-spring-petclinic` 47 · `rust-pyo3` 2859 · `ts-codebase-memory-mcp` 8440 call edges. +- **Counterfactual resolver-factor switch** (`CODESCOPE_RESOLVER_DISABLE_FACTOR=`): drops one factor's weight and score at the single accumulation point, so a factor's real effect is measured by A/B instead of argued from the weight table — `relation.reason`'s `decided_by=` names ImportMatch for every scored edge on this repository, while disabling factors shows which ones actually move the graph. No weight is changed until that table says otherwise. +- **`codescope parse-failures` / `reset-failures`** (the `parse_failures` table was write-only), **`get_communities` restored** (synchronous label propagation; deterministic; summary-first — the earlier revision returned ~100K tokens for one query), **`graph_query` honours `LIMIT`** and rejects trailing text instead of dropping it silently. -### Bug fixes +### 🐛 Bug Fixes -- `FilterPolicy::setMode()` never rebuilt the active skip sets (FAST rules were inert when the mode was set after construction). -- Discovery `seen_dirs` counted files too (inflated ~44k for a 215-file project) — now directories only. -- `import.source_scope_id` UPDATE failures were silently swallowed, leaving imports at scope 0 — now checked and logged. -- Skills scripts/docs updated to the current MCP tool set (`index_project`/`get_hotspots` removed from `TOOL_HANDLERS`; indexing now via `codescope worker` / `index-parallel`, hotspots via `get_knowledge_graph`). -- **Fuzzy prefix/suffix binary-search results truncated in name order instead of rowid order** (`fuzzy_resolver.cpp`): when a prefix/suffix query matched more than `kFuzzyCandidateLimit` (5) entities, the retained subset differed from the old SQL `LIKE ... LIMIT ?` path, so the resolved CALLS edge could diverge on large projects — contradicting the in-memory rewrite's byte-identical contract. Both paths now collect all matches, sort by id (= rowid = load order), then truncate. A/B on CodeScope self index: edges 1,249 → 1,189, nodes/files unchanged, accuracy gate still P/R/F1 = 1.0. -- **`buildModuleSummaries` bound 9 parameters for 8 `?` placeholders** (`state_builder.cpp`): the extra bind hit a non-existent parameter (SQLITE_RANGE, silently ignored); the loop now binds exactly 8 and the comment is corrected. -- **`skills/analyze.sh` / `skills/index.sh` indexed and queried different DBs** when `CODESCOPE_DB_PATH` was unset: `worker` took the DB as a positional argument (default `/tmp/codescope_index.db`) while `cli` read the env var (default `.codescope/codescope.db`), so stats were silently reported from an empty/stale DB. Both scripts now `export CODESCOPE_DB_PATH="$DB"` so worker and cli share one database. +**Indexing and symbol names** -### Documentation +- **A Maven tree indexed nothing**: the engine's test/docs/samples skip collided with Java package components, so `src/main/java/…/samples/petclinic/` was skipped at any depth and `discover_modules` reported no modules. Both entry points now share `applyProjectLanguageContext()`; spring-petclinic goes 0 → 30 files. +- **Java methods were named after their return type** (`JavaVisitor::extractName` accepted `type_identifier` and returned the first match): `public String processUpdateForm(…)` was recorded as an entity called `String` — 27 such entities in petclinic, 4 `Pet`, 3 `LocalDate` — while `void` methods were correct, and those type-named entities then collected edges. Names are read from the grammar's `name` field; petclinic 33 → 47 edges and `find_definition("processUpdateForm")` resolves. +- **The same call shape resolved for Go, Java and Python and was dropped for C++, Rust and TypeScript**: a parameter is a declaration, but only variable declarations fed the visitors' type table, so a call through a parameter carried a receiver with no type and the ambiguity gate abstained. All three now record parameter types; 6 of 6 languages resolve to the owner's method in `test_resolver_language_consistency`. +- **`arity` and `param_count` had no producer**: `param_count` was structurally 0 (no visitor emits a `Parameter` record) and `arity` — the input the resolver's overload step reads — was 0 for all 24604 goagent rows, so every candidate scored as "unknown arity". Both come from the CST declaration now, with a variadic tail or a default reported as unknown rather than a count that would penalise a valid call. +- **TypeScript annotations lost array element types and leaked object-type fragments**: `extractTsTypeAnnotation` unwrapped element and member types as if they were annotations, so `Foo[]` yielded nothing, and it returned raw text for shapes it does not model, so `const tabs: { id: TabId; … }[]` recorded the receiver type of `tabs.map(…)` as `id: TabId`. Element types resolve and anything that is not a plain type name is reported as unknown. +- **Builtin-name filters deleted calls to the file's own functions** (a user `format()`, `valueOf()`, `write()` or `free()` collided with the filter's list and produced no call record at all): each visitor now keeps the names the file declares, and the filter applies only to unqualified calls. +- **C/C++ functions with a qualified return type were invisible** (`std::string`, `char *`, namespace-nested): the name scan matched the return type and dropped the whole definition — on this repository `engine/src/query/**` contributed 3 definitions out of 1972 statements. Names are read from the `declarator` field. +- **Aliased imports resolved to nothing** (`import { pick as pickLib }` bound a local name with no record of the imported symbol) and **a call omitting a default argument lost its edge** (C++ declares the default in the header while the entity's `arity` comes from the definition). -- **Benchmark section re-measured after the fuzzy ordering fix** (`README.md`, `README.zh.md`): §7 now reflects `target/release/codescope` (v0.2.6) in `CODESCOPE_INDEX_MODE=normal` on the 2026-08-14 run — CodeScope self 0.95 s / 1,189 edges (pre-fix: 1,249; nodes/files unchanged), tinygo 1.77 s / 4,485 edges, rustc 38.94 s / 117,284 edges; per-query MCP latency (median of 7), micro benchmarks, cross-file CALLS ratios, and the corrected `graph_query` figure (88.8 ms with LIMIT 100 — the previously published 0.03 ms was the error-path response of an invalid DSL, not a real query). +**Determinism** — one tree, one binary, one graph: the two resolver loader tables are ordered by content (a worker-dependent rowid order decided which interface six call sites on goagent resolved to), and entity ids derive from the record's semantic identity instead of SQLite's scan order. -### Verification +**Silent success and silent drops** — a run that indexed nothing no longer reports `ok:true` (0-file and no-symbol runs report `ok:false`/`complete:false`); failed CSR builds, bulk COMMITs, nested transactions, migrations and batch inserts roll back and report instead of committing whatever landed; `index-parallel` installs its database at the resolved path (it only reported a temp path, so the next command queried an empty DB) and runs the post-index pass, so the DB it hands back is searchable; `force_index_files` is no longer filtered at the graph stage (forcing `engine/tests` added 13174 records and zero entities); `createSchema` no longer has a use-after-free; the CMake generator is resolved from the build tree, because a mismatched generator silently kept new translation units out of the archive. + +**MCP surface** — `isError` was serialized as `is_error`, so a spec-compliant client saw no failure flag for any tool result; an MCP-indexed project had no `projects` row (every entity landed under `project_id = 0`, next to a full graph); front-end build output was indexed (`dist/`: 700 minified nodes beside 224 real symbols); array-shaped `node_types`/`edge_types` filters were silently ignored; `project_overview.entry_points` read the legacy table and was always empty; `verify_summary` and `verify_review` scored the same empty input 1.0 and 0.0; ambiguous names returned an empty list with no next step. + +**Paths and tool contracts** — `detect_changes` reported "nothing changed" for the input shape its own schema advertises and for any path spelling other than the stored one; `find_definition` / `find_references` / `find_callers` / `find_callees` missed a file named through a symlink; `shortest_path` and `codescope_trace` advertised no required arguments; a failing engine test could pass the gate (the `CHECK` counter was consulted only by `return checkFailures() ? 1 : 0`, which 8 test files never wrote). + +### 🧹 Cleanup + +- **One JSON implementation** (TD-3): every hand-built envelope migrated to `util::JsonWriter`, escaping converged on `util::jsonEscapeString`, and the duplicate escapers and declarations deleted. +- **~1.5k lines of dead code removed** (`scanner_visitor`, the superseded resolver stack, the zero-call-site scorers, five duplicate copies of one scoring rule), the superseded `Planner`/`VerdictBuilder` chain deleted, and `verify_statement` retired now that `verify_claim` is the single entry point. + +### ✅ Testing + +- Every engine test uses `CHECK()` (929 assertion sites migrated, no live `assert()` remains) and a recorded failure makes the binary exit non-zero — enforced by an exit hook, because 8 files printed a banner and returned 0. +- New suites: `test_resolver_language_consistency` (six languages, one call shape), `test_import_alias_resolution`, `test_function_arity`, `test_receiver_type_canonical`, `test_dispatch_order_determinism`, `test_metrics_persist`, `test_force_index_files`, `test_engine_handles` (instance and ABI boundaries), `test_resolver_language_filter`, `test_gitignore_build_dirs`, plus property-based tests over clamping, exclude-path escaping and MCP payload round-trips. +- `make check` green: 106 engine test binaries, 136 server tests; call-graph accuracy gate TP 36 / FP 0 / FN 0; `make bench-graph` matches all six baselines. -All C++ engine tests pass (including `test_call_graph_accuracy`, `test_metrics_readiness`, `test_step11_go_smoke`), accuracy gate P/R/F1 = 1.0 with FP/FN injection correctly rejected, Rust server tests 88/88, clippy + clang-format clean. See [CHANGELOG.md](./CHANGELOG.md) for the complete list. diff --git a/benchmarks/README.md b/benchmarks/README.md index e872c34..2e28ebf 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -1,58 +1,108 @@ -# CodeScope 性能基准 +# CodeScope Performance Benchmarks -## 目录结构 +## Directory Structure ``` benchmarks/ -├── README.md # 本文件 -├── run_benchmark.sh # 基准测试运行脚本 -├── baselines/ # 各项目的基准基线（JSON） -│ ├── rustc.json +├── README.md # This file +├── run_benchmark.sh # Performance benchmark execution script +├── graph_baseline.sh # Graph-SHAPE regression (see below) +├── baselines/ # Baseline results for each project (JSON) +│ ├── rustc.json # performance: index time, latency, memory │ ├── jdk.json │ ├── bun.json │ ├── cpython.json │ ├── memscope.json -│ └── ares.json -└── results/ # 最新运行结果 - └── latest.json # 当前最新结果（symlink） +│ ├── ares.json +│ └── graph_*.json # graph shape: edges + resolution distribution +└── results/ # Latest benchmark results + └── latest.json # Current latest result (symlink) ``` -## 测量指标 +## Measurement Metrics -| 指标 | 说明 | 重要性 | +| Metric | Description | Importance | |------|------|:------:| -| `index_time` | 完整索引耗时 | 核心 | -| `index_speed` | 文件/秒 | 核心 | -| `query_latency.p50` | 各查询中位数延迟 | 核心 | -| `query_latency.p99` | 各查询 P99 延迟 | 核心 | -| `memory.rss_peak` | 峰值 RSS 内存 | 重要 | -| `memory.rss_post_gc` | 索引后 GC 归还的 RSS | 重要 | -| `node_count` | 总节点数 | 参考 | -| `edge_count` | 总边数 | 参考 | +| `index_time` | Total index time | Core | +| `index_speed` | Files per second | Core | +| `query_latency.p50` | Median latency per query | Core | +| `query_latency.p99` | P99 latency per query | Core | +| `memory.rss_peak` | Peak RSS memory | Important | +| `memory.rss_post_gc` | RSS after GC cleanup | Important | +| `node_count` | Total node count | Reference | +| `edge_count` | Total edge count | Reference | -## 运行基准测试 +## Running Benchmarks ```bash -# 对所有基线项目运行基准测试 +# Run benchmarks for all baseline projects ./run_benchmark.sh --all -# 对特定项目运行 +# Run for a specific project ./run_benchmark.sh --project /path/to/repo --name my_project -# 比较结果 +# Compare results ./run_benchmark.sh --compare ``` -## 基线 +## Baselines -当前基线在 `baselines/` 目录下。`run_benchmark.sh --all` 会将结果写入 `results/` 并按日期命名。 -`--compare` 模式输出表格对比最新结果与基线。 +Current baselines are in the `baselines/` directory. `run_benchmark.sh --all` will write results to `results/` named by date. +`--compare` mode outputs a table comparing latest results with baselines. -## 回归检测 +## Regression Detection -`--check` 模式会检测： -- 索引时间退化 >20% -- 查询延迟退化 >30% -- 内存增长 >15% +`--check` mode detects: +- Index time regression >20% +- Query latency regression >30% +- Memory growth >15% -超出阈值会返回非零退出码和退化报告。 +Exceeding thresholds will return a non-zero exit code with a regression report. + +## Graph-Shape Regression + +The benchmarks above measure SPEED. `graph_baseline.sh` measures the RESULT: +how many call edges the resolver produces on a fixed tree, and how they are +distributed across resolution kinds, call-site strategies and deciding factors. +A visitor or resolver-factor change can add, drop or re-point edges on real +projects while `make test-engine` stays green, because the unit tests pin +hand-built fixtures. + +```bash +make bench-graph # compare against baselines/my checkouts +make bench-graph-update # rewrite them (intended changes only) +./benchmarks/graph_baseline.sh --only self # one project +CODESCOPE_BENCH_PROJECTS=/path/to/checkouts make bench-graph +``` + +The comparison is exact — no tolerance window — because the shape is +reproducible: two consecutive `index-parallel` runs of the same binary over the +same tree produce identical edge sets (0 added, 0 removed). A baseline that +drifts after a code change is the signal; a baseline that drifts between two +runs of one binary would be a determinism bug in the indexer. + +Current baselines: `self` (C++), `c-redis` (C), `go-tinygo` (Go), +`java-spring-petclinic` (Java), `rust-pyo3` (Rust) and +`ts-codebase-memory-mcp` (TypeScript, C). + +`self` is the one baseline that tracks the WORKING TREE rather than an external +checkout: it indexes this repository, so adding or removing any indexed source +file under `engine/src` changes its shape by construction and the run reports a +diff (`files`, `entities`, `call_edges` all move) until `--update` rewrites the +file. A diff on `self` alone therefore means "this tree changed", not +"resolution changed" — the other five index fixed checkouts under +`$CODESCOPE_BENCH_PROJECTS` and only move when the resolver moves, which makes +them the ones to read when deciding whether a code change altered resolution. + +Java was missing until the discovery defect behind it was fixed: +`discover::discover_modules` asks the engine (`engine_path_is_skipped`) which +directories hold source, and the policy behind that query was built with no +language context, so the test/docs/samples skip names took their any-depth +branch. spring-petclinic keeps its entire source tree under +`src/main/java/org/springframework/samples/petclinic/`, so the scheduler found +**zero** modules, printed `note: "no source modules found"` and indexed nothing +— 0 files, 0 nodes — while `codescope discover ` counted all 49 +files. The Java relaxation (top-only, depth ≤ 3) existed, but only the indexer's +own walk knew to set it. Both entry points now share one +`applyProjectLanguageContext()`; petclinic indexes 30 files / 137 nodes / 33 +edges with `src/test` still excluded, and the baseline above is its graph. \ No newline at end of file diff --git a/benchmarks/README.zh.md b/benchmarks/README.zh.md new file mode 100644 index 0000000..e872c34 --- /dev/null +++ b/benchmarks/README.zh.md @@ -0,0 +1,58 @@ +# CodeScope 性能基准 + +## 目录结构 + +``` +benchmarks/ +├── README.md # 本文件 +├── run_benchmark.sh # 基准测试运行脚本 +├── baselines/ # 各项目的基准基线（JSON） +│ ├── rustc.json +│ ├── jdk.json +│ ├── bun.json +│ ├── cpython.json +│ ├── memscope.json +│ └── ares.json +└── results/ # 最新运行结果 + └── latest.json # 当前最新结果（symlink） +``` + +## 测量指标 + +| 指标 | 说明 | 重要性 | +|------|------|:------:| +| `index_time` | 完整索引耗时 | 核心 | +| `index_speed` | 文件/秒 | 核心 | +| `query_latency.p50` | 各查询中位数延迟 | 核心 | +| `query_latency.p99` | 各查询 P99 延迟 | 核心 | +| `memory.rss_peak` | 峰值 RSS 内存 | 重要 | +| `memory.rss_post_gc` | 索引后 GC 归还的 RSS | 重要 | +| `node_count` | 总节点数 | 参考 | +| `edge_count` | 总边数 | 参考 | + +## 运行基准测试 + +```bash +# 对所有基线项目运行基准测试 +./run_benchmark.sh --all + +# 对特定项目运行 +./run_benchmark.sh --project /path/to/repo --name my_project + +# 比较结果 +./run_benchmark.sh --compare +``` + +## 基线 + +当前基线在 `baselines/` 目录下。`run_benchmark.sh --all` 会将结果写入 `results/` 并按日期命名。 +`--compare` 模式输出表格对比最新结果与基线。 + +## 回归检测 + +`--check` 模式会检测： +- 索引时间退化 >20% +- 查询延迟退化 >30% +- 内存增长 >15% + +超出阈值会返回非零退出码和退化报告。 diff --git a/benchmarks/README_EN.md b/benchmarks/README_EN.md deleted file mode 100644 index fc1d89c..0000000 --- a/benchmarks/README_EN.md +++ /dev/null @@ -1,58 +0,0 @@ -# CodeScope Performance Benchmarks - -## Directory Structure - -``` -benchmarks/ -├── README.md # This file -├── run_benchmark.sh # Benchmark execution script -├── baselines/ # Baseline results for each project (JSON) -│ ├── rustc.json -│ ├── jdk.json -│ ├── bun.json -│ ├── cpython.json -│ ├── memscope.json -│ └── ares.json -└── results/ # Latest benchmark results - └── latest.json # Current latest result (symlink) -``` - -## Measurement Metrics - -| Metric | Description | Importance | -|------|------|:------:| -| `index_time` | Total index time | Core | -| `index_speed` | Files per second | Core | -| `query_latency.p50` | Median latency per query | Core | -| `query_latency.p99` | P99 latency per query | Core | -| `memory.rss_peak` | Peak RSS memory | Important | -| `memory.rss_post_gc` | RSS after GC cleanup | Important | -| `node_count` | Total node count | Reference | -| `edge_count` | Total edge count | Reference | - -## Running Benchmarks - -```bash -# Run benchmarks for all baseline projects -./run_benchmark.sh --all - -# Run for a specific project -./run_benchmark.sh --project /path/to/repo --name my_project - -# Compare results -./run_benchmark.sh --compare -``` - -## Baselines - -Current baselines are in the `baselines/` directory. `run_benchmark.sh --all` will write results to `results/` named by date. -`--compare` mode outputs a table comparing latest results with baselines. - -## Regression Detection - -`--check` mode detects: -- Index time regression >20% -- Query latency regression >30% -- Memory growth >15% - -Exceeding thresholds will return a non-zero exit code with a regression report. \ No newline at end of file diff --git a/benchmarks/baselines/graph_c-redis.json b/benchmarks/baselines/graph_c-redis.json new file mode 100644 index 0000000..943ff54 --- /dev/null +++ b/benchmarks/baselines/graph_c-redis.json @@ -0,0 +1,19 @@ +{ + "files": 452, + "entities": 12046, + "call_edges": 27474, + "call_sites": 61289, + "resolution_kind": { + "exact_local": 24673, + "imported": 2801 + }, + "resolve_strategy": { + "external": 2312, + "p1_intra": 19047, + "unresolved": 39930 + }, + "deciding_factor": { + "ImportMatch": 2801 + }, + "unresolved_call_sites": 33815 +} diff --git a/benchmarks/baselines/graph_go-tinygo.json b/benchmarks/baselines/graph_go-tinygo.json new file mode 100644 index 0000000..52e4a84 --- /dev/null +++ b/benchmarks/baselines/graph_go-tinygo.json @@ -0,0 +1,21 @@ +{ + "files": 816, + "entities": 21587, + "call_edges": 4313, + "call_sites": 27804, + "resolution_kind": { + "dispatch": 147, + "exact_local": 2684, + "imported": 1480, + "qualified": 2 + }, + "resolve_strategy": { + "external": 3411, + "p1_intra": 3512, + "unresolved": 20881 + }, + "deciding_factor": { + "ImportMatch": 1480 + }, + "unresolved_call_sites": 23491 +} diff --git a/benchmarks/baselines/graph_java-spring-petclinic.json b/benchmarks/baselines/graph_java-spring-petclinic.json new file mode 100644 index 0000000..1e00614 --- /dev/null +++ b/benchmarks/baselines/graph_java-spring-petclinic.json @@ -0,0 +1,19 @@ +{ + "files": 25, + "entities": 137, + "call_edges": 47, + "call_sites": 234, + "resolution_kind": { + "exact_local": 43, + "imported": 4 + }, + "resolve_strategy": { + "external": 10, + "p1_intra": 15, + "unresolved": 209 + }, + "deciding_factor": { + "ImportMatch": 4 + }, + "unresolved_call_sites": 187 +} diff --git a/benchmarks/baselines/graph_rust-pyo3.json b/benchmarks/baselines/graph_rust-pyo3.json new file mode 100644 index 0000000..6e1055d --- /dev/null +++ b/benchmarks/baselines/graph_rust-pyo3.json @@ -0,0 +1,21 @@ +{ + "files": 266, + "entities": 8586, + "call_edges": 2859, + "call_sites": 14341, + "resolution_kind": { + "dispatch": 61, + "exact_local": 1243, + "imported": 1118, + "qualified": 437 + }, + "resolve_strategy": { + "external": 304, + "p1_intra": 2736, + "unresolved": 11301 + }, + "deciding_factor": { + "ImportMatch": 1118 + }, + "unresolved_call_sites": 11482 +} diff --git a/benchmarks/baselines/graph_self.json b/benchmarks/baselines/graph_self.json new file mode 100644 index 0000000..d542eae --- /dev/null +++ b/benchmarks/baselines/graph_self.json @@ -0,0 +1,19 @@ +{ + "files": 213, + "entities": 2833, + "call_edges": 2274, + "call_sites": 16743, + "resolution_kind": { + "exact_local": 1026, + "imported": 1248 + }, + "resolve_strategy": { + "external": 2383, + "p1_intra": 1501, + "unresolved": 12859 + }, + "deciding_factor": { + "ImportMatch": 1248 + }, + "unresolved_call_sites": 14469 +} diff --git a/benchmarks/baselines/graph_ts-codebase-memory-mcp.json b/benchmarks/baselines/graph_ts-codebase-memory-mcp.json new file mode 100644 index 0000000..7c9dde4 --- /dev/null +++ b/benchmarks/baselines/graph_ts-codebase-memory-mcp.json @@ -0,0 +1,21 @@ +{ + "files": 204, + "entities": 4143, + "call_edges": 8440, + "call_sites": 29482, + "resolution_kind": { + "exact_local": 6946, + "import_module": 19, + "imported": 1475 + }, + "resolve_strategy": { + "external": 1875, + "p1_intra": 8468, + "unresolved": 19139 + }, + "deciding_factor": { + "ImportMatch": 1475, + "ImportModuleMatch": 19 + }, + "unresolved_call_sites": 21042 +} diff --git a/benchmarks/graph_baseline.sh b/benchmarks/graph_baseline.sh new file mode 100755 index 0000000..5bde44b --- /dev/null +++ b/benchmarks/graph_baseline.sh @@ -0,0 +1,214 @@ +#!/usr/bin/env bash +# graph_baseline.sh — graph-SHAPE regression baseline. +# +# The other benchmarks here measure speed and memory. This one measures the +# RESULT: how many call edges the resolver produces on a fixed tree, and how +# they are distributed across resolution kinds, parse-time strategies and +# deciding factors. A change to any visitor or to a resolver factor can add, +# drop or re-point edges, and `make test-engine` cannot see it — the unit tests +# pin hand-built fixtures, not the shape of a real project's graph. +# +# ./benchmarks/graph_baseline.sh # compare against the baselines +# ./benchmarks/graph_baseline.sh --update # (re)write the baselines +# ./benchmarks/graph_baseline.sh --only self # one project +# +# Exit status is non-zero when any baseline differs, so CI (or a pre-commit +# hook) can gate on it. Requires bin/codescope, built by `make build`. +# +# Determinism: the shape is reproducible for a fixed binary and tree — two +# consecutive index runs of the fallback project produced identical edge sets +# (0 added, 0 removed) — so the comparison is exact, with no tolerance window. +# A baseline that drifts because the CODE changed is the point; a baseline that +# drifts between two runs of the same binary would be a bug in the indexer, +# which is what makes this script's failure mode informative. +# +# Projects are local checkouts, named by language, so a change that helps Go and +# hurts Rust is visible instead of averaged away. Set CODESCOPE_BENCH_PROJECTS +# to relocate them; missing projects are skipped with a warning. +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +BASELINE_DIR="$ROOT/benchmarks/baselines" +# One work directory per PROCESS: a fixed path let two concurrent runs index +# into the same files, which showed up as "no such table: entity" for whichever +# project lost the race rather than as a clear failure. +WORK_DIR="${TMPDIR:-/tmp}/codescope-graph-baseline/$$" +PROJECTS_ROOT="${CODESCOPE_BENCH_PROJECTS:-/Users/scc/code/researcher}" +BIN="$ROOT/bin/codescope" + +UPDATE=0 +ONLY="" +while [[ $# -gt 0 ]]; do + case "$1" in + --update) UPDATE=1 ;; + --only) + shift + ONLY="${1:-}" + ;; + *) + echo "usage: $0 [--update] [--only ]" >&2 + exit 2 + ;; + esac + shift +done + +# name|path — the name is the baseline file's suffix, the path is a checkout. +# `self` is this repository: the C++ tree the engine is developed against. +# +# One project per language, so a change that helps Go and hurts Rust is visible +# instead of averaged away. A Java checkout is deliberately NOT listed: +# index-parallel could not index a Maven layout at all — discover_modules +# applied the any-depth branch of the test/docs/samples skip rules to +# `src/main/java/org/.../samples/petclinic/` and reported "no source modules +# found" for the whole tree. Fixed in this release (the policy behind +# engine_path_is_skipped now learns its language the way the indexer does), so +# the Java baseline is back. +PROJECTS=( + "self|$ROOT" + "c-redis|$PROJECTS_ROOT/redis" + "go-tinygo|$PROJECTS_ROOT/tinygo" + "java-spring-petclinic|$PROJECTS_ROOT/spring-petclinic" + "rust-pyo3|$PROJECTS_ROOT/pyo3" + "ts-codebase-memory-mcp|$PROJECTS_ROOT/codebase-memory-mcp" +) + +[[ -x "$BIN" ]] || { + echo "graph_baseline: $BIN not found — run 'make build' first" >&2 + exit 2 +} +mkdir -p "$BASELINE_DIR" "$WORK_DIR" + +# The factor that decided each scored match, extracted from +# relation.reason ("decided_by= name=... arity=... score=..."). +FACTOR_SQL="SELECT substr(r, 1, instr(r || ' ', ' ') - 1) AS c, COUNT(*) AS n + FROM (SELECT substr(reason, instr(reason, 'decided_by=') + 11) AS r + FROM relation WHERE type = 1 AND reason LIKE '%decided_by=%') + GROUP BY c ORDER BY c" + +# dump_shape — the canonical, sorted, diffable shape of one graph. +dump_shape() +{ + local db="$1" out="$2" + { + printf '{\n' + printf ' "files": %s,\n' \ + "$(sqlite3 "$db" 'SELECT COUNT(DISTINCT file_path) FROM entity;')" + printf ' "entities": %s,\n' \ + "$(sqlite3 "$db" 'SELECT COUNT(*) FROM entity;')" + printf ' "call_edges": %s,\n' \ + "$(sqlite3 "$db" 'SELECT COUNT(*) FROM relation WHERE type = 1;')" + printf ' "call_sites": %s,\n' \ + "$(sqlite3 "$db" 'SELECT COUNT(*) FROM reference;')" + dump_group "$db" "resolution_kind" \ + "SELECT COALESCE(NULLIF(resolution_kind, ''), '(empty)') AS c, + COUNT(*) AS n + FROM relation WHERE type = 1 GROUP BY c ORDER BY c" + # The parse-time strategy lives on the call SITE (reference), not on + # the edge: it is what the visitor guessed before the resolver ran, so + # its distribution is a different question from resolution_kind. + dump_group "$db" "resolve_strategy" \ + "SELECT COALESCE(NULLIF(resolve_strategy, ''), '(empty)') AS c, + COUNT(*) AS n + FROM reference GROUP BY c ORDER BY c" + dump_group "$db" "deciding_factor" "$FACTOR_SQL" + printf ' "unresolved_call_sites": %s\n' \ + "$(sqlite3 "$db" 'SELECT (SELECT COUNT(*) FROM reference) - + (SELECT COUNT(*) FROM relation WHERE type = 1);')" + printf '}\n' + } >"$out" +} + +# dump_group — one JSON object of value→count, last value +# without a trailing comma so the file is valid JSON. +dump_group() +{ + local db="$1" key="$2" sql="$3" + printf ' "%s": {\n' "$key" + sqlite3 "$db" \ + "SELECT ' \"' || c || '\": ' || n || ',' FROM ($sql);" | + sed '$ s/,$//' + printf ' },\n' +} + +total=0 +failures=0 +for entry in "${PROJECTS[@]}"; do + name="${entry%%|*}" + path="${entry#*|}" + [[ -n "$ONLY" && "$name" != "$ONLY" ]] && continue + if [[ ! -d "$path" ]]; then + printf ' %-26s skipped (no checkout at %s)\n' "$name" "$path" + continue + fi + + db="$WORK_DIR/$name.db" + current="$WORK_DIR/$name.json" + baseline="$BASELINE_DIR/graph_$name.json" + rm -f "$db" "$db-wal" "$db-shm" + + CODESCOPE_DB_PATH="$db" "$BIN" index-parallel "$path" -w 4 \ + >"$WORK_DIR/$name.log" 2>&1 || + { + printf ' %-26s INDEX FAILED (see %s)\n' "$name" \ + "$WORK_DIR/$name.log" + failures=$((failures + 1)) + continue + } + # `index-parallel` exits 0 when discovery found nothing to index, so the + # exit status alone cannot tell a real index from an empty one; the + # scheduler's own `note` is the only actionable diagnosis. + if [[ "$(sqlite3 "$db" 'SELECT COUNT(*) FROM entity;' 2>/dev/null || echo 0)" == 0 ]]; then + printf ' %-26s INDEXED NOTHING: %s\n' "$name" \ + "$(grep -o '"note":"[^"]*"' "$WORK_DIR/$name.log" | tail -1)" + printf ' (the indexer saw no source files in %s)\n' "$path" + failures=$((failures + 1)) + continue + fi + dump_shape "$db" "$current" + total=$((total + 1)) + + if [[ "$UPDATE" == 1 ]]; then + cp "$current" "$baseline" + printf ' %-26s baseline updated (%s edges)\n' "$name" \ + "$(sqlite3 "$db" 'SELECT COUNT(*) FROM relation WHERE type=1;')" + continue + fi + if [[ ! -f "$baseline" ]]; then + printf ' %-26s no baseline yet — run with --update\n' "$name" + failures=$((failures + 1)) + continue + fi + if diff -q "$baseline" "$current" >/dev/null; then + printf ' %-26s ✓ identical (%s edges)\n' "$name" \ + "$(sqlite3 "$db" 'SELECT COUNT(*) FROM relation WHERE type=1;')" + else + printf ' %-26s ✗ SHAPE CHANGED\n' "$name" + # `diff` exits 1 when it finds differences, which under `set -e` + + # `pipefail` aborted the whole run right here — the first project to + # drift ended the sweep, so the projects after it were never compared. + diff -u "$baseline" "$current" | sed -n '1,60p' || true + failures=$((failures + 1)) + fi +done + +if [[ "$total" == 0 ]]; then + printf 'graph_baseline: no project could be measured (work dir kept: %s)\n' \ + "$WORK_DIR" >&2 + exit 2 +fi +if [[ "$UPDATE" == 1 ]]; then + printf 'graph_baseline: %s baseline(s) written to %s\n' "$total" \ + "$BASELINE_DIR" + [[ "$failures" == 0 ]] && rm -rf "$WORK_DIR" + exit 0 +fi +if [[ "$failures" != 0 ]]; then + printf 'graph_baseline: %s of %s project(s) differ from their baseline\n' \ + "$failures" "$total" + printf 'inspect the diff above; if the change is intended, re-run with --update\n' + printf '(the databases are kept for inspection: %s)\n' "$WORK_DIR" + exit 1 +fi +rm -rf "$WORK_DIR" +printf 'graph_baseline: %s project(s) match their baseline\n' "$total" diff --git a/bootstrap.sh b/bootstrap.sh index f39b64c..075a385 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -189,7 +189,7 @@ echo " Quick start:" echo " codescope discover /path/to/project" echo "" echo " Index a project:" -echo " codescope cli index_project '{\"project_path\":\"/path/to/project\"}'" +echo " codescope index-parallel /path/to/project" echo "" echo " Start MCP server:" echo " codescope" diff --git a/docs/ACCURACY_PROGRESS.md b/docs/ACCURACY_PROGRESS.md deleted file mode 100644 index 709731d..0000000 --- a/docs/ACCURACY_PROGRESS.md +++ /dev/null @@ -1,98 +0,0 @@ -# CodeScope Accuracy Improvement — Execution Progress - -> Tracking document for `ACCURACY_IMPROVEMENT_DEVELOPMENT_PLAN.md` -> Baseline branch: `dev` @ `eca4bd0` -> Started: 2026-07-31 -> Coding standard: `plan/rules/code_rules.md` - -## Legend - -- ☐ Not started -- 🚧 In progress -- ✅ Completed & verified by `make check` -- ⚠️ Completed with caveats -- ❌ Blocked - -## Per-Step Status - -| Step | Title | Status | Owner Agent | make check | Notes | -|------|-------|--------|-------------|------------|-------| -| 0 | Freeze relation contract & baseline | ✅ | main | 13/13 pass | Helpers in `graph_types.{h,cpp}`; contract in `plan/rules/relation_contract.md`; baseline test `test_accuracy_baseline.cpp` emits JSON | -| 1 | CALLS query boundary + dedup | ✅ | agent-A | 15/15 pass | `CALLS`-only Cypher + `edge_type=1` filter; `UNIQUE(project_id,source_id,target_id,type)` on `relation`; Ladybug schema v2→v3; defensive result dedup; counter-example `test_typed_relation_query` | -| 2 | Accuracy Benchmark | ✅ | agent-A | 16/16 pass | TP/FP/FN/P/R/F1 runner; 7-language portable fixtures; `make accuracy-check` target with fault injection; baseline P=1.0 R=1.0 F1=1.0 | -| 3 | Call fact schema (receiver/qualified) | ✅ | agent-A | verified | `reference` 表新增 `qualified_target/receiver_text/receiver_type/import_alias/call_site_file` 列（store_schema.cpp §3.1）；`semantic_records` 同列镜像 + migration；round-trip 由 test_step11_go_smoke/accuracy fixtures 覆盖 | -| 4 | Per-language fact extraction | ✅ | agent-A | verified | Go/Py/C++/Rust/Java/JS/TS visitors 填充 receiver/qualified/import_alias（setCallFacts）；reference 表实测 `p.helper|p|Point`、`obj.render|obj|Timeline` 等证据正确 | -| 5 | Resolver exact-first refactor | ✅ | agent-A | verified | exact-first 候选链 + ambiguity gate + evidence-gated fuzzy + receiver-type factor；修复 receiver 强证据被归一化 margin 误杀（0.08<0.15）→ 新增 receiver bypass，accuracy gate 0 FP/0 FN | -| 6 | relation provenance | ✅ | agent-A | verified | relation 表 confidence/resolver/resolution_kind/reason/call_site_* 全列写入；Ladybug CALLS schema v3→v4 增加 compact provenance（confidence/resolver/resolution_kind）；Graph Compiler 双路径 CSV 对齐 10 列；callers/callees/ByEntity API 返回真实 confidence/resolver/resolution_kind（resolve_strategy 由 resolution_kind 映射，不再恒为空） | -| 7 | Query identity model | ✅ | agent-A | verified | `getCallersByEntity/getCalleesByEntity`（C++ + FFI + engine.h + server FFI 绑定）；裸名 API 歧义检测 `ambiguous=true + candidates`（detectBareNameAmbiguity，接入 getCallers/getCallees）；test_homonym_filter 改为精确身份测试（无过滤=ambiguous+2 candidates，过滤后各自命中） | -| 8 | Dynamic dispatch modeling | ✅ | agent-A | verified | interface_impl_index_ 预加载（semantic_records kind=20）+ Interface/Virtual 调用展开为 bounded candidate set（resolution_kind="dispatch"），receiver 未知时不伪造唯一实现；Java/Rust visitor 产出 InterfaceImpl 记录 | -| 9 | Verifier registry/coverage/evidence | ✅ | agent-B | 28/28 pass | Lifecycle fix (A15), FunctionImplementsVerifier (A16), canonical entity/relation evidence (A17), distinguishable error codes (A21); introspection API `engine_get_verifier_registry_status` (9.2); `test_verifier_registry` 10/10 + `test_verifier_lifecycle` 4/4 + `test_verifier_claim_coverage` 8/8 + `test_verifier_ground_truth` 6/6 | -| 10 | Metrics/Embedding/Semantic | ✅ | agent-C | 53/53 pass | SUNSET for metrics + embedding/semantic; FTS kept. See verification log + decisions log | -| 11 | Real project calibration & CI gate | ✅ | main | verified | `test_step11_go_smoke` L1-L5 全链路（source→reference→relation→Ladybug→adaptive API）PASS；CI 接入 accuracy gate（_ci.yml 新增 "Accuracy gate" 步骤：baseline 必须过 + FP/FN 注入必须失败）；legacy graph_edges CSV 列数对齐 v4；`make test-engine` 全绿 | -| Review | Final code review & `make check` | ✅ | main | verified | `make test-engine` 158 个 passed 标记 + exit 0；accuracy gate 0 FP/0 FN；`cargo check` server 通过 | - -## Acceptance Gates (from §10 of plan) - -Will be ticked as each step lands. Final completion requires: - -1. ✅ No non-Calls relations or duplicate typed edges in callers/callees — Step 1: CALLS-only Cypher + `edge_type=1`; `test_typed_relation_query` 反例验证 -2. ✅ All main languages have portable multi-file accuracy fixtures — Step 2: 7 languages (cpp/go/python/rust/java/js/ts) with ground_truth.json -3. ✅ CI auto-emits Precision/Recall/F1 — Step 2: `make accuracy-check` runs baseline + fault injection; `test_call_graph_accuracy` in TEST_EXES; Step 11: CI accuracy gate step in `_ci.yml` -4. ✅ receiver/qualified/import evidence flows to Resolver — Step 3/4: reference 表列 + visitors 填充 + RefRow 贯通（实测 `p.helper|p|Point`、`obj.render|obj|Timeline`） -5. ✅ Resolver abstains on ambiguous calls (no insertion-order dependence) — Step 5: ambiguity gate + receiver strong-evidence bypass; accuracy gate 0 FP/0 FN -6. ✅ relation & Ladybug CALLS carry queryable provenance — Step 6: relation 全列 + Ladybug CALLS v4 compact provenance；callers/callees API 返回 confidence/resolver/resolution_kind -7. ✅ Same-name entities queryable by stable identity — Step 7: `getCallersByEntity/getCalleesByEntity` + 裸名 `ambiguous=true + candidates` -8. ✅ SQLite ↔ LadybugDB typed-graph diff = 0 — `test_ladybug_diff` 11/11 passed（本地含 LadybugDB 验证） -9. ✅ Go positive-control calls verified end-to-end (source→reference→relation→Ladybug→API) — `test_step11_go_smoke` L1-L5 PASS -10. ✅ verifier passes lifecycle/coverage/evidence regression -11. ✅ metrics/embedding/semantic either real or explicitly sunset (no placeholder 0) — Step 10 sunset: metrics + embedding/semantic marked `available:false`/`unavailable_reason:"sunset"`; complexity APIs return `{"complexity":null,"unavailable":true}` (A18 fixed); FTS kept -12. ✅ readiness matches canonical data coverage — Step 10: `vector_ready` conditional on `node_vectors` row count (A19 fixed); `metrics_ready` structurally 0 (sunset); `embedding_ready` reads canonical `node_vectors` count; `engine_get_enhancement_status` returns real entity/relation/node_vectors counts (A20 fixed) -13. ✅ correctness/stability/performance gates green — `make test-engine` exit 0（158 passed 标记）；`test_call_graph_accuracy` 连续运行确定；`cargo check` server 通过 -14. ✅ all "actual values" come from reproducible commands — baseline/accuracy JSON 由 `make accuracy-check` 与测试二进制生成，非估算值 - -## Build & Test Verification Log - -| Date | Step | Command | Result | -|------|------|---------|--------| -| 2026-07-31 | baseline | `make build` | ✅ all targets link | -| 2026-07-31 | Step 0 | 13 accuracy tests | ✅ all pass after contract helpers + Graph Compiler split change | -| 2026-07-31 | Step 0 | `test_accuracy_baseline` | ✅ emits `/tmp/codescope_accuracy_baseline.json` (entity=7, relation.total=3, relation.type_1_calls=3, duplicate_typed=0, both probes true) | -| 2026-07-31 | Step 0 | `test_homonym_filter` | ⚠️ passes (73 → 2 with filter) but depends on local `/Users/scc/code/pycode/Transformer_Explorer`; output contains duplicates (node_id 113 appears 3×). Step 2 will replace with portable fixture. | -| 2026-07-31 | Step 1 | `cmake --build engine/build` | ✅ builds (cleared stale ccache/PCH that pre-dated Step 1 — unrelated verifier `kEdgeTypeCalls`→`kRelationTypeCalls` rename from Track B was served stale; not a Step 1 regression) | -| 2026-07-31 | Step 1 | 15 accuracy tests (13 FP/precision + baseline + typed_relation) | ✅ all pass | -| 2026-07-31 | Step 1 | `test_typed_relation_query` | ✅ counter-example: same source→target with References(0)+Calls(1)+Defines(2)+Contains(3) → getCallees/getCallers return ONLY the Calls edge (total=1); duplicate Calls(1) rejected by unique index | -| 2026-07-31 | Step 9 | `cmake --build engine/build` | ✅ all 71 targets link (incl. new `test_verifier_lifecycle`, `test_verifier_claim_coverage`, `test_verifier_ground_truth`); clang-format `--dry-run --Werror` clean on all modified verifier files | -| 2026-07-31 | Step 9 | `test_verifier_lifecycle` | ✅ 4/4 pass: 3-cycle init/verify/shutdown; restore DB without `engine_create_project`; multi-project sequential verify; unknown claim type → input error | -| 2026-07-31 | Step 9 | `test_verifier_claim_coverage` | ✅ 8/8 pass: 100% claim-type coverage (4/4 supported); all types dispatch; FunctionImplements/Capability/Contract/Architecture ground truth; evidence backend not ready → Unknown; distinct error codes | -| 2026-07-31 | Step 9 | `test_verifier_ground_truth` | ✅ 6/6 pass: introspection API healthy; FunctionImplements Supported + evidence_facts; isolated function → Unknown; non-existent → Contradicted; backend not ready → Unknown for ALL types; introspection with project_id=0 | -| 2026-07-31 | Step 9 | `test_verifier_registry` | ✅ 10/10 pass (no regression): names, dispatch per type, type-exclusive accepts, supported_claim_types covers all public, wire names, ensureDefaultVerifiers idempotent, clear→ensure re-populates | -| 2026-07-31 | Step 9 | `test_accuracy_baseline` | ✅ no regression: entity=7, relation.total=3, type_1_calls=3, duplicate_typed=0, both probes true | -| 2026-07-31 | Step 9 | full `make test-engine` suite (54 tests) | ✅ 54/54 pass (CMake reconfigured to pick up `test_verifier_ground_truth` via `file(GLOB)`; all 4 verifier test binaries green) | -| 2026-07-31 | Step 10 | `cmake --build engine/build` | ✅ all targets link (incl. new `test_metrics_readiness`); clang-format clean on all 8 modified C++ files | -| 2026-07-31 | Step 10 | `test_metrics_readiness` | ✅ 12/12 pass: enhancement_status real counts (total=2 cg=2 metrics=0 emb=0); complexity returns `{"complexity":null,"unavailable":true,"reason":"metrics_sunset"}` (A18); capabilities mark metrics/semantic `available:false`+`unavailable_reason:"sunset"`; A19 guard (vector_ready=0 when node_vectors empty); metrics_ready structurally 0; FTS search works; corruption (insert→emb=1, drop→emb=0); A19 full cycle (re-index resets vector_ready=0); stale-flag regression (manual vector_ready=1→0 on no-op re-index) | -| 2026-07-31 | Step 10 | full `make test-engine` suite (53 tests) | ✅ 53/53 pass (test_verifier_lifecycle transient stale-DB failure passes in isolation) | -| 2026-07-31 | Step 2 | `make accuracy-check` | ✅ baseline P=1.0 R=1.0 F1=1.0 (TP=17 FP=0 FN=0); FP injection → exit 1 (P=0.708); FN injection → exit 1 (R=0.588); gate catches both fault types | -| 2026-07-31 | Step 2 | `test_call_graph_accuracy` | ✅ 7-language fixtures (cpp/go/python/rust/java/js/ts); deterministic across 3 runs; fixtures copied to /tmp to bypass FilterPolicy "tests" skip-dir | -| 2026-07-31 | Step 2 | `test_homonym_filter` | ✅ portable local fixture (2 Go files with same-name `handler`); file_filter disambiguates (no-filter=2, first.go=1, second.go=1); returns nonzero on failure | -| 2026-07-31 | Step 2 | clang-format --dry-run --Werror | ✅ clean on test_call_graph_accuracy.cpp and test_homonym_filter.cpp | - -## Decisions Log - -- 2026-07-31: Parallel agent topology — Track A (Steps 0-8 + 11, call graph accuracy) is sequential due to schema dependencies; Track B (Step 9 verifier) and Track C (Step 10 metrics) run in parallel with Track A once Step 0 contract is frozen. -- 2026-07-31: Step 0 changes the Graph Compiler split (was `rtype >= 4 → RELATES`, now `isCallsEdge(rtype) → CALLS else RELATES`). User-visible query results unchanged because `getCallers/getCallees` still match `CALLS|RELATES` (the union is preserved). Step 1 will tighten the query to `CALLS only` with `edge_type=1` filter. -- 2026-07-31: Legacy `graph_edges.edge_type` numbering (1=call_graph, 3=symbol_reference) is left untouched — it's the deprecated fallback path and uses different semantics from the canonical `relation.type` contract. A clarifying comment was added. -- 2026-07-31 (Step 1): `getCallers`/`getCallees` Cypher changed from `CALLS|RELATES` to `CALLS` only with defensive `r.edge_type = 1` filter. The filter is redundant with the Step 0 Graph Compiler split but guards against stale `.lbug` files compiled by older binaries. -- 2026-07-31 (Step 1): Added `UNIQUE(project_id, source_id, target_id, type)` index on `relation` with a dedup migration (`DELETE ... WHERE id NOT IN (SELECT MIN(id) ... GROUP BY ...)`), mirroring the existing `graph_edges` unique-index pattern. `INSERT OR IGNORE INTO relation` now actually deduplicates. -- 2026-07-31 (Step 1): Bumped Ladybug schema version 2→3 to force a full `.lbug` recompile so stale non-Calls edges written by pre-Step-0 binaries are purged. -- 2026-07-31 (Step 1): Added defensive result-layer dedup in `getCallers`/`getCallees` keyed on `node_id|file_path|start_row` so stale duplicate CALLS edges from pre-migration `.lbug` files collapse to a single entry. -- 2026-07-31: Step 9 lifecycle fix (A15) — replaced the `static bool initialized` flag in `engine_verify_ffi.cpp` with `VerifierRegistry::ensureDefaultVerifiers()`, which checks the actual registry state (not a process-level flag) and is idempotent. This makes `engine_shutdown()` → `engine_init()` → verify symmetric: shutdown clears the registry, and the next verify call re-populates it. Sentinel verifiers use nullptr/0 for store/pid because `accepts()` only inspects `claim.type`; the actual `verify()` dispatch constructs a fresh project-bound verifier via `makeVerifierForClaim`, avoiding cross-project state leaks. -- 2026-07-31: Step 9 `FunctionImplements` product decision — implemented a dedicated `FunctionImplementsVerifier` (not a CapabilityVerifier fallback). It reads canonical `entity` (kind 0/1) + `relation` (type=1 Calls) tables: Supported when the function exists and participates in the call graph, Contradicted when absent, Unknown when isolated or evidence backend not ready. This closes the "factory has fallback but registry never matches" gap. -- 2026-07-31: Step 9 evidence migration (A17) — all four verifiers (Capability, Contract, Architecture, FunctionImplements) now read `entity`/`relation` as the production source of truth. No `FROM graph_nodes` / `FROM graph_edges` SQL remains in verifier source; only migration-comments reference the legacy names. `evidence_backend_ready()` gates verifiers on `entity > 0 AND relation > 0` so they return Unknown + reason instead of fabricating verdicts from empty tables. -- 2026-07-31: Step 9 error code unification (A21) — `verify_claim` now emits machine-readable `error_code` fields: `registry_empty` (verifier_count == 0), `claim_type_unsupported` (no verifier accepts the type, or unknown type string), `evidence_backend_not_ready` (entity/relation empty), `verifier_execution_failed` (verifier threw). Callers can distinguish a broken subsystem from a normal Unknown verdict. `parseClaimType` returns `std::optional` so unknown type strings surface an input error instead of silently falling back to `CapabilityExists`. -- 2026-07-31: Step 10 ADR — **SUNSET** metrics + embedding/semantic search for the current sprint; **KEEP** FTS. Rationale: the metrics producer (`resolveStagedMetrics`) and the embedding builder (`buildVectorsFromGraph`) are no-ops with no real implementation behind them; returning placeholder `0`/empty values (A18) and setting `vector_ready=1` unconditionally in DEEP mode (A19) masqueraded as working features. Sunset means these capabilities are now explicitly marked `available:false`/`unavailable_reason:"sunset"` in `engine_get_capabilities`, complexity APIs return a structured `{"complexity":null,"unavailable":true}` marker, and misleading "run codescope_enhance to enable" descriptions were removed. FTS5 remains the only supported search path. This is reversible: re-enabling metrics or embedding later only requires implementing the producer and flipping the `available` flag + removing the sunset reason. -- 2026-07-31: Step 10 readiness-vs-canonical-data invariant (A19/A20) — `vector_ready` is now conditional on `node_vectors` row count > 0 (was unconditionally 1 in DEEP mode); `metrics_ready` is structurally 0 (producer sunset, `markCallgraphAndMetricsReady` no longer sets it); `embedding_ready` in `engine_get_enhancement_status` reads the canonical `node_vectors` count directly (was hardcoded `SELECT 0,0,0`). A no-op re-index (all files unchanged) now also refreshes `vector_ready` so a stale flag cannot persist when the early-return path skips `engine_index_post_parse`. `test_metrics_readiness` guards both directions: a stale `vector_ready=1` is reset to 0 on re-index, and an externally-inserted vector row raises `embedding_ready` to 1 (then drops to 0 on delete). -- 2026-07-31 (Step 2): Accuracy fixtures live under `engine/tests/accuracy/fixtures//`. The engine FilterPolicy's `normal_skip_dirs_` includes `"tests"`, so indexing the fixture path in-place yields zero entities. The runner (`test_call_graph_accuracy.cpp`) copies fixture source files to `/tmp/codescope_acc_src_/` before indexing, bypassing the skip filter. `ground_truth.json` is NOT copied (it is not source code). -- 2026-07-31 (Step 2): Constructor calls (`new B()`, `Timeline()`, `new Renderer()`) produce CALLS edges to the class entity (kind=2). These are legitimate call edges, so they were added to `expected_calls` in the Java/Python/TS ground truth — this completes the ground truth rather than hiding a defect. The plan explicitly requires fixtures to cover "constructor/static/virtual/interface dispatch". -- 2026-07-31 (Step 2): FN fault injection removes an expected edge from the ACTUAL set (not the EXPECTED set). Removing from expected would turn a TP into an FP (precision drops, not recall). Removing from actual creates a true FN (expected but missing → recall drops). FP injection adds a fake edge to actual (precision drops). Both cause nonzero exit. -- 2026-07-31 (Step 2): `test_homonym_filter` replaced with a portable local fixture (2 Go files, same-name `handler` function). The old test depended on `/Users/scc/code/pycode/Transformer_Explorer` and returned 0 even on failure (A10). The new test returns nonzero on failure and verifies file_filter disambiguation: without filter both helpers appear, with filter only the filtered file's helper appears. -- 2026-07-31 (Step 2): `test_call_graph_accuracy` added to `TEST_EXES` (baseline passes: 0 FP, 0 FN). `make accuracy-check` is a separate target that runs the baseline + FP injection + FN injection and verifies the gate catches both fault types. diff --git a/docs/REAL_PROJECT_TOOL_REPORT_2026-09-21.md b/docs/REAL_PROJECT_TOOL_REPORT_2026-09-21.md new file mode 100644 index 0000000..0a83107 --- /dev/null +++ b/docs/REAL_PROJECT_TOOL_REPORT_2026-09-21.md @@ -0,0 +1,186 @@ +# Real-project MCP tool test — CodeScope + goagent + +Date: 2026-09-21 · revision: `5a25b67` (working tree carries the uncommitted fixes below) · binary: `target/release/codescope` + +## What was run + +Both MCP sessions are real: a server subprocess on stdio, `initialize` (which indexes the project), `tools/list`, then one `tools/call` per tool in `tools/list` order, all in a single long-lived session. Arguments are harvested from the freshly built index (a real symbol name, entity id, file path, module name) and passed as each schema requires. + +| | CodeScope (self) | goagent | +|---|---|---| +| root | `/Users/scc/code/cppCode/CodeScope` | `/Users/scc/go/src/goagent` | +| source files (CLI index) | 250 | 1594 | +| index time via CLI `index-parallel -w4` | 0.93 s | 4.87 s | +| nodes after `initialize` (MCP session) | 1788 | 24987 | +| `initialize` wall time | 0.8 s | 3.8 s | +| tools listed / called | 46 / 46 | 46 / 46 | +| problematic results | 0 | 0 | + +## Per-tool results + +`status` classifies the JSON-RPC + tool envelope: `OK` = a payload came back with no `error`/`isError`. Latency is the round trip in that session. + +| tool | CodeScope | goagent | payload (self) | payload (goagent) | +|---|---|---|---|---| +| `build_evidence` | OK / 1ms | OK / 1ms | keys=[] len=4958 | keys=[] len=4078 | +| `build_project_state` | OK / 2ms | OK / 2ms | keys=['overall', 'capability', 'architecture', 'workflow', | keys=['overall', 'capability', 'architecture', 'workflow', | +| `codescope_trace` | OK / 0ms | OK / 1ms | keys=['name', 'file', 'line', 'callers', 'callees'] len=17 | keys=['name', 'file', 'line', 'callers', 'callees'] len=63 | +| `connected_components` | OK / 1ms | OK / 2ms | keys=['components', 'total', 'approximation', 'note'] len= | keys=['components', 'total', 'approximation', 'note'] len= | +| `count_tokens` | OK / 0ms | OK / 0ms | keys=['chars_ascii', 'chars_non_ascii', 'chars_total', 'me | keys=['chars_ascii', 'chars_non_ascii', 'chars_total', 'me | +| `detect_architecture_drift` | OK / 4ms | OK / 18ms | keys=['drifts', 'drifts_found'] len=30 | keys=['drifts', 'drifts_found'] len=30 | +| `detect_capability_drift` | OK / 2ms | OK / 47ms | keys=['total_capabilities', 'drifts', 'drifts_found'] len= | keys=['total_capabilities', 'drifts', 'drifts_found'] len= | +| `detect_changes` | OK / 0ms | OK / 0ms | keys=['error', 'modified', 'callers', 'callees', 'total_im | keys=['error', 'modified', 'callers', 'callees', 'total_im | +| `detect_documentation_drift` | OK / 2ms | OK / 7ms | keys=['claimed_languages', 'found_languages', 'missing_lan | keys=['claimed_languages', 'found_languages', 'missing_lan | +| `detect_drift` | OK / 0ms | OK / 1ms | keys=['drifts', 'drifts_found'] len=503 | keys=['drifts', 'drifts_found'] len=1522 | +| `detect_ffi_boundaries` | OK / 3ms | OK / 18ms | keys=['languages', 'cross_language_files', 'ffi_symbols', | keys=['languages', 'cross_language_files', 'ffi_symbols', | +| `enhance_project` | OK / 420ms | OK / 529ms | keys=['status', 'time_ms'] len=29 | keys=['status', 'time_ms'] len=29 | +| `explain_module` | OK / 0ms | OK / 3ms | keys=['module', 'summary', 'entities', 'capabilities', 'co | keys=['module', 'summary', 'entities', 'capabilities', 'co | +| `explain_symbol` | OK / 1ms | OK / 6ms | keys=['symbol', 'definition', 'callers', 'callees'] len=10 | keys=['symbol', 'definition', 'callers', 'callees'] len=90 | +| `find_callees` | OK / 0ms | OK / 2ms | keys=['callees', 'total', 'ambiguous', 'candidates'] len=3 | keys=['callees', 'total', 'ambiguous', 'candidates'] len=2 | +| `find_callees_by_entity` | OK / 0ms | OK / 0ms | keys=['callees', 'total', 'entity_id'] len=38 | keys=['callees', 'total', 'entity_id'] len=254 | +| `find_callers` | OK / 0ms | OK / 3ms | keys=['callers', 'total', 'ambiguous', 'candidates'] len=3 | keys=['callers', 'total', 'ambiguous', 'candidates'] len=2 | +| `find_callers_by_entity` | OK / 0ms | OK / 0ms | keys=['callers', 'total', 'entity_id'] len=38 | keys=['callers', 'total', 'entity_id'] len=40 | +| `find_definition` | OK / 0ms | OK / 3ms | keys=['results', 'total'] len=4362 | keys=['results', 'total'] len=3862 | +| `find_references` | OK / 0ms | OK / 3ms | keys=['results', 'total'] len=24 | keys=['results', 'total'] len=23870 | +| `find_symbol` | OK / 0ms | OK / 0ms | keys=['results'] len=3503 | keys=['results'] len=3016 | +| `force_index_files` | OK / 470ms | OK / 6135ms | keys=['discovery', 'files_indexed', 'ok', 'paths_requested | keys=['discovery', 'files_indexed', 'ok', 'paths_requested | +| `get_communities` | OK / 1ms | OK / 7ms | keys=['communities', 'total_communities', 'returned_commun | keys=['communities', 'total_communities', 'returned_commun | +| `get_entry_points` | OK / 0ms | OK / 2ms | keys=['entry_points', 'total'] len=3987 | keys=['entry_points', 'total'] len=12049 | +| `get_graph` | OK / 18ms | OK / 62ms | keys=['totals', 'nodes', 'edges', 'has_more'] len=970078 | keys=['totals', 'nodes', 'edges', 'has_more'] len=3158682 | +| `get_graph_stats` | OK / 64ms | OK / 326ms | keys=['total_nodes', 'total_edges', 'total_files'] len=57 | keys=['total_nodes', 'total_edges', 'total_files'] len=58 | +| `get_knowledge_graph` | OK / 0ms | OK / 1ms | keys=['table', 'rows', 'total', 'truncated'] len=19041 | keys=['table', 'rows', 'total', 'truncated'] len=16143 | +| `get_module_tree` | OK / 0ms | OK / 0ms | keys=['modules'] len=4810 | keys=['modules'] len=17567 | +| `get_neighbors` | OK / 0ms | OK / 0ms | keys=['radius_applied', 'radius_requested', 'note', 'neigh | keys=['radius_applied', 'radius_requested', 'note', 'neigh | +| `get_project_state` | OK / 0ms | OK / 0ms | keys=['overall', 'capability', 'architecture', 'workflow', | keys=['overall', 'capability', 'architecture', 'workflow', | +| `get_routes` | OK / 0ms | OK / 0ms | keys=['routes'] len=13 | keys=['routes'] len=6139 | +| `get_subgraph` | OK / 0ms | OK / 1ms | keys=['nodes', 'total'] len=196 | keys=['nodes', 'total'] len=8337 | +| `get_type_info` | OK / 0ms | OK / 2ms | keys=['types'] len=18792 | keys=['types'] len=17566 | +| `get_verifier_registry_status` | OK / 0ms | OK / 0ms | keys=['registry_empty', 'verifier_count', 'verifier_names' | keys=['registry_empty', 'verifier_count', 'verifier_names' | +| `graph_query` | OK / 121ms | OK / 2087ms | keys=['results', 'total'] len=500476 | keys=['results', 'total'] len=2027087 | +| `index_file` | OK / 50ms | OK / 218ms | keys=['ok', 'nodes', 'edges'] len=31 | keys=['ok', 'nodes', 'edges'] len=31 | +| `project_overview` | OK / 0ms | OK / 2ms | keys=['languages', 'total_modules', 'total_symbols', 'anal | keys=['languages', 'total_modules', 'total_symbols', 'anal | +| `search` | OK / 0ms | OK / 1ms | keys=['method', 'results'] len=1440 | keys=['method', 'results'] len=1329 | +| `search_code` | OK / 0ms | OK / 0ms | keys=['results', 'total'] len=2215 | keys=['results', 'total'] len=2084 | +| `shortest_path` | OK / 0ms | OK / 2ms | keys=['path', 'found', 'approximation', 'note', 'hops'] le | keys=['path', 'found', 'approximation', 'note', 'hops'] le | +| `trace_flow` | OK / 0ms | OK / 0ms | keys=['name', 'file', 'line', 'callees'] len=157 | keys=['name', 'file', 'line', 'callees'] len=1232 | +| `verify_claim` | OK / 0ms | OK / 1ms | keys=['claim_id', 'verdict', 'confidence', 'verifier', 'de | keys=['claim_id', 'verdict', 'confidence', 'verifier', 'de | +| `verify_integrity` | OK / 10016ms | OK / 10002ms | keys=['findings', 'total', 'trust_score', 'supported', 'co | keys=['findings', 'total', 'trust_score', 'supported', 'co | +| `verify_reality` | OK / 0ms | OK / 0ms | keys=['statement', 'claims_parsed', 'verdict', 'confidence | keys=['statement', 'claims_parsed', 'verdict', 'confidence | +| `verify_review` | OK / 0ms | OK / 0ms | keys=['claims_parsed', 'results', 'summary'] len=105 | keys=['claims_parsed', 'results', 'summary'] len=105 | +| `verify_summary` | OK / 8ms | OK / 71ms | keys=['claims_parsed', 'results', 'drifts', 'summary'] len | keys=['claims_parsed', 'results', 'drifts', 'summary'] len | + +## What worked, on real data + +- `detect_changes` (after the fix below) resolves real symbols: `engine/src/store/store.h` returns `GraphStore`, `QueryDeadlineGuard`, `BulkPragmaGuard`, `dbPath`, `handle`, `error` with their callers. +- `build_evidence` surfaced a TODO marker from goagent's own source (`// TODO(tech-debt): the separate output.LLMAdapter assembly …`) and the markers in CodeScope. +- `verify_claim` returns registry verdicts naming the verifier (`CapabilityVerifier`) and the reason (`Capability 'engine' not declared in knowledge layer`). +- `explain_symbol` returned 10.5 KB / 9.1 KB of real definitions+callers+callees; `get_module_tree` 17.6 KB for goagent; `get_entry_points` 12 KB; `get_type_info` 18.8 KB. +- `search` reports which method answered, and `find_callers`/`find_callees` report `ambiguous` + `candidates`, i.e. they abstain rather than guess. + +## Findings + +### 1. `index-parallel`'s merged DB cannot be used as a project (HIGH) + +The CLI index path produces a standalone `{prefix}_main.db` in which entities are merged but + +- the `projects` table is **empty** (no row for the indexed root), and +- the rows keep the **per-module project_id** (`1, 2, …`); they are not unified. + +Measured: CodeScope 1788 entities split as project 1 → 1346, project 2 → 442; goagent 24987 split as 1 → 22517, 2 → 1298, 3 → 969, 4 → 160, 5 → …. Consequences: a tool called with `project_id=1` sees only the first module (1346 of 1788), and the MCP server cannot adopt the DB at all — `get_project_id_by_path` finds nothing, so `initialize` re-indexes from scratch. That is why the sessions here index themselves and report complete counts. + +Evidence: `sqlite3 _main.db "SELECT project_id, COUNT(*) FROM entity GROUP BY project_id"`, and `SELECT * FROM projects` returning 0 rows. + +### 2. `detect_changes` silently meant "no changes" for the shape its schema advertises (FIXED) + +The schema declares `modified_files` as a `string` ("JSON array of modified file paths"), so a caller passing a path sent a bare path — invalid JSON — and the engine parsed nothing: `{"modified": [], "total_impacted": 0}` with no error. The engine's parameter is actually `modified_files_json` (an array). The handler now accepts a real array and still takes a pre-encoded string verbatim. Verified with real data: `store.h` now returns its entities and callers. + +Caller note: paths must match the stored form (`file_path` is absolute in the index). + +### 3. The first knowledge-dependent call blocks ~10.0 s on both projects + +`verify_integrity` took **10.016 s** (CodeScope, whose whole index took 0.93 s) and **10.002 s** (goagent, 4.87 s) — the same ~10 s for very different sizes, which points at a fixed cost rather than proportional work. I verified what it is *not*: `joinAsyncKnowledgeBuilder` is a plain `thread::join()` with no timeout, and the `10000` at `engine_verify_ffi.cpp:387` is a query deadline guard. The time is spent in the background knowledge-builder thread; I found no pacing constant, so this is reported as **measured and unexplained** rather than diagnosed. Every tool that calls `waitForKnowledgeBuilder()` inherits it once per session — later calls are fast. + +### 4. `project_overview` contradicts itself about readiness + +In one payload, `analysis_progress` reports every entity through all stages (`scanned = callgraph = metrics = 1480` for CodeScope, `6267` for goagent) while `ready_features` reports `call_graph: false, metrics: false, semantic_search: false` on **both** projects. `ready_features` comes from `getReadyRatio(project_id, "_ready") > 0.5` (`engine_queries.cpp:772`), so the two fields read different sources and one of them is wrong for a fully indexed project. Not fixed — I did not establish which one is right. + +### 5. `get_graph_stats.total_files` does not mean "files indexed" + +For goagent it reports **685** while 1594 files were indexed (the CLI and the session agree on 1594). Node counts match, so this is a definitional mismatch — presumably "files that have graph nodes". Read as "how much of my project is indexed" it understates by more than a factor of two. + +### 6. Response sizes worth knowing about + +`get_graph` returned 3.16 MB (goagent) and 0.97 MB (CodeScope) in a single MCP message; `graph_query` returned 2.03 MB in 2.1 s. No pagination failure was hit here, but a client with a message-size cap will fail on these. + +## Disclosure + +The first run reported three tool failures (`shortest_path`, `codescope_trace`, `graph_query`). They were **the test driver's** fault: it nested the per-tool extras under the tool name (`{"graph_query": {...}}`) instead of merging them, so those tools received no usable arguments. After fixing the driver both projects ran 46/46 clean. No product defect was involved — the three tools work. + +## Reproduction + +``` +# index (CLI; see finding 1 for why the merged DB is not adoptable) +CODESCOPE_DB_PATH=/tmp/cs_real_test.db CODESCOPE_DB_PREFIX=/tmp/csrt_self \ + ./target/release/codescope index-parallel --workers 4 --parallel 4 + +# drive every tool over a real MCP session +python3 /tmp/mcp_drive.py self # /tmp/mcp_drive.py goa +``` + +Raw material: `/tmp/mcp_report/{codescope_self,goagent}.json` (per-call args, latency, status, raw head) and the matching `.stderr.log` files. + + +## Fixes applied after this run (batch 23) + +**#1 (the one that matters for node purity) — FIXED and verified on both projects.** +`merge_driver::unify_project` now runs at the end of every merge: it discovers the **39** tables that carry a +`project_id` (discovery, not a hardcoded list — a hardcoded one would have missed 25 of them) and rewrites them onto a +single project id, then writes the `projects` row for the indexed directory, creating the table with the engine's own DDL +if the merge did not carry it. + +| check | before | after | +|---|---|---| +| `projects` rows | 0 | **1** — `(1, '', 'CodeScope')` | +| tables with more than one `project_id` | 39 of 39 possible | **none** | +| entities reachable as one project | 1346 of 1788 (project 1) | **1793 of 1793** | +| `initialize` on that DB | re-indexed from scratch (0.8 s) | **0.01 s**, log says `Reusing existing project CodeScope (id=1, has data)` | +| entities / DB size across a session | — | **unchanged** (1793 → 1793, same byte size: no duplication) | +| a `server/` symbol (`index_parallel`) | invisible from project 1 | found | + +**#6 (search) — FIXED.** A scheduler-built DB never received the engine's post-index pass, so `code_fts`, `name_trgm`, +`fts_node_map` and `node_vectors` were empty and `search` silently answered from its graph fallback — which finds classes +but not functions. The CLI's index branch now opens the merged DB and runs the pass while it owns it: `project_readiness` +0 → 1 row, `code_fts` 0 → 1793, `name_trgm` 0 → 1793, and `search` for `index_parallel` / `is_skip_dir` (previously +unfindable) now reports `method: "legacy_fts"` with real hits. Cost: CodeScope's CLI index 0.93 s → 2.34 s. + +## Refined diagnoses after the fixes + +Updated 2026-09-30: **all four findings below are closed.** Each entry keeps the +diagnosis and records what fixed it. + +- **#3 (the ~10 s first call)** — that call was the post-index knowledge build, + not a fixed timeout. Resolved by moving the build into indexing: the CLI's + index branch now opens the merged DB and runs the engine's post-index pass + while it owns it, so a session that adopts the database has nothing left to + build. Measured on goagent in a fresh session: `explain_module` 0.13 s, + `get_knowledge_graph` 0.04 s. +- **#4 (`ready_features` false) — cause pinned, then fixed.** `getReadyRatio` + computed `SUM(gn._ready)/COUNT(*)` **FROM `graph_nodes`**, empty in + every DB the scheduler produces, so the ratio was 0.0 by construction. It now + reads the canonical tables (`entity`/`relation`/`node_vectors`) with the same + SQL as `engine_get_enhancement_status`, and `build_context`'s + `sample_call_edges` reads `relation`/`entity` instead of the empty + `graph_edges`. Measured on goagent: `ready_features.call_graph` false → true. + Regression test: `engine/tests/test_readiness_canonical.cpp`. +- **#5 (`get_graph_stats.total_files`)** — it now counts the `files` table, i.e. + files indexed (goagent: 672 → 1,579), and reports the previous number as + `files_with_symbols`. +- **#6 (response sizes)** — the 1 MiB transport cap is documented (`README.md` / + `README.zh.md` §5 and the tool descriptions) and `graph_query` honours its + documented `LIMIT`: the broad call-graph pattern that produced 2.16 MB now + returns 56 KB with `LIMIT 200`. + +## Verification after the fixes + +All 46 tools were re-run against both projects over fresh real sessions: **46/46 clean on CodeScope and 46/46 on +goagent**, with no regression in the accuracy gate (TP 36 / FP 0 / FN 0), 108 server tests and 80 engine tests. diff --git a/docs/articles/en/codescope-deepdive-01-dual-language.md b/docs/articles/en/codescope-deepdive-01-dual-language.md index fc99de9..a984e52 100644 --- a/docs/articles/en/codescope-deepdive-01-dual-language.md +++ b/docs/articles/en/codescope-deepdive-01-dual-language.md @@ -1,7 +1,6 @@ # CodeScope Deep Dive (1): Dual-Language Architecture — Why We Wrote a C++ Project in Rust > *"Sometimes the best tool for the job isn't the one you'd like to use."* -> 有时最好的工具并不是你喜欢的那个。 ## Why Two Languages? diff --git a/docs/articles/en/codescope-deepdive-02-worker-isolation.md b/docs/articles/en/codescope-deepdive-02-worker-isolation.md index 247b365..b29948d 100644 --- a/docs/articles/en/codescope-deepdive-02-worker-isolation.md +++ b/docs/articles/en/codescope-deepdive-02-worker-isolation.md @@ -1,7 +1,6 @@ # CodeScope Deep Dive (2): Worker Subprocess Isolation — Why Indexing Must Run in Subprocesses > *"The only way to make a C++ program crash-proof is to run it in a separate process."* -> 让 C++ 程序不崩溃的唯一方法，是把它跑在单独的进程里。 ## Starting with an OOM Crash diff --git a/docs/articles/en/codescope-deepdive-05-parser-ir.md b/docs/articles/en/codescope-deepdive-05-parser-ir.md index f74c975..dee8fbe 100644 --- a/docs/articles/en/codescope-deepdive-05-parser-ir.md +++ b/docs/articles/en/codescope-deepdive-05-parser-ir.md @@ -1,7 +1,6 @@ # CodeScope Deep Dive (5): tree-sitter Parser & Unified IR — 8 Languages, One Model > *"A programming language is a tool. Code analysis should not care which tool you used."* -> 编程语言是工具。代码分析不应该关心你用了哪个工具。 ## The Problem: 8 Languages, One Format diff --git a/docs/articles/en/codescope-deepdive-06-sqlite-storage.md b/docs/articles/en/codescope-deepdive-06-sqlite-storage.md index 17261f5..27f93c0 100644 --- a/docs/articles/en/codescope-deepdive-06-sqlite-storage.md +++ b/docs/articles/en/codescope-deepdive-06-sqlite-storage.md @@ -1,7 +1,14 @@ # CodeScope Deep Dive (6): SQLite Graph Storage — Schema Design & FTS > *"A graph database is a great idea. A graph database that requires a server process is a deployment nightmare."* -> 图数据库是个好主意。但需要启动一个服务进程的图数据库，就是部署噩梦了。 + +> **Historical note (2026-10-01)**: the storage layer moved on after this +> article. The canonical tables are `entity` (symbols) and `relation` (edges), +> `graph_nodes` is no longer populated, and full-text search lives in `code_fts` +> + `name_trgm`. See `docs/en/technical_breakdown.md` §4.1 for the current +> schema. LadybugDB, described further down as an optional acceleration +> layer, was removed from the product as well — that section is history, not +> a supported path. The design reasoning still holds. ## The Problem: Where to Store the Code Graph? diff --git a/docs/articles/en/codescope-deepdive-07-verification-pipeline.md b/docs/articles/en/codescope-deepdive-07-verification-pipeline.md index ad4fe6e..fc9c6ad 100644 --- a/docs/articles/en/codescope-deepdive-07-verification-pipeline.md +++ b/docs/articles/en/codescope-deepdive-07-verification-pipeline.md @@ -1,7 +1,13 @@ # CodeScope Deep Dive (7): Verification Pipeline — Claim → Evidence → Verdict > *"The most dangerous assumption in software engineering is that the code matches the docs."* -> 软件工程中最危险的假设，就是认为代码和文档是一致的。 + +> **Historical note (2026-10-01)**: the storage layer moved on after this +> article. The canonical tables are `entity` (symbols) and `relation` (edges), +> `graph_nodes` is no longer populated, and full-text search lives in `code_fts` +> + `name_trgm`. See `docs/en/technical_breakdown.md` §4.1 for the current +> schema. LadybugDB was removed from the product as well; where the text below +> names it, it is history rather than a supported path. ## The Problem: How to Keep Code and Docs in Sync? diff --git a/docs/articles/en/codescope-deepdive-08-drift-detection.md b/docs/articles/en/codescope-deepdive-08-drift-detection.md index 3ecb5d5..2aa39e0 100644 --- a/docs/articles/en/codescope-deepdive-08-drift-detection.md +++ b/docs/articles/en/codescope-deepdive-08-drift-detection.md @@ -1,7 +1,6 @@ # CodeScope Deep Dive (8): Drift Detection — The Trinity of Documentation, Capability, and Architecture > *"Documentation is a love letter to your future self. Drift is the heartbreak when you realize it was never updated."* -> 文档是写给未来自己的情书。漂移是当你发现它从未被更新时的心碎。 ## The Problem: When Does Documentation Start to Diverge from Code? diff --git a/docs/articles/zh/codescope-deepdive-01-dual-language.md b/docs/articles/zh/codescope-deepdive-01-dual-language.md index 449c740..f152529 100644 --- a/docs/articles/zh/codescope-deepdive-01-dual-language.md +++ b/docs/articles/zh/codescope-deepdive-01-dual-language.md @@ -1,6 +1,5 @@ # CodeScope 拆解 (一)：双语言架构 — 为什么我们用 Rust 写了一个 C++ 项目 -> *"Sometimes the best tool for the job isn't the one you'd like to use."* > 有时最好的工具并不是你喜欢的那个。 ## 为什么是两种语言？ diff --git a/docs/articles/zh/codescope-deepdive-02-worker-isolation.md b/docs/articles/zh/codescope-deepdive-02-worker-isolation.md index dd5e65b..b252ff0 100644 --- a/docs/articles/zh/codescope-deepdive-02-worker-isolation.md +++ b/docs/articles/zh/codescope-deepdive-02-worker-isolation.md @@ -1,6 +1,5 @@ # CodeScope 拆解 (二)：Worker 子进程隔离 — 为什么索引必须跑在子进程里 -> *"The only way to make a C++ program crash-proof is to run it in a separate process."* > 让 C++ 程序不崩溃的唯一方法，是把它跑在单独的进程里。 ## 从 OOM 崩溃说起 diff --git a/docs/articles/zh/codescope-deepdive-05-parser-ir.md b/docs/articles/zh/codescope-deepdive-05-parser-ir.md index abe989a..d3c3477 100644 --- a/docs/articles/zh/codescope-deepdive-05-parser-ir.md +++ b/docs/articles/zh/codescope-deepdive-05-parser-ir.md @@ -1,6 +1,5 @@ # CodeScope 拆解 (五)：tree-sitter 解析器与统一 IR — 8 种语言一个模型 -> *"A programming language is a tool. Code analysis should not care which tool you used."* > 编程语言是工具。代码分析不应该关心你用了哪个工具。 ## 问题：8 种语言，一个格式 diff --git a/docs/articles/zh/codescope-deepdive-06-sqlite-storage.md b/docs/articles/zh/codescope-deepdive-06-sqlite-storage.md index d978fd4..0b93453 100644 --- a/docs/articles/zh/codescope-deepdive-06-sqlite-storage.md +++ b/docs/articles/zh/codescope-deepdive-06-sqlite-storage.md @@ -1,8 +1,12 @@ # CodeScope 拆解 (六)：SQLite 图谱存储 — Schema 设计与 FTS -> *"A graph database is a great idea. A graph database that requires a server process is a deployment nightmare."* > 图数据库是个好主意。但需要启动一个服务进程的图数据库，就是部署噩梦了。 +> **历史注记（2026-10-01）**：本文写作之后存储层已经演进。权威表是 `entity`（符号）与 +> `relation`（边），`graph_nodes` 不再写入，全文检索由 `code_fts` + `name_trgm` 承担。 +> 当前 schema 见 `docs/zh/technical_breakdown.md` §4.1。文中作为「可选加速层」介绍的 +> LadybugDB 也已从产品中移除 —— 那一节是历史，而非可用路径。下文的其余设计思路仍然成立。 + ## 问题：代码图存哪里？ CodeScope 的核心输出是**代码符号图**：节点是函数、类、变量，边是调用、引用、继承关系。这个图必须持久化，支持后续查询。 diff --git a/docs/articles/zh/codescope-deepdive-07-verification-pipeline.md b/docs/articles/zh/codescope-deepdive-07-verification-pipeline.md index 3e038aa..9b4e039 100644 --- a/docs/articles/zh/codescope-deepdive-07-verification-pipeline.md +++ b/docs/articles/zh/codescope-deepdive-07-verification-pipeline.md @@ -1,8 +1,12 @@ # CodeScope 拆解 (七)：验证管线 — Claim → Evidence → Verdict -> *"The most dangerous assumption in software engineering is that the code matches the docs."* > 软件工程中最危险的假设，就是认为代码和文档是一致的。 +> **历史注记（2026-10-01）**：本文写作之后存储层已经演进。权威表是 `entity`（符号）与 +> `relation`（边），`graph_nodes` 不再写入，全文检索由 `code_fts` + `name_trgm` 承担。 +> 当前 schema 见 `docs/zh/technical_breakdown.md` §4.1。LadybugDB 也已从产品中移除， +> 下文提到它的地方是历史，而非可用路径。 + ## 问题：代码和文档怎么对得上？ 当项目增长到一定规模，代码和文档之间必然产生偏差。README 里说"支持增量索引"，但代码里可能只实现了全量重建。API 文档说"线程安全"，但实现里全是裸锁。 diff --git a/docs/articles/zh/codescope-deepdive-08-drift-detection.md b/docs/articles/zh/codescope-deepdive-08-drift-detection.md index 94bc80b..1de9143 100644 --- a/docs/articles/zh/codescope-deepdive-08-drift-detection.md +++ b/docs/articles/zh/codescope-deepdive-08-drift-detection.md @@ -1,6 +1,5 @@ # CodeScope 拆解 (八)：漂移检测 — 文档、能力、架构的三位一体 -> *"Documentation is a love letter to your future self. Drift is the heartbreak when you realize it was never updated."* > 文档是写给未来自己的情书。漂移是当你发现它从未被更新时的心碎。 ## 问题：文档什么时候开始偏离代码？ diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md deleted file mode 100644 index ffcd7ab..0000000 --- a/docs/benchmarks/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# CodeScope Benchmarks Archive - -This directory contains performance benchmark reports and project analysis results. - -## Document Structure - -### Chinese Reports (Original) -- `large_project_index_report.md` - Large project indexing benchmarks (rustc, Bun, JDK, Linux kernel) -- `timbre_model_analysis.md` - InstrumentTimbre model architecture analysis -- `linux-kernel.md` - Linux kernel benchmark results (English) - -### English Translations -All English versions are available in `../en/` directory: -- `en/large_project_index_report.md` - English translation -- `en/timbre_model_analysis.md` - English translation -- `en/linux-analysis.md` - Linux kernel analysis (English) - -## Benchmark Coverage - -| Project | Files | Language | Index Time | Nodes | Status | -|---------|-------|----------|------------|-------|--------| -| rustc | 36,807 | Rust | 1m44s | 4.13M | ✅ | -| Bun | 9,641 | Zig/C++/JS | 1m1s | 2.95M | ✅ | -| JDK | 19,821 | Java | 3m31s | 8.05M | ✅ | -| Linux kernel | 60,468 | C | ~300s (timeout) | — | ⚠️ | -| InstrumentTimbre | 142 | Python | 0.94s | — | ✅ | - -## Performance Insights - -**Key Findings:** -1. Parse time accounts for only 1-3% of total indexing time -2. SQLite write is the main bottleneck (30-45%) -3. buildGraph phase takes 29-38% of time -4. FTS indexing adds 19-29% overhead -5. Token savings: ~98.9% vs raw source files, ~85% vs codebase-memory-mcp - -**Throughput:** -- Peak: 11,523 files/s (rustc) -- Typical: 150-350 files/s (medium projects) -- Lower: 93 files/s (JDK - high node density) - ---- - -For detailed analysis, refer to individual reports in this directory or their English translations in `../en/`. \ No newline at end of file diff --git a/docs/benchmarks/linux-kernel.md b/docs/benchmarks/linux-kernel.md deleted file mode 100644 index ce7b2d2..0000000 --- a/docs/benchmarks/linux-kernel.md +++ /dev/null @@ -1,32 +0,0 @@ -# Linux Kernel Benchmark - -## Environment -- **Machine**: Mac15,11 (Apple Silicon) -- **CPU**: 14 cores -- **RAM**: 36 GB -- **OS**: macOS -- **Date**: 2026-07-05 -- **Binary**: ARM64, optimized release build - -## Results - -| Metric | Value | -|--------|-------| -| Files indexed | 64,694 | -| Total nodes | 12,029,605 | -| Functions | 3,840,680 | -| CALLS edges | 3,727,864 | -| Cross-file CALLS | 1,502,432 (40%) | -| DB size | ~1.2 GB | -| Index time | ~3 min 7 sec | -| Peak memory | ~3 GB | -| Workers | 14 | - -## Architecture Improvements - -| Change | Before | After | -|--------|--------|-------| -| engine.cpp size | 3,499 lines | 48 lines (split into 6 files) | -| Cross-file resolution | ✗ | 1.5M edges | -| Pipeline | monolithic 2-phase | Translte→Link→Emit 4-phase | -| Linux kernel indexing | SIGILL crash | ✅ 64K files in 3 min | diff --git a/docs/bugs/bug_incremental_dedup.zh.md b/docs/bugs/bug_incremental_dedup.zh.md index fbe098c..30e9b28 100644 --- a/docs/bugs/bug_incremental_dedup.zh.md +++ b/docs/bugs/bug_incremental_dedup.zh.md @@ -21,6 +21,11 @@ ## 1. 如何复现 +> **注（2026-09-19）**：下面的 `codescope cli index_project ...` 是当时的调用方式， +> 现已被 `codescope index-parallel ` 取代 —— `index_project` 现在是 MCP +> 会话工具（派生子进程隔离内存），不进 `cli` 的分发表。复现步骤保持原样，以免 +> 改写当时的记录。 + ```bash # 1) 干净索引项目 codescope cli index_project '{"project_path":"/Users/scc/code/researcher/spring-petclinic"}' diff --git a/docs/bugs/bug_resolve_strategy.en.md b/docs/bugs/bug_resolve_strategy.en.md index 2907363..be4acfa 100644 --- a/docs/bugs/bug_resolve_strategy.en.md +++ b/docs/bugs/bug_resolve_strategy.en.md @@ -188,7 +188,7 @@ Measured `engine_get_callees` output (Transformer_Explorer): | `generate_attention_patterns` | p1_intra | in-project ✅ | | `__init__` → `MultiHeadAttention` | p1_intra | in-project ✅ | -The frontend can now filter `external` / `unresolved` via the `resolve_strategy` field,彻底解决 third-party false positives彻底 eliminated. +The frontend can now filter `external` / `unresolved` via the `resolve_strategy` field, which eliminates third-party false positives. ### 4.5 test_bun parameterization acceptance diff --git a/docs/comparison_report.md b/docs/comparison_report.md deleted file mode 100644 index a00ecd2..0000000 --- a/docs/comparison_report.md +++ /dev/null @@ -1,143 +0,0 @@ -# CodeScope vs codebase-memory-mcp 横向对比报告 - -> **被测项目**：一个基于 ARES 实现的 Agent 助手（Go 语言，~95 个源文件） -> **测试日期**：2026-07-06 -> **测试环境**：macOS arm64, 36GB RAM - ---- - -## 一、索引性能 - -| 指标 | CodeScope | codebase-memory-mcp | -|------|-----------|---------------------| -| **索引耗时** | **307 ms** | 545 ms | -| **索引文件数** | **95**（仅项目代码，正确排除 .venv） | ~82（含误排除） | -| **图节点** | **24,924** | 2,057 | -| **图边** | **23,184** | 7,025 | -| **Parse 耗时** | 12 ms | — | -| **SQLite 写入** | 113 ms | — | -| **BuildGraph** | 99 ms | — | -| **FTS 索引** | 36 ms | — | -| **发现目录数** | 3,944 | — | -| **跳过的目录** | **3,777**（.gitignore 规则生效） | 13（硬编码排除列表） | - -### 分析 - -CodeScope 索引速度 **快 1.8 倍**（307ms vs 545ms），且通过 `.gitignore` + `.codescopeignore` 任意深度匹配正确排除了噪音目录（.venv、node_modules 等）。而 codebase-memory-mcp 使用硬编码的排除列表，且存在一个严重缺陷： - -``` -排除的目录：.trae, bin, plans, .claude, docs, examples, scripts, - .git, .codescope, data, services/embedding/.venv, - cmd/server, cmd/bot ← 🐛 错误排除 -``` - -它错误地将 `cmd/server` 和 `cmd/bot` 也排除了——这两个是项目的主要入口目录。 - ---- - -## 二、信息完整性 - -| 维度 | CodeScope | codebase-memory-mcp | -|------|-----------|---------------------| -| **入口点发现** | ✅ 4/4 完整 | ❌ 3/4（遗漏 cmd/server/main.go） | -| **HTTP 路由** | ✅ 所有 handler 均索引 | ❌ 因 cmd/server 被排除不可见 | -| **热点分析** | ✅ 有完整文件路径 | ❌ 部分节点缺失文件路径 | -| **高复杂度函数** | ✅ 支持 `get_complexity` 查询 | ✅ `query_graph` 支持 | -| **模块树** | ✅ 支持 | ❌ 不支持 | -| **社区检测** | ❌ C++ 引擎有实现，MCP 未接入 | ❌ 不支持 | - ---- - -## 三、查询 Token 消耗 - -采用 DeepSeek 公式：`tokens = ASCII字符数 × 0.3 + 非ASCII字符数 × 0.6` - -### 3.1 图统计 - -| 工具 | 命令 | 响应字符数 | Token 消耗 | -|------|------|-----------|-----------| -| **CodeScope** | `get_graph_stats` | **58** | **17** | -| codebase-memory-mcp | `get_architecture`（含统计） | 16,707 | 5,012 | - -> codebase-memory-mcp 无法单独查询图统计，必须通过 `get_architecture` 一次性获取所有信息，token 贵约 **295 倍**。 - -### 3.2 代码搜索 - -| 工具 | 命令 | 响应字符数 | Token | -|------|------|-----------|-------| -| **CodeScope** | `search("handler")` | 1,243 | **372** | -| codebase-memory-mcp | `search_graph("handler")` | 2,768 | 830 | -| codebase-memory-mcp | `search_code("handler")` | 2,763 | 828 | - -### 3.3 其他查询 - -| 查询 | CodeScope | Tokens | codebase-memory-mcp | Tokens | -|------|-----------|--------|---------------------|--------| -| 热点 Top10 | `get_hotspots` ❌ 已移除 | — | — | 改用 explain_symbol | -| 入口点 | `get_entry_points` | **5** | `get_architecture`（含） | 已含 | -| 项目信息 | `get_project_info` | **43** | — | — | -| 调用者查询 | `find_callers("main")` | **7** | — | — | -| 项目总览 | `project_overview` | **71** | — | — | -| 模块树 | `get_module_tree` | **4** | ❌ 不支持 | — | -| 高复杂度函数 | — | — | `query_graph(complexity≥10)` | 206 | - -### 3.4 典型分析组合 - -| 方案 | 包含的查询 | 总 Token 消耗 | -|------|-----------|-------------| -| **CodeScope** | stats(17) + hotspots(466) + entry(5) + search(372) + info(43) + callers(7) | **~910** | -| codebase-memory-mcp | get_architecture(5012) + search_graph(830) + query_graph(206) | **~6,048** | - -> CodeScope **节省约 85% 的 token**，因为每个工具只返回所需信息，不含冗余字段。 - ---- - -## 四、社区检测（CodeScope 独有） - -| 参数 | 输出字符数 | Token 消耗 | -|------|-----------|-----------| -| `max_members=5, max_communities=10` | 342,219 | **102,665** ⚠️ | -| `max_members=0, max_communities=0`（全部） | ~数千万 | ~~17,010,398~~ ❌ | - -> **建议**：社区检测输出较大，建议使用更严格的限制（如 `max_members=3, max_communities=5`）或将 `total_communities` 单独作为摘要返回。 - ---- - -## 五、发现的 Bug 线索 - -通过高复杂度分析，在项目中发现了以下需关注的代码： - -| 函数 | 圈复杂度 | 认知复杂度 | 循环深度 | 行数 | 风险 | -|------|---------|-----------|---------|------|------| -| `cmd/chat/main.run` | **31** 🔴 | **63** 🔴 | **9** 🔴 | 188 | **极高** | -| `strategy.scoreParamReasonability` | **22** 🔴 | **41** 🔴 | 0 | 67 | **高** | -| `cmd/chat/main.handleCommand` | **18** 🔴 | **42** 🔴 | **3** | 65 | **高** | -| `extraction.findBalancedBraceEnd` | 9 | **21** 🔴 | 1 | 33 | 中 | -| `stream_test.TestChatStreamSuccess` | 11 | 31 🔴 | **7** 🔴 | 83 | 中 | - -这些函数建议进行重构以降低复杂度和维护成本。 - ---- - -## 六、总结 - -| 维度 | CodeScope | codebase-memory-mcp | -|------|-----------|---------------------| -| **索引速度** | ✅ **307 ms（快 1.8×）** | 545 ms | -| **数据完整性** | ✅ 95 文件，24,924 节点，完整覆盖 | ❌ **cmd/server 被错误排除** | -| **查询效率** | ✅ 最小查询仅 **17 tokens** | ❌ 最小查询 5,012 tokens | -| **典型分析成本** | ✅ **~910 tokens** | ❌ **~6,048 tokens** | -| **社区检测** | ❌ C++ 引擎有实现，MCP 未接入 | ❌ 不支持 | -| **模块树** | ✅ 支持（4 tokens） | ❌ 不支持 | -| **入口点** | ✅ 4/4 完整 | ❌ 3/4，遗漏 HTTP 服务 | -| **错误排除** | ✅ 无误排除（.gitignore 驱动） | ❌ 误排除 cmd/server 和 cmd/bot | -| **路由分析** | ✅ 完整 | ❌ 把 GitHub URL 当路由 | - -### 最终结论 - -CodeScope 经过本轮优化后（.gitignore 任意深度匹配、社区检测输出限流、项目 ID 继承修复），在 **索引速度**、**查询 token 效率** 和 **数据完整性** 三个维度上均显著优于 codebase-memory-mcp： - -1. **索引快 1.8 倍**，且信息量多 12 倍（24K vs 2K 节点） -2. **查询 token 成本低 85%**（910 vs 6,048 tokens） -3. **数据更完整**，入口点和路由全部覆盖 -4. **社区检测、模块树** 等功能是 codebase-memory-mcp 不具备的差异化能力 diff --git a/docs/en/QUICK_START.md b/docs/en/QUICK_START.md new file mode 100644 index 0000000..8cdfa81 --- /dev/null +++ b/docs/en/QUICK_START.md @@ -0,0 +1,140 @@ +# CodeScope Quick Start + +> Index your first project in five minutes. + +## 1. Install + +### Option 1: one-shot build (recommended) + +```bash +bash <(curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/bootstrap.sh) +``` + +The script detects your operating system, installs what is missing (Xcode, Homebrew, +LLVM, cmake, Rust) and then builds CodeScope. + +### Option 2: pre-built binary + +```bash +# macOS ARM64 +curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/install.sh | bash + +# Windows PowerShell +irm https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/install.ps1 | iex +``` + +### Option 3: build from source + +```bash +# 1. system dependencies +# macOS: +brew install llvm@21 cmake pkg-config sqlite3 + +# Linux (Ubuntu): +sudo apt-get install -y build-essential cmake llvm-dev libclang-dev libsqlite3-dev + +# 2. Rust +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + +# 3. build +git clone https://github.com/Timwood0x10/CodeScope.git +cd CodeScope +cargo build --release +``` + +## 2. Sanity check + +```bash +# the binary +./target/release/codescope --help + +# scan a project to see how many source files it finds +./target/release/codescope discover /path/to/your/project +``` + +## 3. Index a project + +```bash +# index the whole project (one subprocess per module, memory-isolated) +./target/release/codescope index-parallel /path/to/your/project + +# large projects: --workers is the total parse-core count (default 8), +# --parallel caps concurrent module workers (default 4) +./target/release/codescope index-parallel /path/to/your/project --workers 8 --parallel 4 +``` + +## 4. Start the MCP server + +```bash +# MCP speaks stdio, so there is no port +./target/release/codescope + +# the database path comes from the environment; there is no --db-path flag +CODESCOPE_DB_PATH=/path/to/codescope.db ./target/release/codescope +``` + +## 5. Index modes + +| Mode | `CODESCOPE_INDEX_MODE` | What it changes | +|:-----|:----------------------|:----------------| +| fast | `fast` | Extra discovery pruning (11 directory classes + 4 cache-file classes) and no FTS build | +| normal | `normal` (default) | Balanced discovery + the full pipeline | +| deep | `deep` | Like `normal`, and rebuilds the n-gram semantic vectors | +| strict | `strict` | **Discovery only**: a whitelist gate on detected languages; the parse pipeline is unchanged | + +## 6. Supported languages + +| Language | Extension | Status | +|:---------|:----------|:-------| +| Python | `.py` | vendored grammar, parsed end-to-end | +| Go | `.go` | vendored grammar | +| Rust | `.rs` | vendored grammar | +| JavaScript | `.js`, `.mjs` | vendored grammar | +| TypeScript | `.ts` | vendored grammar | +| TSX | `.tsx` | vendored grammar | +| C | `.c`, `.h` | vendored grammar (compiled `-O0`, see the SIGILL note below) | +| C++ | `.cpp`, `.cc`, `.cxx`, `.hpp`, `.hxx` | vendored grammar | +| Java | `.java` | vendored grammar | + +`.kt`/`.kts`, `.rb`, `.scala` and `.swift` are detected by extension but no grammar is +vendored: they are recorded in `parse_failures` as `language_missing` (see +`codescope parse-failures`) and never count towards the fail-fast skip. + +## 7. FAQ + +### Q: the build fails with "LLVM not found" + +Make sure LLVM is on `PATH`: + +```bash +# macOS Homebrew +export PATH="/opt/homebrew/opt/llvm@21/bin:$PATH" +export LDFLAGS="-L/opt/homebrew/opt/llvm@21/lib" +export CPPFLAGS="-I/opt/homebrew/opt/llvm@21/include" +``` + +### Q: indexing a C project crashes with SIGILL + +That is tree-sitter's C grammar v0.24.2 on Apple Silicon. CodeScope compiles that +grammar with `-O0` to avoid it (`engine/CMakeLists.txt`), so a current build +(`cargo build --release`) does not hit it. If you still see SIGILL, rebuild. + +### Q: how do I use it from an AI client? + +CodeScope speaks MCP, so it drops into Claude Desktop, Cursor and similar clients: + +```json +{ + "mcpServers": { + "codescope": { + "command": "/path/to/codescope" + } + } +} +``` + +## 8. Next + +- [Architecture](architecture.md) — how CodeScope works +- [Optimization notes](../optimization/optimization.md) — performance work +- [Benchmark record](../optimization/perf-full-index-2026-08-11.zh.md) — full-index measurements (written in Chinese) diff --git a/docs/en/architecture.md b/docs/en/architecture.md index 4129f46..c0a51e7 100644 --- a/docs/en/architecture.md +++ b/docs/en/architecture.md @@ -1,7 +1,7 @@ # CodeScope Architecture -**Version**: 0.4.0 -**Date**: 2026-07-12 +**Version**: v0.2.7 +**Date**: 2026-10-01 --- @@ -256,7 +256,7 @@ sequenceDiagram Client->>Server: tools/call index_project Server->>Worker: spawn subprocess - Worker->>DB: Write semantic_records + graph_nodes + Worker->>DB: Write semantic_records + entity + relation Worker-->>Server: stdout JSON result Server->>Server: RUNTIME.spawn(build_fts) Server-->>Client: {"ok":true, "files_indexed":N} @@ -279,7 +279,7 @@ sequenceDiagram | `search` (graph fallback) | **<10 ms** | LIKE fallback search | | `get_module_tree` | **<1 ms** | Lightweight query | | `get_entry_points` | **<1 ms** | Indexed lookup | -| `get_communities` | ❌ Engine impl exists, no MCP tool — use `connected_components` instead | — | +| `get_communities` | ✅ Available — deterministic label propagation over CALLS edges; summary-first (`max_communities` / `include_members` / `max_members`) | — | | `get_index_progress` | **<1 ms** | Atomic global read | --- diff --git a/docs/en/comparison_report.md b/docs/en/comparison_report.md deleted file mode 100644 index 4b1622d..0000000 --- a/docs/en/comparison_report.md +++ /dev/null @@ -1,143 +0,0 @@ -# CodeScope vs codebase-memory-mcp Cross-Comparison Report - -> **Project Under Test**: An Agent assistant based on ARES implementation (Go language, ~95 source files) -> **Test Date**: 2026-07-06 -> **Test Environment**: macOS arm64, 36GB RAM - ---- - -## I. Indexing Performance - -| Metric | CodeScope | codebase-memory-mcp | -|--------|-----------|---------------------| -| **Indexing Time** | **307 ms** | 545 ms | -| **Files Indexed** | **95** (project code only, correctly excluded .venv) | ~82 (including incorrect exclusions) | -| **Graph Nodes** | **24,924** | 2,057 | -| **Graph Edges** | **23,184** | 7,025 | -| **Parse Time** | 12 ms | — | -| **SQLite Write** | 113 ms | — | -| **BuildGraph** | 99 ms | — | -| **FTS Indexing** | 36 ms | — | -| **Directories Discovered** | 3,944 | — | -| **Directories Skipped** | **3,777** (.gitignore rules applied) | 13 (hardcoded exclusion list) | - -### Analysis - -CodeScope indexing is **1.8x faster** (307ms vs 545ms) and correctly excludes noise directories (`.venv`, `node_modules`, etc.) through `.gitignore` + `.codescopeignore` arbitrary depth matching. In contrast, codebase-memory-mcp uses a hardcoded exclusion list and has a critical flaw: - -``` -Excluded directories: .trae, bin, plans, .claude, docs, examples, scripts, - .git, .codescope, data, services/embedding/.venv, - cmd/server, cmd/bot ← 🐛 Incorrectly excluded -``` - -It incorrectly excluded `cmd/server` and `cmd/bot` — these are the project's main entry point directories. - ---- - -## II. Information Completeness - -| Dimension | CodeScope | codebase-memory-mcp | -|------------|-----------|---------------------| -| **Entry Point Discovery** | ✅ 4/4 complete | ❌ 3/4 (missing cmd/server/main.go) | -| **HTTP Routes** | ✅ All handlers indexed | ❌ Not visible due to cmd/server exclusion | -| **Hotspot Analysis** | ✅ Complete file paths | ❌ Some nodes missing file paths | -| **High Complexity Functions** | ✅ Supports `get_complexity` query | ✅ `query_graph` supports | -| **Module Tree** | ✅ Supported | ❌ Not supported | -| **Community Detection** | ❌ C++ engine has impl, no MCP tool | ❌ Not supported | - ---- - -## III. Query Token Consumption - -Using DeepSeek formula: `tokens = ASCII characters × 0.3 + non-ASCII characters × 0.6` - -### 3.1 Graph Statistics - -| Tool | Command | Response Characters | Token Consumption | -|------|---------|---------------------|-------------------| -| **CodeScope** | `get_graph_stats` | **58** | **17** | -| codebase-memory-mcp | `get_architecture` (includes stats) | 16,707 | 5,012 | - -> codebase-memory-mcp cannot query graph statistics separately; it must retrieve all information at once through `get_architecture`, costing approximately **295x more tokens**. - -### 3.2 Code Search - -| Tool | Command | Response Characters | Tokens | -|------|---------|---------------------|--------| -| **CodeScope** | `search("handler")` | 1,243 | **372** | -| codebase-memory-mcp | `search_graph("handler")` | 2,768 | 830 | -| codebase-memory-mcp | `search_code("handler")` | 2,763 | 828 | - -### 3.3 Other Queries - -| Query | CodeScope | Tokens | codebase-memory-mcp | Tokens | -|-------|-----------|--------|---------------------|--------| -| Hotspots Top10 | `get_hotspots` ❌ removed | — | — | Use explain_symbol instead | -| Entry Points | `get_entry_points` | **5** | `get_architecture` (included) | Already included | -| Project Info | `get_project_info` | **43** | — | — | -| Caller Query | `find_callers("main")` | **7** | — | — | -| Project Overview | `project_overview` | **71** | — | — | -| Module Tree | `get_module_tree` | **4** | ❌ Not supported | — | -| High Complexity Functions | — | — | `query_graph(complexity≥10)` | 206 | - -### 3.4 Typical Analysis Combination - -| Approach | Included Queries | Total Token Consumption | -|----------|------------------|------------------------| -| **CodeScope** | stats(17) + hotspots(466) + entry(5) + search(372) + info(43) + callers(7) | **~910** | -| codebase-memory-mcp | get_architecture(5012) + search_graph(830) + query_graph(206) | **~6,048** | - -> CodeScope **saves approximately 85% tokens** because each tool only returns the required information without redundant fields. - ---- - -## IV. Community Detection (CodeScope Exclusive) - -| Parameters | Output Characters | Token Consumption | -|------------|-------------------|-------------------| -| `max_members=5, max_communities=10` | 342,219 | **102,665** ⚠️ | -| `max_members=0, max_communities=0` (all) | ~tens of millions | ~~17,010,398~~ ❌ | - -> **Recommendation**: Community detection output is large; it's recommended to use stricter limits (e.g., `max_members=3, max_communities=5`) or return `total_communities` separately as a summary. - ---- - -## V. Bug Clues Discovered - -Through high complexity analysis, the following code areas of concern were identified in the project: - -| Function | Cyclomatic Complexity | Cognitive Complexity | Loop Depth | Lines | Risk | -|----------|----------------------|---------------------|------------|-------|------| -| `cmd/chat/main.run` | **31** 🔴 | **63** 🔴 | **9** 🔴 | 188 | **Very High** | -| `strategy.scoreParamReasonability` | **22** 🔴 | **41** 🔴 | 0 | 67 | **High** | -| `cmd/chat/main.handleCommand` | **18** 🔴 | **42** 🔴 | **3** | 65 | **High** | -| `extraction.findBalancedBraceEnd` | 9 | **21** 🔴 | 1 | 33 | Medium | -| `stream_test.TestChatStreamSuccess` | 11 | 31 🔴 | **7** 🔴 | 83 | Medium | - -These functions are recommended for refactoring to reduce complexity and maintenance costs. - ---- - -## VI. Summary - -| Dimension | CodeScope | codebase-memory-mcp | -|------------|-----------|---------------------| -| **Indexing Speed** | ✅ **307 ms (1.8× faster)** | 545 ms | -| **Data Completeness** | ✅ 95 files, 24,924 nodes, full coverage | ❌ **cmd/server incorrectly excluded** | -| **Query Efficiency** | ✅ Minimum query only **17 tokens** | ❌ Minimum query 5,012 tokens | -| **Typical Analysis Cost** | ✅ **~910 tokens** | ❌ **~6,048 tokens** | -| **Community Detection** | ❌ C++ engine has impl, no MCP tool | ❌ Not supported | -| **Module Tree** | ✅ Supported (4 tokens) | ❌ Not supported | -| **Entry Points** | ✅ 4/4 complete | ❌ 3/4, missing HTTP service | -| **Incorrect Exclusions** | ✅ No incorrect exclusions (.gitignore-driven) | ❌ Incorrectly excluded cmd/server and cmd/bot | -| **Route Analysis** | ✅ Complete | ❌ Mistook GitHub URLs as routes | - -### Final Conclusion - -After this round of optimization (.gitignore arbitrary depth matching, community detection output limiting, project ID inheritance fix), CodeScope significantly outperforms codebase-memory-mcp in three dimensions: **indexing speed**, **query token efficiency**, and **data completeness**: - -1. **1.8× faster indexing** with 12× more information (24K vs 2K nodes) -2. **85% lower query token cost** (910 vs 6,048 tokens) -3. **More complete data** with full coverage of entry points and routes -4. **Community detection and module tree** are differentiating capabilities not available in codebase-memory-mcp \ No newline at end of file diff --git a/docs/en/e2e_benchmark_report.md b/docs/en/e2e_benchmark_report.md deleted file mode 100644 index d09a11f..0000000 --- a/docs/en/e2e_benchmark_report.md +++ /dev/null @@ -1,59 +0,0 @@ -# CodeScope E2E Benchmark Report - -**Project**: goagent (Go, 1,134 files) -**Date**: 2026-07-13 -**Machine**: Apple M3 Max, 64GB RAM, macOS 15.0 -**Engine**: 14 workers × 8MB stack, FilterPolicy Normal - ---- - -## Pipeline Overview - -``` -1,134 Go files - │ - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Phase 1: Index │ -│ 2,739ms parse → 0ms sqlite → 23,624ms buildGraph │ -│ 17,127 nodes, 3,341 edges, 2,949 references │ -│ call_kind: 2,402/2,949 (81%) classified │ -└──────────────────────────────────────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Phase 2: Knowledge Graph │ -│ 159 modules, 176 module_edge rows │ -│ Architecture: 1,151 items │ -│ 3 workflows, 7 capabilities, 4 contracts │ -└──────────────────────────────────────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Phase 3: Resolver │ -│ 552 / 2,949 references resolved (18.7%) │ -│ 2,397 unresolved (mostly fmt.Errorf etc. stdlib calls) │ -└──────────────────────────────────────────────────────────┘ -``` - -## Feature Verification - -| Feature | Status | Data | -|---|---|---| -| Type Registry | ✅ | 2,164 type defs + 2,850 refs | -| Route Detection | ✅ | 46 HTTP routes | -| Call Classification | ✅ | 6 languages, 81% classified | -| Interface Method Capture | ✅ | 663 methods with correct parent_id | -| Module Role Labels | ✅ | 159 modules, classified by role | -| Architecture Communities | ✅ | 14 communities, 176 module edges | -| Multi-factor Resolver | ✅ | 8 factors, call_kind weighted | - -## Resource Usage - -| Metric | Value | -|---|---| -| Total time | ~26s (1,134 files) | -| Parse phase | 2,739ms | -| buildGraph phase | 23,624ms | -| Peak memory | ~150MB | -| Database size | ~176MB | diff --git a/docs/en/goagent_verification.md b/docs/en/goagent_verification.md deleted file mode 100644 index 8be557b..0000000 --- a/docs/en/goagent_verification.md +++ /dev/null @@ -1,90 +0,0 @@ -# CodeScope x goagent — Bridge Verification & Token Savings - -> Date: 2026-07-11 -> Project: [goagent](https://github.com/Timwood0x10/ares) (Go) -> 1,124 files, 331,390 lines of code - ---- - -## Bridge Verification - -The old GA system (scheduler) and the new system (coordinator) were connected via `GenomePopulationAdapter` with `submitToCoordinator` bridge. The call chain was verified by grep (not by call graph, which missed indirect struct method calls). - -### Call Chain - -| Link | Status | Location | -|------|--------|----------| -| `Run()` → `submitToCoordinator()` | ✅ | `genome_wiring.go:462` | -| `submitToCoordinator()` → `generateDiffPatches()` | ✅ | `genome_wiring.go:578` | -| `submitToCoordinator()` → `Coordinator.Submit()` | ✅ | `genome_wiring.go:584` | -| Production path `bootstrap.go` → `Coordinator.Submit()` | ✅ | `bootstrap.go:303,363` | -| Production path `genome_wiring_system.go:743` → `Coordinator.Submit()` | ✅ | | -| Self-healing path `evolution_bridge.go:70` → `Coordinator.Submit()` | ✅ | | - -### Architecture - -``` -Old System Scheduler (event-driven) - → GenomePopulationAdapter.Run() - → scoring + evolution (population.Evolve()) - → submitToCoordinator() ← New Bridge - → generateDiffPatches() - → Coordinator.Submit(PatchProposal{Source: SourceGA}) - → Coordinator.Evaluate() - -New System (decision layer) - → Coordinator receives patch proposals - → decides Apply / Delay / Reject - → applies via Patch Executor -``` - ---- - -## Token Savings - -| Metric | Value | -|--------|-------| -| Source files | 1,124 / 1,253 | -| Lines of code | 331,390 | -| Raw tokens | 3,313,900 | -| Knowledge graph tokens | 140,470 | -| **Token savings** | **95.8%** | -| Query latency | 0.5 ms | -| Index time | 31 s | -| Entities | 13,535 | -| Relations | 1,024 | -| Resolved references | 1,022 | - -### Calculation - -``` -Raw tokens: 331,390 lines × 10 = 3,313,900 -Knowledge graph: 13,535 × 10 + 1,024 × 5 = 140,470 -Savings: 95.8% -``` - -**AI's context is compressed from 3.3M tokens to 140K tokens — 23x information density gain.** - ---- - -## Orphan Modules Found (grep-based, excludes self-imports) - -| Module | Entities | External Imports | Verdict | -|--------|----------|-----------------|---------| -| `compat/` | 173 | 0 | ✅ Orphan | -| `api/evolution/` | 66 | 0 | ✅ Orphan | -| `api/service/ga/` | 15 | 0 | ✅ Orphan (deleted) | -| `graphservice/` | 39 | 0 | ✅ Orphan (deleted) | -| `EvolutionBridge` | — | 0 | ✅ Orphan (never instantiated) | -| `retrieval_api/` | 20 | 0 | ✅ Orphan (deleted) | -| `ares_quant/` | 992 | 0 | ✅ Orphan (new discovery) | - -### Audit Document Match - -All 6 orphan modules from the manual audit (`ga_audit/*.md`) matched 100%. CodeScope's automated analysis is consistent with manual grep. - ---- - -## Key Insight - -**Call graph (relation table) is insufficient for detecting connections in Go.** Interface-based calls and struct method calls (e.g., `a.submitToCoordinator(ctx)`) are not captured by the current Resolver Pipeline. The correct approach for orphan detection is **import-path-based analysis** (grep for import paths, excluding self-imports), which matches the manual audit methodology. \ No newline at end of file diff --git a/docs/en/ladybugdb_explorer.md b/docs/en/ladybugdb_explorer.md deleted file mode 100644 index 1febdbb..0000000 --- a/docs/en/ladybugdb_explorer.md +++ /dev/null @@ -1,262 +0,0 @@ -# LadybugDB Explorer with CodeScope - -> **Last Updated**: 2026-07-20 - -This guide explains how to use [LadybugDB Explorer](https://docs.ladybugdb.com/visualization/lbug-explorer/) to visually explore CodeScope's code graph data. - ---- - -## Prerequisites - -- **Docker**: LadybugDB Explorer runs as a Docker container. [Install Docker](https://docs.docker.com/get-docker/) if you don't have it. -- **CodeScope data**: A project must be indexed first (via `codescope index_project`). This creates the `.codescope/codescope.lbug` file. - ---- - -## Quick Start - -### Step 1: Verify the LadybugDB file exists - -```bash -ls -la .codescope/codescope.lbug -``` - -If the file doesn't exist or is 0 bytes, re-index the project: - -```bash -codescope index_project -``` - -### Step 2: Sync data from SQLite to LadybugDB - -The sync from SQLite to LadybugDB is **automatic**. When you run `codescope index_project`, the engine automatically syncs graph data from SQLite to LadybugDB using batched Cypher `CREATE` statements. No manual steps are required. - -> **Note**: The engine uses Cypher `CREATE` instead of `COPY FROM` because `COPY FROM` (CSV bulk import) has a known bug in the vendored LadybugDB 0.18.2 that causes it to fail with `state=1`. Cypher `CREATE` is the reliable path. - -```mermaid -sequenceDiagram - participant SQL as SQLite (.db) - participant ENG as CodeScope Engine - participant LBUG as LadybugDB (.lbug) - - SQL->>ENG: Read graph_nodes / graph_edges - ENG->>LBUG: Batched Cypher CREATE (nodes) - ENG->>LBUG: Batched Cypher CREATE (edges) - Note over LBUG: Sync complete ✓ -``` - -#### Troubleshooting - -If the `.lbug` file is empty after indexing, verify that the engine was compiled with `HAS_LADYBUG` (check build output for `LadybugDB: found at ...`). The engine uses Cypher `CREATE` as a reliable alternative to `COPY FROM`, which has a known issue in LadybugDB 0.18.2. - -### Step 3: Launch LadybugDB Explorer - -```bash -docker run -p 8000:8000 \ - -v "$(pwd)/.codescope:/database" \ - -e LBUG_FILE=codescope.lbug \ - --rm ghcr.io/ladybugdb/explorer:latest -``` - -### Step 4: Open in browser - -Go to [http://localhost:8000](http://localhost:8000) - -```mermaid -flowchart LR - A["Docker Container
ghcr.io/ladybugdb/explorer"] -->|"mount"| B[".codescope/codescope.lbug"] - A -->|"expose port 8000"| C["Browser opens
http://localhost:8000"] - C --> D["LadybugDB Explorer
Interactive graph exploration"] -``` - ---- - -## Usage Examples - -### 1. Explore all symbols - -In the Query panel, run: - -```cypher -MATCH (n:GraphNode) RETURN n.name, n.language, n.file_path LIMIT 50; -``` - -Click **Graph** view to see the nodes as an interactive graph. - -### 2. Find all Go functions - -```cypher -MATCH (n:GraphNode) -WHERE n.language = 'go' -RETURN n.name, n.file_path, n.start_row -ORDER BY n.name; -``` - -### 3. Visualize call relationships - -```cypher -MATCH (n:GraphNode)-[c:CALLS]->(m:GraphNode) -RETURN n.name, c.call_site_line, m.name -LIMIT 100; -``` - -Switch to **Graph** view to see the call graph as an interactive network. - -```mermaid -graph LR - subgraph "Visualization Example" - A["parseJSON
(bootstrap.go:12)"] -->|"CALLS"| B["json.Unmarshal
(encoding/json:456)"] - A -->|"CALLS"| C["strings.TrimSpace
(strings:123)"] - D["buildContext
(context.go:89)"] -->|"CALLS"| A - E["handleRequest
(handler.go:45)"] -->|"CALLS"| A - E -->|"CALLS"| D - end -``` - -### 4. Find callers of a specific function - -```cypher -MATCH (caller:GraphNode)-[c:CALLS]->(callee:GraphNode) -WHERE callee.name = 'targetFunction' -RETURN caller.name, caller.file_path, c.call_site_line; -``` - -### 5. Find the most-called functions - -```cypher -MATCH (caller:GraphNode)-[c:CALLS]->(callee:GraphNode) -RETURN callee.name, count(c) AS call_count -ORDER BY call_count DESC -LIMIT 20; -``` - -### 6. Find hot functions (high fan-in + high complexity) - -```cypher -MATCH (caller:GraphNode)-[c:CALLS]->(callee:GraphNode) -WITH callee, count(c) AS incoming -MATCH (callee)-[c2:CALLS]->(other:GraphNode) -WITH callee, incoming, count(c2) AS outgoing -RETURN callee.name, incoming, outgoing -ORDER BY incoming DESC, outgoing DESC -LIMIT 20; -``` - -### 7. Shortest path between two functions - -```cypher -MATCH p = shortestPath( - (a:GraphNode {name: 'funcA'})-[*..10]->(b:GraphNode {name: 'funcB'}) -) -RETURN p; -``` - ---- - -## Data Flow Overview - -```mermaid -flowchart TB - subgraph "CodeScope Indexing Pipeline" - A["Source Code"] --> B["tree-sitter Parser"] - B --> C["SQLite Storage
codescope.db"] - C --> E["Automatic Sync (Cypher CREATE)
codescope.lbug"] - end - - subgraph "LadybugDB Explorer Visualization" - E --> F["Docker Container
ghcr.io/ladybugdb/explorer"] - F --> G["Browser
http://localhost:8000"] - G --> H["Cypher Queries"] - G --> I["Interactive Graph View"] - G --> J["Table View"] - G --> K["JSON View"] - end - - subgraph "Terminal Queries" - E --> L["lbug CLI"] - L --> M["Cypher Query Results"] - end -``` - ---- - -## Schema Reference - -### GraphNode (Node Table) - -| Column | Type | Description | -|--------|------|-------------| -| `id` | INT64 | Primary key (matches SQLite graph_nodes.id) | -| `project_id` | INT64 | Project identifier | -| `ir_node_id` | INT64 | Internal IR node reference | -| `node_type` | INT32 | 0=function, 1=class, 2=variable, 3=type, ... | -| `name` | STRING | Symbol name | -| `qualified_name` | STRING | Fully qualified name (e.g., `pkg.FuncName`) | -| `signature` | STRING | Function signature (parameters + return type) | -| `module_path` | STRING | Module/package path | -| `file_path` | STRING | Source file path | -| `language` | STRING | Programming language | -| `start_row` | INT32 | Start line in source file | -| `start_col` | INT32 | Start column | -| `end_row` | INT32 | End line | -| `end_col` | INT32 | End column | -| `parent_id` | INT64 | Parent node ID (for containment) | -| `is_entry_point` | BOOL | Whether this is an entry point (main, init, etc.) | -| `embedding_ready` | BOOL | Whether vector embedding is computed | -| `metrics_ready` | BOOL | Whether metrics are computed | - -### CALLS (Relationship Table) - -| Column | Type | Description | -|--------|------|-------------| -| `FROM` | GraphNode | Caller node | -| `TO` | GraphNode | Callee node | -| `project_id` | INT64 | Project identifier | -| `edge_type` | INT32 | 3=call, 1=contain, 2=reference | -| `call_site_line` | INT32 | Line number of the call site | -| `label` | STRING | Optional label | - -### RELATES (Relationship Table) - -| Column | Type | Description | -|--------|------|-------------| -| `FROM` | GraphNode | Source node | -| `TO` | GraphNode | Target node | -| `project_id` | INT64 | Project identifier | -| `type` | INT32 | Relation type | - ---- - -## Tips - -### Performance - -- For large graphs (100k+ nodes), use `LIMIT` to avoid overloading the browser -- Use indexed properties (`name`, `language`, `file_path`) in WHERE clauses for faster queries -- The Explorer runs in your browser — all computation is on the server side - -### Export - -You can export query results as JSON or CSV from the Query panel's results view. - -### Read-Only Mode - -To launch in read-only mode (prevent accidental modifications): - -```bash -docker run -p 8000:8000 \ - -v "$(pwd)/.codescope:/database" \ - -e LBUG_FILE=codescope.lbug \ - -e MODE=READ_ONLY \ - --rm ghcr.io/ladybugdb/explorer:latest -``` - -### Troubleshooting - -**"File not found" error**: Ensure the `.lbug` file path is correct and the Docker volume mount points to the directory containing it. - -**"Connection refused"**: Make sure Docker is running (`docker ps` to check) and port 8000 is not in use. - -**Empty results**: Run `MATCH (n) RETURN count(n)` to verify data exists. If count is 0, re-run the sync step. - -**Slow queries**: Add `LIMIT` to your queries. For large graphs, start with `LIMIT 50` and increase as needed. \ No newline at end of file diff --git a/docs/en/large_project_index_report.md b/docs/en/large_project_index_report.md deleted file mode 100644 index bc5b82f..0000000 --- a/docs/en/large_project_index_report.md +++ /dev/null @@ -1,597 +0,0 @@ -# CodeScope Large Project Index Report - -> Date: 2026-07-06 | Hardware: Apple M3 Max, 36 GB RAM | macOS - ---- - -## 1. rustc (Rust Compiler) — Full Index - -### Complete Log - -``` -Start: 2026-07-06 19:50:33 - -worker: project=0 starting index_project dir=～/code/rustcode/rust lang= -BATCH [0..99] of 36807 (100 files) -BATCH [0..99] done (0 passes ok), total indexed: 100 -... -BATCH [36800..36806] of 36807 (7 files) -BATCH [36800..36806] done (0 passes ok), total indexed: 36807 -POST_BUILD: symbol graph... -buildGraph: 36800 files | file_list=339ms delete=457ms rf=21ms r2n=7317ms - nodes=8442ms edges=14066ms calls=0ms total=30702ms -{"ok":true,"files_indexed":36807,"workers":14, - "time_parse_ms":3193,"time_sqlite_ms":38932, - "time_buildgraph_ms":30703,"time_fts_ms":22995,"time_vector_ms":0, - "discovery":{"seen_dirs":65437,"seen_files":59339, - "skipped_dirs":1698,"skipped_files":49, - "skipped_suffix":76,"candidate_files":36807}} - -End: 2026-07-06 19:52:18 -Total: 1m44s -``` - -### Performance Metrics - -| Metric | Value | -|--------|:-----:| -| **Total time** | **1m44s** | -| Source files | 36,807 (all .rs) | -| Graph nodes | **4,130,017** | -| Graph edges | **3,044,162** | -| Parse | 3,193ms (3.0%) | -| SQLite write | 38,932ms (37.4%) | -| buildGraph | 30,703ms (29.5%) | -| FTS index | 22,995ms (22.1%) | -| Directories discovered | 65,437 | -| Directories skipped | 1,698 | -| Workers | 14 | - -### Bottleneck Analysis - -``` -Parse 3,193ms ███░░░░░░░░░░░░░░░░░░ 3.0% -SQLite 38,932ms █████████████████████ 37.4% -Graph 30,703ms █████████████████░░░░ 29.5% -FTS 22,995ms █████████████░░░░░░░░ 22.1% -``` - -**Still SQLite-bound** (SQLite + buildGraph + FTS = 89%). Parsing only took 3.2s (3%), achieving throughput of **11,523 files/sec**. - -### Query Token Consumption - -| Query | Tokens | Percentage | -|-------|:------:|:----------:| -| `get_graph_stats` | 19 | — | -| `get_hotspots` top10 | ❌ 未实现 — 无 MCP 工具 | — | — | -| `project_overview` | ~71 | — | -| `get_module_tree` | 4 | — | -| `get_entry_points` | 5 | — | -| `search` (single) | ~300-1000 | — | -| **Typical analysis combo** | **~910** | — | - ---- - -## 2. Bun Runtime — Full Index - -### Complete Log - -``` -Start: 2026-07-06 19:55:37 - -worker: project=0 starting index_project dir=～/code/researcher/bun lang= -BATCH [0..99] of 9641 (100 files) -BATCH [0..99] done (0 passes ok), total indexed: 100 -... -BATCH [9600..9640] of 9641 (41 files) -BATCH [9600..9640] done (0 passes ok), total indexed: 9641 -POST_BUILD: symbol graph... -buildGraph: 9623 files | file_list=238ms delete=116ms rf=5ms r2n=4842ms - nodes=5632ms edges=9450ms calls=0ms total=20320ms -{"ok":true,"files_indexed":9641,"workers":14, - "time_parse_ms":1024,"time_sqlite_ms":25736, - "time_buildgraph_ms":20321,"time_fts_ms":11568,"time_vector_ms":0, - "discovery":{"seen_dirs":21672,"seen_files":13883, - "skipped_dirs":6415,"skipped_files":4, - "skipped_suffix":504,"candidate_files":9641}} - -End: 2026-07-06 19:56:39 -Total: 1m1s -``` - -### Performance Metrics - -| Metric | Value | -|--------|:-----:| -| **Total time** | **1m1s** | -| Source files | 9,641 | -| Graph nodes | **2,951,664** | -| Graph edges | **2,567,205** | -| Parse | 1,024ms (1.7%) | -| SQLite write | 25,736ms (43.6%) | -| buildGraph | 20,321ms (34.4%) | -| FTS index | 11,568ms (19.6%) | -| Directories discovered | 21,672 | -| Directories skipped | 6,415 | -| Workers | 14 | - -### Bottleneck Analysis - -``` -Parse 1,024ms ██░░░░░░░░░░░░░░░░░░░ 1.7% -SQLite 25,736ms █████████████████████ 43.6% -Graph 20,321ms ██████████████████░░░ 34.4% -FTS 11,568ms ███████████████░░░░░░ 19.6% -``` - ---- - -## 3. Bun Runtime Core Functionality Analysis - -### Tech Stack - -Bun is primarily written in **Zig**, integrating **WebKit's JavaScriptCore** as the JS engine, with extensive C++ FFI binding layers: - -| Language | Files | Purpose | -|----------|:-----:|---------| -| Zig | 1,270 | Core runtime, CLI, bundler, package manager | -| C++ | 579 | JSC binding layer (`src/jsc/bindings/`) | -| C | 113 | WebCore integration, system calls | -| JS/TS | 6,592 | Tests, API definitions | - -### Core Architecture - -```mermaid -graph TB - subgraph "Bun CLI (main.zig)" - CLI["Cli.start()"] - end - subgraph "Commands" - RUN["bun run"] - TEST["bun test"] - INSTALL["bun install"] - BUILD["bun build"] - SERVE["bun serve"] - end - subgraph "JS Runtime (bun.js.zig)" - VM["VirtualMachine"] - TRANS["Transpiler
(TS/JSX→JS)"] - RES["Resolver"] - end - subgraph "JavaScriptCore (C++)" - JSC["JSC::JSGlobalObject"] - BIND["bindings: JSCommonJSModule
JSCommonJSExtensions"] - WC["WebCore bindings
TypedArrayController"] - end - subgraph "System Layer" - MIM["mimalloc allocator"] - SSL["BoringSSL"] - end - - CLI --> RUN & TEST & INSTALL & BUILD & SERVE - RUN --> VM - VM --> JSC - VM --> TRANS - VM --> RES - JSC --> BIND & WC - CLI --> MIM & SSL -``` - -### Core Feature Breakdown - -| Feature | Implementation Location | Technology | -|---------|------------------------|-----------| -| **JS/TS execution** | `src/bun.js.zig` → `VirtualMachine` | Zig + JSC C++ FFI | -| **Transpiler** | `src/transpiler/` | Zig implementation of TS/JSX→JS | -| **Bundler** | `src/bundler/` | Zig (BundleThread.zig) | -| **Package manager** | `src/install/` | Zig (npm protocol implementation) | -| **HTTP server** | `src/http/` | Zig + libuv | -| **SQLite** | `src/sqlite/` | Zig wrapper | -| **WebSocket** | `src/websocket/` | Zig | -| **Crypto** | `src/boringssl/` | BoringSSL (OpenSSL fork) | -| **Memory allocator** | `src/bun_alloc/` | mimalloc | - -### Key Technical Details - -1. **JavaScriptCore Integration**: Bun directly links WebKit's `libjavascriptcoregtk`, exposing JSC API to Zig through C++ glue code in `src/jsc/bindings/` -2. **Main Loop**: `Cli.start()` in `main.zig` parses subcommands and dispatches -3. **Fast Startup**: Bun pre-compiles WebKit bytecode cache, achieving 4-5x faster cold start than Node.js -4. **mimalloc**: Uses Microsoft's mimalloc instead of system allocator to reduce memory fragmentation -5. **Zig Zero-Cost Abstractions**: Zig's comptime and no implicit allocation policies are the foundation of Bun's performance advantages - ---- - -## 4. Arc/Rc & Multithreading Implementation in rustc - -> Based on source analysis of `library/alloc/src/sync.rs` (4,974 lines) and `library/alloc/src/rc.rs` (4,599 lines). - -### 4.1 Arc Core Data Structure - -```rust -// sync.rs:269-276 -pub struct Arc { - ptr: NonNull>, // heap-allocated ArcInner - phantom: PhantomData>, - alloc: A, // allocator (default Global) -} - -// sync.rs:388-401 -#[repr(C, align(2))] -struct ArcInner { - strong: Atomic, // thread-safe strong ref count - weak: Atomic, // thread-safe weak ref count - data: T, // actual data -} -``` - -Key design points: -- **`Atomic`** replaces Rc's `Cell` — the fundamental difference between Arc thread-safety and Rc single-threading -- **`#[repr(C, align(2))]`** — 2-byte alignment enables low-bit tagging (for weak reference count locking) - -### 4.2 Send/Sync Derivation - -```rust -unsafe impl Send for Arc {} -unsafe impl Sync for Arc {} -``` - -`Arc: Send` if and only if `T: Sync + Send` and allocator is Send. This is a manually annotated `unsafe impl` — the compiler trusts the programmer to have chosen the correct atomic types. - -### 4.3 Arc::new Allocation Path - -```rust -pub fn new(data: T) -> Arc { - let x: Box<_> = Box::new(ArcInner { - strong: atomic::AtomicUsize::new(1), // strong=1 - weak: atomic::AtomicUsize::new(1), // weak=1 - data, - }); - unsafe { Self::from_inner(Box::leak(x).into()) } - // ^^^ Box::leak prevents Box from freeing memory, - // transferring lifetime control to reference count -} -``` - -### 4.4 Arc::clone Atomic Increment - -```rust -fn clone(&self) -> Self { - let inner = self.ptr.as_ref(); - inner.strong.fetch_add(1, Ordering::Relaxed); - // Relaxed is sufficient: clone doesn't involve data dependency on other memory - Self { ptr: self.ptr, phantom: PhantomData, alloc: ... } -} -``` - -### 4.5 Arc::drop Atomic Decrement & Memory Deallocation - -```rust -fn drop(&mut self) { - let inner = self.ptr.as_ref(); - if inner.strong.fetch_sub(1, Ordering::Release) != 1 { - return; // other strong references exist - } - atomic::fence(Ordering::Acquire); // synchronize with last Release - unsafe { ptr::drop_in_place(&mut (*ptr).data); } - - if inner.weak.fetch_sub(1, Ordering::Release) == 1 { - atomic::fence(Ordering::Acquire); - unsafe { dealloc(ptr as *mut u8, layout); } - } -} -``` - -**Memory Ordering Strategy**: -- `fetch_sub(Release)`: ensures all writes before drop are visible to other threads -- `fence(Acquire)`: ensures we see the last thread's final writes to data before releasing -- Release/Acquire forms a **happens-before** relationship — the foundation of lock-free programming - -### 4.6 Arc vs Rc Comparison - -```mermaid -flowchart TD - subgraph "Rc (single-threaded)" - RC_CELL["strong: Cell"] - RC_WEAK["weak: Cell"] - RC_CLONE["clone: cell.get()+1"] - RC_DROP["drop: cell.get()-1"] - RC_SEND["Send: ❌"] - RC_SYNC["Sync: ❌"] - end - subgraph "Arc (multi-threaded)" - ARC_ATOMIC["strong: Atomic"] - ARC_WEAK["weak: Atomic"] - ARC_CLONE["clone: fetch_add(Relaxed)"] - ARC_DROP["drop: fetch_sub(Release)+fence(Acquire)"] - ARC_SEND["Send: ✅ T: Sync+Send"] - ARC_SYNC["Sync: ✅ T: Sync+Send"] - end -``` - -| Dimension | Arc (sync.rs) | Rc (rc.rs) | -|-----------|--------------|-----------| -| Ref count type | `Atomic` | `Cell` | -| Send | ✅ conditional | ❌ not supported | -| Sync | ✅ conditional | ❌ not supported | -| Memory ordering | Relaxed / Release / Acquire + fence | None (single-threaded) | -| Overhead | Atomic operation instructions | Zero | - -### 4.7 Thread Underlying Implementation - -```rust -// library/std/src/thread/mod.rs -// Platform dispatch (via cfg macro): -// Linux/macOS: sys::unix::thread → libc::pthread_create -// Windows: sys::windows::thread → CreateThread - -pub fn spawn(f: F) -> JoinHandle -where F: FnOnce() -> T + Send + 'static, T: Send + 'static -{ - // Builder::new().spawn(f) - // → imp::Thread::new(name, Box::new(f)) - // → pthread_create / CreateThread -} -``` - -The `Send` constraint on `spawn()` is checked at **compile time** by the type system, not a runtime check. The `unsafe impl Send for Arc` is the key that allows `Arc` to be moved across threads. - -### Summary - -Arc's thread safety relies on three underlying mechanisms: - -1. **`Atomic`** hardware-level atomic operations (x86 `LOCK XADD` / ARM `LDREX+STREX`) -2. **`Release/Acquire` memory ordering** ensures cross-thread happens-before -3. **`unsafe impl Send/Sync`** manual annotation — the compiler trusts the programmer chose correct atomic types - ---- - -## 5. Cross-Project Comparison - -| Metric | rustc (36,807 .rs) | Bun (9,641 Zig/C++/JS) | Linux kernel (60,468 C/H) | -|--------|:-----------------:|:----------------------:|:-------------------------:| -| **Index time** | **1m44s** | **1m1s** | **~300s (timeout)** | -| Parse throughput | 11,523 files/s | 9,414 files/s | — | -| Graph nodes | **4,130,017** | 2,951,664 | — | -| Graph edges | 3,044,162 | 2,567,205 | — | -| Parse % | 3.0% | 1.7% | 2.5% | -| SQLite % | 37.4% | 43.6% | 30.5% | -| buildGraph % | 29.5% | 34.4% | 37.3% | -| FTS % | 22.1% | 19.6% | 29.3% | - -## 6. Token Consumption Comparison - -Aggregated from three projects in this test: - -| Query | rustc | Bun | ARES Agent | memscope-rs | -|-------|:-----:|:---:|:----------:|:-----------:| -| `get_graph_stats` | 19 | 19 | 18 | 18 | -| `get_hotspots` top10 | ❌ 未实现 — 无 MCP 工具 | — | — | — | -| `project_overview` | 71 | 71 | 71 | 71 | -| `get_module_tree` | 4 | 4 | 4 | 4 | -| `get_entry_points` | 5 | 5 | 5 | 5 | -| `get_project_info` | 44 | 39 | 44 | 44 | - -**Cross-project consistency**: Token consumption depends only on **returned data volume**, not project size. Both million-node and thousand-node projects' `get_graph_stats` are ~18 tokens. - -**vs raw source code**: -- rustc 36,807 files: CodeScope returns architecture info in **~910 tokens** vs reading source code would require **millions of tokens** -- Bun 9,641 files: CodeScope returns architecture info in **~910 tokens** vs reading source code would require **hundreds of thousands of tokens** - -**vs codebase-memory-mcp**: codebase-memory-mcp cannot complete full indexing for either project (rustc: no Rust multi-file graph support, Bun: no Zig support, JDK: no Java support), while CodeScope covers all languages through tree-sitter multi-language parsers. - ---- - -## 6. JDK 20K Java Files — Full Index - -### Complete Log - -``` -Start: 2026-07-06 20:50:45 - -worker: project=0 starting index_project dir=～/code/researcher/jdk/src lang= -BATCH [0..99] of 19821 (100 files) -... -BATCH [19800..19820] of 19821 (21 files) -BATCH [19800..19820] done, total indexed: 19821 -POST_BUILD: symbol graph... -buildGraph: 19821 files | file_list=644ms delete=239ms rf=10ms r2n=14567ms - nodes=22609ms edges=38139ms calls=0ms total=76344ms -{"ok":true,"files_indexed":19821,"workers":14, - "time_parse_ms":2410,"time_sqlite_ms":83556, - "time_buildgraph_ms":76344,"time_fts_ms":38349,"time_vector_ms":0, - "discovery":{"seen_dirs":25805,"seen_files":22935, - "skipped_dirs":113,"skipped_files":0, - "skipped_suffix":765,"candidate_files":19821}} - -End: 2026-07-06 20:54:16 -Total: 3m31s -``` - -### Performance - -| Metric | Value | -|--------|:-----:| -| **Total time** | **3m31s** | -| Source files | 19,821 (src/ only, test/make/doc excluded) | -| Graph nodes | **8,047,762** | -| Graph edges | **6,723,123** | -| Parse | 2,410ms (1.2%) | -| SQLite write | 83,556ms (41.7%) | -| buildGraph | 76,344ms (38.1%) | -| FTS (deferred) | 38,349ms (19.1%) | -| DB size | **8.9 GB** | - -### Bottleneck - -``` -Parse 2,410ms ██░░░░░░░░░░░░░░░░░░░ 1.2% -SQLite 83,556ms █████████████████████ 41.7% -Graph 76,344ms ████████████████████░ 38.1% -FTS 38,349ms █████████████████░░░░ 19.1% -``` - -### Why JDK Is Slower Than rustc - -JDK has fewer files (19,821 vs 36,807) but takes longer (3m31s vs 1m44s) because: - -| Project | Files | Nodes/File | Total Nodes | Throughput | -|---------|:-----:|:----------:|:-----------:|:----------:| -| rustc | 36,807 | **112** | 4.13M | **353 files/s** | -| Bun | 9,641 | **306** | 2.95M | 155 files/s | -| **JDK** | **19,821** | **405** | **8.05M** | 93 files/s | - -JDK produces **405 nodes per file** (3.6× rustc). Java classes have more fields, methods, annotations, and inner classes than Rust modules, resulting in more SQLite writes and buildGraph work. - -### Before vs After FilterPolicy Fix - -| State | Files | Result | -|-------|:-----:|:-------| -| ❌ Before FilterPolicy fix | 59,923 (45,586 test files) | 300s timeout, incomplete | -| ✅ After FilterPolicy fix | **19,821** (src/ only) | **3m31s, complete** | - ---- - -## 7. Full Comparison - -| Metric | rustc | Bun | JDK | Linux kernel | -|--------|:-----:|:---:|:---:|:------------:| -| **Primary language** | Rust | Zig/C++/JS | **Java** | C | -| **Source files** | 36,807 | 9,641 | **19,821** | ~60,000 | -| **Index time** | **1m44s** | 1m1s | **3m31s** | ~300s (timeout) | -| **Graph nodes** | 4,130,017 | 2,951,664 | **8,047,762** | — | -| **Graph edges** | 3,044,162 | 2,567,205 | **6,723,123** | — | -| **Nodes/file** | 112 | 306 | **405** | — | -| **DB size** | — | — | **8.9 GB** | 12 GB | -| **Parse %** | 3.0% | 1.7% | 1.2% | 2.5% | -| **SQLite %** | 37.4% | 43.6% | **41.7%** | 30.5% | -| **buildGraph %** | 29.5% | 34.4% | **38.1%** | 37.3% | -| **FTS %** | 22.1% | 19.6% | 19.1% | 29.3% | - ---- - -## 8. Additional Projects - -### codebase-memory-mcp (Competitor Source) - -| Metric | Value | -|--------|:-----:| -| **Index time** | **2m0s** | -| Source files | 1,257 | -| Graph nodes | **6,221,406** | -| Graph edges | **1,065,570** | -| DB size | ~1.2 GB | -| Languages | **Pure C** (src/ + vendored/) — rewritten from Go to pure C | - -### CPython - -| Part | Files | Time | Nodes | Status | -|------|:-----:|:----:|:-----:|:------:| -| Python source | **1,022** | **6s** | 446,618 | ✅ Done | -| C source | ~1,100 | — | — | ❌ tree-sitter-c parser bug | - -**Why CPython C files failed**: tree-sitter-c 0.24.1's parser crashes with `Illegal instruction: 4` (SIGILL) when parsing core CPython C files (e.g., `Python/Python-ast.c`, `Objects/unicodeobject.c`). This is a tree-sitter upstream bug in the generated parser.c using unsupported CPU instructions. - ---- - -## 9. Complete Index Summary - -| Project | Language | Files | Time | Nodes | Status | -|---------|----------|:-----:|:----:|:-----:|:------:| -| **rustc** | Rust | 36,807 | 1m44s | 4,130,017 | ✅ | -| **JDK** | Java/C++ | 19,821 | 3m31s | 8,047,762 | ✅ | -| **Bun** | Zig/C++/JS | 9,641 | 1m1s | 2,951,664 | ✅ | -| **codebase-memory-mcp** | Pure C (src/ + vendored/) | 1,257 | 2m0s | 6,221,406 | ✅ | -| **CPython (Python)** | Python | 1,022 | 6s | 446,618 | ✅ | -| **CPython (C)** | C | ~1,100 | — | — | ❌ tree-sitter bug | -| **Linux kernel** | C | ~60,000 | 300s timeout | — | ⚠️ Partial | -| **memscope-rs** | Rust/C | 238 | 2s | 123,270 | ✅ | -| **InstrumentTimbre** | Python | 142 | 0.9s | — | ✅ | -| **ARES Agent** | Go | 95 | 0.3s | 24,924 | ✅ | - -### Key Takeaways - -1. **Bottleneck is always SQLite** (SQLite + buildGraph + FTS = ~90%), parse is only 1-3% -2. **Nodes/file determines index speed**: JDK 405 nodes/file (most complex) → 3m31s; rustc 112 nodes/file → 1m44s -3. **FilterPolicy + .gitignore** drastically reduces noise (JDK from 59,923 to 19,821 files) -4. **Deferred FTS** allows graph queries before FTS completes -5. **tree-sitter-c parser** has compatibility issues with complex C codebases (CPython/Linux kernel), awaiting upstream fix - ---- - -## 10. Query Performance Benchmarks - -### Query Latency - -| Query | Avg Latency | Notes | -|-------|:-----------:|-------| -| `get_graph_stats` | **<1 ms** | Pure SQL COUNT | -| `get_hotspots` | ❌ Not implemented — no MCP tool | — | -| `find_callers` / `find_callees` | **<1 ms** | Index-covered JOIN | -| `search` (FTS) | **<5 ms** | FTS5 full-text | -| `search` (graph fallback) | **<10 ms** | LIKE fallback search | -| `get_module_tree` | **<1 ms** | Lightweight | -| `get_entry_points` | **<1 ms** | Indexed lookup | -| `get_communities` | ❌ Engine impl exists, no MCP tool | — | -| `codescope_trace` | **<5 ms** | BFS path search | - -### Cross-Project Benchmarks - -| Project | Files | Index Time | Parse | SQLite | buildGraph | FTS | Nodes/File | -|---------|:-----:|:----------:|:-----:|:------:|:----------:|:---:|:----------:| -| rustc | 36,807 | 1m44s | 3.2s | 38.9s | 30.7s | 23.0s | 112 | -| Bun | 9,641 | 1m1s | 1.0s | 25.7s | 20.3s | 11.6s | 306 | -| JDK | 19,821 | 3m31s | 2.4s | 83.6s | 76.3s | 38.3s | **405** | -| codebase-memory-mcp | 1,257 | 2m0s | — | — | 48.0s | — | — | -| memscope-rs | 238 | 2s | 31ms | 589ms | 577ms | 253ms | — | -| ARES Agent | 95 | 0.3s | 9ms | 96ms | 81ms | 30ms | — | - -### Bottleneck Summary - -``` -All projects show consistent bottleneck distribution: - SQLite + buildGraph = ~70-80% - FTS = ~20% - Parse = ~1-3% - -Conclusion: CodeScope is SQLite-bound, not CPU-bound. -Optimizing SQLite writes and buildGraph is the only effective speedup path. -``` - ---- - -## 11. buildGraph SQL Audit - -### EXPLAIN QUERY PLAN Results (50-file C++ project) - -``` ---- QPLAN _r2n --- - id=6 parent=3 SEARCH sr USING INDEX idx_sr_kind (project_id=? AND kind=?) - id=50 parent=3 USING INDEX sqlite_autoindex__rf_1 FOR IN-OPERATOR - id=69 parent=0 SCAN (subquery-3) -``` - -| Phase | SQL | Time | Analysis | -|-------|-----|:----:|----------| -| file_list | `SELECT DISTINCT file_path FROM semantic_records` | 1ms | ✅ Uses `idx_sr_kind` | -| delete | `deleteGraphEdgesByFile` + `deleteGraphNodesByFile` | 0ms | ✅ Index-covered | -| _rf | Create temp file filter table | 0ms | ✅ Lightweight | -| r2n | `CREATE TEMP TABLE _r2n AS ... ROW_NUMBER() OVER ()` | 36ms | ⚠️ Window function non-indexed | -| nodes | `INSERT INTO graph_nodes SELECT ... JOIN _r2n` | 32ms | ✅ Indexed JOIN | -| edges | `INSERT INTO graph_edges SELECT ... JOIN _r2n x2` | 41ms | ⚠️ Slowest phase | -| calls | `INSERT ... 4x JOIN + semantic_records self-join` | 0ms | ⚠️ O(N²) risk on large projects | - -### Large Project Bottleneck Predictions - -| Bottleneck | Complexity | Impact | Suggestion | -|------------|:----------:|--------|------------| -| `ROW_NUMBER() OVER ()` window function | O(N) | ~1-2s for 8M rows | Cannot index, but still O(N) | -| containment edges double _r2n JOIN | O(N log N) | ~4-8s for 8M rows | Already uses DISTINCT + indexes, acceptable | -| call edges 4x JOIN + self-join | O(N²) risk | Main bottleneck | Needs `idx_sr_kind(project_id, kind)` + `idx_sr_fp_oid(file_path, original_id)` | - -### Optimization Recommendations - -1. `_r2n` `ROW_NUMBER() OVER ()` without ORDER BY is already optimal -2. Call edges `sr.name = callee.name` JOIN has no index — add `idx_sr_name(project_id, name)` to accelerate -3. Consider batch INSERT instead of row-by-row INSERT-SELECT for containment edges - ---- \ No newline at end of file diff --git a/docs/en/linux-analysis.md b/docs/en/linux-analysis.md deleted file mode 100644 index b54e900..0000000 --- a/docs/en/linux-analysis.md +++ /dev/null @@ -1,310 +0,0 @@ -# Linux Kernel Analysis with CodeScope - -> Comprehensive analysis of Linux kernel v6.13 using CodeScope's Fast Scan and Enhancement tools. -> All timing, CPU, and memory data captured from real runs (Apple M-series, macOS). - ---- - -## 1. USB Driver Model — How Mice & Keyboards Are Identified - -### Scan Results - -| Metric | Value | -|--------|-------| -| Directory | `drivers/usb/` | -| Scan time | **351.49 ms** | -| Symbols | 37,286 | -| Modules | 40 | -| Language | c | -| Peak CPU | ~100% (single core) | -| Peak memory | ~8 MB | -| Full log | `scan_usb_raw.log` | - -### Module Tree (40 subdirectories) - -```mermaid -graph TD - USB["usb/"] - USB --> CORE["core/
(27 files)"] - USB --> HOST["host/
(110 files)"] - USB --> GADGET["gadget/
(9 files)"] - GADGET --> FUNC["function/
(63 files)"] - GADGET --> UDC["udc/
(40 files)"] - USB --> SERIAL["serial/
(67 files)"] - USB --> STORAGE["storage/
(32 files)"] - USB --> TYPEC["typec/
(17 files)"] - USB --> MISC["misc/
(32 files)"] - USB --> DWC3["dwc3/
(32 files)"] - USB --> MUSBC["musb/
(28 files)"] -``` - -### How Mice vs Keyboards Are Distinguished - -The mechanism is at **`drivers/hid/usbhid/hid-core.c`** (scan: 5.58 ms, 385 symbols). - -```mermaid -flowchart TD - A["USB device insertion"] --> B["usb_hid_probe()
(hid-core.c:1004)"] - B --> C{"Check bInterfaceProtocol"} - C -->|"1 = KEYBOARD"| D["HID_TYPE_USBNONE
register EV_KEY events"] - C -->|"2 = MOUSE"| E["HID_TYPE_USBMOUSE
register EV_REL + EV_KEY"] - D --> F["hid_parse_report()"] - E --> F - F --> G["hidinput_configure_usage()
(hid-input.c:711)"] - G --> H["Map HID usage codes to input events"] - H --> I["input_register_device()"] - I --> J["User-space: input_event"] -``` - -**Key file locations:** - -| File | Line | Function | Purpose | -|------|------|----------|---------| -| `drivers/hid/usbhid/hid-core.c` | 1004 | `usb_hid_probe()` | Read `bInterfaceProtocol` | -| `drivers/hid/usbhid/hid-core.c` | 1136 | `HID_GD_MOUSE` handler | Mouse input mapping | -| `drivers/hid/usbhid/hid-core.c` | 1144 | `HID_GD_KEYBOARD` handler | Keyboard input mapping | -| `drivers/hid/hid-input.c` | 711 | `hidinput_configure_usage()` | Core HID→input translation | -| `include/linux/hid.h` | 625 | `enum hid_type` | `HID_TYPE_USBMOUSE` / `HID_TYPE_USBNONE` | - -### Entry Points Found - -| Symbol | Kind | File | Line | -|--------|------|------|------| -| `start` | function | `drivers/usb/host/sl811-hcd.c` | 303 | - ---- - -## 2. Process Scheduling — Parent-Child Resource Handling - -### Scan Results - -| Metric | Value | -|--------|-------| -| Directory | `kernel/sched/` | -| Scan time | **45 ms** | -| Symbols | 4,913 | -| Modules | 2 (sched, sched/ext) | -| Language | c | -| Enhancement | 291 ms, 4,800 call edges | - -### Scheduler Code Layout - -```mermaid -graph TD - SCHED["kernel/sched/"] - SCHED --> CORE["core.c
__schedule(), schedule()"] - SCHED --> FAIR["fair.c
CFS Completely Fair Scheduler"] - SCHED --> RT["rt.c
Real-time scheduler"] - SCHED --> DEADLINE["deadline.c
Deadline scheduler"] - SCHED --> IDLE["idle.c
Idle task"] - SCHED --> H["sched.h
Data structures"] - SCHED --> EXT["ext/
Extensible scheduler API"] -``` - -### Parent-Child Resource Flow (Copy-On-Write) - -```mermaid -flowchart TD - A["copy_process(kernel/fork.c:1994)"] --> B["dup_task_struct(current)"] - B --> C["sched_fork(clone_flags, p)"] - C --> D["copy_mm()"] - D --> E["dup_mm(mm)"] - E --> F["dup_mmap(mm, oldmm)"] - F --> G["copy_page_range(src_mm, dst_mm)
COW: share pages as read-only"] - D --> H["copy_sighand()"] - D --> I["copy_files()"] -``` - -**COW (Copy-On-Write) mechanism:** `copy_page_range()` shares physical memory pages between parent and child, marked as read-only. The first write by either process triggers a page fault, duplicating the page. - -### Preemption Prevention (Three Layers) - -``` -Layer 1: Per-task counter - preempt_count() > 0 → __schedule() returns immediately - Location: include/linux/preempt.h:92 - -Layer 2: Spinlocks - spin_lock() → preempt_disable() automatically - spin_unlock() → preempt_enable() automatically - -Layer 3: Interrupt context - Hard/soft IRQ → preempt_count incremented - Preemption blocked until handler returns -``` - -**Call path traced by CodeScope:** - -| From | To | Verified | -|------|----|----------| -| `copy_process` | `sched_fork` | ✅ `kernel/fork.c:1994 → kernel/sched/core.c:4803` | -| `copy_process` | `dup_mm` | ✅ `kernel/fork.c:1994 → kernel/fork.c:1568 → kernel/fork.c:1527` | -| `__schedule` | `pick_next_task_fair` | ⚠️ cross-file call, needs enhancement | - -### Key Source Locations - -``` -kernel/fork.c:914 dup_task_struct() — Copy process structure -kernel/fork.c:1994 copy_process() — Process creation entry -kernel/sched/core.c:7061 __schedule() — Main scheduler -include/linux/preempt.h:108 preempt_count() — Preemption counter -``` - ---- - -## 3. Memory Page Allocation - -### Scan Results - -| Metric | Value | -|--------|-------| -| Directory | `mm/` | -| Scan time | **182.93 ms** | -| Symbols | 16,111 | -| Modules | 7 | -| Language | c | - -### Page Allocator Flow - -```mermaid -flowchart TD - A["alloc_pages(gfp_mask, order)"] --> B["__alloc_pages(gfp_mask, order)
(page_alloc.c:4034)"] - B --> C["get_page_from_freelist()"] - C --> D["rmqueue()"] - D --> E["__free_one_page()
buddy merge on free"] -``` - -### Page Reclaim (Memory Pressure) - -```mermaid -flowchart TD - A["try_to_free_pages()
memory reclaim entry"] --> B["shrink_node()
per-node LRU scanning"] - B --> C["shrink_lruvec()
scan LRU lists"] - C --> D["shrink_active_list()
demote active→inactive"] - C --> E["shrink_inactive_list()
reclaim inactive pages"] -``` - -### IO Strategy: Readahead - -``` -ondemand_readahead() ← mm/readahead.c:501 - ↓ -Initial read: 4 pages (16 KB) -Sequential: double → 32 → 64 → 128 pages (512 KB max) -Random: auto-degrade, no readahead -``` - -### Key Source Locations - -``` -mm/page_alloc.c:936 __free_one_page() — Buddy merge release -mm/page_alloc.c:3401 rmqueue() — Core page allocation -mm/page_alloc.c:3792 get_page_from_freelist() — Zone selection -mm/page_alloc.c:4034 __alloc_pages() — Allocation entry -mm/readahead.c:160 read_pages() — Readahead IO dispatch -include/linux/fs.h:401 address_space_operations — Page IO vtable -``` - ---- - -## 4. Tool Chain Summary - -| Operation | Time | CPU | Memory | Tool Used | -|-----------|------|-----|--------|-----------| -| USB scan (`drivers/usb/`) | 351 ms | ~100% | ~8 MB | `codescope_scan` | -| USB HID scan (`drivers/hid/usbhid/`) | 5.58 ms | ~100% | ~4 MB | `codescope_scan` | -| Scheduler scan (`kernel/sched/`) | 45 ms | ~100% | ~6 MB | `codescope_scan` | -| Memory scan (`mm/`) | 183 ms | ~100% | ~8 MB | `codescope_scan` | -| Scheduler enhancement | 291 ms | ~100% | ~12 MB | `codescope_enhance` | -| Full kernel/ enhancement | 27 s | ~100% | ~30 MB | `codescope_enhance` | -| `find_symbol("usb_register")` | ~15 µs | — | — | `codescope_find_symbol` | -| `trace_path(copy_process, sched_fork)` | <1 ms | — | — | `codescope_trace` | - -**Total analysis time (all scans combined):** ~800 ms -**Total peak memory:** ~30 MB -**Token savings vs reading raw source:** ~98.9% - ---- - -## 5. Full Kernel Index (codebase-memory-mcp) - -### Index Results (89,465 files) - -| Metric | Value | -|--------|-------| -| Files discovered | **89,465** | -| File discovery | **2,336 ms** | -| **Total nodes** | **4,877,492** | -| **Total edges** | **9,326,238** | -| **Database size** | **7.06 GB** | -| **Index time** | **183 s (≈3 min)** | -| **Peak memory** | **11.6 GB** | -| Parallel workers | 14 | -| Throughput | 489 files/s, 50,966 edges/s | - -### Phase Timing Breakdown - -| Phase | Time | Description | -|-------|:----:|-------------| -| `parallel_extract` | **60.4 s** | Parse 89K files, extract tokens and symbols | -| `parallel_resolve` | **38.9 s** | Resolve call relations, usages, semantic edges | -| `dump` (DB persist) | **23.8 s** | Write to SQLite database | -| `semantic_edges` | **14.5 s** | Semantic similarity edges | -| `registry_build` | **11.1 s** | Build symbol registry | -| `similarity` | 1.8 s | Function fingerprint similarity | -| Other phases | 3.5 s | tests/k8s/githistory/decorator/configlink | -| **Total** | **~183 s** | | - -### Memory Profile - -``` -Peak: 11,644 MB (during parallel_extract — 14 workers holding ASTs) -Stable: 7,500 MB (post-extract, during resolve and registry phases) -Budget: 18,432 MB (per-worker: 1,316 MB) -``` - -### Disk Usage - -``` -SQLite DB: 7.06 GB (~/.cache/codebase-memory-mcp/) -Cache: 6.70 GB (total indexed projects) -─── -Total: ~14 GB -``` - -### Performance Scaling - -| Scope | Files | Nodes | Time | Memory | -|-------|:----:|:-----:|:----:|:------:| -| Single file | 1 | 31 | 4.94 ms | ~50 MB | -| kernel/sched/ | 51 | 3,854 | 841 ms | ~500 MB | -| kernel/ | 697 | 33,546 | 2,302 ms | ~1.5 GB | -| **Full kernel** | **89,465** | **4,877,492** | **183 s** | **11.6 GB** | - -## 6. CodeScope MCP Analysis Examples - -The following analyses were conducted using **CodeScope's own MCP tools** — the project's MCP server was configured in `.mcp.json` and used to query the Linux kernel source code. - -### Example 1: USB HID Device Identification - -**Tools used:** `index_file`, `find_definition`, `graph_query ❌` (MATCH (Function)-[Calls]->(Function)) - -Key symbols located via CodeScope MCP: - -| Symbol | File | Line | CodeScope Tool | -|--------|------|:----:|---------------| -| `usbhid_probe` | `hid-core.c` | 1363 | `find_definition` | -| `usbhid_parse` | `hid-core.c` | 982 | `find_definition` | -| `usb_mouse_probe` | `usbmouse.c` | 106 | `find_definition` | -| `usb_kbd_probe` | `usbkbd.c` | 261 | `find_definition` | -| `hid_usb_ids` | `hid-core.c` | 1678 | `find_definition` | - -Call graph (via `graph_query ❌`): 45 Call edges discovered across the USB HID driver chain. - -Full analysis: `Runtimelog/scan_usb_hid_analysis.log` - -### Example 2: Process Scheduling & COW Resource Management - -**Tools used:** `find_definition`, `index_file` on kernel/sched/core.c, kernel/fork.c, kernel/exit.c - -Full analysis: `Runtimelog/scan_linux_scheduler.log` diff --git a/docs/en/skills.md b/docs/en/skills.md index 9a13202..8b709b8 100644 --- a/docs/en/skills.md +++ b/docs/en/skills.md @@ -73,7 +73,7 @@ graph TB Client["Claude Desktop / Cursor
Custom MCP Client"] end subgraph "Rust MCP Server (codescope)" - TD["Tool Dispatch
(35+ tools)"] + TD["Tool Dispatch
(46 tools)"] TQ["Task Queue
(Tokio async)"] FFI["FFI (extern \"C\")"] end @@ -85,9 +85,10 @@ graph TB LS["LSP Client"] end subgraph "SQLite (WAL)" - N["graph_nodes
(nodes)"] - E["graph_edges
(edges)"] - SR["semantic_records
(IR + FTS)"] + N["entity
(symbols)"] + E["relation
(edges)"] + SR["semantic_records
(IR)"] + FTSN["code_fts + name_trgm
(FTS5)"] end Client -->|JSON-RPC 2.0| TD TD --> TQ @@ -115,7 +116,7 @@ flowchart LR ## 3. MCP Tools (38 tools) -### Index (索引) +### Index | Tool | Purpose | Input | Output | Token | |------|---------|-------|--------|-------| @@ -123,7 +124,7 @@ flowchart LR | `index_file` | Index a single file | `file_path` | JSON: file result | ~30 | | `search` | Unified search (FTS) | `query`, `limit?` | JSON: matched results | ~30 | -### Locate (查位置) +### Locate | Tool | Purpose | Input | Output | Token | |------|---------|-------|--------|-------| @@ -134,7 +135,7 @@ flowchart LR | `trace_flow` | Trace call path (BFS) | `function_name`, `depth` | call chain | ~50-200 | | `search_code` | Full-text search | `query`, `limit?` | matched lines | ~30 | -### Understand (理解项目) +### Understand | Tool | Purpose | Input | Output | Token | |------|---------|-------|--------|-------| @@ -145,7 +146,7 @@ flowchart LR | `get_entry_points` | Entry points (main/init/run) | (none) | entry functions | ~5 | | `get_graph_stats` | Graph statistics | (none) | nodes, edges, files | ~10 | -### Verify (验证) +### Verify | Tool | Purpose | Input | Output | Token | |------|---------|-------|--------|-------| @@ -181,7 +182,7 @@ Find entry points → get_entry_points (5 tok) # Find hotspots → ❌ get_hotspots not implemented Search code → search (300-1000 tok) Trace calls → find_callers / find_callees (10-50 tok) -Architecture → get_module_tree (4 tok) + optional get_communities ❌ (engine impl exists, no MCP tool) (1K-200K tok) +Architecture → get_module_tree (4 tok) + optional get_communities ✅ (label propagation, summary-first by default) Change impact → detect_changes (100-500 tok) AI Q&A → codescope_build_context (200-1000 tok) ``` @@ -195,7 +196,7 @@ AI Q&A → codescope_build_context (200-1000 tok) | `CODESCOPE_DB_PATH` | `.codescope/codescope.db` | SQLite database path | | `GRAMMARS_DIR` | `grammars/` | tree-sitter grammar .so directory | | `CODESCOPE_LSP` | (unset) | LSP server command for type enhancement | -| `CODESCOPE_INDEX_MODE` | `normal` | Index mode: `fast` / `normal` / `strict`（见下方 Index Modes） | +| `CODESCOPE_INDEX_MODE` | `normal` | Index mode: `fast` / `normal` / `deep`, plus the discovery-only `strict` (see Index Modes below) | | `CODESCOPE_WORKERS` | `min(hw,4)` | Parse worker thread count | | `CODESCOPE_SKIP_ASYNC` | (unset) | Set to 1 to skip async model/state/FTS stages (set by the parallel scheduler path) | | `CODESCOPE_PROFILE_RESOLVER` | (unset) | Enable resolver per-phase timing (`[module=resolver, method=run]` breakdown) | @@ -217,7 +218,7 @@ Set via `CODESCOPE_INDEX_MODE`（filter_policy.h enum NORMAL/FAST/STRICT — the | `normal` | `NORMAL` | base skip table only (build/dist/out/target/test/docs/vendor/node_modules/.venv ...) | ✅ | Default | | `strict` | `STRICT` | base skip + detectLanguage whitelist gate (only source files of recognized languages) | ✅ | Most restrictive, leanest data | -> Known issue (fixed 2026-08-11): `fast` used to be ≈ `normal` — `fast_extra_skip_dirs_` was an empty reserved set and `setMode()` did not rebuild `active_skip_dirs_`. See `docs/optimization/perf-full-index-2026-08-11.md` §9/§10. +> Known issue (fixed 2026-08-11): `fast` used to be ≈ `normal` — `fast_extra_skip_dirs_` was an empty reserved set and `setMode()` did not rebuild `active_skip_dirs_`. See `docs/optimization/perf-full-index-2026-08-11.zh.md` §9/§10. ## 7. Performance Benchmarks @@ -272,6 +273,10 @@ make bench-full # Full benchmark ## 10. Supported Languages +**Vendored grammars — parsed end-to-end.** These are the 8 languages in the +README table; each has a visitor (the live IR pipeline) covered by +`engine/tests/test_ir_edge_coverage.cpp`. + | Language | Extension | Parser | |----------|-----------|--------| | Python | `.py` | tree-sitter-python | @@ -283,7 +288,16 @@ make bench-full # Full benchmark | C | `.c`, `.h` | tree-sitter-c | | C++ | `.cpp`, `.cc`, `.cxx`, `.hpp`, `.hxx` | tree-sitter-cpp | | Java | `.java` | tree-sitter-java | + +**Detected by extension, NOT parsed — no grammar is vendored.** These files are +discovered (so they appear in the candidate count) and then reported as +`language_missing` parse failures in `parse_failures`; they are never indexed, +are re-attempted on every run and never count towards +`CODESCOPE_FAIL_RETRY_MAX`. + +| Language | Extension | Grammar (not vendored) | +|----------|-----------|------------------------| | Kotlin | `.kt`, `.kts` | tree-sitter-kotlin | | Ruby | `.rb` | tree-sitter-ruby | | Scala | `.scala` | tree-sitter-scala | -| Swift | `.swift` | tree-sitter-swift | +| Swift | `.swift` | tree-sitter-swift — its `parser.c` is ABI-incompatible with the vendored tree-sitter core, so it is absent from `GRAMMAR_SOURCES` and the Swift visitor/translator were removed with it (see `engine/src/parser/parser.cpp`) | diff --git a/docs/en/technical_breakdown.md b/docs/en/technical_breakdown.md index b74ba23..8d059e0 100644 --- a/docs/en/technical_breakdown.md +++ b/docs/en/technical_breakdown.md @@ -1,6 +1,6 @@ # CodeScope Technical Breakdown -> **Version**: v0.2.1 | **Last Updated**: 2026-07-15 +> **Version**: v0.2.7 | **Last Updated**: 2026-10-09 This document provides a deep dive into CodeScope's internal architecture, design decisions, and implementation details. It's intended for developers who want to understand how the system works, contribute to it, or integrate with it. @@ -15,7 +15,7 @@ This document provides a deep dive into CodeScope's internal architecture, desig 5. [Resolver Pipeline](#5-resolver-pipeline) 6. [Verification System](#6-verification-system) 7. [Performance Characteristics](#7-performance-characteristics) -8. [LadybugDB Integration](#8-ladybugdb-integration) +8. [LadybugDB Integration (removed)](#8-ladybugdb-integration-removed) 9. [How to Extend](#9-how-to-extend) --- @@ -36,7 +36,7 @@ flowchart TB MCPServer["MCP Server (JSON-RPC 2.0)"] FFIBridge["FFI Bridge (Rust ↔ C++)"] TokioRT["Tokio Runtime (Background Tasks)"] - tools["35+ MCP Tools
Locate / Understand / Verify / Index"] + tools["46 MCP Tools
Locate / Understand / Verify / Index"] end subgraph "C++ Worker Subprocess" @@ -44,12 +44,11 @@ flowchart TB Parser["tree-sitter Parser (8 languages)"] Resolver["Resolver Pipeline"] GraphBuilder["Graph Builder"] - Store["SQLite + LadybugDB Writer"] + Store["SQLite Writer"] end subgraph "Storage" SQLite["SQLite DB (WAL mode)"] - Ladybug["LadybugDB (.lbug)"] end Client -->|"MCP stdio"| MCPServer @@ -57,7 +56,6 @@ flowchart TB MCPServer -->|"FFI (read-only)"| FFIBridge FFIBridge -->|"poll progress"| SQLite Worker -->|"write"| SQLite - Worker -->|"sync"| Ladybug TokioRT -->|"async FTS build"| SQLite ``` @@ -79,13 +77,13 @@ flowchart LR B --> C["IR (SemanticUnit)"] C --> D["SQLite"] D --> E["Graph Builder"] - E --> F["LadybugDB"] + E --> F["SQLite: entity + relation"] end subgraph "Query Flow" G["MCP Tool"] --> H["FFI Bridge"] H --> I["C++ Query Engine"] - I --> J["SQLite / LadybugDB"] + I --> J["SQLite (entity / relation)"] J --> K["JSON Response"] end ``` @@ -108,7 +106,6 @@ flowchart LR | `filter_policy.cpp` | — | Dir/file/suffix filtering + .gitignore matching | | `store/store.cpp` | ~3,060 | SQLite storage + FTS + progress tracking | | `store/store_core.cpp` | — | Core CRUD operations | -| `store/store_ladybug.cpp` | 365 | LadybugDB sync (CSV → COPY FROM) | | `ir/` | — | IR types (SemanticUnit, Record, Reference) | | `parser/` | — | tree-sitter wrappers per language | | `graph/` | — | Graph builder + call chain resolution | @@ -245,7 +242,7 @@ flowchart TB subgraph "MCP Protocol Layer" Transport["transport.rs
stdio transport
1MB read line limit"] - Server["server.rs
Request dispatch + routing
35+ tools registered"] + Server["server.rs
Request dispatch + routing
46 tools registered"] end Main --> MCP @@ -259,7 +256,7 @@ flowchart TB The MCP server implements the [Model Context Protocol](https://modelcontextprotocol.io/) over stdio: -- **`tools/list`**: Returns the list of available tools (35+ tools) +- **`tools/list`**: Returns the list of available tools (46 tools) - **`tools/call`**: Invokes a tool by name with arguments - **Transport**: JSON-RPC 2.0 over stdin/stdout - **Error handling**: Parse errors return `ReadResult::ParseError` instead of crashing (1MB read line limit) @@ -305,15 +302,19 @@ The primary storage is SQLite with WAL mode for concurrent reads. The schema has ```mermaid flowchart TB - subgraph "Core Tables" - GN["graph_nodes
All symbols
~17k-200k rows"] --> GE["graph_edges
Relationships
~3k-50k rows"] - ENT["entity
Production code only
~10k-150k rows"] --> REL["relation
Production code only
~2k-40k rows"] + subgraph "Core Tables (canonical)" + ENT["entity
Symbols
~10k-200k rows"] --> REL["relation
Edges
~2k-50k rows"] + end + + subgraph "Legacy (deprecated, read by no query)" + GN["graph_nodes
not populated on the scheduler paths"] + GE["graph_edges
still mirrored by the resolver"] end subgraph "Auxiliary Tables" REF["reference
Unresolved references"] SCOPE["scope
Scope chains"] - FTS["search_index
FTS5 full-text search"] + FTS["code_fts + name_trgm
FTS5 full-text search"] VEC["node_vectors
Vector embeddings"] end @@ -322,29 +323,30 @@ flowchart TB PROJ["project
Metadata + readiness flags"] end - GN --> FTS - GN --> VEC - ENT --> REL + ENT --> FTS + ENT --> VEC ``` **Indexes:** All queries use index-only scans: -- `idx_sr_kind_name` on `(kind, name)` — symbol lookup -- `idx_sr_fp_parent` on `(file_path, parent_id)` — tree traversal -- `idx_ge_source` on `(source_node_id)` — call edge lookup -- `idx_ge_target` on `(target_node_id)` — callee lookup +- `idx_entity_name` on `entity(project_id, name)` — symbol lookup +- `idx_entity_file` on `entity(project_id, file_path)` — per-file re-index and delete +- `idx_relation_source` on `relation(project_id, source_id)` — callee lookup +- `idx_relation_target` on `relation(project_id, target_id)` — caller lookup ### 4.2 Dual-Write Strategy -During v0.1.3 migration, production-code symbols are dual-written to both: -- Legacy `graph_nodes`/`graph_edges` tables -- New `entity`/`relation` tables +The dual write that came with the v0.1.3 migration was retired for **symbols**: +they exist only in `entity`. It survives for **edges**, where the resolver still +mirrors every resolved edge into the legacy `graph_edges` beside the canonical +`relation` (`engine/src/resolver/pipeline_flush.cpp`). -This enables: -- Backward compatibility with existing queries -- Gradual migration without downtime -- A/B comparison for correctness verification +Measured on a freshly indexed project (goagent, 1,579 files, 2026-10-01): +`entity` 24,545 rows, `relation` 7,374, `graph_edges` 7,377, `graph_nodes` 0. No +query reads either legacy table (`engine/src/engine_queries_context.cpp` records +the switch to the canonical ones), so the edge mirror costs disk and nothing +else — but it is why the two edge tables can drift apart by a few rows. ### 4.3 String Interning @@ -560,118 +562,18 @@ Compares code structure against documentation: 1. **tree-sitter parse time**: Dominant factor for large files (>10k lines). Parsing is single-threaded per file. 2. **SQLite write throughput**: ~80k rows/s is the limit for WAL mode on consumer SSDs. 3. **FTS build time**: Linear in number of symbols. For 150k symbols, takes ~30s. -4. **LadybugDB COPY FROM**: CSV import is fast but requires a full sync — incremental sync not yet implemented. --- -## 8. LadybugDB Integration - -### 8.1 Overview - -[LadybugDB](https://ladybugdb.com/) is an embedded graph database that CodeScope uses as an optional secondary storage backend. It provides: - -```mermaid -flowchart LR - subgraph "SQLite" - SQL["Relational queries
Reliable, zero-config"] - end - - subgraph "LadybugDB" - LBUG["Graph-native queries (Cypher)
Faster multi-hop traversal
Visualization support"] - end - - SQL -->|"CSV export → COPY FROM sync"| LBUG - LBUG -->|"LadybugDB Explorer
Interactive graph exploration"| VIZ["Visualization"] -``` - -### 8.2 Schema - -LadybugDB stores the same data as SQLite but in a graph-native format: - -```cypher -// Node table (mirrors graph_nodes) -CREATE NODE TABLE IF NOT EXISTS GraphNode ( - id INT64 PRIMARY KEY, - project_id INT64, - ir_node_id INT64, - node_type INT32, - name STRING, - qualified_name STRING, - signature STRING, - module_path STRING, - file_path STRING, - language STRING, - start_row INT32, - start_col INT32, - end_row INT32, - end_col INT32, - parent_id INT64, - is_entry_point BOOL, - embedding_ready BOOL, - metrics_ready BOOL -); - -// Call edges (from caller to callee) -CREATE REL TABLE IF NOT EXISTS CALLS ( - FROM GraphNode TO GraphNode, - project_id INT64, - edge_type INT32, - call_site_line INT32, - label STRING -); +## 8. LadybugDB Integration (removed) -// Generic relations -CREATE REL TABLE IF NOT EXISTS RELATES ( - FROM GraphNode TO GraphNode, - project_id INT64, - type INT32 -); -``` - -### 8.3 Sync Mechanism - -Data is synced from SQLite to LadybugDB via CSV export + COPY FROM: - -```mermaid -sequenceDiagram - participant SQL as SQLite - participant CSV as CSV File - participant LBUG as LadybugDB - - SQL->>CSV: Export graph_nodes → nodes.csv - SQL->>CSV: Export graph_edges → edges.csv - CSV->>LBUG: COPY GraphNode FROM 'nodes.csv' (header=false) - CSV->>LBUG: COPY CALLS FROM 'edges.csv' (header=false) - Note over LBUG: Sync complete ✓ -``` - -The sync is triggered via `codescope_export_graph` or the `syncGraphToLadybugDB()` C++ function. - -### 8.4 Querying LadybugDB - -```bash -# Open the LadybugDB shell -lbug /path/to/project/.codescope/codescope.lbug - -# Count all nodes -MATCH (n:GraphNode) RETURN count(n); - -# Find all Go functions -MATCH (n:GraphNode) -WHERE n.language = 'go' -RETURN n.name, n.file_path, n.start_row; - -# Find callers of a specific function -MATCH (n:GraphNode)-[c:CALLS]->(m:GraphNode) -WHERE m.name = 'targetFunction' -RETURN n.name, c.call_site_line; - -# Find the shortest call path between two functions -MATCH p = shortestPath( - (a:GraphNode {name: 'funcA'})-[*..10]->(b:GraphNode {name: 'funcB'}) -) -RETURN p; -``` +CodeScope used to mirror the graph into [LadybugDB](https://ladybugdb.com/), an +embedded graph database, as an optional acceleration layer. **It was removed**: +the tree has no `store_ladybug*.cpp`, `server/build.rs` does not link `liblbug`, +and no `.codescope/codescope.lbug` file is produced any more — the backend is +pure SQLite (the same note appears in `README.md` §7). The design and the +integration steps are kept in git history; every graph and search tool now reads +the SQLite tables described in §4. --- @@ -759,10 +661,14 @@ class MyInspector : public Inspector { - **Async runtime**: Tokio provides lightweight async tasks for FTS building and progress polling. - **Ecosystem**: MCP client libraries, serde for JSON, cargo for dependency management. -### Why SQLite + LadybugDB? +### Why SQLite, and not an embedded graph database? - **SQLite**: Universal, zero-config, battle-tested. Every system has SQLite support. -- **LadybugDB**: Graph-native queries for multi-hop traversals. Cypher is more expressive than recursive SQL for graph patterns. -- **Dual strategy**: SQLite for reliability and portability, LadybugDB for performance and visualization. +- **An embedded graph DB was tried and removed** (LadybugDB, §8): it added a linked + dependency and a second store to keep in sync, while the graph workloads CodeScope + actually runs — caller/callee lookups, BFS over the call graph, hotspots — are served + by indexed `relation` lookups plus the CSR adjacency blob. +- **One store**: every tool reads the same tables, so the graph and the index cannot + disagree. ### Why two processes? - **Crash isolation**: A parser segfault doesn't kill the MCP server. diff --git a/docs/en/timbre_model_analysis.md b/docs/en/timbre_model_analysis.md deleted file mode 100644 index 86df528..0000000 --- a/docs/en/timbre_model_analysis.md +++ /dev/null @@ -1,150 +0,0 @@ -# InstrumentTimbre Model Architecture Analysis - -> Project: Timbre Recognition/Conversion Model Based on ARES -> Indexing Time: 0.94s | 142 .py files | Date: 2026-07-06 - ---- - -## 1. Index Log - -``` -Start Time: Monday, July 6, 2026 20:59:53 - -worker: project=0 starting index_project dir=～/code/AIproject/music_ai/InstrumentTimbre lang= -BATCH [0..99] of 142 (100 files) -BATCH [0..99] done, total indexed: 100 -BATCH [100..141] of 142 (42 files) -BATCH [100..141] done, total indexed: 142 -POST_BUILD: symbol graph... -buildGraph: 142 files | file_list=3ms delete=1ms rf=0ms r2n=54ms - nodes=46ms edges=92ms calls=0ms total=199ms -{"ok":true,"files_indexed":142,"workers":14, - "time_parse_ms":21,"time_sqlite_ms":234, - "time_buildgraph_ms":199,"time_fts_ms":92,"time_vector_ms":0, - "discovery":{"seen_dirs":296747,"seen_files":293437, - "skipped_dirs":3270,"skipped_files":0, - "skipped_suffix":293213,"candidate_files":142}} - -End Time: Monday, July 6, 2026 20:59:54 -Total Time: 0.94s -``` - -### About the ./data Directory - -The `.gitignore` already contains a `data/` rule, which FilterPolicy reads and automatically skips. From the discovery statistics: -- **293,437 files scanned** (of which 293,213 were filtered by suffix — the majority being audio data under data/) -- **142 candidate files** (only .py source code) -- Clean indexing phase with no noise - ---- - -## 2. Model Architecture Overview - -```mermaid -flowchart TB - subgraph "Input" - AUDIO["Audio Input
(.wav files)"] - end - subgraph "Encoder (encoders.py)" - W2V["wav2vec2 Pretrained
(Optional, 768-dim)"] - CNN["Multi-scale CNN
3×3 / 5×5 / 7×7 Parallel
MultiScaleExtractor"] - SA["Self-Attention
(Conv1d Q/K/V)"] - MHA["Multi-Head Attention
(8 heads)"] - POOL["Adaptive Pooling
→ 128-dim Feature Vector"] - end - subgraph "Decoder (decoders.py)" - FC["Fully Connected Layers
128→256→num_classes"] - end - subgraph "Attention Mechanisms (attention.py)" - ATTN["SelfAttention
FeatureAttention
CrossAttention
TimbreTransformer
HybridAttention"] - end - - AUDIO --> W2V - AUDIO --> CNN - W2V --> ADAPTER["Adapter MLP
768→512→256→128"] - CNN --> SA - SA --> MHA - ADAPTER --> MHA - MHA --> POOL - POOL --> FC -``` - ---- - -## 3. Core Components - -### 3.1 Encoder (`models/encoders.py`) - -**InstrumentTimbreEncoder** — Main encoder that converts audio to 128-dimensional timbre feature vectors. - -#### Dual-Path Architecture - -| Path | Input | Processing | Output Dimension | -|------|-------|------------|------------------| -| **wav2vec2** (optional) | Raw audio | `torchaudio Wav2Vec2Base → Adapter MLP(768→512→256→128)` | 128 | -| **Multi-scale CNN** (default) | Spectrogram | `Conv2d 3×3 / 5×5 / 7×7 parallel → Concatenate → Self-Attention → 8-head attention → Pooling` | 128 | - -```python -class InstrumentTimbreEncoder(nn.Module): - def __init__(self, input_channels=1, output_dim=128, use_pretrained=False): - self.multi_scale_extractor = nn.ModuleDict({ - 'scale_3': nn.Conv2d(input_channels, 64, 3, padding=1), # 3×3 convolution - 'scale_5': nn.Conv2d(input_channels, 64, 5, padding=2), # 5×5 convolution - 'scale_7': nn.Conv2d(input_channels, 64, 7, padding=3), # 7×7 convolution - }) - # Followed by: residual connection → Self-Attention → multi-head attention → adaptive pooling → 128-dim -``` - -### 3.2 Decoder (`models/decoders.py`) - -| Decoder | Architecture | Purpose | -|---------|--------------|---------| -| InstrumentTimbreDecoder | `Linear(128, 256) → ReLU → Dropout → Linear(256, num_classes)` | Basic classification | -| EnhancedTimbreDecoder | Multi-layer MLP + LayerNorm + GELU + Dropout | High-quality classification | - -### 3.3 Attention Mechanisms (`models/attention.py`) - -| Module | Principle | Input→Output | -|--------|-----------|--------------| -| **SelfAttention** | Conv1d Q/K/V projection → dot-product attention → residual connection | `[B,C,L] → [B,C,L]` | -| **FeatureAttention** | Fully-connected layer attention weights | `[B,D] → [B,D]` | -| **CrossAttention** | Cross-attention between two sequences | `[B,C,L1],[B,C,L2] → [B,C,L2]` | -| **TimbreTransformer** | Transformer encoder + LayerNorm | `[B,D,L] → [B,D,L]` | -| **HybridAttention** | Self + Cross hybrid | Combined | - -SelfAttention implementation details: -```python -class SelfAttention(nn.Module): - def __init__(self, in_channels): - self.query = Conv1d(in_channels, in_channels//8, 1) # Q projection - self.key = Conv1d(in_channels, in_channels//8, 1) # K projection - self.value = Conv1d(in_channels, in_channels, 1) # V projection - self.gamma = Parameter(torch.zeros(1)) # Learnable residual weight - - def forward(self, x): - energy = bmm(proj_query.permute(0,2,1), proj_key) # [B,L,L] attention matrix - attention = softmax(energy) - out = bmm(proj_value, attention.permute(0,2,1)) - return self.gamma * out + x # Weighted residual -``` - ---- - -## 4. Technology Stack - -| Component | Purpose | -|-----------|---------| -| PyTorch | Deep learning framework | -| torchaudio | Audio loading + wav2vec2 pretrained models | -| librosa | Audio feature extraction (MFCC, spectrogram, etc.) | -| wav2vec2 | Self-supervised pretrained speech model (optional) | - -## 5. Architecture Features - -1. **Dual-Path Encoding**: wav2vec2 pretrained + multi-scale CNN parallel extraction, fused into 128-dimensional timbre vectors -2. **Multi-Scale Features**: 3×3 / 5×5 / 7×7 CNN kernels in parallel to capture acoustic features at different granularities -3. **Attention Mechanisms**: Self-Attention via Conv1d + 8-head multi-head attention + optional Transformer -4. **Residual Connections**: Each layer has dimension-adaptive residual connections (`ResidualConnection`) -5. **GELU Activation**: Replaces traditional ReLU for more stable training -6. **LayerNorm**: Replaces BatchNorm, suitable for variable-length audio inputs -7. **Output**: 128-dimensional timbre feature vectors for classification, similarity matching, or timbre conversion \ No newline at end of file diff --git a/docs/en/token_savings_report.md b/docs/en/token_savings_report.md deleted file mode 100644 index 21883e7..0000000 --- a/docs/en/token_savings_report.md +++ /dev/null @@ -1,99 +0,0 @@ -# Token Savings Report - -> CodeScope vs. reading raw source files — and vs. codebase-memory-mcp — measured across real analysis scenarios. -> Hardware: Apple M3 Max, 36 GB RAM. OS: macOS. - -## Why Token Savings Matter - -When an LLM needs to understand code, it has two options: -1. **Read raw source files** — thousands of lines of code, mostly irrelevant context -2. **Query CodeScope** — get exactly the structured knowledge needed (symbols, graphs, metrics) - -CodeScope reduces token consumption by ~98.9% on average vs raw source, and by ~85% vs codebase-memory-mcp. - -## Measured Savings by Scenario - -### vs Raw Source Files - -| Scenario | CodeScope (tokens) | Raw Source (tokens) | Savings | -|----------|-------------------|---------------------|---------| -| Find function definition | ~21 | ~2,265 | **99.1%** | -| Trace callers of a function | ~18 | ~2,000 | **99.1%** | -| Project module tree (`get_module_tree`) | ~4 | ~1,875 | **99.8%** | -| Project overview (project_overview) | ~71 | ~1,875 | **96.2%** | -| Function complexity analysis | ~43 | ~4,733 | **99.1%** | -| Symbol search by name | ~23 | ~958 | **97.6%** | -| USB driver subsystem overview | ~250 | ~24,000 | **99.0%** | -| Linux kernel scheduler analysis | ~180 | ~15,000 | **98.8%** | -| **Average** | **~76** | **~6,744** | **98.9%** | - -### vs codebase-memory-mcp - -| Query | CodeScope (tokens) | codebase-memory-mcp (tokens) | Savings | -|-------|:------------------:|:---------------------------:|:-------:| -| Graph statistics | **~18** | ~5,012 (via get_architecture) | **99.6%** | -| Hotspots Top 10 | **~488** | ~5,012 (included in arch report) | **90.3%** | -| Search "handler" | **~372** | ~830 (search_graph) + ~828 (search_code) | **55.2%** | -| Entry points | **~5** | ~5,012 (included in arch report) | **99.9%** | -| Module tree | **~4** | ❌ not supported | — | -| Community detection (default) | **~1K-50K** | ❌ not supported | — | -| Caller query | **~7** | — | — | -| **Typical analysis combo** | **~910** | **~6,048** | **85.0%** | - -## Real-World Benchmarks - -### Project Indexes (Final Optimized) - -| Project | Files | Index Time | Nodes | Edges | DB Size | -|---------|:----:|:----------:|:-----:|:-----:|:-------:| -| ARES Agent (Go) | 95 | **0.31s** | 24,924 | 23,184 | ~250 MB | -| memscope-rs (Rust+C) | 238 | **2.36s** | 123,270 | 108,905 | ~1.2 GB | -| GoAgent (1,167 Go) | 1,167 | **2.89s** | 261,743 | 244,078 | ~2.8 GB | -| Linux kernel 6.14.7 (partial) | 6,173 | **~300s** (timeout) | — | — | 12 GB | - -### Query Token Cost (Per-Tool Reference) - -| Tool | Token Cost | When to Use | -|------|:----------:|-------------| -| `get_graph_stats` | **~18** | Quick project size check | -| `get_module_tree` | **~4** | Module structure overview | -| `get_entry_points` | **~5** | Find entry points | -| `find_callers` / `find_callees` | **~10-50** | Call graph tracing | -| `get_project_info` | **~44** | Project metadata | -| `project_overview` | **~71** | First step on new project | -| `find_definition` / `find_references` | **~20-30** | Symbol location | -| `get_complexity` | **~20** | Complexity metrics | -| `locate_code ❌` | **~50-300** | Source context | -| `codescope_capabilities` | **~309** | Feature readiness | -| `search` | **~300-1000** | Code search | -| `codescope_trace` | **~50-200** | Call path tracing | -| `detect_changes` | **~100-500** | Change impact | -| `get_hotspots` | ❌ Not implemented — no MCP tool | — | -| `codescope_build_context` | **~200-1000** | AI context bundle | -| `get_communities` (default) | ❌ Engine impl exists, no MCP tool | — | - -### Typical Analysis Pipeline - -``` -Analysis pipeline: project_overview + entry_points + hotspots + search + modules -CodeScope: 71 + 5 + 500 + 1000 + 4 = ~1,580 tokens -codebase-memory: get_architecture(5012) + search_graph(830) + query_graph(206) = ~6,048 tokens - -CodeScope saves ~74% over codebase-memory-mcp for a complete analysis. -CodeScope saves ~98.9% over reading raw source files for the same analysis. -``` - -## Why It Works - -CodeScope pre-indexes the codebase into a structured knowledge base (SQLite with 15+ tables). When an LLM asks a question, CodeScope returns only the **relevant facts** — not the entire source file. - -``` -Source Code → CodeScope (Facts + Index + Graph + Metrics) → LLM gets only relevant context - ↓ - ~98.9% fewer tokens vs raw - ~85% fewer tokens vs codebase-memory -``` - -## Methodology - -Token count estimated using DeepSeek formula: `tokens = ASCII_chars × 0.3 + non-ASCII_chars × 0.6`. CodeScope output counted from actual JSON responses. Raw source estimated at 4 chars/token for code. codebase-memory-mcp output counted from actual JSON responses via fast mode indexing. diff --git a/docs/en/why_codescope.md b/docs/en/why_codescope.md index caeb849..d75150d 100644 --- a/docs/en/why_codescope.md +++ b/docs/en/why_codescope.md @@ -542,7 +542,7 @@ index_project → poll get_index_progress ``` Index returns (1s) → FTS builds asynchronously (background) -Before FTS ready → auto-fallback to graph_nodes.name LIKE search +Before FTS ready → auto-fallback to entity.name LIKE search After FTS ready → auto-switch to FTS5 full-text search ``` diff --git a/docs/optimization/optimization-english.md b/docs/optimization/optimization.md similarity index 95% rename from docs/optimization/optimization-english.md rename to docs/optimization/optimization.md index 9f66ff5..36cd57a 100644 --- a/docs/optimization/optimization-english.md +++ b/docs/optimization/optimization.md @@ -436,6 +436,19 @@ ORDER BY COUNT(*) DESC LIMIT 10; | **Layer pairs** | Correct | **Correct** | | Precision | Theoretically more precise | Practically equivalent (architecture_edge is pre-validated) | +> **Superseded (2026-09-18):** this table records the trade-offs of that +> optimization as they stood at the time. `architecture_edge` was subsequently +> found NOT to be layer-validated — `ArchitecturePlugin` writes a row for any +> cross-module call, and `layer_lower` / `layer_upper` hold module NAMES, not +> layer names. (The columns were renamed to `callee_module` / `caller_module` +> in v0.7 — see the migration in `store_schema_migrations.cpp` — so the SQL +> snippets above use the names that existed when the analysis was written.) +> layer names. The "Violation count" and "Compliance flag" rows above therefore +> no longer describe current behaviour: a normal cross-module dependency is not +> counted as a violation, and the count now lives in `cross_module_edges`. See +> `CHANGELOG.md`, "Normal cross-module dependencies are no longer reported as +> architecture violations". + ### Index support ```sql diff --git a/docs/optimization/optimization-chinese.md b/docs/optimization/optimization.zh.md similarity index 96% rename from docs/optimization/optimization-chinese.md rename to docs/optimization/optimization.zh.md index 80491fb..8960093 100644 --- a/docs/optimization/optimization-chinese.md +++ b/docs/optimization/optimization.zh.md @@ -449,6 +449,16 @@ ORDER BY COUNT(*) DESC LIMIT 10; | **层对** | 正确 | **正确** | | 精度 | 理论更精确 | 实际等价（architecture_edge 已验证） | +> **已失效（2026-09-18）**：本表记录的是当时那次优化的取舍。后续核实发现 +> `architecture_edge` **并不**经过层校验 —— `ArchitecturePlugin` 对任意跨模块 +> 调用都写一行，`layer_lower` / `layer_upper` 存的是**模块名**而非层名。因此上表 +> （这两列在 v0.7 已改名为 `callee_module` / `caller_module`，见 +> `store_schema_migrations.cpp` 中的迁移；上方 SQL 保留的是分析当时的列名。） +> 的 `violations` 计数与"合规性标志"两行已不再描述当前行为：正常跨模块依赖不再 +> 计为违规，计数改由 `cross_module_edges` 承载。详见 `CHANGELOG.md` 的 +> "Normal cross-module dependencies are no longer reported as architecture +> violations"。 + ### 索引支持 ```sql diff --git a/docs/optimization/perf-full-index-2026-08-11.md b/docs/optimization/perf-full-index-2026-08-11.zh.md similarity index 99% rename from docs/optimization/perf-full-index-2026-08-11.md rename to docs/optimization/perf-full-index-2026-08-11.zh.md index 414b955..2ac0428 100644 --- a/docs/optimization/perf-full-index-2026-08-11.md +++ b/docs/optimization/perf-full-index-2026-08-11.zh.md @@ -491,6 +491,9 @@ buildGraph resolver = 7192ms ← 占 buildGraph ~75% |---|---|---| | ✅ 返回合法 JSON 且数据准确 | 44 | 查询类返回真实数据（get_graph_stats=129893 nodes、find_callers/callees 真实边、codescope_trace 真实调用链、get_module_tree 169KB 等） | | ✅ 参数格式需正确（已验证正确调用） | 2 | `verify_claim`/`verify_statement` 需要 `claim.type` 字段（capability_exists 等）；正确参数后返回 verdict | + +> 注（2026-09-21）：`verify_statement` 已下线，其自然语言短语匹配前端一并删除，统一由 +> `verify_claim` 承担；上表中的 `verify_statement` 仅为当时实测记录。 | ❌ 空转 / Unknown tool | 0 | 无 | **关键数据核验**：`get_graph_stats` 返回 `total_nodes=129893 / total_edges=117154` 与 DB 实测一致；`find_callers(transmute)` 返回 ambiguous 候选（非空）；`get_entry_points` 44KB 真实入口；`verify_integrity` 122KB findings；`get_module_tree` 169KB 模块树——全部非空转。 diff --git a/docs/zh/GRAMMARS_CI.md b/docs/zh/GRAMMARS_CI.md deleted file mode 100644 index 4561170..0000000 --- a/docs/zh/GRAMMARS_CI.md +++ /dev/null @@ -1,378 +0,0 @@ -# Grammars 编译方案 - -## 问题背景 - -CodeScope 使用 tree-sitter 解析多种编程语言。`grammars/*.so` 文件是编译后的语法库，运行时通过 `dlopen` 加载。 - -### 原有问题 - -1. **依赖 npm 包名不正确** - `npm install -g tree-sitter-python` 可能失败 -2. **缺少编译步骤** - 仅仅安装 npm 包无法生成 `.so` 文件 -3. **CI 失败风险高** - 使用 `2>/dev/null || true` 隐藏错误 - ---- - -## 解决方案 - -### 方案 1：使用 tree-sitter CLI（推荐） - -**CI 配置：** -```bash -# 安装官方 CLI -npm install -g tree-sitter-cli - -# 编译语法 -cd grammars -bash build_ci.sh -``` - -**优点：** -- ✅ 官方推荐方式 -- ✅ 可靠性高 -- ✅ 支持所有语言 - -**缺点：** -- ⚠️ 编译耗时（每个语法 ~30秒） -- ⚠️ 需要网络下载源码 - ---- - -### 方案 2：预编译 + Git Submodules - -**结构：** -``` -grammars/ -├── prebuilt/ -│ ├── linux-x86_64/ -│ │ ├── tree-sitter-python.so -│ │ ├── tree-sitter-c.so -│ │ └── ... -│ ├── macos-arm64/ -│ │ ├── tree-sitter-python.so -│ │ └── ... -│ └── macos-x86_64/ -│ └── ... -├── tree-sitter-python/ # Git submodule -├── tree-sitter-c/ # Git submodule -└── build.sh -``` - -**CI 配置：** -```yaml -- name: Checkout submodules - run: git submodule update --init --recursive - -- name: Build grammars - run: | - cd grammars - bash build.sh --platform ${{ matrix.target }} -``` - -**优点：** -- ✅ 无需网络下载 -- ✅ 编译速度快（有缓存） -- ✅ 版本控制精确 - -**缺点：** -- ⚠️ 仓库体积增大 -- ⚠️ 需要维护多平台预编译文件 - ---- - -### 方案 3：容器化编译（最可靠） - -**Dockerfile：** -```dockerfile -FROM ubuntu:22.04 - -RUN apt-get update && \ - apt-get install -y cmake gcc git nodejs npm && \ - npm install -g tree-sitter-cli - -WORKDIR /grammars -COPY build_ci.sh . - -RUN bash build_ci.sh - -VOLUME /output -CMD cp *.so /output/ -``` - -**CI 配置：** -```yaml -- name: Build grammars (Docker) - run: | - docker build -t codescope-grammars grammars/ - docker run --rm -v $PWD/grammars:/output codescope-grammars -``` - -**优点：** -- ✅ 环境完全一致 -- ✅ 无需处理平台差异 -- ✅ 可复用编译结果 - -**缺点：** -- ⚠️ Docker 构建开销 - ---- - -## CI 支持的平台 - -### 自动编译（CI） - -| 平台 | 编译脚本 | 输出文件 | CI 状态 | -|------|---------|----------|---------| -| **Linux (ubuntu-22.04)** | build_ci.sh | *.so | ✅ 自动编译 | -| **macOS (macos-14)** | build_ci.sh | *.so | ✅ 自动编译 | - -### 手动编译（Windows） - -| 平台 | 编译脚本 | 输出文件 | 状态 | -|------|---------|----------|------| -| **Windows** | build_win_guide.sh → build_ci.ps1 | *.dll | ⚠️ 用户自行编译 | - ---- - -## Windows 手动编译指南 - -### 1. 准备环境 - -运行环境准备脚本： -```bash -# Git Bash 环境下运行 -cd grammars -bash build_win_guide.sh -``` - -脚本会检查并提示缺失的依赖： -- Git -- GCC (MinGW-w64) -- Node.js -- tree-sitter CLI - -### 2. 安装依赖（如缺失） - -**PowerShell (Chocolatey):** -```powershell -choco install -y git mingw nodejs -npm install -g tree-sitter-cli -``` - -**手动下载:** -1. Git: https://git-scm.com/download/win -2. MinGW-w64: https://www.mingw-w64.org/ -3. Node.js: https://nodejs.org/ -4. tree-sitter: `npm install -g tree-sitter-cli` - -### 3. 编译 grammars - -**PowerShell (推荐):** -```powershell -cd grammars -.\build_ci.ps1 # 完整编译 -.\build_ci.ps1 -Languages python,c # 指定语言 -``` - -**Git Bash:** -```bash -cd grammars -bash build_ci.sh # 会自动检测 Windows 环境 -``` - -### 4. 验证编译结果 - -```powershell -dir *.dll # PowerShell -``` - -```bash -ls -lh *.dll # Git Bash -``` - -**预期输出文件：** -- tree-sitter-python.dll -- tree-sitter-c.dll -- tree-sitter-cpp.dll -- tree-sitter-rust.dll -- tree-sitter-javascript.dll -- tree-sitter-typescript.dll -- tree-sitter-tsx.dll -- tree-sitter-go.dll -- tree-sitter-java.dll - ---- - -## CI 工作流程 - -### Build & Test (build.yml) -```yaml -支持平台: Linux (ubuntu-22.04) + macOS (macos-14) - -步骤： -1. Install dependencies (apt/brew + tree-sitter CLI) -2. Build grammars → bash build_ci.sh → *.so -3. Build engine → cmake + ninja -4. Build server → cargo build --release -5. Run tests → cargo test -6. Upload artifacts → codescope + grammars/*.so -``` - -### Release (release.yml) -```yaml -支持平台: Linux + macOS - -步骤： -1. Build per platform → *.so -2. Package with grammars → tar.gz -3. Generate checksums -4. Create GitHub Release - -注意: Windows 用户需要手动编译 .dll 文件 -``` - ---- - -## Windows 用户注意事项 - -### 为什么 Windows 不在 CI 中自动编译？ - -1. **编译环境复杂** - Windows 需要额外的 MinGW-w64 设置 -2. **CI 时间过长** - Windows 编译耗时增加 CI 运行时间 -3. **失败风险高** - Windows 环境差异导致编译失败概率较高 -4. **用户可控** - Windows 用户可以根据自己的环境灵活调整 - -### Windows 用户的责任 - -- ✅ 自行准备编译环境（Git + MinGW + Node.js） -- ✅ 运行 `build_win_guide.sh` 检查环境 -- ✅ 使用 `build_ci.ps1` 或 `build_ci.sh` 编译 -- ✅ 确保 `.dll` 文件与 `codescope.exe` 在同一目录 - ---- - -## 常见问题 - -### Q1: 编译失败怎么办？ - -**检查项：** -1. tree-sitter CLI 是否安装成功 -2. Git 是否可以克隆语法仓库 -3. GCC/Clang 是否可用 -4. 检查具体错误日志 - -**解决方法：** -```bash -# 手动测试 -cd grammars -bash build_ci.sh 2>&1 | tee build.log - -# 检查单个语法 -tree-sitter generate # 在语法目录内 -gcc -fPIC -shared src/parser.c -I src -o test.so -``` - -### Q2: 如何加速编译？ - -**优化策略：** -1. **缓存** - 使用 `actions/cache` 缓存编译结果 -2. **并发** - 并行编译多个语法 -3. **预构建** - 在本地预编译并提交 - -**缓存配置：** -```yaml -- name: Cache grammars - uses: actions/cache@v3 - with: - path: grammars/tree-sitter-* - key: grammars-${{ hashFiles('grammars/build_ci.sh') }} -``` - -### Q3: 不同平台的 .so 兼容性？ - -**关键点：** -- Linux: `.so` 文件（ELF 格式） -- macOS: `.so` 文件（Mach-O 格式，实际是 `.dylib`） -- Windows: `.dll` 文件（PE 格式） - -**解决方案：** -```bash -# Linux/macOS (build_ci.sh) -gcc -fPIC -shared src/parser.c -I src -o tree-sitter-python.so - -# macOS 需要特殊处理 -gcc -dynamiclib -o tree-sitter-python.dylib src/parser.c -I src - -# Windows PowerShell (build_ci.ps1) -gcc -shared -o tree-sitter-python.dll src\parser.c -Isrc - -# Windows Batch (build_ci.bat) -gcc -shared -o tree-sitter-python.dll -Isrc src\parser.c -``` - -### Q4: Windows 编译的注意事项？ - -**依赖安装：** -```powershell -# Chocolatey 方式 -choco install -y cmake ninja nodejs git mingw - -# 手动安装 -1. Git: https://git-scm.com/download/win -2. MinGW-w64: https://www.mingw-w64.org/ -3. Node.js: https://nodejs.org/ -4. tree-sitter CLI: npm install -g tree-sitter-cli -``` - -**编译器选择：** -- GCC（MinGW-w64）- 推荐，兼容性好 -- Clang（LLVM）- 可选，现代编译器 - -**常见错误：** -1. `gcc not found` - 安装 MinGW-w64 并添加到 PATH -2. `tree-sitter not found` - 运行 `npm install -g tree-sitter-cli` -3. `Permission denied` - 以管理员权限运行 PowerShell -4. `.dll vs .so` - Windows 必须使用 `.dll` 扩展名 - ---- - -## 下一步优化 - -1. ✅ 使用 tree-sitter CLI（已完成） -2. ✅ 支持 Windows 平台（已完成） -3. ⏭️ 添加缓存机制 -4. ⏭️ 并行编译加速 - ---- - -## 本地测试指南 - -### Linux/macOS -```bash -cd grammars -bash build_ci.sh # 完整编译 -bash build_ci.sh python c # 仅编译指定语言 -ls -lh *.so # 查看结果 -``` - -### Windows (PowerShell) -```powershell -cd grammars -.\build_ci.ps1 # 完整编译 -.\build_ci.ps1 -Languages python,c # 仅编译指定语言 -dir *.dll # 查看结果 -``` - -### Windows (Batch - 备用) -```cmd -cd grammars -build_ci.bat # 完整编译 -dir *.dll # 查看结果 -``` - ---- - -## 参考资料 - -- [tree-sitter CLI 文档](https://tree-sitter.github.io/tree-sitter/creating-parsers) -- [官方语法列表](https://github.com/tree-sitter/tree-sitter/wiki/List-of-parsers) -- [CI 最佳实践](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions) \ No newline at end of file diff --git a/QUICK_START.md b/docs/zh/QUICK_START.md similarity index 55% rename from QUICK_START.md rename to docs/zh/QUICK_START.md index 6842266..1324e0f 100644 --- a/QUICK_START.md +++ b/docs/zh/QUICK_START.md @@ -7,7 +7,7 @@ ### 方式 1：一键构建（推荐） ```bash -bash <(curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/master/bootstrap.sh) +bash <(curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/bootstrap.sh) ``` 脚本会自动检测你的操作系统，安装缺失的依赖（Xcode、Homebrew、LLVM、cmake、Rust），然后编译 CodeScope。 @@ -16,10 +16,10 @@ bash <(curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/master ```bash # macOS ARM64 -curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/master/install.sh | bash +curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/install.sh | bash # Windows PowerShell -irm https://raw.githubusercontent.com/Timwood0x10/CodeScope/master/install.ps1 | iex +irm https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/install.ps1 | iex ``` ### 方式 3：手动构建 @@ -54,11 +54,11 @@ cargo build --release ## 三、索引项目 ```bash -# 索引整个项目 -./target/release/codescope cli index_project '{"project_path":"/path/to/your/project"}' +# 索引整个项目（按模块派生子进程，内存相互隔离） +./target/release/codescope index-parallel /path/to/your/project -# 或者用内置调度器（更快，适合大项目） -codescope index-parallel /path/to/your/project +# 大项目可调参：--workers 解析核心总数（默认 8）、--parallel 并发模块 worker 上限（默认 4） +./target/release/codescope index-parallel /path/to/your/project --workers 8 --parallel 4 ``` ## 四、启动 MCP 服务器 @@ -73,22 +73,30 @@ CODESCOPE_DB_PATH=/path/to/codescope.db ./target/release/codescope ## 五、支持的索引模式 -| 模式 | 速度 | 用途 | 命令 | -|:----|:----:|:-----|:----| -| 快速 | 最快 | 日常开发，快速查询 | `CODESCOPE_INDEX_MODE=fast` | -| 标准 | 中等 | 默认，平衡速度和精度 | 默认 | -| 严格 | 最慢 | 生产环境，完整验证 | `CODESCOPE_INDEX_MODE=strict` | +| 模式 | `CODESCOPE_INDEX_MODE` | 作用 | +|:----|:----|:-----| +| fast | `fast` | 发现阶段额外剪枝（11 类目录 + 4 类缓存文件），且不构建 FTS | +| normal | `normal`（默认） | 平衡的发现策略 + 完整管线 | +| deep | `deep` | 同 `normal`，但重建 n-gram 语义向量 | +| strict | `strict` | **仅**影响发现：对识别出的语言加白名单 gate，不改动解析管线 | ## 六、支持的语言 -| 语言 | 状态 | 说明 | -|:----|:----:|:-----| -| Rust | ✅ 稳定 | 所有项目测试通过 | -| Go | ✅ 稳定 | 标准库 54/54 模块通过 | -| Java | ✅ 可用 | JDK 56/69 模块通过 | -| C/C++ | ✅ 稳定 | tree-sitter C grammar SIGILL 已通过 -O0 编译修复 | -| Python | ✅ 稳定 | 基础支持 | -| JavaScript/TypeScript | ✅ 稳定 | 基础支持 | +| 语言 | 扩展名 | 状态 | +|:----|:------|:-----| +| Python | `.py` | 已内置语法，端到端解析 | +| Go | `.go` | 已内置语法 | +| Rust | `.rs` | 已内置语法 | +| JavaScript | `.js`, `.mjs` | 已内置语法 | +| TypeScript | `.ts` | 已内置语法 | +| TSX | `.tsx` | 已内置语法 | +| C | `.c`, `.h` | 已内置语法（以 `-O0` 编译，见下方 SIGILL 说明） | +| C++ | `.cpp`, `.cc`, `.cxx`, `.hpp`, `.hxx` | 已内置语法 | +| Java | `.java` | 已内置语法 | + +`.kt`/`.kts`、`.rb`、`.scala`、`.swift` 会按扩展名被识别，但没有内置语法：它们会在 +`parse_failures` 中记录为 `language_missing`（用 `codescope parse-failures` 查看）， +且不计入 fail-fast 跳过次数。 ## 七、常见问题 @@ -123,6 +131,6 @@ CodeScope 实现 MCP 协议，可以直接集成到 Claude Desktop、Cursor 等 ## 八、下一步 -- [架构文档](docs/zh/architecture.md) — 了解 CodeScope 的工作原理 -- [优化文档](docs/optimization/optimization-chinese.md) — 性能优化记录 -- [性能报告](docs/zh/e2e_benchmark_report.md) — 大规模项目索引测试 +- [架构文档](architecture.md) — 了解 CodeScope 的工作原理 +- [优化文档](../optimization/optimization.zh.md) — 性能优化记录 +- [性能报告](../optimization/perf-full-index-2026-08-11.zh.md) — 全量索引性能统计 diff --git a/docs/zh/architecture.md b/docs/zh/architecture.md index 41c5e1a..8f04939 100644 --- a/docs/zh/architecture.md +++ b/docs/zh/architecture.md @@ -1,7 +1,7 @@ # CodeScope 架构文档 -**版本**：0.2.1 -**更新日期**：2026-07-06 +**版本**：v0.2.7 +**更新日期**：2026-10-01 --- @@ -195,7 +195,7 @@ sequenceDiagram Client->>Server: tools/call index_project Server->>Worker: spawn subprocess - Worker->>DB: 写入 semantic_records + graph_nodes + Worker->>DB: 写入 semantic_records + entity + relation Worker-->>Server: stdout JSON result Server->>Server: RUNTIME.spawn(build_fts) Server-->>Client: {"ok":true, "files_indexed":N} @@ -218,7 +218,7 @@ sequenceDiagram | `search`（graph fallback） | **<10 ms** | LIKE 降级搜索 | | `get_module_tree` | **<1 ms** | 轻量查询 | | `get_entry_points` | **<1 ms** | 索引查询 | -| `get_communities` | ❌ 引擎有实现，未接 MCP — 改用 `connected_components` | — | +| `get_communities` | ✅ 已接入 — 基于 CALLS 边做确定性标签传播；默认只返回摘要（`max_communities` / `include_members` / `max_members`） | — | | `get_index_progress` | **<1 ms** | 原子读全局变量 | --- diff --git a/docs/zh/e2e_benchmark_report.md b/docs/zh/e2e_benchmark_report.md deleted file mode 100644 index dcb0c4c..0000000 --- a/docs/zh/e2e_benchmark_report.md +++ /dev/null @@ -1,59 +0,0 @@ -# CodeScope E2E 基准报告 - -**项目**: goagent (Go, 1,134 文件) -**日期**: 2026-07-13 -**机器**: Apple M3 Max, 64GB RAM, macOS 15.0 -**引擎**: 14 workers × 8MB 栈, FilterPolicy Normal - ---- - -## 流水线概览 - -``` -1,134 Go 文件 - │ - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Phase 1: Index │ -│ 2,739ms parse → 0ms sqlite → 23,624ms buildGraph │ -│ 17,127 nodes, 3,341 edges, 2,949 references │ -│ call_kind: 2,402/2,949 (81%) 有分类 │ -└──────────────────────────────────────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Phase 2: Knowledge Graph │ -│ 159 modules, 176 module_edge rows │ -│ Architecture: 1,151 items │ -│ 3 workflows, 7 capabilities, 4 contracts │ -└──────────────────────────────────────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Phase 3: Resolver │ -│ 552 / 2,949 references resolved (18.7%) │ -│ 2,397 未解析（主要是 fmt.Errorf 等标准库引用） │ -└──────────────────────────────────────────────────────────┘ -``` - -## 功能清单验证 - -| 功能 | 状态 | 数据 | -|---|---|---| -| 类型注册表 | ✅ | 2,164 类型定义 + 2,850 引用 | -| 路由检测 | ✅ | 46 条 HTTP 路由 | -| 调用分类 | ✅ | 6 语言支持, 81% 有分类 | -| 接口方法捕获 | ✅ | 663 个方法有正确的 parent_id | -| 模块角色 | ✅ | 159 模块, 按 role 分类 | -| 架构社区 | ✅ | 14 社区, 176 模块边 | -| 多因子 Resolver | ✅ | 8 因子打分, call_kind 加权 | - -## 资源消耗 - -| 指标 | 值 | -|---|---| -| 总耗时 | ~26 秒（1,134 文件） | -| parse 阶段 | 2,739ms | -| buildGraph 阶段 | 23,624ms | -| 峰值内存 | ~150MB | -| 数据库大小 | ~176MB | diff --git a/docs/zh/final_benchmark_report.md b/docs/zh/final_benchmark_report.md deleted file mode 100644 index d184883..0000000 --- a/docs/zh/final_benchmark_report.md +++ /dev/null @@ -1,147 +0,0 @@ -# CodeScope 完整基准报告 - -**日期**: 2026-07-07 -**机器**: Apple M3 Max, 64GB RAM, macOS -**引擎**: 14 workers, 8MB 栈/worker, FilterPolicy Normal - ---- - -## 1. 索引性能 - -| 项目 | 语言 | 文件 | 索引耗时 | 峰值内存 | Parse | SQLite | buildGraph | 节点 | 边 | 调用边 | 函数 | 节点/文件 | -|------|------|:----:|:--------:|:--------:|:-----:|:------:|:----------:|:----:|:--:|:------:|:----:|:---------:| -| **CodeScope** | C++/Rust | 122 | **2s** | **62MB** | 16ms | 238ms | 381ms | 40,838 | 25,308 | 346 | 287 | 335 | -| **ffi-demo** | JS/TS/Go/Zig/C | 77 | <1s | **48MB** | 8ms | 158ms | 386ms | 24,538 | 21,126 | 435 | 703 | 319 | -| **others** | JS/TS/Go | 55 | <1s | **48MB** | 9ms | 161ms | 373ms | 24,348 | 21,180 | 430 | 696 | 443 | -| **garbage-code-hunter** | Rust | 94 | **2s** | **67MB** | 10ms | 233ms | 1,000ms | 36,010 | 35,657 | 2,663 | 1,114 | 383 | -| **memscope-rs** | Rust | 219 | **7s** | **128MB** | 31ms | 824ms | 5,987ms | 115,274 | 102,280 | 1,010 | 2,904 | 526 | - -### 历史大项目（前会话数据，仅供参考） - -| 项目 | 文件 | 索引耗时 | Parse | SQLite | buildGraph | FTS | 节点 | 节点/文件 | -|------|:----:|:--------:|:-----:|:------:|:----------:|:---:|:----:|:---------:| -| **rustc** | 36,807 | 1m44s | 3.2s | 38.9s | 30.7s | 23.0s | 4,130,017 | 112 | -| **Bun** | 9,641 | 1m1s | 1.0s | 25.7s | 20.3s | 11.6s | 2,951,664 | 306 | -| **JDK** | 19,821 | **3m31s** | 2.4s | 83.6s | 76.3s | 38.3s | 8,047,762 | **405** | -| **CPython** | 1,022 | 6s | 0.5s | 2.5s | 2.0s | 0.8s | 446,618 | 437 | - ---- - -## 2. 瓶颈分布 - -``` -CodeScope (122 files): - Parse: 16ms (2%) - SQLite: 238ms (27%) - buildGraph:381ms (44%) ← 主瓶颈 - FTS: 240ms (27%) - -memscope-rs (219 files): - Parse: 31ms (0.4%) - SQLite: 824ms (11%) - buildGraph:5987ms (82%) ← 绝对主瓶颈 - FTS: 456ms (6%) - -所有项目一致性： - buildGraph = 44-82% ← SQLite-bound - Parse = <3% ← 不是 CPU-bound -``` - ---- - -## 3. 查询延迟 - -| 查询 | CodeScope | ffi-demo | others | garbage-code-hunter | memscope-rs | **平均** | -|------|:---------:|:--------:|:------:|:-------------------:|:-----------:|:--------:| -| `get_graph_stats` | 7ms | 6ms | 7ms | 6ms | 9ms | **7ms** | -| `get_entry_points` | 5ms | 5ms | 5ms | 5ms | 5ms | **5ms** | -| `get_module_tree` | 5ms | 5ms | 5ms | 5ms | 5ms | **5ms** | -| `codescope_trace` | 6ms | 6ms | 6ms | 5ms | 5ms | **5.6ms** | - -> 注：CLI 模式每次查询含 ~4ms 引擎初始化开销，实际 SQLite 查询 <1ms。 - ---- - -## 4. Token 消耗 - -### 响应 Token 估算（DeepSeek 公式） - -| 查询 | CodeScope | ffi-demo | others | garbage-code-hunter | memscope-rs | -|------|:---------:|:--------:|:------:|:-------------------:|:-----------:| -| `get_graph_stats` | **18** | **18** | **18** | **18** | **19** | -| `codescope_trace` (depth=1) | 53 | 79 | 173 | 271 | 156 | - -### 各工具 Token 估算（独立请求） - -| 工具 | 典型响应大小 | Token | 说明 | -|------|:-----------:|:-----:|------| -| `get_graph_stats` | ~60 bytes | **~18** | 纯 SQL COUNT | -| `get_entry_points` | ~50 bytes | **~15** | 入口点列表 | -| `get_module_tree` | ~100 bytes | **~30** | 模块树 JSON | -| `find_callers` | ~100-500 bytes | **~30-150** | 调用者列表 | -| `find_callees` | ~100-500 bytes | **~30-150** | 被调用者列表 | -| `codescope_trace` (depth=1) | ~200-900 bytes | **~50-270** | 交互展开深度1 | -| `codescope_trace` (depth=2) | ~500-3000 bytes | **~150-900** | 交互展开深度2 | -| `get_hotspots` | ❌ 未实现 — 无 MCP 工具 | — | — | -| `search` (FTS) | ~1000-5000 bytes | **~300-1500** | 全文搜索 | -| `get_communities ❌` | 1K-200K bytes | **~300-60K** ⚠️ | 全图社区检测 | - -### 与 CBM 对比 - -| 指标 | codebase-memory-mcp | CodeScope | 差距 | -|------|:-------------------:|:---------:|:----:| -| 最小可用响应 | 56,183 bytes | **629 bytes** | **89x** | -| 等价 tokens | ~14,046 | **~157** | **89x** | -| 索引大小 | 64 MB | **270 KB** | **243x** | -| 典型 trace 查询 tokens | ~14,046 | **~53-271** | **52-265x** | - ---- - -## 5. `codescope_trace` 准确性验证 - -| 项目 | 函数 | 调用者 | 被调用者 | JSON 验证 | 耗时 | -|------|------|:------:|:--------:|:---------:|:----:| -| CodeScope | `CppVisitor` | 0 | 1 | ✅ 有效 JSON | 6ms | -| ffi-demo | `CapitalSigma0` | 1 | 1 | ✅ 有效 JSON | 6ms | -| others | `ComputeProgressMetrics` | 0 | 4 | ✅ 有效 JSON | 6ms | -| garbage-code-hunter | `activate` | 0 | 6 | ✅ 有效 JSON | 5ms | -| garbage-code-hunter | `analyze` | **18** | **10** | ✅ **高扇入测试通过** | <1ms | -| memscope-rs | `analyze_cycle` | **2** | **1** | ✅ 有效 JSON | 5ms | -| memscope-rs | `build_circular_reference_report` | **1** | **1** | ✅ 1:1 路径 | <1ms | - ---- - -## 6. 资源消耗总表 - -| 项目 | 峰值 RSS | 索引 CPU 时间 | 查询 CPU 时间 | 每个查询 Token | 边/文件 | -|------|:--------:|:-------------:|:-------------:|:--------------:|:-------:| -| **CodeScope** | 62 MB | 0.6s | 5-7ms | ~18 | 207 | -| **ffi-demo** | 48 MB | 0.5s | 5-6ms | ~18 | 274 | -| **others** | 48 MB | 0.5s | 5-7ms | ~18 | 385 | -| **garbage-code-hunter** | 67 MB | 1.2s | 5-6ms | ~18-271 | 379 | -| **memscope-rs** | 128 MB | 6.8s | 5-9ms | ~19-156 | 467 | - -### 关键观察 - -1. **内存与文件数非线性** — memscope-rs 219 文件用 128MB，比 122 文件的 CodeScope 高 2x。Rust `memscope-rs` 的节点密度（526 节点/文件）远高于其他项目。 -2. **buildGraph 始终是主瓶颈** — 占索引时间的 44%-82%，项目越大比例越高。 -3. **Token 消耗极小** — 基础查询 15-30 tokens，比 CBM 的 14K tokens 低 500-900x。 -4. **Worker 内存隔离有效** — Worker 退出后 RSS 100% 归还，Server 常驻内存约 12MB。 -5. **查询延迟稳定** — 不论项目大小（55-219 文件），查询延迟保持在 5-9ms（含引擎初始化）。 - ---- - -## 7. 本次轮次代码变动汇总 - -在优化前后对比中，以下代码变动贡献了主要提升： - -| 改动 | 影响 | 文件 | -|------|------|------| -| FTS 后置构建 + Tokio async trigger | 索引阶段不阻塞 | `engine_index.cpp`, `tools/mod.rs` | -| `buildGraph(true)` + kind 修复 | 调用边从 0 → 正确生成 | `store.cpp` | -| SUBSTR 后缀匹配替代精确匹配 | 调用边生成率 0% → 100% | `store.cpp` | -| 新增 `idx_sr_kind_name` + `idx_sr_fp_parent` 索引 | buildGraph 加速 | `store.cpp` createSchema | -| Worker 300s 超时 + 3 次重试 | 防止 server 永久挂起 | `tools/mod.rs` | -| 交互式 `codescope_trace` | 新功能：递归展开调用者/被调用者 | `store.h/cpp`, `engine_queries.cpp`, `ffi/mod.rs` | -| 索引进度 `get_index_progress` | 新功能：Client 轮询进度 | `store.h/cpp`, `engine_ffi.cpp`, `ffi/mod.rs` | -| `count_tokens` 估算 | 新功能：DeepSeek 公式 token 估算 | `tools/mod.rs`（已有） | diff --git a/docs/zh/goagent_verification.md b/docs/zh/goagent_verification.md deleted file mode 100644 index db0c792..0000000 --- a/docs/zh/goagent_verification.md +++ /dev/null @@ -1,90 +0,0 @@ -# CodeScope × goagent — 桥接验证 & Token 节省 - -> 日期：2026-07-11 -> 项目：[goagent](https://github.com/Timwood0x10/ares)（Go） -> 1,124 文件，331,390 行代码 - ---- - -## 桥接验证 - -旧 GA 系统（scheduler）与新系统（coordinator）通过 `GenomePopulationAdapter` + `submitToCoordinator` 桥打通。调用链通过 grep 验证（非调用图——图为 struct 方法调用未捕获）。 - -### 调用链 - -| 链路 | 状态 | 位置 | -|------|------|------| -| `Run()` → `submitToCoordinator()` | ✅ | `genome_wiring.go:462` | -| `submitToCoordinator()` → `generateDiffPatches()` | ✅ | `genome_wiring.go:578` | -| `submitToCoordinator()` → `Coordinator.Submit()` | ✅ | `genome_wiring.go:584` | -| 生产路径 `bootstrap.go` → `Coordinator.Submit()` | ✅ | `bootstrap.go:303,363` | -| 生产路径 `genome_wiring_system.go:743` → `Coordinator.Submit()` | ✅ | | -| 自愈路径 `evolution_bridge.go:70` → `Coordinator.Submit()` | ✅ | | - -### 架构 - -``` -旧系统 Scheduler（事件驱动） - → GenomePopulationAdapter.Run() - → 评分 + 进化 (population.Evolve()) - → submitToCoordinator() ← 新桥 - → generateDiffPatches() - → Coordinator.Submit(PatchProposal{Source: SourceGA}) - → Coordinator.Evaluate() - -新系统（决策层） - → Coordinator 接收 patch proposals - → 决定 Apply / Delay / Reject - → 通过 Patch Executor 应用 -``` - ---- - -## Token 节省 - -| 指标 | 值 | -|------|-----| -| 源文件 | 1,124 / 1,253 | -| 代码行数 | 331,390 | -| 原始 token | 3,313,900 | -| 知识图谱 token | 140,470 | -| **Token 节省** | **95.8%** | -| 查询延迟 | 0.5 ms | -| 索引时间 | 31 s | -| 实体数 | 13,535 | -| 关系边 | 1,024 | -| 已解析引用 | 1,022 | - -### 计算方式 - -``` -原始 token: 331,390 行 × 10 = 3,313,900 -知识图谱 token: 13,535 × 10 + 1,024 × 5 = 140,470 -节省: 95.8% -``` - -**AI 的上下文从 331 万 token 压缩到 14 万 token，信息密度高 23 倍。** - ---- - -## 孤儿模块（基于 grep 导入路径，排除自引用） - -| 模块 | 实体数 | 外部导入 | 判定 | -|------|--------|---------|------| -| `compat/` | 173 | 0 | ✅ 孤儿 | -| `api/evolution/` | 66 | 0 | ✅ 孤儿 | -| `api/service/ga/` | 15 | 0 | ✅ 孤儿（已删除） | -| `graphservice/` | 39 | 0 | ✅ 孤儿（已删除） | -| `EvolutionBridge` | — | 0 | ✅ 孤儿（从未实例化） | -| `retrieval_api/` | 20 | 0 | ✅ 孤儿（已删除） | -| `ares_quant/` | 992 | 0 | ✅ 孤儿（新发现） | - -### 与审计文档一致性 - -人工审计的 6 个孤儿模块全部命中，100% 一致。CodeScope 自动分析与手动 grep 结论一致。 - ---- - -## 关键结论 - -**Go 的调用图（relation 表）不足以检测连接。** 接口调用和 struct 方法调用（如 `a.submitToCoordinator(ctx)`）不会被当前 Resolver Pipeline 捕获。正确的孤儿检测方法是**基于导入路径的分析**（grep 导入路径，排除自引用），与人工审计方法一致。 \ No newline at end of file diff --git a/docs/zh/ladybugdb_explorer.md b/docs/zh/ladybugdb_explorer.md deleted file mode 100644 index 13fb0b0..0000000 --- a/docs/zh/ladybugdb_explorer.md +++ /dev/null @@ -1,261 +0,0 @@ -# CodeScope 与 LadybugDB Explorer 可视化指南 - -> **最后更新**: 2026-07-20 - -本指南介绍如何使用 [LadybugDB Explorer](https://docs.ladybugdb.com/visualization/lbug-explorer/) 对 CodeScope 的代码图数据进行交互式可视化探索。 - ---- - -## 前提条件 - -- **Docker**：LadybugDB Explorer 以 Docker 容器运行。[安装 Docker](https://docs.docker.com/get-docker/) -- **CodeScope 数据**：项目需要先完成索引（通过 `codescope index_project`），这会生成 `.codescope/codescope.lbug` 文件 - ---- - -## 快速开始 - -### 步骤1: 确认 LadybugDB 文件存在 - -```bash -ls -la .codescope/codescope.lbug -``` - -如果文件不存在或大小为 0，重新索引项目： - -```bash -codescope index_project -``` - -### 步骤2: 从 SQLite 同步数据到 LadybugDB - -从 SQLite 到 LadybugDB 的同步是**自动完成的**。当你运行 `codescope index_project` 时，引擎会自动使用批处理的 Cypher `CREATE` 语句将图数据从 SQLite 同步到 LadybugDB。无需任何手动步骤。 - -> **注意**：引擎使用 Cypher `CREATE` 而非 `COPY FROM`，因为内嵌的 LadybugDB 0.18.2 中的 `COPY FROM`（CSV 批量导入）存在已知缺陷，会以 `state=1` 错误失败。Cypher `CREATE` 是可靠的替代方案。 - -```mermaid -sequenceDiagram - participant SQL as SQLite (.db) - participant ENG as CodeScope 引擎 - participant LBUG as LadybugDB (.lbug) - - SQL->>ENG: 读取 graph_nodes / graph_edges - ENG->>LBUG: 批处理 Cypher CREATE（节点） - ENG->>LBUG: 批处理 Cypher CREATE（关系边） - Note over LBUG: 同步完成 ✓ -``` - -#### 故障排查 - -如果索引后 `.lbug` 文件为空，请确认引擎已使用 `HAS_LADYBUG` 编译（检查构建输出中是否有 `LadybugDB: found at ...`）。引擎使用 Cypher `CREATE` 作为 `COPY FROM` 的可靠替代方案，后者在 LadybugDB 0.18.2 中存在已知问题。 - -### 步骤3: 启动 LadybugDB Explorer - -```bash -docker run -p 8000:8000 \ - -v "$(pwd)/.codescope:/database" \ - -e LBUG_FILE=codescope.lbug \ - --rm ghcr.io/ladybugdb/explorer:latest -``` - -### 步骤4: 在浏览器中打开 - -访问 [http://localhost:8000](http://localhost:8000) - -```mermaid -flowchart LR - A["Docker 容器
ghcr.io/ladybugdb/explorer"] -->|"挂载"| B[".codescope/codescope.lbug"] - A -->|"暴露端口 8000"| C["浏览器打开
http://localhost:8000"] - C --> D["LadybugDB Explorer
交互式图探索界面"] -``` - ---- - -## 使用示例 - -### 1. 浏览所有符号 - -在 Query 面板中运行： - -```cypher -MATCH (n:GraphNode) RETURN n.name, n.language, n.file_path LIMIT 50; -``` - -点击 **Graph** 视图，以交互式图的形式查看节点。 - -### 2. 查找所有 Go 函数 - -```cypher -MATCH (n:GraphNode) -WHERE n.language = 'go' -RETURN n.name, n.file_path, n.start_row -ORDER BY n.name; -``` - -### 3. 可视化调用关系 - -```cypher -MATCH (n:GraphNode)-[c:CALLS]->(m:GraphNode) -RETURN n.name, c.call_site_line, m.name -LIMIT 100; -``` - -切换到 **Graph** 视图，查看调用关系的交互式网络图。 - -```mermaid -graph LR - subgraph "可视化效果" - A["parseJSON
(bootstrap.go:12)"] -->|"CALLS"| B["json.Unmarshal
(encoding/json:456)"] - A -->|"CALLS"| C["strings.TrimSpace
(strings:123)"] - D["buildContext
(context.go:89)"] -->|"CALLS"| A - E["handleRequest
(handler.go:45)"] -->|"CALLS"| A - E -->|"CALLS"| D - end -``` - -### 4. 查找特定函数的调用者 - -```cypher -MATCH (caller:GraphNode)-[c:CALLS]->(callee:GraphNode) -WHERE callee.name = 'targetFunction' -RETURN caller.name, caller.file_path, c.call_site_line; -``` - -### 5. 查找被调用最多的函数 - -```cypher -MATCH (caller:GraphNode)-[c:CALLS]->(callee:GraphNode) -RETURN callee.name, count(c) AS call_count -ORDER BY call_count DESC -LIMIT 20; -``` - -### 6. 查找热点函数（高入度 + 高出度） - -```cypher -MATCH (caller:GraphNode)-[c:CALLS]->(callee:GraphNode) -WITH callee, count(c) AS incoming -MATCH (callee)-[c2:CALLS]->(other:GraphNode) -WITH callee, incoming, count(c2) AS outgoing -RETURN callee.name, incoming, outgoing -ORDER BY incoming DESC, outgoing DESC -LIMIT 20; -``` - -### 7. 两个函数之间的最短调用路径 - -```cypher -MATCH p = shortestPath( - (a:GraphNode {name: 'funcA'})-[*..10]->(b:GraphNode {name: 'funcB'}) -) -RETURN p; -``` - ---- - -## Schema 参考 - -### GraphNode（节点表） - -| 列名 | 类型 | 说明 | -|------|------|------| -| `id` | INT64 | 主键（对应 SQLite graph_nodes.id） | -| `project_id` | INT64 | 项目标识符 | -| `ir_node_id` | INT64 | 内部 IR 节点引用 | -| `node_type` | INT32 | 0=函数, 1=类, 2=变量, 3=类型, ... | -| `name` | STRING | 符号名称 | -| `qualified_name` | STRING | 完全限定名（如 `pkg.FuncName`） | -| `signature` | STRING | 函数签名（参数 + 返回值类型） | -| `module_path` | STRING | 模块/包路径 | -| `file_path` | STRING | 源文件路径 | -| `language` | STRING | 编程语言 | -| `start_row` | INT32 | 源文件中的起始行 | -| `start_col` | INT32 | 起始列 | -| `end_row` | INT32 | 结束行 | -| `end_col` | INT32 | 结束列 | -| `parent_id` | INT64 | 父节点 ID（用于包含关系） | -| `is_entry_point` | BOOL | 是否为入口点（main, init 等） | -| `embedding_ready` | BOOL | 向量嵌入是否已计算 | -| `metrics_ready` | BOOL | 指标是否已计算 | - -### CALLS（关系表） - -| 列名 | 类型 | 说明 | -|------|------|------| -| `FROM` | GraphNode | 调用者节点 | -| `TO` | GraphNode | 被调用者节点 | -| `project_id` | INT64 | 项目标识符 | -| `edge_type` | INT32 | 3=调用, 1=包含, 2=引用 | -| `call_site_line` | INT32 | 调用点所在行号 | -| `label` | STRING | 可选标签 | - -### RELATES（关系表） - -| 列名 | 类型 | 说明 | -|------|------|------| -| `FROM` | GraphNode | 源节点 | -| `TO` | GraphNode | 目标节点 | -| `project_id` | INT64 | 项目标识符 | -| `type` | INT32 | 关系类型 | - ---- - -## 数据流总览 - -```mermaid -flowchart TB - subgraph "CodeScope 索引流程" - A["源代码"] --> B["tree-sitter 解析器"] - B --> C["SQLite 存储
codescope.db"] - C --> E["自动同步（Cypher CREATE）
codescope.lbug"] - end - - subgraph "LadybugDB Explorer 可视化" - E --> F["Docker 容器
ghcr.io/ladybugdb/explorer"] - F --> G["浏览器
http://localhost:8000"] - G --> H["Cypher 查询"] - G --> I["交互式图视图"] - G --> J["表格视图"] - G --> K["JSON 视图"] - end - - subgraph "终端查询" - E --> L["lbug CLI
命令行"] - L --> M["Cypher 查询结果"] - end -``` - ---- - -## 小贴士 - -### 性能建议 - -- 对于大图（10 万+ 节点），使用 `LIMIT` 避免浏览器过载 -- 在 `WHERE` 子句中使用索引属性（`name`、`language`、`file_path`）加速查询 -- Explorer 在浏览器中运行——所有计算在服务端执行 - -### 导出数据 - -从查询面板的结果视图可以将结果导出为 JSON 或 CSV。 - -### 只读模式 - -以只读模式启动（防止意外修改）： - -```bash -docker run -p 8000:8000 \ - -v "$(pwd)/.codescope:/database" \ - -e LBUG_FILE=codescope.lbug \ - -e MODE=READ_ONLY \ - --rm ghcr.io/ladybugdb/explorer:latest -``` - -### 故障排查 - -| 问题 | 解决方案 | -|------|----------| -| **"File not found" 错误** | 确保 `.lbug` 文件路径正确，Docker 挂载卷指向正确的目录 | -| **"Connection refused"** | 确认 Docker 正在运行（`docker ps` 检查），端口 8000 未被占用 | -| **查询结果为空** | 运行 `MATCH (n) RETURN count(n)` 验证是否有数据。如果为 0，重新执行同步步骤 | -| **查询缓慢** | 在查询中添加 `LIMIT`。对于大图，从 `LIMIT 50` 开始逐步增加 | \ No newline at end of file diff --git a/docs/zh/large_project_index_report.md b/docs/zh/large_project_index_report.md deleted file mode 100644 index fadb625..0000000 --- a/docs/zh/large_project_index_report.md +++ /dev/null @@ -1,600 +0,0 @@ -# CodeScope 大规模项目索引报告 - -> 日期：2026-07-06 | 硬件：Apple M3 Max, 36 GB RAM | macOS - -*** - -## 一、rustc（Rust 编译器）全量索引 - -### 完整日志 - -``` -开始时间: 2026年 7月 6日 星期一 19时50分33秒 - -worker: project=0 starting index_project dir=～/code/rustcode/rust lang= -BATCH [0..99] of 36807 (100 files) -BATCH [0..99] done (0 passes ok), total indexed: 100 -... -BATCH [36800..36806] of 36807 (7 files) -BATCH [36800..36806] done (0 passes ok), total indexed: 36807 -POST_BUILD: symbol graph... -buildGraph: 36800 files | file_list=339ms delete=457ms rf=21ms r2n=7317ms - nodes=8442ms edges=14066ms calls=0ms total=30702ms -{"ok":true,"files_indexed":36807,"workers":14, - "time_parse_ms":3193,"time_sqlite_ms":38932, - "time_buildgraph_ms":30703,"time_fts_ms":22995,"time_vector_ms":0, - "discovery":{"seen_dirs":65437,"seen_files":59339, - "skipped_dirs":1698,"skipped_files":49, - "skipped_suffix":76,"candidate_files":36807}} - -结束时间: 2026年 7月 6日 星期一 19时52分18秒 -总耗时: 1m44s -``` - -### 性能数据 - -| 指标 | 数值 | -| ---------------- | :--------------: | -| **总耗时** | **1m44s** | -| 源文件数 | 36,807（全部 .rs） | -| 图节点 | **4,130,017** | -| 图边 | **3,044,162** | -| 解析 (Parse) | 3,193ms (3.0%) | -| SQLite 写入 | 38,932ms (37.4%) | -| 图构建 (buildGraph) | 30,703ms (29.5%) | -| FTS 索引 | 22,995ms (22.1%) | -| 发现目录 | 65,437 | -| 跳过目录 | 1,698 | -| 工作线程 | 14 | - -### 瓶颈分析 - -``` -Parse 3,193ms ███░░░░░░░░░░░░░░░░░░ 3.0% -SQLite 38,932ms █████████████████████ 37.4% -Graph 30,703ms █████████████████░░░░ 29.5% -FTS 22,995ms █████████████░░░░░░░░ 22.1% -``` - -**瓶颈依然是 SQLite-bound**（SQLite + buildGraph + FTS 占 89%）。解析仅 3.2 秒（3%），吞吐量达到 **11,523 文件/秒**。 - -### 查询 Token 消耗 - -| 查询 | Tokens | 占比 | -| -------------------- | :--------: | :-: | -| `get_graph_stats` | 19 | — | -| `get_hotspots ❌` top10 | \~500 | — | -| `project_overview` | \~71 | — | -| `get_module_tree` | 4 | — | -| `get_entry_points` | 5 | — | -| `search`（单次） | \~300-1000 | — | -| **典型分析组合** | **\~910** | — | - -*** - -## 二、Bun Runtime 全量索引 - -### 完整日志 - -``` -开始时间: 2026年 7月 6日 星期一 19时55分37秒 - -worker: project=0 starting index_project dir=～/code/researcher/bun lang= -BATCH [0..99] of 9641 (100 files) -BATCH [0..99] done (0 passes ok), total indexed: 100 -... -BATCH [9600..9640] of 9641 (41 files) -BATCH [9600..9640] done (0 passes ok), total indexed: 9641 -POST_BUILD: symbol graph... -buildGraph: 9623 files | file_list=238ms delete=116ms rf=5ms r2n=4842ms - nodes=5632ms edges=9450ms calls=0ms total=20320ms -{"ok":true,"files_indexed":9641,"workers":14, - "time_parse_ms":1024,"time_sqlite_ms":25736, - "time_buildgraph_ms":20321,"time_fts_ms":11568,"time_vector_ms":0, - "discovery":{"seen_dirs":21672,"seen_files":13883, - "skipped_dirs":6415,"skipped_files":4, - "skipped_suffix":504,"candidate_files":9641}} - -结束时间: 2026年 7月 6日 星期一 19时56分39秒 -总耗时: 1m1s -``` - -### 性能数据 - -| 指标 | 数值 | -| ---------------- | :--------------: | -| **总耗时** | **1m1s** | -| 源文件数 | 9,641 | -| 图节点 | **2,951,664** | -| 图边 | **2,567,205** | -| 解析 (Parse) | 1,024ms (1.7%) | -| SQLite 写入 | 25,736ms (43.6%) | -| 图构建 (buildGraph) | 20,321ms (34.4%) | -| FTS 索引 | 11,568ms (19.6%) | -| 发现目录 | 21,672 | -| 跳过目录 | 6,415 | -| 工作线程 | 14 | - -### 瓶颈分析 - -``` -Parse 1,024ms ██░░░░░░░░░░░░░░░░░░░ 1.7% -SQLite 25,736ms █████████████████████ 43.6% -Graph 20,321ms ██████████████████░░░ 34.4% -FTS 11,568ms ███████████████░░░░░░ 19.6% -``` - -*** - -## 三、Bun Runtime 核心功能分析 - -### 技术栈 - -Bun 主要用 **Zig** 编写，集成 **WebKit 的 JavaScriptCore** 作为 JS 引擎，同时有大量 C++ FFI 绑定层： - -| 语言 | 文件数 | 用途 | -| ----- | :---: | ---------------------------- | -| Zig | 1,270 | 核心运行时、CLI、bundler、包管理 | -| C++ | 579 | JSC 绑定层（`src/jsc/bindings/`） | -| C | 113 | WebCore 集成、系统调用 | -| JS/TS | 6,592 | 测试、API 定义 | - -### 核心架构 - -```mermaid -graph TB - subgraph "Bun CLI (main.zig)" - CLI["Cli.start()"] - end - subgraph "命令" - RUN["bun run"] - TEST["bun test"] - INSTALL["bun install"] - BUILD["bun build"] - SERVE["bun serve"] - end - subgraph "JS 运行时 (bun.js.zig)" - VM["VirtualMachine"] - TRANS["Transpiler
(TS/JSX→JS)"] - RES["Resolver"] - end - subgraph "JavaScriptCore (C++)" - JSC["JSC::JSGlobalObject"] - BIND["bindings: JSCommonJSModule
JSCommonJSExtensions"] - WC["WebCore bindings
TypedArrayController"] - end - subgraph "系统层" - MIM["mimalloc 分配器"] - SSL["BoringSSL"] - end - - CLI --> RUN & TEST & INSTALL & BUILD & SERVE - RUN --> VM - VM --> JSC - VM --> TRANS - VM --> RES - JSC --> BIND & WC - CLI --> MIM & SSL -``` - -### 核心功能拆解 - -| 功能 | 实现位置 | 技术 | -| ------------- | ----------------------------------- | --------------------- | -| **JS/TS 执行** | `src/bun.js.zig` → `VirtualMachine` | Zig + JSC C++ FFI | -| **转译器** | `src/transpiler/` | Zig 实现的 TS/JSX→JS | -| **打包器** | `src/bundler/` | Zig（BundleThread.zig） | -| **包管理器** | `src/install/` | Zig（npm 协议实现） | -| **HTTP 服务器** | `src/http/` | Zig + libuv | -| **SQLite** | `src/sqlite/` | Zig 封装 | -| **WebSocket** | `src/websocket/` | Zig | -| **加密** | `src/boringssl/` | BoringSSL（OpenSSL 分支） | -| **内存分配器** | `src/bun_alloc/` | mimalloc | - -### 关键技术细节 - -1. **JavaScriptCore 集成**：Bun 直接链接 WebKit 的 `libjavascriptcoregtk`，通过 `src/jsc/bindings/` 中的 C++ 胶水代码暴露 JSC API 给 Zig -2. **主循环**：`main.zig` 中的 `Cli.start()` 解析子命令并分发 -3. **快速启动**：Bun 预编译 WebKit 的字节码缓存，冷启动比 Node.js 快 4-5 倍 -4. **mimalloc**：使用微软的 mimalloc 替代系统分配器，减少内存碎片 -5. **Zig 零开销抽象**：Zig 的 comptime 和无隐式分配策略是 Bun 性能优势的基础 - -*** - -## 三、rustc 中 Arc/Rc 与多线程的底层实现 - -> 基于 `library/alloc/src/sync.rs`（4,974 行）和 `library/alloc/src/rc.rs`（4,599 行）的源码分析。 - -### 3.1 Arc 的核心数据结构 - -```rust -// sync.rs:269-276 -pub struct Arc { - ptr: NonNull>, // 堆分配的 ArcInner - phantom: PhantomData>, - alloc: A, // 分配器（默认 Global） -} - -// sync.rs:388-401 -#[repr(C, align(2))] -struct ArcInner { - strong: Atomic, // 线程安全的强引用计数 - weak: Atomic, // 线程安全的弱引用计数 - data: T, // 实际数据 -} -``` - -关键设计点： - -- **`Atomic`** 代替 Rc 的 `Cell`——这是 Arc 线程安全和 Rc 单线程的根本区别 -- **`#[repr(C, align(2))]`**——2 字节对齐使指针可以低位标记（用于弱引用计数锁定） - -### 3.2 Send/Sync 的精确推导 - -```rust -unsafe impl Send for Arc {} -unsafe impl Sync for Arc {} -``` - -`Arc: Send` 当且仅当 `T: Sync + Send` 且分配器也 Send。这是手工标注的 `unsafe impl`——编译器信任程序员选对了原子类型。 - -### 3.3 Arc::new 的分配路径 - -```rust -pub fn new(data: T) -> Arc { - let x: Box<_> = Box::new(ArcInner { - strong: atomic::AtomicUsize::new(1), // strong=1 - weak: atomic::AtomicUsize::new(1), // weak=1 - data, - }); - unsafe { Self::from_inner(Box::leak(x).into()) } - // ^^^ Box::leak 阻止 Box 释放内存， - // 将生命周期控制权交给引用计数 -} -``` - -### 3.4 Arc::clone 的原子递增 - -```rust -fn clone(&self) -> Self { - let inner = self.ptr.as_ref(); - inner.strong.fetch_add(1, Ordering::Relaxed); - // Relaxed 就够了：clone 不涉及对其他内存的读取依赖 - Self { ptr: self.ptr, phantom: PhantomData, alloc: ... } -} -``` - -### 3.5 Arc::drop 的原子递减与内存释放 - -```rust -fn drop(&mut self) { - let inner = self.ptr.as_ref(); - if inner.strong.fetch_sub(1, Ordering::Release) != 1 { - return; // 还有别的强引用 - } - atomic::fence(Ordering::Acquire); // 与最后一个 Release 同步 - unsafe { ptr::drop_in_place(&mut (*ptr).data); } - - if inner.weak.fetch_sub(1, Ordering::Release) == 1 { - atomic::fence(Ordering::Acquire); - unsafe { dealloc(ptr as *mut u8, layout); } - } -} -``` - -**内存序策略**： - -- `fetch_sub(Release)`：确保 drop 之前的所有写入对其他线程可见 -- `fence(Acquire)`：确保释放前看到最后一个线程对 data 的最终写入 -- Release/Acquire 构成 **happens-before** 关系——无锁编程的基石 - -### 3.6 Arc 与 Rc 的对比 - -```mermaid -flowchart TD - subgraph "Rc（单线程）" - RC_CELL["strong: Cell<usize>"] - RC_WEAK["weak: Cell<usize>"] - RC_CLONE["clone: cell.get()+1"] - RC_DROP["drop: cell.get()-1"] - RC_SEND["Send: ❌"] - RC_SYNC["Sync: ❌"] - end - subgraph "Arc（多线程）" - ARC_ATOMIC["strong: Atomic<usize>"] - ARC_WEAK["weak: Atomic<usize>"] - ARC_CLONE["clone: fetch_add(Relaxed)"] - ARC_DROP["drop: fetch_sub(Release)+fence(Acquire)"] - ARC_SEND["Send: ✅ T: Sync+Send"] - ARC_SYNC["Sync: ✅ T: Sync+Send"] - end -``` - -| 维度 | Arc (sync.rs) | Rc (rc.rs) | -| ------ | ----------------------------------- | ------------- | -| 引用计数类型 | `Atomic` | `Cell` | -| Send | ✅ 有条件 | ❌ 不支持 | -| Sync | ✅ 有条件 | ❌ 不支持 | -| 内存序 | Relaxed / Release / Acquire + fence | 无（单线程） | -| 额外开销 | 原子操作指令 | 零 | - -### 3.7 线程的底层实现 - -```rust -// library/std/src/thread/mod.rs -// 平台分发（通过 cfg 宏）： -// Linux/macOS: sys::unix::thread → libc::pthread_create -// Windows: sys::windows::thread → CreateThread - -pub fn spawn(f: F) -> JoinHandle -where F: FnOnce() -> T + Send + 'static, T: Send + 'static -{ - // Builder::new().spawn(f) - // → imp::Thread::new(name, Box::new(f)) - // → pthread_create / CreateThread -} -``` - -`spawn()` 上的 `Send` 约束在**编译期**由类型系统检查，不是运行时检查。`unsafe impl Send for Arc` 正是允许 `Arc` 跨线程传递的关键。 - -### 总结 - -Arc 的线程安全性依赖于三个底层机制： - -1. **`Atomic`** 的硬件级原子操作（x86 `LOCK XADD` / ARM `LDREX+STREX`） -2. **`Release/Acquire`** **内存序**保证跨线程的 happens-before -3. **`unsafe impl Send/Sync`** 的手工标注——编译器信任程序员选择了正确的原子类型 - -*** - -## 四、横向对比 - -| 指标 | rustc (36,807 .rs) | Bun (9,641 Zig/C++/JS) | Linux 内核 (60,468 C/H) | -| ------------ | :----------------: | :--------------------: | :-------------------: | -| **索引耗时** | **1m44s** | **1m1s** | **\~300s（超时）** | -| 解析吞吐 | 11,523 文件/s | 9,414 文件/s | — | -| 图节点 | **4,130,017** | 2,951,664 | — | -| 图边 | 3,044,162 | 2,567,205 | — | -| Parse % | 3.0% | 1.7% | 2.5% | -| SQLite % | 37.4% | 43.6% | 30.5% | -| buildGraph % | 29.5% | 34.4% | 37.3% | -| FTS % | 22.1% | 19.6% | 29.3% | - -## 五、Token 消耗对比 - -从本次测试的三个项目汇总： - -| 查询 | rustc | Bun | ARES Agent | memscope-rs | -| -------------------- | :---: | :---: | :--------: | :---------: | -| `get_graph_stats` | 19 | 19 | 18 | 18 | -| `get_hotspots ❌` top10 | 500 | \~488 | 488 | 488 | -| `project_overview` | 71 | 71 | 71 | 71 | -| `get_module_tree` | 4 | 4 | 4 | 4 | -| `get_entry_points` | 5 | 5 | 5 | 5 | -| `get_project_info` | 44 | 39 | 44 | 44 | - -**跨项目一致性**：Token 消耗仅取决于**返回的数据量**，与项目大小无关。百万节点和千节点项目的 `get_graph_stats` 都是 \~18 tokens。 - -**vs 直接读源码**： - -- rustc 36,807 文件：CodeScope 返回架构信息 **\~910 tokens** vs 直接读源码 **数百万 tokens** -- Bun 9,641 文件：CodeScope 返回架构信息 **\~910 tokens** vs 直接读源码 **数十万 tokens** - -**vs codebase-memory-mcp**：codebase-memory-mcp 对这三个项目均无法完成全量索引（rustc: 不支持 Rust 多文件图构建，Bun: Zig 不支持，JDK: 不支持 Java），CodeScope 通过 tree-sitter 多语言解析器覆盖了全部语言。 - -*** - -## 六、JDK 20K Java 文件全量索引 - -### 完整日志 - -``` -开始时间: 2026年 7月 6日 星期一 20时50分45秒 - -worker: project=0 starting index_project dir=～/code/researcher/jdk/src lang= -BATCH [0..99] of 19821 (100 files) -... -BATCH [19800..19820] of 19821 (21 files) -BATCH [19800..19820] done, total indexed: 19821 -POST_BUILD: symbol graph... -buildGraph: 19821 files | file_list=644ms delete=239ms rf=10ms r2n=14567ms - nodes=22609ms edges=38139ms calls=0ms total=76344ms -{"ok":true,"files_indexed":19821,"workers":14, - "time_parse_ms":2410,"time_sqlite_ms":83556, - "time_buildgraph_ms":76344,"time_fts_ms":38349,"time_vector_ms":0, - "discovery":{"seen_dirs":25805,"seen_files":22935, - "skipped_dirs":113,"skipped_files":0, - "skipped_suffix":765,"candidate_files":19821}} - -结束时间: 2026年 7月 6日 星期一 20时54分16秒 -总耗时: 3m31s -``` - -### 性能数据 - -| 指标 | 数值 | -| ---------------- | :-----------------------------: | -| **总耗时** | **3m31s** | -| 源文件数 | 19,821（仅 src/，排除 test/make/doc） | -| 图节点 | **8,047,762** | -| 图边 | **6,723,123** | -| 解析 (Parse) | 2,410ms (1.2%) | -| SQLite 写入 | 83,556ms (41.7%) | -| 图构建 (buildGraph) | 76,344ms (38.1%) | -| FTS 索引（后置） | 38,349ms (19.1%) | -| 数据库大小 | **8.9 GB** | - -### 瓶颈分析 - -``` -Parse 2,410ms ██░░░░░░░░░░░░░░░░░░░ 1.2% -SQLite 83,556ms █████████████████████ 41.7% -Graph 76,344ms ████████████████████░ 38.1% -FTS 38,349ms █████████████████░░░░ 19.1% -``` - -### 耗时反直觉分析 - -JDK 19,821 文件比 rustc 36,807 文件耗时更长（3m31s vs 1m44s），原因是： - -| 项目 | 文件数 | 节点/文件 | 总节点 | 吞吐量 | -| ------- | :--------: | :-----: | :-------: | :----------: | -| rustc | 36,807 | **112** | 4.13M | **353 文件/s** | -| Bun | 9,641 | **306** | 2.95M | 155 文件/s | -| **JDK** | **19,821** | **405** | **8.05M** | 93 文件/s | - -JDK 每文件平均产生 **405 个节点**（是 rustc 的 3.6 倍）。Java 类的字段、方法、注解、内部类比 Rust 的模块结构更复杂，导致 SQLite 写入和 buildGraph 的总工作量更大。 - -### 与旧版的对比 - -| 状态 | 文件数 | 索引结果 | -| ------------------ | :-------------------: | :----------: | -| ❌ FilterPolicy 修复前 | 59,923（含 45,586 测试文件） | 300s 超时，未完成 | -| ✅ FilterPolicy 修复后 | **19,821**（仅 src/ 源码） | **3m31s，完成** | - -*** - -## 七、全量横向对比 - -| 指标 | rustc | Bun | JDK | Linux 内核 | -| ---------------- | :-------: | :--------: | :-----------: | :--------: | -| **主要语言** | Rust | Zig/C++/JS | **Java** | C | -| **源文件数** | 36,807 | 9,641 | **19,821** | \~60,000 | -| **索引耗时** | **1m44s** | 1m1s | **3m31s** | \~300s（超时） | -| **图节点** | 4,130,017 | 2,951,664 | **8,047,762** | — | -| **图边** | 3,044,162 | 2,567,205 | **6,723,123** | — | -| **节点/文件** | 112 | 306 | **405** | — | -| **DB 大小** | — | — | **8.9 GB** | 12 GB | -| **Parse %** | 3.0% | 1.7% | 1.2% | 2.5% | -| **SQLite %** | 37.4% | 43.6% | **41.7%** | 30.5% | -| **buildGraph %** | 29.5% | 34.4% | **38.1%** | 37.3% | -| **FTS %** | 22.1% | 19.6% | 19.1% | 29.3% | - -*** - -## 八、补充项目 - -### codebase-memory-mcp（竞品源码） - -| 指标 | 数值 | -| -------- | :--------------------------------------------------: | -| **索引耗时** | **2m0s** | -| 源文件 | 1,257 | -| 图节点 | **6,221,406** | -| 图边 | **1,065,570** | -| 数据库大小 | \~1.2 GB | -| 源码语言 | **纯 C**（src/ + vendored/）— 已从 Go 重写为纯 C | - -### CPython - -| 部分 | 文件数 | 耗时 | 节点 | 状态 | -| --------- | :-------: | :----: | :-----: | :---------------------: | -| Python 源码 | **1,022** | **6s** | 446,618 | ✅ 完成 | -| C 源码 | \~1,100 | — | — | ❌ tree-sitter-c 解析器 bug | - -**CPython C 文件无法索引的原因**：tree-sitter-c 0.24.1 解析器在解析 CPython 核心 C 文件（如 `Python/Python-ast.c`、`Objects/unicodeobject.c`）时触发 `Illegal instruction: 4`（SIGILL）。这是 tree-sitter 生成的 parser.c 使用当前 CPU 不支持指令所致，非 CodeScope 本身的问题。需等待 tree-sitter-c 上游修复。 - -*** - -## 九、全量索引总表 - -| 项目 | 语言 | 文件数 | 耗时 | 图节点 | 状态 | -| ----------------------- | ---------- | :------: | :-----: | :-------: | :---: | -| **rustc** | Rust | 36,807 | 1m44s | 4,130,017 | ✅ | -| **JDK** | Java/C++ | 19,821 | 3m31s | 8,047,762 | ✅ | -| **Bun** | Zig/C++/JS | 9,641 | 1m1s | 2,951,664 | ✅ | -| **codebase-memory-mcp** | 纯 C | 1,257 | 2m0s | 6,221,406 | ✅ | -| **CPython (Python)** | Python | 1,022 | 6s | 446,618 | ✅ | -| **CPython (C)** | C | \~1,100 | — | — | ❌  | -| **Linux 内核** | C | \~60,000 | 300s 超时 | — | ⚠️ 部分 | -| **memscope-rs** | Rust/C | 238 | 2s | 123,270 | ✅ | -| **InstrumentTimbre** | Python | 142 | 0.9s | — | ✅ | -| **ARES Agent** | Go | 95 | 0.3s | 24,924 | ✅ | - -### 关键结论 - -1. **瓶颈始终在 SQLite**（SQLite + buildGraph + FTS 占 \~90%），解析仅占 1-3% -2. **每文件节点数决定索引速度**：JDK 405 节点/文件（最复杂）→ 3m31s；rustc 112 节点/文件 → 1m44s -3. **FilterPolicy + .gitignore** 大幅减少噪音索引（JDK 从 59,923 降至 19,821 文件） -4. **FTS 后置**让 graph 查询在 FTS 完成前即可用 -5. **tree-sitter-c 解析器**对复杂 C 代码（CPython/Linux 内核）存在兼容性问题，需要上游修复 - ---- - -## 十、查询性能基准 - -### 查询延迟 - -| 查询 | 平均延迟 | 说明 | -|------|:--------:|------| -| `get_graph_stats` | **<1 ms** | 纯 SQL COUNT | -| `get_hotspots ❌` | **<2 ms** | SQL JOIN + GROUP BY | -| `find_callers` / `find_callees` | **<1 ms** | 索引覆盖的 JOIN | -| `search`（FTS） | **<5 ms** | FTS5 全文搜索 | -| `search`（graph fallback） | **<10 ms** | LIKE 降级搜索 | -| `get_module_tree` | **<1 ms** | 轻量查询 | -| `get_entry_points` | **<1 ms** | 索引查询 | -| `get_communities` | ❌ 引擎有实现，未接 MCP | — | -| `codescope_trace` | **<5 ms** | BFS 路径搜索 | - -### 跨项目基准 - -| 项目 | 文件 | 索引耗时 | Parse | SQLite | buildGraph | FTS | 节点/文件 | -|------|:----:|:--------:|:-----:|:------:|:----------:|:---:|:----------:| -| rustc | 36,807 | 1m44s | 3.2s | 38.9s | 30.7s | 23.0s | 112 | -| Bun | 9,641 | 1m1s | 1.0s | 25.7s | 20.3s | 11.6s | 306 | -| JDK | 19,821 | 3m31s | 2.4s | 83.6s | 76.3s | 38.3s | **405** | -| codebase-memory-mcp | 1,257 | 2m0s | — | — | 48.0s | — | — | -| memscope-rs | 238 | 2s | 31ms | 589ms | 577ms | 253ms | — | -| ARES Agent | 95 | 0.3s | 9ms | 96ms | 81ms | 30ms | — | - -### 瓶颈分布总结 - -``` -所有项目的瓶颈分布高度一致： - SQLite + buildGraph = ~70-80% - FTS = ~20% - Parse = ~1-3% - -结论：CodeScope 是 SQLite-bound，不是 CPU-bound。 -优化 SQLite 写入和 buildGraph 是唯一有效的提速路径。 -``` - ---- - -## 十一、buildGraph SQL 审计 - -### EXPLAIN QUERY PLAN 结果（50 文件 C++ 项目） - -``` ---- QPLAN _r2n --- - id=6 parent=3 SEARCH sr USING INDEX idx_sr_kind (project_id=? AND kind=?) - id=50 parent=3 USING INDEX sqlite_autoindex__rf_1 FOR IN-OPERATOR - id=69 parent=0 SCAN (subquery-3) -``` - -| 阶段 | SQL | 耗时 | 分析 | -|------|-----|:----:|------| -| file_list | `SELECT DISTINCT file_path FROM semantic_records` | 1ms | ✅ 使用 `idx_sr_kind` 索引 | -| delete | `deleteGraphEdgesByFile` + `deleteGraphNodesByFile` | 0ms | ✅ 索引覆盖 | -| _rf | 创建临时文件过滤表 | 0ms | ✅ 轻量 | -| r2n | `CREATE TEMP TABLE _r2n AS ... ROW_NUMBER() OVER ()` | 36ms | ⚠️ 窗口函数不走索引 | -| nodes | `INSERT INTO graph_nodes SELECT ... JOIN _r2n` | 32ms | ✅ 索引 JOIN | -| edges | `INSERT INTO graph_edges SELECT ... JOIN _r2n x2` | 41ms | ⚠️ 最慢阶段 | -| calls | `INSERT ... 4x JOIN + semantic_records 自连接` | 0ms | ⚠️ 大项目可能 O(N²) | - -### 大项目瓶颈预测 - -| 瓶颈 | 复杂度 | 影响 | 建议 | -|------|:------:|------|------| -| `ROW_NUMBER() OVER ()` 窗口函数 | O(N) | 8M 行时 ~1-2s | SQLite 窗口函数无法索引，但仍是 O(N) | -| containment edges 双 _r2n JOIN | O(N log N) | 8M 行时 ~4-8s | 已用 DISTINCT + 索引，可接受 | -| call edges 4x JOIN + 自连接 | O(N²) 风险 | 大项目主要瓶颈 | 需要 `idx_sr_kind(project_id, kind)` + `idx_sr_fp_oid(file_path, original_id)` | - -### 优化建议 - -1. `_r2n` 创建时 `ROW_NUMBER() OVER ()` 不需要 ORDER BY，目前已是最优 -2. call edges 的 `sr.name = callee.name` JOIN 无索引——添加 `idx_sr_name(project_id, name)` 可加速 -3. 考虑对 containment edges 使用批量 INSERT 而非逐条 INSERT-SELECT - ---- diff --git a/docs/zh/linux-analysis.md b/docs/zh/linux-analysis.md deleted file mode 100644 index dbf779c..0000000 --- a/docs/zh/linux-analysis.md +++ /dev/null @@ -1,323 +0,0 @@ -# CodeScope Linux 内核分析 - -> 使用 CodeScope 的快速扫描和增强工具对 Linux 内核 v6.13 进行全面分析。 -> 所有时间、CPU 和内存数据均来自实际运行（Apple M 系列芯片，macOS）。 - ---- - -## 1. USB 驱动模型——鼠标和键盘如何被区分 - -### 扫描结果 - -| 指标 | 数值 | -|------|------| -| 目录 | `drivers/usb/` | -| 扫描时间 | **351.49 ms** | -| 符号数 | 37,286 | -| 模块数 | 40 | -| 语言 | c | -| 峰值 CPU | ~100%（单核） | -| 峰值内存 | ~8 MB | -| 完整日志 | `scan_usb_raw.log` | - -### 模块树（40 个子目录） - -```mermaid -graph TD - USB["usb/"] - USB --> CORE["core/
(27 文件)"] - USB --> HOST["host/
(110 文件)"] - USB --> GADGET["gadget/
(9 文件)"] - GADGET --> FUNC["function/
(63 文件)"] - GADGET --> UDC["udc/
(40 文件)"] - USB --> SERIAL["serial/
(67 文件)"] - USB --> STORAGE["storage/
(32 文件)"] - USB --> TYPEC["typec/
(17 文件)"] - USB --> MISC["misc/
(32 文件)"] - USB --> DWC3["dwc3/
(32 文件)"] - USB --> MUSBC["musb/
(28 文件)"] - - style USB fill:#4472C4,color:#fff - style CORE fill:#E6F3FF - style HOST fill:#E6F3FF - style GADGET fill:#E6F3FF - style SERIAL fill:#E6F3FF - style STORAGE fill:#E6F3FF - style TYPEC fill:#E6F3FF - style MISC fill:#E6F3FF - style DWC3 fill:#E6F3FF - style MUSBC fill:#E6F3FF - style FUNC fill:#FFF2CC - style UDC fill:#FFF2CC -``` - -### 鼠标与键盘的区分机制 - -核心代码在 **`drivers/hid/usbhid/hid-core.c`**（扫描：5.58 ms，385 个符号）。 - -```mermaid -flowchart TD - A["USB 设备插入"] --> B["usb_hid_probe()
(hid-core.c:1004)"] - B --> C{"检查 bInterfaceProtocol"} - C -->|"1 = KEYBOARD"| D["HID_TYPE_USBNONE
注册 EV_KEY 事件"] - C -->|"2 = MOUSE"| E["HID_TYPE_USBMOUSE
注册 EV_REL + EV_KEY"] - D --> F["hid_parse_report()"] - E --> F - F --> G["hidinput_configure_usage()
(hid-input.c:711)"] - G --> H["映射 HID usage 码到输入事件"] - H --> I["input_register_device()"] - I --> J["用户态: input_event"] -``` - -**关键源码位置：** - -| 文件 | 行号 | 函数 | 作用 | -|------|------|------|------| -| `drivers/hid/usbhid/hid-core.c` | 1004 | `usb_hid_probe()` | 读取 `bInterfaceProtocol` | -| `drivers/hid/usbhid/hid-core.c` | 1136 | `HID_GD_MOUSE` 处理器 | 鼠标输入映射 | -| `drivers/hid/usbhid/hid-core.c` | 1144 | `HID_GD_KEYBOARD` 处理器 | 键盘输入映射 | -| `drivers/hid/hid-input.c` | 711 | `hidinput_configure_usage()` | 核心 HID→input 翻译 | -| `include/linux/hid.h` | 625 | `enum hid_type` | `HID_TYPE_USBMOUSE` / `HID_TYPE_USBNONE` | - -### 发现的入口点 - -| 符号 | 类型 | 文件 | 行号 | -|------|------|------|------| -| `start` | function | `drivers/usb/host/sl811-hcd.c` | 303 | - ---- - -## 2. 进程调度——父子进程资源处理 - -### 扫描结果 - -| 指标 | 数值 | -|------|------| -| 目录 | `kernel/sched/` | -| 扫描时间 | **45 ms** | -| 符号数 | 4,913 | -| 模块数 | 2（sched, sched/ext） | -| 语言 | c | -| 增强阶段 | 291 ms，4,800 条调用边 | - -### 调度代码结构 - -```mermaid -graph TD - SCHED["kernel/sched/"] - SCHED --> CORE["core.c
__schedule()、schedule()"] - SCHED --> FAIR["fair.c
CFS 完全公平调度器"] - SCHED --> RT["rt.c
实时调度器"] - SCHED --> DEADLINE["deadline.c
截止时间调度器"] - SCHED --> IDLE["idle.c
空闲任务"] - SCHED --> H["sched.h
数据结构"] - SCHED --> EXT["ext/
可扩展调度接口"] -``` - -### 父子进程资源流程（写时复制 COW） - -```mermaid -flowchart TD - A["copy_process(kernel/fork.c:1994)"] --> B["dup_task_struct(current)
复制内核栈 + task_struct"] - B --> C["sched_fork(clone_flags, p)
初始子进程调度状态"] - C --> D["copy_mm()
复制地址空间"] - D --> E["dup_mm(mm)
复制内存描述符"] - E --> F["dup_mmap(mm, oldmm)
复制内存映射"] - F --> G["copy_page_range(src_mm, dst_mm)
COW：共享只读页面"] - D --> H["copy_sighand()
复制信号处理器"] - D --> I["copy_files()
复制文件描述符表"] -``` - -**COW（写时复制）机制：** `copy_page_range()` 让父子进程共享同一物理内存页，标记为只读。任一进程首次写入时触发缺页中断，复制该页。 - -### 防止抢占（三层防线） - -``` -第一层：每进程计数器 - preempt_count() > 0 → __schedule() 直接返回 - 位置：include/linux/preempt.h:92 - -第二层：自旋锁 - spin_lock() → 自动 preempt_disable() - spin_unlock() → 自动 preempt_enable() - -第三层：中断上下文 - 硬/软中断 → preempt_count 递增 - 抢占被阻止直到处理器返回 -``` - -**CodeScope 追踪的调用路径：** - -| 起点 | 终点 | 验证 | -|------|------|------| -| `copy_process` | `sched_fork` | ✅ `kernel/fork.c:1994 → kernel/sched/core.c:4803` | -| `copy_process` | `dup_mm` | ✅ `kernel/fork.c:1994 → kernel/fork.c:1568 → kernel/fork.c:1527` | -| `__schedule` | `pick_next_task_fair` | ⚠️ 跨文件调用，需要增强阶段 | - -### 关键源码位置 - -``` -kernel/fork.c:914 dup_task_struct() — 复制进程结构 -kernel/fork.c:1994 copy_process() — 进程创建入口 -kernel/sched/core.c:7061 __schedule() — 主调度器 -include/linux/preempt.h:108 preempt_count() — 抢占计数器 -``` - ---- - -## 3. 内存页分配 - -### 扫描结果 - -| 指标 | 数值 | -|------|------| -| 目录 | `mm/` | -| 扫描时间 | **182.93 ms** | -| 符号数 | 16,111 | -| 模块数 | 7 | -| 语言 | c | - -### 页分配器流程 - -```mermaid -flowchart TD - A["alloc_pages(gfp_mask, order)"] --> B["__alloc_pages(gfp_mask, order)
(page_alloc.c:4034)"] - B --> C["get_page_from_freelist()"] - C --> D["rmqueue()"] - D --> E["__free_one_page()
伙伴合并释放"] -``` - -### 页回收（内存压力下） - -```mermaid -flowchart TD - A["try_to_free_pages()
内存回收入口"] --> B["shrink_node()
按 NUMA 节点扫描 LRU"] - B --> C["shrink_lruvec()
扫描 LRU 链表"] - C --> D["shrink_active_list()
活跃 → 非活跃降级"] - C --> E["shrink_inactive_list()
回收非活跃页"] -``` - -### IO 策略：预读 - -``` -ondemand_readahead() ← mm/readahead.c:501 - ↓ -首次读取：4 页（16 KB） -顺序读取：倍增 → 32 → 64 → 128 页（最大 512 KB） -随机读取：自动降级，不预读 -``` - -### 关键源码位置 - -``` -mm/page_alloc.c:936 __free_one_page() — 伙伴合并释放 -mm/page_alloc.c:3401 rmqueue() — 核心页分配 -mm/page_alloc.c:3792 get_page_from_freelist() — Zone 选择 -mm/page_alloc.c:4034 __alloc_pages() — 分配入口 -mm/readahead.c:160 read_pages() — 预读 IO 下发 -include/linux/fs.h:401 address_space_operations — 页 IO 虚函数表 -``` - ---- - -## 4. 工具链汇总 - -| 操作 | 时间 | CPU | 内存 | 工具 | -|------|------|-----|------|------| -| USB 扫描（`drivers/usb/`） | 351 ms | ~100% | ~8 MB | `codescope_scan` | -| USB HID 扫描（`drivers/hid/usbhid/`） | 5.58 ms | ~100% | ~4 MB | `codescope_scan` | -| 调度器扫描（`kernel/sched/`） | 45 ms | ~100% | ~6 MB | `codescope_scan` | -| 内存扫描（`mm/`） | 183 ms | ~100% | ~8 MB | `codescope_scan` | -| 调度器增强 | 291 ms | ~100% | ~12 MB | `codescope_enhance` | -| 全量内核增强 | 27 s | ~100% | ~30 MB | `codescope_enhance` | -| `find_symbol("usb_register")` | ~15 µs | — | — | `codescope_find_symbol` | -| `trace_path(copy_process, sched_fork)` | <1 ms | — | — | `codescope_trace` | - -**全部分析总耗时：** ~800 ms -**峰值内存：** ~30 MB -**相比直接读源码的 Token 节省：** ~98.9% - ---- - -## 5. 全量内核索引 (codebase-memory-mcp) - -### 索引结果 (89,465 文件) - -| 指标 | 数值 | -|------|------| -| 发现文件数 | **89,465** | -| 文件发现耗时 | **2,336 ms** | -| **总节点数** | **4,877,492** | -| **总边数** | **9,326,238** | -| **数据库大小** | **7.06 GB** | -| **索引耗时** | **183 s（约 3 分钟）** | -| **峰值内存** | **11.6 GB** | -| 并行 workers | 14 | -| 吞吐量 | 489 files/s，50,966 edges/s | - -### 阶段耗时 - -| 阶段 | 耗时 | 说明 | -|------|:----:|------| -| `parallel_extract` | **60.4 s** | 并行解析 89K 文件，提取 token 和符号 | -| `parallel_resolve` | **38.9 s** | 解析调用关系、引用、语义边 | -| `dump` (DB 持久化) | **23.8 s** | 写入 SQLite 数据库 | -| `semantic_edges` | **14.5 s** | 语义相似度边 | -| `registry_build` | **11.1 s** | 构建符号注册表 | -| `similarity` | 1.8 s | 函数指纹相似度 | -| 其他阶段 | 3.5 s | tests/k8s/githistory/decorator/configlink | -| **总计** | **~183 s** | | - -### 内存配置文件 - -``` -峰值： 11,644 MB (parallel_extract 阶段 — 14 workers 持有 AST) -稳定： 7,500 MB (提取后，resolve 和 registry 阶段) -预算： 18,432 MB (每个 worker: 1,316 MB) -``` - -### 磁盘使用 - -``` -SQLite DB: 7.06 GB (~/.cache/codebase-memory-mcp/) -缓存： 6.70 GB (所有已索引项目) -─── -总计： ~14 GB -``` - -### 扩展性 - -| 范围 | 文件数 | 节点 | 耗时 | 内存 | -|------|:-----:|:----:|:----:|:----:| -| 单文件 | 1 | 31 | 4.94 ms | ~50 MB | -| kernel/sched/ | 51 | 3,854 | 841 ms | ~500 MB | -| kernel/ | 697 | 33,546 | 2,302 ms | ~1.5 GB | -| **全量内核** | **89,465** | **4,877,492** | **183 s** | **11.6 GB** | - -## 6. CodeScope MCP 分析示例 - -以下分析使用 **CodeScope 自身的 MCP 工具**完成——MCP 服务器配置在 `.mcp.json` 中，用于查询 Linux 内核源码。 - -### 示例 1：USB HID 设备识别 - -**使用工具：** `index_file`, `find_definition`, `graph_query ❌` (MATCH (Function)-[Calls]->(Function)) - -关键符号定位（通过 CodeScope MCP）： - -| 符号 | 文件 | 行号 | CodeScope 工具 | -|------|------|:----:|---------------| -| `usbhid_probe` | `hid-core.c` | 1363 | `find_definition` | -| `usbhid_parse` | `hid-core.c` | 982 | `find_definition` | -| `usb_mouse_probe` | `usbmouse.c` | 106 | `find_definition` | -| `usb_kbd_probe` | `usbkbd.c` | 261 | `find_definition` | -| `hid_usb_ids` | `hid-core.c` | 1678 | `find_definition` | - -调用图（通过 `graph_query ❌`）：在 USB HID 驱动链中发现 45 条调用边。 - -完整分析：`Runtimelog/scan_usb_hid_analysis.log` - -### 示例 2：进程调度与父子资源管理 - -**使用工具：** `find_definition`, `index_file` 针对 kernel/sched/core.c、kernel/fork.c、kernel/exit.c - -完整分析：`Runtimelog/scan_linux_scheduler.log` diff --git a/docs/zh/skills.md b/docs/zh/skills.md index 13ef185..d05dd08 100644 --- a/docs/zh/skills.md +++ b/docs/zh/skills.md @@ -73,7 +73,7 @@ graph TB Client["Claude Desktop / Cursor
自定义 MCP 客户端"] end subgraph "Rust MCP Server (codescope)" - TD["工具调度
(35+ 工具)"] + TD["工具调度
(46 个工具)"] TQ["任务队列
(Tokio async)"] FFI["FFI (extern \"C\")"] end @@ -86,9 +86,10 @@ graph TB CD["社区检测"] end subgraph "SQLite (WAL)" - N["graph_nodes
(节点)"] - E["graph_edges
(边)"] - SR["semantic_records
(IR + FTS)"] + N["entity
(符号)"] + E["relation
(边)"] + SR["semantic_records
(IR)"] + FTSN["code_fts + name_trgm
(FTS5)"] end Client -->|JSON-RPC 2.0| TD TD --> TQ @@ -182,7 +183,7 @@ flowchart LR # 查热点 → ❌ get_hotspots 未实现 搜代码 → search (300-1000 tok) 查调用链 → find_callers / find_callees (10-50 tok) -架构分析 → get_module_tree (4 tok) + 可选 get_communities ❌ (引擎有实现，未接 MCP) (1K-200K tok) +架构分析 → get_module_tree (4 tok) + 可选 get_communities ✅（标签传播，默认只返回摘要） 变更影响 → detect_changes (100-500 tok) AI 问答 → codescope_build_context (200-1000 tok) ``` @@ -218,7 +219,7 @@ AI 问答 → codescope_build_context (200-1000 tok) | `normal` | `NORMAL` | 仅基础 skip 表（build/dist/out/target/test/docs/vendor/node_modules/.venv 等） | ✅ | 默认 | | `strict` | `STRICT` | 基础 skip + detectLanguage 白名单 gate（仅索引可识别语言的源码文件） | ✅ | 最严格，数据最精简 | -> 已知问题（2026-08-11 已修复）：fast 此前≈normal——`fast_extra_skip_dirs_` 为空集（预留未实现）、`setMode()` 未重建 `active_skip_dirs_`。已补全剪枝集合并修复。详见 `docs/optimization/perf-full-index-2026-08-11.md` §9/§10。 +> 已知问题（2026-08-11 已修复）：fast 此前≈normal——`fast_extra_skip_dirs_` 为空集（预留未实现）、`setMode()` 未重建 `active_skip_dirs_`。已补全剪枝集合并修复。详见 `docs/optimization/perf-full-index-2026-08-11.zh.md` §9/§10。 ## 7. 性能基准 @@ -273,6 +274,9 @@ make bench-full # 完整基准测试 ## 10. 支持的语言 +**已内置语法 —— 端到端解析。** 即 README 表格中的 8 种语言；每种都有 visitor（LIVE IR 管线），并由 +`engine/tests/test_ir_edge_coverage.cpp` 覆盖。 + | 语言 | 扩展名 | 解析器 | |------|--------|--------| | Python | `.py` | tree-sitter-python | @@ -284,7 +288,14 @@ make bench-full # 完整基准测试 | C | `.c`, `.h` | tree-sitter-c | | C++ | `.cpp`, `.cc`, `.cxx`, `.hpp`, `.hxx` | tree-sitter-cpp | | Java | `.java` | tree-sitter-java | + +**按扩展名可识别，但**不解析** —— 未内置语法。** 这些文件会被发现（因此计入候选文件数），随后在 +`parse_failures` 中记录为 `language_missing` 解析失败；它们不会被索引，每次运行都会重试，且不计入 +`CODESCOPE_FAIL_RETRY_MAX`。 + +| 语言 | 扩展名 | 语法（未内置） | +|------|--------|----------------| | Kotlin | `.kt`, `.kts` | tree-sitter-kotlin | | Ruby | `.rb` | tree-sitter-ruby | | Scala | `.scala` | tree-sitter-scala | -| Swift | `.swift` | tree-sitter-swift | +| Swift | `.swift` | tree-sitter-swift —— 其 `parser.c` 与内置的 tree-sitter core ABI 不兼容，因此不在 `GRAMMAR_SOURCES` 中，Swift 的 visitor/translator 也已随之删除（见 `engine/src/parser/parser.cpp`） | diff --git a/docs/zh/technical_breakdown.md b/docs/zh/technical_breakdown.md index 10b8baf..4e6eec3 100644 --- a/docs/zh/technical_breakdown.md +++ b/docs/zh/technical_breakdown.md @@ -1,6 +1,6 @@ # CodeScope 技术拆解 -> **版本**: v0.2.1 | **最后更新**: 2026-07-15 +> **版本**: v0.2.7 | **最后更新**: 2026-10-09 本文档深入剖析 CodeScope 的内部架构、设计决策和实现细节，适合希望理解系统原理、参与贡献或进行集成的开发者阅读。 @@ -15,7 +15,7 @@ 5. [解析器管线](#5-解析器管线) 6. [验证系统](#6-验证系统) 7. [性能指标](#7-性能指标) -8. [LadybugDB 集成](#8-ladybugdb-集成) +8. [LadybugDB 集成（已移除）](#8-ladybugdb-集成已移除) 9. [如何扩展](#9-如何扩展) --- @@ -36,7 +36,7 @@ flowchart TB MCPServer["MCP 服务器 (JSON-RPC 2.0)"] FFIBridge["FFI 桥接层 (Rust ↔ C++)"] TokioRT["Tokio 运行时 (后台任务)"] - Tools["35+ 个 MCP 工具
定位 / 理解 / 验证 / 索引"] + Tools["46 个 MCP 工具
定位 / 理解 / 验证 / 索引"] end subgraph "C++ Worker 子进程" @@ -44,12 +44,11 @@ flowchart TB Parser["tree-sitter 解析器 (8种语言)"] Resolver["解析器管线"] GraphBuilder["图构建器"] - Store["SQLite + LadybugDB 写入器"] + Store["SQLite 写入器"] end subgraph "存储层" SQLite["SQLite 数据库 (WAL 模式)"] - Ladybug["LadybugDB (.lbug)"] end Client -->|"MCP stdio"| MCPServer @@ -57,7 +56,6 @@ flowchart TB MCPServer -->|"FFI (只读)"| FFIBridge FFIBridge -->|"轮询进度"| SQLite Worker -->|"写入"| SQLite - Worker -->|"同步"| Ladybug TokioRT -->|"异步 FTS 构建"| SQLite ``` @@ -79,13 +77,13 @@ flowchart LR B --> C["IR (SemanticUnit)"] C --> D["SQLite 写入"] D --> E["图构建"] - E --> F["LadybugDB 同步"] + E --> F["写入 SQLite：entity + relation"] end subgraph "查询流程" G["MCP 工具调用"] --> H["FFI 桥接"] H --> I["C++ 查询引擎"] - I --> J["SQLite / LadybugDB"] + I --> J["SQLite（entity / relation）"] J --> K["JSON 响应"] end ``` @@ -108,7 +106,6 @@ flowchart LR | `filter_policy.cpp` | — | 目录/文件/后缀过滤 + .gitignore 匹配 | | `store/store.cpp` | ~3,060 | SQLite 存储 + FTS + 进度 + 增量索引 | | `store/store_core.cpp` | — | 核心 CRUD 操作 | -| `store/store_ladybug.cpp` | 365 | LadybugDB 同步（CSV → COPY FROM） | | `ir/` | — | IR 类型（SemanticUnit、Record、Reference） | | `parser/` | — | 各语言的 tree-sitter 封装 | | `graph/` | — | 图构建 + 调用链解析 | @@ -212,7 +209,7 @@ flowchart TB subgraph "MCP 协议层" Transport["transport.rs
stdio 传输层
1MB 行读取限制"] - Server["server.rs
请求分发 + 工具路由
35+ 工具注册"] + Server["server.rs
请求分发 + 工具路由
46 个工具注册"] end Main --> MCP @@ -226,7 +223,7 @@ flowchart TB MCP 服务器通过 stdio 实现 [Model Context Protocol](https://modelcontextprotocol.io/)： -- **`tools/list`**：返回可用工具列表（35+ 个工具） +- **`tools/list`**：返回可用工具列表（46 个工具） - **`tools/call`**：通过参数调用指定工具 - **传输层**：通过 stdin/stdout 的 JSON-RPC 2.0 - **错误处理**：解析错误返回 `ReadResult::ParseError` 而非崩溃（1MB 行读取限制） @@ -270,17 +267,20 @@ extern "C" { ```mermaid flowchart TB - subgraph "SQLite 核心表" - GN["graph_nodes
所有符号（函数、类型、变量等）
~17k-200k 行"] - GE["graph_edges
关系（调用、包含、引用）
~3k-50k 行"] - ENT["entity
实体（仅生产代码）
~10k-150k 行"] - REL["relation
关系（仅生产代码）
~2k-40k 行"] + subgraph "SQLite 核心表（权威）" + ENT["entity
符号
~10k-200k 行"] + REL["relation
关系（边）
~2k-50k 行"] + end + + subgraph "遗留表（已废弃，无查询读取）" + GN["graph_nodes
调度路径不再写入"] + GE["graph_edges
解析器仍会镜像写入"] end subgraph "辅助表" REF["reference
符号引用（未解析）"] SCOPE["scope
作用域链"] - FTS["search_index
FTS5 全文搜索"] + FTS["code_fts + name_trgm
FTS5 全文搜索"] VEC["node_vectors
向量嵌入 (vec0)"] end @@ -289,28 +289,26 @@ flowchart TB PROJ["project
项目元数据 + readiness 标志"] end - GN --> GE - GN --> FTS - GN --> VEC - ENT --> REL + ENT --> FTS + ENT --> VEC ``` **索引设计**：所有查询使用索引覆盖扫描，无全表扫描 -- `idx_sr_kind_name` on `(kind, name)` — 符号查找 -- `idx_sr_fp_parent` on `(file_path, parent_id)` — 树遍历 -- `idx_ge_source` on `(source_node_id)` — 调用边查找 -- `idx_ge_target` on `(target_node_id)` — 被调用者查找 +- `idx_entity_name` on `entity(project_id, name)` — 符号查找 +- `idx_entity_file` on `entity(project_id, file_path)` — 单文件重建与删除 +- `idx_relation_source` on `relation(project_id, source_id)` — 被调用者查找 +- `idx_relation_target` on `relation(project_id, target_id)` — 调用者查找 ### 4.2 双写策略 -在 v0.1.3 迁移期间，生产代码符号同时写入： -- 旧版 `graph_nodes`/`graph_edges` 表 -- 新版 `entity`/`relation` 表 +v0.1.3 迁移引入的双写对**符号**已经退役：符号只存在于 `entity`。对**边**仍然保留 —— +解析器在写入权威表 `relation` 的同时，仍会把每条已解析的边镜像进遗留表 `graph_edges` +（`engine/src/resolver/pipeline_flush.cpp`）。 -这实现了： -- 向后兼容现有查询 -- 无需停机即可逐步迁移 -- 正确性验证的 A/B 对比 +实测（goagent，1,579 文件，2026-10-01，全新索引）：`entity` 24,545 行、`relation` 7,374 行、 +`graph_edges` 7,377 行、`graph_nodes` 0 行。没有任何查询读取这两张遗留表 +（`engine/src/engine_queries_context.cpp` 记录了这一切换），因此这份边镜像只占磁盘、不影响正确性 —— +但它正是「两张边表可能相差几行」的原因。 ### 4.3 字符串驻留（String Interning） @@ -549,114 +547,15 @@ flowchart LR 1. **tree-sitter 解析时间**：大文件（>1 万行）的主要耗时因素，解析为单线程 2. **SQLite 写入吞吐量**：消费级 SSD 上 WAL 模式约 8 万行/秒 3. **FTS 构建时间**：与符号数线性相关，15 万符号约需 30 秒 -4. **LadybugDB COPY FROM**：CSV 导入快速但需要全量同步——增量同步尚未实现 --- -## 8. LadybugDB 集成 - -### 8.1 概述 - -[LadybugDB](https://ladybugdb.com/) 是一个嵌入式图数据库，CodeScope 将其作为可选的辅助存储后端，提供： - -```mermaid -flowchart LR - subgraph "SQLite" - SQL["关系型查询
精确、可靠、零配置"] - end - - subgraph "LadybugDB" - LBUG["图原生查询 (Cypher)
多跳遍历更高效
可视化支持"] - end - - SQL -->|"CSV 导出 → COPY FROM 同步"| LBUG - LBUG -->|"LadybugDB Explorer
交互式图探索"| Viz["可视化"] -``` - -### 8.2 Schema - -```cypher -// 节点表（对应 graph_nodes） -CREATE NODE TABLE IF NOT EXISTS GraphNode ( - id INT64 PRIMARY KEY, - project_id INT64, - ir_node_id INT64, - node_type INT32, - name STRING, - qualified_name STRING, - signature STRING, - module_path STRING, - file_path STRING, - language STRING, - start_row INT32, - start_col INT32, - end_row INT32, - end_col INT32, - parent_id INT64, - is_entry_point BOOL, - embedding_ready BOOL, - metrics_ready BOOL -); - -// 调用边（从调用者到被调用者） -CREATE REL TABLE IF NOT EXISTS CALLS ( - FROM GraphNode TO GraphNode, - project_id INT64, - edge_type INT32, - call_site_line INT32, - label STRING -); - -// 通用关系 -CREATE REL TABLE IF NOT EXISTS RELATES ( - FROM GraphNode TO GraphNode, - project_id INT64, - type INT32 -); -``` +## 8. LadybugDB 集成（已移除） -### 8.3 同步机制 - -数据通过 CSV 导出 + COPY FROM 从 SQLite 同步到 LadybugDB： - -```mermaid -sequenceDiagram - participant SQL as SQLite - participant CSV as CSV 文件 - participant LBUG as LadybugDB - - SQL->>CSV: 导出 graph_nodes → nodes.csv - SQL->>CSV: 导出 graph_edges → edges.csv - CSV->>LBUG: COPY GraphNode FROM 'nodes.csv' - CSV->>LBUG: COPY CALLS FROM 'edges.csv' - Note over LBUG: 17,127 节点 + 3,341 边
同步完成 -``` - -### 8.4 查询示例 - -```bash -# 打开 LadybugDB shell -lbug .codescope/codescope.lbug - -# 统计所有节点 -MATCH (n:GraphNode) RETURN count(n); - -# 查找所有 Go 函数 -MATCH (n:GraphNode) -WHERE n.language = 'go' -RETURN n.name, n.file_path, n.start_row; - -# 查找目标函数的调用者 -MATCH (n:GraphNode)-[c:CALLS]->(m:GraphNode) -WHERE m.name = 'targetFunction' -RETURN n.name, c.call_site_line; - -# 查找两个函数之间的最短调用路径 -MATCH p = shortestPath( - (a:GraphNode {name: 'funcA'})-[*..10]->(b:GraphNode {name: 'funcB'}) -) -RETURN p; -``` +CodeScope 曾把图镜像进 [LadybugDB](https://ladybugdb.com/)（嵌入式图数据库）作为可选的加速层。 +**该集成已被移除**：代码里不再有 `store_ladybug*.cpp`，`server/build.rs` 也不再链接 `liblbug`， +产品不再生成 `.codescope/codescope.lbug` —— 后端现在是纯 SQLite（`README.zh.md` §7 有同样的说明）。 +设计与集成步骤保留在 git 历史里；所有图查询与搜索工具现在读取 §4 描述的 SQLite 表。 --- @@ -748,10 +647,10 @@ class MyInspector : public Inspector { - **异步运行时**：Tokio 为 FTS 构建和进度轮询提供轻量级异步任务 - **生态系统**：MCP 客户端库、serde JSON 序列化、cargo 依赖管理 -### 为什么 SQLite + LadybugDB 双存储？ +### 为什么用 SQLite，而不是嵌入式图数据库？ - **SQLite**：通用、零配置、久经考验，每个系统都有 SQLite 支持 -- **LadybugDB**：图原生查询，多跳遍历更高效，Cypher 比递归 SQL 更表达力 -- **双策略**：SQLite 保证可靠性和可移植性，LadybugDB 提供性能和可视化 +- **嵌入式图数据库试用后被移除**（LadybugDB，见 §8）：它引入了一个链接依赖和一份需要保持同步的第二存储；而 CodeScope 实际的图负载 —— 调用者/被调用者查询、调用图 BFS、热点分析 —— 由带索引的 `relation` 查询加 CSR 邻接表即可承担 +- **单一存储**：所有工具读取同一批表，图与索引不可能互相矛盾 ### 为什么双进程？ - **崩溃隔离**：解析器段错误不会杀死 MCP 服务器 diff --git a/docs/zh/timbre_model_analysis.md b/docs/zh/timbre_model_analysis.md deleted file mode 100644 index 163060b..0000000 --- a/docs/zh/timbre_model_analysis.md +++ /dev/null @@ -1,150 +0,0 @@ -# InstrumentTimbre 模型架构分析 - -> 项目：基于 ARES 的音色识别/转换模型 -> 索引耗时：0.94s | 142 个 .py 文件 | 日期：2026-07-06 - ---- - -## 一、索引日志 - -``` -开始时间: 2026年 7月 6日 星期一 20时59分53秒 - -worker: project=0 starting index_project dir=～/code/AIproject/music_ai/InstrumentTimbre lang= -BATCH [0..99] of 142 (100 files) -BATCH [0..99] done, total indexed: 100 -BATCH [100..141] of 142 (42 files) -BATCH [100..141] done, total indexed: 142 -POST_BUILD: symbol graph... -buildGraph: 142 files | file_list=3ms delete=1ms rf=0ms r2n=54ms - nodes=46ms edges=92ms calls=0ms total=199ms -{"ok":true,"files_indexed":142,"workers":14, - "time_parse_ms":21,"time_sqlite_ms":234, - "time_buildgraph_ms":199,"time_fts_ms":92,"time_vector_ms":0, - "discovery":{"seen_dirs":296747,"seen_files":293437, - "skipped_dirs":3270,"skipped_files":0, - "skipped_suffix":293213,"candidate_files":142}} - -结束时间: 2026年 7月 6日 星期一 20时59分54秒 -总耗时: 0.94s -``` - -### 关于 ./data 目录 - -`.gitignore` 中已有 `data/` 规则，FilterPolicy 读取后自动跳过。从发现统计可见： -- **293,437 个文件被扫描**（其中 293,213 个因后缀被过滤 — 绝大多数是 data/ 下的音频数据） -- **142 个候选文件**（仅 .py 源码） -- index 阶段干净，无噪音 - ---- - -## 二、模型架构总览 - -```mermaid -flowchart TB - subgraph "输入" - AUDIO["音频输入
(.wav 文件)"] - end - subgraph "Encoder (encoders.py)" - W2V["wav2vec2 预训练
(可选, 768维)"] - CNN["多尺度 CNN
3×3 / 5×5 / 7×7 并行
MultiScaleExtractor"] - SA["Self-Attention
(Conv1d Q/K/V)"] - MHA["多头注意力
(8 heads)"] - POOL["自适应池化
→ 128维特征向量"] - end - subgraph "Decoder (decoders.py)" - FC["全连接层
128→256→num_classes"] - end - subgraph "注意力机制 (attention.py)" - ATTN["SelfAttention
FeatureAttention
CrossAttention
TimbreTransformer
HybridAttention"] - end - - AUDIO --> W2V - AUDIO --> CNN - W2V --> ADAPTER["Adapter MLP
768→512→256→128"] - CNN --> SA - SA --> MHA - ADAPTER --> MHA - MHA --> POOL - POOL --> FC -``` - ---- - -## 三、核心组件 - -### 3.1 Encoder (`models/encoders.py`) - -**InstrumentTimbreEncoder** — 主编码器，将音频转为 128 维音色特征向量。 - -#### 双路径结构 - -| 路径 | 输入 | 处理 | 输出维度 | -|------|------|------|---------| -| **wav2vec2**（可选） | 原始音频 | `torchaudio Wav2Vec2Base → Adapter MLP(768→512→256→128)` | 128 | -| **多尺度 CNN**（默认） | 频谱图 | `Conv2d 3×3 / 5×5 / 7×7 并行 → 拼接 → Self-Attention → 8头注意力 → 池化` | 128 | - -```python -class InstrumentTimbreEncoder(nn.Module): - def __init__(self, input_channels=1, output_dim=128, use_pretrained=False): - self.multi_scale_extractor = nn.ModuleDict({ - 'scale_3': nn.Conv2d(input_channels, 64, 3, padding=1), # 3×3 卷积 - 'scale_5': nn.Conv2d(input_channels, 64, 5, padding=2), # 5×5 卷积 - 'scale_7': nn.Conv2d(input_channels, 64, 7, padding=3), # 7×7 卷积 - }) - # 后续：残差连接 → Self-Attention → 多头注意力 → 自适应池化 → 128维 -``` - -### 3.2 Decoder (`models/decoders.py`) - -| 解码器 | 结构 | 用途 | -|--------|------|------| -| InstrumentTimbreDecoder | `Linear(128, 256) → ReLU → Dropout → Linear(256, num_classes)` | 基础分类 | -| EnhancedTimbreDecoder | 多层 MLP + LayerNorm + GELU + Dropout | 高质量分类 | - -### 3.3 注意力机制 (`models/attention.py`) - -| 模块 | 原理 | 输入→输出 | -|------|------|----------| -| **SelfAttention** | Conv1d Q/K/V 投影 → 点积注意力 → 残差连接 | `[B,C,L] → [B,C,L]` | -| **FeatureAttention** | 全连接层注意力权重 | `[B,D] → [B,D]` | -| **CrossAttention** | 两个序列间的交叉注意力 | `[B,C,L1],[B,C,L2] → [B,C,L2]` | -| **TimbreTransformer** | Transformer 编码器 + LayerNorm | `[B,D,L] → [B,D,L]` | -| **HybridAttention** | Self + Cross 混合 | 组合 | - -SelfAttention 实现细节： -```python -class SelfAttention(nn.Module): - def __init__(self, in_channels): - self.query = Conv1d(in_channels, in_channels//8, 1) # Q投影 - self.key = Conv1d(in_channels, in_channels//8, 1) # K投影 - self.value = Conv1d(in_channels, in_channels, 1) # V投影 - self.gamma = Parameter(torch.zeros(1)) # 可学习的残差权重 - - def forward(self, x): - energy = bmm(proj_query.permute(0,2,1), proj_key) # [B,L,L] 注意力矩阵 - attention = softmax(energy) - out = bmm(proj_value, attention.permute(0,2,1)) - return self.gamma * out + x # 加权残差 -``` - ---- - -## 四、技术栈 - -| 组件 | 用途 | -|------|------| -| PyTorch | 深度学习框架 | -| torchaudio | 音频加载 + wav2vec2 预训练模型 | -| librosa | 音频特征提取（MFCC、频谱等） | -| wav2vec2 | 自监督预训练语音模型（可选） | - -## 五、架构特点 - -1. **双路径编码**：wav2vec2 预训练 + 多尺度 CNN 并行提取，最后融合为 128 维音色向量 -2. **多尺度特征**：3×3 / 5×5 / 7×7 三种 kernel 的 CNN 并行，捕获不同粒度的声学特征 -3. **注意力机制**：Conv1d 实现的 Self-Attention + 8头多头注意力 + 可选的 Transformer -4. **残差连接**：每层都有维度自适应的残差连接（`ResidualConnection`） -5. **GELU 激活**：替代传统 ReLU，训练更稳定 -6. **LayerNorm**：替代 BatchNorm，适合变长音频输入 -7. **输出**：128 维音色特征向量，可用于分类、相似度匹配或音色转换 diff --git a/docs/zh/token_savings_report.md b/docs/zh/token_savings_report.md deleted file mode 100644 index 0bb71d3..0000000 --- a/docs/zh/token_savings_report.md +++ /dev/null @@ -1,99 +0,0 @@ -# Token 节省报告 - -> CodeScope 对比直接读取源文件 —— 以及对比 codebase-memory-mcp —— 基于真实分析场景的实测数据。 -> 硬件：Apple M3 Max, 36 GB RAM。系统：macOS。 - -## 为什么 Token 节省很重要 - -当 LLM 需要理解代码时，有两个选择： -1. **读取原始源文件**——数千行代码，大部分是无关上下文 -2. **查询 CodeScope**——精确获取需要的结构化知识（符号、图、指标） - -CodeScope 相对于原始源码平均减少 **~98.9%** 的 token 消耗，相对于 codebase-memory-mcp 减少 **~85%**。 - -## 各场景实测节省 - -### 对比原始源码 - -| 场景 | CodeScope (tokens) | 原始源码 (tokens) | 节省 | -|------|:------------------:|:-----------------:|:----:| -| 查找函数定义 | ~21 | ~2,265 | **99.1%** | -| 追踪函数调用者 | ~18 | ~2,000 | **99.1%** | -| 项目模块树 (`get_module_tree`) | ~4 | ~1,875 | **99.8%** | -| 项目总览 (`project_overview`) | ~71 | ~1,875 | **96.2%** | -| 函数复杂度分析 | ~43 | ~4,733 | **99.1%** | -| 符号名称搜索 | ~23 | ~958 | **97.6%** | -| USB 驱动子系统概览 | ~250 | ~24,000 | **99.0%** | -| Linux 内核调度器分析 | ~180 | ~15,000 | **98.8%** | -| **平均** | **~76** | **~6,744** | **98.9%** | - -### 对比 codebase-memory-mcp - -| 查询 | CodeScope (tokens) | codebase-memory-mcp (tokens) | 节省 | -|------|:------------------:|:---------------------------:|:----:| -| 图统计 | **~18** | ~5,012 (get_architecture) | **99.6%** | -| 热点 Top 10 | **~488** | ~5,012（含在架构报告） | **90.3%** | -| 搜索 "handler" | **~372** | ~830 (search_graph) + ~828 (search_code) | **55.2%** | -| 入口点 | **~5** | ~5,012（含在架构报告） | **99.9%** | -| 模块树 | **~4** | ❌ 不支持 | — | -| 社区检测（默认） | ❌ 引擎有实现，MCP 未接入 | ❌ 不支持 | — | -| 调用者查询 | **~7** | — | — | -| **典型分析组合** | **~910** | **~6,048** | **85.0%** | - -## 真实基准测试 - -### 项目索引（最终优化版） - -| 项目 | 文件数 | 索引时间 | 节点 | 边 | 数据库大小 | -|------|:-----:|:--------:|:----:|:--:|:---------:| -| ARES Agent (Go) | 95 | **0.31s** | 24,924 | 23,184 | ~250 MB | -| memscope-rs (Rust+C) | 238 | **2.36s** | 123,270 | 108,905 | ~1.2 GB | -| GoAgent (1,167 Go) | 1,167 | **2.89s** | 261,743 | 244,078 | ~2.8 GB | -| Linux 内核 6.14.7（部分） | 6,173 | **~300s**（超时） | — | — | 12 GB | - -### 查询 Token 消耗（逐工具参考） - -| 工具 | Token 消耗 | 何时使用 | -|------|:----------:|---------| -| `get_graph_stats` | **~18** | 快速了解项目规模 | -| `get_module_tree` | **~4** | 项目模块结构概览 | -| `get_entry_points` | **~5** | 查找入口点 | -| `find_callers` / `find_callees` | **~10-50** | 调用图追踪 | -| `get_project_info` | **~44** | 项目元信息 | -| `project_overview` | **~71** | 新项目第一步 | -| `find_definition` / `find_references` | **~20-30** | 符号定位 | -| `get_complexity` | **~20** | 复杂度指标 | -| `locate_code ❌` | **~50-300** | 源码上下文 | -| `codescope_capabilities` | **~309** | 功能就绪状态 | -| `search` | **~300-1000** | 代码搜索 | -| `codescope_trace` | **~50-200** | 调用路径追踪 | -| `detect_changes` | **~100-500** | 变更影响分析 | -| `get_hotspots` | ❌ 未实现 — 无 MCP 工具 | — | -| `codescope_build_context` | **~200-1000** | AI 上下文构建 | -| `get_communities`（默认） | ❌ 引擎有实现，MCP 未接入 | — | - -### 典型分析流程 - -``` -分析管道: project_overview + entry_points + hotspots + search + modules -CodeScope: 71 + 5 + 500 + 1000 + 4 = ~1,580 tokens -codebase-memory: get_architecture(5012) + search_graph(830) + query_graph(206) = ~6,048 tokens - -完整分析：CodeScope 比 codebase-memory-mcp 节省 ~74% tokens - CodeScope 比直接读源码节省 ~98.9% tokens -``` - -## 原理 - -CodeScope 将代码库预索引为结构化知识库（SQLite，15+ 张表）。当 LLM 提问时，CodeScope 只返回**相关事实**——而不是整个源文件。 - -``` -源代码 → CodeScope（事实 + 索引 + 图 + 指标）→ LLM 只获取相关上下文 - ↓ - 对比直接读源码节省 ~98.9% tokens - 对比 codebase-memory 节省 ~85% tokens -``` - -## 方法论 - -采用 DeepSeek 公式估算：`tokens = ASCII字符数 × 0.3 + 非ASCII字符数 × 0.6`。CodeScope 输出从实际 JSON 响应中计数。原始源码按 4 chars/token 估算。codebase-memory-mcp 输出通过 fast 模式索引的实际 JSON 响应计数。 diff --git a/docs/zh/WHY_CODESCOPE.md b/docs/zh/why_codescope.md similarity index 99% rename from docs/zh/WHY_CODESCOPE.md rename to docs/zh/why_codescope.md index e6502cc..e667a6e 100644 --- a/docs/zh/WHY_CODESCOPE.md +++ b/docs/zh/why_codescope.md @@ -543,7 +543,7 @@ index_project → get_index_progress 轮询 ``` 索引返回(1s) → FTS 异步构建(后台) -FTS 就绪前 → 自动降级到 graph_nodes.name LIKE 搜索 +FTS 就绪前 → 自动降级到 entity.name LIKE 搜索 FTS 就绪后 → 自动切换到 FTS5 全文搜索 ``` diff --git a/engine/CMakeLists.txt b/engine/CMakeLists.txt index bc671b6..9bd30bd 100644 --- a/engine/CMakeLists.txt +++ b/engine/CMakeLists.txt @@ -2,12 +2,12 @@ cmake_minimum_required(VERSION 3.30) project(astgraph_engine VERSION 0.1.0 LANGUAGES C CXX) # ── Build Optimization ─────────────────────────────────────────── -if(NOT CMAKE_GENERATOR) - find_program(NINJA ninja) - if(NINJA) - set(CMAKE_GENERATOR "Ninja" CACHE STRING "Build generator") - endif() -endif() +# NOTE: the generator is chosen by the CALLER, not here. CMake sets +# CMAKE_GENERATOR itself before this file is processed, so +# `set(CMAKE_GENERATOR ... CACHE ...)` is a no-op and a project cannot switch +# generators. Selection lives in the Makefile (engine/build) and +# server/build.rs (engine/build-release*); both reuse the generator recorded +# in an existing CMakeCache.txt and only pick Ninja for a fresh tree. find_program(CCACHE_PROGRAM ccache) if(CCACHE_PROGRAM) # Cache BOTH C and C++ — the heavy translation units (sqlite3.c, @@ -174,7 +174,9 @@ message(STATUS "sqlite-vec: source at ${SQLITE_VEC_SRC}") set(ENGINE_SOURCES ${GRAMMAR_SOURCES} src/engine.cpp + src/engine_context.cpp src/engine_helpers.cpp + src/util/json_writer.cpp src/engine_lifecycle.cpp src/engine_index.cpp src/engine_index_project.cpp @@ -182,12 +184,18 @@ set(ENGINE_SOURCES src/engine_index_sched.cpp src/engine_index_discover.cpp src/engine_index_project_membulk.cpp - src/engine_index_post_parse.cpp + src/post_parse_phase.cpp src/engine_index_metrics.cpp src/engine_scanner.cpp src/engine_queries.cpp + src/engine_queries_status.cpp + src/engine_queries_context.cpp src/engine_ffi.cpp + src/engine_filter_ffi.cpp + src/engine_ffi_index.cpp + src/engine_ffi_graph.cpp src/engine_verify_ffi.cpp + src/engine_verify_explain.cpp src/engine_verify_drift_ffi.cpp src/parser/parser.cpp src/ir/ir.cpp @@ -196,6 +204,8 @@ set(ENGINE_SOURCES src/ir/semantic_unit.cpp src/ir/semantic_emitter.cpp src/ir/builtin_registry.cpp + src/ir/builtin_registry_systems.cpp + src/ir/builtin_registry_app.cpp src/ir/translators/python_translator.cpp src/ir/translators/go_translator.cpp src/ir/translators/c_translator.cpp @@ -204,27 +214,34 @@ set(ENGINE_SOURCES src/ir/translators/javascript_translator.cpp src/ir/translators/typescript_translator.cpp src/ir/translators/js_visitor.cpp + src/ir/translators/js_visitor_calls.cpp + src/ir/translators/js_visitor_defined_names.cpp src/ir/translators/ts_visitor.cpp src/ir/translators/tsx_visitor.cpp src/ir/translators/c_visitor.cpp src/ir/translators/cpp_visitor.cpp src/ir/translators/go_visitor.cpp + src/ir/translators/go_visitor_calls.cpp src/ir/translators/python_visitor.cpp src/ir/translators/rust_visitor.cpp src/ir/translators/java_visitor.cpp src/ir/translators/java_translator.cpp - src/ir/translators/scanner_visitor.cpp src/ir/translators/tsx_translator.cpp src/graph/graph_types.cpp src/graph/graph_builder.cpp src/store/store_core.cpp src/store/store_schema.cpp + src/store/store_schema_migrations.cpp + src/store/store_schema_migrations_types.cpp src/store/store.cpp src/store/store_insert.cpp src/store/store_search.cpp + src/store/store_search_vector.cpp src/store/store_batch.cpp - src/store/store_type.cpp src/store/store_graph.cpp + src/store/store_metrics.cpp + src/store/store_graph_csr.cpp + src/store/store_graph_imports.cpp src/store/store_query.cpp src/store/store_project.cpp src/store/store_parse_failure.cpp @@ -232,11 +249,12 @@ set(ENGINE_SOURCES src/store/store_membulk.cpp src/store/store_semantic_fact.cpp src/query/query_engine.cpp + src/query/query_engine_traverse.cpp src/query/query_analysis.cpp + src/query/query_communities.cpp src/query/graph_query.cpp src/query/impact_analysis.cpp src/lsp/lsp_client.cpp - src/resolver/resolver.cpp src/resolver/pipeline.cpp src/resolver/pipeline_apply.cpp src/resolver/pipeline_load.cpp @@ -251,8 +269,6 @@ set(ENGINE_SOURCES src/model/state_builder.cpp src/model/project_state_builder.cpp src/model/semantic_fact_extractor.cpp - src/resolver/project_index.cpp - src/resolver/project_resolver.cpp src/verify/capability_verifier.cpp src/verify/contract_verifier.cpp src/verify/architecture_verifier.cpp @@ -264,13 +280,9 @@ set(ENGINE_SOURCES src/verify/documentation_drift.cpp src/verify/capability_drift.cpp src/verify/architecture_drift.cpp - src/verify/intent_parser.cpp - src/verify/planner.cpp - src/verify/verdict_builder.cpp - src/engine_verify_planner_ffi.cpp src/async_knowledge.cpp - src/resolver/resolve_cache.cpp src/filter_policy.cpp + src/filter_policy_detect.cpp src/filter_policy_ignore.cpp src/platform_win.cpp src/evidence/rule.cpp @@ -352,6 +364,13 @@ target_compile_definitions(astgraph_engine PUBLIC SQLITE_ENABLE_FTS5 SQLITE_THREADSAFE=1) +# Absolute rules-dir default baked in at build time so +# evidence::resolveRulesDir() can find engine/src/evidence/rules +# regardless of the process working directory (the MCP/CLI vehicles +# run inside the target project, where the relative path is absent). +target_compile_definitions(astgraph_engine PRIVATE + CODESCOPE_RULES_DIR_DEFAULT="${CMAKE_CURRENT_SOURCE_DIR}/src/evidence/rules") + target_include_directories(astgraph_engine PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include ${CMAKE_CURRENT_SOURCE_DIR}/src @@ -397,6 +416,13 @@ if(BUILD_TESTS) foreach(test_src ${TEST_SOURCES}) get_filename_component(test_name ${test_src} NAME_WE) add_executable(${test_name} ${test_src}) + # Tests use CHECK() from tests/test_check.h, which NDEBUG cannot + # compile out (REVIEW_0.2.7 TEST-3). -UNDEBUG is kept as a second line + # of defence for anything that still routes through the C library, so + # a Release/RelWithDebInfo configure cannot run the suite vacuously. + target_compile_options(${test_name} PRIVATE + $<$:/UNDEBUG> + $<$>:-UNDEBUG>) target_link_libraries(${test_name} astgraph_engine) add_test(NAME ${test_name} COMMAND ${test_name}) endforeach() diff --git a/engine/include/engine.h b/engine/include/engine.h index adf85fc..a5f3780 100644 --- a/engine/include/engine.h +++ b/engine/include/engine.h @@ -7,14 +7,56 @@ extern "C" { #endif -// ─── Lifecycle ──────────────────────────────────────────────── - -int engine_init(const char *db_path); -void engine_shutdown(); +// ─── Engine instance (opaque handle) ────────────────────────── +// +// TD-1 (docs/REVIEW_0.2.7.md): the instance used to be process-global state +// that every entry point reached for implicitly. It is now created explicitly +// and passed in: every stateful function below takes an `engine_t` as its +// first parameter. +// +// Ownership: engine_create() transfers one instance to the caller, which MUST +// release it with engine_destroy() exactly once. Destroying a handle twice, +// or using it after engine_destroy(), is undefined behaviour. +// Lifetime: a handle returned by engine_create() is valid until +// engine_destroy(handle). Nothing else frees it. +// Thread safety: one instance serialises its own calls over a single store +// connection, so calls that mutate the store must not overlap an indexing +// call on the SAME handle. Distinct handles have distinct stores, but the +// verifier registry and the async knowledge builder are still process-wide +// (documented limitation, engine_context.h), so run one instance per process +// unless those subsystems have been audited for your use. +typedef struct CodescopeEngine *engine_t; + +// Create an engine instance bound to `db_path`, opening (and if necessary +// creating) its SQLite database and registering the statically linked +// tree-sitter grammars. +// @param db_path Path of the SQLite file. NULL or "" is rejected. +// @return A new instance, or NULL when the input is rejected or the store +// cannot be opened. The reason is logged to stderr with module/method +// tags (module=ffi, method=engine_create). +// Ownership: the caller owns the returned handle and releases it with +// engine_destroy(). +// Lifetime: input string is borrowed for the call only. +// Thread safety: must not run concurrently with engine_destroy() on the same +// handle, nor with any other call on it. +engine_t engine_create(const char *db_path); + +// Destroy an engine instance: waits for the background knowledge builder, +// clears the verifier registry, closes the store, and releases the handle. +// Accepts NULL as a no-op, so calling it on a failed engine_create() is safe. +// @param handle Instance returned by engine_create(); may be NULL. +// Ownership: consumes `handle` — it is invalid after this call, and passing it +// to any other entry point is undefined behaviour. +// Lifetime: no pointers are retained. +// Thread safety: must not run while any call on `handle` is in flight. +void engine_destroy(engine_t handle); /// Returns the engine version string. The returned pointer is static /// and does NOT need to be freed. /// @return Static C string like "0.2.1" +// Ownership: nothing is allocated; no free is needed. +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. const char *engine_version(void); // ─── FFI binding status (L3 fix) ─────────────────────────────── @@ -24,7 +66,7 @@ const char *engine_version(void); // for completeness and are kept (not removed) because some are used by // scripts or a future CLI, but they must NOT be treated as server-backed // APIs. Functions NOT bound by the server include: -// engine_get_communities, engine_get_hotspots, engine_get_module_map, +// engine_get_hotspots, engine_get_module_map, // engine_get_entry_points, engine_trace_call_chain, engine_get_callers, // engine_get_callees, engine_get_complexity, engine_get_capabilities, // engine_get_index_progress, engine_scan_project, engine_search_semantic, @@ -34,43 +76,121 @@ const char *engine_version(void); // ─── Project ────────────────────────────────────────────────── -uint64_t engine_create_project(const char *root_path, const char *name); +// Ownership: nothing is allocated; no free is needed. +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +uint64_t engine_create_project(engine_t handle, const char *root_path, + const char *name); // ─── Parsing & Indexing ─────────────────────────────────────── // Parse one file and build IR + graph; returns JSON status {"ok": true/false, "error": "..."} -char *engine_index_file(uint64_t project_id, const char *file_path); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_index_file(engine_t handle, uint64_t project_id, + const char *file_path); // Index an entire directory recursively; returns JSON progress summary -char *engine_index_project(uint64_t project_id, const char *dir_path, - const char *language_filter); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_index_project(engine_t handle, uint64_t project_id, + const char *dir_path, const char *language_filter); // Index a list of files (JSON array of file paths); returns JSON progress summary. // Skips directory scanning — uses the same parallel worker infrastructure. // file_list_json: ["/path/to/file1.c", "/path/to/file2.c", ...] -char *engine_index_files(uint64_t project_id, const char *file_list_json); +// +// bypass_fail_fast selects the fail-fast policy for the requested files: +// 0 — honour it, like the automatic project path: a file whose parse has +// failed CODESCOPE_FAIL_RETRY_MAX times (default +// engine_index_sched::kDefaultFailRetryMax) is skipped. Used by the +// scheduler-driven callers (worker --file-list, chunk worker). +// 1 — always re-attempt every listed file. Used by force_index_files, whose +// contract is "index these paths regardless of the default skip rules". +// Either way, parse failures of this run are recorded (and flushed) into +// parse_failures; files failing only because no grammar is vendored for their +// language are exempt from the skip in both modes. +// +// Ownership: the engine allocates the returned JSON string; the caller frees it +// with engine_free_string(). Nothing else is retained — the file list is parsed +// and copied during the call. +// Thread safety: NOT safe to call while another indexing call is running. The +// engine holds one store connection, and this call joins the background +// knowledge builder and takes the store write guard for the duration. +char *engine_index_files(engine_t handle, uint64_t project_id, + const char *file_list_json, int bypass_fail_fast); + +// ─── Parse-failure Maintenance ──────────────────────────────── + +// Read the project's parse_failures rows (the files the indexer could not +// parse, with their reason and retry count). +// limit: maximum rows to return; <= 0 means the store default (100). +// Returns {"ok":true,"parse_failures":[...]} or an {"ok":false,...} envelope. +// Ownership: caller frees with engine_free_string(); no other state is touched. +// Thread safety: read-only, but it shares the engine's single store +// connection, so it must not run concurrently with an indexing call. +char *engine_get_parse_failures(engine_t handle, uint64_t project_id, + int limit); + +// Delete every parse_failures row of the project, so the files become eligible +// for the fail-fast skip again (they are re-attempted on the next index run). +// Returns {"ok":true,"removed":N} or an {"ok":false,...} envelope. +// Ownership: caller frees with engine_free_string(). +// Thread safety: same connection as above — not concurrent with indexing. +char *engine_reset_parse_failures(engine_t handle, uint64_t project_id); // ─── Queries ────────────────────────────────────────────────── -char *engine_find_definition(uint64_t project_id, const char *symbol_name, - const char *file_filter); -char *engine_find_references(uint64_t project_id, const char *symbol_name, - const char *file_filter); -char *engine_get_callers(uint64_t project_id, const char *function_name, - const char *file_filter); -char *engine_get_callees(uint64_t project_id, const char *function_name, - const char *file_filter); -char *engine_get_neighbors(uint64_t project_id, uint64_t node_id, - int edge_type_filter, int radius); -char *engine_find_shortest_path(uint64_t project_id, uint64_t source_id, - uint64_t target_id); -char *engine_get_subgraph(uint64_t project_id, uint64_t center_node_id, - int radius, const char *node_type_filter, +char *engine_find_definition(engine_t handle, uint64_t project_id, + const char *symbol_name, const char *file_filter); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_find_references(engine_t handle, uint64_t project_id, + const char *symbol_name, const char *file_filter); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_callers(engine_t handle, uint64_t project_id, + const char *function_name, const char *file_filter); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_callees(engine_t handle, uint64_t project_id, + const char *function_name, const char *file_filter); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_neighbors(engine_t handle, uint64_t project_id, + uint64_t node_id, int edge_type_filter, int radius); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_find_shortest_path(engine_t handle, uint64_t project_id, + uint64_t source_id, uint64_t target_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_subgraph(engine_t handle, uint64_t project_id, + uint64_t center_node_id, int radius, + const char *node_type_filter, const char *edge_type_filter); -char *engine_locate_node(uint64_t project_id, uint64_t node_id, +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_locate_node(engine_t handle, uint64_t project_id, uint64_t node_id, int context_lines); -char *engine_locate_by_name(uint64_t project_id, const char *name); -char *engine_get_graph_stats(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_locate_by_name(engine_t handle, uint64_t project_id, + const char *name); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_graph_stats(engine_t handle, uint64_t project_id); // Find connected components in the call graph via BFS over name-matched // relation edges. Returns JSON: @@ -80,7 +200,10 @@ char *engine_get_graph_stats(uint64_t project_id); // "total":N,"approximation":"heuristic", // "note":"Connected components computed on name-matched call edges."} // On error returns JSON with an "error" field. -char *engine_find_connected_components(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_find_connected_components(engine_t handle, uint64_t project_id); // ─── Interactive exploration ────────────────────────────────── // Explore a function's callers/callees recursively as a JSON tree. @@ -88,51 +211,100 @@ char *engine_find_connected_components(uint64_t project_id); // @param function_name Starting function. // @param depth How many levels to recurse (max 5). // @param direction "callers", "callees", or "both". -char *engine_explore_function(uint64_t project_id, const char *function_name, - int depth, const char *direction); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_explore_function(engine_t handle, uint64_t project_id, + const char *function_name, int depth, + const char *direction); // Get the latest project ID from the database. // Returns the project with the most indexed data (graph_nodes), // not just the highest id. This prevents project_id misalignment // when an empty "shell" project has a higher id than the data-bearing // project. Returns 0 if no projects exist. -uint64_t engine_get_latest_project_id(); +// Ownership: nothing is allocated; no free is needed. +// Lifetime: no pointers are passed in. +// Thread safety: shares one store connection; not concurrent with indexing. +uint64_t engine_get_latest_project_id(engine_t handle); // Get a project ID by its root_path. // Returns the project_id, or 0 if no project matches the root_path. // Unlike engine_create_project, this is a pure lookup — it does NOT // create a new project if the root_path is not found. -uint64_t engine_get_project_id_by_path(const char *root_path); +// Ownership: nothing is allocated; no free is needed. +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +uint64_t engine_get_project_id_by_path(engine_t handle, const char *root_path); // Count graph_nodes for a project. // Returns the node count, or 0 if the project has no indexed data. // Used by MCP to decide whether to reuse existing data or re-index. -uint64_t engine_get_project_node_count(uint64_t project_id); +// Ownership: nothing is allocated; no free is needed. +// Lifetime: no pointers are passed in. +// Thread safety: shares one store connection; not concurrent with indexing. +uint64_t engine_get_project_node_count(engine_t handle, uint64_t project_id); // ─── Full-text search ────────────────────────────────────────── // Search code by name or content using FTS5; returns JSON with matching nodes -char *engine_search_code(uint64_t project_id, const char *query, int limit); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_search_code(engine_t handle, uint64_t project_id, + const char *query, int limit); // Semantic search using n-gram vector similarity (matches similar names) -char *engine_search_semantic(uint64_t project_id, const char *query, int limit); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_search_semantic(engine_t handle, uint64_t project_id, + const char *query, int limit); // Build FTS index from graph data (async, non-blocking for graph queries) -char *engine_build_fts(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_build_fts(engine_t handle, uint64_t project_id); // Get current index progress as JSON (for client polling) -char *engine_get_index_progress(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_get_index_progress(engine_t handle, uint64_t project_id); // ─── Complexity analysis ─────────────────────────────────────── // Get cyclomatic complexity, cognitive complexity, and nesting depth for a graph node -char *engine_get_complexity(uint64_t project_id, uint64_t graph_node_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_complexity(engine_t handle, uint64_t project_id, + uint64_t graph_node_id); // ─── Graph Query DSL ────────────────────────────────────────── // Execute a minimal graph pattern query: MATCH (srcType[:name])-[edgeType]->(tgtType[:name]) // Returns JSON array of {source, edge, target} triples. -char *engine_graph_query(uint64_t project_id, const char *dsl_query); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_graph_query(engine_t handle, uint64_t project_id, + const char *dsl_query); + +// Direct graph slice: up to node_limit nodes starting at node_offset and +// edge_limit edges starting at edge_offset, each optionally filtered by type +// (NULL or "" = no filter). This is the paged viewer path behind the MCP +// `get_graph` tool; the full graph of a large project is never materialised in +// one call. +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: reads the shared store connection; not concurrent with +// indexing on the same handle. +char *engine_get_graph(engine_t handle, uint64_t project_id, + int64_t node_offset, int node_limit, int64_t edge_offset, + int edge_limit, const char *node_type_filter, + const char *edge_type_filter); // ─── Change Impact Analysis ───────────────────────────────────── @@ -144,32 +316,48 @@ char *engine_graph_query(uint64_t project_id, const char *dsl_query); * e.g. '["/path/to/file1.py","/path/to/file2.rs"]'. * @return JSON with "modified", "callers", "callees", and "total_impacted". */ -char *engine_detect_changes(uint64_t project_id, +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_detect_changes(engine_t handle, uint64_t project_id, const char *modified_files_json); // ─── Community Detection ──────────────────────────────────────── /** - * Run label-propagation community detection on the code graph. + * Run deterministic label-propagation community detection over the CALLS + * graph (relation type=1, treated as undirected; self-loops and isolated + * nodes are excluded). Each participating node starts in its own community + * and iteratively adopts the most common label among its neighbors; ties + * prefer the current label, then the smallest label id, so the same database + * always yields the same communities. * - * Each node starts in its own community and iteratively adopts the - * most common community label among its neighbors. Returns module - * clusters and their inter-relationships. + * Implemented in engine/src/query/query_communities.cpp. * * @param project_id The project to analyze. - * @param max_members Maximum members per community in output. - * Set to a small value (e.g. 5-10) to avoid large token output. - * 0 = include all members. - * @param max_communities Maximum communities to return. - * Set to e.g. 20 to limit output size and token cost. - * 0 = return all communities. - * @param include_members If non-zero, include member list in each community. - * If 0 (default), only return {id, label, member_count} summary. - * @return JSON with "communities" array, "inter_community_edges", - * and "total_communities". + * @param max_members Maximum members emitted per community. Non-positive + * means "use the default" (10), NOT "unlimited"; + * clamped to 200. Only used when include_members is set. + * @param max_communities Maximum communities returned. Non-positive means + * "use the default" (20), NOT "unlimited"; clamped + * to 500. + * @param include_members If non-zero, include the member list. If 0 (the + * default) only {id, label, member_count} summaries + * are returned, keeping the payload small. + * @return JSON: + * {"communities":[{"id":,"label":, + * "member_count":N[,"members":[...]]}], + * "total_communities":N,"returned_communities":N, + * "inter_community_edges":N,"truncated":bool, + * "approximation":"heuristic","note":"..."} + * On error the JSON contains an "error" field and an empty community list. */ -char *engine_get_communities(uint64_t project_id, int max_members, - int max_communities, int include_members); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_communities(engine_t handle, uint64_t project_id, + int max_members, int max_communities, + int include_members); // ─── Hotspot Analysis ─────────────────────────────────────────── @@ -180,22 +368,37 @@ char *engine_get_communities(uint64_t project_id, int max_members, * @param top_n Number of top hotspots to return (default 10). * @return JSON: { "hotspots": [ { "name","caller_count","complexity",... } ] } */ -char *engine_get_hotspots(uint64_t project_id, int top_n); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_hotspots(engine_t handle, uint64_t project_id, int top_n); // ─── Code Understanding ───────────────────────────────────────── // Get a module map: all directories with their functions and methods -char *engine_get_module_map(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_module_map(engine_t handle, uint64_t project_id); // Find likely entry points (main, run, start, init, setup) -char *engine_get_entry_points(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_entry_points(engine_t handle, uint64_t project_id); // Trace call chain between two named functions -char *engine_trace_call_chain(uint64_t project_id, const char *from, - const char *to); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_trace_call_chain(engine_t handle, uint64_t project_id, + const char *from, const char *to); // Complete project overview: modules + entry points + hotspots + stats -char *engine_get_project_overview(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_project_overview(engine_t handle, uint64_t project_id); // ─── Phase A: Fast Scan & Query ─────────────────────────────── @@ -205,20 +408,30 @@ char *engine_get_project_overview(uint64_t project_id); * Returns JSON with modules, entry_points, and total_symbols count. * Designed for ms-level response time. */ -char *engine_scan_project(uint64_t project_id, const char *dir_path, - const char *language_filter); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_scan_project(engine_t handle, uint64_t project_id, + const char *dir_path, const char *language_filter); /** * Get hierarchical module tree for a project. * Returns JSON with modules array containing id, parent_id, name, path, file_count. */ -char *engine_get_module_tree(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_module_tree(engine_t handle, uint64_t project_id); /** * Find symbol(s) by exact name match. * Returns JSON with results array. */ -char *engine_find_symbol(uint64_t project_id, const char *symbol_name); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_find_symbol(engine_t handle, uint64_t project_id, + const char *symbol_name); // ─── Phase B: Background Enhancement ────────────────────────── @@ -228,13 +441,19 @@ char *engine_find_symbol(uint64_t project_id, const char *symbol_name); * Returns JSON summary: { "files_processed": N, "symbols_enhanced": N, "call_edges": N } * Call this asynchronously after engine_scan_project returns. */ -char *engine_enhance_project(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_enhance_project(engine_t handle, uint64_t project_id); /** * Get enhancement status — how many symbols have ready flags set. * Returns JSON: { "total_symbols": N, "callgraph_ready": N, "metrics_ready": N, "embedding_ready": N } */ -char *engine_get_enhancement_status(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_enhancement_status(engine_t handle, uint64_t project_id); // ─── Phase C: Unified MCP Tools (adaptive backend) ──────────── @@ -242,7 +461,11 @@ char *engine_get_enhancement_status(uint64_t project_id); * Unified search: auto-selects between FTS5 and semantic search. * Returns JSON with results and method field indicating which engine was used. */ -char *engine_unified_search(uint64_t project_id, const char *query, int limit); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_unified_search(engine_t handle, uint64_t project_id, + const char *query, int limit); /** * Find callers (adaptive): uses new call_edges table if callgraph_ready, @@ -253,7 +476,11 @@ char *engine_unified_search(uint64_t project_id, const char *query, int limit); * homonyms (same name across files/classes). NULL = aggregate * all files (legacy behavior). */ -char *engine_find_callers_adaptive(uint64_t project_id, const char *symbol_name, +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_find_callers_adaptive(engine_t handle, uint64_t project_id, + const char *symbol_name, const char *file_filter); /** @@ -264,7 +491,11 @@ char *engine_find_callers_adaptive(uint64_t project_id, const char *symbol_name, * restricts the caller to the given file, disambiguating * homonyms. NULL = aggregate all files (legacy behavior). */ -char *engine_find_callees_adaptive(uint64_t project_id, const char *symbol_name, +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_find_callees_adaptive(engine_t handle, uint64_t project_id, + const char *symbol_name, const char *file_filter); /** @@ -278,7 +509,11 @@ char *engine_find_callees_adaptive(uint64_t project_id, const char *symbol_name, * @param entity_id The entity.id from the entity table. * @return JSON: {"callers":[...],"total":N,"entity_id":ID} */ -char *engine_find_callers_by_entity(uint64_t project_id, uint64_t entity_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_find_callers_by_entity(engine_t handle, uint64_t project_id, + uint64_t entity_id); /** * Step 7 (plan §7.2): Find callees by entity ID. @@ -288,28 +523,44 @@ char *engine_find_callers_by_entity(uint64_t project_id, uint64_t entity_id); * @param entity_id The entity.id from the entity table. * @return JSON: {"callees":[...],"total":N,"entity_id":ID} */ -char *engine_find_callees_by_entity(uint64_t project_id, uint64_t entity_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_find_callees_by_entity(engine_t handle, uint64_t project_id, + uint64_t entity_id); /** * Get entry points from the new schema. */ -char *engine_get_entry_points_new(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_entry_points_new(engine_t handle, uint64_t project_id); /** * Get a comprehensive project overview: languages, modules, symbols, * entry points, analysis progress, and ready features. * This is the first tool AI should call after initialization. */ -char *engine_project_overview(uint64_t project_id); -char *engine_detect_ffi_boundaries(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_project_overview(engine_t handle, uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_detect_ffi_boundaries(engine_t handle, uint64_t project_id); /** * Trace the shortest call path between two functions using BFS on call_edges. * Returns JSON: {"path": [{"name":"...","file":"...","line":N}, ...]} * Requires callgraph_ready (run enhance_project first). */ -char *engine_trace_path(uint64_t project_id, const char *from_name, - const char *to_name); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_trace_path(engine_t handle, uint64_t project_id, + const char *from_name, const char *to_name); /** * Build an intelligent context bundle for a natural language query. @@ -317,14 +568,21 @@ char *engine_trace_path(uint64_t project_id, const char *from_name, * entry points, call graph) based on the query and ready flags. * This is the primary tool LLM should call — replaces manual multi-tool chains. */ -char *engine_build_context(uint64_t project_id, const char *query); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_build_context(engine_t handle, uint64_t project_id, + const char *query); /** * Get a standardized capabilities report — what features are available * and their readiness status. Returns JSON with each capability's name, * available flag, and ready flag. */ -char *engine_get_capabilities(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_capabilities(engine_t handle, uint64_t project_id); /** * Direct-query the knowledge graph layer (v0.2.1). @@ -343,18 +601,32 @@ char *engine_get_capabilities(uint64_t project_id); * On error: `{"error":"[module=ffi, method=engine_get_knowledge_graph] ..."}`. * Caller must call `engine_free_string` on the result. */ -char *engine_get_knowledge_graph(uint64_t project_id, const char *table_name, - int32_t limit); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_knowledge_graph(engine_t handle, uint64_t project_id, + const char *table_name, int32_t limit); // Get type info for a project: returns type definitions and their references. // Returns JSON: { "types": [ { "name","qualified_name","kind","file_path","ref_count" } ] } -char *engine_get_type_info(uint64_t project_id, const char *type_name_filter); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_type_info(engine_t handle, uint64_t project_id, + const char *type_name_filter); // Get HTTP routes for a project. Returns JSON: { "routes": [ { "method","path","handler_name","file_path","line" } ] } -char *engine_get_routes(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_routes(engine_t handle, uint64_t project_id); // ─── Memory ─────────────────────────────────────────────────── +// Ownership: takes back a pointer the engine allocated (e.g. a JSON +// envelope) and releases it; the caller must not free it again. +// Lifetime: `ptr` is invalid after this call — do not use or free it again. +// Thread safety: safe to call at any time; it only releases memory. void engine_free_string(char *ptr); /// Rebuild a project's CSR adjacency/adjacency_rev tables on the given DB. @@ -369,6 +641,9 @@ void engine_free_string(char *ptr); /// @param project_id Project whose CSR to rebuild. /// @return JSON `{"ok":true,"project_id":N}` on success, or a JSON error /// object. Caller MUST free via engine_free_string(). +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. char *engine_rebuild_csr(const char *db_path, uint64_t project_id); // ─── Batch Indexing ──────────────────────────────────────────── @@ -380,7 +655,11 @@ char *engine_rebuild_csr(const char *db_path, uint64_t project_id); * @param file_paths_json JSON array of file paths, e.g. ["/a.go","/b.rs"]. * @return JSON summary: {"ok":true, "files":N, "nodes":N, "edges":N, "errors":[...]} */ -char *engine_index_batch(uint64_t project_id, const char *file_paths_json); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_index_batch(engine_t handle, uint64_t project_id, + const char *file_paths_json); // ─── Project Metadata ───────────────────────────────────────── @@ -393,7 +672,10 @@ char *engine_index_batch(uint64_t project_id, const char *file_paths_json); * @return JSON: {"name":"...", "license":"Apache-2.0", "language":"go", * "file_count":N, "dependency_count":N, "detected_files":["..."]} */ -char *engine_get_project_info(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_project_info(engine_t handle, uint64_t project_id); // ─── Shared Artifact ──────────────────────────────────────────── @@ -407,7 +689,11 @@ char *engine_get_project_info(uint64_t project_id); * @return JSON: {"ok":true, "size_bytes":N, "compressed_bytes":N} * or {"ok":false, "error":"..."} on failure. */ -char *engine_export_artifact(uint64_t project_id, const char *output_path); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_export_artifact(engine_t handle, uint64_t project_id, + const char *output_path); /** * Import a previously exported artifact. @@ -418,7 +704,11 @@ char *engine_export_artifact(uint64_t project_id, const char *output_path); * @return JSON: {"ok":true, "project_id":N} * or {"ok":false, "error":"..."} on failure. */ -char *engine_import_artifact(uint64_t project_id, const char *artifact_path); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_import_artifact(engine_t handle, uint64_t project_id, + const char *artifact_path); // ─── Evidence Builder (v0.3 Phase 2) ──────────────────────────── @@ -436,39 +726,11 @@ char *engine_import_artifact(uint64_t project_id, const char *artifact_path); * combine). On error returns a JSON object with an "error" * field. Caller MUST free via engine_free_string(). */ -char *engine_build_evidence(uint64_t project_id, const char *category_filter); - -// ─── Verification Planner (v0.3 Phase 3) ─────────────────────── - -/** - * Verify a natural-language claim against the project's indexed - * evidence. The claim is parsed into an Intent by IntentParser, - * planned into evidence rule executions by Planner, executed via - * EvidenceBuilder, and aggregated into a Verdict by VerdictBuilder. - * - * The returned JSON has the shape: - * { - * "verdict": "Supported|Contradicted|PartiallyVerified|Unknown", - * "confidence": 0.0..1.0, - * "requirements": [ - * {"id":"...","weight":N,"satisfied":bool,"confidence":N}, ... - * ], - * "evidence": [ - * {"category":"...","title":"...","confidence":N, - * "item_count":N}, ... - * ] - * } - * - * On error (engine not initialized, empty claim, etc.) returns a - * JSON object with an "error" field. Caller MUST free via - * engine_free_string(). - * - * @param project_id The project whose semantic_fact rows to query. - * @param claim_text The natural-language claim (e.g. "does this - * project safely handle CString?"). - * @return Heap-allocated JSON string (caller frees). Never null. - */ -char *engine_verify_statement(uint64_t project_id, const char *claim_text); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_build_evidence(engine_t handle, uint64_t project_id, + const char *category_filter); // ─── Claim-driven Verification (v0.3) ────────────────────────── @@ -503,7 +765,11 @@ char *engine_verify_statement(uint64_t project_id, const char *claim_text); * @return Heap-allocated JSON string (caller frees via * engine_free_string). Never null. */ -char *engine_verify_claim(uint64_t project_id, const char *claim_json); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_verify_claim(engine_t handle, uint64_t project_id, + const char *claim_json); // ─── Project State (v0.3 Phase 4) ────────────────────────────── @@ -519,7 +785,10 @@ char *engine_verify_claim(uint64_t project_id, const char *claim_json); * returns a JSON object with an "error" field. Caller MUST * free via engine_free_string(). */ -char *engine_build_project_state(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: writes the store; not concurrent with another engine call. +char *engine_build_project_state(engine_t handle, uint64_t project_id); /** * Get the persisted project state snapshot (without rebuilding). @@ -531,7 +800,10 @@ char *engine_build_project_state(uint64_t project_id); * returns a JSON object with an "error" field and the * project_id. Caller MUST free via engine_free_string(). */ -char *engine_get_project_state(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_project_state(engine_t handle, uint64_t project_id); // ─── Missing FFI declarations (v0.2.5 completeness fix) ─────── // These entry points are implemented in the engine and bound by the Rust @@ -541,34 +813,73 @@ char *engine_get_project_state(uint64_t project_id); // that the caller MUST free via engine_free_string(). /// Run integrity verification (dispatch via VerifierRegistry). Returns JSON. -char *engine_verify_integrity(uint64_t project_id); +/// `max_findings` caps the serialized `findings` array (0 = default cap). +/// `truncated` is true when the array was cut short; `total` counts every +/// verdict (including Supported, which are not listed as findings). +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_verify_integrity(engine_t handle, uint64_t project_id, + int max_findings); /// Parse a natural-language summary into claims and verify them. -char *engine_verify_summary(uint64_t project_id, const char *text); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_verify_summary(engine_t handle, uint64_t project_id, + const char *text); /// Verify the project against a review checklist (natural language text). -char *engine_verify_review(uint64_t project_id, const char *text); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_verify_review(engine_t handle, uint64_t project_id, + const char *text); /// Verify whether the project's reality matches a described expectation. -char *engine_verify_reality(uint64_t project_id, const char *text); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_verify_reality(engine_t handle, uint64_t project_id, + const char *text); /// Scan declared capabilities/contracts for documentation-vs-code drift. -char *engine_detect_drift(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_detect_drift(engine_t handle, uint64_t project_id); /// Detect documentation drift specifically (comments vs actual code). -char *engine_detect_documentation_drift(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_detect_documentation_drift(engine_t handle, uint64_t project_id); /// Detect capability drift specifically (declared vs implemented). -char *engine_detect_capability_drift(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_detect_capability_drift(engine_t handle, uint64_t project_id); /// Detect architecture drift specifically (modules/layers vs actual deps). -char *engine_detect_architecture_drift(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_detect_architecture_drift(engine_t handle, uint64_t project_id); /// Explain a symbol in natural language. Returns JSON. -char *engine_explain_symbol(uint64_t project_id, const char *symbol_name); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_explain_symbol(engine_t handle, uint64_t project_id, + const char *symbol_name); /// Explain a module in natural language. Returns JSON. -char *engine_explain_module(uint64_t project_id, const char *module_name); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_explain_module(engine_t handle, uint64_t project_id, + const char *module_name); /** * Inspect the VerifierRegistry health and claim-type coverage. @@ -597,7 +908,47 @@ char *engine_explain_module(uint64_t project_id, const char *module_name); * Pass 0 to skip the backend probe. * @return Heap-allocated JSON string (caller frees). Never null. */ -char *engine_get_verifier_registry_status(uint64_t project_id); +// Ownership: the caller frees the returned JSON string with engine_free_string(). +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +char *engine_get_verifier_registry_status(engine_t handle, uint64_t project_id); + +/** + * Ask the engine's FilterPolicy whether a path would be skipped. + * + * Exported so the server's module discovery asks the same implementation the + * indexer uses instead of keeping a copy of the skip rules: the copy is what + * let a `.gitignore`d top-level directory be reported as a module (it read no + * ignore file at all). `project_root` selects which .gitignore / + * .codescopeignore rules apply and caches one FilterPolicy per root; + * `rel_path` is relative to that root, exactly as the worker passes it. + * + * @param project_root Project root whose ignore files apply (may be NULL). + * @param rel_path Path relative to project_root; NULL or "" → not skipped. + * @param is_dir Non-zero when the path names a directory. + * @return 1 if the path would be skipped, 0 otherwise (also 0 on any error). + */ +// Ownership: nothing is allocated; no free is needed. +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +int engine_path_is_skipped(const char *project_root, const char *rel_path, + int is_dir); + +/** + * Ask whether the engine can parse a file (by extension, or shebang for + * extensionless scripts). + * + * Exported so the server's file counts cannot advertise a language the engine + * has no support for — `.zig` was counted while the engine has no Zig + * mapping or translator. + * + * @param file_path Path or file name; NULL or "" → 0. + * @return 1 if the engine recognizes a language for it, 0 otherwise. + */ +// Ownership: nothing is allocated; no free is needed. +// Lifetime: input strings are borrowed for the call only. +// Thread safety: shares one store connection; not concurrent with indexing. +int engine_is_indexable_source(const char *file_path); #ifdef __cplusplus } diff --git a/engine/src/ahocorasick.h b/engine/src/ahocorasick.h index ffeccb2..4decea2 100644 --- a/engine/src/ahocorasick.h +++ b/engine/src/ahocorasick.h @@ -25,6 +25,16 @@ class ACAutomaton { public: ACAutomaton() = default; + // The automaton owns the nodes it allocates in addPattern(), so a copy + // would leave two objects pointing at the same nodes and free them twice. + // Deleting the copy constructor also removes the floor under a subtler + // trap: because a destructor is user-declared, the move constructor is not + // implicitly generated, so `return ac;` from a builder silently fell back + // to the COPY constructor. Callers that need a shared instance must hold it + // in a function-local static object instead (see getJsAC()). + ACAutomaton(const ACAutomaton &) = delete; + ACAutomaton &operator=(const ACAutomaton &) = delete; + ~ACAutomaton() { // Free all dynamically allocated nodes diff --git a/engine/src/async_knowledge.cpp b/engine/src/async_knowledge.cpp index 9c91070..f3be14a 100644 --- a/engine/src/async_knowledge.cpp +++ b/engine/src/async_knowledge.cpp @@ -10,10 +10,12 @@ #include #include +#include #include #include #include #include +#include #include // ─── module_edge population ───────────────────────────────────── @@ -34,12 +36,75 @@ namespace // Atomic flag ensuring only one async builder runs at a time. std::atomic g_async_running{ false }; -// Joinable thread handle — NOT detached, so engine_shutdown can join it -// before destroying g_store. Protected by g_thread_mutex to prevent +// Joinable thread handle — NOT detached, so engine_destroy can join it +// before destroying the engine store. Protected by g_thread_mutex to prevent // data races between launch (writer) and join (reader in shutdown). std::mutex g_thread_mutex; std::thread g_builder_thread; +// Serializes the builder body against every FFI entry that touches the +// shared engine store connection. The builder holds it for its whole body; +// waitForKnowledgeBuilder() hands it to the caller for the duration of +// one read/write. Without this, a builder launched by a later index call +// could BEGIN/COMMIT while a read is still stepping statements on the +// same connection (prior review residual #16). +// +// recursive_timed_mutex: FFI entry points nest (e.g. enhance calls through +// paths that also call waitForKnowledgeBuilder), so a plain mutex deadlocks +// the same thread on the second acquisition. The timed variant lets the +// caller bound the wait (see kStoreLockTimeoutMs) instead of hanging +// forever if the builder thread wedges. +std::recursive_timed_mutex g_store_mutex; + +// Signalled when the builder thread body finishes, so join can wait with +// a timeout instead of blocking forever on a wedged builder. +std::mutex g_done_mutex; +std::condition_variable g_done_cv; +bool g_builder_done = true; + +// Upper bound on how long joinAsyncKnowledgeBuilder waits for a running +// builder. Named (no magic number): 60 s is far longer than every measured +// build (the slowest observed was ~10 s on a 1.5k-file project) and short +// enough that a wedged builder cannot hang the MCP server forever. +inline constexpr int kBuilderJoinTimeoutMs = 60000; + +// Upper bound on how long a read/write entry point waits to acquire the +// shared store connection (waitForKnowledgeBuilder). Shorter than the join +// timeout so a read that arrives while a wedged builder holds the lock +// returns a queryable error envelope well within the MCP call timeout, +// instead of blocking the server forever. Named (no magic number). +inline constexpr int kStoreLockTimeoutMs = 30000; + +// How long the shared store lock may be held CONTINUOUSLY (measured from the +// first failed acquisition) before a read declares the builder permanently +// wedged and issues a single sqlite3_interrupt() to unblock a stuck SQLite +// call. Deliberately generous: normal builds finish in ~10 s, and a +// legitimately slow build must not be aborted, so this only fires for a +// genuine permanent stall. After the interrupt the builder's current +// statement returns SQLITE_INTERRUPT, it unwinds, and its lock_guard releases +// the mutex — self-healing without killing a thread. +inline constexpr int kBuilderStallInterruptMs = 300000; + +// Start of a continuous stall on the shared store lock (0 = not stalled). +// Set on the first failed acquisition and cleared whenever one succeeds, so +// it measures true continuous occupancy, not cumulative wait time. +// Connection the running builder is working on, recorded by +// launchAsyncKnowledgeBuilder so waitForKnowledgeBuilder() can +// interrupt *that* connection without reaching for global engine state. +std::atomic g_builder_db{ nullptr }; + +std::atomic g_store_stall_start_ms{ 0 }; +std::atomic g_store_interrupt_issued{ false }; + +// Monotonic milliseconds for stall measurement (unaffected by wall-clock +// changes). +int64_t steadyNowMs() +{ + return std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()) + .count(); +} + // Maximum number of module_edge rows to insert. Prevents unbounded // growth on projects with many small modules (one per file). inline constexpr int64_t kMaxModuleEdges = 5000; @@ -124,6 +189,10 @@ int64_t buildKnowledgeGraphSync(store::GraphStore &store, uint64_t project_id) "[module=async, method=buildKnowledgeGraphSync] " "COMMIT failed: %s\n", store.error().c_str()); + // Roll back so the connection is not left inside a transaction + // (that would make every later BEGIN fail until the process + // restarts). + store.exec("ROLLBACK"); return -1; } @@ -405,7 +474,17 @@ void runModelIndexSync(store::GraphStore &store, uint64_t project_id, auto t_fts_start = Clock::now(); try { store.buildFTSFromGraph(project_id); - store.setProjectReadiness(project_id, "fts_ready", 1); + // exec() does not throw; only mark fts_ready when the + // store reports no error (code_rules: no silent success). + if (store.error().empty()) { + store.setProjectReadiness(project_id, + "fts_ready", 1); + } else { + fprintf(stderr, + "[module=async, method=runModelIndexSync] " + "FTS build failed: %s\n", + store.error().c_str()); + } } catch (const std::exception &e) { fprintf(stderr, "[module=async, method=runModelIndexSync] " @@ -436,7 +515,8 @@ void runModelIndexSync(store::GraphStore &store, uint64_t project_id, (long long)ms(t0, Clock::now())); } -void launchAsyncKnowledgeBuilder(uint64_t project_id, bool run_fts) +void launchAsyncKnowledgeBuilder(EngineContext *ctx, uint64_t project_id, + bool run_fts) { std::lock_guard lock(g_thread_mutex); @@ -459,11 +539,25 @@ void launchAsyncKnowledgeBuilder(uint64_t project_id, bool run_fts) return; } - g_builder_thread = std::thread([project_id, run_fts]() { - if (!g_store) { + { + std::lock_guard done_lock(g_done_mutex); + g_builder_done = false; + } + // `ctx` (the caller's engine instance) is captured by value: the + // handle outlives the thread because engine_destroy() joins the + // builder before it releases the instance. + g_builder_thread = std::thread([project_id, run_fts, ctx]() { + // Hold the connection mutex for the whole body so no FFI read + // or write can interleave BEGIN/COMMIT on the engine store. + std::lock_guard store_lock( + g_store_mutex); + if (!ctx || !ctx->store) { fprintf(stderr, - "[module=async] g_store is null, aborting\n"); + "[module=async] engine store is null, aborting\n"); g_async_running.store(false); + std::lock_guard done_lock(g_done_mutex); + g_builder_done = true; + g_done_cv.notify_all(); return; } fprintf(stderr, @@ -475,8 +569,8 @@ void launchAsyncKnowledgeBuilder(uint64_t project_id, bool run_fts) // background thread BEFORE the knowledge builder, so the // synchronous index path returns as soon as the core graph // + indexes are ready. - runModelIndexSync(*g_store, project_id, run_fts); - buildKnowledgeGraphSync(*g_store, project_id); + runModelIndexSync(*ctx->store, project_id, run_fts); + buildKnowledgeGraphSync(*ctx->store, project_id); } catch (const std::exception &e) { fprintf(stderr, "[module=async] build failed with exception: %s\n", @@ -486,6 +580,11 @@ void launchAsyncKnowledgeBuilder(uint64_t project_id, bool run_fts) "[module=async] build failed with unknown exception\n"); } g_async_running.store(false); + { + std::lock_guard done_lock(g_done_mutex); + g_builder_done = true; + } + g_done_cv.notify_all(); fprintf(stderr, "[module=async] knowledge graph build complete " "for project %llu\n", @@ -495,12 +594,108 @@ void launchAsyncKnowledgeBuilder(uint64_t project_id, bool run_fts) void joinAsyncKnowledgeBuilder() { + // Hold g_thread_mutex for the whole wait+join so launch cannot spawn a + // second builder while this join is still reaping the first (moving the + // thread out and joining outside the lock re-opened that race). std::lock_guard lock(g_thread_mutex); - if (g_builder_thread.joinable()) - g_builder_thread.join(); + if (!g_builder_thread.joinable()) + return; + // Bounded wait before the blocking join. On timeout the thread is + // DETACHED, not joined: the caller may already hold g_store_mutex (the + // write entry points join then lock), and a still-running builder is + // blocked on that same mutex — joining it here would deadlock. The + // detached thread finishes on its own once the caller releases the + // mutex; the log makes the stall traceable. + { + std::unique_lock done_lock(g_done_mutex); + if (!g_done_cv.wait_for( + done_lock, + std::chrono::milliseconds(kBuilderJoinTimeoutMs), + [] { return g_builder_done; })) { + fprintf(stderr, + "[module=async, method=joinAsyncKnowledgeBuilder] " + "builder still running after %d ms — detaching " + "(shared connection stays locked until it " + "finishes)\n", + kBuilderJoinTimeoutMs); + g_builder_thread.detach(); + return; + } + } + g_builder_thread.join(); } bool isAsyncKnowledgeBuilderRunning() { return g_async_running.load(); } + +std::unique_lock waitForKnowledgeBuilder() +{ + // Acquire the connection mutex the builder holds for its whole body. + // This both waits for an in-flight builder AND keeps a later builder + // (launched while this guard is alive) from touching the engine store until the + // caller's read/write finishes — the race a plain join left open. + // recursive_timed_mutex: FFI entry points nest (enhance reaches paths + // that also call this), so the same thread must be able to re-acquire. + // Safe to call from any read entry point: the builder never re-enters + // the read entry points, so this cannot self-deadlock. + std::unique_lock lock(g_store_mutex, + std::defer_lock); + if (lock.try_lock_for(std::chrono::milliseconds(kStoreLockTimeoutMs))) { + // Healthy: clear any stall bookkeeping. + g_store_stall_start_ms.store(0, std::memory_order_relaxed); + g_store_interrupt_issued.store(false, + std::memory_order_relaxed); + return lock; + } + + // The lock has been held past every measured build. Record when the + // stall began; once it is long enough to be permanent, interrupt the + // builder's in-flight SQLite statement so it errors out, unwinds, and + // releases the lock (self-healing). sqlite3_interrupt is documented as + // safe to call from another thread, and only one read runs at a time on + // the single-threaded dispatch loop. + const int64_t now = steadyNowMs(); + int64_t stall_start = + g_store_stall_start_ms.load(std::memory_order_relaxed); + if (stall_start == 0) { + stall_start = now; + g_store_stall_start_ms.store(stall_start, + std::memory_order_relaxed); + } + if (!g_store_interrupt_issued.load(std::memory_order_relaxed) && + now - stall_start >= kBuilderStallInterruptMs) { + g_store_interrupt_issued.store(true, std::memory_order_relaxed); + fprintf(stderr, + "[module=async, method=waitForKnowledgeBuilder] " + "shared store lock held > %d ms — issuing " + "sqlite3_interrupt() to unwedge the builder\n", + kBuilderStallInterruptMs); + if (sqlite3 *builder_db = g_builder_db.load()) + sqlite3_interrupt(builder_db); + // Give the unwinding builder a bounded grace period to release the + // lock; if it does, this read proceeds normally. + if (lock.try_lock_for( + std::chrono::milliseconds(kStoreLockTimeoutMs))) { + g_store_stall_start_ms.store(0, + std::memory_order_relaxed); + g_store_interrupt_issued.store( + false, std::memory_order_relaxed); + return lock; + } + } + + // Still stuck. Proceed-anyway would corrupt the shared connection + // (frames interleaving BEGIN/COMMIT), and blocking forever would hang + // the whole MCP server. Throw instead — every call site sits behind an + // extern "C" try/catch wrapper that turns this into an error envelope. + fprintf(stderr, + "[module=async, method=waitForKnowledgeBuilder] " + "shared store lock still held after %d ms " + "(knowledge builder wedged?)\n", + kStoreLockTimeoutMs); + throw std::runtime_error( + "knowledge builder holds the shared store lock; " + "store temporarily unavailable"); +} diff --git a/engine/src/async_knowledge.h b/engine/src/async_knowledge.h index 74036a5..763ccaa 100644 --- a/engine/src/async_knowledge.h +++ b/engine/src/async_knowledge.h @@ -1,8 +1,12 @@ +// The async builder takes the engine instance explicitly (TD-1 knife 3). +#include "engine_context.h" + #ifndef CODESCOPE_ASYNC_KNOWLEDGE_H #define CODESCOPE_ASYNC_KNOWLEDGE_H #include #include +#include #include namespace store @@ -28,8 +32,8 @@ class GraphStore; // // THREAD SAFETY: the builder uses a global atomic flag to ensure only // one instance runs at a time. The background thread is joinable (not -// detached) so that engine_shutdown can wait for it to finish before -// destroying g_store, preventing use-after-free. Callers must call +// detached) so that engine_destroy can wait for it to finish before +// destroying the engine store, preventing use-after-free. Callers must call // joinAsyncKnowledgeBuilder() before destroying the GraphStore singleton. /// Launch the async post-index builder for a project. @@ -37,16 +41,65 @@ class GraphStore; /// joinable background thread. Returns immediately. /// @param project_id The project to enrich. /// @param run_fts Whether to build the FTS index (skipped in fast mode). -void launchAsyncKnowledgeBuilder(uint64_t project_id, bool run_fts = true); +void launchAsyncKnowledgeBuilder(EngineContext *ctx, uint64_t project_id, + bool run_fts = true); /// Wait for the async knowledge builder to finish (if running). -/// Must be called before destroying g_store to prevent use-after-free. +/// Must be called before destroying the engine store to prevent use-after-free. +/// The wait is bounded by kBuilderJoinTimeoutMs; on timeout the builder +/// thread is detached (not joined) and the function returns, so a wedged +/// builder cannot deadlock a caller that already holds the connection +/// lock. The log makes the cause traceable. void joinAsyncKnowledgeBuilder(); /// Check whether the async knowledge builder is currently running. /// @return true if the builder thread is active. bool isAsyncKnowledgeBuilderRunning(); +/// Wait for the background knowledge builder before a READ touches the shared +/// store, and hold the shared-connection lock until the returned guard is +/// destroyed. +/// +/// The builder runs on the same sqlite3 connection as the caller and opens its +/// own transactions (module_summary / module_edge / FTS / model tables). A +/// plain join only waits for the *current* builder: a later index call can +/// launch a new writer while a read is still using the instance's store, and the two +/// interleave BEGIN/COMMIT on the one shared connection. The returned guard +/// is a `std::unique_lock` on the connection mutex the builder also holds for +/// its whole body, so no builder SQL can run while the guard is alive. +/// +/// Call sites MUST bind the guard for the duration of the store access: +/// `auto _store_guard = waitForKnowledgeBuilder();`. Dropping it immediately +/// (calling the function as a statement) re-opens the race. +/// +/// BOUNDED WAIT: the acquisition is bounded by kStoreLockTimeoutMs (30 s). If +/// the builder thread wedges while holding the lock (e.g. a SQLite backoff +/// loop), the caller must NOT proceed to touch the instance's store concurrently, so a +/// `std::runtime_error` is thrown instead. The FFI wrappers already catch +/// `std::exception` and return an `[module=..., method=...]` error envelope, +/// so a wedged builder degrades to "store unavailable" rather than hanging +/// the MCP server forever. +/// +/// SELF-HEALING: if the lock stays continuously held past +/// kBuilderStallInterruptMs (5 min), the waiter issues a single +/// `sqlite3_interrupt()` on the shared connection. That unblocks a stuck +/// SQLite call in the builder, the builder unwinds, and its destructor +/// releases the lock, so the store recovers without killing a thread. The +/// threshold is far above any measured build, so it cannot abort a +/// legitimately slow one. A stall in non-SQLite code cannot be interrupted; +/// recovery is best-effort and logged. +/// +/// Safe to call from any read entry point: the builder thread calls +/// runModelIndexSync()/buildKnowledgeGraphSync() directly and never re-enters +/// the read entry points, so this can never self-deadlock. The mutex is +/// recursive so nested FFI entry points on one thread can re-acquire it. +/// Never call joinAsyncKnowledgeBuilder while holding the guard across a +/// running builder (the builder thread needs the same mutex to finish). +/// +/// @throws std::runtime_error if the shared store lock cannot be acquired +/// within kStoreLockTimeoutMs. +std::unique_lock waitForKnowledgeBuilder(); + /// Synchronous entry point: build the module_edge table and set the /// knowledge_ready flag. Called by the background thread, but can also /// be called directly for testing or synchronous enrichment. diff --git a/engine/src/codescope_grammars.h b/engine/src/codescope_grammars.h index 9f9d5a8..5a6e0b3 100644 --- a/engine/src/codescope_grammars.h +++ b/engine/src/codescope_grammars.h @@ -16,7 +16,6 @@ const TSLanguage *tree_sitter_java(); const TSLanguage *tree_sitter_javascript(); const TSLanguage *tree_sitter_python(); const TSLanguage *tree_sitter_rust(); -const TSLanguage *tree_sitter_swift(); const TSLanguage *tree_sitter_typescript(); const TSLanguage *tree_sitter_tsx(); diff --git a/engine/src/engine.cpp b/engine/src/engine.cpp index 208128d..68c3069 100644 --- a/engine/src/engine.cpp +++ b/engine/src/engine.cpp @@ -28,21 +28,19 @@ // ─── Sqlite-vec extension loading (dlopen-based, portable) ── -// ─── Global singletons ───────────────────────────────────────── -// NOT static — declared extern in engine_internal.h so that -// other engine_*.cpp translation units can access them. -// Using unique_ptr for exception-safe memory management. - -std::unique_ptr g_store; -std::unique_ptr g_query; -std::unique_ptr g_parser; +// ─── Engine state ────────────────────────────────────────────── +// The store / query / parser state lives in the EngineContext declared in +// engine_context.h and created by engine_create(), so no state is defined +// in this file any more. See engine_context.h for the knife-1/2/3 migration +// notes (TD-1). // Core engine functions are split into separate translation units: // engine_helpers.cpp — readFile, jsonEscape, detectLanguage, dupString, etc. -// engine_lifecycle.cpp — engine_init, engine_shutdown, engine_create_project +// engine_lifecycle.cpp — engine_create, engine_destroy, engine_create_project // engine_index.cpp — engine_index_file, engine_index_project, engine_index_batch // engine_scanner.cpp — fast scanner + engine_scan_project // engine_queries.cpp — enhancement, search, callers/callees, trace, context // engine_ffi.cpp — find_definition, get_callers, search_code, complexity, DSL // -// engine.cpp retains only the preamble, global singletons, and sqlite-vec init. +// engine.cpp retains only the includes and these layout notes; the engine +// state lives in engine_context.cpp (see engine_context.h). diff --git a/engine/src/engine_context.cpp b/engine/src/engine_context.cpp new file mode 100644 index 0000000..035da95 --- /dev/null +++ b/engine/src/engine_context.cpp @@ -0,0 +1,20 @@ +#include "engine_context.h" +#include "parser/parser.h" +#include "query/query_engine.h" +#include "store/store.h" + +// Completes the two out-of-line special members declared in engine_context.h. +// Instantiating the unique_ptr constructors/deleters here is exactly why the +// header can keep its member types forward-declared. +// +// This TU holds no state: after TD-1 knife 3 the instance is heap-allocated by +// engine_create() and released by engine_destroy() (engine_lifecycle.cpp), so +// there is neither a namespace-scope object nor a function-local static left. + +CodescopeEngine::CodescopeEngine() = default; + +/// Destructor. Members are destroyed in reverse declaration order +/// (parser → query → store), which is the order engine_destroy() establishes +/// explicitly: the query engine may issue SQLite work, so it must go before the +/// store closes. +CodescopeEngine::~CodescopeEngine() = default; diff --git a/engine/src/engine_context.h b/engine/src/engine_context.h new file mode 100644 index 0000000..4eedd9f --- /dev/null +++ b/engine/src/engine_context.h @@ -0,0 +1,112 @@ +#ifndef ENGINE_CONTEXT_H +#define ENGINE_CONTEXT_H + +// Engine instance state — TD-1 (docs/REVIEW_0.2.7.md), knife 3. +// +// History, so the shape below is not mistaken for an accident: +// +// * Before TD-1 the engine kept three process-global `unique_ptr` +// singletons (`g_store`, `g_query`, `g_parser`) declared in +// engine_internal.h and defined in engine.cpp. +// * Knife 1 moved them into one `EngineContext` object and kept the old +// names as reference aliases, so no call site had to change. +// * Knife 2 migrated all 340 call sites to `engineContext()` and turned the +// object into a function-local static: no global object was constructed +// before `main()` any more, but the instance was still reached for +// implicitly. +// * Knife 3 (this revision) deletes that accessor. The instance is created +// by `engine_create()` (engine.h) and reaches engine code only as the +// opaque `engine_t` handle every FFI entry point now takes as its first +// parameter — the state is passed in, never looked up (code_rules §2: +// "Avoid global/static mutable state unless strictly necessary"). +// +// What is per-instance after knife 3: `store`, `query`, `parser` and their +// lifetime, which is exactly the state the FFI previously reached for. +// +// What is still process-wide, and therefore still a limit on running two +// *independent* engines in one process (recorded in the FFI contract in +// engine.h and in the CHANGELOG rather than left implicit): +// +// * `verify::VerifierRegistry::instance()` — a process-wide registry bound +// to (store, project_id) at construction. Pre-existing and already +// documented as a v0.3 limitation inside engine_lifecycle.cpp. +// * the async knowledge builder's thread/flags (async_knowledge.cpp) and the +// shared-store lock it holds; one builder runs per process. +// * `store::IndexProgress` and the parse-failure buffer in the store layer. +// +// The header deliberately FORWARD-DECLARES the member types and defines both +// special members out-of-line (engine_context.cpp), so a translation unit that +// only passes a handle through — e.g. store/store_parse_failure.cpp — can +// include it without pulling parser.h / query_engine.h / store.h into the +// build. + +#include + +// The `engine_t` handle type (engine.h). Including the public C header here is +// cheap (it pulls in stdint.h only) and keeps engineInstance() below typed +// exactly like the ABI. +#include "engine.h" + +namespace store +{ +class GraphStore; +} +namespace query +{ +class QueryEngine; +} +class Parser; + +/// One engine instance. +/// +/// The C ABI exposes this type opaquely as `engine_t` +/// (`typedef struct CodescopeEngine *engine_t;` in engine.h): callers get a +/// handle from engine_create(), pass it to every stateful FFI function, and +/// release it with engine_destroy(). Engine code spells the same type +/// `EngineContext` through the alias below. +struct CodescopeEngine { + /// Open graph store (SQLite). Non-null for the whole life of a + /// successfully created instance; closed by engine_destroy(). + std::unique_ptr store; + /// Query engine bound to `store`. + std::unique_ptr query; + /// Parser with the statically linked grammars registered. + std::unique_ptr parser; + + /// Both special members are declared here and defined in + /// engine_context.cpp. That is what lets this header keep its member types + /// forward-declared: engine_create()/engine_destroy() construct and destroy + /// the object in a translation unit that sees the concrete types, and a + /// translation unit that only forwards a handle never instantiates the + /// `unique_ptr` deleters (clang otherwise fails the deleter's + /// incomplete-type static_assert). + CodescopeEngine(); + ~CodescopeEngine(); +}; + +/// Internal spelling of the instance type. Every engine_*.cpp file uses this +/// name; the ABI sees the same type as `engine_t`. +using EngineContext = CodescopeEngine; + +/// Resolve an ABI engine handle to the instance it names. +/// +/// A null handle means the caller never created an instance, or already +/// destroyed it. Every FFI entry point funnels that case through its existing +/// "engine not initialized" guard (`if (!ctx || !ctx->store) ...`), so the +/// null contract is stated once here instead of being re-derived at 70 sites. +/// +/// @param handle Handle returned by engine_create(); may be null. +/// @return The instance named by `handle`, or null when `handle` is null. +/// +/// Ownership: borrowed — the instance is owned by the caller of +/// engine_create() and stays valid until engine_destroy(handle). +/// Lifetime: the returned pointer must not outlive engine_destroy(handle). +/// Thread safety: none is taken here. The caller obeys the instance's +/// single-connection contract; see the thread-safety contract in +/// engine_internal.h. +inline EngineContext *engineInstance(engine_t handle) +{ + return handle; +} + +#endif // ENGINE_CONTEXT_H diff --git a/engine/src/engine_evidence_ffi.cpp b/engine/src/engine_evidence_ffi.cpp index d12badc..cc525bd 100644 --- a/engine/src/engine_evidence_ffi.cpp +++ b/engine/src/engine_evidence_ffi.cpp @@ -6,12 +6,12 @@ // char *engine_build_evidence(uint64_t project_id, // const char *category_filter); // -// The function loads rule files from the directory pointed to by -// CODESCOPE_RULES_DIR (falling back to "engine/src/evidence/rules" -// relative to CWD), runs all rules (or one category's rules when -// `category_filter` is non-empty), and returns a JSON array of -// Evidence objects. The caller MUST release the returned pointer via -// engine_free_string(). +// The function loads rule files from the directory returned by +// evidence::resolveRulesDir() ($CODESCOPE_RULES_DIR, then the build- +// time default, then the in-tree relative path), runs all rules (or +// one category's rules when `category_filter` is non-empty), and +// returns a JSON array of Evidence objects. The caller MUST release +// the returned pointer via engine_free_string(). // // Output shape (JSON array of objects): // [ @@ -29,17 +29,18 @@ // ] // // All errors return a JSON object with an "error" field instead of -// crashing. Null `g_store` returns +// crashing. A null handle (or a store-less instance) returns // {"error":"engine not initialized"}. #include "engine_internal.h" +#include "async_knowledge.h" #include "evidence/evidence_builder.h" #include "platform_win.h" +#include "util/json_writer.h" #include #include #include -#include #include #include @@ -48,75 +49,34 @@ namespace { -// JSON-escape a string for inclusion in a JSON string literal. -// Mirrors the jsonEscape helper in engine_internal.h but kept -// local to avoid pulling that header's full set of includes. -std::string escapeJson(const std::string &s) +// Serialize one EvidenceItem into the object currently open on `w`. +void writeItem(util::JsonWriter &w, const evidence::EvidenceItem &item) { - std::string out; - out.reserve(s.size() + 8); - for (char c : s) { - switch (c) { - case '"': - out += "\\\""; - break; - case '\\': - out += "\\\\"; - break; - case '\n': - out += "\\n"; - break; - case '\r': - out += "\\r"; - break; - case '\t': - out += "\\t"; - break; - default: - if (static_cast(c) < 0x20) { - char buf[8]; - std::snprintf(buf, sizeof(buf), "\\u%04x", - static_cast(c)); - out += buf; - } else { - out += c; - } - } - } - return out; + w.beginObject(); + w.key("fact_id").value(item.fact_id); + w.key("category").value(item.category); + w.key("primitive").value(item.primitive); + w.key("kind").value(item.kind); + w.key("symbol").value(item.symbol); + w.key("file").value(item.file); + w.key("line").value(item.line); + w.key("snippet").value(item.snippet); + w.endObject(); } -// Serialize one EvidenceItem to a JSON object string. -std::string serializeItem(const evidence::EvidenceItem &item) +// Serialize one Evidence as an object; `items` is always emitted as an +// array (empty for the Count combine mode). +void writeEvidence(util::JsonWriter &w, const evidence::Evidence &ev) { - std::ostringstream ss; - ss << "{\"fact_id\":" << item.fact_id << ",\"category\":\"" - << escapeJson(item.category) << "\"" - << ",\"primitive\":\"" << escapeJson(item.primitive) << "\"" - << ",\"kind\":\"" << escapeJson(item.kind) << "\"" - << ",\"symbol\":\"" << escapeJson(item.symbol) << "\"" - << ",\"file\":\"" << escapeJson(item.file) << "\"" - << ",\"line\":" << item.line << ",\"snippet\":\"" - << escapeJson(item.snippet) << "\"" - << "}"; - return ss.str(); -} - -// Serialize one Evidence to a JSON object string. `items` is always -// emitted as an array (empty for Count combine). -std::string serializeEvidence(const evidence::Evidence &ev) -{ - std::ostringstream ss; - ss << "{\"category\":\"" << escapeJson(ev.category) << "\"" - << ",\"title\":\"" << escapeJson(ev.title) << "\"" - << ",\"confidence\":" << ev.confidence << ",\"items\":["; - for (size_t i = 0; i < ev.items.size(); ++i) { - if (i) - ss << ","; - ss << serializeItem(ev.items[i]); - } - ss << "]}"; - return ss.str(); + w.beginObject(); + w.key("category").value(ev.category); + w.key("title").value(ev.title); + w.key("confidence").value(ev.confidence); + w.key("items").beginArray(); + for (const auto &item : ev.items) + writeItem(w, item); + w.endArray(); + w.endObject(); } } // namespace @@ -125,35 +85,46 @@ std::string serializeEvidence(const evidence::Evidence &ev) // Returns JSON array of evidence for a project. Optionally filter by // category. Caller must free the returned string via -// engine_free_string. Path: rules_dir defaults to -// "engine/src/evidence/rules" relative to CWD, or override via -// CODESCOPE_RULES_DIR env var. -char *engine_build_evidence(uint64_t project_id, const char *category_filter) +// engine_free_string. Rule files are located by +// evidence::resolveRulesDir() so the result does not depend on the +// process working directory. +char *engine_build_evidence(engine_t handle, uint64_t project_id, + const char *category_filter) { - if (!g_store) - return dupString("{\"error\":\"engine not initialized\"}"); + EngineContext *ctx = engineInstance(handle); - const char *env_dir = std::getenv("CODESCOPE_RULES_DIR"); - std::string rules_dir = - (env_dir && *env_dir) ? env_dir : "engine/src/evidence/rules"; + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) + return dupString( + "{\"error\":\"engine not initialized\"}"); - evidence::EvidenceBuilder builder(g_store.get()); - builder.loadRules(rules_dir); + std::string rules_dir = evidence::resolveRulesDir(); + if (rules_dir.empty()) + return dupString("[]"); - std::vector evidences; - if (category_filter && *category_filter) { - evidences = - builder.buildByCategory(project_id, category_filter); - } else { - evidences = builder.buildAll(project_id); - } + evidence::EvidenceBuilder builder(ctx->store.get()); + builder.loadRules(rules_dir); + + std::vector evidences; + if (category_filter && *category_filter) { + evidences = builder.buildByCategory(project_id, + category_filter); + } else { + evidences = builder.buildAll(project_id); + } - std::string json = "["; - for (size_t i = 0; i < evidences.size(); ++i) { - if (i) - json += ","; - json += serializeEvidence(evidences[i]); + util::JsonWriter w; + w.beginArray(); + for (const auto &ev : evidences) + writeEvidence(w, ev); + w.endArray(); + return dupString(w.str()); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_build_evidence", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_build_evidence", "unknown exception")); } - json += "]"; - return dupString(json); } diff --git a/engine/src/engine_ffi.cpp b/engine/src/engine_ffi.cpp index 240122d..2f338b3 100644 --- a/engine/src/engine_ffi.cpp +++ b/engine/src/engine_ffi.cpp @@ -1,11 +1,11 @@ #include "engine_internal.h" +#include "async_knowledge.h" #include "platform_win.h" +#include "util/json_writer.h" #include #include -#include #include -#include #include #include "verify/dead_code_inspector.h" @@ -41,13 +41,13 @@ // every capability coverage ratio. Reads the canonical `entity` table — never // the deprecated `graph_nodes`/`symbols` tables — so the count reflects real // indexed data. -static int ffi_count_eligible_entities(uint64_t project_id) +static int ffi_count_eligible_entities(EngineContext *ctx, uint64_t project_id) { sqlite3_stmt *stmt = nullptr; int total = 0; const char *sql = "SELECT COUNT(*) FROM entity WHERE project_id = ? AND kind IN (0,1)"; - if (sqlite3_prepare_v2(g_store->handle(), sql, -1, &stmt, nullptr) == + if (sqlite3_prepare_v2(ctx->store->handle(), sql, -1, &stmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64(stmt, 1, static_cast(project_id)); if (sqlite3_step(stmt) == SQLITE_ROW) @@ -57,7 +57,7 @@ static int ffi_count_eligible_entities(uint64_t project_id) fprintf(stderr, "engine_get_capabilities: entity count probe failed: %s " "[module=ffi, method=engine_get_capabilities]\n", - sqlite3_errmsg(g_store->handle())); + sqlite3_errmsg(ctx->store->handle())); } return total; } @@ -66,7 +66,8 @@ static int ffi_count_eligible_entities(uint64_t project_id) // one Calls relation (relation.type=1) as source or target. This is the // canonical signal that the call graph has been built for them. Returns 0 on // any error. Used to compute the call_graph coverage ratio. -static int ffi_count_callgraph_ready_entities(uint64_t project_id) +static int ffi_count_callgraph_ready_entities(EngineContext *ctx, + uint64_t project_id) { sqlite3_stmt *stmt = nullptr; int ready = 0; @@ -80,7 +81,7 @@ static int ffi_count_callgraph_ready_entities(uint64_t project_id) " SELECT target_id AS src FROM relation WHERE project_id=? AND type=1" " )" ")"; - if (sqlite3_prepare_v2(g_store->handle(), sql, -1, &stmt, nullptr) == + if (sqlite3_prepare_v2(ctx->store->handle(), sql, -1, &stmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); @@ -91,7 +92,7 @@ static int ffi_count_callgraph_ready_entities(uint64_t project_id) fprintf(stderr, "engine_get_capabilities: callgraph count probe failed: %s " "[module=ffi, method=engine_get_capabilities]\n", - sqlite3_errmsg(g_store->handle())); + sqlite3_errmsg(ctx->store->handle())); } return ready; } @@ -99,14 +100,15 @@ static int ffi_count_callgraph_ready_entities(uint64_t project_id) // Count function/method entities whose code metrics were resolved onto the // canonical entity rows (cyclomatic > 0). This is the metrics_ready signal — // it reflects real producer output from resolveStagedMetrics, never a flag. -static int ffi_count_metrics_ready_entities(uint64_t project_id) +static int ffi_count_metrics_ready_entities(EngineContext *ctx, + uint64_t project_id) { sqlite3_stmt *stmt = nullptr; int ready = 0; const char *sql = "SELECT COUNT(*) FROM entity " "WHERE project_id = ? AND kind IN (0,1) AND cyclomatic > 0"; - if (sqlite3_prepare_v2(g_store->handle(), sql, -1, &stmt, nullptr) == + if (sqlite3_prepare_v2(ctx->store->handle(), sql, -1, &stmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64(stmt, 1, static_cast(project_id)); if (sqlite3_step(stmt) == SQLITE_ROW) @@ -116,7 +118,7 @@ static int ffi_count_metrics_ready_entities(uint64_t project_id) fprintf(stderr, "engine_get_capabilities: metrics count probe failed: %s " "[module=ffi, method=engine_get_capabilities]\n", - sqlite3_errmsg(g_store->handle())); + sqlite3_errmsg(ctx->store->handle())); } return ready; } @@ -124,13 +126,13 @@ static int ffi_count_metrics_ready_entities(uint64_t project_id) // Count node_vectors rows for the project — the canonical embedding_ready // signal for semantic search. 0 when the builder has not run or wrote nothing // (avoids the A19 "fake ready" regression). -static int ffi_count_vector_entities(uint64_t project_id) +static int ffi_count_vector_entities(EngineContext *ctx, uint64_t project_id) { sqlite3_stmt *stmt = nullptr; int ready = 0; const char *sql = "SELECT COUNT(*) FROM node_vectors WHERE project_id = ?"; - if (sqlite3_prepare_v2(g_store->handle(), sql, -1, &stmt, nullptr) == + if (sqlite3_prepare_v2(ctx->store->handle(), sql, -1, &stmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64(stmt, 1, static_cast(project_id)); if (sqlite3_step(stmt) == SQLITE_ROW) @@ -140,15 +142,18 @@ static int ffi_count_vector_entities(uint64_t project_id) fprintf(stderr, "engine_get_capabilities: vector count probe failed: %s " "[module=ffi, method=engine_get_capabilities]\n", - sqlite3_errmsg(g_store->handle())); + sqlite3_errmsg(ctx->store->handle())); } return ready; } -char *engine_get_capabilities(uint64_t project_id) +char *engine_get_capabilities(engine_t handle, uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString( "{\"error\":\"engine not initialized\"}"); @@ -159,537 +164,143 @@ char *engine_get_capabilities(uint64_t project_id) // whether the producer has actually populated data for this // project, so clients can distinguish "built" from "not yet // indexed in DEEP mode". FTS stays available (already wired). - const int total = ffi_count_eligible_entities(project_id); + const int total = ffi_count_eligible_entities(ctx, project_id); const int cg_ready_count = - ffi_count_callgraph_ready_entities(project_id); + ffi_count_callgraph_ready_entities(ctx, project_id); const bool cg_ready = cg_ready_count > 0; - - std::ostringstream json; - json << "{" - << "\"project_id\":" << project_id << "," - << "\"total_symbols\":" << total << "," - << "\"capabilities\":{" - << "\"fast_scan\":{\"available\":true,\"ready\":true,\"description\":\"ms-level declaration extraction\"}," - << "\"module_tree\":{\"available\":true,\"ready\":true,\"description\":\"hierarchical module view\"}," - << "\"symbol_search\":{\"available\":true,\"ready\":true,\"description\":\"exact name match\"}," - << "\"entry_points\":{\"available\":true,\"ready\":" - << (total > 0 ? "true" : "false") - << ",\"description\":\"main/initcall/probe detection\"}," - << "\"call_graph\":{\"available\":true,\"ready\":" - << (cg_ready ? "true" : "false") - << ",\"coverage\":{\"eligible\":" << total - << ",\"ready\":" << cg_ready_count << "}" - << ",\"description\":\"function call edges (built during index)\"}," - << "\"path_tracing\":{\"available\":true,\"ready\":" - << (cg_ready ? "true" : "false") - << ",\"description\":\"BFS shortest path between functions\"}," - // v0.2.5: metrics + semantic search restored. Metrics are - // produced by computeMetricsFromCST in the parse worker and - // resolved onto entity by resolveStagedMetrics; semantic - // search is an n-gram hash vector (buildVectorsFromGraph). - // `ready` reflects canonical data (entity cyclomatic > 0 / - // node_vectors rows), never a hardcoded flag. - << "\"metrics\":{\"available\":true,\"ready\":" - << (ffi_count_metrics_ready_entities(project_id) > 0 ? - "true" : - "false") - << ",\"description\":\"cyclomatic/cognitive/nesting complexity (computed during index, resolved onto entity)\"}," - << "\"semantic_search\":{\"available\":true,\"ready\":" - << (ffi_count_vector_entities(project_id) > 0 ? "true" : - "false") - << ",\"mode\":\"ngram_hash\"," - << "\"description\":\"n-gram hash vector lexical similarity (restored in v0.2.5); complements FTS exact search\"}," - // FTS remains the exact-match workhorse; semantic search - // is additive (never replaces it). - << "\"fts\":{\"available\":true,\"ready\":" - << (g_store->getProjectReadiness(project_id, "fts_ready") ? - "true" : - "false") - << ",\"description\":\"FTS5 full-text search for exact/prefix matching\"}," - << "\"context_builder\":{\"available\":true,\"ready\":true,\"description\":\"intelligent context assembly\"}" - << "}," - << "\"enhancement_needed\":\"Call graph, complexity metrics, and n-gram semantic vectors are built during index; FTS powers exact search.\"" - << "}"; - return dupString(json.str()); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_capabilities] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_capabilities] unknown exception\"}"); - } -} - -// ─── Knowledge Graph direct query (v0.2.1) ───────────────────────────── -// -// Surfaces the knowledge-layer tables (entity / relation / architecture_edge / -// module_edge / capability / document / module_summary) so MCP clients can -// browse the knowledge graph directly, instead of only benefiting from it -// indirectly via explain_module / detect_capability_drift / get_module_tree. -// -// Per plan/rules/code_rules.md §FFI: this is a block-level transfer — one -// FFI call returns the entire result set (bounded by `limit`), never one -// row per call. Error paths emit a stderr line tagged with module=ffi, -// method=engine_get_knowledge_graph per §"Additional Rules". -char *engine_get_knowledge_graph(uint64_t project_id, const char *table_name, - int32_t limit) -{ - try { - if (!g_store) - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_knowledge_graph] engine not initialized\"}"); - if (!table_name || !*table_name) - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_knowledge_graph] table_name is required\"}"); - - // Whitelist of knowledge-layer tables. We never let the caller pass - // arbitrary SQL — the table_name is matched against this fixed set - // and the SELECT is built with a hard-coded column list per table. - // This prevents SQL injection via the table_name parameter. - struct TableSpec { - const char *name; - const char * - select; // hard-coded column list, no user input - }; - static const TableSpec kTables[] = { - { "entity", - "SELECT id, name, qualified_name, kind, " - "file_path, start_row, start_col FROM entity " - "WHERE project_id=? ORDER BY id LIMIT ?" }, - { "relation", "SELECT id, source_id, target_id, type " - "FROM relation WHERE project_id=? " - "ORDER BY id LIMIT ?" }, - { "architecture_edge", - "SELECT id, layer_lower, layer_upper, " - "entity_id FROM architecture_edge " - "WHERE project_id=? ORDER BY id LIMIT ?" }, - { "module_edge", - "SELECT id, src_module, tgt_module, " - "edge_count FROM module_edge " - "WHERE project_id=? ORDER BY id LIMIT ?" }, - { "capability", - "SELECT id, name, summary FROM capability " - "WHERE project_id=? ORDER BY id LIMIT ?" }, - { "document", - "SELECT id, type, file_path, start_line, end_line " - "FROM document " - "WHERE project_id=? ORDER BY id LIMIT ?" }, - { "module_summary", - "SELECT id, module_id, state, incoming_count, " - "outgoing_count, internal_edges, " - "dead_entities, utilization, confidence " - "FROM module_summary " - "WHERE project_id=? ORDER BY id LIMIT ?" }, - }; - const TableSpec *spec = nullptr; - for (const auto &t : kTables) { - if (strcmp(t.name, table_name) == 0) { - spec = &t; - break; - } - } - if (!spec) { - std::string err = - "{\"error\":\"[module=ffi, " - "method=engine_get_knowledge_graph] unknown table '"; - err += table_name; - err += "'. Supported: entity, relation, architecture_edge, " - "module_edge, capability, document, module_summary\"}"; - return dupString(err); - } - - // Clamp limit to [0, 1000] — bounds the FFI transfer per - // block-level rule and prevents unbounded allocation. - int32_t clamped = limit < 0 ? 0 : (limit > 1000 ? 1000 : limit); - - sqlite3 *db = g_store->handle(); - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db, spec->select, -1, &stmt, nullptr) != - SQLITE_OK) { - fprintf(stderr, - "[module=ffi, method=engine_get_knowledge_graph] " - "prepare failed for table '%s': %s\n", - table_name, sqlite3_errmsg(db)); - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_knowledge_graph] prepare failed\"}"); - } - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_int(stmt, 2, clamped); - - std::string json = "{\"table\":\""; - json += table_name; - json += "\",\"rows\":["; - bool first = true; - int col_count = sqlite3_column_count(stmt); - // Count every row emitted so total/truncated are accurate. The - // previous code declared total after the loop and never - // incremented it, so total was always 0 and truncated was - // always false — any caller paginating on total missed rows - // beyond the clamped limit. - int64_t total = 0; - while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first) - json.push_back(','); - first = false; - json.push_back('{'); - for (int c = 0; c < col_count; ++c) { - if (c > 0) - json.push_back(','); - const char *cn = sqlite3_column_name(stmt, c); - json += '"'; - json += cn; - json += "\":"; - if (sqlite3_column_type(stmt, c) == - SQLITE_NULL) { - json += "null"; - continue; - } - // Numeric columns emit bare numbers; text columns - // get JSON-escaped via jsonEscape to stay safe - // against names containing quotes / newlines. - if (sqlite3_column_type(stmt, c) == - SQLITE_INTEGER) { - json += std::to_string( - sqlite3_column_int64(stmt, c)); - } else { - const char *t = - reinterpret_cast( - sqlite3_column_text( - stmt, c)); - json += '"'; - json += jsonEscape(t ? t : ""); - json += '"'; - } - } - json.push_back('}'); - total++; - } - sqlite3_finalize(stmt); - json += "],\"total\":"; - json += std::to_string(total); - json += ",\"truncated\":"; - json += (total >= clamped && clamped > 0) ? "true" : "false"; - json += "}"; - return dupString(json); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_knowledge_graph] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_knowledge_graph] unknown exception\"}"); - } -} - -char *engine_find_definition(uint64_t project_id, const char *symbol_name, - const char *file_filter) -{ - try { - if (!symbol_name || !*symbol_name) - return dupString( - "{\"error\":\"[module=ffi, method=engine_find_definition] symbol_name is required\"}"); - if (!g_query) - return dupString( - "{\"total\":0,\"results\":[],\"error\":\"not initialized\"}"); - return dupString(g_query->findDefinition( - project_id, symbol_name, file_filter)); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_find_definition] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_find_definition] unknown exception\"}"); - } -} - -char *engine_find_references(uint64_t project_id, const char *symbol_name, - const char *file_filter) -{ - try { - if (!symbol_name || !*symbol_name) - return dupString( - "{\"error\":\"[module=ffi, method=engine_find_references] symbol_name is required\"}"); - if (!g_query) - return dupString( - "{\"total\":0,\"results\":[],\"error\":\"not initialized\"}"); - return dupString(g_query->findReferences( - project_id, symbol_name, file_filter)); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_find_references] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_find_references] unknown exception\"}"); - } -} - -char *engine_get_callers(uint64_t project_id, const char *function_name, - const char *file_filter) -{ - try { - if (!function_name || !*function_name) - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_callers] function_name is required\"}"); - if (!g_query) - return dupString( - "{\"total\":0,\"callers\":[],\"error\":\"not initialized\"}"); - return dupString(g_query->getCallers(project_id, function_name, - file_filter)); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_callers] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_callers] unknown exception\"}"); - } -} - -char *engine_get_callees(uint64_t project_id, const char *function_name, - const char *file_filter) -{ - try { - if (!function_name || !*function_name) - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_callees] function_name is required\"}"); - if (!g_query) - return dupString( - "{\"total\":0,\"callees\":[],\"error\":\"not initialized\"}"); - return dupString(g_query->getCallees(project_id, function_name, - file_filter)); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_callees] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_callees] unknown exception\"}"); - } -} - -char *engine_get_neighbors(uint64_t project_id, uint64_t node_id, - int edge_type_filter, int radius) -{ - try { - if (!g_query) - return dupString( - "{\"total\":0,\"neighbors\":[],\"error\":\"not initialized\"}"); - return dupString(g_query->getNeighbors( - project_id, node_id, edge_type_filter, radius)); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_neighbors] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_neighbors] unknown exception\"}"); - } -} - -char *engine_find_shortest_path(uint64_t project_id, uint64_t source_id, - uint64_t target_id) -{ - try { - if (!g_query) - return dupString( - "{\"path\":[],\"error\":\"not initialized\"}"); - return dupString(g_query->findShortestPath( - project_id, source_id, target_id)); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_find_shortest_path] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_find_shortest_path] unknown exception\"}"); - } -} - -// ─── Connected Components ───────────────────────────────────── - -// Serialize a verify::Finding to a JSON object fragment (without the -// surrounding braces). Used by engine_find_connected_components below. -// Each finding becomes: -// "type":"...","description":"...","confidence":N,"evidence":[...] -static void appendFindingJson(std::ostringstream &json, - const verify::Finding &f) -{ - json << "\"type\":\"" << jsonEscape(f.type) << "\"," - << "\"description\":\"" << jsonEscape(f.description) << "\"," - << "\"confidence\":" << f.confidence << "," - << "\"evidence\":["; - for (size_t i = 0; i < f.evidence.size(); i++) { - if (i > 0) - json << ","; - const verify::Evidence &e = f.evidence[i]; - json << "{\"entity_name\":\"" << jsonEscape(e.entity_name) - << "\",\"file_path\":\"" << jsonEscape(e.file_path) - << "\",\"line\":" << e.line << ",\"detail\":\"" - << jsonEscape(e.detail) << "\"}"; - } - json << "]"; -} - -char *engine_find_connected_components(uint64_t project_id) -{ - try { - // Module/method tag for error messages per code_rules.md. - static const char *kModule = "ffi"; - static const char *kMethod = "engine_find_connected_components"; - - if (!g_store) { - std::ostringstream err; - err << "{\"error\":\"engine not initialized [module=" - << kModule << ", method=" << kMethod << "]\"," - << "\"components\":[],\"total\":0," - << "\"approximation\":\"heuristic\"," - << "\"note\":\"Connected components computed on name-matched " - "call edges.\"}"; - return dupString(err.str()); - } - - verify::DeadCodeInspector dci(g_store.get(), project_id); - std::vector findings = - dci.findConnectedComponents(); - - std::ostringstream json; - json << "{\"components\":["; - for (size_t i = 0; i < findings.size(); i++) { - if (i > 0) - json << ","; - json << "{"; - appendFindingJson(json, findings[i]); - json << "}"; - } - json << "],\"total\":" << findings.size() - << ",\"approximation\":\"heuristic\"," - << "\"note\":\"Connected components computed on name-matched " - "call edges.\"}"; - return dupString(json.str()); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_find_connected_components] ") + - jsonEscape(e.what()) + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_find_connected_components] unknown exception\"}"); - } -} - -char *engine_get_subgraph(uint64_t project_id, uint64_t center_node_id, - int radius, const char *node_type_filter, - const char *edge_type_filter) -{ - try { - if (!g_query) - return dupString( - "{\"total\":0,\"nodes\":[],\"error\":\"not initialized\"}"); - return dupString(g_query->getSubgraph( - project_id, center_node_id, radius, node_type_filter, - edge_type_filter)); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_subgraph] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_subgraph] unknown exception\"}"); - } -} - -char *engine_locate_node(uint64_t project_id, uint64_t node_id, - int context_lines) -{ - try { - if (!g_query) - return dupString( - "{\"total\":0,\"locations\":[],\"error\":\"not initialized\"}"); - return dupString(g_query->locateNode(project_id, node_id, - context_lines)); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_locate_node] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_locate_node] unknown exception\"}"); - } -} - -char *engine_locate_by_name(uint64_t project_id, const char *name) -{ - try { - if (!name || !*name) - return dupString( - "{\"error\":\"[module=ffi, method=engine_locate_by_name] name is required\"}"); - if (!g_query) - return dupString( - "{\"total\":0,\"locations\":[],\"error\":\"not initialized\"}"); - return dupString(g_query->locateByName(project_id, name)); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_locate_by_name] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_locate_by_name] unknown exception\"}"); - } -} - -char *engine_get_graph_stats(uint64_t project_id) -{ - try { - if (!g_query) - return dupString("{\"error\":\"not initialized\"}"); - return dupString(g_query->getGraphStats(project_id)); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_graph_stats] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_graph_stats] unknown exception\"}"); + const bool metrics_ready = + ffi_count_metrics_ready_entities(ctx, project_id) > 0; + const bool semantic_ready = + ffi_count_vector_entities(ctx, project_id) > 0; + const bool fts_ready = ctx->store->getProjectReadiness( + project_id, "fts_ready"); + + // Built with util::JsonWriter: every string is escaped and every + // separator is inserted structurally, so a stray description + // character can never produce invalid JSON (code_rules §5). + util::JsonWriter w; + w.beginObject(); + w.key("project_id").value(project_id); + w.key("total_symbols").value(total); + w.key("capabilities").beginObject(); + w.key("fast_scan").beginObject(); + w.key("available").value(true); + w.key("ready").value(true); + w.key("description") + .value(std::string("ms-level declaration extraction")); + w.endObject(); + w.key("module_tree").beginObject(); + w.key("available").value(true); + w.key("ready").value(true); + w.key("description") + .value(std::string("hierarchical module view")); + w.endObject(); + w.key("symbol_search").beginObject(); + w.key("available").value(true); + w.key("ready").value(true); + w.key("description").value(std::string("exact name match")); + w.endObject(); + w.key("entry_points").beginObject(); + w.key("available").value(true); + w.key("ready").value(total > 0); + w.key("description") + .value(std::string("main/initcall/probe detection")); + w.endObject(); + w.key("call_graph").beginObject(); + w.key("available").value(true); + w.key("ready").value(cg_ready); + w.key("coverage").beginObject(); + w.key("eligible").value(total); + w.key("ready").value(cg_ready_count); + w.endObject(); + w.key("description") + .value(std::string( + "function call edges (built during index)")); + w.endObject(); + w.key("path_tracing").beginObject(); + w.key("available").value(true); + w.key("ready").value(cg_ready); + w.key("description") + .value(std::string( + "BFS shortest path between functions")); + w.endObject(); + // v0.2.5: metrics + semantic search restored. `ready` reflects + // canonical data (entity cyclomatic > 0 / node_vectors rows), + // never a hardcoded flag. + w.key("metrics").beginObject(); + w.key("available").value(true); + w.key("ready").value(metrics_ready); + w.key("description") + .value(std::string( + "cyclomatic/cognitive/nesting complexity (computed during " + "index, resolved onto entity)")); + w.endObject(); + w.key("semantic_search").beginObject(); + w.key("available").value(true); + w.key("ready").value(semantic_ready); + w.key("mode").value(std::string("ngram_hash")); + w.key("description") + .value(std::string( + "n-gram hash vector lexical similarity (restored in " + "v0.2.5); complements FTS exact search")); + w.endObject(); + // FTS remains the exact-match workhorse; semantic search is + // additive (never replaces it). + w.key("fts").beginObject(); + w.key("available").value(true); + w.key("ready").value(fts_ready); + w.key("description") + .value(std::string( + "FTS5 full-text search for exact/prefix matching")); + w.endObject(); + w.key("context_builder").beginObject(); + w.key("available").value(true); + w.key("ready").value(true); + w.key("description") + .value(std::string("intelligent context assembly")); + w.endObject(); + w.endObject(); // capabilities + w.key("enhancement_needed") + .value(std::string( + "Call graph, complexity metrics, and n-gram semantic " + "vectors are built during index; FTS powers exact " + "search.")); + w.endObject(); + return dupString(w.str()); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_capabilities", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_capabilities", "unknown exception")); } } // ─── Full-text search ───────────────────────────────────────── -char *engine_search_code(uint64_t project_id, const char *query, int limit) +char *engine_search_code(engine_t handle, uint64_t project_id, + const char *query, int limit) { + EngineContext *ctx = engineInstance(handle); + try { if (!query || !*query) return dupString( "{\"error\":\"[module=ffi, method=engine_search_code] query is required\"}"); - if (!g_query) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) return dupString( "{\"total\":0,\"results\":[],\"error\":\"not initialized\"}"); if (limit <= 0 || limit > 100) limit = 20; - return dupString(g_query->searchCode(project_id, query, limit)); - } catch (const std::exception &e) { return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_search_code] ") + - e.what() + "\"}"); + ctx->query->searchCode(project_id, query, limit)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_search_code", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_search_code] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_search_code", "unknown exception")); } } @@ -701,60 +312,71 @@ char *engine_search_code(uint64_t project_id, const char *query, int limit) // entities by cosine similarity. When no vectors exist for the project it // returns an empty result with reason="embedding_not_built" so callers can // fall back to FTS — never a misleading "not implemented". -char *engine_search_semantic(uint64_t project_id, const char *query, int limit) +char *engine_search_semantic(engine_t handle, uint64_t project_id, + const char *query, int limit) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store || !g_store->handle() || !query) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store || !ctx->store->handle() || !query) return dupString("{\"total\":0,\"results\":[]," "\"error\":\"not initialized\"}"); - return dupString( - g_store->searchSemanticJson(project_id, query, limit)); + return dupString(ctx->store->searchSemanticJson(project_id, + query, limit)); } catch (const std::exception &e) { - return dupString(std::string("{\"error\":\"") + e.what() + - "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_search_semantic", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_search_semantic", "unknown exception")); } } // ─── Complexity Analysis ────────────────────────────────────── -char *engine_get_complexity(uint64_t project_id, uint64_t graph_node_id) +char *engine_get_complexity(engine_t handle, uint64_t project_id, + uint64_t graph_node_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_query) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) return dupString("{\"error\":\"not initialized\"}"); return dupString( - g_query->getComplexity(project_id, graph_node_id)); + ctx->query->getComplexity(project_id, graph_node_id)); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_complexity] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_complexity", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_complexity] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_complexity", "unknown exception")); } } // ─── Graph Query DSL ───────────────────────────────────────── -char *engine_graph_query(uint64_t project_id, const char *dsl_query) +char *engine_graph_query(engine_t handle, uint64_t project_id, + const char *dsl_query) { + EngineContext *ctx = engineInstance(handle); + try { if (!dsl_query || !*dsl_query) return dupString( "{\"error\":\"[module=ffi, method=engine_graph_query] dsl_query is required\"}"); - if (!g_query) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) return dupString( "{\"total\":0,\"results\":[],\"error\":\"not initialized\"}"); - return dupString(g_query->graphQuery(project_id, dsl_query)); + return dupString(ctx->query->graphQuery(project_id, dsl_query)); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_graph_query] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_graph_query", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_graph_query] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_graph_query", "unknown exception")); } } @@ -770,144 +392,159 @@ char *engine_graph_query(uint64_t project_id, const char *dsl_query) // // # Thread safety // Delegates to QueryEngine over read-only SQLite queries under the global -// g_store guard. Safe to call from the MCP server thread. -extern "C" char *engine_get_graph(uint64_t project_id, int64_t node_offset, - int node_limit, int64_t edge_offset, - int edge_limit, const char *node_type_filter, +// store guard. Safe to call from the MCP server thread. +extern "C" char *engine_get_graph(engine_t handle, uint64_t project_id, + int64_t node_offset, int node_limit, + int64_t edge_offset, int edge_limit, + const char *node_type_filter, const char *edge_type_filter) { + EngineContext *ctx = engineInstance(handle); try { - if (!g_query) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) return dupString("{\"error\":\"not initialized\"}"); - return dupString(g_query->getGraph( + return dupString(ctx->query->getGraph( project_id, node_offset, node_limit, edge_offset, edge_limit, node_type_filter, edge_type_filter)); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_graph] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope("ffi", "engine_get_graph", + e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_graph] unknown exception\"}"); + return dupString(util::errorEnvelope("ffi", "engine_get_graph", + "unknown exception")); } } // ─── Change Impact Analysis ───────────────────────────────── -char *engine_detect_changes(uint64_t project_id, +char *engine_detect_changes(engine_t handle, uint64_t project_id, const char *modified_files_json) { + EngineContext *ctx = engineInstance(handle); + try { + auto _store_guard = waitForKnowledgeBuilder(); if (!modified_files_json || !*modified_files_json) return dupString( "{\"error\":\"[module=ffi, method=engine_detect_changes] modified_files_json is required\"}"); - if (!g_query) { + if (!ctx || !ctx->query) { return dupString( "{\"error\":\"not initialized\"," "\"modified\":[],\"callers\":[],\"callees\":[],\"total_impacted\":0}"); } - return dupString(g_query->detectChanges(project_id, - modified_files_json)); + return dupString(ctx->query->detectChanges( + project_id, modified_files_json)); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_detect_changes] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_detect_changes", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_detect_changes] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_detect_changes", "unknown exception")); } } // ─── Community Detection ──────────────────────────────────── -// engine_get_communities — active. Runs label-propagation community -// detection (see engine.h). Wrapping follows the FFI Safety Contract. -char *engine_get_communities(uint64_t project_id, int max_members, - int max_communities, int include_members) +// engine_get_communities — runs label-propagation community detection over +// the CALLS graph (see engine.h and query_communities.cpp). Wrapping follows +// the FFI Safety Contract. +char *engine_get_communities(engine_t handle, uint64_t project_id, + int max_members, int max_communities, + int include_members) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_query) { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) { return dupString( "{\"error\":\"not initialized\"," - "\"communities\":[],\"inter_community_edges\":[],\"total_" - "communities\":0}"); + "\"communities\":[],\"total_communities\":0," + "\"returned_communities\":0," + "\"inter_community_edges\":0}"); } - return dupString(g_query->getCommunities( + return dupString(ctx->query->getCommunities( project_id, max_members, max_communities, include_members != 0)); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_communities] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_communities", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_communities] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_communities", "unknown exception")); } } // ─── Index Progress ────────────────────────────────────────────── -char *engine_get_index_progress(uint64_t project_id) +char *engine_get_index_progress(engine_t handle, uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"not initialized\"}"); return dupString( store::getIndexProgressJson(project_id).c_str()); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_index_progress] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_index_progress", e.what())); } catch (...) { return dupString( - "{\"error\":\"[module=ffi, method=engine_get_index_progress] unknown exception\"}"); + util::errorEnvelope("ffi", "engine_get_index_progress", + "unknown exception")); } } // ─── Async FTS Build ──────────────────────────────────────── -char *engine_build_fts(uint64_t project_id) +char *engine_build_fts(engine_t handle, uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"not initialized\"}"); - g_store->buildFTSFromGraph(project_id); - g_store->setProjectReadiness(project_id, "fts_ready", 1); - g_store->setProjectReadiness(project_id, "normal_ready", 1); + ctx->store->buildFTSFromGraph(project_id); + if (!ctx || !ctx->store->error().empty()) { + return dupString( + util::errorEnvelope("ffi", "engine_build_fts", + ctx->store->error())); + } + ctx->store->setProjectReadiness(project_id, "fts_ready", 1); + ctx->store->setProjectReadiness(project_id, "normal_ready", 1); return dupString("{\"ok\":true}"); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_build_fts] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope("ffi", "engine_build_fts", + e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_build_fts] unknown exception\"}"); + return dupString(util::errorEnvelope("ffi", "engine_build_fts", + "unknown exception")); } } // ─── Hotspot Analysis ────────────────────────────────────── -char *engine_get_hotspots(uint64_t project_id, int top_n) +char *engine_get_hotspots(engine_t handle, uint64_t project_id, int top_n) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_query) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) return dupString("{\"error\":\"not initialized\"}"); if (top_n <= 0) top_n = 10; - return dupString(g_query->getHotspots(project_id, top_n)); + return dupString(ctx->query->getHotspots(project_id, top_n)); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_hotspots] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_hotspots", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_hotspots] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_hotspots", "unknown exception")); } } @@ -917,161 +554,161 @@ char *engine_get_hotspots(uint64_t project_id, int top_n) // engine_explain_module live in engine_verify_ffi.cpp (split out to keep // this file under the 1000-line limit; see code_rules.md §1). -extern "C" char *engine_explain_symbol(uint64_t project_id, +extern "C" char *engine_explain_symbol(engine_t handle, uint64_t project_id, const char *symbol_name) { + EngineContext *ctx = engineInstance(handle); + try { if (!symbol_name || !*symbol_name) return dupString( "{\"error\":\"[module=ffi, method=engine_explain_symbol] symbol_name is required\"}"); - if (!g_query) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) return dupString("{\"error\":\"not initialized\"}"); return dupString( - g_query->explainSymbol(project_id, symbol_name)); + ctx->query->explainSymbol(project_id, symbol_name)); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_explain_symbol] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_explain_symbol", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_explain_symbol] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_explain_symbol", "unknown exception")); } } -char *engine_get_module_map(uint64_t project_id) +char *engine_get_module_map(engine_t handle, uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_query) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) return dupString("{\"error\":\"not initialized\"}"); - return dupString(g_query->getModuleMap(project_id)); + return dupString(ctx->query->getModuleMap(project_id)); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_module_map] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_module_map", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_module_map] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_module_map", "unknown exception")); } } -char *engine_get_entry_points(uint64_t project_id) +char *engine_get_entry_points(engine_t handle, uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_query) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) return dupString("{\"error\":\"not initialized\"}"); - return dupString(g_query->getEntryPoints(project_id)); + return dupString(ctx->query->getEntryPoints(project_id)); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_entry_points] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_entry_points", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_entry_points] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_entry_points", "unknown exception")); } } -char *engine_trace_call_chain(uint64_t project_id, const char *from, - const char *to) +char *engine_trace_call_chain(engine_t handle, uint64_t project_id, + const char *from, const char *to) { + EngineContext *ctx = engineInstance(handle); + try { if (!from || !*from || !to || !*to) return dupString( "{\"error\":\"[module=ffi, method=engine_trace_call_chain] from and to are required\"}"); - if (!g_query) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) return dupString("{\"error\":\"not initialized\"}"); - return dupString(g_query->traceCallChain(project_id, from, to)); - } catch (const std::exception &e) { return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_trace_call_chain] ") + - e.what() + "\"}"); + ctx->query->traceCallChain(project_id, from, to)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_trace_call_chain", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_trace_call_chain] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_trace_call_chain", "unknown exception")); } } -char *engine_get_project_overview(uint64_t project_id) +char *engine_get_project_overview(engine_t handle, uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_query) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) return dupString("{\"error\":\"not initialized\"}"); - return dupString(g_query->getProjectOverview(project_id)); + return dupString(ctx->query->getProjectOverview(project_id)); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_project_overview] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_project_overview", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_project_overview] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_project_overview", + "unknown exception")); } } -char *engine_get_type_info(uint64_t project_id, const char *type_name_filter) +char *engine_get_type_info(engine_t handle, uint64_t project_id, + const char *type_name_filter) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"not initialized\"}"); - // Query type_info table for type definitions + // Query type_info table for type definitions. project_id and the + // name filter are bound with sqlite3_bind_* — never concatenated + // into the SQL text (code_rules §5: no injection surface). std::string sql = "SELECT ti.name, ti.qualified_name, ti.kind, ti.file_path, " " ti.language, ti.start_row, " " (SELECT COUNT(*) FROM type_ref tr WHERE tr.type_name = ti.name " " AND tr.project_id = ti.project_id) AS ref_count " - "FROM type_info ti WHERE ti.project_id=" + - std::to_string(project_id); - - if (type_name_filter && *type_name_filter) { - // Escape special characters for SQL LIKE: % _ and ' - std::string filter(type_name_filter); - size_t pos = 0; - while ((pos = filter.find('\\', pos)) != - std::string::npos) { - filter.replace(pos, 1, "\\\\"); - pos += 2; - } - pos = 0; - while ((pos = filter.find('%', pos)) != - std::string::npos) { - filter.replace(pos, 1, "\\%"); - pos += 2; - } - pos = 0; - while ((pos = filter.find('_', pos)) != - std::string::npos) { - filter.replace(pos, 1, "\\_"); - pos += 2; + "FROM type_info ti WHERE ti.project_id = ?"; + + const bool has_filter = type_name_filter && *type_name_filter; + std::string like_pattern; + if (has_filter) { + // Escape LIKE metacharacters (\ first, then % and _) so the + // caller's filter matches literally. The value is passed via + // sqlite3_bind_text, so quote escaping is unnecessary. + like_pattern = "%"; + for (const char c : std::string(type_name_filter)) { + if (c == '\\' || c == '%' || c == '_') + like_pattern.push_back('\\'); + like_pattern.push_back(c); } - pos = 0; - while ((pos = filter.find('\'', pos)) != - std::string::npos) { - filter.replace(pos, 1, "''"); - pos += 2; - } - sql += " AND ti.name LIKE '%" + filter + - "%' ESCAPE '\\'"; + like_pattern.push_back('%'); + sql += " AND ti.name LIKE ? ESCAPE '\\'"; } sql += " ORDER BY ref_count DESC LIMIT 100"; sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(g_store->handle(), sql.c_str(), -1, + if (sqlite3_prepare_v2(ctx->store->handle(), sql.c_str(), -1, &stmt, nullptr) != SQLITE_OK) { return dupString( "{\"error\":\"query failed\",\"types\":[]}"); } + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + if (has_filter) { + sqlite3_bind_text(stmt, 2, like_pattern.c_str(), -1, + SQLITE_TRANSIENT); + } - std::string result = "{\"types\":["; - bool first = true; + util::JsonWriter w; + w.beginObject(); + w.key("types").beginArray(); while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first) - result += ","; - first = false; const char *n = reinterpret_cast( sqlite3_column_text(stmt, 0)); const char *qn = reinterpret_cast( @@ -1084,37 +721,36 @@ char *engine_get_type_info(uint64_t project_id, const char *type_name_filter) int row = sqlite3_column_int(stmt, 5); int64_t ref_count = sqlite3_column_int64(stmt, 6); - result += - "{\"name\":\"" + jsonEscape(n ? n : "") + "\""; - result += ",\"qualified_name\":\"" + - jsonEscape(qn ? qn : "") + "\""; - result += ",\"kind\":" + std::to_string(kind); - result += ",\"file_path\":\"" + - jsonEscape(fp ? fp : "") + "\""; - result += ",\"language\":\"" + - jsonEscape(lang ? lang : "") + "\""; - result += ",\"line\":" + std::to_string(row); - result += ",\"ref_count\":" + - std::to_string(ref_count) + "}"; + w.beginObject(); + w.key("name").value(n ? n : ""); + w.key("qualified_name").value(qn ? qn : ""); + w.key("kind").value(kind); + w.key("file_path").value(fp ? fp : ""); + w.key("language").value(lang ? lang : ""); + w.key("line").value(row); + w.key("ref_count").value(ref_count); + w.endObject(); } sqlite3_finalize(stmt); - result += "]}"; - return dupString(result.c_str()); + w.endArray(); + w.endObject(); + return dupString(w.str()); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_type_info] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_type_info", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_type_info] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_type_info", "unknown exception")); } } -char *engine_get_routes(uint64_t project_id) +char *engine_get_routes(engine_t handle, uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"not initialized\"}"); const char *sql = @@ -1122,19 +758,17 @@ char *engine_get_routes(uint64_t project_id) "FROM route WHERE project_id=? ORDER BY method, path LIMIT 500"; sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(g_store->handle(), sql, -1, &stmt, + if (sqlite3_prepare_v2(ctx->store->handle(), sql, -1, &stmt, nullptr) != SQLITE_OK) { return dupString( "{\"error\":\"query failed\",\"routes\":[]}"); } sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - std::string result = "{\"routes\":["; - bool first = true; + util::JsonWriter w; + w.beginObject(); + w.key("routes").beginArray(); while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first) - result += ","; - first = false; const char *m = reinterpret_cast( sqlite3_column_text(stmt, 0)); const char *p = reinterpret_cast( @@ -1145,27 +779,24 @@ char *engine_get_routes(uint64_t project_id) sqlite3_column_text(stmt, 3)); int line = sqlite3_column_int(stmt, 4); - result += "{\"method\":\"" + jsonEscape(m ? m : "") + - "\""; - result += - ",\"path\":\"" + jsonEscape(p ? p : "") + "\""; - result += ",\"handler\":\"" + jsonEscape(h ? h : "") + - "\""; - result += - ",\"file\":\"" + jsonEscape(f ? f : "") + "\""; - result += ",\"line\":" + std::to_string(line) + "}"; + w.beginObject(); + w.key("method").value(m ? m : ""); + w.key("path").value(p ? p : ""); + w.key("handler").value(h ? h : ""); + w.key("file").value(f ? f : ""); + w.key("line").value(line); + w.endObject(); } sqlite3_finalize(stmt); - result += "]}"; - return dupString(result.c_str()); + w.endArray(); + w.endObject(); + return dupString(w.str()); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_routes] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope("ffi", "engine_get_routes", + e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_routes] unknown exception\"}"); + return dupString(util::errorEnvelope("ffi", "engine_get_routes", + "unknown exception")); } } @@ -1179,444 +810,49 @@ void engine_free_string(char *ptr) free(ptr); } -// ─── Batch Indexing ────────────────────────────────────────── - -char *engine_index_batch(uint64_t project_id, const char *file_paths_json) -{ - try { - if (!file_paths_json || !*file_paths_json) - return dupString( - "{\"error\":\"[module=ffi, method=engine_index_batch] file_paths_json is required\"}"); - if (!g_store || !g_parser) - return dupString( - "{\"ok\":false,\"error\":\"not initialized\"}"); - - // Parse JSON array of file paths - std::vector paths; - { - const char *p = file_paths_json; - if (!p || !*p) - return dupString( - "{\"ok\":false,\"error\":\"empty file list\"}"); - while (*p && *p != '[') - p++; - if (!*p) - return dupString( - "{\"ok\":false,\"error\":\"expected [\"}"); - p++; - while (*p) { - while (*p == ' ' || *p == '\t' || *p == '\n' || - *p == '\r' || *p == ',') - p++; - if (*p == ']') - break; - if (*p != '"') - return dupString( - "{\"ok\":false,\"error\":\"expected string\"}"); - p++; - std::string path; - while (*p && *p != '"') { - if (*p == '\\') { - p++; - switch (*p) { - case '"': - path += '"'; - break; - case '\\': - path += '\\'; - break; - case '/': - path += '/'; - break; - case 'n': - path += '\n'; - break; - case 't': - path += '\t'; - break; - case 'r': - path += '\r'; - break; - case 'b': - path += '\b'; - break; - case 'f': - path += '\f'; - break; - case 'u': { - // Simple pass-through for \uXXXX. - // Buffer layout: '\' 'u' + up to 5 hex - // digits + NUL = 8 bytes. The loop below - // lets i reach 6, then writes the NUL at - // index 7, so the buffer must hold 8. - char unicode_buf[8] = - "\\u"; - int i = 1; - while (*++p && i < 6 && - ((*p >= '0' && - *p <= '9') || - (*p >= 'a' && - *p <= 'f') || - (*p >= 'A' && - *p <= 'F'))) - unicode_buf[++i] = - *p; - unicode_buf[++i] = '\0'; - path += unicode_buf; - p--; // loop will advance p - break; - } - default: - path += '\\'; - path += *p; - break; - } - } else { - path += *p; - } - p++; - } - if (*p != '"') - return dupString( - "{\"ok\":false,\"error\":\"unterminated string\"}"); - p++; - if (!path.empty()) - paths.push_back(path); - } - } - if (paths.empty()) - return dupString( - "{\"ok\":false,\"error\":\"empty file list\"}"); - - // Phase 1: Parse all files in memory (no DB I/O) - struct FileBatch { - std::unique_ptr unit; - std::string source; - std::string language; - std::string file_path; - FileBatch(std::unique_ptr u, - std::string s, std::string l, std::string fp) - : unit(std::move(u)) - , source(std::move(s)) - , language(std::move(l)) - , file_path(std::move(fp)) - { - } - }; - std::vector batches; - std::vector errors; - - for (const auto &fp : paths) { - const char *lang = detectLanguage(fp.c_str()); - if (!lang) { - errors.push_back(fp + ": unsupported"); - continue; - } - - std::string source = readFile(fp.c_str()); - if (source.empty()) { - errors.push_back(fp + ": cannot read"); - continue; - } - - TSTree *tree = g_parser->parse(fp.c_str(), - source.c_str(), lang, - source.size()); - if (!tree) { - errors.push_back(fp + ": parse failed"); - continue; - } - - std::unique_ptr translator( - ir::createTranslator(lang)); - if (!translator) { - ts_tree_delete(tree); - errors.push_back(fp + ": no translator"); - continue; - } - - ir::TranslationUnit *unit = translator->translate( - tree, source.c_str(), fp.c_str()); - ts_tree_delete(tree); - if (!unit) { - errors.push_back(fp + ": translation failed"); - continue; - } - - batches.push_back(FileBatch{ - std::unique_ptr(unit), - std::move(source), lang, fp }); - } - - // Phase 2: Persist in single transaction - g_store->beginTransaction(); - - uint64_t start_id = 1; - { - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2( - g_store->handle(), - "SELECT COALESCE(MAX(id),0)+1 FROM graph_nodes", - -1, &stmt, nullptr) == SQLITE_OK) { - if (sqlite3_step(stmt) == SQLITE_ROW) - start_id = static_cast( - sqlite3_column_int64(stmt, 0)); - sqlite3_finalize(stmt); - } - } - - graph::GraphBuilder builder(project_id, start_id); - int total_nodes = 0, total_edges = 0; - - for (auto &b : batches) { - std::string hash = simpleHash(b.source); - g_store->upsertFile(project_id, b.file_path.c_str(), - b.language.c_str(), hash.c_str()); - // Delete graph-layer data only (relation, graph_edges, - // graph_nodes, entity, type_ref, type_info, import, route). - // NOT semantic_records — this batch path re-inserts graph - // data directly from in-memory `b.unit` and never repopulates - // semantic_records. buildGraph later uses semantic_records to - // decide the file rebuild set; wiping it here would make - // subsequent engine_index_project skip rebuilding this file, - // leaving stale graph_nodes forever. - g_store->deleteGraphDataByFile(project_id, - b.file_path.c_str()); - - // No ir_nodes/ir_semantic_edges write — graph_nodes is canonical. - // FTS/vector writes skipped for single-file index path. - - auto sg = builder.buildSymbolGraph(b.unit.get()); - auto cg = builder.buildCallGraph(b.unit.get()); - for (auto &gn : sg.nodes) { - g_store->insertGraphNode(project_id, gn); - g_store->insertEntity(project_id, gn); - total_nodes++; - } - for (auto &e : sg.edges) { - g_store->insertGraphEdge(project_id, e); - total_edges++; - } - for (auto &e : cg.edges) { - g_store->insertGraphEdge(project_id, e); - total_edges++; - } - - // ComplexityAnalyzer removed (Phase 0) - for (auto &gn : sg.nodes) - if (gn.type == graph::NodeType::Function || - gn.type == graph::NodeType::Method) - for (auto *in : b.unit->all_nodes) - if (in->id == gn.ir_node_id) { - break; - } - } - - g_store->commitTransaction(); - - std::ostringstream r; - r << "{\"ok\":true,\"files\":" - << (batches.size() + errors.size()) - << ",\"indexed\":" << batches.size() - << ",\"nodes\":" << total_nodes - << ",\"edges\":" << total_edges << ",\"errors\":["; - for (size_t i = 0; i < errors.size(); i++) { - if (i > 0) - r << ","; - r << "\"" << jsonEscape(errors[i]) << "\""; - } - r << "]}"; - return dupString(r.str()); - } catch (const std::exception &e) { - g_store->rollbackTransaction(); - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_index_batch] ") + - e.what() + "\"}"); - } catch (...) { - g_store->rollbackTransaction(); - return dupString( - "{\"error\":\"[module=ffi, method=engine_index_batch] unknown exception\"}"); - } -} - -// ─── Project Metadata ─────────────────────────────────────── - -static const char *detectLicense(const std::string &content) -{ - if (content.find("Apache License") != std::string::npos || - content.find("Version 2.0, January 2004") != std::string::npos) - return "Apache-2.0"; - if (content.find("MIT License") != std::string::npos || - content.find("Permission is hereby granted") != std::string::npos) - return "MIT"; - if (content.find("GNU GENERAL PUBLIC LICENSE") != std::string::npos) - return content.find("Version 3") != std::string::npos ? - "GPL-3.0" : - "GPL-2.0"; - if (content.find("BSD") != std::string::npos) - return "BSD"; - if (content.find("Mozilla Public") != std::string::npos) - return "MPL-2.0"; - return "Unknown"; -} - -char *engine_get_project_info(uint64_t project_id) -{ - try { - if (!g_store) - return dupString("{\"error\":\"not initialized\"}"); - - sqlite3 *db = g_store->handle(); - std::string name, root; - - { - sqlite3_stmt *stmt = nullptr; - const char *sql = - "SELECT name, root_path FROM projects WHERE id=?"; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64( - stmt, 1, - static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) { - if (sqlite3_column_text(stmt, 0)) - name = reinterpret_cast< - const char *>( - sqlite3_column_text( - stmt, 0)); - if (sqlite3_column_text(stmt, 1)) - root = reinterpret_cast< - const char *>( - sqlite3_column_text( - stmt, 1)); - } - sqlite3_finalize(stmt); - } - } - - // Detect license - std::string license = "Unknown"; - const char *lfs[] = { "LICENSE", "LICENSE.txt", - "LICENSE.md", "LICENSE-APACHE", - "COPYING", nullptr }; - for (int i = 0; lfs[i]; i++) { - std::string c = readFile((root + "/" + lfs[i]).c_str()); - if (!c.empty()) { - license = detectLicense(c); - break; - } - } - - // Primary language - std::string lang; - { - sqlite3_stmt *stmt = nullptr; - const char *sql = - "SELECT language,COUNT(*) FROM files WHERE project_id=? " - "GROUP BY language ORDER BY 2 DESC LIMIT 1"; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64( - stmt, 1, - static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW && - sqlite3_column_text(stmt, 0)) - lang = reinterpret_cast( - sqlite3_column_text(stmt, 0)); - sqlite3_finalize(stmt); - } - } - - int file_count = 0, dep_count = 0; - { - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2( - db, - "SELECT COUNT(*) FROM files WHERE project_id=?", - -1, &stmt, nullptr) == SQLITE_OK) { - sqlite3_bind_int64( - stmt, 1, - static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) - file_count = - sqlite3_column_int(stmt, 0); - sqlite3_finalize(stmt); - } - } - - // Try to parse dep files - const char *dfs[] = { "go.mod", "Cargo.toml", - "pyproject.toml", "package.json", - "requirements.txt", nullptr }; - for (int i = 0; dfs[i]; i++) { - std::string c = readFile((root + "/" + dfs[i]).c_str()); - if (!c.empty()) { - int lines = 0; - for (size_t p = 0; - (p = c.find('\n', p)) != std::string::npos; - lines++, p++) - ; - dep_count = lines / 3; - break; - } - } - - std::ostringstream j; - j << "{\"name\":\"" << jsonEscape(name) << "\",\"license\":\"" - << jsonEscape(license) << "\",\"language\":\"" - << jsonEscape(lang) << "\",\"file_count\":" << file_count - << ",\"dependency_count\":" << dep_count << "}"; - return dupString(j.str()); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_get_project_info] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_get_project_info] unknown exception\"}"); - } -} - // ─── Shared Artifact ───────────────────────────────────────────── -char *engine_export_artifact(uint64_t project_id, const char *output_path) +char *engine_export_artifact(engine_t handle, uint64_t project_id, + const char *output_path) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store || !output_path || !*output_path) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store || !output_path || !*output_path) return dupString( "{\"ok\":false,\"error\":\"invalid arguments\"}"); - auto result = g_store->exportArtifact(project_id, output_path); + auto result = + ctx->store->exportArtifact(project_id, output_path); return dupString(result); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_export_artifact] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_export_artifact", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_export_artifact] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_export_artifact", "unknown exception")); } } -char *engine_import_artifact(uint64_t project_id, const char *artifact_path) +char *engine_import_artifact(engine_t handle, uint64_t project_id, + const char *artifact_path) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store || !artifact_path || !*artifact_path) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store || !artifact_path || !*artifact_path) return dupString( "{\"ok\":false,\"error\":\"invalid arguments\"}"); auto result = - g_store->importArtifact(project_id, artifact_path); + ctx->store->importArtifact(project_id, artifact_path); return dupString(result); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_import_artifact] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_import_artifact", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_import_artifact] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_import_artifact", "unknown exception")); } } @@ -1630,7 +866,7 @@ char *engine_import_artifact(uint64_t project_id, const char *artifact_path) // merge, the scheduler calls this to rebuild CSR from the globally-remapped // relation table (buildCSR reads relation type=1 edges), so all neighbor // ids are global again. It opens a LOCAL GraphStore on db_path so it does -// not disturb the process-wide g_store. +// not disturb the process-wide engine store. // // @param db_path Path to the (merged) SQLite DB. // @param project_id Project whose CSR to rebuild. @@ -1646,10 +882,9 @@ extern "C" char *engine_rebuild_csr(const char *db_path, uint64_t project_id) } store::GraphStore local_store; if (!local_store.open(db_path)) { - return dupString( - "{\"error\":\"[module=ffi, " - "method=engine_rebuild_csr] cannot open db: " + - std::string(db_path) + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_rebuild_csr", + std::string("cannot open db: ") + db_path)); } if (!local_store.buildCSR(project_id)) { return dupString( @@ -1661,9 +896,11 @@ extern "C" char *engine_rebuild_csr(const char *db_path, uint64_t project_id) return dupString("{\"ok\":true,\"project_id\":" + std::to_string(project_id) + "}"); } catch (const std::exception &e) { - return dupString(std::string("{\"error\":\"[module=ffi, " - "method=engine_rebuild_csr] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_rebuild_csr", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_rebuild_csr", "unknown exception")); } } @@ -1675,6 +912,6 @@ const char *engine_version(void) // No try/catch required — only a static string literal is returned, // so no exceptions are possible. // Keep in sync with RELEASE.md and Cargo.toml version. - static const char kVersion[] = "0.2.6"; + static const char kVersion[] = "0.2.7"; return kVersion; } diff --git a/engine/src/engine_ffi_graph.cpp b/engine/src/engine_ffi_graph.cpp new file mode 100644 index 0000000..84549a0 --- /dev/null +++ b/engine/src/engine_ffi_graph.cpp @@ -0,0 +1,490 @@ +// engine_ffi_graph.cpp — knowledge-graph direct query + navigation. +// +// Split out of engine_ffi.cpp (see plan/rules/code_rules.md 1000-line +// rule). Surfaces the knowledge-layer tables (entity / relation / +// architecture_edge / module_edge / capability / document / +// module_summary) so MCP clients can browse the graph directly instead +// of only benefiting from it indirectly via explain_module / +// detect_capability_drift / get_module_tree. +// Every function follows the FFI safety contract in engine_ffi.cpp: +// try/catch around the body, null-checked inputs, dupString() result +// the caller frees with engine_free_string(). + +#include "engine_internal.h" +#include "async_knowledge.h" +#include "platform_win.h" +#include "util/json_writer.h" + +#include +#include +#include +#include +#include + +// engine_find_connected_components walks findings and dead-code evidence. +#include "verify/dead_code_inspector.h" +#include "verify/finding.h" + +// ─── Knowledge Graph direct query (v0.2.1) ───────────────────────────── +// +// Surfaces the knowledge-layer tables (entity / relation / architecture_edge / +// module_edge / capability / document / module_summary) so MCP clients can +// browse the knowledge graph directly, instead of only benefiting from it +// indirectly via explain_module / detect_capability_drift / get_module_tree. +// +// Per plan/rules/code_rules.md §FFI: this is a block-level transfer — one +// FFI call returns the entire result set (bounded by `limit`), never one +// row per call. Error paths emit a stderr line tagged with module=ffi, +// method=engine_get_knowledge_graph per §"Additional Rules". +char *engine_get_knowledge_graph(engine_t handle, uint64_t project_id, + const char *table_name, int32_t limit) +{ + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) + return dupString( + "{\"error\":\"[module=ffi, method=engine_get_knowledge_graph] engine not initialized\"}"); + if (!table_name || !*table_name) + return dupString( + "{\"error\":\"[module=ffi, method=engine_get_knowledge_graph] table_name is required\"}"); + + // Whitelist of knowledge-layer tables. We never let the caller pass + // arbitrary SQL — the table_name is matched against this fixed set + // and the SELECT is built with a hard-coded column list per table. + // This prevents SQL injection via the table_name parameter. + struct TableSpec { + const char *name; + const char * + select; // hard-coded column list, no user input + }; + static const TableSpec kTables[] = { + { "entity", + "SELECT id, name, qualified_name, kind, " + "file_path, start_row, start_col FROM entity " + "WHERE project_id=? ORDER BY id LIMIT ?" }, + { "relation", "SELECT id, source_id, target_id, type " + "FROM relation WHERE project_id=? " + "ORDER BY id LIMIT ?" }, + { "architecture_edge", + "SELECT id, callee_module, caller_module, " + "entity_id FROM architecture_edge " + "WHERE project_id=? ORDER BY id LIMIT ?" }, + { "module_edge", + "SELECT id, src_module, tgt_module, " + "edge_count FROM module_edge " + "WHERE project_id=? ORDER BY id LIMIT ?" }, + { "capability", + "SELECT id, name, summary FROM capability " + "WHERE project_id=? ORDER BY id LIMIT ?" }, + { "document", + "SELECT id, type, file_path, start_line, end_line " + "FROM document " + "WHERE project_id=? ORDER BY id LIMIT ?" }, + { "module_summary", + "SELECT id, module_id, state, incoming_count, " + "outgoing_count, internal_edges, " + "dead_entities, utilization, confidence " + "FROM module_summary " + "WHERE project_id=? ORDER BY id LIMIT ?" }, + }; + const TableSpec *spec = nullptr; + for (const auto &t : kTables) { + if (strcmp(t.name, table_name) == 0) { + spec = &t; + break; + } + } + if (!spec) { + util::JsonWriter w; + w.beginObject(); + w.key("error").value( + std::string( + "[module=ffi, " + "method=engine_get_knowledge_graph] " + "unknown table '") + + table_name + + "'. Supported: entity, relation, " + "architecture_edge, module_edge, capability, " + "document, module_summary"); + w.endObject(); + return dupString(w.str()); + } + + // Clamp limit to [0, 1000] — bounds the FFI transfer per + // block-level rule and prevents unbounded allocation. + int32_t clamped = limit < 0 ? 0 : (limit > 1000 ? 1000 : limit); + + sqlite3 *db = ctx->store->handle(); + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, spec->select, -1, &stmt, nullptr) != + SQLITE_OK) { + fprintf(stderr, + "[module=ffi, method=engine_get_knowledge_graph] " + "prepare failed for table '%s': %s\n", + table_name, sqlite3_errmsg(db)); + return dupString( + "{\"error\":\"[module=ffi, method=engine_get_knowledge_graph] prepare failed\"}"); + } + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_int(stmt, 2, clamped); + + util::JsonWriter w; + w.beginObject(); + w.key("table").value(table_name); + w.key("rows").beginArray(); + int col_count = sqlite3_column_count(stmt); + // Count every row emitted so total/truncated are accurate. The + // previous code declared total after the loop and never + // incremented it, so total was always 0 and truncated was + // always false — any caller paginating on total missed rows + // beyond the clamped limit. + int64_t total = 0; + while (sqlite3_step(stmt) == SQLITE_ROW) { + w.beginObject(); + for (int c = 0; c < col_count; ++c) { + const char *cn = sqlite3_column_name(stmt, c); + // key() escapes the column name; null for missing. + if (sqlite3_column_type(stmt, c) == + SQLITE_NULL) { + w.key(cn ? cn : "").nullValue(); + continue; + } + // Numeric columns emit bare numbers; text columns + // are escaped by JsonWriter (names can contain + // quotes / newlines). + if (sqlite3_column_type(stmt, c) == + SQLITE_INTEGER) { + w.key(cn ? cn : "") + .value(sqlite3_column_int64( + stmt, c)); + } else { + const char *t = + reinterpret_cast( + sqlite3_column_text( + stmt, c)); + w.key(cn ? cn : "").value(t ? t : ""); + } + } + w.endObject(); + total++; + } + sqlite3_finalize(stmt); + w.endArray(); + w.key("total").value(total); + w.key("truncated").value(total >= clamped && clamped > 0); + w.endObject(); + return dupString(w.str()); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_knowledge_graph", e.what())); + } catch (...) { + return dupString( + util::errorEnvelope("ffi", "engine_get_knowledge_graph", + "unknown exception")); + } +} + +char *engine_find_definition(engine_t handle, uint64_t project_id, + const char *symbol_name, const char *file_filter) +{ + EngineContext *ctx = engineInstance(handle); + + try { + if (!symbol_name || !*symbol_name) + return dupString( + "{\"error\":\"[module=ffi, method=engine_find_definition] symbol_name is required\"}"); + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) + return dupString( + "{\"total\":0,\"results\":[],\"error\":\"not initialized\"}"); + return dupString(ctx->query->findDefinition( + project_id, symbol_name, file_filter)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_definition", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_definition", "unknown exception")); + } +} + +char *engine_find_references(engine_t handle, uint64_t project_id, + const char *symbol_name, const char *file_filter) +{ + EngineContext *ctx = engineInstance(handle); + + try { + if (!symbol_name || !*symbol_name) + return dupString( + "{\"error\":\"[module=ffi, method=engine_find_references] symbol_name is required\"}"); + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) + return dupString( + "{\"total\":0,\"results\":[],\"error\":\"not initialized\"}"); + return dupString(ctx->query->findReferences( + project_id, symbol_name, file_filter)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_references", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_references", "unknown exception")); + } +} + +char *engine_get_callers(engine_t handle, uint64_t project_id, + const char *function_name, const char *file_filter) +{ + EngineContext *ctx = engineInstance(handle); + + try { + if (!function_name || !*function_name) + return dupString( + "{\"error\":\"[module=ffi, method=engine_get_callers] function_name is required\"}"); + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) + return dupString( + "{\"total\":0,\"callers\":[],\"error\":\"not initialized\"}"); + return dupString(ctx->query->getCallers( + project_id, function_name, file_filter)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_callers", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_callers", "unknown exception")); + } +} + +char *engine_get_callees(engine_t handle, uint64_t project_id, + const char *function_name, const char *file_filter) +{ + EngineContext *ctx = engineInstance(handle); + + try { + if (!function_name || !*function_name) + return dupString( + "{\"error\":\"[module=ffi, method=engine_get_callees] function_name is required\"}"); + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) + return dupString( + "{\"total\":0,\"callees\":[],\"error\":\"not initialized\"}"); + return dupString(ctx->query->getCallees( + project_id, function_name, file_filter)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_callees", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_callees", "unknown exception")); + } +} + +char *engine_get_neighbors(engine_t handle, uint64_t project_id, + uint64_t node_id, int edge_type_filter, int radius) +{ + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) + return dupString( + "{\"total\":0,\"neighbors\":[],\"error\":\"not initialized\"}"); + return dupString(ctx->query->getNeighbors( + project_id, node_id, edge_type_filter, radius)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_neighbors", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_neighbors", "unknown exception")); + } +} + +char *engine_find_shortest_path(engine_t handle, uint64_t project_id, + uint64_t source_id, uint64_t target_id) +{ + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) + return dupString( + "{\"path\":[],\"error\":\"not initialized\"}"); + return dupString(ctx->query->findShortestPath( + project_id, source_id, target_id)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_shortest_path", e.what())); + } catch (...) { + return dupString( + util::errorEnvelope("ffi", "engine_find_shortest_path", + "unknown exception")); + } +} + +// ─── Connected Components ───────────────────────────────────── + +// Serialize a verify::Finding to a JSON object fragment (without the +// surrounding braces). Used by engine_find_connected_components below. +// Each finding becomes: +// "type":"...","description":"...","confidence":N,"evidence":[...] +static void appendFindingJson(util::JsonWriter &w, const verify::Finding &f) +{ + w.key("type").value(f.type); + w.key("description").value(f.description); + w.key("confidence").value(f.confidence); + w.key("evidence").beginArray(); + for (const verify::Evidence &e : f.evidence) { + w.beginObject(); + w.key("entity_name").value(e.entity_name); + w.key("file_path").value(e.file_path); + w.key("line").value(e.line); + w.key("detail").value(e.detail); + w.endObject(); + } + w.endArray(); +} + +char *engine_find_connected_components(engine_t handle, uint64_t project_id) +{ + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + // Module/method tag for error messages per code_rules.md. + static const char *kModule = "ffi"; + static const char *kMethod = "engine_find_connected_components"; + static const char *kNote = + "Connected components computed on name-matched call " + "edges."; + + if (!ctx || !ctx->store) { + util::JsonWriter w; + w.beginObject(); + w.key("error").value( + std::string("engine not initialized [module=") + + kModule + ", method=" + kMethod + "]"); + w.key("components").beginArray().endArray(); + w.key("total").value(0); + w.key("approximation").value(std::string("heuristic")); + w.key("note").value(std::string(kNote)); + w.endObject(); + return dupString(w.str()); + } + + verify::DeadCodeInspector dci(ctx->store.get(), project_id); + std::vector findings = + dci.findConnectedComponents(); + + util::JsonWriter w; + w.beginObject(); + w.key("components").beginArray(); + for (const auto &f : findings) { + w.beginObject(); + appendFindingJson(w, f); + w.endObject(); + } + w.endArray(); + w.key("total").value(findings.size()); + w.key("approximation").value(std::string("heuristic")); + w.key("note").value(std::string(kNote)); + w.endObject(); + return dupString(w.str()); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_connected_components", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_connected_components", + "unknown exception")); + } +} + +char *engine_get_subgraph(engine_t handle, uint64_t project_id, + uint64_t center_node_id, int radius, + const char *node_type_filter, + const char *edge_type_filter) +{ + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) + return dupString( + "{\"total\":0,\"nodes\":[],\"error\":\"not initialized\"}"); + return dupString(ctx->query->getSubgraph( + project_id, center_node_id, radius, node_type_filter, + edge_type_filter)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_subgraph", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_subgraph", "unknown exception")); + } +} + +char *engine_locate_node(engine_t handle, uint64_t project_id, uint64_t node_id, + int context_lines) +{ + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) + return dupString( + "{\"total\":0,\"locations\":[],\"error\":\"not initialized\"}"); + return dupString(ctx->query->locateNode(project_id, node_id, + context_lines)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_locate_node", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_locate_node", "unknown exception")); + } +} + +char *engine_locate_by_name(engine_t handle, uint64_t project_id, + const char *name) +{ + EngineContext *ctx = engineInstance(handle); + + try { + if (!name || !*name) + return dupString( + "{\"error\":\"[module=ffi, method=engine_locate_by_name] name is required\"}"); + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) + return dupString( + "{\"total\":0,\"locations\":[],\"error\":\"not initialized\"}"); + return dupString(ctx->query->locateByName(project_id, name)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_locate_by_name", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_locate_by_name", "unknown exception")); + } +} + +char *engine_get_graph_stats(engine_t handle, uint64_t project_id) +{ + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query) + return dupString("{\"error\":\"not initialized\"}"); + return dupString(ctx->query->getGraphStats(project_id)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_graph_stats", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_graph_stats", "unknown exception")); + } +} diff --git a/engine/src/engine_ffi_index.cpp b/engine/src/engine_ffi_index.cpp new file mode 100644 index 0000000..526e94d --- /dev/null +++ b/engine/src/engine_ffi_index.cpp @@ -0,0 +1,504 @@ +// engine_ffi_index.cpp — batch indexing and project metadata. +// +// Split out of engine_ffi.cpp (see plan/rules/code_rules.md 1000-line +// rule). engine_index_batch is the worker subprocess entry point that +// parses and inserts a whole file list in one call; +// engine_get_project_info reports what was indexed (license, counts, +// timestamps). Both describe the project on disk rather than querying +// the graph. +// Every function follows the FFI safety contract in engine_ffi.cpp: +// try/catch around the body, null-checked inputs, dupString() result +// the caller frees with engine_free_string(). + +#include "engine_internal.h" +#include "async_knowledge.h" +#include "platform_win.h" +#include "util/json_writer.h" +#include "store/store_parse_failure.h" + +#include +#include +#include +#include +#include + +// ─── Batch Indexing ────────────────────────────────────────── + +char *engine_index_batch(engine_t handle, uint64_t project_id, + const char *file_paths_json) +{ + EngineContext *ctx = engineInstance(handle); + + try { + if (!file_paths_json || !*file_paths_json) + return dupString( + "{\"error\":\"[module=ffi, method=engine_index_batch] file_paths_json is required\"}"); + if (!ctx || !ctx->store || !ctx->parser) + return dupString( + "{\"ok\":false,\"error\":\"not initialized\"}"); + + // Parse JSON array of file paths + std::vector paths; + { + const char *p = file_paths_json; + if (!p || !*p) + return dupString( + "{\"ok\":false,\"error\":\"empty file list\"}"); + while (*p && *p != '[') + p++; + if (!*p) + return dupString( + "{\"ok\":false,\"error\":\"expected [\"}"); + p++; + while (*p) { + while (*p == ' ' || *p == '\t' || *p == '\n' || + *p == '\r' || *p == ',') + p++; + if (*p == ']') + break; + if (*p != '"') + return dupString( + "{\"ok\":false,\"error\":\"expected string\"}"); + p++; + std::string path; + while (*p && *p != '"') { + if (*p == '\\') { + p++; + switch (*p) { + case '"': + path += '"'; + break; + case '\\': + path += '\\'; + break; + case '/': + path += '/'; + break; + case 'n': + path += '\n'; + break; + case 't': + path += '\t'; + break; + case 'r': + path += '\r'; + break; + case 'b': + path += '\b'; + break; + case 'f': + path += '\f'; + break; + case 'u': { + // Simple pass-through for \uXXXX. + // Buffer layout: '\' 'u' + up to 5 hex + // digits + NUL = 8 bytes. The loop below + // lets i reach 6, then writes the NUL at + // index 7, so the buffer must hold 8. + char unicode_buf[8] = + "\\u"; + int i = 1; + while (*++p && i < 6 && + ((*p >= '0' && + *p <= '9') || + (*p >= 'a' && + *p <= 'f') || + (*p >= 'A' && + *p <= 'F'))) + unicode_buf[++i] = + *p; + unicode_buf[++i] = '\0'; + path += unicode_buf; + p--; // loop will advance p + break; + } + default: + path += '\\'; + path += *p; + break; + } + } else { + path += *p; + } + p++; + } + if (*p != '"') + return dupString( + "{\"ok\":false,\"error\":\"unterminated string\"}"); + p++; + if (!path.empty()) + paths.push_back(path); + } + } + if (paths.empty()) + return dupString( + "{\"ok\":false,\"error\":\"empty file list\"}"); + + // Phase 1: Parse all files in memory (no DB I/O) + struct FileBatch { + std::unique_ptr unit; + std::string source; + std::string language; + std::string file_path; + FileBatch(std::unique_ptr u, + std::string s, std::string l, std::string fp) + : unit(std::move(u)) + , source(std::move(s)) + , language(std::move(l)) + , file_path(std::move(fp)) + { + } + }; + std::vector batches; + std::vector errors; + + for (const auto &fp : paths) { + const char *lang = detectLanguage(fp.c_str()); + if (!lang) { + errors.push_back(fp + ": unsupported"); + continue; + } + + std::string source = readFile(fp.c_str()); + if (source.empty()) { + errors.push_back(fp + ": cannot read"); + continue; + } + + TSTree *tree = ctx->parser->parse(fp.c_str(), + source.c_str(), lang, + source.size()); + if (!tree) { + errors.push_back(fp + ": parse failed"); + continue; + } + + std::unique_ptr translator( + ir::createTranslator(lang)); + if (!translator) { + ts_tree_delete(tree); + errors.push_back(fp + ": no translator"); + continue; + } + + ir::TranslationUnit *unit = translator->translate( + tree, source.c_str(), fp.c_str()); + ts_tree_delete(tree); + if (!unit) { + errors.push_back(fp + ": translation failed"); + continue; + } + + batches.push_back(FileBatch{ + std::unique_ptr(unit), + std::move(source), lang, fp }); + } + + // Phase 2: Persist in single transaction + ctx->store->beginTransaction(); + + uint64_t start_id = 1; + { + // Allocate past BOTH tables: insertEntity is INSERT OR + // IGNORE (drops on collision) while insertGraphNode does a + // bare INSERT on graph_nodes.id (PRIMARY KEY). Sourcing only + // one table lets the other collide and silently lose rows. + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2( + ctx->store->handle(), + "SELECT COALESCE(MAX(t.id),0)+1 FROM (SELECT id " + "FROM entity UNION ALL SELECT id FROM " + "graph_nodes) t", + -1, &stmt, nullptr) == SQLITE_OK) { + if (sqlite3_step(stmt) == SQLITE_ROW) + start_id = static_cast( + sqlite3_column_int64(stmt, 0)); + sqlite3_finalize(stmt); + } + } + + graph::GraphBuilder builder(project_id, start_id); + int total_nodes = 0, total_edges = 0; + + for (auto &b : batches) { + std::string hash = simpleHash(b.source); + ctx->store->upsertFile(project_id, b.file_path.c_str(), + b.language.c_str(), + hash.c_str()); + // Delete graph-layer data only (relation, graph_edges, + // graph_nodes, entity, type_ref, type_info, import, route). + // NOT semantic_records — this batch path re-inserts graph + // data directly from in-memory `b.unit` and never repopulates + // semantic_records. buildGraph later uses semantic_records to + // decide the file rebuild set; wiping it here would make + // subsequent engine_index_project skip rebuilding this file, + // leaving stale graph_nodes forever. + ctx->store->deleteGraphDataByFile(project_id, + b.file_path.c_str()); + + // No ir_nodes/ir_semantic_edges write — graph_nodes is canonical. + // FTS/vector writes skipped for single-file index path. + + auto sg = builder.buildSymbolGraph(b.unit.get()); + auto cg = builder.buildCallGraph(b.unit.get()); + for (auto &gn : sg.nodes) { + ctx->store->insertGraphNode(project_id, gn); + ctx->store->insertEntity(project_id, gn); + total_nodes++; + } + for (auto &e : sg.edges) { + ctx->store->insertGraphEdge(project_id, e); + total_edges++; + } + for (auto &e : cg.edges) { + ctx->store->insertGraphEdge(project_id, e); + total_edges++; + } + + // ComplexityAnalyzer removed (Phase 0) + for (auto &gn : sg.nodes) + if (gn.type == graph::NodeType::Function || + gn.type == graph::NodeType::Method) + for (auto *in : b.unit->all_nodes) + if (in->id == gn.ir_node_id) { + break; + } + } + + ctx->store->commitTransaction(); + + util::JsonWriter w; + w.beginObject(); + w.key("ok").value(true); + w.key("files").value(batches.size() + errors.size()); + w.key("indexed").value(batches.size()); + w.key("nodes").value(total_nodes); + w.key("edges").value(total_edges); + w.key("errors").beginArray(); + for (const auto &err : errors) + w.value(err); + w.endArray(); + w.endObject(); + return dupString(w.str()); + } catch (const std::exception &e) { + ctx->store->rollbackTransaction(); + return dupString(util::errorEnvelope( + "ffi", "engine_index_batch", e.what())); + } catch (...) { + ctx->store->rollbackTransaction(); + return dupString(util::errorEnvelope( + "ffi", "engine_index_batch", "unknown exception")); + } +} + +// ─── Project Metadata ─────────────────────────────────────── + +static const char *detectLicense(const std::string &content) +{ + if (content.find("Apache License") != std::string::npos || + content.find("Version 2.0, January 2004") != std::string::npos) + return "Apache-2.0"; + if (content.find("MIT License") != std::string::npos || + content.find("Permission is hereby granted") != std::string::npos) + return "MIT"; + if (content.find("GNU GENERAL PUBLIC LICENSE") != std::string::npos) + return content.find("Version 3") != std::string::npos ? + "GPL-3.0" : + "GPL-2.0"; + if (content.find("BSD") != std::string::npos) + return "BSD"; + if (content.find("Mozilla Public") != std::string::npos) + return "MPL-2.0"; + return "Unknown"; +} + +char *engine_get_project_info(engine_t handle, uint64_t project_id) +{ + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) + return dupString("{\"error\":\"not initialized\"}"); + + sqlite3 *db = ctx->store->handle(); + std::string name, root; + + { + sqlite3_stmt *stmt = nullptr; + const char *sql = + "SELECT name, root_path FROM projects WHERE id=?"; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64( + stmt, 1, + static_cast(project_id)); + if (sqlite3_step(stmt) == SQLITE_ROW) { + if (sqlite3_column_text(stmt, 0)) + name = reinterpret_cast< + const char *>( + sqlite3_column_text( + stmt, 0)); + if (sqlite3_column_text(stmt, 1)) + root = reinterpret_cast< + const char *>( + sqlite3_column_text( + stmt, 1)); + } + sqlite3_finalize(stmt); + } + } + + // Detect license + std::string license = "Unknown"; + const char *lfs[] = { "LICENSE", "LICENSE.txt", + "LICENSE.md", "LICENSE-APACHE", + "COPYING", nullptr }; + for (int i = 0; lfs[i]; i++) { + std::string c = readFile((root + "/" + lfs[i]).c_str()); + if (!c.empty()) { + license = detectLicense(c); + break; + } + } + + // Primary language + std::string lang; + { + sqlite3_stmt *stmt = nullptr; + const char *sql = + "SELECT language,COUNT(*) FROM files WHERE project_id=? " + "GROUP BY language ORDER BY 2 DESC LIMIT 1"; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64( + stmt, 1, + static_cast(project_id)); + if (sqlite3_step(stmt) == SQLITE_ROW && + sqlite3_column_text(stmt, 0)) + lang = reinterpret_cast( + sqlite3_column_text(stmt, 0)); + sqlite3_finalize(stmt); + } + } + + int file_count = 0, dep_count = 0; + { + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2( + db, + "SELECT COUNT(*) FROM files WHERE project_id=?", + -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64( + stmt, 1, + static_cast(project_id)); + if (sqlite3_step(stmt) == SQLITE_ROW) + file_count = + sqlite3_column_int(stmt, 0); + sqlite3_finalize(stmt); + } + } + + // Try to parse dep files + const char *dfs[] = { "go.mod", "Cargo.toml", + "pyproject.toml", "package.json", + "requirements.txt", nullptr }; + for (int i = 0; dfs[i]; i++) { + std::string c = readFile((root + "/" + dfs[i]).c_str()); + if (!c.empty()) { + int lines = 0; + for (size_t p = 0; + (p = c.find('\n', p)) != std::string::npos; + lines++, p++) + ; + dep_count = lines / 3; + break; + } + } + + util::JsonWriter w; + w.beginObject(); + w.key("name").value(name); + w.key("license").value(license); + w.key("language").value(lang); + w.key("file_count").value(file_count); + w.key("dependency_count").value(dep_count); + w.endObject(); + return dupString(w.str()); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_project_info", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_project_info", "unknown exception")); + } +} + +// ─── Parse-failure Maintenance ─────────────────────────────── +// parse_failures is written by the indexing paths and drives the fail-fast +// skip. Nothing else exported it, so the table could be written and never read +// or cleared from the CLI — the `codescope parse-failures` and +// `codescope reset-failures` subcommands were documented but had no entry +// point, and store::getParseFailuresJson / store::resetParseFailures were +// unreachable. These two exports are that entry point. + +/// Read the project's parse_failures rows as JSON. +/// \param project_id Project whose rows are read. +/// \param limit Maximum rows (<= 0 means the store default of 100). +/// \return JSON `{"ok":true,"parse_failures":[...]}`; on failure an +/// `{"ok":false,"error":"...","module":...,"method":...}` envelope. +/// The caller frees it with engine_free_string(). +char *engine_get_parse_failures(engine_t handle, uint64_t project_id, int limit) +{ + EngineContext *ctx = engineInstance(handle); + + try { + if (!ctx || !ctx->store) + return dupString( + "{\"ok\":false,\"error\":\"[module=ffi, method=engine_get_parse_failures] engine not initialized\"}"); + const std::string rows = store::getParseFailuresJson( + ctx, project_id, limit > 0 ? limit : 100); + return dupString("{\"ok\":true,\"parse_failures\":" + rows + + "}"); + } catch (const std::exception &e) { + return dupString(util::okFalseEnvelope( + "ffi", "engine_get_parse_failures", e.what())); + } catch (...) { + return dupString(util::okFalseEnvelope( + "ffi", "engine_get_parse_failures", + "unknown exception")); + } +} + +/// Delete every parse_failures row of the project. +/// \param project_id Project whose rows are cleared. +/// \return JSON `{"ok":true,"removed":N}` (N may be 0); on failure an +/// `{"ok":false,"error":"..."}` envelope. The caller frees it with +/// engine_free_string(). +char *engine_reset_parse_failures(engine_t handle, uint64_t project_id) +{ + EngineContext *ctx = engineInstance(handle); + + try { + if (!ctx || !ctx->store) + return dupString( + "{\"ok\":false,\"error\":\"[module=ffi, method=engine_reset_parse_failures] engine not initialized\"}"); + const int removed = store::resetParseFailures(ctx, project_id); + if (removed < 0) + return dupString( + "{\"ok\":false,\"error\":\"[module=ffi, method=engine_reset_parse_failures] delete failed\"}"); + util::JsonWriter w; + w.beginObject(); + w.key("ok").value(true); + w.key("removed").value(removed); + w.endObject(); + return dupString(w.str()); + } catch (const std::exception &e) { + return dupString(util::okFalseEnvelope( + "ffi", "engine_reset_parse_failures", e.what())); + } catch (...) { + return dupString(util::okFalseEnvelope( + "ffi", "engine_reset_parse_failures", + "unknown exception")); + } +} diff --git a/engine/src/engine_filter_ffi.cpp b/engine/src/engine_filter_ffi.cpp new file mode 100644 index 0000000..1b89fe5 --- /dev/null +++ b/engine/src/engine_filter_ffi.cpp @@ -0,0 +1,100 @@ +// engine_filter_ffi.cpp — the filter decisions the server needs, answered by +// the SAME FilterPolicy the indexer uses. +// +// Why they are exported instead of reimplemented in Rust: the server's module +// discovery (`server/src/discover.rs`) needs to know which directories and +// files the worker will actually index, so the scheduler's allocation and the +// quarantine file list match reality. It used to keep hand-written copies of +// the C++ skip lists and of the source-extension list, and those copies drifted +// twice in one review: +// +// * a `.gitignore`d TOP-LEVEL directory (this repository's `build-*`) was +// still reported as a module, because the copy reads no ignore file at all +// — the worker then indexed zero files for it; +// * `.zig` was counted as a source extension although the engine has no Zig +// support and none is planned: no extension mapping in detectLanguage(), +// no translator. Measured on OmniScope, 362 `.zig` files were counted as +// source while the index contained zero zig entities — the count was +// advertising a capability the engine does not have. (Ruby, which the same +// list contains, IS recognized by the engine and stays countable: the fix +// is to follow the engine, not to shorten a list.) +// +// Both questions now have exactly one implementation — this file delegates to +// FilterPolicy — so a skip rule or a supported extension is added once. + +#include "filter_policy.h" + +#include +#include +#include +#include + +namespace +{ +// One policy per project root: constructing a FilterPolicy fills a dozen skip +// sets from ~200 literals, and loading its ignore files touches the disk, so +// the per-path queries below must not repeat that work. The map is keyed by +// root because .gitignore/.codescopeignore are per project; the queries +// themselves are read-only. Guarded by a mutex: the server calls these from +// its walker. +std::mutex g_policy_mutex; +std::unordered_map> g_policies; + +FilterPolicy &policyFor(const std::string &project_root) +{ + std::lock_guard lock(g_policy_mutex); + auto it = g_policies.find(project_root); + if (it == g_policies.end()) { + auto policy = std::make_unique(); + if (!project_root.empty()) { + // Same order as engine_index_project: the scan directory's own + // rules first (they take precedence), project root's second. + policy->loadIgnoreFile(project_root); + policy->loadGitignore(project_root); + } + policy->loadExcludeEnv(); + // Same language context the indexer's own discovery establishes: the + // test/docs/samples skip names are only relaxed for Java, and without + // knowing the project is Java this policy skipped them at ANY depth, so + // a Maven tree (`src/main/java/org/.../samples/petclinic/`) reported no + // modules at all while `codescope discover` counted every file. + applyProjectLanguageContext(*policy, project_root); + it = g_policies.emplace(project_root, std::move(policy)).first; + } + return *it->second; +} +} // namespace + +extern "C" int engine_path_is_skipped(const char *project_root, + const char *rel_path, int is_dir) +{ + // A filter query must never take the caller down: on any failure the + // answer is "do not skip", which is the behaviour the callers had before + // this function existed. + if (!rel_path || !*rel_path) + return 0; + try { + return policyFor(project_root ? project_root : "") + .shouldSkipEntry(rel_path, is_dir != 0) ? + 1 : + 0; + } catch (...) { + return 0; + } +} + +extern "C" int engine_is_indexable_source(const char *file_path) +{ + // "Can the engine parse this file?" — asked by extension (and shebang for + // extensionless scripts), NOT "is it in someone's allow-list". This is + // what keeps the server's file count from advertising languages the + // engine cannot read. + if (!file_path || !*file_path) + return 0; + try { + return policyFor("").detectLanguage(file_path) != nullptr ? 1 : + 0; + } catch (...) { + return 0; + } +} diff --git a/engine/src/engine_helpers.cpp b/engine/src/engine_helpers.cpp index f838e4b..2bb7884 100644 --- a/engine/src/engine_helpers.cpp +++ b/engine/src/engine_helpers.cpp @@ -1,6 +1,7 @@ #include "engine_internal.h" #include "filter_policy.h" #include "platform_win.h" +#include "util/json_writer.h" #include #include @@ -92,43 +93,6 @@ std::string fileContentHash(const char *path) } // Escape a string for safe embedding in JSON (RFC 8259) -std::string jsonEscape(const std::string &s) -{ - std::string out; - out.reserve(s.size() + 4); - for (char c : s) { - switch (c) { - case '"': - out += "\\\""; - break; - case '\\': - out += "\\\\"; - break; - case '\n': - out += "\\n"; - break; - case '\r': - out += "\\r"; - break; - case '\t': - out += "\\t"; - break; - default: - // Escape control characters (0x00-0x1f) as \uXXXX - if (static_cast(c) < 0x20) { - char buf[8]; - snprintf(buf, sizeof(buf), "\\u%04x", - static_cast(c)); - out += buf; - } else { - out += c; - } - break; - } - } - return out; -} - std::string simpleHash(const std::string &s) { // Simple djb2 hash for content fingerprint diff --git a/engine/src/engine_index.cpp b/engine/src/engine_index.cpp index 7ec8c0c..30becbd 100644 --- a/engine/src/engine_index.cpp +++ b/engine/src/engine_index.cpp @@ -1,4 +1,6 @@ +#include "util/json_writer.h" #include "engine_internal.h" +#include "engine_index_paths.h" #include "filter_policy.h" #include "platform_win.h" #include "async_knowledge.h" @@ -31,16 +33,27 @@ // ─── Constants ───────────────────────────────────────────────── // ─── Index File ──────────────────────────────────────────────── -char *engine_index_file(uint64_t project_id, const char *file_path) +char *engine_index_file(engine_t handle, uint64_t project_id, + const char *file_path) { + EngineContext *ctx = engineInstance(handle); + try { if (!file_path || !*file_path) return dupString( "{\"error\":\"[module=ffi, method=engine_index_file] file_path is required\"}"); - if (!g_store || !g_parser) + if (!ctx || !ctx->store || !ctx->parser) return dupString( "{\"ok\":false,\"error\":\"engine not initialized\"}"); + // Serialize with the background enrichment thread: this path + // writes semantic_records + graph and then launches the builder, + // so it must join any running builder first (the builder holds + // g_store_mutex) and then hold the store guard for the writes. + // Order matters — guarding first would deadlock the join. + joinAsyncKnowledgeBuilder(); + auto _store_guard = waitForKnowledgeBuilder(); + const char *language = detectLanguage(file_path); if (!language) return dupString( @@ -63,12 +76,22 @@ char *engine_index_file(uint64_t project_id, const char *file_path) fsize = static_cast(file_stat_buf.st_size); } + // Store the spelling the index already uses for this file (see + // engine_index_paths.h): callers pass an absolute path, while the + // walk-based index may have stored `./src/a.go`, and writing the + // absolute form verbatim gave the same file a second identity. + // `file_path` stays the real path for the file-system work below (the + // LSP block opens it). + const std::string stored_path = indexSpellingFor( + ctx->store.get(), project_id, file_path); + const char *store_path = stored_path.c_str(); + // Parse - TSTree *tree = g_parser->parse(file_path, source.c_str(), - language, source.size()); + TSTree *tree = ctx->parser->parse(store_path, source.c_str(), + language, source.size()); if (!tree) { return dupString("{\"ok\":false,\"error\":\"" + - g_parser->error() + "\"}"); + ctx->parser->error() + "\"}"); } // ── Build IR: visitor (rust/java) or translator (others) ── @@ -95,7 +118,7 @@ char *engine_index_file(uint64_t project_id, const char *file_path) std::unique_ptr unit_guard; auto visitor = ir::createJsVisitor(language); if (visitor) { - su = visitor->visit(tree, source.c_str(), file_path); + su = visitor->visit(tree, source.c_str(), store_path); su_guard.reset(su); } if (!su) { @@ -107,7 +130,7 @@ char *engine_index_file(uint64_t project_id, const char *file_path) "for language\"}"); } unit_raw = translator->translate(tree, source.c_str(), - file_path); + store_path); unit_guard.reset(unit_raw); } ts_tree_delete(tree); @@ -119,7 +142,7 @@ char *engine_index_file(uint64_t project_id, const char *file_path) // ── Convert IR → store::FileResult (flat records) ────── store::FileResult fr; - fr.file_path = file_path; + fr.file_path = store_path; fr.language = language; fr.mtime = mtime; fr.fsize = fsize; @@ -240,7 +263,7 @@ char *engine_index_file(uint64_t project_id, const char *file_path) rec.loc.start_col = n->loc.start_col; rec.loc.end_row = n->loc.end_row; rec.loc.end_col = n->loc.end_col; - rec.file_path = file_path; + rec.file_path = store_path; fr.records.push_back(std::move(rec)); for (auto *c : n->children) flatten(c, my_id); @@ -256,19 +279,20 @@ char *engine_index_file(uint64_t project_id, const char *file_path) // deletes the file's old graph/entity data before rebuilding from // those records. The pair is idempotent, so re-indexing a file // never accumulates duplicates. - g_store->beginTransaction(); + ctx->store->beginTransaction(); std::string hash = simpleHash(source); - g_store->upsertFile(project_id, file_path, language, - hash.c_str()); - if (!g_store->insertFileResultBatch( + ctx->store->upsertFile(project_id, store_path, language, + hash.c_str()); + if (!ctx || + !ctx->store->insertFileResultBatch( project_id, std::vector{ fr })) { - g_store->rollbackTransaction(); + ctx->store->rollbackTransaction(); return dupString( "{\"ok\":false,\"error\":\"insertFileResultBatch " "failed: " + - g_store->error() + "\"}"); + ctx->store->error() + "\"}"); } - g_store->commitTransaction(); + ctx->store->commitTransaction(); // Rebuild graph for THIS file only — wrap in its own transaction // exactly like engine_index_project. buildGraph uses a SAVEPOINT @@ -276,15 +300,16 @@ char *engine_index_file(uint64_t project_id, const char *file_path) // without it the SAVEPOINT leaks an implicit transaction that // breaks the async knowledge builder. { - g_store->beginTransaction(); + ctx->store->beginTransaction(); std::unordered_set changed{ std::string( - file_path) }; - if (!g_store->buildGraph(project_id, true, &changed)) { - g_store->rollbackTransaction(); + store_path) }; + if (!ctx || !ctx->store->buildGraph(project_id, true, + &changed)) { + ctx->store->rollbackTransaction(); return dupString( "{\"ok\":false,\"error\":\"buildGraph " "failed: " + - g_store->error() + "\"}"); + ctx->store->error() + "\"}"); } // Indexing now builds the full call graph (buildGraph above), // so mark every node callgraph_ready. This makes trace_path and @@ -303,9 +328,9 @@ char *engine_index_file(uint64_t project_id, const char *file_path) "UPDATE graph_nodes SET callgraph_ready=1 " "WHERE project_id=" + std::to_string(project_id); - if (!g_store->exec(up.c_str())) { - std::string err = g_store->error(); - g_store->rollbackTransaction(); + if (!ctx || !ctx->store->exec(up.c_str())) { + std::string err = ctx->store->error(); + ctx->store->rollbackTransaction(); fprintf(stderr, "engine_index_file: callgraph_ready UPDATE " "failed: %s " @@ -316,25 +341,27 @@ char *engine_index_file(uint64_t project_id, const char *file_path) "UPDATE failed: " + err + "\"}"); } - g_store->commitTransaction(); + ctx->store->commitTransaction(); } // Async knowledge builder (modules/role/summary) — keeps the // knowledge layer consistent with engine_index_project so MCP // tools return complete results after a single-file re-index. - launchAsyncKnowledgeBuilder(project_id, true); + launchAsyncKnowledgeBuilder(ctx, project_id, true); - // Report the actual persisted graph node/edge counts for the file. + // Report the actual persisted counts for the file, from the canonical + // tables the pipeline writes. Counting `graph_nodes`/`graph_edges` + // (which only the legacy engine_index_batch path populates) reported + // `nodes:0, edges:0` for a file that had just been indexed. auto countRows = [&](const char *table) -> int64_t { - // graph_edges has no file_path column; count its rows by - // joining through graph_nodes (which does). + // `relation` has no file_path column; count its rows through the + // source entity, which does. std::string sql; - if (std::string(table) == "graph_edges") { - sql = "SELECT COUNT(*) FROM graph_edges ge " - "JOIN graph_nodes gn ON ge.source_node_id " - "= gn.id " - "WHERE gn.project_id = ? AND " - "gn.file_path = ?"; + if (std::string(table) == "relation") { + sql = "SELECT COUNT(*) FROM relation r " + "JOIN entity e ON e.id = r.source_id " + "WHERE r.project_id = ? AND " + "e.file_path = ?"; } else { sql = std::string("SELECT COUNT(*) FROM ") + table + @@ -342,40 +369,41 @@ char *engine_index_file(uint64_t project_id, const char *file_path) } sqlite3_stmt *st = nullptr; int64_t n = 0; - if (sqlite3_prepare_v2(g_store->handle(), sql.c_str(), - -1, &st, nullptr) != SQLITE_OK) { + if (sqlite3_prepare_v2(ctx->store->handle(), + sql.c_str(), -1, &st, + nullptr) != SQLITE_OK) { fprintf(stderr, "engine_index_file: countRows prepare " "failed for %s: %s " "[module=ffi, method=engine_index_file]\n", table, - sqlite3_errmsg(g_store->handle())); + sqlite3_errmsg(ctx->store->handle())); return 0; } sqlite3_bind_int64(st, 1, static_cast(project_id)); - sqlite3_bind_text(st, 2, file_path, -1, SQLITE_STATIC); + sqlite3_bind_text(st, 2, store_path, -1, SQLITE_STATIC); if (sqlite3_step(st) == SQLITE_ROW) n = sqlite3_column_int64(st, 0); sqlite3_finalize(st); return n; }; - int64_t node_count = countRows("graph_nodes"); - int64_t edge_count = countRows("graph_edges"); + int64_t node_count = countRows("entity"); + int64_t edge_count = countRows("relation"); std::ostringstream result_os; result_os << "{\"ok\":true,\"nodes\":" << node_count << ",\"edges\":" << edge_count << "}"; return dupString(result_os.str()); } catch (const std::exception &e) { - g_store->rollbackTransaction(); + ctx->store->rollbackTransaction(); return dupString( std::string( "{\"error\":\"[module=ffi, method=engine_index_file] ") + e.what() + "\"}"); } catch (...) { - g_store->rollbackTransaction(); - return dupString( - "{\"error\":\"[module=ffi, method=engine_index_file] unknown exception\"}"); + ctx->store->rollbackTransaction(); + return dupString(util::errorEnvelope("ffi", "engine_index_file", + "unknown exception")); } } diff --git a/engine/src/engine_index_discover.cpp b/engine/src/engine_index_discover.cpp index 9b3299c..058ee1b 100644 --- a/engine/src/engine_index_discover.cpp +++ b/engine/src/engine_index_discover.cpp @@ -1,3 +1,4 @@ +#include "util/json_writer.h" #include "engine_index_discover.h" #include @@ -13,44 +14,174 @@ namespace engine_index_discover { +// ── README / knowledge documents ──────────────────────────────────────── +// +// One implementation for every caller: the discovery walk below and the +// enhance pass (engine_enhance_project) both route through these helpers, so +// the accepted names, the replace-not-append rule and the log tags cannot +// drift apart between the paths that ingest READMEs. + +namespace +{ + +/// README names the engine ingests, lower-cased. Kept in one place so the walk +/// and the project-root lookup cannot disagree (code_rules §5: no duplicated +/// literals). +const char *const kReadmeNames[] = { "readme.md", "readme.markdown", "readme" }; + +/// @param name File name to normalise. +/// @return The name lower-cased (ASCII; file names here are ASCII in practice). +std::string toLowerName(const std::string &name) +{ + std::string lower = name; + for (auto &c : lower) + c = static_cast( + std::tolower(static_cast(c))); + return lower; +} + +/// @param file_name Base name without any directory part. +/// @return true when it is one of the README names the engine ingests. +bool isReadmeName(const std::string &file_name) +{ + const std::string lower = toLowerName(file_name); + for (const char *candidate : kReadmeNames) { + if (lower == candidate) + return true; + } + return false; +} + +/// @param path Absolute path. +/// @return The part after the last path separator. +std::string baseName(const std::string &path) +{ + const size_t slash = path.find_last_of("/\\"); + return slash == std::string::npos ? path : path.substr(slash + 1); +} + +/// @param path Absolute path. +/// @return The directory containing `path`, or an empty string when it has none. +std::string parentDir(const std::string &path) +{ + const size_t slash = path.find_last_of("/\\"); + return slash == std::string::npos ? std::string() : + path.substr(0, slash); +} + +/// @param content File body. +/// @return 1-based line count, which insertDocument records as `end_line`. +int countLines(const std::string &content) +{ + int lines = 1; + for (char c : content) + if (c == '\n') + ++lines; + return lines; +} + +} // namespace + +bool ingestReadmeDocument(EngineContext *ctx, uint64_t project_id, + const std::string &readme_path) +{ + if (!ctx || !ctx->store) + return false; + + const std::string content = readFile(readme_path.c_str()); + if (content.empty()) + return false; + + // type 0 (kDocumentTypeReadme) signals the knowledge layer to parse + // capabilities out of the content. + const int doc_type = 0; + // Replace, never append: the drift tools concatenate every README row of a + // project, so a duplicate would silently double the weight of its claims. + if (!ctx->store->deleteDocument(project_id, doc_type, readme_path)) { + fprintf(stderr, + "engine: deleteDocument failed for %s: %s " + "[module=engine, method=ingestReadmeDocument]\n", + readme_path.c_str(), ctx->store->error().c_str()); + return false; + } + if (!ctx->store->insertDocument(project_id, doc_type, readme_path, + content, 1, countLines(content))) { + fprintf(stderr, + "engine: insertDocument failed for %s: %s " + "[module=engine, method=ingestReadmeDocument]\n", + readme_path.c_str(), ctx->store->error().c_str()); + return false; + } + return true; +} + +int ingestProjectRootReadme(EngineContext *ctx, uint64_t project_id) +{ + if (!ctx || !ctx->store) + return -1; + + const std::string root = ctx->store->getProjectRootPath(project_id); + if (root.empty()) { + fprintf(stderr, + "engine: project %llu has no root path; cannot ingest its README " + "[module=engine, method=ingestProjectRootReadme]\n", + (unsigned long long)project_id); + return -1; + } + + // List the root instead of guessing name casing: a directory listing with a + // case-insensitive comparison also catches README.MD / Readme.md, which a + // case-sensitive probe would miss on Linux. + std::error_code ec; + std::filesystem::directory_iterator it( + root, + std::filesystem::directory_options::skip_permission_denied, ec); + if (ec) { + fprintf(stderr, + "engine: cannot list project root %s: %s " + "[module=engine, method=ingestProjectRootReadme]\n", + root.c_str(), ec.message().c_str()); + return -1; + } + const std::filesystem::directory_iterator end; + for (; it != end; it.increment(ec)) { + if (ec) + break; + std::error_code entry_ec; + if (!it->is_regular_file(entry_ec)) + continue; + const std::string name = it->path().filename().string(); + if (!isReadmeName(name)) + continue; + return ingestReadmeDocument(ctx, project_id, + it->path().string()) ? + 1 : + -1; + } + return 0; +} + // Walk `dir` and collect candidate source files, applying the same // FilterPolicy rules as the scanner (skip dirs, gitignore, // .codescopeignore, bundle suffixes, filename/suffix skips, language // filter). Also ingests the project-root README as a knowledge // document and runs the incremental scan-state gate. -int collectFileJobs(uint64_t project_id, const std::string &dir, - FilterPolicy &filter, +int collectFileJobs(EngineContext *ctx, uint64_t project_id, + const std::string &dir, FilterPolicy &filter, const std::unordered_set &scan_state, std::vector &jobs, bool &is_reindex, std::string &err_json) { - // Pre-detect Java projects BEFORE the directory walk. The FilterPolicy - // Java carve-out defers test/docs/example/samples/... dirs to a - // top-only check ONLY when lang_context_ == "java", but lang_context_ - // previously flipped only upon seeing the FIRST .java file during the - // walk — and that file may itself live under an example/samples/... - // dir which is skipped at any depth while lang_context_ is still - // empty. That chicken-and-egg made Java projects with such package - // dirs index 0 files (e.g. spring-petclinic's - // org/springframework/samples/petclinic). Fix: cheap recursive scan - // for any *.java before the main walk and flip lang_context_ early. - { - std::error_code ec; - auto pit = std::filesystem::recursive_directory_iterator( - dir, - std::filesystem::directory_options::skip_permission_denied, - ec); - std::filesystem::recursive_directory_iterator pend; - while (!ec && pit != pend) { - const auto &pent = *pit; - if (pent.is_regular_file() && - pent.path().extension() == ".java") { - filter.setLangContext("java"); - break; - } - pit.increment(ec); - } - } + // Pre-detect Java projects BEFORE the directory walk: the Java carve-out + // defers test/docs/example/samples/... dirs to a top-only check, but the + // file that reveals the project is Java may itself live under such a + // directory and is skipped while the language is still unset — the + // chicken-and-egg that made Java projects with package dirs of those names + // index 0 files (spring-petclinic's org/springframework/samples/petclinic). + // The scan is shared with the server's module discovery, which queries the + // same policy through engine_path_is_skipped and used to lack the context + // entirely (see filter_policy.h). + applyProjectLanguageContext(filter, dir); try { // P0-2: standalone discovery timing. Previously this phase only @@ -62,7 +193,22 @@ int collectFileJobs(uint64_t project_id, const std::string &dir, auto it = std::filesystem::recursive_directory_iterator( dir, std::filesystem::directory_options:: skip_permission_denied); - for (auto &entry : it) { + // An explicit iterator loop, NOT `for (auto &entry : it)`. + // + // A range-for over an iterator copies it: `begin()` returns the + // iterator by value, the loop advances that copy, and the pruning + // call below would then operate on the original object that nobody + // is iterating. The result was measured, not assumed: with + // `.gitignore` holding `build-x/`, the walk reported + // `skipped_dirs=4` and STILL descended into all four levels, + // because `disable_recursion_pending()` was applied to a copy. + // Directory pruning was therefore cosmetic for every entry point + // that relies on it — the hard-skip names and `.gitignore` rules + // both counted their matches and then walked in anyway. + const auto it_end = + std::filesystem::recursive_directory_iterator(); + for (; it != it_end; ++it) { + const auto &entry = *it; // seen_dirs counts ONLY directory entries — recursive_ // directory_iterator yields files too, so counting every // entry here inflated the metric with file visits. JSON @@ -77,75 +223,25 @@ int collectFileJobs(uint64_t project_id, const std::string &dir, rel.clear(); // ── README / document ingestion (BEFORE skip filter) ── - // .md files are in skip_suffixes_ (filter_policy.cpp:422) - // so they never reach the source-code indexing path. - // But the knowledge layer (CapabilityPlugin, - // ContractPlugin) needs README content in the - // document table to extract capabilities/contracts. - // Therefore we intercept README.md here — BEFORE - // shouldSkipEntry() drops it — and ingest it via - // insertDocument(). + // .md files are in skip_suffixes_ (filter_policy.cpp:422) so they + // never reach the source-code indexing path, but the knowledge + // layer (CapabilityPlugin, ContractPlugin) and the drift tools need + // README content in the document table. Intercept the README at the + // root of the directory being scanned — BEFORE shouldSkipEntry() + // drops it — and ingest it through the shared helper. // - // Only the project-root README is ingested as a - // knowledge document; nested READMEs are ignored - // to avoid noise from vendored deps. + // A README at a nested directory's root is picked up when that + // directory is scanned; the project root itself is additionally + // covered by engine_enhance_project (ingestProjectRootReadme), which + // every index path runs. if (entry.is_regular_file()) { - const std::string &fp = entry.path().string(); - std::string fname = fp; - size_t sl = fp.find_last_of("/\\"); - if (sl != std::string::npos) - fname = fp.substr(sl + 1); - // Case-insensitive README.md match - std::string fname_lower = fname; - for (auto &c : fname_lower) - c = static_cast(std::tolower(c)); - - // Check if this README is at project root - bool is_root_readme = false; - if (fname_lower == "readme.md" || - fname_lower == "readme.markdown" || - fname_lower == "readme") { - // Project-root README: its parent dir == dir - std::string parent = fp; - size_t ps = parent.find_last_of("/\\"); - parent = (ps != std::string::npos) ? - parent.substr(0, ps) : - ""; - is_root_readme = (parent == dir); - } - - if (is_root_readme) { - // Ingest README content into document table. - // type=0 (kDocumentTypeReadme) signals the - // knowledge layer to parse capabilities. - std::string content = - readFile(fp.c_str()); - if (!content.empty()) { - int doc_type = - 0; // kDocumentTypeReadme - // insertDocument's 5th/6th params are - // start_line / end_line (1-based line - // numbers), NOT byte offsets. Count - // newlines to compute the line range. - int line_count = 1; - for (char c : content) - if (c == '\n') - ++line_count; - if (!g_store->insertDocument( - project_id, - doc_type, fp, - content, 1, - line_count)) { - fprintf(stderr, - "engine: insertDocument failed for %s: %s " - "[module=engine, method=collectFileJobs]\n", - fp.c_str(), - g_store->error() - .c_str()); - } - } - // README is ingested as a document, NOT as - // source code — skip the rest of the loop. + const std::string readme_path = + entry.path().string(); + if (isReadmeName(baseName(readme_path)) && + parentDir(readme_path) == dir) { + ingestReadmeDocument(ctx, project_id, + readme_path); + // A README is a knowledge document, never source code. continue; } } @@ -266,9 +362,12 @@ int collectFileJobs(uint64_t project_id, const std::string &dir, jobs.size()); } catch (const std::exception &e) { std::ostringstream err; - err << "{\"ok\":false,\"error\":\"scan error: " - << jsonEscape(e.what()) << "\"}"; - err_json = err.str(); + // The helper adds the [module=…, method=…] trace chain the hand-built + // envelope was missing (plan/rules/code_rules.md: every error must be + // traceable to a module and a method). + err_json = util::okFalseEnvelope("ffi", "engine_index_discover", + std::string("scan error: ") + + e.what()); return -1; } return 0; diff --git a/engine/src/engine_index_discover.h b/engine/src/engine_index_discover.h index f6b9250..4b68d5a 100644 --- a/engine/src/engine_index_discover.h +++ b/engine/src/engine_index_discover.h @@ -6,6 +6,7 @@ #include #include +#include "engine_context.h" #include "filter_policy.h" namespace engine_index_discover @@ -36,12 +37,50 @@ struct FileJob { // @param err_json [out] JSON error payload when returning -1. // @return 0 on success (jobs populated), -1 on scan error (err_json set). // @throws nothing — filesystem exceptions are caught internally. -int collectFileJobs(uint64_t project_id, const std::string &dir, - FilterPolicy &filter, +int collectFileJobs(EngineContext *ctx, uint64_t project_id, + const std::string &dir, FilterPolicy &filter, const std::unordered_set &scan_state, std::vector &jobs, bool &is_reindex, std::string &err_json); +// Ingest one README file into the `document` table as a knowledge document +// (type 0), replacing any previous row for the same path. +// +// The knowledge layer (CapabilityPlugin, ContractPlugin) and the drift tools +// read language claims out of README content, so the row must exist for those +// tools to be able to answer at all. `.md` files are in the FilterPolicy's +// skip list, which is why ingestion happens before the skip check rather than +// through the source-code path. +// +// Idempotent on purpose: the drift tools concatenate every README row of a +// project, so a duplicate would silently double the weight of its claims. +// +// @param ctx Engine instance; its store receives the row. Must not be +// null and must have an open store. +// @param project_id Project the document belongs to. +// @param readme_path Absolute path of the README file. +// @return true when the row was (re)written; false when the file had no +// content or the store rejected it (logged here, with the store error). +bool ingestReadmeDocument(EngineContext *ctx, uint64_t project_id, + const std::string &readme_path); + +// Ingest the project's root README, if it has one. +// +// Every index path must end with the same documentation state, but ingestion +// used to live only in the discovery walk and only for the directories it +// scanned: the parallel scheduler indexes module directories, so a project +// whose root holds no source files never had its root README ingested +// (six of ten real projects checked had zero documents while the file was +// present), and the file-list path (force-index, force_index_files) ingested +// none at all. Enhancing is the one step every path runs last, so the root +// README is ingested there, before the knowledge layer reads it. +// +// @param ctx Engine instance; its store must be open. +// @param project_id Project whose root README should be ingested. +// @return 1 when a README was ingested, 0 when the project has none, +// -1 when the project row could not be read (logged). +int ingestProjectRootReadme(EngineContext *ctx, uint64_t project_id); + } // namespace engine_index_discover #endif // ENGINE_INDEX_DISCOVER_H diff --git a/engine/src/engine_index_files.cpp b/engine/src/engine_index_files.cpp index 077b93d..c2699ca 100644 --- a/engine/src/engine_index_files.cpp +++ b/engine/src/engine_index_files.cpp @@ -1,4 +1,6 @@ +#include "util/json_writer.h" #include "engine_internal.h" +#include "engine_index_paths.h" #include "filter_policy.h" #include "platform_win.h" @@ -34,6 +36,7 @@ #include "ir/translators/js_visitor.h" #include "engine_index_metrics.h" +#include "engine_index_sched.h" #include "async_knowledge.h" #include "store/store_parse_failure.h" @@ -49,12 +52,31 @@ constexpr uint64_t kMaxFileSize = 5 * 1024 * 1024; // 5 MB default // --file-list path). Split out of engine_index_project.cpp into its own // translation unit so each file stays under the 1000-line rule // (plan/rules/code_rules.md §1). -char *engine_index_files(uint64_t project_id, const char *file_list_json) +// +// Two kinds of callers share this entry point: +// * the scheduler-driven worker paths (worker --file-list, chunk worker), +// which mirror the automatic project index and honour the fail-fast skip; +// * the `force_index_files` tool, whose documented contract is "index these +// paths regardless of the default skip rules" and therefore bypasses it. +// The caller states which policy applies through `bypass_fail_fast` (see +// engine.h) rather than this file guessing from an environment variable. +/// Body of engine_index_files. Kept as a separate function so the extern "C" +/// entry point below can stay a thin try/catch wrapper: no C++ exception may +/// cross the C ABI boundary (the MCP server is long-running, so an escaping +/// exception would terminate it). +static char *indexFilesImpl(EngineContext *ctx, uint64_t project_id, + const char *file_list_json, int bypass_fail_fast) { - if (!g_store) + if (!ctx || !ctx->store) return dupString( "{\"ok\":false,\"error\":\"engine not initialized\"}"); + // See engine_index_project: the background enrichment thread shares + // this connection, so it must not run while we write. Join first (the + // builder holds g_store_mutex), then hold the store guard for the write. + joinAsyncKnowledgeBuilder(); + auto _store_guard = waitForKnowledgeBuilder(); + if (!file_list_json || !file_list_json[0]) return dupString( "{\"ok\":false,\"error\":\"file_list_json is empty\"}"); @@ -78,6 +100,10 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) // stat). Without it, result->mtime defaults to 0 and a later // incremental run's isFileUnchanged can never skip (M-13). int64_t mtime = 0; + // The path as the caller gave it, used to OPEN the file (it must work + // regardless of the current directory). `path` above is the spelling + // the index stores — see existingSpellingFor. + std::string abs_path; }; std::vector jobs; std::string json(file_list_json); @@ -115,27 +141,41 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) filter.setLangContext("java"); } - jobs.push_back({ path, lang, - static_cast(file_stat.st_size), - static_cast(file_stat.st_mtime) }); + // Reuse the spelling this file is already stored under, so an + // already-indexed file is not given a second identity. + const std::string stored = + indexSpellingFor(ctx->store.get(), project_id, path); + jobs.push_back( + { stored, lang, static_cast(file_stat.st_size), + static_cast(file_stat.st_mtime), path }); } if (jobs.empty()) return dupString( "{\"ok\":true,\"files_indexed\":0,\"nodes\":0,\"edges\":0,\"errors\":0}"); - // Fail-fast: pre-load known parse failures so the parse loop can - // skip them without per-file DB queries. Mirrors the logic in - // engine_index_project. - const int kFailRetryMax = [] { - const char *e = getenv("CODESCOPE_FAIL_RETRY_MAX"); - return e ? std::max(1, std::atoi(e)) : 3; - }(); + // Fail-fast skip, applied exactly like the automatic project path + // (engine_index_project loads the same set): a file whose parse has + // failed >= CODESCOPE_FAIL_RETRY_MAX times is dropped without a re-parse. + // The threshold comes from the shared constant — this file used to + // hard-code 3, three times the documented default. + // + // force_index_files passes bypass_fail_fast = 1 and gets an EMPTY set: + // its documented contract is "index these paths regardless of the default + // skip rules", so silently dropping a file the user explicitly named + // contradicts both that contract and plan/rules/code_rules.md (no silent + // handling). A forced file is always re-attempted; a genuinely broken one + // costs one parse per call and re-records its failure. std::unordered_set known_failures; - { + if (!bypass_fail_fast) { + const int kFailRetryMax = [] { + const char *e = getenv("CODESCOPE_FAIL_RETRY_MAX"); + return e ? std::max(1, std::atoi(e)) : + engine_index_sched::kDefaultFailRetryMax; + }(); std::vector fail_vec; - if (!store::loadKnownParseFailures(project_id, kFailRetryMax, - /*out*/ fail_vec)) { + if (!store::loadKnownParseFailures(ctx, project_id, + kFailRetryMax, fail_vec)) { fprintf(stderr, "engine: loadKnownParseFailures failed " "(continuing) " @@ -171,7 +211,7 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) for (auto &j : jobs) langs.insert(j.lang); for (auto &l : langs) - lang_ptrs[l] = g_parser->getLanguage(l.c_str()); + lang_ptrs[l] = ctx->parser->getLanguage(l.c_str()); } // ── Streaming Pipeline ───────────────────────────────────── @@ -195,7 +235,7 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) // ── Writer thread ────────────────────────────────────────── std::thread writer_thread([&]() { - g_store->beginTransaction(); + ctx->store->beginTransaction(); std::vector batch; batch.reserve(kWriterBatchSize); while (true) { @@ -203,7 +243,7 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) bool ok = result_queue.pop(fr); if (!ok) { if (!batch.empty()) { - if (!g_store->insertFileResultBatch( + if (!ctx->store->insertFileResultBatch( project_id, batch)) { writer_error = 1; } @@ -223,8 +263,8 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) } if (batch.size() >= kWriterBatchSize || result_queue.isDone()) { - if (!g_store->insertFileResultBatch(project_id, - batch)) { + if (!ctx->store->insertFileResultBatch( + project_id, batch)) { writer_error = 1; } files_written += static_cast(batch.size()); @@ -232,9 +272,9 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) } } if (writer_error) - g_store->rollbackTransaction(); + ctx->store->rollbackTransaction(); else - g_store->commitTransaction(); + ctx->store->commitTransaction(); }); // ── Parse workers ────────────────────────────────────────── @@ -280,12 +320,12 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) current_path = job.path; current_lang = job.lang; - // Fail-fast: skip files that have failed >= - // CODESCOPE_FAIL_RETRY_MAX times. + // Fail-fast. The set is empty when the caller requested + // bypass_fail_fast (force_index_files), which makes this a + // no-op there — see the note where it is loaded. if (known_failures.find(job.path) != - known_failures.end()) { + known_failures.end()) continue; - } int done = next_job.load(); if (done % progress_interval == 0 && done > 0) @@ -298,7 +338,32 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) (int)(done * 100 / total_files)); - std::string source = readFile(job.path.c_str()); + // Grammar availability, checked BEFORE reading the + // file: reading a file the engine has no grammar for is + // wasted I/O. A registered-but-NULL grammar (the + // language is mapped but its grammar is unavailable: + // "swift"/"kotlin"/"ruby"/"scala" while no grammar is + // vendored, or a .so that failed to load) is reported as + // LanguageMissing — handing nullptr to + // ts_parser_set_language would yield a null tree recorded + // as "parse_null_tree", a wrong reason that also disguised + // an unsupported language as a broken file. + { + auto lit = lang_ptrs.find(job.lang); + if (lit == lang_ptrs.end() || + lit->second == nullptr) { + store::bufferParseFailure( + project_id, job.path, + job.lang, + store::failReasonToString( + store::FailReason:: + LanguageMissing)); + continue; + } + } + + std::string source = + readFile(job.abs_path.c_str()); if (source.empty()) { store::bufferParseFailure( project_id, job.path, job.lang, @@ -308,7 +373,10 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) continue; } - // Per-thread parser + // Per-thread parser. A language with no registered + // grammar was already rejected above, so the lookup below + // only needs the presence check (kept as a guard against + // reordering). auto pit = tl_parsers.find(job.lang); if (pit == tl_parsers.end()) { auto lit = lang_ptrs.find(job.lang); @@ -375,6 +443,12 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) if (visitor) { ir::SemanticUnit *su = nullptr; + // The visitor transfers ownership of the + // returned unit to the caller (see + // js_visitor.h); without this guard every + // parsed file leaks one SemanticUnit. + std::unique_ptr + su_guard; try { su = visitor->visit( tree.get(), @@ -399,6 +473,7 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) VisitorUnknownThrow)); continue; } + su_guard.reset(su); if (su) { result->records = su->allRecords(); @@ -421,12 +496,17 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) LanguageMissing)); continue; } - ir::TranslationUnit *unit = nullptr; + // RAII ownership of the TranslationUnit: the + // translator contract says the caller frees it + // (ir_translator.h); unique_ptr keeps that true + // on the exception/fallback paths too. + std::unique_ptr + unit; try { - unit = translator->translate( + unit.reset(translator->translate( tree.get(), source.c_str(), - job.path.c_str()); + job.path.c_str())); } catch (const std::exception &e) { store::bufferParseFailure( project_id, job.path, @@ -552,6 +632,23 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) result_queue.markDone(); if (writer_thread.joinable()) writer_thread.join(); + // Write the parse failures this run buffered. Like the memBulk branch, + // this path never reached the streaming path's flush, so an unparseable + // file handed to force_index_files was re-attempted (see the note at the + // top) and then recorded nowhere — parse_failures kept the stale + // fail_count and get_parse_failures could never show it. Runs after + // every writer has stopped, so the auxiliary connection does not + // contend with the bulk writer's transaction. + const int flushed = store::flushParseFailures(ctx); + if (flushed < 0) { + // Reported, not swallowed: the run itself may still succeed, but its + // parse failures are missing from the table (code_rules.md, no silent + // error handling). + fprintf(stderr, + "engine: flushParseFailures failed; this run's parse " + "failures are not recorded " + "[module=engine, method=engine_index_files]\n"); + } time_parse_ms = duration_cast(steady_clock::now() - t_parse_start) .count(); @@ -561,14 +658,21 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) t_parse_start = steady_clock::now(); // buildGraph(...true) is a FULL rebuild: it drops the lookup // + unique-edge indexes. Unlike engine_index_project (which - // reaches this via engine_index_post_parse), this path must + // reaches this via postParsePhase), this path must // recreate those indexes itself or they stay missing (M-12). // P2 fix: a resolver-pipeline failure makes buildGraph roll back // its graph savepoint and return false; flag it as a writer error // so the result JSON reports failure instead of a false success // (the outer transaction is rolled back by the caller when it // sees ok:false). - if (!g_store->buildGraph(project_id, true)) { + // `bypass_fail_fast` is 1 only for force_index_files, whose contract + // is "index these paths regardless of the default skip rules" — so + // the graph must accept the test/bench/spec declarations this call + // just parsed, or the tool reports files_indexed while the graph + // stays empty (see buildGraph's include_test_files and the + // test_file_filter it guards). + if (!ctx || !ctx->store->buildGraph(project_id, true, nullptr, + bypass_fail_fast != 0)) { writer_error = 1; } time_buildgraph_ms = @@ -578,18 +682,18 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) // Mark every node callgraph_ready: the call graph is now // committed, so trace_path / enhancement-status report - // readiness correctly (mirrors engine_index_post_parse, M-15). + // readiness correctly (mirrors postParsePhase, M-15). { std::string up = "UPDATE graph_nodes SET callgraph_ready=1 " "WHERE project_id=" + std::to_string(project_id); - if (!g_store->exec(up.c_str())) { + if (!ctx || !ctx->store->exec(up.c_str())) { fprintf(stderr, "engine_index_files: callgraph_ready " "UPDATE failed: %s " "[module=engine, method=engine_index_files]\n", - g_store->error().c_str()); + ctx->store->error().c_str()); } } @@ -599,9 +703,11 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) // createIndexesAfterBulkLoad(project_id, !is_reindex) with // is_reindex=false (M-12). { - store::GraphStore::BulkPragmaGuard guard(g_store.get()); + store::GraphStore::BulkPragmaGuard guard( + ctx->store.get()); auto t_idx = steady_clock::now(); - g_store->createIndexesAfterBulkLoad(project_id, true); + ctx->store->createIndexesAfterBulkLoad(project_id, + true); fprintf(stderr, "engine: createIndexesAfterBulkLoad=%lldms " "[module=engine, method=engine_index_files]\n", @@ -612,7 +718,7 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) // Set the core-graph readiness flag so the project is // queryable immediately after a file-list index (M-15). - g_store->setProjectReadiness(project_id, "normal_ready", 1); + ctx->store->setProjectReadiness(project_id, "normal_ready", 1); } // ── Build result JSON ────────────────────────────────────── @@ -625,7 +731,7 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) // Query node/edge counts { - sqlite3 *db = g_store->handle(); + sqlite3 *db = ctx->store->handle(); sqlite3_stmt *stmt = nullptr; std::string sql = "SELECT COUNT(*) FROM entity WHERE project_id = " + @@ -662,6 +768,23 @@ char *engine_index_files(uint64_t project_id, const char *file_list_json) store::setIndexProgress(p); } - launchAsyncKnowledgeBuilder(project_id, !mode_fast); + launchAsyncKnowledgeBuilder(ctx, project_id, !mode_fast); return dupString(result.str()); } + +char *engine_index_files(engine_t handle, uint64_t project_id, + const char *file_list_json, int bypass_fail_fast) +{ + EngineContext *ctx = engineInstance(handle); + + try { + return indexFilesImpl(ctx, project_id, file_list_json, + bypass_fail_fast); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_index_files", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_index_files", "unknown exception")); + } +} diff --git a/engine/src/engine_index_metrics.cpp b/engine/src/engine_index_metrics.cpp index 4f04741..199fa14 100644 --- a/engine/src/engine_index_metrics.cpp +++ b/engine/src/engine_index_metrics.cpp @@ -6,13 +6,272 @@ #include "engine_index_metrics.h" #include +#include #include +#include #include +#include #include #include #include #include +namespace +{ + +/// Parameter counts of one function declaration, in the two forms the two +/// consumers need. `declared` is what the source spells out (the code metric); +/// `arity` is how many arguments a call site supplies, which is what the +/// Resolver compares against a reference row. They differ by an implicit +/// receiver: Python and Rust write it as a parameter (`self` / `&self`) while +/// the call syntax hides it, so a call site that supplies n arguments has +/// arity n for a method that declares n + 1 parameters. +struct ParameterCount { + int declared = 0; + int arity = 0; +}; + +/// True when the node IS a parameter list. The `parameters` field of a function +/// declaration holds one of these in every grammar this indexer parses; +/// anything else in that field is a single bare parameter, which JavaScript and +/// TypeScript allow (`x => x`). +bool isParameterListType(const char *type) +{ + return strcmp(type, "parameter_list") == 0 || // C, C++, Go + strcmp(type, "parameters") == 0 || // Rust, Python + strcmp(type, "formal_parameters") == + 0; // JavaScript, TypeScript, Java +} + +/// Text of a node with whitespace, parentheses and comma removed — enough to +/// recognise C's `(void)`, whose single `parameter_declaration` is the one +/// parameter that declares no parameter. `(void *)` normalises to `void*` and +/// is therefore not mistaken for it. +std::string normalizedParameterText(TSNode node, const char *source) +{ + const uint32_t start = ts_node_start_byte(node); + const std::string text(source + start, ts_node_end_byte(node) - start); + std::string norm; + norm.reserve(text.size()); + for (char c : text) + if (c != '(' && c != ')' && c != ',' && + !isspace(static_cast(c))) + norm.push_back(c); + return norm; +} + +/// The identifier a parameter starts with: `self` in `self: Node = None`, +/// `cls` in `cls`, `self` in `&self` (Rust spells it as a node type instead). +/// Stops at the first separator that ends a name. +std::string leadingParameterName(TSNode param, const char *source) +{ + const uint32_t start = ts_node_start_byte(param); + const std::string text(source + start, ts_node_end_byte(param) - start); + std::string name; + for (char c : text) { + if (c == ':' || c == '=' || c == '&' || c == '<' || + isspace(static_cast(c))) + break; + name.push_back(c); + } + return name; +} + +/// True when the declaration sits inside a class body. Only Python needs this: +/// it is the one grammar here whose receiver is an ordinary parameter named by +/// convention, so the name alone would also drop a free function's first +/// argument (a function named `self` outside a class is not a method). +/// +/// Cost: one ancestor walk, run only for a parameter whose name is +/// `self`/`cls`, and bounded by the tree depth — the walk's visit cap keeps +/// nesting finite, so this is invisible next to the CST traversal that called +/// it. If that cap is ever lifted, cache the verdict per declaration node +/// rather than walking the same ancestors once per parameter. +bool isMethodDeclaration(TSNode decl) +{ + for (TSNode n = ts_node_parent(decl); !ts_node_is_null(n); + n = ts_node_parent(n)) + if (strcmp(ts_node_type(n), "class_definition") == 0) + return true; + return false; +} + +/// True when a parameter node type makes the number of arguments a call may +/// supply differ from the declared count: a variadic tail accepts any number of +/// extras and a default value lets a call supply fewer. `arity` is the count a +/// call site must supply, so such a declaration has no exact answer. +bool hasOpenParameterCount(const char *type) +{ + return strcmp(type, "variadic_parameter") == 0 || // C, C++ (…) + strcmp(type, "variadic_parameter_declaration") == + 0 || // Go (rest ...T) + strcmp(type, "list_splat_pattern") == 0 || // Python *args + strcmp(type, "dictionary_splat_pattern") == 0 || // Python **kw + strcmp(type, "rest_pattern") == + 0 || // JavaScript/TypeScript ...rest + strcmp(type, "default_parameter") == 0 || // Python (b=1) + strcmp(type, "typed_default_parameter") == + 0 || // Python (b: int = 1) + strcmp(type, "optional_parameter_declaration") == + 0 || // C++ (b = 0) + strcmp(type, "assignment_pattern") == 0; // JavaScript/TypeScript +} + +/// True when the parameter's source text contains `...`, the one marker every +/// variadic spelling shares (`opts ...any`, `...rest`, `T...`) and that no other +/// parameter declaration contains. +bool textHasEllipsis(TSNode param, const char *source) +{ + const uint32_t start = ts_node_start_byte(param); + const std::string text(source + start, ts_node_end_byte(param) - start); + return text.find("...") != std::string::npos; +} + +/// True when the parameter's source text assigns it a value (`bool append = +/// false`, `std::string name = "x"`). +/// +/// C++ spells a default INSIDE the ordinary `parameter_declaration`; the +/// `optional_parameter_declaration` node the type list above expects is not what +/// tree-sitter-cpp produces for it. So `bool loadIgnoreFile(const std::string +/// &root, bool append = false)` was recorded with arity 2 while a legitimate +/// one-argument call - which the default makes valid - was then PENALISED by +/// factorSignatureMatch (arity 1 against 2 scores kScorePenalty), and the edge +/// disappeared. Measured on this repository by rebuilding the previous commit +/// and diffing call edges: `policyFor -> loadIgnoreFile`, `indexProjectImpl -> +/// loadIgnoreFile` / `loadGitignore` / `insertFileResultBatch` and every other +/// lost edge were calls that omit a defaulted argument. `=` cannot otherwise +/// appear in a parameter declaration: the function's own name is outside the +/// list (`bool operator==(const X &o)` has no `=` among the parameters). +bool textHasDefaultValue(TSNode param, const char *source) +{ + const uint32_t start = ts_node_start_byte(param); + const std::string text(source + start, ts_node_end_byte(param) - start); + return text.find('=') != std::string::npos; +} + +/// Parameter counts of a parameter-list node: the number of NAMED children, +/// minus the two traps. `comment` is a named node in every grammar and is not a +/// parameter, and C's `(void)` is a single parameter_declaration that declares +/// none. +ParameterCount countParameters(TSNode params, TSNode decl, const char *source) +{ + ParameterCount out; + if (!isParameterListType(ts_node_type(params))) { + out.declared = out.arity = 1; // a bare parameter, e.g. `x => x` + return out; + } + + std::vector declared; + bool open_count = false; + for (uint32_t i = 0, n = ts_node_named_child_count(params); i < n; + ++i) { + TSNode child = ts_node_named_child(params, i); + const char *type = ts_node_type(child); + if (strcmp(type, "comment") == 0) + continue; + if (hasOpenParameterCount(type) || + textHasEllipsis(child, source) || + textHasDefaultValue(child, source)) + open_count = true; + declared.push_back(child); + } + if (declared.size() == 1 && + normalizedParameterText(declared.front(), source) == "void") + return out; // `(void)` — no parameter at all + + out.declared = static_cast(declared.size()); + if (declared.empty()) + return out; + + // A variadic tail or a default value means a valid call may pass a + // different number of arguments than the declaration lists (155 functions + // in goagent alone are variadic), so the count a call site must supply is + // unknown — 0 keeps the Resolver on its pre-existing behaviour for these + // instead of penalising the call that matches them. + if (open_count) + return out; + + out.arity = out.declared; + const TSNode first = declared.front(); + const std::string first_name = leadingParameterName(first, source); + if (strcmp(ts_node_type(first), "self_parameter") == 0 || + (isMethodDeclaration(decl) && + (first_name == "self" || first_name == "cls"))) + out.arity -= 1; // the call site never passes the receiver + return out; +} + +/// The parameter list declared under a declaration node, for the grammars whose +/// declaration does not own the list itself: C and C++ put it on the nested +/// declarator (`function_definition` → `function_declarator`), so a direct field +/// lookup on the declaration finds nothing. +/// +/// Bounded to two levels, which keeps the search inside the declaration: a +/// callback's parameter list (C's `void f(int (*cb)(int a))`) hangs off a +/// different declarator and a lambda in the body sits deeper than that, so +/// neither can be mistaken for the declaration's own parameters. +TSNode parameterListUnderDeclarator(TSNode decl) +{ + std::vector level{ decl }; + for (int depth = 0; depth < 2; ++depth) { + std::vector next; + for (TSNode n : level) { + for (uint32_t i = 0, + n_children = ts_node_child_count(n); + i < n_children; ++i) { + TSNode child = ts_node_child(n, i); + TSNode params = ts_node_child_by_field_name( + child, "parameters", 10); + if (!ts_node_is_null(params)) + return params; + next.push_back(child); + } + } + level.swap(next); + } + return TSNode{}; +} + +/// Parameter counts of the function declaration owning the record spanning +/// [start_byte, end_byte). Returns false when the CST exposes no such +/// declaration (the legacy translator path, or a grammar whose declaration node +/// carries no `parameters` field), in which case the caller leaves the counts +/// at 0 — absent, not wrong. +/// +/// Two lookups, because a record's range starts either at the declaration or at +/// the declared name: +/// * the record's own node may span the whole declaration, in which case the +/// list is a field of it (Go, Java, JavaScript, Python, Rust) or one level +/// down on its declarator (C, C++); +/// * or it spans only the name, in which case climbing to the nearest +/// ancestor carrying a `parameters` field finds that function's own list. +/// Either way the list belongs to this declaration: a Go method's receiver is a +/// separate field and is not counted as a parameter. +bool parameterCountForRecord(TSNode root, uint32_t start_byte, + uint32_t end_byte, const char *source, + ParameterCount &out) +{ + TSNode node = ts_node_descendant_for_byte_range( + root, start_byte, + end_byte > start_byte ? end_byte - 1 : start_byte); + bool own_node = true; + while (!ts_node_is_null(node)) { + TSNode params = + ts_node_child_by_field_name(node, "parameters", 10); + if (ts_node_is_null(params) && own_node) + params = parameterListUnderDeclarator(node); + if (!ts_node_is_null(params)) { + out = countParameters(params, node, source); + return true; + } + own_node = false; + node = ts_node_parent(node); + } + return false; +} + +} // namespace + namespace index_metrics { @@ -20,7 +279,7 @@ namespace index_metrics std::vector computeMetricsFromCST(TSTree *tree, const char *source, - const std::vector &records) + std::vector &records) { std::vector result; if (!tree || !source || records.empty()) @@ -56,7 +315,7 @@ computeMetricsFromCST(TSTree *tree, const char *source, struct FuncEntry { uint32_t start_byte; uint32_t end_byte; - const ir::Record *rec; + ir::Record *rec; }; std::vector funcs; funcs.reserve(records.size() / 4); @@ -224,6 +483,33 @@ computeMetricsFromCST(TSTree *tree, const char *source, eff_depth, eff_func }); } + // Parameter counts come from the CST, not from the record subtree walked + // above: RecordKind::Parameter (8) exists and the walk counts it, but no + // Visitor ever emits one — the record layer carries parameters as + // arguments of a declaration it does not model — so param_count was 0 on + // every function of every project (goagent: 6163 function entities, 0 with + // param_count > 0) while cyclomatic/lines/cognitive were correct. The + // count is read from the function declaration's own `parameters` field. + // + // The same number is written to the record's `arity`: buildGraph fills + // entity.arity from semantic_records.arity and the Resolver Pipeline reads + // entity.arity to weigh same-name overloads (factorSignatureMatch), so + // without a producer for it every candidate scored as "unknown arity" — + // the factor could neither prefer the overload whose parameter count + // matches the call site nor penalise the ones that do not. + { + const TSNode cst_root = ts_tree_root_node(tree); + for (auto &fe : funcs) { + ParameterCount params; + if (!parameterCountForRecord(cst_root, fe.start_byte, + fe.end_byte, source, + params)) + continue; // no declaration to read — leave 0 + metrics_map[fe.rec].param_count = params.declared; + fe.rec->arity = params.arity; + } + } + // Finalize: cyclomatic = 1 + branches + loops, // cognitive = cyclomatic + nesting_depth (approximation) result.reserve(funcs.size()); diff --git a/engine/src/engine_index_metrics.h b/engine/src/engine_index_metrics.h index 66f8117..1405cee 100644 --- a/engine/src/engine_index_metrics.h +++ b/engine/src/engine_index_metrics.h @@ -25,16 +25,25 @@ namespace index_metrics /// and the flattened record list produced by the Visitor pipeline. /// /// The CST supplies control-flow nodes (if/for/while/switch/case) that -/// RecordKind intentionally elides. Records provide param/call counts with -/// correct RecordKind values (Parameter=8, CallExpr=9). +/// RecordKind intentionally elides, and the parameter lists: no Visitor emits +/// RecordKind::Parameter, so a record-side parameter count is always 0 (see +/// the note on `arity` below). Records provide call counts with correct +/// RecordKind values (CallExpr=9). +/// +/// Records are NOT const: this also writes each Function/Method record's +/// `arity` (its parameter count), because buildGraph fills entity.arity from +/// semantic_records.arity and the Resolver Pipeline reads that column to tell +/// same-name overloads apart. The records must therefore be passed before they +/// are written to the database, which is what all three index paths do. /// /// \param tree The tree-sitter parse tree. Must outlive the call. /// \param source Null-terminated source text the tree was parsed from. -/// \param records Flattened records from the Visitor (SemanticUnit). +/// \param records Flattened records from the Visitor (SemanticUnit); function +/// and method records get their `arity` set in place. /// \return One MetricRow per Function/Method record, in record order. std::vector computeMetricsFromCST(TSTree *tree, const char *source, - const std::vector &records); + std::vector &records); /// Compute per-function metrics from the legacy IR TranslationUnit tree. /// Used as a fallback when no Visitor is available for a language. diff --git a/engine/src/engine_index_paths.h b/engine/src/engine_index_paths.h new file mode 100644 index 0000000..79ee4fe --- /dev/null +++ b/engine/src/engine_index_paths.h @@ -0,0 +1,390 @@ +#ifndef ENGINE_INDEX_PATHS_H +#define ENGINE_INDEX_PATHS_H + +// How a file's path is spelled in the index. +// +// One file must have one identity, but two things decide the spelling and they +// disagree: +// +// * the walk-based entry points store `filesystem::path(root_argument) / +// entry`, so the spelling depends on the argument the caller passed — +// `/abs/proj/src/a.go` for an absolute root, `./src/a.go` for ".", and +// `proj/src/a.go` for a relative `proj`; +// * the single-file entry points (`index_file`, `force_index_files`, the +// scheduler's `--file-list` retry) are handed ABSOLUTE paths by their +// callers and used to store them verbatim. +// +// Same file, two spellings → the entities exist twice, a query by the indexed +// spelling misses the new rows, and one run can count the file twice. Since the +// projects table only records the canonicalised root, the walk's spelling +// cannot be recomputed from it, so the existing spelling is looked up and +// reused. The common spellings are enumerated explicitly (no suffix guessing, +// which could match a different file), and the ones that are not covered leave +// the file with the old behaviour rather than a wrong one. +// +// "Existing" covers both tables that store a file path: `entity` for files that +// were indexed, and `parse_failures` for files that never were. The second is +// what keeps a persistently unparseable file from collecting one identity per +// spelling — see knownSpellingFor. +// +// Enumerating spellings is not enough when the two sides differ by a SYMLINKED +// ANCESTOR (`/tmp/x` vs `/private/tmp/x`): no spelling of the canonical path +// yields the aliased one, because realpath only goes one way. That case is +// closed by a second pass that narrows on the path tail and accepts a stored +// row only when its canonical form equals the incoming path's — equality, not +// a guess, so two files can never be merged. + +#include "store/store.h" + +#include + +#include +#include +#include + +#include +#include + +/// The project's root directory as recorded by create_project ("" if unknown). +inline std::string projectRootPath(store::GraphStore *store, + uint64_t project_id) +{ + if (!store || !store->handle()) + return ""; + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(store->handle(), + "SELECT root_path FROM projects WHERE id=?", -1, + &stmt, nullptr) != SQLITE_OK) + return ""; + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + std::string root; + if (sqlite3_step(stmt) == SQLITE_ROW) { + const char *p = reinterpret_cast( + sqlite3_column_text(stmt, 0)); + if (p) + root = p; + } + sqlite3_finalize(stmt); + return root; +} + +/// The candidate spellings to test for `path`, most specific first. +/// \param root Project root as recorded by create_project ("" if unknown). +/// \param path Path as the caller spells it. +/// \return `path` itself plus, when it lies under `root`, its path relative to +/// the root, its "./"-prefixed form and its "/" form. +inline std::vector spellingCandidates(const std::string &root, + const std::string &path) +{ + std::vector candidates{ path }; + if (!root.empty() && root != "/" && !path.empty() && path[0] == '/') { + std::string prefix = root; + if (prefix.back() != '/') + prefix.push_back('/'); + if (path.compare(0, prefix.size(), prefix) == 0) { + const std::string rel = path.substr(prefix.size()); + if (!rel.empty()) { + candidates.push_back(rel); + candidates.push_back("./" + rel); + const size_t slash = root.find_last_of('/'); + const std::string base = + slash == std::string::npos ? + root : + root.substr(slash + 1); + if (!base.empty()) + candidates.push_back(base + "/" + rel); + } + } + } + return candidates; +} + +/// Look `candidates` up in one stored-path column, or return "". +/// \param table Internal table name — a literal from this header, never +/// caller input, so the concatenation cannot be injected into. +inline std::string +lookupStoredSpelling(store::GraphStore *store, uint64_t project_id, + const char *table, + const std::vector &candidates) +{ + if (!store || !store->handle() || candidates.empty()) + return ""; + std::string sql = std::string("SELECT file_path FROM ") + table + + " WHERE project_id=? AND file_path IN ("; + for (size_t i = 0; i < candidates.size(); i++) + sql += (i > 0 ? ",?" : "?"); + sql += ") LIMIT 1"; + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(store->handle(), sql.c_str(), -1, &stmt, + nullptr) != SQLITE_OK) { + // Reported, then treated as "not stored": the caller falls back to + // the spelling it was given (code_rules.md: no silent errors). + fprintf(stderr, + "engine: spelling lookup prepare failed for %s " + "[module=engine, method=lookupStoredSpelling]\n", + table); + return ""; + } + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + for (size_t i = 0; i < candidates.size(); i++) + sqlite3_bind_text(stmt, static_cast(i + 2), + candidates[i].c_str(), -1, SQLITE_TRANSIENT); + std::string found; + const int rc = sqlite3_step(stmt); + if (rc == SQLITE_ROW) { + const char *p = reinterpret_cast( + sqlite3_column_text(stmt, 0)); + if (p) + found = p; + } else if (rc != SQLITE_DONE) { + fprintf(stderr, + "engine: spelling lookup step failed (%d) for %s " + "[module=engine, method=lookupStoredSpelling]\n", + rc, table); + } + sqlite3_finalize(stmt); + return found; +} + +/// How many stored-path candidates the alias fallback inspects per table +/// before giving up. Bounds the work when many rows share a path tail. +constexpr int kAliasCandidateLimit = 16; + +/// Absolute normalised form of `path`, or "" when it cannot be compared. +/// +/// Uses weakly_canonical: it resolves the symlinked part of the path without +/// requiring the file to exist, so a file that was deleted (or that a crashed +/// run left behind) still compares equal to its other spelling. Relative paths +/// return "" — the walk recorded them relative to a working directory this +/// function cannot know, so resolving them here would be a guess. +inline std::string canonicalForComparison(const std::string &path) +{ + if (path.empty() || !std::filesystem::path(path).is_absolute()) + return ""; + std::error_code ec; + const std::string canon = + std::filesystem::weakly_canonical(path, ec).string(); + return ec ? std::string() : canon; +} + +/// Escape the LIKE metacharacters in `s` so it matches literally. +/// \param s Text to match against `file_path`. +/// \return `s` with `%`, `_` and `\` backslash-escaped (use with ESCAPE '\'). +inline std::string escapeLikePattern(const std::string &s) +{ + std::string out; + out.reserve(s.size() + 8); + for (const char c : s) { + if (c == '%' || c == '_' || c == '\\') + out.push_back('\\'); + out.push_back(c); + } + return out; +} + +/// Canonical form of a possibly relative `path` resolved against `base_dir`. +/// \param base_dir Directory the relative path is resolved against. +/// \param path Path to normalise. +/// \return Absolute normalised form, or "" when it cannot be resolved. +inline std::string canonicalRelativeTo(const std::string &base_dir, + const std::string &path) +{ + if (base_dir.empty() || path.empty()) + return ""; + std::error_code ec; + const std::string joined = + (std::filesystem::path(base_dir) / path).string(); + const std::string canon = + std::filesystem::weakly_canonical(joined, ec).string(); + return ec ? std::string() : canon; +} + +/// Second pass over the stored spellings: those that cannot be enumerated. +/// Rows are narrowed by their path tail and accepted only when they are +/// provably the same file as the incoming path — never on resemblance. +/// +/// * an ABSOLUTE row is accepted when its canonical form equals +/// `canonical_incoming`. This is the symlinked-ancestor case (macOS +/// `/tmp/x` is really `/private/tmp/x`): indexing through the alias stores +/// "/tmp/x/f.c" while the single-file entry points are handed +/// "/private/tmp/x/f.c", and no spelling of the latter enumerates the +/// former. Without it the file reached `entity` twice and `find_symbol` +/// answered with its symbol twice. +/// * a RELATIVE row is accepted when it is "/" and `prefix` +/// resolves, from the current working directory, to the project root. That +/// is what the walk stored for a relative root argument (`./proj/f.c`, +/// `../proj/f.c`) when the indexer ran from this directory; when it did not, +/// the check fails and the row is left alone rather than guessed at. +/// +/// \param rel Path relative to the project root (no leading /). +/// \param canonical_incoming canonicalForComparison() of the incoming path. +/// \param canonical_root canonicalForComparison() of the project root. +inline std::string +lookupStoredSpellingByCanonical(store::GraphStore *store, uint64_t project_id, + const char *table, const std::string &rel, + const std::string &canonical_incoming, + const std::string &canonical_root) +{ + if (!store || !store->handle() || rel.empty() || + canonical_incoming.empty()) + return ""; + std::string sql = std::string("SELECT file_path FROM ") + table + + " WHERE project_id=? AND file_path LIKE ? ESCAPE '\\'" + " LIMIT " + + std::to_string(kAliasCandidateLimit); + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(store->handle(), sql.c_str(), -1, &stmt, + nullptr) != SQLITE_OK) { + // Not fatal: a failed probe leaves the file with the ordinary + // behaviour (it is stored under the spelling it came in with), but + // it must not pass unnoticed (code_rules.md: no silent errors). + fprintf(stderr, + "engine: alias spelling probe prepare failed for %s " + "[module=engine, method=lookupStoredSpellingByCanonical]\n", + table); + return ""; + } + const std::string pattern = "%/" + escapeLikePattern(rel); + const std::string suffix = "/" + rel; + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_text(stmt, 2, pattern.c_str(), -1, SQLITE_TRANSIENT); + + // Resolved once; empty disables the relative branch (see the contract). + std::error_code cwd_ec; + const std::string cwd = std::filesystem::current_path(cwd_ec).string(); + const bool can_check_relative = !cwd_ec && !cwd.empty() && + !canonical_root.empty(); + + std::string found; + int rc = SQLITE_DONE; + while ((rc = sqlite3_step(stmt)) == SQLITE_ROW) { + const char *p = reinterpret_cast( + sqlite3_column_text(stmt, 0)); + if (!p) + continue; + const std::string stored(p); + // LIKE narrowed it, but the acceptance test below is exact: the row + // must really end with "/". + if (stored.size() <= suffix.size() || + stored.compare(stored.size() - suffix.size(), suffix.size(), + suffix) != 0) + continue; + if (std::filesystem::path(stored).is_absolute()) { + if (canonicalForComparison(stored) == + canonical_incoming) { + found = stored; + break; + } + continue; + } + if (!can_check_relative) + continue; + const std::string prefix = + stored.substr(0, stored.size() - suffix.size()); + if (prefix.empty()) + continue; + if (canonicalRelativeTo(cwd, prefix) == canonical_root) { + found = stored; + break; + } + } + if (rc != SQLITE_DONE && rc != SQLITE_ROW) { + fprintf(stderr, + "engine: alias spelling probe step failed (%d) for %s " + "[module=engine, method=lookupStoredSpellingByCanonical]\n", + rc, table); + } + sqlite3_finalize(stmt); + return found; +} + +/// The spelling under which `path` is ALREADY stored for this project, or "" +/// when the project has not indexed it yet. +inline std::string existingSpellingFor(store::GraphStore *store, + uint64_t project_id, + const std::string &root, + const std::string &path) +{ + if (path.empty()) + return ""; + return lookupStoredSpelling(store, project_id, "entity", + spellingCandidates(root, path)); +} + +/// The spelling under which `path` is already KNOWN to the project — indexed, +/// or recorded as a parse failure — or "" when it is a new file. +/// +/// Two passes, each over `entity` first and `parse_failures` second: +/// 1. the enumerated spellings (path, root-relative, "./"-prefixed, +/// "/" — see spellingCandidates); +/// 2. spellings that CANNOT be enumerated, accepted only on canonical +/// equality (a symlinked ancestor, see lookupStoredSpellingByCanonical). +/// +/// The parse_failures half is not redundant: a file that never parses has no +/// entity row, and those are exactly the files that accumulate parse_failures +/// rows. Looking only at `entity` therefore left them free to acquire a second +/// identity, because the walk-based entry points spell a path as the caller +/// passed it while the single-file entry points are handed canonicalised paths +/// (on macOS `/tmp` is a symlink, so the same file arrived as both +/// "/tmp/p/a.py" and "/private/tmp/p/a.py"). One file then held two +/// parse_failures rows with independent fail_count values, so no retry policy +/// could reason about it, and `get_parse_failures` listed the file twice — +/// and, for files that DO parse, `entity` held the symbols twice. +inline std::string knownSpellingFor(store::GraphStore *store, + uint64_t project_id, + const std::string &root, + const std::string &path) +{ + if (path.empty()) + return ""; + const std::vector candidates = + spellingCandidates(root, path); + std::string found = + lookupStoredSpelling(store, project_id, "entity", candidates); + if (!found.empty()) + return found; + found = lookupStoredSpelling(store, project_id, "parse_failures", + candidates); + if (!found.empty()) + return found; + + // Second pass: `path` under the recorded root gives the tail to match on. + if (root.empty() || root == "/" || + !std::filesystem::path(path).is_absolute()) + return ""; + std::string prefix = root; + if (prefix.back() != '/') + prefix.push_back('/'); + if (path.compare(0, prefix.size(), prefix) != 0) + return ""; + const std::string rel = path.substr(prefix.size()); + if (rel.empty()) + return ""; + const std::string canon = canonicalForComparison(path); + if (canon.empty()) + return ""; + const std::string canonical_root = canonicalForComparison(root); + found = lookupStoredSpellingByCanonical(store, project_id, "entity", + rel, canon, canonical_root); + if (!found.empty()) + return found; + return lookupStoredSpellingByCanonical(store, project_id, + "parse_failures", rel, canon, + canonical_root); +} + +/// The spelling to store `path` under: the project's existing one when it +/// already knows the file (indexed, or a recorded parse failure), otherwise +/// `path` as given. +inline std::string indexSpellingFor(store::GraphStore *store, + uint64_t project_id, + const std::string &path) +{ + if (path.empty()) + return path; + const std::string stored = knownSpellingFor( + store, project_id, projectRootPath(store, project_id), path); + return stored.empty() ? path : stored; +} + +#endif // ENGINE_INDEX_PATHS_H diff --git a/engine/src/engine_index_project.cpp b/engine/src/engine_index_project.cpp index 0de0ac4..23d4c84 100644 --- a/engine/src/engine_index_project.cpp +++ b/engine/src/engine_index_project.cpp @@ -1,3 +1,4 @@ +#include "util/json_writer.h" #include "engine_internal.h" #include "filter_policy.h" #include "platform_win.h" @@ -53,13 +54,32 @@ using namespace engine_index_sched; // ─── Index Project (Parallel) ────────────────────────────────── -char *engine_index_project(uint64_t project_id, const char *dir_path, - const char *language_filter) +/// Body of engine_index_project. Kept as a separate function so the extern "C" +/// entry point below can stay a thin try/catch wrapper: no C++ exception may +/// cross the C ABI boundary (the MCP server is long-running, so an escaping +/// exception would terminate it). +static char *indexProjectImpl(EngineContext *ctx, uint64_t project_id, + const char *dir_path, const char *language_filter) { - if (!g_store) + // The instance arrives as a parameter (TD-1 knife 3): indexProjectImpl + // drives the whole index run and touches the state at every phase + // boundary, so it validates the handle once, here. + if (!ctx || !ctx->store) return dupString( "{\"ok\":false,\"error\":\"engine not initialized\"}"); + // Serialize with the background enrichment thread. It writes to the + // same engine store connection (model / state / FTS / knowledge) and opens + // its own transactions, so indexing concurrently would interleave + // BEGIN/COMMIT on one connection ("cannot start a transaction within + // a transaction") and one side could commit the other's half-written + // state. Joining is a no-op when the builder has already finished. + // Order matters: join FIRST (the builder holds g_store_mutex and + // would deadlock against a guard we already held), then take the + // store guard so a later builder launched mid-write cannot interleave. + joinAsyncKnowledgeBuilder(); + auto _store_guard = waitForKnowledgeBuilder(); + // Fail-fast: pre-load known parse failures so the parse loop can // skip them without per-file DB queries. CODESCOPE_FAIL_RETRY_MAX // sets the threshold (default kDefaultFailRetryMax = 1) — files @@ -71,7 +91,8 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, std::unordered_set known_failures; { std::vector fail_vec; - if (!store::loadKnownParseFailures(project_id, kFailRetryMax, + if (!store::loadKnownParseFailures(ctx, project_id, + kFailRetryMax, /*out*/ fail_vec)) { // Non-fatal: continue indexing, just no skip set. fprintf(stderr, @@ -96,18 +117,10 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, std::string lang_filter = language_filter ? language_filter : ""; - // Pre-parse language filter and max file size ONCE before file discovery - // (not per-file in the worker or file-collection loop) - std::unordered_set lang_filter_set; - if (!lang_filter.empty()) { - size_t start = 0, end; - do { - end = lang_filter.find(',', start); - lang_filter_set.insert( - lang_filter.substr(start, end - start)); - start = end + 1; - } while (end != std::string::npos); - } + // Parse max file size ONCE before file discovery (not per-file in the + // worker or file-collection loop). The language filter is applied only + // through FilterPolicy::setLanguageFilter below — the single + // implementation with alias folding — so no local label set is kept. uint64_t max_file_size = kMaxFileSize; const char *env_max = getenv("CODESCOPE_MAX_FILE_SIZE"); if (env_max) @@ -128,6 +141,39 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, // Load .codescopeignore + .gitignore patterns from project root filter.loadIgnoreFile(dir); filter.loadGitignore(dir); + // A per-module worker (the parallel scheduler spawns one per module) + // scans `/`, and that module directory usually has + // no ignore file of its own — the rules live at the project root, which + // the scheduler passes in CODESCOPE_PROJECT_ROOT. Without this, the + // parallel path applied none of the project's .gitignore rules: a + // `**/build-*/` rule pruned nothing and a build tree's own CMake probe + // sources (CMakeFiles//CompilerIdC*/CMakeCCompilerId.c) became + // entities and modules in the merged graph, exactly as the whole-project + // path did before the same rules were honoured there. + // + // Loaded AFTER the scan directory's own rules so that the more specific + // file still wins: the matcher stops at the first positive match. + const char *env_project_root = getenv("CODESCOPE_PROJECT_ROOT"); + if (env_project_root && *env_project_root && + std::string(env_project_root) != dir) { + filter.loadIgnoreFile(env_project_root, /*append=*/true); + filter.loadGitignore(env_project_root, /*append=*/true); + // Those rules are anchored at the project root while this worker's + // paths are relative to `dir`, so give the policy the missing prefix + // (`/`). Without it a rule like `**/build-*/` cannot match a + // path that begins inside the module — the module-level twin of the + // file-level case this same change fixed. + std::string root(env_project_root); + std::string prefix; + if (dir.size() > root.size() + 1 && + dir.compare(0, root.size(), root) == 0 && + dir[root.size()] == '/') { + prefix = dir.substr(root.size() + 1); + if (!prefix.empty() && prefix.back() != '/') + prefix += '/'; + } + filter.setScanPrefix(prefix); + } // Load CODESCOPE_EXCLUDE_PATHS env var (comma-separated globs) so // users can exclude non-core dirs (test/, docs/, vendor/) at index // time to reduce node count on very large projects. @@ -135,7 +181,7 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, // Batch-load file scan state ONCE to avoid N per-file DB queries // during discovery (1254 files × ~2ms prepare/finalize = ~2.5s saved). - auto scan_state = g_store->loadFileScanStateBatch(project_id); + auto scan_state = ctx->store->loadFileScanStateBatch(project_id); // Phase 1: collect file paths (single-threaded). The directory walk, // README ingestion and incremental scan-state gate now live in @@ -144,7 +190,7 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, std::vector jobs; bool is_reindex = false; std::string discover_err; - if (engine_index_discover::collectFileJobs(project_id, dir, filter, + if (engine_index_discover::collectFileJobs(ctx, project_id, dir, filter, scan_state, jobs, is_reindex, discover_err) != 0) { return dupString(discover_err.c_str()); @@ -166,11 +212,11 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, // rebuilt row count — preserving the A19 "readiness matches // canonical data" invariant. if (is_reindex && env_mode && strcmp(env_mode, "deep") == 0) { - g_store->buildVectorsFromGraph(project_id); + ctx->store->buildVectorsFromGraph(project_id); sqlite3_stmt *vstmt = nullptr; const char *vsql = "SELECT COUNT(*) FROM node_vectors WHERE project_id = ?"; - if (sqlite3_prepare_v2(g_store->handle(), vsql, -1, + if (sqlite3_prepare_v2(ctx->store->handle(), vsql, -1, &vstmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64( vstmt, 1, @@ -180,7 +226,7 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, vec_rows = sqlite3_column_int64(vstmt, 0); sqlite3_finalize(vstmt); - g_store->setProjectReadiness( + ctx->store->setProjectReadiness( project_id, "vector_ready", vec_rows > 0 ? 1 : 0); } else { @@ -188,7 +234,7 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, "engine_index_project: node_vectors count " "probe failed (no-op re-index): %s " "[module=engine, method=engine_index_project]\n", - sqlite3_errmsg(g_store->handle())); + sqlite3_errmsg(ctx->store->handle())); } } return dupString( @@ -210,7 +256,7 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, active_files.reserve(jobs.size()); for (auto &job : jobs) active_files.push_back(job.path); - g_store->cleanupStaleFiles(project_id, active_files); + ctx->store->cleanupStaleFiles(project_id, active_files); } // Sort jobs by file size descending — large files first @@ -228,7 +274,7 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, for (auto &j : jobs) langs.insert(j.lang); for (auto &l : langs) - lang_ptrs[l] = g_parser->getLanguage(l.c_str()); + lang_ptrs[l] = ctx->parser->getLanguage(l.c_str()); } // Index mode (from env): "fast" | "normal" (default) | "deep" @@ -251,9 +297,11 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, job_lang.reserve(jobs.size()); for (auto &job : jobs) job_lang.push_back({ job.path, job.lang }); - return engine_index_project_membulk( - project_id, dir, max_file_size, filter, job_lang, - lang_ptrs, is_reindex, mode_fast, mode_deep); + return engine_index_project_membulk(ctx, project_id, dir, + max_file_size, filter, + job_lang, lang_ptrs, + known_failures, is_reindex, + mode_fast, mode_deep); } // ── Dynamic-scheduler init ──────────────────────────────── @@ -313,7 +361,9 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, const size_t kWriterBatchSize = 50; std::thread writer_thread([&]() { - g_store->beginTransaction(); + // `ctx` is the caller's handle-validated instance, captured by + // reference; the writer owns the only SQLite write path. + ctx->store->beginTransaction(); std::vector batch; batch.reserve(kWriterBatchSize); @@ -324,7 +374,8 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, if (!ok) { // Queue is done and empty — flush any remaining batch if (!batch.empty()) { - if (!g_store->insertFileResultBatch( + if (!ctx || + !ctx->store->insertFileResultBatch( project_id, batch)) { writer_error = 1; } @@ -348,8 +399,8 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, // Flush batch to DB (within the single transaction) if (batch.size() >= kWriterBatchSize || result_queue.isDone()) { - if (!g_store->insertFileResultBatch(project_id, - batch)) { + if (!ctx || !ctx->store->insertFileResultBatch( + project_id, batch)) { writer_error = 1; fprintf(stderr, "writer: insertFileResultBatch" @@ -362,9 +413,9 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, } if (writer_error) - g_store->rollbackTransaction(); + ctx->store->rollbackTransaction(); else - g_store->commitTransaction(); + ctx->store->commitTransaction(); }); // ── Parse workers ────────────────────────────────────────── @@ -470,6 +521,30 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, continue; } + // Grammar availability, checked BEFORE reading the file: + // reading a file the engine has no grammar for is wasted I/O, + // and this check must run on EVERY attempt (a language_missing + // row never becomes a permanent skip), so its cost matters. + // A registered-but-NULL grammar (the language is mapped but no + // grammar is vendored for it: "swift", "kotlin", "ruby", + // "scala", or a .so that failed to load) is reported as + // LanguageMissing — handing nullptr to ts_parser_set_language + // would yield a null tree recorded as "parse_null_tree", a wrong + // reason that also disguised an unsupported language as a broken + // file. + { + auto lit = lang_ptrs.find(job.lang); + if (lit == lang_ptrs.end() || + lit->second == nullptr) { + store::bufferParseFailure( + project_id, job.path, job.lang, + store::failReasonToString( + store::FailReason:: + LanguageMissing)); + continue; + } + } + // v0.6 (perf): st_size was just obtained above, so reuse it to // skip readFile's ate-seek + tellg round-trip per file. std::string source = readFilePrealloc( @@ -486,6 +561,9 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, // Per-thread parser auto pit = tl_parsers.find(job.lang); if (pit == tl_parsers.end()) { + // Grammar availability was validated above (a null + // grammar never reaches this point), so the lookup only + // needs the presence check. auto lit = lang_ptrs.find(job.lang); if (lit == lang_ptrs.end()) { store::bufferParseFailure( @@ -538,6 +616,11 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, if (visitor) { ir::SemanticUnit *su = nullptr; + // The visitor transfers ownership of the returned + // unit to the caller (see js_visitor.h); without + // this guard every parsed file leaks one + // SemanticUnit. + std::unique_ptr su_guard; try { su = visitor->visit(tree.get(), source.c_str(), @@ -558,6 +641,7 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, VisitorUnknownThrow)); continue; } + su_guard.reset(su); if (su) { result->records = su->allRecords(); result->metrics = index_metrics:: @@ -578,11 +662,13 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, LanguageMissing)); continue; } - ir::TranslationUnit *unit = nullptr; + // RAII ownership: the translator contract says + // the caller frees the unit (ir_translator.h). + std::unique_ptr unit; try { - unit = translator->translate( + unit.reset(translator->translate( tree.get(), source.c_str(), - job.path.c_str()); + job.path.c_str())); } catch (const std::exception &e) { store::bufferParseFailure( project_id, job.path, job.lang, @@ -644,7 +730,8 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, if (unit->root) flatten(unit->root, 0); result->metrics = index_metrics:: - computeMetricsFromUnit(unit); + computeMetricsFromUnit( + unit.get()); } } @@ -800,7 +887,7 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, // Flush any buffered parse failures to SQLite. // This is done AFTER the writer thread joins so there's no // concurrent write contention on the parse_failures table. - store::flushParseFailures(); + store::flushParseFailures(ctx); int total_indexed = files_written.load(); @@ -822,7 +909,24 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, for (auto &job : jobs) job_paths.push_back(job.path); - return engine_index_post_parse(project_id, dir, job_paths, filter, - is_reindex, mode_fast, mode_deep, - time_parse_ms, 0, total_indexed); + return postParsePhase(ctx, project_id, dir, job_paths, filter, + is_reindex, mode_fast, mode_deep, time_parse_ms, + 0, total_indexed); +} + +char *engine_index_project(engine_t handle, uint64_t project_id, + const char *dir_path, const char *language_filter) +{ + EngineContext *ctx = engineInstance(handle); + + try { + return indexProjectImpl(ctx, project_id, dir_path, + language_filter); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_index_project", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_index_project", "unknown exception")); + } } diff --git a/engine/src/engine_index_project_membulk.cpp b/engine/src/engine_index_project_membulk.cpp index 64b0ab7..55024cd 100644 --- a/engine/src/engine_index_project_membulk.cpp +++ b/engine/src/engine_index_project_membulk.cpp @@ -9,7 +9,7 @@ // unique_ptr into a BoundedQueue, each worker appends to a // thread-local vector and merges into the aggregator when the buffer is // full or the worker exits. The post-parse graph-building sequence is shared -// with the streaming path via engine_index_post_parse(). +// with the streaming path via postParsePhase(). #include "engine_internal.h" #include "store/store_membulk.h" @@ -33,15 +33,17 @@ #include "ir/semantic_unit.h" #include "ir/translators/js_visitor.h" #include "engine_index_metrics.h" +#include "store/store_parse_failure.h" char *engine_index_project_membulk( - uint64_t project_id, const std::string &dir, uint64_t max_file_size, - const FilterPolicy &filter, + EngineContext *ctx, uint64_t project_id, const std::string &dir, + uint64_t max_file_size, const FilterPolicy &filter, const std::vector> &job_lang, const std::unordered_map &lang_ptrs, - bool is_reindex, bool mode_fast, bool mode_deep) + const std::unordered_set &known_failures, bool is_reindex, + bool mode_fast, bool mode_deep) { - if (!g_store) + if (!ctx || !ctx->store) return dupString( "{\"ok\":false,\"error\":\"engine not initialized\"}"); @@ -120,6 +122,17 @@ char *engine_index_project_membulk( break; auto &job = jobs[idx]; + // Fail-fast: skip files that have failed >= + // CODESCOPE_FAIL_RETRY_MAX times. Mirrors the streaming path + // (engine_index_project.cpp) — this path used to ignore the + // skip set entirely, so on a project of <=2000 files (the + // common case) every run re-parsed and re-recorded the same + // unparseable files, and the documented "skipped entirely on + // the next run" behaviour never happened. + if (known_failures.find(job.path) != + known_failures.end()) + continue; + // Progress log every 10% int done = next_job.load(); if (done % progress_interval == 0 && done > 0) @@ -130,23 +143,76 @@ char *engine_index_project_membulk( (long long)done, (long long)total_files, (int)(done * 100 / total_files)); - // File size check + // File size check. stat() must be checked BEFORE the size + // comparison: the previous `stat(...) == 0 && ...` form + // fell through on stat failure and then read the + // uninitialized file_stat at result.mtime/result.fsize + // below, writing garbage into the incremental-index + // freshness baseline. Mirrors engine_index_project. struct stat file_stat; - if (stat(job.path.c_str(), &file_stat) == 0 && - static_cast(file_stat.st_size) > - max_file_size) + if (stat(job.path.c_str(), &file_stat) != 0) { + store::bufferParseFailure( + project_id, job.path, job.lang, + store::failReasonToString( + store::FailReason::StatFailed)); + continue; + } + if (static_cast(file_stat.st_size) > + max_file_size) { + // Policy skip, not a parse failure: a large + // generated file must not be permanently + // blacklisted by the fail-fast table. + fprintf(stderr, + "engine: skip oversize file (%s): %s " + "[module=engine, " + "method=engine_index_project_membulk]\n", + store::failReasonToString( + store::FailReason::FileTooLarge), + job.path.c_str()); continue; + } + + // Grammar availability, checked BEFORE reading the file: see + // the streaming path (engine_index_project.cpp) for the full + // reasoning — no wasted I/O, and the LanguageMissing reason is + // recorded instead of a misleading parse_null_tree. + { + auto lit = lang_ptrs.find(job.lang); + if (lit == lang_ptrs.end() || + lit->second == nullptr) { + store::bufferParseFailure( + project_id, job.path, job.lang, + store::failReasonToString( + store::FailReason:: + LanguageMissing)); + continue; + } + } std::string source = readFile(job.path.c_str()); - if (source.empty()) + if (source.empty()) { + store::bufferParseFailure( + project_id, job.path, job.lang, + store::failReasonToString( + store::FailReason::ReadEmpty)); continue; + } // Per-thread parser auto pit = tl_parsers.find(job.lang); if (pit == tl_parsers.end()) { + // Grammar availability was validated above (a null + // grammar never reaches this point), so the lookup only + // needs the presence check. auto lit = lang_ptrs.find(job.lang); - if (lit == lang_ptrs.end()) + if (lit == lang_ptrs.end()) { + store::bufferParseFailure( + project_id, job.path, job.lang, + store::failReasonToString( + store::FailReason:: + LanguageMissing)); continue; + } std::unique_ptr np( ts_parser_new()); ts_parser_set_language(np.get(), lit->second); @@ -158,8 +224,14 @@ char *engine_index_project_membulk( pit->second.get(), nullptr, source.c_str(), static_cast(source.size()))); - if (!tree) + if (!tree) { + store::bufferParseFailure( + project_id, job.path, job.lang, + store::failReasonToString( + store::FailReason:: + ParseNullTree)); continue; + } store::FileResult result; result.file_path = job.path; @@ -182,9 +254,39 @@ char *engine_index_project_membulk( } if (visitor) { - ir::SemanticUnit *su = visitor->visit( - tree.get(), source.c_str(), - job.path.c_str()); + ir::SemanticUnit *su = nullptr; + // The visitor transfers ownership of the returned + // unit to the caller (see js_visitor.h); without + // this guard every parsed file leaks one + // SemanticUnit. + std::unique_ptr su_guard; + // A visitor exception must not escape this worker + // thread: it would reach std::terminate and abort + // the whole process mid-index. The streaming path + // already catches these; this path previously did + // not, so any malformed file could kill the + // default (<=2000 files) index run. + try { + su = visitor->visit(tree.get(), + source.c_str(), + job.path.c_str()); + } catch (const std::exception &e) { + store::bufferParseFailure( + project_id, job.path, job.lang, + std::string(store::failReasonToString( + store::FailReason:: + VisitorException)) + + ": " + e.what()); + continue; + } catch (...) { + store::bufferParseFailure( + project_id, job.path, job.lang, + store::failReasonToString( + store::FailReason:: + VisitorUnknownThrow)); + continue; + } + su_guard.reset(su); if (su) { result.records = su->allRecords(); result.metrics = index_metrics:: @@ -198,12 +300,36 @@ char *engine_index_project_membulk( auto translator = ir::createTranslator(job.lang.c_str()); if (!translator) { + store::bufferParseFailure( + project_id, job.path, job.lang, + store::failReasonToString( + store::FailReason:: + LanguageMissing)); + continue; + } + // RAII ownership: the translator contract says + // the caller frees the unit (ir_translator.h). + std::unique_ptr unit; + try { + unit.reset(translator->translate( + tree.get(), source.c_str(), + job.path.c_str())); + } catch (const std::exception &e) { + store::bufferParseFailure( + project_id, job.path, job.lang, + std::string(store::failReasonToString( + store::FailReason:: + VisitorException)) + + ": " + e.what()); + continue; + } catch (...) { + store::bufferParseFailure( + project_id, job.path, job.lang, + store::failReasonToString( + store::FailReason:: + VisitorUnknownThrow)); continue; } - ir::TranslationUnit *unit = - translator->translate(tree.get(), - source.c_str(), - job.path.c_str()); if (unit) { // Convert TranslationUnit nodes to flat records. uint64_t flat_id = 1; @@ -242,7 +368,8 @@ char *engine_index_project_membulk( if (unit->root) flatten(unit->root, 0); result.metrics = index_metrics:: - computeMetricsFromUnit(unit); + computeMetricsFromUnit( + unit.get()); } } @@ -303,10 +430,18 @@ char *engine_index_project_membulk( // indexes and skip the per-file DELETE on a fresh DB. int total_indexed = static_cast(agg.size()); auto t_flush_start = steady_clock::now(); - if (!agg.flush(*g_store, project_id, is_reindex)) { + if (!agg.flush(*ctx->store, project_id, is_reindex)) { return dupString( "{\"ok\":false,\"error\":\"membulk flush failed\"}"); } + // Flush the parse failures this path buffered. The streaming path does + // this at its own end (engine_index_project.cpp), but memBulk RETURNS + // EARLY from the dispatcher and never reached that call — so for every + // project of <=2000 files (the common case) `parse_failures` stayed + // empty, and an unparseable file was dropped with no record anywhere. + // Runs after the bulk transaction (agg.flush) has closed, so the + // auxiliary connection does not contend with it. + store::flushParseFailures(ctx); auto t_flush_end = steady_clock::now(); fprintf(stderr, "engine: membulk_flush=%lldms (files=%d, is_reindex=%d) " @@ -321,7 +456,7 @@ char *engine_index_project_membulk( post_paths.reserve(job_lang.size()); for (const auto &pl : job_lang) post_paths.push_back(pl.first); - return engine_index_post_parse(project_id, dir, post_paths, filter, - is_reindex, mode_fast, mode_deep, - time_parse_ms, 0, total_indexed); + return postParsePhase(ctx, project_id, dir, post_paths, filter, + is_reindex, mode_fast, mode_deep, time_parse_ms, + 0, total_indexed); } diff --git a/engine/src/engine_internal.h b/engine/src/engine_internal.h index 96d0046..22392ca 100644 --- a/engine/src/engine_internal.h +++ b/engine/src/engine_internal.h @@ -24,20 +24,23 @@ #include "query/impact_analysis.h" // community_detection removed — Phase 0 cut #include "lsp/lsp_client.h" -// ─── Global Singletons ─────────────────────────────────────────── +// ─── Engine State (TD-1) ───────────────────────────────────────── // -// Shared across all engine_*.cpp translation units. -// Initialized by engine_init() and cleaned up by engine_shutdown(). -// Using unique_ptr for exception-safe memory management. +// The three state members (store, query, parser) live in the CodescopeEngine +// instance declared in engine_context.h. Since TD-1 knife 3 there is no +// process-global accessor: the instance arrives as the `engine_t` handle the +// FFI entry points take and is passed on to internal helpers as +// `EngineContext *ctx`. It is created by engine_create() and torn down by +// engine_destroy(). Using unique_ptr for exception-safe memory management. -extern std::unique_ptr g_store; -extern std::unique_ptr g_query; -extern std::unique_ptr g_parser; +#include "engine_context.h" // ═══════════════════════════════════════════════════════════════════ -// Global Singleton Thread-Safety Contract +// Engine State Thread-Safety Contract // ═══════════════════════════════════════════════════════════════════ -// g_store, g_query, g_parser are process-global unique_ptr singletons. +// One EngineContext instance is one engine: its store, query engine and +// parser. It is not shared between instances, but a few auxiliary subsystems +// still are process-wide (see engine_context.h for the list). // // Thread-safety model: // - The Rust MCP server calls FFI functions SEQUENTIALLY from a single @@ -69,36 +72,45 @@ std::string readFilePrealloc(const char *path, size_t known_size); // unchanged before the incremental skip. Computed on the file path so the // caller does not need to hold file bytes in memory. std::string fileContentHash(const char *path); -std::string jsonEscape(const std::string &s); std::string simpleHash(const std::string &s); const char *detectLanguage(const char *file_path); char *dupString(const std::string &s); +// engine_get_enhancement_status body. Lives in engine_queries_status.cpp +// (split out of engine_queries.cpp for the 1000-line rule); the FFI wrapper +// in engine_queries.cpp catches exceptions and formats the error envelope. +char *getEnhancementStatusImpl(EngineContext *ctx, uint64_t project_id); + // ─── Index Project: in-memory bulk path + shared post-parse ─────────── // // For small modules (<= kMemBulkFileThreshold files) the parse workers // aggregate FileResult in memory instead of pushing through BoundedQueue, // then flush once via insertFileResultBatch. The post-parse graph-building -// sequence is shared with the streaming path via engine_index_post_parse. +// sequence is shared with the streaming path via postParsePhase. /// In-memory bulk index path for small modules. +/// `known_failures` is the pre-loaded fail-fast skip set (files whose +/// fail_count reached CODESCOPE_FAIL_RETRY_MAX); the streaming path and this +/// one must both honour it or the documented "skipped entirely on the next +/// run" behaviour only applies to projects above the memBulk threshold. char *engine_index_project_membulk( - uint64_t project_id, const std::string &dir, uint64_t max_file_size, - const FilterPolicy &filter, + EngineContext *ctx, uint64_t project_id, const std::string &dir, + uint64_t max_file_size, const FilterPolicy &filter, const std::vector> &job_lang, const std::unordered_map &lang_ptrs, - bool is_reindex, bool mode_fast, bool mode_deep); + const std::unordered_set &known_failures, bool is_reindex, + bool mode_fast, bool mode_deep); /// Shared post-parse sequence: buildGraph -> callgraph_ready UPDATE -> -/// populateSymbols/resolveStagedMetrics -> (deep) vectors -> +/// resolveStagedMetrics -> (deep) vectors -> /// createIndexesAfterBulkLoad -> readiness -> result JSON. /// Returns a dupString()'d JSON result. Caller owns the pointer. -char *engine_index_post_parse(uint64_t project_id, const std::string &dir, - const std::vector &job_paths, - const FilterPolicy &filter, bool is_reindex, - bool mode_fast, bool mode_deep, - int64_t time_parse_ms, int64_t time_buildgraph_ms, - int total_indexed); +char *postParsePhase(EngineContext *ctx, uint64_t project_id, + const std::string &dir, + const std::vector &job_paths, + const FilterPolicy &filter, bool is_reindex, + bool mode_fast, bool mode_deep, int64_t time_parse_ms, + int64_t time_buildgraph_ms, int total_indexed); // ─── Evidence Builder FFI (v0.3 Phase 2) ───────────────────────── // @@ -114,7 +126,8 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, // "pattern"/"framework"/"ffi"); NULL or "" // means run all categories. // @return Heap-allocated JSON array string (caller frees). -char *engine_build_evidence(uint64_t project_id, const char *category_filter); +char *engine_build_evidence(engine_t handle, uint64_t project_id, + const char *category_filter); // ─── Path Helpers ───────────────────────────────────────────────── // Cross-platform path separator check: '/' on Unix, '/' and '\\' on diff --git a/engine/src/engine_lifecycle.cpp b/engine/src/engine_lifecycle.cpp index fb21695..6b8afee 100644 --- a/engine/src/engine_lifecycle.cpp +++ b/engine/src/engine_lifecycle.cpp @@ -21,14 +21,33 @@ extern "C" int sqlite3_vec_init(sqlite3 *db, char **pzErrMsg, // ─── Lifecycle ───────────────────────────────────────────────── -int engine_init(const char *db_path) +// Create one engine instance (TD-1 knife 3): the state used to live in a +// process-global accessor, and the caller now owns it through the returned +// handle. Raw `new` is confined to this FFI boundary (code_rules §2): the +// unique_ptr keeps every failure path exception-safe, and the final release() +// IS the documented ownership transfer to the caller. +engine_t engine_create(const char *db_path) { + if (!db_path || !*db_path) { + fprintf(stderr, + "[module=ffi, method=engine_create] db_path is required\n"); + return nullptr; + } + std::unique_ptr engine; + try { + engine = std::make_unique(); + } catch (const std::exception &e) { + fprintf(stderr, "[module=ffi, method=engine_create] %s\n", + e.what()); + return nullptr; + } catch (...) { + fprintf(stderr, + "[module=ffi, method=engine_create] unknown exception\n"); + return nullptr; + } + + EngineContext &ctx = *engine; try { - if (!db_path || !*db_path) { - fprintf(stderr, - "[module=ffi, method=engine_init] db_path is required\n"); - return -1; - } #ifdef HAVE_SQLITE_VEC // Register sqlite-vec once for the whole process. auto_extension fires on // every sqlite3_open, so the `vec0` module is available on the store's @@ -39,40 +58,42 @@ int engine_init(const char *db_path) // Use unique_ptr for exception-safe initialization // If any constructor throws, previous allocations are auto-freed. - g_store = std::make_unique(); + ctx.store = std::make_unique(); // Time GraphStore::open() (PRAGMAs + createSchema) so the // per-worker startup cost is visible in logs. createSchema runs // ~100+ CREATE TABLE/INDEX IF NOT EXISTS statements on a fresh // DB — significant in short-lived worker subprocesses. auto t_open_start = std::chrono::steady_clock::now(); - if (!g_store->open(db_path)) { + if (!ctx.store->open(db_path)) { fprintf(stderr, - "engine_init: open failed: %s [module=engine, method=engine_init]\n", - g_store ? g_store->error().c_str() : "(null)"); - g_store.reset(); // Auto-cleanup via unique_ptr - return -1; + "engine_create: open failed: %s [module=engine, method=engine_create]\n", + ctx.store ? ctx.store->error().c_str() : + "(null)"); + ctx.store.reset(); // Auto-cleanup via unique_ptr + return nullptr; } auto t_open_end = std::chrono::steady_clock::now(); fprintf(stderr, - "engine_init: GraphStore::open=%lldms " - "[module=engine, method=engine_init]\n", + "engine_create: GraphStore::open=%lldms " + "[module=engine, method=engine_create]\n", (long long)std::chrono::duration_cast< std::chrono::milliseconds>(t_open_end - t_open_start) .count()); - g_query = std::make_unique(g_store.get()); + ctx.query = + std::make_unique(ctx.store.get()); // Initialize parser and register available grammars - g_parser = std::make_unique(); + ctx.parser = std::make_unique(); // Register all statically-linked tree-sitter grammars. // Grammars are compiled into the binary — no .so loading needed. const char *langs[] = { "python", "cpp", "c", "rust", "javascript", "typescript", "tsx", "go", "java" }; for (auto lang : langs) { - g_parser->registerLanguage(lang); + ctx.parser->registerLanguage(lang); } // sqlite-vec is statically compiled into the binary — no runtime @@ -81,65 +102,81 @@ int engine_init(const char *db_path) // Note: vec0 embeddings table is no longer created — node_vectors // is the sole vector storage. searchSemantic reads node_vectors directly. - return 0; + return engine.release(); // ownership transfer to the caller } catch (const std::exception &e) { - fprintf(stderr, "[module=ffi, method=engine_init] %s\n", + fprintf(stderr, "[module=ffi, method=engine_create] %s\n", e.what()); - return -1; + return nullptr; } catch (...) { fprintf(stderr, - "[module=ffi, method=engine_init] unknown exception\n"); - return -1; + "[module=ffi, method=engine_create] unknown exception\n"); + return nullptr; } } -void engine_shutdown() +// Destroy one engine instance: the handle is consumed, and the destructor +// order is the one the managed state requires. +void engine_destroy(engine_t handle) { + // A null handle means engine_create() failed or already destroyed the + // instance; destroying nothing is a no-op so callers do not need to guard. + if (!handle) + return; + + // Rebuild the owning unique_ptr: this is the ownership transfer back from + // the caller (documented in engine.h), and it guarantees the instance is + // released exactly once even if a step below throws. + std::unique_ptr engine(handle); try { // Destruct in reverse construction order: - // constructed: g_store → g_query (depends on store) → g_parser (independent) - // destruct: g_parser → g_query → g_store - // This guarantees that g_query's destructor (which may issue SQLite calls) - // runs while g_store is still alive, and g_store is closed last. + // constructed: store → query (depends on store) → parser (independent) + // destruct: parser → query → store + // This guarantees that the query engine's destructor (which may issue + // SQLite calls) runs while the store is still alive, and the store is + // closed last. - // Clear the verifier registry BEFORE g_store is torn down: every - // registered Verifier holds a raw pointer to g_store, so dropping + // Clear the verifier registry BEFORE the store is torn down: every + // registered Verifier holds a raw pointer to the store, so dropping // them first avoids any dangling-pointer access during store close. verify::VerifierRegistry::instance().clear(); // Wait for the async knowledge builder to finish before destroying - // g_store. The builder thread dereferences g_store, so failing to - // join here would cause a use-after-free. + // the store. The builder thread dereferences it, so failing to join + // here would cause a use-after-free. joinAsyncKnowledgeBuilder(); - g_parser.reset(); // independent, safe to drop first - g_query.reset(); // may do SQLite work via g_store, destruct BEFORE store closes - if (g_store) { - g_store->close(); - g_store.reset(); + engine->parser.reset(); // independent, safe to drop first + // may issue SQLite work; drop before the store closes + engine->query.reset(); + if (engine->store) { + engine->store->close(); + engine->store.reset(); } } catch (const std::exception &e) { - fprintf(stderr, "[module=ffi, method=engine_shutdown] %s\n", + fprintf(stderr, "[module=ffi, method=engine_destroy] %s\n", e.what()); } catch (...) { fprintf(stderr, - "[module=ffi, method=engine_shutdown] unknown exception\n"); + "[module=ffi, method=engine_destroy] unknown exception\n"); } } // ─── Project ─────────────────────────────────────────────────── -uint64_t engine_create_project(const char *root_path, const char *name) +uint64_t engine_create_project(engine_t handle, const char *root_path, + const char *name) { + EngineContext *ctx = engineInstance(handle); + try { if (!root_path || !*root_path || !name || !*name) { fprintf(stderr, "[module=ffi, method=engine_create_project] root_path and name are required\n"); return 0; } - if (!g_store) + if (!ctx || !ctx->store) return 0; - uint64_t pid = g_store->createProject(root_path, name); + uint64_t pid = ctx->store->createProject(root_path, name); if (pid == 0) return 0; @@ -152,7 +189,7 @@ uint64_t engine_create_project(const char *root_path, const char *name) // future revision should key the registry by project_id. verify::VerifierRegistry::instance().clear(); verify::VerifierRegistry::instance().register_default_verifiers( - g_store.get(), pid); + ctx->store.get(), pid); return pid; } catch (const std::exception &e) { @@ -167,12 +204,14 @@ uint64_t engine_create_project(const char *root_path, const char *name) } } -uint64_t engine_get_latest_project_id() +uint64_t engine_get_latest_project_id(engine_t handle) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + if (!ctx || !ctx->store) return 0; - return g_store->getLatestProjectId(); + return ctx->store->getLatestProjectId(); } catch (const std::exception &e) { fprintf(stderr, "[module=ffi, method=engine_get_latest_project_id] %s\n", @@ -185,17 +224,19 @@ uint64_t engine_get_latest_project_id() } } -uint64_t engine_get_project_id_by_path(const char *root_path) +uint64_t engine_get_project_id_by_path(engine_t handle, const char *root_path) { + EngineContext *ctx = engineInstance(handle); + try { if (!root_path || !*root_path) { fprintf(stderr, "[module=ffi, method=engine_get_project_id_by_path] root_path is required\n"); return 0; } - if (!g_store) + if (!ctx || !ctx->store) return 0; - return g_store->getProjectId(root_path); + return ctx->store->getProjectId(root_path); } catch (const std::exception &e) { fprintf(stderr, "[module=ffi, method=engine_get_project_id_by_path] %s\n", @@ -208,12 +249,14 @@ uint64_t engine_get_project_id_by_path(const char *root_path) } } -uint64_t engine_get_project_node_count(uint64_t project_id) +uint64_t engine_get_project_node_count(engine_t handle, uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + if (!ctx || !ctx->store) return 0; - return g_store->getProjectNodeCount(project_id); + return ctx->store->getProjectNodeCount(project_id); } catch (const std::exception &e) { fprintf(stderr, "[module=ffi, method=engine_get_project_node_count] %s\n", diff --git a/engine/src/engine_project_state_ffi.cpp b/engine/src/engine_project_state_ffi.cpp index 518791e..ef8bf5c 100644 --- a/engine/src/engine_project_state_ffi.cpp +++ b/engine/src/engine_project_state_ffi.cpp @@ -15,10 +15,12 @@ // // Both functions return a heap-allocated JSON string that the caller // MUST release via engine_free_string(). On error, the returned JSON -// object contains an "error" field. Null `g_store` returns +// object contains an "error" field. A null handle returns // {"error":"engine not initialized"}. +#include "util/json_writer.h" #include "engine_internal.h" +#include "async_knowledge.h" #include "model/project_state_builder.h" #include "platform_win.h" @@ -34,16 +36,29 @@ /// @param project_id Project to analyze. /// @return Heap-allocated JSON string. On error returns a JSON /// object with an "error" field. -char *engine_build_project_state(uint64_t project_id) +char *engine_build_project_state(engine_t handle, uint64_t project_id) { - if (!g_store) - return dupString("{\"error\":\"engine not initialized\"}"); - model::ProjectStateBuilder builder(g_store.get()); - if (!builder.build(project_id)) { - return dupString( - "{\"error\":\"failed to build project state\"}"); + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) + return dupString( + "{\"error\":\"engine not initialized\"}"); + model::ProjectStateBuilder builder(ctx->store.get()); + if (!builder.build(project_id)) { + return dupString("{\"error\":\"failed to build project " + "state\"}"); + } + return dupString(builder.getSnapshotJson(project_id)); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_build_project_state", e.what())); + } catch (...) { + return dupString("{\"error\":\"[module=ffi, " + "method=engine_build_project_state] unknown " + "exception\"}"); } - return dupString(builder.getSnapshotJson(project_id)); } /// Get the persisted project state snapshot (without rebuilding). @@ -55,16 +70,30 @@ char *engine_build_project_state(uint64_t project_id) /// @return Heap-allocated JSON string. If no snapshot exists /// returns a JSON object with an "error" field and the /// project_id. -char *engine_get_project_state(uint64_t project_id) +char *engine_get_project_state(engine_t handle, uint64_t project_id) { - if (!g_store) - return dupString("{\"error\":\"engine not initialized\"}"); - model::ProjectStateBuilder builder(g_store.get()); - std::string snapshot = builder.getSnapshotJson(project_id); - if (snapshot.empty()) { - return dupString("{\"error\":\"project state not yet built\"," - "\"project_id\":" + - std::to_string(project_id) + "}"); + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) + return dupString( + "{\"error\":\"engine not initialized\"}"); + model::ProjectStateBuilder builder(ctx->store.get()); + std::string snapshot = builder.getSnapshotJson(project_id); + if (snapshot.empty()) { + return dupString( + "{\"error\":\"project state not yet built\"," + "\"project_id\":" + + std::to_string(project_id) + "}"); + } + return dupString(snapshot); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_project_state", e.what())); + } catch (...) { + return dupString("{\"error\":\"[module=ffi, " + "method=engine_get_project_state] unknown " + "exception\"}"); } - return dupString(snapshot); } diff --git a/engine/src/engine_queries.cpp b/engine/src/engine_queries.cpp index e9572f6..8225bb0 100644 --- a/engine/src/engine_queries.cpp +++ b/engine/src/engine_queries.cpp @@ -1,27 +1,15 @@ +#include "util/json_writer.h" +#include "engine_index_discover.h" #include "engine_internal.h" #include "model/semantic_fact_extractor.h" #include "async_knowledge.h" #include "platform_win.h" -#include -#include #include #include -#include -#include -#include -#include -#include -#include #include #include #include -#include -#include -#include -#include -#include -#include // ─── Constants ───────────────────────────────────────────────── // Above this node count, a name LIKE '%query%' fallback scan is too @@ -31,24 +19,28 @@ static constexpr int64_t kLargeProjectNodeThreshold = 100000; // ─── Phase A: engine_get_module_tree ────────────────────────── -char *engine_get_module_tree(uint64_t project_id) +static char *getModuleTreeImpl(EngineContext *ctx, uint64_t project_id) { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"engine not initialized\"}"); - return dupString(g_store->getModuleTreeJson(project_id)); + return dupString(ctx->store->getModuleTreeJson(project_id)); } // ─── Phase A: engine_find_symbol ────────────────────────────── -char *engine_find_symbol(uint64_t project_id, const char *symbol_name) +static char *findSymbolImpl(EngineContext *ctx, uint64_t project_id, + const char *symbol_name) { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"engine not initialized\"}"); if (!symbol_name || !*symbol_name) return dupString( "{\"error\":\"symbol_name is empty\",\"results\":[]}"); - std::string result = g_store->findSymbolJson(project_id, symbol_name); + std::string result = + ctx->store->findSymbolJson(project_id, symbol_name); // Check if empty and add smart hints if (result.find("\"results\":") != std::string::npos && @@ -59,7 +51,7 @@ char *engine_find_symbol(uint64_t project_id, const char *symbol_name) const char *lsql = "SELECT DISTINCT language || ',' FROM entity WHERE project_id = ? AND kind IN (0,1) LIMIT 5"; sqlite3_stmt *lstmt = nullptr; - if (sqlite3_prepare_v2(g_store->handle(), lsql, -1, &lstmt, + if (sqlite3_prepare_v2(ctx->store->handle(), lsql, -1, &lstmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64(lstmt, 1, static_cast(project_id)); @@ -81,7 +73,7 @@ char *engine_find_symbol(uint64_t project_id, const char *symbol_name) const char *csql = "SELECT COUNT(*) FROM entity WHERE project_id = ? AND kind IN (0,1)"; sqlite3_stmt *cstmt = nullptr; - if (sqlite3_prepare_v2(g_store->handle(), csql, -1, &cstmt, + if (sqlite3_prepare_v2(ctx->store->handle(), csql, -1, &cstmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64(cstmt, 1, static_cast(project_id)); @@ -90,18 +82,31 @@ char *engine_find_symbol(uint64_t project_id, const char *symbol_name) sqlite3_finalize(cstmt); } - // Check if this looks like a kernel project - bool is_kernel = (langs.find("c") != std::string::npos); + // Whole-token language membership. `langs` is a comma-joined + // list ("javascript", "c,cpp", …), so a plain substring find + // treats "javascript" as containing "c" and labels every such + // project a C/C++ kernel tree (T5 finding #11). + auto hasLangToken = [](const std::string &list, + const char *token) { + std::string padded = "," + list + ","; + std::string needle = std::string(",") + token + ","; + return padded.find(needle) != std::string::npos; + }; + const bool is_c_family = hasLangToken(langs, "c") || + hasLangToken(langs, "cpp") || + hasLangToken(langs, "c++"); // Check callgraph/enhancement readiness - double cg_ready = - g_store->getReadyRatio(project_id, "callgraph_ready"); - double emb_ready = - g_store->getReadyRatio(project_id, "embedding_ready"); + double cg_ready = ctx->store->getReadyRatio(project_id, + "callgraph_ready"); + double emb_ready = ctx->store->getReadyRatio(project_id, + "embedding_ready"); // Build smart message std::string hint = "{\"results\":[],\"hint\":{"; hint += "\"message\":\"No symbol named '" + - jsonEscape(std::string(symbol_name)) + "' found\","; - hint += "\"project_language\":\"" + jsonEscape(langs) + "\","; + util::jsonEscapeString(std::string(symbol_name)) + + "' found\","; + hint += "\"project_language\":\"" + + util::jsonEscapeString(langs) + "\","; hint += "\"total_symbols\":" + std::to_string(total) + ","; hint += "\"callgraph_ready\":" + std::to_string(cg_ready) + ","; hint += "\"embedding_ready\":" + std::to_string(emb_ready) + @@ -111,15 +116,21 @@ char *engine_find_symbol(uint64_t project_id, const char *symbol_name) hint += "Call graph is not ready — run codescope_enhance for deeper analysis. "; else if (total > 0) hint += "The symbol exists in the project but was not found by exact name match — try search or a different spelling. "; - hint += "\""; - if (total > 0 && is_kernel) { - hint += "\"suggestion\":\"This appears to be a C/C++ project. "; - // Check common kernel entry points + if (total > 0) { + // Close the note field and continue into `suggestion`. + // Emitting just `"` here left `"note":"..." "suggestion":…` + // — two adjacent strings with no comma, which is not + // valid JSON for the MCP client. + hint += "\","; + // Suggest the entry-point KINDS this project actually + // has. Never a canned symbol list: the old hard-coded + // kernel names (module_init/usb_register/probe) were + // wrong for every non-kernel project. std::string ep_hints; const char *epsql = "SELECT DISTINCT kind FROM entry_points WHERE project_id = ? LIMIT 5"; sqlite3_stmt *estmt = nullptr; - if (sqlite3_prepare_v2(g_store->handle(), epsql, -1, + if (sqlite3_prepare_v2(ctx->store->handle(), epsql, -1, &estmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64( estmt, 1, @@ -142,14 +153,25 @@ char *engine_find_symbol(uint64_t project_id, const char *symbol_name) ", ") == 0) ep_hints.erase(ep_hints.size() - 2); } + hint += "\"suggestion\":\"This is a " + + util::jsonEscapeString(langs) + " project. "; if (!ep_hints.empty()) { - hint += "Known entry point types: " + ep_hints + - ". "; - hint += "Try searching for 'probe', 'init', or a driver-specific function name."; + // entry_points.kind is free TEXT — escape it like + // every other interpolated value so a quote or + // backslash cannot break the JSON frame. + hint += "Entry point kinds present: " + + util::jsonEscapeString(ep_hints) + ". "; + hint += "Try one of those names, or use search."; + } else if (is_c_family) { + hint += "Try 'main' or an exported function name."; } else { - hint += "Possible entry points: module_init(), usb_register(), probe(), init()."; + hint += "Use search to look up a symbol by a different spelling."; } hint += "\""; + } else { + // No suggestion block: `note` is the last field, so + // close its value without a trailing comma. + hint += "\""; } hint += "}}"; return dupString(hint); @@ -170,13 +192,35 @@ char *engine_find_symbol(uint64_t project_id, const char *symbol_name) // // No re-parse, no re-translate, no regex extraction. -char *engine_enhance_project(uint64_t project_id) +static char *enhanceProjectImpl(EngineContext *ctx, uint64_t project_id) { - if (!g_store || !g_parser) + if (!ctx || !ctx->store || !ctx->parser) return dupString("{\"error\":\"engine not initialized\"}"); + // The background enrichment thread shares this connection and opens + // its own transactions (module_summary / module_edge / FTS / model + // tables). Wait for it before rebuilding the graph so the two never + // interleave BEGIN/COMMIT on one connection. No-op when finished. + // Join first (the builder holds g_store_mutex), then hold the store + // guard for the duration of the rebuild. + joinAsyncKnowledgeBuilder(); + auto _store_guard = waitForKnowledgeBuilder(); + using Clock = std::chrono::steady_clock; auto t_start = Clock::now(); + // Per-step timings + output counts for the response body. The catalog + // documents files_processed / symbols_enhanced / call_edges / timing + // breakdowns; the old body returned only {status,time_ms} (T5 #4). + int64_t t_semantic = 0, t_buildgraph = 0, t_fts = 0, t_metrics = 0, + t_model = 0; + int64_t semantic_facts = 0; + + // Honest failure reporting (2026-09-27 review D1-5): every step + // that can fail sets failed_step + failure_detail so the response below + // reports ok:false instead of claiming success over a rolled-back or + // truncated graph. Stays null when the whole pipeline succeeds. + const char *failed_step = nullptr; + std::string failure_detail; // Step 0.5: Extract semantic facts // @@ -189,20 +233,47 @@ char *engine_enhance_project(uint64_t project_id) // rule (Count combine mode that does not depend on semantic_facts). { auto t = Clock::now(); - g_store->beginTransaction(); - model::SemanticFactExtractor extractor(g_store.get()); - extractor.extractAll(project_id); - g_store->commitTransaction(); + ctx->store->beginTransaction(); + model::SemanticFactExtractor extractor(ctx->store.get()); + semantic_facts = extractor.extractAll(project_id); + ctx->store->commitTransaction(); + t_semantic = + std::chrono::duration_cast( + Clock::now() - t) + .count(); fprintf(stderr, "enhance: semantic_facts %lldms\n", - (long long)std::chrono::duration_cast< - std::chrono::milliseconds>(Clock::now() - t) - .count()); + (long long)t_semantic); + } + + // Step 0.6: knowledge documents + // + // Every index path must end with the same documentation state, but README + // ingestion used to happen only in the discovery walk and only for the + // directories it scanned. The parallel scheduler indexes module + // directories, so a project whose root holds no source files never had its + // root README ingested (six of the ten real projects checked answered + // detect_documentation_drift with nothing at all while a README was + // present), and the file-list path (force-index) ingested none whatsoever. + // Enhancing is the step every path runs last, so the root README is + // ingested here, before the knowledge layer reads capabilities out of it. + { + const int readme = + engine_index_discover::ingestProjectRootReadme( + ctx, project_id); + if (readme > 0) + fprintf(stderr, + "enhance: project README ingested as a knowledge document\n"); + else if (readme < 0) + fprintf(stderr, + "enhance: project README NOT ingested (unreadable project row, " + "unlistable root or store failure) " + "[module=engine_queries, method=engine_enhance_project]\n"); } // Step 1: buildGraph (skip if already finalized) { - int ready = g_store->getProjectReadiness(project_id, - "normal_ready"); + int ready = ctx->store->getProjectReadiness(project_id, + "normal_ready"); if (ready) { fprintf(stderr, "enhance: project %llu already finalized (semantic_facts re-extracted), " @@ -219,14 +290,18 @@ char *engine_enhance_project(uint64_t project_id) } { auto t = Clock::now(); - g_store->beginTransaction(); + ctx->store->beginTransaction(); // P2 fix: a resolver-pipeline failure makes buildGraph roll back its // graph savepoint and return false. Committing here would persist a // truncated graph and report success, so propagate the failure and // skip the graph-commit step (the outer enhance continues to the // model build below, which is independent of buildGraph). - if (!g_store->buildGraph(project_id, true)) { - g_store->rollbackTransaction(); + if (!ctx || !ctx->store->buildGraph(project_id, true)) { + ctx->store->rollbackTransaction(); + failed_step = "buildGraph"; + failure_detail = + "buildGraph failed; graph transaction rolled back, " + "the persisted graph may be truncated"; fprintf(stderr, "enhance: buildGraph failed for project %llu — " "skipping graph rebuild [module=engine, " @@ -234,37 +309,52 @@ char *engine_enhance_project(uint64_t project_id) (unsigned long long)project_id); goto run_model_build; } - g_store->commitTransaction(); + ctx->store->commitTransaction(); + t_buildgraph = + std::chrono::duration_cast( + Clock::now() - t) + .count(); fprintf(stderr, "enhance: buildGraph %lldms\n", - (long long)std::chrono::duration_cast< - std::chrono::milliseconds>(Clock::now() - t) - .count()); + (long long)t_buildgraph); } // Step 2: Build FTS (symbols no longer synced — graph_nodes is canonical) { auto t = Clock::now(); - g_store->buildFTSFromGraph(project_id); - fprintf(stderr, "enhance: buildFTS %lldms\n", - (long long)std::chrono::duration_cast< - std::chrono::milliseconds>(Clock::now() - t) - .count()); + ctx->store->buildFTSFromGraph(project_id); + if (!ctx || !ctx->store->error().empty()) { + failed_step = "buildFTSFromGraph"; + failure_detail = ctx->store->error(); + fprintf(stderr, + "enhance: buildFTS failed: %s " + "[module=engine, method=engine_enhance_project]\n", + ctx->store->error().c_str()); + // Leave fts_ready unset so search does not take the + // FTS path against an incomplete index. + goto run_model_build; + } + t_fts = std::chrono::duration_cast( + Clock::now() - t) + .count(); + fprintf(stderr, "enhance: buildFTS %lldms\n", (long long)t_fts); } // Step 5: Resolve pre-computed metrics { auto t = Clock::now(); - g_store->resolveStagedMetrics(project_id); + ctx->store->resolveStagedMetrics(project_id); + t_metrics = + std::chrono::duration_cast( + Clock::now() - t) + .count(); fprintf(stderr, "enhance: resolveMetrics %lldms\n", - (long long)std::chrono::duration_cast< - std::chrono::milliseconds>(Clock::now() - t) - .count()); + (long long)t_metrics); } // Finalize - g_store->createIndexesAfterBulkLoad(project_id); - g_store->setProjectReadiness(project_id, "normal_ready", 1); - g_store->setProjectReadiness(project_id, "fts_ready", 1); + ctx->store->createIndexesAfterBulkLoad(project_id); + ctx->store->setProjectReadiness(project_id, "normal_ready", 1); + ctx->store->setProjectReadiness(project_id, "fts_ready", 1); run_model_build: // ── Model building (module_summary, architecture_edge, etc.) ── @@ -273,19 +363,13 @@ char *engine_enhance_project(uint64_t project_id) // was done with SKIP_ASYNC=1. { auto t = Clock::now(); - runModelIndexSync(*g_store, project_id, true); - fprintf(stderr, "enhance: runModelIndexSync %lldms\n", - (long long)std::chrono::duration_cast< - std::chrono::milliseconds>(Clock::now() - t) - .count()); - } - { - auto t = Clock::now(); - buildKnowledgeGraphSync(*g_store, project_id); - fprintf(stderr, "enhance: buildKnowledgeGraphSync %lldms\n", - (long long)std::chrono::duration_cast< - std::chrono::milliseconds>(Clock::now() - t) - .count()); + runModelIndexSync(*ctx->store, project_id, true); + buildKnowledgeGraphSync(*ctx->store, project_id); + t_model = std::chrono::duration_cast( + Clock::now() - t) + .count(); + fprintf(stderr, "enhance: model/knowledge %lldms\n", + (long long)t_model); } int64_t total_ms = @@ -294,226 +378,59 @@ char *engine_enhance_project(uint64_t project_id) .count(); fprintf(stderr, "enhance: done %lldms total\n", (long long)total_ms); - std::ostringstream json; - json << "{" - << "\"status\":\"ok\"" - << ",\"time_ms\":" << total_ms << "}"; - return dupString(json.str()); -} - -// ─── Phase B: engine_get_enhancement_status ──────────────────── - -// Helper: count eligible function/method entities (entity.kind IN 0,1) for a -// project — the denominator for every capability coverage ratio. Reads the -// canonical `entity` table. Returns 0 on any error. -static int queries_count_eligible_entities(sqlite3 *db, uint64_t project_id) -{ - sqlite3_stmt *stmt = nullptr; - int total = 0; - const char *sql = - "SELECT COUNT(*) FROM entity WHERE project_id=? AND kind IN (0,1)"; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) - total = sqlite3_column_int(stmt, 0); - sqlite3_finalize(stmt); - } else { - fprintf(stderr, - "engine_get_enhancement_status: entity count probe failed: %s " - "[module=queries, method=engine_get_enhancement_status]\n", - sqlite3_errmsg(db)); - } - return total; -} - -// Helper: count distinct function/method entities that participate in at least -// one Calls relation (relation.type=1). This is the canonical callgraph-ready -// count. Returns 0 on any error. -static int queries_count_callgraph_ready(sqlite3 *db, uint64_t project_id) -{ - sqlite3_stmt *stmt = nullptr; - int ready = 0; - const char *sql = - "SELECT COUNT(*) FROM (" - " SELECT DISTINCT src FROM (" - " SELECT source_id AS src FROM relation WHERE project_id=? AND type=1" - " UNION" - " SELECT target_id AS src FROM relation WHERE project_id=? AND type=1" - " )" - ")"; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_int64(stmt, 2, static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) - ready = sqlite3_column_int(stmt, 0); - sqlite3_finalize(stmt); - } else { - fprintf(stderr, - "engine_get_enhancement_status: callgraph count probe failed: %s " - "[module=queries, method=engine_get_enhancement_status]\n", - sqlite3_errmsg(db)); - } - return ready; -} - -// Helper: count node_vectors rows for a project — the canonical embedding -// coverage count. Returns 0 if the table is missing or empty. Used both for -// the embedding_ready count and to guard the project_readiness.vector_ready -// flag against the A19 "fake ready" regression. -static int64_t queries_count_node_vectors(sqlite3 *db, uint64_t project_id) -{ - sqlite3_stmt *stmt = nullptr; - int64_t ready = 0; - const char *sql = - "SELECT COUNT(*) FROM node_vectors WHERE project_id=?"; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) - ready = sqlite3_column_int64(stmt, 0); - sqlite3_finalize(stmt); - } else { - // node_vectors table may not exist on legacy DBs — log and treat - // as 0 (readiness tracks canonical data; an absent table means 0). - fprintf(stderr, - "engine_get_enhancement_status: node_vectors count probe failed: %s " - "[module=queries, method=engine_get_enhancement_status]\n", - sqlite3_errmsg(db)); - } - return ready; -} - -// Helper: count function/method entities that carry resolved code metrics -// (cyclomatic > 0), i.e. the canonical metrics_ready count. metrics are -// resolved onto entity by resolveStagedMetrics after buildGraph, so this -// probe reflects real producer output — never a placeholder. Returns 0 on -// any error or on a pre-metrics database. -static int queries_count_metrics_ready(sqlite3 *db, uint64_t project_id) -{ - sqlite3_stmt *stmt = nullptr; - int ready = 0; - const char *sql = - "SELECT COUNT(*) FROM entity " - "WHERE project_id = ? AND kind IN (0,1) AND cyclomatic > 0"; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) - ready = sqlite3_column_int(stmt, 0); - sqlite3_finalize(stmt); - } else { - fprintf(stderr, - "engine_get_enhancement_status: metrics count probe failed: %s " - "[module=queries, method=engine_get_enhancement_status]\n", - sqlite3_errmsg(db)); - } - return ready; -} - -// Helper: compute coverage ratio as a JSON-friendly string in [0.0, 1.0]. -// Returns "0.0" when eligible == 0 to avoid divide-by-zero. -static std::string queries_coverage_ratio(int ready, int eligible) -{ - if (eligible <= 0) - return "0.0"; - char buf[32]; - std::snprintf(buf, sizeof(buf), "%.4f", - static_cast(ready) / - static_cast(eligible)); - return std::string(buf); -} - -char *engine_get_enhancement_status(uint64_t project_id) -{ - if (!g_store) - return dupString("{\"error\":\"engine not initialized\"}"); - - auto db = g_store->handle(); - - // v0.2.5: report real counts from canonical data (entity / relation / - // node_vectors). The legacy int fields (total_symbols/callgraph_ready/ - // metrics_ready/embedding_ready) are preserved at the start of the JSON - // so existing MCP clients / sscanf parsers keep working; the richer - // `capabilities` block carries eligible/ready/coverage/ - // producer_version so callers can tell "not yet run" from "not built". - const int total = queries_count_eligible_entities(db, project_id); - const int cg_ready = queries_count_callgraph_ready(db, project_id); - const int64_t vec_rows = queries_count_node_vectors(db, project_id); - // metrics_ready comes from canonical data: entity rows (kind 0/1) whose - // cyclomatic was resolved by resolveStagedMetrics. It is a real count — - // the metrics producer was restored in v0.2.5, so a fresh index produces - // a positive value, while a pre-metrics DB reports 0 honestly. - const int metrics_ready = queries_count_metrics_ready(db, project_id); - const int embedding_ready = static_cast(vec_rows); - - // fts_ready is read from project_readiness (set by the async path). - const int fts_ready = - g_store->getProjectReadiness(project_id, "fts_ready"); - - // coverage ratios — real numbers in [0.0, 1.0], never a placeholder 0. - const std::string cg_coverage = queries_coverage_ratio(cg_ready, total); - const std::string metrics_coverage = - queries_coverage_ratio(metrics_ready, total); - const std::string embedding_coverage = - queries_coverage_ratio(embedding_ready, total); + // Post-pass counts from canonical tables: what the run actually + // produced, not a per-step tally (the index pipeline's per-file + // counts are not visible from here). + auto countRows = [&](const char *sql) -> int64_t { + sqlite3_stmt *st = nullptr; + int64_t n = 0; + if (sqlite3_prepare_v2(ctx->store->handle(), sql, -1, &st, + nullptr) == SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int64(st, 0); + sqlite3_finalize(st); + } + return n; + }; + const int64_t symbols = + countRows("SELECT COUNT(*) FROM entity WHERE project_id=?"); + const int64_t call_edges = countRows( + "SELECT COUNT(*) FROM relation WHERE project_id=? AND type=1"); + const int64_t files = + countRows("SELECT COUNT(DISTINCT file_path) FROM entity " + "WHERE project_id=?"); std::ostringstream json; - // Legacy int fields — kept stable for backward-compat parsers. - json << "{" - << "\"total_symbols\":" << total << "," - << "\"callgraph_ready\":" << cg_ready << "," - << "\"metrics_ready\":" << metrics_ready << "," - << "\"embedding_ready\":" << embedding_ready - << "," - // Richer per-capability block: eligible/ready/failed/coverage + - // producer_version + unavailable_reason. NO hardcoded 0 — every - // count comes from a canonical table probe above. - << "\"capabilities\":{" - << "\"callgraph\":{" - << "\"available\":true," - << "\"ready\":" << (cg_ready > 0 ? "true" : "false") << "," - << "\"eligible\":" << total << "," - << "\"ready_count\":" << cg_ready << "," - << "\"failed\":0," - << "\"coverage\":" << cg_coverage << "," - << "\"producer_version\":\"buildGraph\"" - << "}," - << "\"metrics\":{" - << "\"available\":true," - << "\"ready\":" << (metrics_ready > 0 ? "true" : "false") << "," - << "\"eligible\":" << total << "," - << "\"ready_count\":" << metrics_ready << "," - << "\"failed\":0," - << "\"coverage\":" << metrics_coverage << "," - << "\"producer_version\":\"resolveStagedMetrics\"" - << "}," - << "\"embedding\":{" - << "\"available\":true," - << "\"ready\":" << (embedding_ready > 0 ? "true" : "false") << "," - << "\"eligible\":" << total << "," - << "\"ready_count\":" << embedding_ready << "," - << "\"failed\":0," - << "\"coverage\":" << embedding_coverage << "," - << "\"producer_version\":\"buildVectorsFromGraph\"" - << "}," - << "\"semantic_search\":{" - << "\"available\":true," - << "\"ready\":" << (embedding_ready > 0 ? "true" : "false") << "," - << "\"mode\":\"ngram_hash\"," - << "\"description\":\"n-gram hash vector lexical similarity " - "(restored in v0.2.5); complements FTS exact search\"" - << "}," - << "\"fts\":{" - << "\"available\":true," - << "\"ready\":" << (fts_ready ? "true" : "false") << "}" - << "}" - << "}"; + json << "{"; + if (failed_step) { + json << "\"ok\":false,\"status\":\"failed\"" + << ",\"failed_step\":\"" << failed_step << "\"" + << ",\"error\":\"" + << util::jsonEscapeString(failure_detail) << "\""; + } else { + json << "\"ok\":true,\"status\":\"ok\""; + } + json << ",\"time_ms\":" << total_ms << ",\"files_processed\":" << files + << ",\"symbols_enhanced\":" << symbols + << ",\"call_edges\":" << call_edges + << ",\"semantic_facts\":" << semantic_facts << ",\"timing\":{" + << "\"semantic_facts_ms\":" << t_semantic << "," + << "\"buildgraph_ms\":" << t_buildgraph << "," + << "\"fts_ms\":" << t_fts << "," + << "\"metrics_ms\":" << t_metrics << "," + << "\"model_ms\":" << t_model << "}}"; return dupString(json.str()); } // ─── Phase C: Unified Search (adaptive FTS / semantic) ─────── -char *engine_unified_search(uint64_t project_id, const char *query, int limit) +static char *unifiedSearchImpl(EngineContext *ctx, uint64_t project_id, + const char *query, int limit) { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"engine not initialized\"}"); if (!query || !*query) return dupString( @@ -522,11 +439,12 @@ char *engine_unified_search(uint64_t project_id, const char *query, int limit) limit = 20; // Check if FTS index is ready; if not, fall back to graph-based search - int fts_ready = g_store->getProjectReadiness(project_id, "fts_ready"); + int fts_ready = + ctx->store->getProjectReadiness(project_id, "fts_ready"); if (fts_ready) { // FTS is ready — use full-text search - return dupString( - g_store->searchUnifiedJson(project_id, query, limit)); + return dupString(ctx->store->searchUnifiedJson(project_id, + query, limit)); } // FTS not ready — check project size before falling back to the @@ -539,7 +457,7 @@ char *engine_unified_search(uint64_t project_id, const char *query, int limit) sqlite3_stmt *stmt = nullptr; const char *sql = "SELECT COUNT(*) FROM entity WHERE project_id = ?"; - if (sqlite3_prepare_v2(g_store->handle(), sql, -1, &stmt, + if (sqlite3_prepare_v2(ctx->store->handle(), sql, -1, &stmt, nullptr) != SQLITE_OK) { // Prepare failed — cannot determine node count. Log and // fall through to the fallback (let it run; the user gets @@ -548,10 +466,10 @@ char *engine_unified_search(uint64_t project_id, const char *query, int limit) fprintf(stderr, "unified_search: COUNT prepare failed: %s " "[module=engine, method=unified_search]\n", - sqlite3_errmsg(g_store->handle())); + sqlite3_errmsg(ctx->store->handle())); if (stmt) sqlite3_finalize(stmt); - return dupString(g_store->searchGraphFallback( + return dupString(ctx->store->searchGraphFallback( project_id, query, limit)); } sqlite3_bind_int64(stmt, 1, static_cast(project_id)); @@ -576,94 +494,104 @@ char *engine_unified_search(uint64_t project_id, const char *query, int limit) // Small project — the fallback scan is fast enough. return dupString( - g_store->searchGraphFallback(project_id, query, limit)); + ctx->store->searchGraphFallback(project_id, query, limit)); } // ─── Phase C: Adaptive Find Callers ────────────────────────── -char *engine_find_callers_adaptive(uint64_t project_id, const char *symbol_name, - const char *file_filter) +static char *findCallersAdaptiveImpl(EngineContext *ctx, uint64_t project_id, + const char *symbol_name, + const char *file_filter) { // v0.2.5: getCallers has its own SQLite/SQLite backend, so the // graph-not-ready guard only requires the SQLite handle (works on // SQLite-only/Windows builds). The [module=engine_queries, // method=find_callers_adaptive] tag is kept so callers can still // distinguish an indexing-pending state from a query error. - if (!g_query || !g_store || !g_store->handle()) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query || !ctx->store || !ctx->store->handle()) return dupString("{\"error\":\"graph not ready [module=engine_" "queries, method=find_callers_adaptive]\"}"); if (!symbol_name || !*symbol_name) return dupString("{\"error\":\"symbol_name is empty\"}"); return dupString( - g_query->getCallers(project_id, symbol_name, file_filter)); + ctx->query->getCallers(project_id, symbol_name, file_filter)); } // ─── Phase C: Adaptive Find Callees ────────────────────────── -char *engine_find_callees_adaptive(uint64_t project_id, const char *symbol_name, - const char *file_filter) +static char *findCalleesAdaptiveImpl(EngineContext *ctx, uint64_t project_id, + const char *symbol_name, + const char *file_filter) { // v0.2.5: getCallees has its own SQLite/SQLite backend, so the // graph-not-ready guard only requires the SQLite handle (works on // SQLite-only/Windows builds). - if (!g_query || !g_store || !g_store->handle()) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query || !ctx->store || !ctx->store->handle()) return dupString("{\"error\":\"graph not ready [module=engine_" "queries, method=find_callees_adaptive]\"}"); if (!symbol_name || !*symbol_name) return dupString("{\"error\":\"symbol_name is empty\"}"); return dupString( - g_query->getCallees(project_id, symbol_name, file_filter)); + ctx->query->getCallees(project_id, symbol_name, file_filter)); } // ─── Step 7 (plan §7.2): Entity-precise caller/callee queries ──── -char *engine_find_callers_by_entity(uint64_t project_id, uint64_t entity_id) +static char *findCallersByEntityImpl(EngineContext *ctx, uint64_t project_id, + uint64_t entity_id) { // v0.2.5: getCallersByEntity has its own SQLite/SQLite backend, so // the graph-not-ready guard is only required on the SQLite path and // is enforced inside that backend; here we only guard the store handle. - if (!g_query || !g_store || !g_store->handle()) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query || !ctx->store || !ctx->store->handle()) return dupString("{\"error\":\"graph not ready [module=engine_" "queries, method=find_callers_by_entity]\"}"); if (entity_id == 0) return dupString("{\"error\":\"entity_id is 0\"}"); - return dupString(g_query->getCallersByEntity(project_id, entity_id)); + return dupString(ctx->query->getCallersByEntity(project_id, entity_id)); } -char *engine_find_callees_by_entity(uint64_t project_id, uint64_t entity_id) +static char *findCalleesByEntityImpl(EngineContext *ctx, uint64_t project_id, + uint64_t entity_id) { // v0.2.5: getCalleesByEntity has its own SQLite/SQLite backend; the // guard here only requires the SQLite handle (works on SQLite-only). - if (!g_query || !g_store || !g_store->handle()) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query || !ctx->store || !ctx->store->handle()) return dupString("{\"error\":\"graph not ready [module=engine_" "queries, method=find_callees_by_entity]\"}"); if (entity_id == 0) return dupString("{\"error\":\"entity_id is 0\"}"); - return dupString(g_query->getCalleesByEntity(project_id, entity_id)); + return dupString(ctx->query->getCalleesByEntity(project_id, entity_id)); } // ─── Phase C: Get Entry Points (new schema) ────────────────── -char *engine_get_entry_points_new(uint64_t project_id) +static char *getEntryPointsNewImpl(EngineContext *ctx, uint64_t project_id) { // SQLite is the only data source. graph-not-ready is reported with // the [module=engine_queries, method=get_entry_points_new] tag. // v0.2.5: getEntryPoints has its own SQLite/SQLite backend; the guard // here only requires the SQLite handle (works on SQLite-only). - if (!g_query || !g_store || !g_store->handle()) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->query || !ctx->store || !ctx->store->handle()) return dupString("{\"error\":\"graph not ready [module=engine_" "queries, method=get_entry_points_new]\"}"); - return dupString(g_query->getEntryPoints(project_id)); + return dupString(ctx->query->getEntryPoints(project_id)); } // ─── Phase C: Project Overview ─────────────────────────────── -char *engine_project_overview(uint64_t project_id) +static char *projectOverviewImpl(EngineContext *ctx, uint64_t project_id) { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"engine not initialized\"}"); - auto db = g_store->handle(); + auto db = ctx->store->handle(); std::ostringstream json; // ── Project info ── @@ -686,17 +614,21 @@ char *engine_project_overview(uint64_t project_id) first = false; const char *l = reinterpret_cast( sqlite3_column_text(stmt, 0)); - json << "\"" << jsonEscape(l ? l : "") << "\""; + json << "\"" + << util::jsonEscapeString(l ? l : "") + << "\""; } sqlite3_finalize(stmt); } json << "],"; } - // Module count + // Module count. `scope` kind=1 is the canonical module registry the + // module tree is built from; the `modules` table is a coarser summary + // that under-counts on large indexes (T5 finding #13). { const char *sql = - "SELECT COUNT(*) FROM modules WHERE project_id = ?"; + "SELECT COUNT(*) FROM scope WHERE project_id = ? AND kind = 1"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK) { @@ -709,40 +641,93 @@ char *engine_project_overview(uint64_t project_id) } } - // Symbol count + analysis state breakdown (via entity) + // Symbol count + analysis state breakdown. The previous SQL selected + // COUNT(*) four times, so scanned/callgraph/metrics/embedding were + // always identical placeholders. Each column is now a real + // readiness probe (same definitions as get_enhancement_status). { - const char *sql = "SELECT COUNT(*), " - "COUNT(*), " - "COUNT(*), " - "COUNT(*) " - "FROM entity e " - "WHERE e.project_id = ? AND e.kind IN (0,1)"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(stmt, 1, - static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) { - json << "\"total_symbols\":" - << sqlite3_column_int(stmt, 0) << ","; - json << "\"analysis_progress\":{" - << "\"scanned\":" - << sqlite3_column_int(stmt, 0) << "," - << "\"callgraph\":" - << sqlite3_column_int(stmt, 2) << "," - << "\"metrics\":" - << sqlite3_column_int(stmt, 3) << "," - << "\"embedding\":" - << sqlite3_column_int(stmt, 4) << "},"; + auto countWith = [&](const char *where) -> int64_t { + std::string sql = "SELECT COUNT(*) FROM entity " + "WHERE project_id = ? AND " + + std::string(where); + sqlite3_stmt *s = nullptr; + int64_t n = 0; + if (sqlite3_prepare_v2(db, sql.c_str(), -1, &s, + nullptr) == SQLITE_OK) { + sqlite3_bind_int64( + s, 1, static_cast(project_id)); + if (sqlite3_step(s) == SQLITE_ROW) + n = sqlite3_column_int64(s, 0); + sqlite3_finalize(s); + } + return n; + }; + const int64_t eligible = countWith("kind IN (0,1)"); + // Callgraph-ready: entities on at least one Calls edge. Needs + // three project_id binds (outer + both UNION arms), so it is + // not expressed through countWith's single-bind helper. + int64_t cg_ready = 0; + { + sqlite3_stmt *s = nullptr; + const char *cg_sql = + "SELECT COUNT(*) FROM entity WHERE " + "project_id = ? AND kind IN (0,1) AND id IN (" + " SELECT source_id FROM relation WHERE " + "project_id = ? AND type=1 " + " UNION SELECT target_id FROM relation WHERE " + "project_id = ? AND type=1)"; + if (sqlite3_prepare_v2(db, cg_sql, -1, &s, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64( + s, 1, static_cast(project_id)); + sqlite3_bind_int64( + s, 2, static_cast(project_id)); + sqlite3_bind_int64( + s, 3, static_cast(project_id)); + if (sqlite3_step(s) == SQLITE_ROW) + cg_ready = sqlite3_column_int64(s, 0); + sqlite3_finalize(s); } - sqlite3_finalize(stmt); } + const int64_t metrics_ready = + countWith("kind IN (0,1) AND cyclomatic > 0"); + int64_t embedding_ready = 0; + { + sqlite3_stmt *s = nullptr; + const char *vsql = + "SELECT COUNT(*) FROM node_vectors WHERE " + "project_id = ?"; + if (sqlite3_prepare_v2(db, vsql, -1, &s, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64( + s, 1, static_cast(project_id)); + if (sqlite3_step(s) == SQLITE_ROW) + embedding_ready = + sqlite3_column_int64(s, 0); + sqlite3_finalize(s); + } + } + json << "\"total_symbols\":" << eligible << ","; + json << "\"analysis_progress\":{" + << "\"scanned\":" << eligible << "," + << "\"callgraph\":" << cg_ready << "," + << "\"metrics\":" << metrics_ready << "," + << "\"embedding\":" << embedding_ready << "},"; } // Entry points { - std::string ep = g_store->getEntryPointsJson(project_id); - // ep already has {"entry_points": [...]} + // Source of truth is the query engine (name whitelist over the + // canonical `entity` table), the same call the get_entry_points tool + // makes. This used to go through GraphStore::getEntryPointsJson, + // which joins the LEGACY `graph_nodes` table — empty in the canonical + // pipeline — so `entry_points` was `{"entry_points":[],"total":0}` in + // every overview ever returned, while get_entry_points listed real + // entries for the same project. + std::string ep = + ctx->query ? ctx->query->getEntryPoints(project_id) : + std::string(); + // ep already has {"entry_points": [...], "total": N} if (!ep.empty() && ep[0] == '{') { json << "\"entry_points\":" << ep.c_str() << ","; } @@ -751,11 +736,12 @@ char *engine_project_overview(uint64_t project_id) // Ready features (which analysis features are complete for >50% of symbols) { json << "\"ready_features\":{"; - double cg = - g_store->getReadyRatio(project_id, "callgraph_ready"); - double me = g_store->getReadyRatio(project_id, "metrics_ready"); - double em = - g_store->getReadyRatio(project_id, "embedding_ready"); + double cg = ctx->store->getReadyRatio(project_id, + "callgraph_ready"); + double me = + ctx->store->getReadyRatio(project_id, "metrics_ready"); + double em = ctx->store->getReadyRatio(project_id, + "embedding_ready"); json << "\"call_graph\":" << (cg > 0.5 ? "true" : "false") << "," << "\"metrics\":" << (me > 0.5 ? "true" : "false") << "," @@ -767,690 +753,193 @@ char *engine_project_overview(uint64_t project_id) return dupString(json.str()); } -// ─── Path Tracing ────────────────────────────────────────────── +// ─── FFI boundary wrappers ─────────────────────────────────────── +// The bodies above are static implementations (`*Impl`). Every extern "C" +// entry point is a thin try/catch wrapper: no C++ exception may cross the C +// ABI boundary, because the MCP server is long-running and an escaping +// exception would terminate the whole session. Error envelopes carry a +// [module=ffi, method=] tag per code_rules.md. -char *engine_trace_path(uint64_t project_id, const char *from_name, - const char *to_name) +char *engine_get_module_tree(engine_t handle, uint64_t project_id) { - // Trace a path from from_function to to_function using the SQLite - // shortest-path backend (QueryEngine::findShortestPath, CSR BFS) and - // hydrate the node ids from the canonical entity table. The legacy - // tracePathJson output schema is preserved: - // {"path":[{"name":"...","file":"...","line":N}, ...]} - // {"path":[],"error":"..."} - // {"path":[{"name":"..."}],"trivial":true} - if (!from_name || !*from_name || !to_name || !*to_name) - return dupString( - "{\"error\":\"empty symbol name\",\"path\":[]}"); - if (!g_store || !g_store->handle()) { - return dupString("{\"error\":\"graph not ready [module=" - "engine_queries, method=trace_path]\"," - "\"path\":[]}"); - } - sqlite3 *db = g_store->handle(); - auto resolveName = [&](const char *name, uint64_t &out_id) -> bool { - const char *sql = "SELECT id FROM entity WHERE project_id=? " - "AND name=? ORDER BY id LIMIT 1"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) != SQLITE_OK) - return false; - sqlite3_bind_int64(st, 1, static_cast(project_id)); - sqlite3_bind_text(st, 2, name, -1, SQLITE_TRANSIENT); - bool ok = false; - if (sqlite3_step(st) == SQLITE_ROW) { - out_id = static_cast( - sqlite3_column_int64(st, 0)); - ok = true; - } - sqlite3_finalize(st); - return ok; - }; - uint64_t from_id = 0, to_id = 0; - if (!resolveName(from_name, from_id) || !resolveName(to_name, to_id)) - return dupString( - "{\"path\":[],\"error\":\"symbol not found\"}"); - std::string bfs_json = - g_query->findShortestPath(project_id, from_id, to_id); - bool found = bfs_json.find("\"found\":true") != std::string::npos; - if (!found) - return dupString("{\"path\":[],\"error\":\"no path found\"}"); - std::vector node_ids; - { - const std::string needle = "\"node_id\":"; - size_t pos = 0; - while ((pos = bfs_json.find(needle, pos)) != - std::string::npos) { - pos += needle.size(); - while (pos < bfs_json.size() && - (bfs_json[pos] == ' ' || bfs_json[pos] == '\t')) - ++pos; - std::string num; - while (pos < bfs_json.size() && - std::isdigit(static_cast( - bfs_json[pos]))) { - num += bfs_json[pos++]; - } - if (!num.empty()) - node_ids.push_back(static_cast( - std::strtoull(num.c_str(), nullptr, - 10))); - } - } - if (node_ids.empty()) - return dupString("{\"path\":[],\"error\":\"no path found\"}"); - std::unordered_map> - lookup; - { - std::string id_list; - for (size_t i = 0; i < node_ids.size(); ++i) { - if (i > 0) - id_list += ","; - id_list += std::to_string(node_ids[i]); - } - std::string sql = "SELECT id, name, file_path, start_row " - "FROM entity WHERE project_id=? AND id IN (" + - id_list + ")"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db, sql.c_str(), -1, &st, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(st, 1, - static_cast(project_id)); - while (sqlite3_step(st) == SQLITE_ROW) { - uint64_t id = static_cast( - sqlite3_column_int64(st, 0)); - std::string name = - reinterpret_cast( - sqlite3_column_text(st, 1)) ? - reinterpret_cast( - sqlite3_column_text( - st, 1)) : - ""; - std::string file = - reinterpret_cast( - sqlite3_column_text(st, 2)) ? - reinterpret_cast( - sqlite3_column_text( - st, 2)) : - ""; - int line = sqlite3_column_int(st, 3); - lookup.emplace(id, std::make_tuple(name, file, - line)); - } - sqlite3_finalize(st); - } - } - std::ostringstream json; - json << "{\"path\":["; - bool first = true; - for (uint64_t id : node_ids) { - if (!first) - json << ","; - first = false; - auto it = lookup.find(id); - if (it == lookup.end()) { - json << "{\"name\":\"?\",\"file\":\"\",\"line\":0}"; - } else { - const auto &tup = it->second; - json << "{\"name\":\"" << jsonEscape(std::get<0>(tup)) - << "\",\"file\":\"" << jsonEscape(std::get<1>(tup)) - << "\",\"line\":" << std::get<2>(tup) << "}"; - } + EngineContext *ctx = engineInstance(handle); + + try { + return getModuleTreeImpl(ctx, project_id); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_module_tree", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_module_tree", "unknown exception")); } - json << "]}"; - return dupString(json.str()); } -// ─── Interactive Function Exploration ───────────────────────── - -char *engine_explore_function(uint64_t project_id, const char *function_name, - int depth, const char *direction) +char *engine_find_symbol(engine_t handle, uint64_t project_id, + const char *symbol_name) { - // SQLite-only recursive exploration. The legacy output schema is - // preserved: - // {"name":"...","file":"...","line":N, - // "callers":[{...recursive...}],"callees":[{...recursive...}]} - // {"error":"function '...' not found","name":"...", - // "callers":[],"callees":[]} - // - // v0.2.5: the graph-not-ready guard is SQLite-specific and lives - // inside the #ifdef; the SQLite backend has its own !g_store->handle() - // guard in the #else branch. - if (!function_name || !*function_name) - return dupString( - "{\"error\":\"empty function name\",\"callers\":[]," - "\"callees\":[]}"); - const char *dir = direction ? direction : "both"; - - // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── - // Recursively explore callers/callees of a function using CSR - // adjacency (getCallerIds / getCalleeIds) and entity metadata. - // JSON shape matches the SQLite branch: nested {name,file,line, - // callers,callees}. - int max_depth = depth > 5 ? 5 : (depth < 0 ? 0 : depth); - if (!g_store || !g_store->handle()) { - return dupString("{\"error\":\"graph not ready [module=engine_" - "queries, method=explore_function]\"," - "\"callers\":[],\"callees\":[]}"); - } - sqlite3 *db = g_store->handle(); - auto fetchNode = [&](uint64_t id, std::string &name, std::string &file, - int &line) { - const char *sql = - "SELECT name, file_path, start_row FROM entity " - "WHERE id=?"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) != SQLITE_OK) - return; - sqlite3_bind_int64(st, 1, static_cast(id)); - if (sqlite3_step(st) == SQLITE_ROW) { - name = reinterpret_cast( - sqlite3_column_text(st, 0)) ? - reinterpret_cast( - sqlite3_column_text(st, 0)) : - ""; - file = reinterpret_cast( - sqlite3_column_text(st, 1)) ? - reinterpret_cast( - sqlite3_column_text(st, 1)) : - ""; - line = sqlite3_column_int(st, 2); - } - sqlite3_finalize(st); - }; - auto fetchNeighbors = [&](uint64_t id, bool callers, - std::vector &out) { - auto ids = callers ? g_store->getCallerIds(id) : - g_store->getCalleeIds(id); - for (uint64_t nid : ids) - out.push_back(nid); - }; - std::function buildNode = - [&](std::ostringstream &json, uint64_t id, int remaining) { - std::string name = "?"; - std::string file_path; - int line = 0; - fetchNode(id, name, file_path, line); - json << "{\"name\":\"" << jsonEscape(name.c_str()) - << "\",\"file\":\"" - << jsonEscape(file_path.c_str()) - << "\",\"line\":" << line; - if (remaining <= 0) { - json << "}"; - return; - } - bool show_callers = strcmp(dir, "callers") == 0 || - strcmp(dir, "both") == 0; - bool show_callees = strcmp(dir, "callees") == 0 || - strcmp(dir, "both") == 0; - if (show_callers) { - json << ",\"callers\":["; - std::vector ids; - fetchNeighbors(id, true, ids); - bool first = true; - for (uint64_t cid : ids) { - if (cid == id) - continue; - if (!first) - json << ","; - first = false; - buildNode(json, cid, remaining - 1); - } - json << "]"; - } - if (show_callees) { - json << ",\"callees\":["; - std::vector ids; - fetchNeighbors(id, false, ids); - bool first = true; - for (uint64_t cid : ids) { - if (cid == id) - continue; - if (!first) - json << ","; - first = false; - buildNode(json, cid, remaining - 1); - } - json << "]"; - } - json << "}"; - }; - // Find the starting function (kind IN (0,1,6)). - uint64_t func_id = 0; - { - const char *sql = - "SELECT id FROM entity WHERE project_id=? AND name=? " - "AND kind IN (0,1,6) ORDER BY id LIMIT 1"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(st, 1, - static_cast(project_id)); - sqlite3_bind_text(st, 2, function_name, -1, - SQLITE_TRANSIENT); - if (sqlite3_step(st) == SQLITE_ROW) - func_id = static_cast( - sqlite3_column_int64(st, 0)); - sqlite3_finalize(st); - } - } - if (!func_id) { - std::ostringstream err; - err << "{\"error\":\"function '" - << jsonEscape(std::string(function_name)) - << "' not found\",\"name\":\"" - << jsonEscape(std::string(function_name)) - << "\",\"callers\":[],\"callees\":[]}"; - return dupString(err.str()); + EngineContext *ctx = engineInstance(handle); + + try { + return findSymbolImpl(ctx, project_id, symbol_name); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_symbol", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_symbol", "unknown exception")); } - std::ostringstream result; - buildNode(result, func_id, max_depth); - return dupString(result.str()); } -// ─── Context Builder ───────────────────────────────────────── - -// Simple intent detection: extract keywords from a natural language query -static std::string detectIntent(const std::string &query) +char *engine_enhance_project(engine_t handle, uint64_t project_id) { - std::string q; - for (char c : query) { - if (isalnum(c) || c == '_' || c == ' ') - q += tolower(c); - else - q += ' '; - } + EngineContext *ctx = engineInstance(handle); - // Module/subdir hints - static const char *modules[] = { "usb", "sound", "net", "block", - "mmc", "gpu", "drm", "i2c", - "spi", "pci", "acpi", "arm", - "x86", "riscv", nullptr }; - for (const char **m = modules; *m; m++) { - if (q.find(*m) != std::string::npos) - return std::string("module:") + *m; + try { + return enhanceProjectImpl(ctx, project_id); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_enhance_project", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_enhance_project", "unknown exception")); } - - // Topic hints - if (q.find("init") != std::string::npos || - q.find("entry") != std::string::npos || - q.find("start") != std::string::npos || - q.find("boot") != std::string::npos) - return "entry_points"; - if (q.find("call") != std::string::npos || - q.find("graph") != std::string::npos || - q.find("trace") != std::string::npos || - q.find("path") != std::string::npos) - return "callgraph"; - if (q.find("driver") != std::string::npos || - q.find("probe") != std::string::npos || - q.find("device") != std::string::npos) - return "drivers"; - if (q.find("memory") != std::string::npos || - q.find("alloc") != std::string::npos || - q.find("free") != std::string::npos || - q.find("mm") != std::string::npos) - return "memory"; - if (q.find("sched") != std::string::npos || - q.find("task") != std::string::npos || - q.find("process") != std::string::npos || - q.find("thread") != std::string::npos) - return "scheduler"; - if (q.find("overview") != std::string::npos || - q.find("architectur") != std::string::npos) - return "overview"; - - return "general"; } -char *engine_build_context(uint64_t project_id, const char *query) +char *engine_get_enhancement_status(engine_t handle, uint64_t project_id) { - if (!g_store) - return dupString("{\"error\":\"engine not initialized\"}"); - - std::string q = query ? query : ""; - std::string intent = detectIntent(q); - auto db = g_store->handle(); - std::ostringstream json; - json << "{"; - - // 1. Project overview (always) - json << "\"project_overview\":" - << g_store->getModuleTreeJson(project_id).c_str() << ","; + EngineContext *ctx = engineInstance(handle); + + try { + return getEnhancementStatusImpl(ctx, project_id); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_enhancement_status", e.what())); + } catch (...) { + return dupString( + "{\"error\":\"[module=ffi, " + "method=engine_get_enhancement_status] unknown " + "exception\"}"); + } +} - // 2. Intent metadata - json << "\"intent\":\"" << intent << "\","; +char *engine_unified_search(engine_t handle, uint64_t project_id, + const char *query, int limit) +{ + EngineContext *ctx = engineInstance(handle); - // 3. Entry points (if relevant or always for general) - if (intent.find("module:") != std::string::npos || - intent == "entry_points" || intent == "general" || - intent == "drivers") { - json << "\"entry_points\":" - << g_store->getEntryPointsJson(project_id).c_str() << ","; + try { + return unifiedSearchImpl(ctx, project_id, query, limit); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_unified_search", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_unified_search", "unknown exception")); } +} - // 4. Focus on specific module if detected - if (intent.find("module:") == 0) { - std::string module_name = intent.substr(7); - std::string msql = - "SELECT name, kind, file_path, line FROM symbols " - "WHERE project_id = ? AND file_path LIKE ? " - "LIMIT 50"; - sqlite3_stmt *mstmt = nullptr; - if (sqlite3_prepare_v2(db, msql.c_str(), -1, &mstmt, nullptr) == - SQLITE_OK) { - std::string pattern = "%/" + module_name + "/%"; - sqlite3_bind_int64(mstmt, 1, - static_cast(project_id)); - sqlite3_bind_text(mstmt, 2, pattern.c_str(), -1, - SQLITE_TRANSIENT); - json << "\"related_symbols\":["; - bool first = true; - while (sqlite3_step(mstmt) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - const char *n = reinterpret_cast( - sqlite3_column_text(mstmt, 0)); - const char *k = reinterpret_cast( - sqlite3_column_text(mstmt, 1)); - const char *f = reinterpret_cast( - sqlite3_column_text(mstmt, 2)); - int ln = sqlite3_column_int(mstmt, 3); - json << "{\"name\":\"" << jsonEscape(n ? n : "") - << "\",\"kind\":\"" - << jsonEscape(k ? k : "") << "\"," - << "\"file\":\"" << jsonEscape(f ? f : "") - << "\",\"line\":" << ln << "}"; - } - sqlite3_finalize(mstmt); - json << "],"; - } +char *engine_find_callers_adaptive(engine_t handle, uint64_t project_id, + const char *symbol_name, + const char *file_filter) +{ + EngineContext *ctx = engineInstance(handle); + + try { + return findCallersAdaptiveImpl(ctx, project_id, symbol_name, + file_filter); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_callers_adaptive", e.what())); + } catch (...) { + return dupString("{\"error\":\"[module=ffi, " + "method=engine_find_callers_adaptive] unknown " + "exception\"}"); } +} - // 5. Call graph data (only if ready AND relevant) - double cg_ratio = g_store->getReadyRatio(project_id, "callgraph_ready"); - bool cg_ready = (cg_ratio > 0.1); - if (cg_ready && (intent == "callgraph" || intent == "general")) { - json << "\"callgraph_available\":true,"; - // Add a sample of call edges (from graph_edges) - const char *csql = - "SELECT gn1.name, gn2.name FROM graph_edges ge " - "JOIN graph_nodes gn1 ON gn1.id = ge.source_node_id " - "JOIN graph_nodes gn2 ON gn2.id = ge.target_node_id " - "WHERE ge.project_id = ? AND ge.edge_type IN (1,3) LIMIT 10"; - sqlite3_stmt *cstmt = nullptr; - if (sqlite3_prepare_v2(db, csql, -1, &cstmt, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(cstmt, 1, - static_cast(project_id)); - json << "\"sample_call_edges\":["; - bool first = true; - while (sqlite3_step(cstmt) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - const char *caller = - reinterpret_cast( - sqlite3_column_text(cstmt, 0)); - const char *callee = - reinterpret_cast( - sqlite3_column_text(cstmt, 1)); - json << "{\"caller\":\"" - << jsonEscape(caller ? caller : "") - << "\"," - << "\"callee\":\"" - << jsonEscape(callee ? callee : "") - << "\"}"; - } - sqlite3_finalize(cstmt); - json << "],"; - } - } else { - json << "\"callgraph_available\":false,"; +char *engine_find_callees_adaptive(engine_t handle, uint64_t project_id, + const char *symbol_name, + const char *file_filter) +{ + EngineContext *ctx = engineInstance(handle); + + try { + return findCalleesAdaptiveImpl(ctx, project_id, symbol_name, + file_filter); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_callees_adaptive", e.what())); + } catch (...) { + return dupString("{\"error\":\"[module=ffi, " + "method=engine_find_callees_adaptive] unknown " + "exception\"}"); } - - // 6. Enhancement progress - json << "\"enhancement_progress\":{" - << "\"callgraph_ready\":" << (cg_ready ? "true" : "false") << "," - << "\"metrics_ready\":" - << (g_store->getReadyRatio(project_id, "metrics_ready") > 0.1 ? - "true" : - "false") - << "," - << "\"embedding_ready\":" - << (g_store->getReadyRatio(project_id, "embedding_ready") > 0.1 ? - "true" : - "false") - << "}"; - - // 7. Ready features summary - json << ",\"ready_features\":{" - << "\"fast_scan\":true," - << "\"module_tree\":true," - << "\"symbol_search\":true," - << "\"call_graph\":" << (cg_ready ? "true" : "false") << "," - << "\"path_tracing\":" << (cg_ready ? "true" : "false") << "," - << "\"semantic_search\":" - << (g_store->getReadyRatio(project_id, "embedding_ready") > 0.1 ? - "true" : - "false") - << "}"; - - json << "}"; - return dupString(json.str()); } -// ─── Phase C: FFI Boundary Detection ────────────────────────── - -char *engine_detect_ffi_boundaries(uint64_t project_id) +char *engine_find_callers_by_entity(engine_t handle, uint64_t project_id, + uint64_t entity_id) { - // SQLite-only FFI boundary detection. The legacy output schema is - // preserved: - // {"languages":[{language,node_count}], - // "cross_language_files":[{file_path,languages,node_count}], - // "ffi_symbols":[{name,file_path,language,line}], - // "orphan_symbols":[{name,file_path,language,line}]} - // - // v0.2.5: the graph-not-ready guard is SQLite-specific and lives - // inside the #ifdef; the SQLite backend has its own !g_store->handle() - // guard in the #else branch. - - // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── - // FFI-boundary diagnosis over the canonical entity table. The four - // sections (languages, cross_language_files, ffi_symbols, - // orphan_symbols) mirror the SQLite branch's output schema. - if (!g_store || !g_store->handle()) { - return dupString("{\"error\":\"graph not ready [module=engine_" - "queries, method=detect_ffi_boundaries]\"}"); + EngineContext *ctx = engineInstance(handle); + + try { + return findCallersByEntityImpl(ctx, project_id, entity_id); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_callers_by_entity", e.what())); + } catch (...) { + return dupString( + "{\"error\":\"[module=ffi, " + "method=engine_find_callers_by_entity] unknown " + "exception\"}"); } - sqlite3 *db = g_store->handle(); - std::ostringstream json; - json << "{"; - auto esc = [](const std::string &s) { return jsonEscape(s.c_str()); }; +} - // 1. Language distribution. - json << "\"languages\":["; - { - const char *sql = "SELECT language, COUNT(*) FROM entity " - "WHERE project_id=? GROUP BY language " - "ORDER BY COUNT(*) DESC"; - sqlite3_stmt *st = nullptr; - bool first = true; - if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(st, 1, - static_cast(project_id)); - while (sqlite3_step(st) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - std::string lang = - reinterpret_cast( - sqlite3_column_text(st, 0)) ? - reinterpret_cast( - sqlite3_column_text( - st, 0)) : - ""; - int64_t count = sqlite3_column_int64(st, 1); - json << "{\"language\":\"" << esc(lang) - << "\",\"node_count\":" << count << "}"; - } - sqlite3_finalize(st); - } +char *engine_find_callees_by_entity(engine_t handle, uint64_t project_id, + uint64_t entity_id) +{ + EngineContext *ctx = engineInstance(handle); + + try { + return findCalleesByEntityImpl(ctx, project_id, entity_id); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_find_callees_by_entity", e.what())); + } catch (...) { + return dupString( + "{\"error\":\"[module=ffi, " + "method=engine_find_callees_by_entity] unknown " + "exception\"}"); } - json << "],"; +} - // 2. Cross-language files. - json << "\"cross_language_files\":["; - { - const char *sql = - "SELECT file_path, GROUP_CONCAT(DISTINCT language), " - " COUNT(*) FROM entity " - "WHERE project_id=? AND language <> '' " - "GROUP BY file_path HAVING COUNT(DISTINCT language) > 1 " - "ORDER BY COUNT(*) DESC LIMIT 20"; - sqlite3_stmt *st = nullptr; - bool first = true; - if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(st, 1, - static_cast(project_id)); - while (sqlite3_step(st) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - std::string fp = - reinterpret_cast( - sqlite3_column_text(st, 0)) ? - reinterpret_cast( - sqlite3_column_text( - st, 0)) : - ""; - std::string langs = - reinterpret_cast( - sqlite3_column_text(st, 1)) ? - reinterpret_cast( - sqlite3_column_text( - st, 1)) : - ""; - int64_t cnt = sqlite3_column_int64(st, 2); - json << "{\"file_path\":\"" << esc(fp) - << "\",\"languages\":\"" << esc(langs) - << "\",\"node_count\":" << cnt << "}"; - } - sqlite3_finalize(st); - } +char *engine_get_entry_points_new(engine_t handle, uint64_t project_id) +{ + EngineContext *ctx = engineInstance(handle); + + try { + return getEntryPointsNewImpl(ctx, project_id); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_get_entry_points_new", e.what())); + } catch (...) { + return dupString("{\"error\":\"[module=ffi, " + "method=engine_get_entry_points_new] unknown " + "exception\"}"); } - json << "],"; +} - // 3. FFI-related symbols (name prefix match, mirroring the Cypher - // STARTS WITH list). - json << "\"ffi_symbols\":["; - { - const char *sql = - "SELECT name, file_path, language, start_row FROM entity " - "WHERE project_id=? AND kind IN (0,1,2) AND (" - "substr(name,1,7)='extern_' OR substr(name,1,4)='ffi_' " - "OR substr(name,1,5)='wasm_' OR substr(name,1,5)='cabi_' " - "OR substr(name,1,4)='jni_' OR substr(name,1,4)='JNI_' " - "OR substr(name,1,9)='CALLBACK_') LIMIT 30"; - sqlite3_stmt *st = nullptr; - bool first = true; - if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(st, 1, - static_cast(project_id)); - while (sqlite3_step(st) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - std::string name = - reinterpret_cast( - sqlite3_column_text(st, 0)) ? - reinterpret_cast( - sqlite3_column_text( - st, 0)) : - ""; - std::string fp = - reinterpret_cast( - sqlite3_column_text(st, 1)) ? - reinterpret_cast( - sqlite3_column_text( - st, 1)) : - ""; - std::string lang = - reinterpret_cast( - sqlite3_column_text(st, 2)) ? - reinterpret_cast( - sqlite3_column_text( - st, 2)) : - ""; - int64_t row = sqlite3_column_int64(st, 3); - json << "{\"name\":\"" << esc(name) - << "\",\"file_path\":\"" << esc(fp) - << "\",\"language\":\"" << esc(lang) - << "\",\"line\":" << row << "}"; - } - sqlite3_finalize(st); - } - } - json << "],"; +char *engine_project_overview(engine_t handle, uint64_t project_id) +{ + EngineContext *ctx = engineInstance(handle); - // 4. Orphan symbols: kind=2 (no edges) excluding test/bench files. - json << "\"orphan_symbols\":["; - { - const char *sql = - "SELECT e.name, e.file_path, e.language, e.start_row " - "FROM entity e WHERE e.project_id=? AND e.kind=2 " - "AND e.file_path NOT LIKE '%test%' " - "AND e.file_path NOT LIKE '%bench%' " - "AND NOT EXISTS (SELECT 1 FROM relation r " - " WHERE r.project_id=? " - " AND (r.source_id=e.id OR " - " r.target_id=e.id)) " - "LIMIT 20"; - sqlite3_stmt *st = nullptr; - bool first = true; - if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(st, 1, - static_cast(project_id)); - sqlite3_bind_int64(st, 2, - static_cast(project_id)); - while (sqlite3_step(st) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - std::string name = - reinterpret_cast( - sqlite3_column_text(st, 0)) ? - reinterpret_cast( - sqlite3_column_text( - st, 0)) : - ""; - std::string fp = - reinterpret_cast( - sqlite3_column_text(st, 1)) ? - reinterpret_cast( - sqlite3_column_text( - st, 1)) : - ""; - std::string lang = - reinterpret_cast( - sqlite3_column_text(st, 2)) ? - reinterpret_cast( - sqlite3_column_text( - st, 2)) : - ""; - int64_t row = sqlite3_column_int64(st, 3); - json << "{\"name\":\"" << esc(name) - << "\",\"file_path\":\"" << esc(fp) - << "\",\"language\":\"" << esc(lang) - << "\",\"line\":" << row << "}"; - } - sqlite3_finalize(st); - } + try { + return projectOverviewImpl(ctx, project_id); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_project_overview", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_project_overview", "unknown exception")); } - json << "]}"; - return dupString(json.str()); } diff --git a/engine/src/engine_queries_context.cpp b/engine/src/engine_queries_context.cpp new file mode 100644 index 0000000..91a2123 --- /dev/null +++ b/engine/src/engine_queries_context.cpp @@ -0,0 +1,909 @@ +// engine_queries_context.cpp — path tracing, interactive exploration, +// context building and FFI-boundary detection. +// +// Split out of engine_queries.cpp (see plan/rules/code_rules.md +// 1000-line rule). These four MCP entry points share the "walk the graph +// and hand the caller a focused JSON view" shape and nothing else, so +// they moved together. Like every other engine_* symbol they are pure C +// ABI export points declared on the Rust side via `extern "C"` — no +// header declaration is needed for the definition itself. + +#include "util/json_writer.h" +#include "engine_internal.h" +#include "async_knowledge.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// ─── Path Tracing ────────────────────────────────────────────── + +static char *tracePathImpl(EngineContext *ctx, uint64_t project_id, + const char *from_name, const char *to_name) +{ + auto _store_guard = waitForKnowledgeBuilder(); + // Trace a path from from_function to to_function using the SQLite + // shortest-path backend (QueryEngine::findShortestPath, CSR BFS) and + // hydrate the node ids from the canonical entity table. The legacy + // tracePathJson output schema is preserved: + // {"path":[{"name":"...","file":"...","line":N}, ...]} + // {"path":[],"error":"..."} + // {"path":[{"name":"..."}],"trivial":true} + if (!from_name || !*from_name || !to_name || !*to_name) + return dupString( + "{\"error\":\"empty symbol name\",\"path\":[]}"); + if (!ctx || !ctx->store || !ctx->store->handle()) { + return dupString("{\"error\":\"graph not ready [module=" + "engine_queries, method=trace_path]\"," + "\"path\":[]}"); + } + sqlite3 *db = ctx->store->handle(); + // Homonym guard: resolveName below takes ORDER BY id LIMIT 1, which + // silently traces the first of several same-named entities (T5 + // finding #9). Surface the candidates instead — same contract as + // getCallers / find_callers. + { + std::string amb = + query::bareNameCandidates(db, project_id, from_name); + if (!amb.empty()) + return dupString("{\"path\":[]," + amb.substr(1)); + amb = query::bareNameCandidates(db, project_id, to_name); + if (!amb.empty()) + return dupString("{\"path\":[]," + amb.substr(1)); + } + auto resolveName = [&](const char *name, uint64_t &out_id) -> bool { + const char *sql = "SELECT id FROM entity WHERE project_id=? " + "AND name=? ORDER BY id LIMIT 1"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) != SQLITE_OK) + return false; + sqlite3_bind_int64(st, 1, static_cast(project_id)); + sqlite3_bind_text(st, 2, name, -1, SQLITE_TRANSIENT); + bool ok = false; + if (sqlite3_step(st) == SQLITE_ROW) { + out_id = static_cast( + sqlite3_column_int64(st, 0)); + ok = true; + } + sqlite3_finalize(st); + return ok; + }; + uint64_t from_id = 0, to_id = 0; + if (!resolveName(from_name, from_id) || !resolveName(to_name, to_id)) + return dupString( + "{\"path\":[],\"error\":\"symbol not found\"}"); + std::string bfs_json = + ctx->query->findShortestPath(project_id, from_id, to_id); + bool found = bfs_json.find("\"found\":true") != std::string::npos; + if (!found) + return dupString("{\"path\":[],\"error\":\"no path found\"}"); + std::vector node_ids; + { + const std::string needle = "\"node_id\":"; + size_t pos = 0; + while ((pos = bfs_json.find(needle, pos)) != + std::string::npos) { + pos += needle.size(); + while (pos < bfs_json.size() && + (bfs_json[pos] == ' ' || bfs_json[pos] == '\t')) + ++pos; + std::string num; + while (pos < bfs_json.size() && + std::isdigit(static_cast( + bfs_json[pos]))) { + num += bfs_json[pos++]; + } + if (!num.empty()) + node_ids.push_back(static_cast( + std::strtoull(num.c_str(), nullptr, + 10))); + } + } + if (node_ids.empty()) + return dupString("{\"path\":[],\"error\":\"no path found\"}"); + std::unordered_map> + lookup; + { + std::string id_list; + for (size_t i = 0; i < node_ids.size(); ++i) { + if (i > 0) + id_list += ","; + id_list += std::to_string(node_ids[i]); + } + std::string sql = "SELECT id, name, file_path, start_row " + "FROM entity WHERE project_id=? AND id IN (" + + id_list + ")"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql.c_str(), -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + uint64_t id = static_cast( + sqlite3_column_int64(st, 0)); + std::string name = + reinterpret_cast( + sqlite3_column_text(st, 1)) ? + reinterpret_cast( + sqlite3_column_text( + st, 1)) : + ""; + std::string file = + reinterpret_cast( + sqlite3_column_text(st, 2)) ? + reinterpret_cast( + sqlite3_column_text( + st, 2)) : + ""; + int line = sqlite3_column_int(st, 3); + lookup.emplace(id, std::make_tuple(name, file, + line)); + } + sqlite3_finalize(st); + } + } + std::ostringstream json; + json << "{\"path\":["; + bool first = true; + for (uint64_t id : node_ids) { + if (!first) + json << ","; + first = false; + auto it = lookup.find(id); + if (it == lookup.end()) { + util::JsonWriter el; + el.beginObject(); + el.key("name").value("?"); + el.key("file").value(""); + el.key("line").value(0); + el.endObject(); + json << el.str(); + } else { + const auto &tup = it->second; + util::JsonWriter el; + el.beginObject(); + el.key("name").value(std::get<0>(tup)); + el.key("file").value(std::get<1>(tup)); + el.key("line").value(std::get<2>(tup)); + el.endObject(); + json << el.str(); + } + } + json << "]}"; + return dupString(json.str()); +} + +// ─── Interactive Function Exploration ───────────────────────── + +static char *exploreFunctionImpl(EngineContext *ctx, uint64_t project_id, + const char *function_name, int depth, + const char *direction) +{ + auto _store_guard = waitForKnowledgeBuilder(); + // SQLite-only recursive exploration. The legacy output schema is + // preserved: + // {"name":"...","file":"...","line":N, + // "callers":[{...recursive...}],"callees":[{...recursive...}]} + // {"error":"function '...' not found","name":"...", + // "callers":[],"callees":[]} + // + // v0.2.5: the graph-not-ready guard is SQLite-specific and lives + // inside the #ifdef; the SQLite backend has its own !ctx->store->handle() + // guard in the #else branch. + if (!function_name || !*function_name) + return dupString( + "{\"error\":\"empty function name\",\"callers\":[]," + "\"callees\":[]}"); + const char *dir = direction ? direction : "both"; + + // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── + // Recursively explore callers/callees of a function using CSR + // adjacency (getCallerIds / getCalleeIds) and entity metadata. + // JSON shape matches the SQLite branch: nested {name,file,line, + // callers,callees}. + int max_depth = depth > 5 ? 5 : (depth < 0 ? 0 : depth); + if (!ctx || !ctx->store || !ctx->store->handle()) { + return dupString("{\"error\":\"graph not ready [module=engine_" + "queries, method=explore_function]\"," + "\"callers\":[],\"callees\":[]}"); + } + sqlite3 *db = ctx->store->handle(); + // Homonym guard: this function resolves the bare name to a single + // entity below, so several same-named functions collapse to whichever + // one the lookup picks and the trace silently explores the wrong + // target (T5 finding #9). Surface the candidates instead — same + // contract as getCallers / find_callers. + { + std::string amb = query::bareNameCandidates(db, project_id, + function_name); + if (!amb.empty()) + // amb is a pre-serialized `{"ambiguous":…}` fragment; drop + // its leading '{' and let it close the object. The name is + // escaped through the shared helper. + return dupString("{\"name\":\"" + + util::jsonEscapeString(function_name) + + "\",\"callers\":[],\"callees\":[]," + + amb.substr(1)); + } + auto fetchNode = [&](uint64_t id, std::string &name, std::string &file, + int &line) { + const char *sql = + "SELECT name, file_path, start_row FROM entity " + "WHERE id=?"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) != SQLITE_OK) + return; + sqlite3_bind_int64(st, 1, static_cast(id)); + if (sqlite3_step(st) == SQLITE_ROW) { + name = reinterpret_cast( + sqlite3_column_text(st, 0)) ? + reinterpret_cast( + sqlite3_column_text(st, 0)) : + ""; + file = reinterpret_cast( + sqlite3_column_text(st, 1)) ? + reinterpret_cast( + sqlite3_column_text(st, 1)) : + ""; + line = sqlite3_column_int(st, 2); + } + sqlite3_finalize(st); + }; + auto fetchNeighbors = [&](uint64_t id, bool callers, + std::vector &out) { + auto ids = callers ? ctx->store->getCallerIds(id) : + ctx->store->getCalleeIds(id); + for (uint64_t nid : ids) + out.push_back(nid); + }; + std::function buildNode = + [&](std::ostringstream &json, uint64_t id, int remaining) { + std::string name = "?"; + std::string file_path; + int line = 0; + fetchNode(id, name, file_path, line); + // The object stays open across the optional callers/callees + // members and is closed at two different points, and the + // lambda recurses over the stream, so its shape is kept; + // escaping goes through the shared helper. + json << "{\"name\":\"" << util::jsonEscapeString(name) + << "\",\"file\":\"" + << util::jsonEscapeString(file_path) + << "\",\"line\":" << line; + if (remaining <= 0) { + json << "}"; + return; + } + bool show_callers = strcmp(dir, "callers") == 0 || + strcmp(dir, "both") == 0; + bool show_callees = strcmp(dir, "callees") == 0 || + strcmp(dir, "both") == 0; + if (show_callers) { + json << ",\"callers\":["; + std::vector ids; + fetchNeighbors(id, true, ids); + bool first = true; + for (uint64_t cid : ids) { + if (cid == id) + continue; + if (!first) + json << ","; + first = false; + buildNode(json, cid, remaining - 1); + } + json << "]"; + } + if (show_callees) { + json << ",\"callees\":["; + std::vector ids; + fetchNeighbors(id, false, ids); + bool first = true; + for (uint64_t cid : ids) { + if (cid == id) + continue; + if (!first) + json << ","; + first = false; + buildNode(json, cid, remaining - 1); + } + json << "]"; + } + json << "}"; + }; + // Find the starting function (kind IN (0,1,6)). + uint64_t func_id = 0; + { + const char *sql = + "SELECT id FROM entity WHERE project_id=? AND name=? " + "AND kind IN (0,1,6) ORDER BY id LIMIT 1"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + sqlite3_bind_text(st, 2, function_name, -1, + SQLITE_TRANSIENT); + if (sqlite3_step(st) == SQLITE_ROW) + func_id = static_cast( + sqlite3_column_int64(st, 0)); + sqlite3_finalize(st); + } + } + if (!func_id) { + util::JsonWriter err; + err.beginObject(); + err.key("error").value(std::string("function '") + + function_name + "' not found"); + err.key("name").value(function_name); + err.key("callers").beginArray().endArray(); + err.key("callees").beginArray().endArray(); + err.endObject(); + return dupString(err.str()); + } + std::ostringstream result; + buildNode(result, func_id, max_depth); + return dupString(result.str()); +} + +// ─── Context Builder ───────────────────────────────────────── + +// Simple intent detection: extract keywords from a natural language query +static std::string detectIntent(const std::string &query) +{ + std::string q; + for (char c : query) { + if (isalnum(c) || c == '_' || c == ' ') + q += tolower(c); + else + q += ' '; + } + + // Module/subdir hints + static const char *modules[] = { "usb", "sound", "net", "block", + "mmc", "gpu", "drm", "i2c", + "spi", "pci", "acpi", "arm", + "x86", "riscv", nullptr }; + for (const char **m = modules; *m; m++) { + if (q.find(*m) != std::string::npos) + return std::string("module:") + *m; + } + + // Topic hints + if (q.find("init") != std::string::npos || + q.find("entry") != std::string::npos || + q.find("start") != std::string::npos || + q.find("boot") != std::string::npos) + return "entry_points"; + if (q.find("call") != std::string::npos || + q.find("graph") != std::string::npos || + q.find("trace") != std::string::npos || + q.find("path") != std::string::npos) + return "callgraph"; + if (q.find("driver") != std::string::npos || + q.find("probe") != std::string::npos || + q.find("device") != std::string::npos) + return "drivers"; + if (q.find("memory") != std::string::npos || + q.find("alloc") != std::string::npos || + q.find("free") != std::string::npos || + q.find("mm") != std::string::npos) + return "memory"; + if (q.find("sched") != std::string::npos || + q.find("task") != std::string::npos || + q.find("process") != std::string::npos || + q.find("thread") != std::string::npos) + return "scheduler"; + if (q.find("overview") != std::string::npos || + q.find("architectur") != std::string::npos) + return "overview"; + + return "general"; +} + +static char *buildContextImpl(EngineContext *ctx, uint64_t project_id, + const char *query) +{ + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) + return dupString("{\"error\":\"engine not initialized\"}"); + + std::string q = query ? query : ""; + std::string intent = detectIntent(q); + auto db = ctx->store->handle(); + std::ostringstream json; + json << "{"; + + // 1. Project overview (always) + json << "\"project_overview\":" + << ctx->store->getModuleTreeJson(project_id).c_str() << ","; + + // 2. Intent metadata + json << "\"intent\":\"" << intent << "\","; + + // 3. Entry points (if relevant or always for general) + if (intent.find("module:") != std::string::npos || + intent == "entry_points" || intent == "general" || + intent == "drivers") { + // Same live source as project_overview / get_entry_points (the legacy + // store accessor joined `graph_nodes`, which the canonical pipeline + // never fills, so this always emitted an empty list). + json << "\"entry_points\":" + << ctx->query->getEntryPoints(project_id).c_str() << ","; + } + + // 4. Focus on specific module if detected + if (intent.find("module:") == 0) { + std::string module_name = intent.substr(7); + std::string msql = + "SELECT name, kind, file_path, line FROM symbols " + "WHERE project_id = ? AND file_path LIKE ? " + "LIMIT 50"; + sqlite3_stmt *mstmt = nullptr; + if (sqlite3_prepare_v2(db, msql.c_str(), -1, &mstmt, nullptr) == + SQLITE_OK) { + std::string pattern = "%/" + module_name + "/%"; + sqlite3_bind_int64(mstmt, 1, + static_cast(project_id)); + sqlite3_bind_text(mstmt, 2, pattern.c_str(), -1, + SQLITE_TRANSIENT); + json << "\"related_symbols\":["; + bool first = true; + while (sqlite3_step(mstmt) == SQLITE_ROW) { + if (!first) + json << ","; + first = false; + const char *n = reinterpret_cast( + sqlite3_column_text(mstmt, 0)); + const char *k = reinterpret_cast( + sqlite3_column_text(mstmt, 1)); + const char *f = reinterpret_cast( + sqlite3_column_text(mstmt, 2)); + int ln = sqlite3_column_int(mstmt, 3); + util::JsonWriter el; + el.beginObject(); + el.key("name").value(n ? n : ""); + el.key("kind").value(k ? k : ""); + el.key("file").value(f ? f : ""); + el.key("line").value(ln); + el.endObject(); + json << el.str(); + } + sqlite3_finalize(mstmt); + json << "],"; + } + } + + // 5. Call graph data (only if ready AND relevant) + double cg_ratio = + ctx->store->getReadyRatio(project_id, "callgraph_ready"); + bool cg_ready = (cg_ratio > 0.1); + if (cg_ready && (intent == "callgraph" || intent == "general")) { + json << "\"callgraph_available\":true,"; + // Sample call edges from the canonical tables (relation + entity). + // It used to join graph_edges/graph_nodes, which the canonical + // pipeline leaves empty, so the sample was always [] even when + // callgraph_available had just been reported true. + const char *csql = + "SELECT s.name, t.name FROM relation r " + "JOIN entity s ON s.id = r.source_id " + "JOIN entity t ON t.id = r.target_id " + "WHERE r.project_id = ? AND r.type IN (1,3) LIMIT 10"; + sqlite3_stmt *cstmt = nullptr; + if (sqlite3_prepare_v2(db, csql, -1, &cstmt, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(cstmt, 1, + static_cast(project_id)); + json << "\"sample_call_edges\":["; + bool first = true; + while (sqlite3_step(cstmt) == SQLITE_ROW) { + if (!first) + json << ","; + first = false; + const char *caller = + reinterpret_cast( + sqlite3_column_text(cstmt, 0)); + const char *callee = + reinterpret_cast( + sqlite3_column_text(cstmt, 1)); + util::JsonWriter el; + el.beginObject(); + el.key("caller").value(caller ? caller : ""); + el.key("callee").value(callee ? callee : ""); + el.endObject(); + json << el.str(); + } + sqlite3_finalize(cstmt); + json << "],"; + } + } else { + json << "\"callgraph_available\":false,"; + } + + // 6. Enhancement progress + json << "\"enhancement_progress\":{" + << "\"callgraph_ready\":" << (cg_ready ? "true" : "false") << "," + << "\"metrics_ready\":" + << (ctx->store->getReadyRatio(project_id, "metrics_ready") > 0.1 ? + "true" : + "false") + << "," + << "\"embedding_ready\":" + << (ctx->store->getReadyRatio(project_id, "embedding_ready") > + 0.1 ? + "true" : + "false") + << "}"; + + // 7. Ready features summary + json << ",\"ready_features\":{" + << "\"fast_scan\":true," + << "\"module_tree\":true," + << "\"symbol_search\":true," + << "\"call_graph\":" << (cg_ready ? "true" : "false") << "," + << "\"path_tracing\":" << (cg_ready ? "true" : "false") << "," + << "\"semantic_search\":" + << (ctx->store->getReadyRatio(project_id, "embedding_ready") > + 0.1 ? + "true" : + "false") + << "}"; + + json << "}"; + return dupString(json.str()); +} + +// ─── Phase C: FFI Boundary Detection ────────────────────────── + +static char *detectFfiBoundariesImpl(EngineContext *ctx, uint64_t project_id) +{ + auto _store_guard = waitForKnowledgeBuilder(); + // SQLite-only FFI boundary detection. The legacy output schema is + // preserved: + // {"languages":[{language,node_count}], + // "cross_language_files":[{file_path,languages,node_count}], + // "ffi_symbols":[{name,file_path,language,line}], + // "orphan_symbols":[{name,file_path,language,line}]} + // + // v0.2.5: the graph-not-ready guard is SQLite-specific and lives + // inside the #ifdef; the SQLite backend has its own !ctx->store->handle() + // guard in the #else branch. + + // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── + // FFI-boundary diagnosis over the canonical entity table. The four + // sections (languages, cross_language_files, ffi_symbols, + // orphan_symbols) mirror the SQLite branch's output schema. + if (!ctx || !ctx->store || !ctx->store->handle()) { + return dupString("{\"error\":\"graph not ready [module=engine_" + "queries, method=detect_ffi_boundaries]\"}"); + } + sqlite3 *db = ctx->store->handle(); + std::ostringstream json; + json << "{"; + // Single source of truth for escaping; the surrounding document is still + // assembled by hand and is tracked as remaining TD-3 work. + auto esc = [](const std::string &s) { + return util::jsonEscapeString(s); + }; + + // 1. Language distribution. + json << "\"languages\":["; + { + const char *sql = "SELECT language, COUNT(*) FROM entity " + "WHERE project_id=? GROUP BY language " + "ORDER BY COUNT(*) DESC"; + sqlite3_stmt *st = nullptr; + bool first = true; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + if (!first) + json << ","; + first = false; + std::string lang = + reinterpret_cast( + sqlite3_column_text(st, 0)) ? + reinterpret_cast( + sqlite3_column_text( + st, 0)) : + ""; + int64_t count = sqlite3_column_int64(st, 1); + json << "{\"language\":\"" << esc(lang) + << "\",\"node_count\":" << count << "}"; + } + sqlite3_finalize(st); + } + } + json << "],"; + + // 2. Cross-language files. + json << "\"cross_language_files\":["; + { + const char *sql = + "SELECT file_path, GROUP_CONCAT(DISTINCT language), " + " COUNT(*) FROM entity " + "WHERE project_id=? AND language <> '' " + "GROUP BY file_path HAVING COUNT(DISTINCT language) > 1 " + "ORDER BY COUNT(*) DESC LIMIT 20"; + sqlite3_stmt *st = nullptr; + bool first = true; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + if (!first) + json << ","; + first = false; + std::string fp = + reinterpret_cast( + sqlite3_column_text(st, 0)) ? + reinterpret_cast( + sqlite3_column_text( + st, 0)) : + ""; + std::string langs = + reinterpret_cast( + sqlite3_column_text(st, 1)) ? + reinterpret_cast( + sqlite3_column_text( + st, 1)) : + ""; + int64_t cnt = sqlite3_column_int64(st, 2); + json << "{\"file_path\":\"" << esc(fp) + << "\",\"languages\":\"" << esc(langs) + << "\",\"node_count\":" << cnt << "}"; + } + sqlite3_finalize(st); + } + } + json << "],"; + + // 3. FFI-related symbols (name prefix match, mirroring the Cypher + // STARTS WITH list). + json << "\"ffi_symbols\":["; + { + const char *sql = + "SELECT name, file_path, language, start_row FROM entity " + "WHERE project_id=? AND kind IN (0,1,2) AND (" + "substr(name,1,7)='extern_' OR substr(name,1,4)='ffi_' " + "OR substr(name,1,5)='wasm_' OR substr(name,1,5)='cabi_' " + "OR substr(name,1,4)='jni_' OR substr(name,1,4)='JNI_' " + "OR substr(name,1,9)='CALLBACK_') LIMIT 30"; + sqlite3_stmt *st = nullptr; + bool first = true; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + if (!first) + json << ","; + first = false; + std::string name = + reinterpret_cast( + sqlite3_column_text(st, 0)) ? + reinterpret_cast( + sqlite3_column_text( + st, 0)) : + ""; + std::string fp = + reinterpret_cast( + sqlite3_column_text(st, 1)) ? + reinterpret_cast( + sqlite3_column_text( + st, 1)) : + ""; + std::string lang = + reinterpret_cast( + sqlite3_column_text(st, 2)) ? + reinterpret_cast( + sqlite3_column_text( + st, 2)) : + ""; + int64_t row = sqlite3_column_int64(st, 3); + json << "{\"name\":\"" << esc(name) + << "\",\"file_path\":\"" << esc(fp) + << "\",\"language\":\"" << esc(lang) + << "\",\"line\":" << row << "}"; + } + sqlite3_finalize(st); + } + } + json << "],"; + + // 3b. One-sided FFI: symbols that are CALLED but have no definition + // entity in the indexed project. This is the C↔Rust shape where one + // side is indexed alone — `int rust_add(int,int);` in a C header is + // a prototype whose body lives in a Rust crate that was never + // indexed. The name-prefix list above cannot see these (no + // extern_/wasm_/jni_ prefix), so without this the tool reports + // ffi_symbols:[] for a genuine boundary (T5 finding #7). + json << "\"external_symbols\":["; + { + const char *sql = + "SELECT DISTINCT sr.name, sr.file_path, sr.language, " + " sr.start_row " + "FROM semantic_records sr " + "WHERE sr.project_id=? AND sr.kind=9 " + "AND sr.name <> '' " + "AND NOT EXISTS (SELECT 1 FROM entity e " + " WHERE e.project_id=sr.project_id " + " AND (e.name=sr.name OR " + " e.qualified_name=sr.name) " + " AND e.kind IN (0,1)) " + "ORDER BY sr.name LIMIT 30"; + sqlite3_stmt *st = nullptr; + bool first = true; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + if (!first) + json << ","; + first = false; + const unsigned char *n = + sqlite3_column_text(st, 0); + const unsigned char *f = + sqlite3_column_text(st, 1); + const unsigned char *l = + sqlite3_column_text(st, 2); + std::string name = + n ? reinterpret_cast(n) : + ""; + std::string fp = + f ? reinterpret_cast(f) : + ""; + std::string lang = + l ? reinterpret_cast(l) : + ""; + int64_t row = sqlite3_column_int64(st, 3); + json << "{\"name\":\"" << esc(name) + << "\",\"file_path\":\"" << esc(fp) + << "\",\"language\":\"" << esc(lang) + << "\",\"line\":" << row << "}"; + } + sqlite3_finalize(st); + } + } + json << "],"; + + // 4. Orphan symbols: kind=2 (no edges) excluding test/bench files. + json << "\"orphan_symbols\":["; + { + const char *sql = + "SELECT e.name, e.file_path, e.language, e.start_row " + "FROM entity e WHERE e.project_id=? AND e.kind=2 " + "AND e.file_path NOT LIKE '%test%' " + "AND e.file_path NOT LIKE '%bench%' " + "AND NOT EXISTS (SELECT 1 FROM relation r " + " WHERE r.project_id=? " + " AND (r.source_id=e.id OR " + " r.target_id=e.id)) " + "LIMIT 20"; + sqlite3_stmt *st = nullptr; + bool first = true; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + sqlite3_bind_int64(st, 2, + static_cast(project_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + if (!first) + json << ","; + first = false; + std::string name = + reinterpret_cast( + sqlite3_column_text(st, 0)) ? + reinterpret_cast( + sqlite3_column_text( + st, 0)) : + ""; + std::string fp = + reinterpret_cast( + sqlite3_column_text(st, 1)) ? + reinterpret_cast( + sqlite3_column_text( + st, 1)) : + ""; + std::string lang = + reinterpret_cast( + sqlite3_column_text(st, 2)) ? + reinterpret_cast( + sqlite3_column_text( + st, 2)) : + ""; + int64_t row = sqlite3_column_int64(st, 3); + json << "{\"name\":\"" << esc(name) + << "\",\"file_path\":\"" << esc(fp) + << "\",\"language\":\"" << esc(lang) + << "\",\"line\":" << row << "}"; + } + sqlite3_finalize(st); + } + } + json << "]}"; + return dupString(json.str()); +} + +// ─── FFI boundary wrappers ─────────────────────────────────────── +// The bodies above are static implementations (`*Impl`). Every extern "C" +// entry point is a thin try/catch wrapper: no C++ exception may cross the C +// ABI boundary, because the MCP server is long-running and an escaping +// exception would terminate the whole session. Error envelopes carry a +// [module=ffi, method=] tag per code_rules.md. + +char *engine_trace_path(engine_t handle, uint64_t project_id, + const char *from_name, const char *to_name) +{ + EngineContext *ctx = engineInstance(handle); + + try { + return tracePathImpl(ctx, project_id, from_name, to_name); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope("ffi", "engine_trace_path", + e.what())); + } catch (...) { + return dupString(util::errorEnvelope("ffi", "engine_trace_path", + "unknown exception")); + } +} + +char *engine_explore_function(engine_t handle, uint64_t project_id, + const char *function_name, int depth, + const char *direction) +{ + EngineContext *ctx = engineInstance(handle); + + try { + return exploreFunctionImpl(ctx, project_id, function_name, + depth, direction); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_explore_function", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_explore_function", "unknown exception")); + } +} + +char *engine_build_context(engine_t handle, uint64_t project_id, + const char *query) +{ + EngineContext *ctx = engineInstance(handle); + + try { + return buildContextImpl(ctx, project_id, query); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_build_context", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_build_context", "unknown exception")); + } +} + +char *engine_detect_ffi_boundaries(engine_t handle, uint64_t project_id) +{ + EngineContext *ctx = engineInstance(handle); + + try { + return detectFfiBoundariesImpl(ctx, project_id); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_detect_ffi_boundaries", e.what())); + } catch (...) { + return dupString("{\"error\":\"[module=ffi, " + "method=engine_detect_ffi_boundaries] unknown " + "exception\"}"); + } +} diff --git a/engine/src/engine_queries_status.cpp b/engine/src/engine_queries_status.cpp new file mode 100644 index 0000000..a64be8a --- /dev/null +++ b/engine/src/engine_queries_status.cpp @@ -0,0 +1,223 @@ +// engine_queries_status.cpp — engine_get_enhancement_status implementation. +// +// Split out of engine_queries.cpp (1000-line rule, plan/rules/code_rules.md). +// Reports per-capability readiness coverage (callgraph / metrics / embedding / +// semantic_search / fts) for a project as a JSON object. + +#include "engine_internal.h" +#include "async_knowledge.h" + +#include +#include +#include +#include +#include + +// ─── Phase B: engine_get_enhancement_status ──────────────────── + +// Helper: count eligible function/method entities (entity.kind IN 0,1) for a +// project — the denominator for every capability coverage ratio. Reads the +// canonical `entity` table. Returns 0 on any error. +static int queries_count_eligible_entities(sqlite3 *db, uint64_t project_id) +{ + sqlite3_stmt *stmt = nullptr; + int total = 0; + const char *sql = + "SELECT COUNT(*) FROM entity WHERE project_id=? AND kind IN (0,1)"; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + if (sqlite3_step(stmt) == SQLITE_ROW) + total = sqlite3_column_int(stmt, 0); + sqlite3_finalize(stmt); + } else { + fprintf(stderr, + "engine_get_enhancement_status: entity count probe failed: %s " + "[module=queries, method=engine_get_enhancement_status]\n", + sqlite3_errmsg(db)); + } + return total; +} + +// Helper: count distinct function/method entities that participate in at least +// one Calls relation (relation.type=1). This is the canonical callgraph-ready +// count. Returns 0 on any error. +static int queries_count_callgraph_ready(sqlite3 *db, uint64_t project_id) +{ + sqlite3_stmt *stmt = nullptr; + int ready = 0; + const char *sql = + "SELECT COUNT(*) FROM (" + " SELECT DISTINCT src FROM (" + " SELECT source_id AS src FROM relation WHERE project_id=? AND type=1" + " UNION" + " SELECT target_id AS src FROM relation WHERE project_id=? AND type=1" + " )" + ")"; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_int64(stmt, 2, static_cast(project_id)); + if (sqlite3_step(stmt) == SQLITE_ROW) + ready = sqlite3_column_int(stmt, 0); + sqlite3_finalize(stmt); + } else { + fprintf(stderr, + "engine_get_enhancement_status: callgraph count probe failed: %s " + "[module=queries, method=engine_get_enhancement_status]\n", + sqlite3_errmsg(db)); + } + return ready; +} + +// Helper: count node_vectors rows for a project — the canonical embedding +// coverage count. Returns 0 if the table is missing or empty. Used both for +// the embedding_ready count and to guard the project_readiness.vector_ready +// flag against the A19 "fake ready" regression. +static int64_t queries_count_node_vectors(sqlite3 *db, uint64_t project_id) +{ + sqlite3_stmt *stmt = nullptr; + int64_t ready = 0; + const char *sql = + "SELECT COUNT(*) FROM node_vectors WHERE project_id=?"; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + if (sqlite3_step(stmt) == SQLITE_ROW) + ready = sqlite3_column_int64(stmt, 0); + sqlite3_finalize(stmt); + } else { + // node_vectors table may not exist on legacy DBs — log and treat + // as 0 (readiness tracks canonical data; an absent table means 0). + fprintf(stderr, + "engine_get_enhancement_status: node_vectors count probe failed: %s " + "[module=queries, method=engine_get_enhancement_status]\n", + sqlite3_errmsg(db)); + } + return ready; +} + +// Helper: count function/method entities that carry resolved code metrics +// (cyclomatic > 0), i.e. the canonical metrics_ready count. metrics are +// resolved onto entity by resolveStagedMetrics after buildGraph, so this +// probe reflects real producer output — never a placeholder. Returns 0 on +// any error or on a pre-metrics database. +static int queries_count_metrics_ready(sqlite3 *db, uint64_t project_id) +{ + sqlite3_stmt *stmt = nullptr; + int ready = 0; + const char *sql = + "SELECT COUNT(*) FROM entity " + "WHERE project_id = ? AND kind IN (0,1) AND cyclomatic > 0"; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + if (sqlite3_step(stmt) == SQLITE_ROW) + ready = sqlite3_column_int(stmt, 0); + sqlite3_finalize(stmt); + } else { + fprintf(stderr, + "engine_get_enhancement_status: metrics count probe failed: %s " + "[module=queries, method=engine_get_enhancement_status]\n", + sqlite3_errmsg(db)); + } + return ready; +} + +// Helper: compute coverage ratio as a JSON-friendly string in [0.0, 1.0]. +// Returns "0.0" when eligible == 0 to avoid divide-by-zero. +static std::string queries_coverage_ratio(int ready, int eligible) +{ + if (eligible <= 0) + return "0.0"; + char buf[32]; + std::snprintf(buf, sizeof(buf), "%.4f", + static_cast(ready) / + static_cast(eligible)); + return std::string(buf); +} + +char *getEnhancementStatusImpl(EngineContext *ctx, uint64_t project_id) +{ + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) + return dupString("{\"error\":\"engine not initialized\"}"); + + auto db = ctx->store->handle(); + + // v0.2.5: report real counts from canonical data (entity / relation / + // node_vectors). The legacy int fields (total_symbols/callgraph_ready/ + // metrics_ready/embedding_ready) are preserved at the start of the JSON + // so existing MCP clients / sscanf parsers keep working; the richer + // `capabilities` block carries eligible/ready/coverage/ + // producer_version so callers can tell "not yet run" from "not built". + const int total = queries_count_eligible_entities(db, project_id); + const int cg_ready = queries_count_callgraph_ready(db, project_id); + const int64_t vec_rows = queries_count_node_vectors(db, project_id); + // metrics_ready comes from canonical data: entity rows (kind 0/1) whose + // cyclomatic was resolved by resolveStagedMetrics. It is a real count — + // the metrics producer was restored in v0.2.5, so a fresh index produces + // a positive value, while a pre-metrics DB reports 0 honestly. + const int metrics_ready = queries_count_metrics_ready(db, project_id); + const int embedding_ready = static_cast(vec_rows); + + // fts_ready is read from project_readiness (set by the async path). + const int fts_ready = + ctx->store->getProjectReadiness(project_id, "fts_ready"); + + // coverage ratios — real numbers in [0.0, 1.0], never a placeholder 0. + const std::string cg_coverage = queries_coverage_ratio(cg_ready, total); + const std::string metrics_coverage = + queries_coverage_ratio(metrics_ready, total); + const std::string embedding_coverage = + queries_coverage_ratio(embedding_ready, total); + + std::ostringstream json; + // Legacy int fields — kept stable for backward-compat parsers. + json << "{" + << "\"total_symbols\":" << total << "," + << "\"callgraph_ready\":" << cg_ready << "," + << "\"metrics_ready\":" << metrics_ready << "," + << "\"embedding_ready\":" << embedding_ready + << "," + // Richer per-capability block: eligible/ready/failed/coverage + + // producer_version + unavailable_reason. NO hardcoded 0 — every + // count comes from a canonical table probe above. + << "\"capabilities\":{" + << "\"callgraph\":{" + << "\"available\":true," + << "\"ready\":" << (cg_ready > 0 ? "true" : "false") << "," + << "\"eligible\":" << total << "," + << "\"ready_count\":" << cg_ready << "," + << "\"failed\":0," + << "\"coverage\":" << cg_coverage << "," + << "\"producer_version\":\"buildGraph\"" + << "}," + << "\"metrics\":{" + << "\"available\":true," + << "\"ready\":" << (metrics_ready > 0 ? "true" : "false") << "," + << "\"eligible\":" << total << "," + << "\"ready_count\":" << metrics_ready << "," + << "\"failed\":0," + << "\"coverage\":" << metrics_coverage << "," + << "\"producer_version\":\"resolveStagedMetrics\"" + << "}," + << "\"embedding\":{" + << "\"available\":true," + << "\"ready\":" << (embedding_ready > 0 ? "true" : "false") << "," + << "\"eligible\":" << total << "," + << "\"ready_count\":" << embedding_ready << "," + << "\"failed\":0," + << "\"coverage\":" << embedding_coverage << "," + << "\"producer_version\":\"buildVectorsFromGraph\"" + << "}," + << "\"semantic_search\":{" + << "\"available\":true," + << "\"ready\":" << (embedding_ready > 0 ? "true" : "false") << "," + << "\"mode\":\"ngram_hash\"," + << "\"description\":\"n-gram hash vector lexical similarity " + "(restored in v0.2.5); complements FTS exact search\"" + << "}," + << "\"fts\":{" + << "\"available\":true," + << "\"ready\":" << (fts_ready ? "true" : "false") << "}" + << "}" + << "}"; + return dupString(json.str()); +} diff --git a/engine/src/engine_scanner.cpp b/engine/src/engine_scanner.cpp index 88111f9..49d854c 100644 --- a/engine/src/engine_scanner.cpp +++ b/engine/src/engine_scanner.cpp @@ -1,4 +1,6 @@ +#include "util/json_writer.h" #include "engine_internal.h" +#include "async_knowledge.h" #include "filter_policy.h" #include "platform_win.h" @@ -28,11 +30,24 @@ // Redirected to engine_index_project with fast mode. // Regex scan eliminated — tree-sitter is fast enough (727ms for 150K lines). -char *engine_scan_project(uint64_t project_id, const char *dir_path, - const char *language_filter) +char *engine_scan_project(engine_t handle, uint64_t project_id, + const char *dir_path, const char *language_filter) { - if (!g_store) - return dupString("{\"error\":\"engine not initialized\"}"); - (void)language_filter; - return engine_index_project(project_id, dir_path, nullptr); + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) + return dupString( + "{\"error\":\"engine not initialized\"}"); + (void)language_filter; + return engine_index_project(handle, project_id, dir_path, + nullptr); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_scan_project", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_scan_project", "unknown exception")); + } } diff --git a/engine/src/engine_verify_drift_ffi.cpp b/engine/src/engine_verify_drift_ffi.cpp index e8fae79..f94e6ed 100644 --- a/engine/src/engine_verify_drift_ffi.cpp +++ b/engine/src/engine_verify_drift_ffi.cpp @@ -1,4 +1,6 @@ +#include "util/json_writer.h" #include "engine_internal.h" +#include "async_knowledge.h" #include "verify/ffi_internal.h" #include "verify/architecture_drift.h" #include "verify/capability_drift.h" @@ -48,7 +50,8 @@ struct AggregateVerdict { double confidence = 0.0; }; -AggregateVerdict aggregateVerdict(int supported, int contradicted, int unknown) +AggregateVerdict aggregateVerdict(EngineContext *ctx, int supported, + int contradicted, int unknown) { AggregateVerdict out; int total = supported + contradicted + unknown; @@ -89,10 +92,14 @@ AggregateVerdict aggregateVerdict(int supported, int contradicted, int unknown) // // MEMORY: caller MUST free the returned char* via engine_free_string(). // THREAD SAFETY: single-threaded (GraphStore writer invariant). -extern "C" char *engine_verify_review(uint64_t project_id, const char *text) +extern "C" char *engine_verify_review(engine_t handle, uint64_t project_id, + const char *text) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"not initialized\"}"); if (!text || !*text) return dupString( @@ -101,7 +108,7 @@ extern "C" char *engine_verify_review(uint64_t project_id, const char *text) std::string src(text); auto batch = verify_ffi::verify_claim_batch( - project_id, src, kSourceKindCodeReview, + ctx, project_id, src, kSourceKindCodeReview, src.substr(0, kSourceRefMaxLen)); std::ostringstream json; @@ -109,23 +116,27 @@ extern "C" char *engine_verify_review(uint64_t project_id, const char *text) << ",\"results\":" << batch.results_json << ",\"summary\":{\"supported\":" << batch.supported << ",\"contradicted\":" << batch.contradicted - << ",\"unknown\":" << batch.unknown << ",\"trust_score\":"; + << ",\"unknown\":" << batch.unknown << ",\"status\":\"" + << batch.status << "\"" + << ",\"verdicts_decided\":" + << (batch.supported + batch.contradicted) + << ",\"trust_score\":"; int denom = batch.supported + batch.contradicted; if (denom > 0) json << (static_cast(batch.supported) / static_cast(denom)); else + // No decided verdicts — `status` says whether the text had no + // claims at all or only undecidable ones. json << "0.0"; json << "}}"; return dupString(json.str()); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_verify_review] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_verify_review", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_verify_review] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_verify_review", "unknown exception")); } } @@ -147,10 +158,14 @@ extern "C" char *engine_verify_review(uint64_t project_id, const char *text) // // MEMORY: caller MUST free the returned char* via engine_free_string(). // THREAD SAFETY: single-threaded (GraphStore writer invariant). -extern "C" char *engine_verify_reality(uint64_t project_id, const char *text) +extern "C" char *engine_verify_reality(engine_t handle, uint64_t project_id, + const char *text) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"not initialized\"}"); if (!text || !*text) return dupString( @@ -159,28 +174,29 @@ extern "C" char *engine_verify_reality(uint64_t project_id, const char *text) std::string src(text); auto batch = verify_ffi::verify_claim_batch( - project_id, src, kSourceKindAiStatement, + ctx, project_id, src, kSourceKindAiStatement, src.substr(0, kSourceRefMaxLen)); - AggregateVerdict agg = aggregateVerdict( - batch.supported, batch.contradicted, batch.unknown); - - std::ostringstream json; - json << "{\"statement\":\"" - << jsonEscape(src.substr(0, kSourceRefMaxLen)) - << "\",\"claims_parsed\":" << batch.claims_count - << ",\"verdict\":\"" << agg.verdict - << "\",\"confidence\":" << agg.confidence - << ",\"results\":" << batch.results_json << "}"; + AggregateVerdict agg = aggregateVerdict(ctx, batch.supported, + batch.contradicted, + batch.unknown); + + util::JsonWriter json; + json.beginObject(); + json.key("statement").value(src.substr(0, kSourceRefMaxLen)); + json.key("claims_parsed").value(batch.claims_count); + json.key("status").value(batch.status); + json.key("verdict").value(agg.verdict); + json.key("confidence").value(agg.confidence); + json.key("results").raw(batch.results_json); + json.endObject(); return dupString(json.str()); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_verify_reality] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_verify_reality", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_verify_reality] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_verify_reality", "unknown exception")); } } @@ -203,13 +219,16 @@ extern "C" char *engine_verify_reality(uint64_t project_id, const char *text) // // MEMORY: caller MUST free the returned char* via engine_free_string(). // THREAD SAFETY: single-threaded (GraphStore writer invariant). -extern "C" char *engine_detect_drift(uint64_t project_id) +extern "C" char *engine_detect_drift(engine_t handle, uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"not initialized\"}"); - sqlite3 *db = g_store->handle(); + sqlite3 *db = ctx->store->handle(); if (!db) return dupString( "{\"error\":\"db handle null " @@ -253,21 +272,24 @@ extern "C" char *engine_detect_drift(uint64_t project_id) "Capability '" + name + "' declared in README but no " "implementing entity with callers"; - g_store->insertFinding( + ctx->store->insertFinding( project_id, "MissingCapability", kDriftSeverityHard, 0, detail, 0.9); if (!first) json << ","; first = false; - json << "{\"type\":\"MissingCapability\"," - << "\"severity\":" - << kDriftSeverityHard - << ",\"capability_id\":" << cid - << ",\"subject\":\"" - << jsonEscape(name) << "\"" - << ",\"detail\":\"" - << jsonEscape(detail) << "\"}"; + util::JsonWriter el; + el.beginObject(); + el.key("type").value( + "MissingCapability"); + el.key("severity") + .value(kDriftSeverityHard); + el.key("capability_id").value(cid); + el.key("subject").value(name); + el.key("detail").value(detail); + el.endObject(); + json << el.str(); drifts_found++; } sqlite3_finalize(stmt); @@ -401,7 +423,7 @@ extern "C" char *engine_detect_drift(uint64_t project_id) "' declared in " + file_path + " but no enforcing code detected"; - g_store->insertFinding( + ctx->store->insertFinding( project_id, "BrokenContract", kDriftSeverityHard, 0, @@ -409,15 +431,17 @@ extern "C" char *engine_detect_drift(uint64_t project_id) if (!first) json << ","; first = false; - json << "{\"type\":\"BrokenContract\"," - << "\"severity\":" - << kDriftSeverityHard - << ",\"contract_id\":" - << cid << ",\"subject\":\"" - << jsonEscape(name) << "\"" - << ",\"detail\":\"" - << jsonEscape(detail) - << "\"}"; + util::JsonWriter el; + el.beginObject(); + el.key("type").value( + "BrokenContract"); + el.key("severity") + .value(kDriftSeverityHard); + el.key("contract_id").value(cid); + el.key("subject").value(name); + el.key("detail").value(detail); + el.endObject(); + json << el.str(); drifts_found++; } } @@ -428,13 +452,11 @@ extern "C" char *engine_detect_drift(uint64_t project_id) json << "],\"drifts_found\":" << drifts_found << "}"; return dupString(json.str()); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_detect_drift] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_detect_drift", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_detect_drift] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_detect_drift", "unknown exception")); } } @@ -458,14 +480,18 @@ extern "C" char *engine_detect_drift(uint64_t project_id) // // MEMORY: caller MUST free the returned char* via engine_free_string(). // THREAD SAFETY: single-threaded (GraphStore writer invariant). -extern "C" char *engine_detect_documentation_drift(uint64_t project_id) +extern "C" char *engine_detect_documentation_drift(engine_t handle, + uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"not initialized\"}"); // Read README content from the document table. - sqlite3 *db = g_store->handle(); + sqlite3 *db = ctx->store->handle(); if (!db) return dupString( "{\"error\":\"db handle null " @@ -509,7 +535,7 @@ extern "C" char *engine_detect_documentation_drift(uint64_t project_id) for (const auto &claim : claims) { claimed.push_back(claim.display); int64_t count = verify::countEntitiesByLanguage( - *g_store, project_id, claim.canonical); + *ctx->store, project_id, claim.canonical); if (count > 0) found.push_back(claim.display); else @@ -527,56 +553,51 @@ extern "C" char *engine_detect_documentation_drift(uint64_t project_id) // Persist a finding row for each missing language. for (size_t i = 0; i < missing.size(); ++i) { - g_store->insertFinding(project_id, "DocumentationDrift", - verify::kDriftSeverityDoc, 0, - missing_details[i], - verify::kDriftConfidenceDoc); + ctx->store->insertFinding(project_id, + "DocumentationDrift", + verify::kDriftSeverityDoc, 0, + missing_details[i], + verify::kDriftConfidenceDoc); } int drifts_found = static_cast(missing.size()); // Serialize to JSON: // {"claimed_languages":[...],"found_languages":[...], // "missing_languages":[...],"drifts":[...],"drifts_found":N} - std::ostringstream json; - json << "{\"claimed_languages\":["; - for (size_t i = 0; i < claimed.size(); ++i) { - if (i > 0) - json << ","; - json << "\"" << jsonEscape(claimed[i]) << "\""; - } - json << "],\"found_languages\":["; - for (size_t i = 0; i < found.size(); ++i) { - if (i > 0) - json << ","; - json << "\"" << jsonEscape(found[i]) << "\""; - } - json << "],\"missing_languages\":["; + util::JsonWriter json; + json.beginObject(); + json.key("claimed_languages").beginArray(); + for (const auto &lang : claimed) + json.value(lang); + json.endArray(); + json.key("found_languages").beginArray(); + for (const auto &lang : found) + json.value(lang); + json.endArray(); + json.key("missing_languages").beginArray(); + for (const auto &lang : missing) + json.value(lang); + json.endArray(); + json.key("drifts").beginArray(); for (size_t i = 0; i < missing.size(); ++i) { - if (i > 0) - json << ","; - json << "\"" << jsonEscape(missing[i]) << "\""; + json.beginObject(); + json.key("type").value("DocumentationDrift"); + json.key("severity").value(verify::kDriftSeverityDoc); + json.key("subject").value(missing[i]); + json.key("detail").value(missing_details[i]); + json.endObject(); } - json << "],\"drifts\":["; - for (size_t i = 0; i < missing.size(); ++i) { - if (i > 0) - json << ","; - json << "{\"type\":\"DocumentationDrift\"" - << ",\"severity\":" << verify::kDriftSeverityDoc - << ",\"subject\":\"" << jsonEscape(missing[i]) - << "\"" - << ",\"detail\":\"" - << jsonEscape(missing_details[i]) << "\"}"; - } - json << "],\"drifts_found\":" << drifts_found << "}"; + json.endArray(); + json.key("drifts_found").value(drifts_found); + json.endObject(); return dupString(json.str()); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_detect_documentation_drift] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_detect_documentation_drift", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_detect_documentation_drift] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_detect_documentation_drift", + "unknown exception")); } } @@ -598,26 +619,30 @@ extern "C" char *engine_detect_documentation_drift(uint64_t project_id) // // MEMORY: caller MUST free the returned char* via engine_free_string(). // THREAD SAFETY: single-threaded (GraphStore writer invariant). -extern "C" char *engine_detect_capability_drift(uint64_t project_id) +extern "C" char *engine_detect_capability_drift(engine_t handle, + uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString( "{\"error\":\"not initialized " "[module=ffi, method=engine_detect_capability_drift]\"}"); - sqlite3 *db = g_store->handle(); + sqlite3 *db = ctx->store->handle(); if (!db) return dupString( "{\"error\":\"db not open " "[module=ffi, method=engine_detect_capability_drift]\"}"); auto drifts = - verify::detectCapabilityDrift(*g_store, project_id); + verify::detectCapabilityDrift(*ctx->store, project_id); // Persist each drift as a finding row. for (const auto &d : drifts) { - g_store->insertFinding( + ctx->store->insertFinding( project_id, "CapabilityDrift", verify::kDriftSeverityCapability, 0, d.detail, verify::kDriftConfidenceCapability); @@ -652,13 +677,15 @@ extern "C" char *engine_detect_capability_drift(uint64_t project_id) for (size_t i = 0; i < drifts.size(); ++i) { if (i > 0) json << ","; - json << "{\"type\":\"CapabilityDrift\"" - << ",\"severity\":" - << verify::kDriftSeverityCapability - << ",\"subject\":\"" - << jsonEscape(drifts[i].subject) << "\"" - << ",\"detail\":\"" << jsonEscape(drifts[i].detail) - << "\"}"; + util::JsonWriter el; + el.beginObject(); + el.key("type").value("CapabilityDrift"); + el.key("severity") + .value(verify::kDriftSeverityCapability); + el.key("subject").value(drifts[i].subject); + el.key("detail").value(drifts[i].detail); + el.endObject(); + json << el.str(); } json << "],\"drifts_found\":" << drifts.size(); // Honest reporting: when the capability table is empty there is @@ -671,13 +698,12 @@ extern "C" char *engine_detect_capability_drift(uint64_t project_id) json << "}"; return dupString(json.str()); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_detect_capability_drift] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_detect_capability_drift", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_detect_capability_drift] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_detect_capability_drift", + "unknown exception")); } } @@ -701,23 +727,28 @@ extern "C" char *engine_detect_capability_drift(uint64_t project_id) // // MEMORY: caller MUST free the returned char* via engine_free_string(). // THREAD SAFETY: single-threaded (GraphStore writer invariant). -extern "C" char *engine_detect_architecture_drift(uint64_t project_id) +extern "C" char *engine_detect_architecture_drift(engine_t handle, + uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString( "{\"error\":\"not initialized " "[module=ffi, method=engine_detect_architecture_drift]\"}"); - auto drifts = - verify::detectArchitectureDrift(*g_store, project_id); + auto drifts = verify::detectArchitectureDrift(*ctx->store, + project_id); // Persist each drift as a finding row. for (const auto &d : drifts) { - g_store->insertFinding(project_id, "ArchitectureDrift", - verify::kDriftSeverityArch, 0, - d.detail, - verify::kDriftConfidenceArch); + ctx->store->insertFinding(project_id, + "ArchitectureDrift", + verify::kDriftSeverityArch, 0, + d.detail, + verify::kDriftConfidenceArch); } std::ostringstream json; @@ -725,22 +756,23 @@ extern "C" char *engine_detect_architecture_drift(uint64_t project_id) for (size_t i = 0; i < drifts.size(); ++i) { if (i > 0) json << ","; - json << "{\"type\":\"ArchitectureDrift\"" - << ",\"severity\":" << verify::kDriftSeverityArch - << ",\"subject\":\"" - << jsonEscape(drifts[i].subject) << "\"" - << ",\"detail\":\"" << jsonEscape(drifts[i].detail) - << "\"}"; + util::JsonWriter el; + el.beginObject(); + el.key("type").value("ArchitectureDrift"); + el.key("severity").value(verify::kDriftSeverityArch); + el.key("subject").value(drifts[i].subject); + el.key("detail").value(drifts[i].detail); + el.endObject(); + json << el.str(); } json << "],\"drifts_found\":" << drifts.size() << "}"; return dupString(json.str()); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_detect_architecture_drift] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_detect_architecture_drift", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_detect_architecture_drift] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_detect_architecture_drift", + "unknown exception")); } } diff --git a/engine/src/engine_verify_explain.cpp b/engine/src/engine_verify_explain.cpp new file mode 100644 index 0000000..1d9cb69 --- /dev/null +++ b/engine/src/engine_verify_explain.cpp @@ -0,0 +1,434 @@ +// engine_verify_explain.cpp — the module Knowledge Card. +// +// Split out of engine_verify_ffi.cpp (see plan/rules/code_rules.md +// 1000-line rule). engine_explain_module is the biggest single export in +// the verify surface: it assembles summary, entities, capabilities, +// contracts, findings and integrity score into one JSON card. Kept +// together because every section is a step of the same card build and +// they share the same verifier-registry bootstrapping. + +#include "util/json_writer.h" +#include "engine_internal.h" +#include "async_knowledge.h" +#include "platform_win.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "verify/architecture_verifier.h" +#include "verify/architecture_drift.h" +#include "verify/capability_drift.h" +#include "verify/capability_verifier.h" +#include "verify/claim.h" +#include "verify/claim_parser.h" +#include "verify/contract_verifier.h" +#include "verify/documentation_drift.h" +#include "verify/function_implements_verifier.h" +#include "verify/registry.h" +#include "verify/dead_code_inspector.h" +#include "verify/ffi_internal.h" + +// ─── Named Constants (local to this translation unit) ────────── +// Duplicated from engine_verify_ffi.cpp rather than shared: they are +// TU-local tuning knobs, and a shared header would make it easy to change +// one export's scoring while silently changing another's. + +// Maximum number of entity sample rows returned by engine_explain_module. +static constexpr int kEntitySampleLimit = 10; +/// Maximum number of cross-module dependency edges to return per direction. +static constexpr int kCrossModuleEdgeLimit = 20; + +// Integrity score parameters for engine_explain_module. +static constexpr int kIntegrityMax = 100; +static constexpr int kIntegritySev2Penalty = 10; +static constexpr int kIntegritySev1Penalty = 5; + +// engine_explain_module — return a Knowledge Card for a named module. +// +// Looks up the module by name (case-insensitive) in the modules table. If no +// row exists, falls back to deriving module info from file paths so the tool +// works on freshly-indexed projects where the modules table has not yet been +// populated. +// +// Output JSON (Knowledge Card): +// {"module":"engine","summary":"...","entities":{"count":N,"sample":[...]}, +// "capabilities":[...],"contracts":[...],"findings":[...], +// "integrity":92} +// On error: {"error":"module not found","module":""} +// +// MEMORY: caller MUST free the returned char* via engine_free_string(). +// THREAD SAFETY: single-threaded (GraphStore writer invariant). +extern "C" char *engine_explain_module(engine_t handle, uint64_t project_id, + const char *module_name) +{ + EngineContext *ctx = engineInstance(handle); + + try { + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) + return dupString( + "{\"error\":\"not initialized " + "[module=ffi, method=engine_explain_module]\"}"); + if (!module_name || !*module_name) + return dupString( + "{\"error\":\"module_name is empty " + "[module=ffi, method=engine_explain_module]\"}"); + + std::string name(module_name); + std::string name_lower; + name_lower.reserve(name.size()); + for (char c : name) + name_lower += static_cast( + std::tolower(static_cast(c))); + + sqlite3 *db = ctx->store->handle(); + if (!db) + return dupString( + "{\"error\":\"db not open " + "[module=ffi, method=engine_explain_module]\"}"); + + // Resolve module row (case-insensitive name match). + // No project_id filter: module names are globally unique in + // both serial (single project) and parallel (merged) products, + // and the MCP layer's restored project_id may differ from the + // owning module's project_id in parallel products. + std::string summary; + bool found = false; + { + const char *sql = "SELECT name FROM modules " + "WHERE LOWER(name)=? " + "LIMIT 1"; + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == + SQLITE_OK) { + sqlite3_bind_text(stmt, 1, name_lower.c_str(), + -1, SQLITE_STATIC); + if (sqlite3_step(stmt) == SQLITE_ROW) { + found = true; + const char *n = + reinterpret_cast( + sqlite3_column_text( + stmt, 0)); + if (n) + summary = "Module: " + + std::string(n); + } + sqlite3_finalize(stmt); + } + } + + // Fallback: derive module existence from file paths so the tool works + // even when the modules table is empty (e.g. freshly indexed project). + // Use "%//%" so paths with a leading "./" or a parent directory + // still match. The slashes prevent partial segment matches (e.g. a + // query for "engine" won't match "./my_engine/foo"). + if (!found) { + // Build the LIKE pattern. The fallback targets files under + // `name`, so an absolute path (leading '/') must not gain a + // second slash: "%/" + "/Users/..." would produce + // "%//Users/..." which never matches a single-slash path. + // Relative module names keep the leading "/" to avoid + // partial-segment matches ("engine" vs "./my_engine"). + std::string like = name[0] == '/' ? "%" + name + "/%" : + "%/" + name + "/%"; + const char *sql = "SELECT COUNT(*) FROM files " + "WHERE path LIKE ?"; + sqlite3_stmt *stmt = nullptr; + int count = 0; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == + SQLITE_OK) { + sqlite3_bind_text(stmt, 1, like.c_str(), -1, + SQLITE_STATIC); + if (sqlite3_step(stmt) == SQLITE_ROW) + count = sqlite3_column_int(stmt, 0); + sqlite3_finalize(stmt); + } + if (count == 0) { + util::JsonWriter j; + j.beginObject(); + j.key("error").value("module not found"); + j.key("module").value(name); + j.endObject(); + return dupString(j.str()); + } + found = true; + summary = "Module derived from " + + std::to_string(count) + " files under " + + name + "/"; + } + + util::JsonWriter json; + json.beginObject(); + json.key("module").value(name); + json.key("summary").value(summary); + + // Entities: count + sample (up to 10) for this module's files. + // Use the same "%//%" pattern as the fallback check above so + // paths with a leading "./" match consistently. + { + std::string like = "%/" + name + "/%"; + // v0.2.5: read from the canonical `entity` table (the legacy + // graph_nodes table is empty in the canonical schema, so this + // previously always returned zero entities). entity.id + // preserves the legacy graph node identity. + std::string sql_str = + "SELECT name, kind, file_path FROM entity " + "WHERE file_path LIKE ? " + "ORDER BY id LIMIT " + + std::to_string(kEntitySampleLimit); + sqlite3_stmt *stmt = nullptr; + int total = 0; + // Count first + const char *csql = "SELECT COUNT(*) FROM entity " + "WHERE file_path LIKE ?"; + if (sqlite3_prepare_v2(db, csql, -1, &stmt, nullptr) == + SQLITE_OK) { + sqlite3_bind_text(stmt, 1, like.c_str(), -1, + SQLITE_STATIC); + if (sqlite3_step(stmt) == SQLITE_ROW) + total = sqlite3_column_int(stmt, 0); + sqlite3_finalize(stmt); + } + json.key("entities").beginObject(); + json.key("count").value(total); + json.key("sample").beginArray(); + if (sqlite3_prepare_v2(db, sql_str.c_str(), -1, &stmt, + nullptr) == SQLITE_OK) { + sqlite3_bind_text(stmt, 1, like.c_str(), -1, + SQLITE_STATIC); + while (sqlite3_step(stmt) == SQLITE_ROW) { + const char *n = + reinterpret_cast( + sqlite3_column_text( + stmt, 0)); + int kind = sqlite3_column_int(stmt, 1); + const char *fp = + reinterpret_cast( + sqlite3_column_text( + stmt, 2)); + json.beginObject(); + json.key("name").value(n ? n : ""); + json.key("kind").value(kind); + json.key("file_path") + .value(fp ? fp : ""); + json.endObject(); + } + sqlite3_finalize(stmt); + } + json.endArray(); + json.endObject(); + } + + // Capabilities + { + json.key("capabilities").beginArray(); + const char *sql = + "SELECT id, name, summary FROM capability " + "ORDER BY id"; + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == + SQLITE_OK) { + while (sqlite3_step(stmt) == SQLITE_ROW) { + int64_t id = + sqlite3_column_int64(stmt, 0); + const char *n = + reinterpret_cast( + sqlite3_column_text( + stmt, 1)); + const char *s = + reinterpret_cast( + sqlite3_column_text( + stmt, 2)); + json.beginObject(); + json.key("id").value(id); + json.key("name").value(n ? n : ""); + json.key("summary").value(s ? s : ""); + json.endObject(); + } + sqlite3_finalize(stmt); + } + json.endArray(); + } + + // Contracts + { + json.key("contracts").beginArray(); + const char *sql = + "SELECT id, name, origin, claim_text FROM contract " + "ORDER BY id"; + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == + SQLITE_OK) { + while (sqlite3_step(stmt) == SQLITE_ROW) { + int64_t id = + sqlite3_column_int64(stmt, 0); + const char *n = + reinterpret_cast( + sqlite3_column_text( + stmt, 1)); + const char *o = + reinterpret_cast( + sqlite3_column_text( + stmt, 2)); + const char *c = + reinterpret_cast( + sqlite3_column_text( + stmt, 3)); + json.beginObject(); + json.key("id").value(id); + json.key("name").value(n ? n : ""); + json.key("origin").value(o ? o : ""); + json.key("claim_text").value(c ? c : ""); + json.endObject(); + } + sqlite3_finalize(stmt); + } + json.endArray(); + } + + // Findings + integrity score + { + json.key("findings").beginArray(); + const char *sql = + "SELECT id, rule, severity, description, confidence " + "FROM finding ORDER BY id"; + sqlite3_stmt *stmt = nullptr; + int sev2 = 0, sev1 = 0; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == + SQLITE_OK) { + while (sqlite3_step(stmt) == SQLITE_ROW) { + int64_t id = + sqlite3_column_int64(stmt, 0); + const char *r = + reinterpret_cast( + sqlite3_column_text( + stmt, 1)); + int sev = sqlite3_column_int(stmt, 2); + const char *d = + reinterpret_cast( + sqlite3_column_text( + stmt, 3)); + double conf = + sqlite3_column_double(stmt, 4); + json.beginObject(); + json.key("id").value(id); + json.key("rule").value(r ? r : ""); + json.key("severity").value(sev); + json.key("description") + .value(d ? d : ""); + json.key("confidence").value(conf); + json.endObject(); + if (sev == 2) + sev2++; + else if (sev == 1) + sev1++; + } + sqlite3_finalize(stmt); + } + json.endArray(); + int integrity = kIntegrityMax - + kIntegritySev2Penalty * sev2 - + kIntegritySev1Penalty * sev1; + if (integrity < 0) + integrity = 0; + if (integrity > kIntegrityMax) + integrity = kIntegrityMax; + json.key("integrity").value(integrity); + } + + // Cross-module dependencies from the pre-computed module_edge table. + // Populated by the async knowledge builder after indexing. When the + // table is empty (e.g. async build not yet complete), both arrays are + // empty — the card still returns successfully. + json.key("cross_module").beginObject(); + + // depends_on: modules that this module calls (outgoing edges). + { + json.key("depends_on").beginArray(); + const char *sql = + "SELECT tgt_module, edge_count " + "FROM module_edge " + "WHERE project_id=? AND src_module LIKE ? " + "ORDER BY edge_count DESC LIMIT ?"; + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == + SQLITE_OK) { + std::string like = "%/" + name + "/%"; + sqlite3_bind_int64( + stmt, 1, + static_cast(project_id)); + sqlite3_bind_text(stmt, 2, like.c_str(), -1, + SQLITE_STATIC); + sqlite3_bind_int(stmt, 3, + kCrossModuleEdgeLimit); + while (sqlite3_step(stmt) == SQLITE_ROW) { + const char *m = + reinterpret_cast( + sqlite3_column_text( + stmt, 0)); + int64_t ec = + sqlite3_column_int64(stmt, 1); + json.beginObject(); + json.key("module").value(m ? m : ""); + json.key("edge_count").value(ec); + json.endObject(); + } + sqlite3_finalize(stmt); + } + json.endArray(); + } + + // depended_by: modules that call this module (incoming edges). + { + json.key("depended_by").beginArray(); + const char *sql = + "SELECT src_module, edge_count " + "FROM module_edge " + "WHERE project_id=? AND tgt_module LIKE ? " + "ORDER BY edge_count DESC LIMIT ?"; + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == + SQLITE_OK) { + std::string like = "%/" + name + "/%"; + sqlite3_bind_int64( + stmt, 1, + static_cast(project_id)); + sqlite3_bind_text(stmt, 2, like.c_str(), -1, + SQLITE_STATIC); + sqlite3_bind_int(stmt, 3, + kCrossModuleEdgeLimit); + while (sqlite3_step(stmt) == SQLITE_ROW) { + const char *m = + reinterpret_cast( + sqlite3_column_text( + stmt, 0)); + int64_t ec = + sqlite3_column_int64(stmt, 1); + json.beginObject(); + json.key("module").value(m ? m : ""); + json.key("edge_count").value(ec); + json.endObject(); + } + sqlite3_finalize(stmt); + } + json.endArray(); + } + + json.endObject(); + json.endObject(); + return dupString(json.str()); + } catch (const std::exception &e) { + return dupString(util::errorEnvelope( + "ffi", "engine_explain_module", e.what())); + } catch (...) { + return dupString(util::errorEnvelope( + "ffi", "engine_explain_module", "unknown exception")); + } +} diff --git a/engine/src/engine_verify_ffi.cpp b/engine/src/engine_verify_ffi.cpp index 9a344d9..47253fc 100644 --- a/engine/src/engine_verify_ffi.cpp +++ b/engine/src/engine_verify_ffi.cpp @@ -10,7 +10,9 @@ // release with engine_free_string(). Null store/query inputs return an // error JSON object instead of crashing. +#include "util/json_writer.h" #include "engine_internal.h" +#include "async_knowledge.h" #include "platform_win.h" #include @@ -45,15 +47,14 @@ namespace // Trust score penalty per non-supported finding in engine_verify_integrity. static constexpr double kTrustScorePenalty = 0.1; -// Maximum number of entity sample rows returned by engine_explain_module. -static constexpr int kEntitySampleLimit = 10; -/// Maximum number of cross-module dependency edges to return per direction. -static constexpr int kCrossModuleEdgeLimit = 20; - -// Integrity score parameters for engine_explain_module. -static constexpr int kIntegrityMax = 100; -static constexpr int kIntegritySev2Penalty = 10; -static constexpr int kIntegritySev1Penalty = 5; +// Default cap on the number of findings serialized into the +// engine_verify_integrity response, and the absolute upper bound the +// caller may request. The findings array is unbounded otherwise: a +// 10k-entity index produces one DeadCode/Architecture/ModuleCoupling +// finding per module/function and the JSON overflows the MCP write cap +// (T5 finding #15). The real totals are always reported alongside. +static constexpr int kDefaultMaxFindings = 200; +static constexpr int kMaxFindingsCap = 2000; // ─── JSON Helpers ─────────────────────────────────────────────── @@ -127,7 +128,7 @@ std::optional parseClaimType(const std::string &s) // global registry. Delegates to VerifierRegistry::ensureDefaultVerifiers, // which checks the actual registry state (not a process-level static flag) // and only re-registers when empty. This fixes the lifecycle bug A15: -// engine_shutdown() cleared the registry but the old `static bool +// engine_destroy() cleared the registry but the old `static bool // initialized` flag stayed true, so the next ensureVerifiersRegistered() // was a no-op and the registry stayed empty → every claim returned // "no verifier registered". @@ -180,11 +181,12 @@ makeVerifierForClaim(const verify::Claim &claim, store::GraphStore *store, namespace verify_ffi { -VerifyResult verify_one_claim(uint64_t project_id, const verify::Claim &claim) +VerifyResult verify_one_claim(EngineContext *ctx, uint64_t project_id, + const verify::Claim &claim) { VerifyResult result; - int64_t claim_id = g_store->insertClaim(project_id, claim); + int64_t claim_id = ctx->store->insertClaim(project_id, claim); if (claim_id < 0) { result.json = dupString("{\"error\":\"failed to persist claim " @@ -208,21 +210,24 @@ VerifyResult verify_one_claim(uint64_t project_id, const verify::Claim &claim) const std::string detail = (reg.verifier_count() == 0) ? std::string("verifier registry is empty " - "(engine_init not called or " - "engine_shutdown cleared it) " + "(engine_create not called or " + "the instance was destroyed) " "[module=ffi, method=" "verify_one_claim]") : (std::string( "no verifier accepts claim type '") + verify::claimTypeWireName(claim.type) + "' [module=ffi, method=verify_one_claim]"); - std::ostringstream j; - j << "{\"claim_id\":" << claim_id - << ",\"verdict\":\"Unknown\",\"confidence\":0" - << ",\"verifier\":null" - << ",\"error_code\":\"" << code << "\"" - << ",\"detail\":\"" << jsonEscape(detail) << "\"" - << ",\"evidence_facts\":[]}"; + util::JsonWriter j; + j.beginObject(); + j.key("claim_id").value(claim_id); + j.key("verdict").value("Unknown"); + j.key("confidence").value(0); + j.key("verifier").nullValue(); + j.key("error_code").value(code); + j.key("detail").value(detail); + j.key("evidence_facts").beginArray().endArray(); + j.endObject(); result.json = dupString(j.str()); result.verdict = verify::Verdict::Unknown; return result; @@ -231,17 +236,20 @@ VerifyResult verify_one_claim(uint64_t project_id, const verify::Claim &claim) // Build a fresh verifier bound to the caller's project_id so verify() // queries the right project's data. The registry's matched pointer is // only used to confirm that SOME verifier accepts this claim type. - auto v = makeVerifierForClaim(claim, g_store.get(), project_id); + auto v = makeVerifierForClaim(claim, ctx->store.get(), project_id); if (!v) { - std::ostringstream j; - j << "{\"claim_id\":" << claim_id - << ",\"verdict\":\"Unknown\",\"confidence\":0" - << ",\"verifier\":null" - << ",\"error_code\":\"verifier_execution_failed\"" - << ",\"detail\":\"verifier " - "implementation unavailable for this claim type " - "[module=ffi, method=verify_one_claim]\"" - << ",\"evidence_facts\":[]}"; + util::JsonWriter j; + j.beginObject(); + j.key("claim_id").value(claim_id); + j.key("verdict").value("Unknown"); + j.key("confidence").value(0); + j.key("verifier").nullValue(); + j.key("error_code").value("verifier_execution_failed"); + j.key("detail").value( + "verifier implementation unavailable for this claim type " + "[module=ffi, method=verify_one_claim]"); + j.key("evidence_facts").beginArray().endArray(); + j.endObject(); result.json = dupString(j.str()); result.verdict = verify::Verdict::Unknown; return result; @@ -254,35 +262,43 @@ VerifyResult verify_one_claim(uint64_t project_id, const verify::Claim &claim) try { rec = v->verify(claim); } catch (const std::exception &e) { - std::ostringstream j; - j << "{\"claim_id\":" << claim_id - << ",\"verdict\":\"Unknown\",\"confidence\":0" - << ",\"verifier\":\"" << jsonEscape(v->name()) << "\"" - << ",\"error_code\":\"verifier_execution_failed\"" - << ",\"detail\":\"verifier threw: " << jsonEscape(e.what()) - << " [module=ffi, method=verify_one_claim]\"" - << ",\"evidence_facts\":[]}"; + util::JsonWriter j; + j.beginObject(); + j.key("claim_id").value(claim_id); + j.key("verdict").value("Unknown"); + j.key("confidence").value(0); + j.key("verifier").value(v->name()); + j.key("error_code").value("verifier_execution_failed"); + j.key("detail").value(std::string("verifier threw: ") + + e.what() + + " [module=ffi, method=verify_one_claim]"); + j.key("evidence_facts").beginArray().endArray(); + j.endObject(); result.json = dupString(j.str()); result.verdict = verify::Verdict::Unknown; return result; } catch (...) { - std::ostringstream j; - j << "{\"claim_id\":" << claim_id - << ",\"verdict\":\"Unknown\",\"confidence\":0" - << ",\"verifier\":\"" << jsonEscape(v->name()) << "\"" - << ",\"error_code\":\"verifier_execution_failed\"" - << ",\"detail\":\"verifier threw unknown exception " - "[module=ffi, method=verify_one_claim]\"" - << ",\"evidence_facts\":[]}"; + util::JsonWriter j; + j.beginObject(); + j.key("claim_id").value(claim_id); + j.key("verdict").value("Unknown"); + j.key("confidence").value(0); + j.key("verifier").value(v->name()); + j.key("error_code").value("verifier_execution_failed"); + j.key("detail").value("verifier threw unknown exception " + "[module=ffi, method=verify_one_claim]"); + j.key("evidence_facts").beginArray().endArray(); + j.endObject(); result.json = dupString(j.str()); result.verdict = verify::Verdict::Unknown; return result; } rec.claim_id = claim_id; - int64_t evidence_id = - g_store->insertEvidence(claim_id, rec.verdict, rec.confidence, - rec.verifier_name, rec.detail); + int64_t evidence_id = ctx->store->insertEvidence(claim_id, rec.verdict, + rec.confidence, + rec.verifier_name, + rec.detail); if (evidence_id < 0) { result.json = dupString("{\"error\":\"failed to persist evidence " @@ -292,7 +308,8 @@ VerifyResult verify_one_claim(uint64_t project_id, const verify::Claim &claim) } for (const auto &f : rec.facts) { - g_store->insertEvidenceFact(evidence_id, f.first, f.second, ""); + ctx->store->insertEvidenceFact(evidence_id, f.first, f.second, + ""); } // Step 9.6: when the verifier returned Unknown because the evidence @@ -300,11 +317,12 @@ VerifyResult verify_one_claim(uint64_t project_id, const verify::Claim &claim) // callers can distinguish "no evidence yet" from a normal Unknown // verdict. The verifier signals this via a low confidence + the // "evidence backend not ready" prefix in the detail string. - std::ostringstream j; - j << "{\"claim_id\":" << claim_id << ",\"verdict\":\"" - << verify::verdictName(rec.verdict) << "\"" - << ",\"confidence\":" << rec.confidence << ",\"verifier\":\"" - << jsonEscape(rec.verifier_name) << "\""; + util::JsonWriter j; + j.beginObject(); + j.key("claim_id").value(claim_id); + j.key("verdict").value(verify::verdictName(rec.verdict)); + j.key("confidence").value(rec.confidence); + j.key("verifier").value(rec.verifier_name); // Tag evidence_backend_not_ready when the verifier reported it. The // detail string is the canonical signal (set by evidence_backend_ready // helpers in each verifier) so we don't need a separate enum field on @@ -312,25 +330,25 @@ VerifyResult verify_one_claim(uint64_t project_id, const verify::Claim &claim) if (rec.verdict == verify::Verdict::Unknown && rec.detail.find("evidence backend not ready") != std::string::npos) { - j << ",\"error_code\":\"evidence_backend_not_ready\""; + j.key("error_code").value("evidence_backend_not_ready"); } - j << ",\"detail\":\"" << jsonEscape(rec.detail) << "\"" - << ",\"evidence_facts\":["; - bool first = true; + j.key("detail").value(rec.detail); + j.key("evidence_facts").beginArray(); for (const auto &f : rec.facts) { - if (!first) - j << ","; - first = false; - j << "{\"kind\":" << f.first << ",\"ref\":" << f.second << "}"; + j.beginObject(); + j.key("kind").value(f.first); + j.key("ref").value(f.second); + j.endObject(); } - j << "]}"; + j.endArray(); + j.endObject(); result.json = dupString(j.str()); result.verdict = rec.verdict; return result; } -BatchResult verify_claim_batch(uint64_t project_id, const std::string &text, - const char *source_kind, +BatchResult verify_claim_batch(EngineContext *ctx, uint64_t project_id, + const std::string &text, const char *source_kind, const std::string &source_ref) { BatchResult out; @@ -345,7 +363,7 @@ BatchResult verify_claim_batch(uint64_t project_id, const std::string &text, if (!first) json << ","; first = false; - VerifyResult result = verify_one_claim(project_id, c); + VerifyResult result = verify_one_claim(ctx, project_id, c); if (result.json) { json << result.json; switch (result.verdict) { @@ -367,6 +385,14 @@ BatchResult verify_claim_batch(uint64_t project_id, const std::string &text, } json << "]"; out.results_json = json.str(); + // Status is decided once here (see BatchResult::status) so the wrapping + // tools cannot drift apart on what "nothing to report" means. + if (out.claims_count == 0) + out.status = "no_claims"; + else if (out.supported + out.contradicted == 0) + out.status = "no_verdicts"; + else + out.status = "verified"; return out; } @@ -383,26 +409,43 @@ BatchResult verify_claim_batch(uint64_t project_id, const std::string &text, // // MEMORY: caller MUST free the returned char* via engine_free_string(). // THREAD SAFETY: single-threaded (GraphStore writer invariant). -extern "C" char *engine_verify_integrity(uint64_t project_id) +// +// `max_findings` caps how many entries are serialized into the +// `findings` array (0 or negative selects the default; the absolute +// upper bound is kMaxFindingsCap). `truncated` is true when the array +// was cut short. `total` counts every verdict (Supported included, +// which are not listed as findings), so it is normally larger than +// the array length. +extern "C" char *engine_verify_integrity(engine_t handle, uint64_t project_id, + int max_findings) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"not initialized\"}"); // Arm the query timeout (10s) so a hung query never blocks // the caller indefinitely. The guard disarms on scope exit. - store::GraphStore::QueryDeadlineGuard guard(g_store.get(), + store::GraphStore::QueryDeadlineGuard guard(ctx->store.get(), 10000); (void)guard; + int limit = (max_findings <= 0) ? kDefaultMaxFindings : + max_findings; + if (limit > kMaxFindingsCap) + limit = kMaxFindingsCap; + int supported = 0, contradicted = 0, unknown = 0, orphans = 0; + int emitted = 0; - std::ostringstream json; - json << "{\"findings\":["; - bool first = true; + util::JsonWriter json; + json.beginObject(); + json.key("findings").beginArray(); // Iterate capabilities -> CapabilityExists claims - auto caps = g_store->listCapabilities(project_id); + auto caps = ctx->store->listCapabilities(project_id); for (const auto &cap : caps) { verify::Claim claim; claim.type = verify::ClaimType::CapabilityExists; @@ -412,7 +455,7 @@ extern "C" char *engine_verify_integrity(uint64_t project_id) claim.source_kind = "capability"; claim.source_ref = std::to_string(cap.first); - auto v = makeVerifierForClaim(claim, g_store.get(), + auto v = makeVerifierForClaim(claim, ctx->store.get(), project_id); if (!v) continue; @@ -435,21 +478,23 @@ extern "C" char *engine_verify_integrity(uint64_t project_id) std::string desc = "Capability '" + cap.second + "' " + verify::verdictName(rec.verdict) + ": " + rec.detail; - g_store->insertFinding(project_id, "CapabilityVerifier", - severity, 0, desc, - rec.confidence); - if (!first) - json << ","; - first = false; - json << "{\"type\":\"CapabilityVerifier\"," - << "\"description\":\"" << jsonEscape(desc) - << "\"," - << "\"confidence\":" << rec.confidence << "}"; + ctx->store->insertFinding(project_id, + "CapabilityVerifier", + severity, 0, desc, + rec.confidence); + if (emitted < limit) { + ++emitted; + json.beginObject(); + json.key("type").value("CapabilityVerifier"); + json.key("description").value(desc); + json.key("confidence").value(rec.confidence); + json.endObject(); + } } // Iterate contracts -> ContractHolds claims. ContractVerifier is // registered, so makeVerifierForClaim returns a valid verifier instance. - auto contracts = g_store->listContracts(project_id); + auto contracts = ctx->store->listContracts(project_id); for (const auto &ct : contracts) { verify::Claim claim; claim.type = verify::ClaimType::ContractHolds; @@ -459,7 +504,7 @@ extern "C" char *engine_verify_integrity(uint64_t project_id) claim.source_kind = "contract"; claim.source_ref = std::to_string(ct.first); - auto v = makeVerifierForClaim(claim, g_store.get(), + auto v = makeVerifierForClaim(claim, ctx->store.get(), project_id); if (!v) continue; @@ -482,16 +527,17 @@ extern "C" char *engine_verify_integrity(uint64_t project_id) std::string desc = "Contract '" + ct.second + "' " + verify::verdictName(rec.verdict) + ": " + rec.detail; - g_store->insertFinding(project_id, "ContractVerifier", - severity, 0, desc, - rec.confidence); - if (!first) - json << ","; - first = false; - json << "{\"type\":\"ContractVerifier\"," - << "\"description\":\"" << jsonEscape(desc) - << "\"," - << "\"confidence\":" << rec.confidence << "}"; + ctx->store->insertFinding(project_id, + "ContractVerifier", severity, + 0, desc, rec.confidence); + if (emitted < limit) { + ++emitted; + json.beginObject(); + json.key("type").value("ContractVerifier"); + json.key("description").value(desc); + json.key("confidence").value(rec.confidence); + json.endObject(); + } } // DeadCodeInspector: find orphan modules and functions. @@ -499,7 +545,7 @@ extern "C" char *engine_verify_integrity(uint64_t project_id) // land inside the JSON array (previously they were appended // after `],"total":N`, producing invalid JSON). { - verify::DeadCodeInspector dci(g_store.get(), + verify::DeadCodeInspector dci(ctx->store.get(), project_id); auto findings = dci.inspect(); for (auto &f : findings) { @@ -511,21 +557,32 @@ extern "C" char *engine_verify_integrity(uint64_t project_id) // actual claim verdicts instead of collapsing to 0 // whenever any orphan exists. orphans++; - if (!first) - json << ","; - first = false; - json << "{\"type\":\"DeadCodeInspector\"," - << "\"rule\":\"" << jsonEscape(f.type) - << "\"," - << "\"description\":\"" - << jsonEscape(f.description) << "\"," - << "\"confidence\":" << f.confidence - << "}"; + if (emitted < limit) { + ++emitted; + json.beginObject(); + json.key("type").value( + "DeadCodeInspector"); + json.key("rule").value(f.type); + json.key("description") + .value(f.description); + json.key("confidence") + .value(f.confidence); + json.endObject(); + } } } - json << "],\"total\":" - << (supported + contradicted + unknown + orphans); + int total = supported + contradicted + unknown + orphans; + // Only non-Supported verdicts become findings; Supported ones + // `continue` above and never enter the array. Truncation must + // therefore compare `emitted` against the emit-eligible count, + // not `total` — otherwise any Supported claim makes the flag + // report a cut that never happened. + int findings_total = contradicted + unknown + orphans; + json.endArray(); + json.key("total").value(total); + json.key("truncated").value(emitted < findings_total); + json.key("limit").value(limit); // Trust score: 1.0 - kTrustScorePenalty per non-supported finding, clamped to [0, 1]. // Orphans are excluded: they are informational findings, not @@ -535,20 +592,19 @@ extern "C" char *engine_verify_integrity(uint64_t project_id) static_cast(contradicted + unknown); if (trust_score < 0.0) trust_score = 0.0; - json << ",\"trust_score\":" << trust_score - << ",\"supported\":" << supported - << ",\"contradicted\":" << contradicted - << ",\"unknown\":" << unknown << ",\"orphans\":" << orphans - << "}"; + json.key("trust_score").value(trust_score); + json.key("supported").value(supported); + json.key("contradicted").value(contradicted); + json.key("unknown").value(unknown); + json.key("orphans").value(orphans); + json.endObject(); return dupString(json.str()); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_verify_integrity] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_verify_integrity", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_verify_integrity] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_verify_integrity", "unknown exception")); } } @@ -567,11 +623,14 @@ extern "C" char *engine_verify_integrity(uint64_t project_id) // // MEMORY: caller MUST free the returned char* via engine_free_string(). // THREAD SAFETY: single-threaded (GraphStore writer invariant). -extern "C" char *engine_verify_claim(uint64_t project_id, +extern "C" char *engine_verify_claim(engine_t handle, uint64_t project_id, const char *claim_json) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString("{\"error\":\"not initialized\"}"); if (!claim_json || !*claim_json) return dupString( @@ -590,14 +649,39 @@ extern "C" char *engine_verify_claim(uint64_t project_id, std::string type_str = jsonField(input, "type"); auto parsed_type = parseClaimType(type_str); if (!parsed_type) { - std::ostringstream err; - err << "{\"error\":\"unknown claim type '" - << jsonEscape(type_str) - << "'. Supported types: capability_exists, " - "contract_holds, architecture_follows, " - "function_implements " - "[module=ffi, method=engine_verify_claim]\"" - << ",\"error_code\":\"claim_type_unsupported\"}"; + util::JsonWriter err; + err.beginObject(); + // Two different mistakes land here and the message used to + // describe neither: an EMPTY type means the `claim` argument was + // not a JSON object carrying a `type` field at all — the common + // case is free text ("the engine supports X"), which the sibling + // verify_summary/verify review/reality tools DO accept, so sending + // prose here is an easy slip. A non-empty type is simply not in the + // contract. The two are separate error codes so a client can tell + // "you sent the wrong shape" from "you named a type that does not + // exist" — which is what this check was introduced to do. + if (type_str.empty()) { + err.key("error").value( + "missing claim type: `claim` must be a JSON object " + "with a \"type\" field (one of: capability_exists, " + "contract_holds, architecture_follows, " + "function_implements), e.g. " + "{\"type\":\"capability_exists\",\"subject\":\"Indexing\"}. " + "For free-form prose use verify_summary instead " + "[module=ffi, method=engine_verify_claim]"); + err.key("error_code") + .value("claim_type_missing"); + } else { + err.key("error").value( + "unknown claim type '" + type_str + + "'. Supported types: capability_exists, " + "contract_holds, architecture_follows, " + "function_implements " + "[module=ffi, method=engine_verify_claim]"); + err.key("error_code") + .value("claim_type_unsupported"); + } + err.endObject(); return dupString(err.str()); } claim.type = *parsed_type; @@ -621,16 +705,14 @@ extern "C" char *engine_verify_claim(uint64_t project_id, } verify_ffi::VerifyResult result = - verify_ffi::verify_one_claim(project_id, claim); + verify_ffi::verify_one_claim(ctx, project_id, claim); return result.json; } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_verify_claim] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_verify_claim", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_verify_claim] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_verify_claim", "unknown exception")); } } @@ -653,10 +735,14 @@ extern "C" char *engine_verify_claim(uint64_t project_id, // // MEMORY: caller MUST free the returned char* via engine_free_string(). // THREAD SAFETY: single-threaded (GraphStore writer invariant). -extern "C" char *engine_verify_summary(uint64_t project_id, const char *text) +extern "C" char *engine_verify_summary(engine_t handle, uint64_t project_id, + const char *text) { + EngineContext *ctx = engineInstance(handle); + try { - if (!g_store) + auto _store_guard = waitForKnowledgeBuilder(); + if (!ctx || !ctx->store) return dupString( "{\"error\":\"not initialized " "[module=ffi, method=engine_verify_summary]\"}"); @@ -667,37 +753,34 @@ extern "C" char *engine_verify_summary(uint64_t project_id, const char *text) std::string src(text); auto batch = verify_ffi::verify_claim_batch( - project_id, src, verify_ffi::kSourceKindAiSummary, + ctx, project_id, src, verify_ffi::kSourceKindAiSummary, src.substr(0, verify_ffi::kSourceRefMaxLen)); // ── End-to-end drift detection ── // Cross-reference AI claims against the actual codebase state. // Each drift finding represents a mismatch between documentation // (or AI summary) and the code. - auto doc_drifts = - verify::detectDocumentationDrift(*g_store, project_id); + auto doc_drifts = verify::detectDocumentationDrift(*ctx->store, + project_id); auto cap_drifts = - verify::detectCapabilityDrift(*g_store, project_id); - auto arch_drifts = - verify::detectArchitectureDrift(*g_store, project_id); + verify::detectCapabilityDrift(*ctx->store, project_id); + auto arch_drifts = verify::detectArchitectureDrift(*ctx->store, + project_id); size_t total_drifts = doc_drifts.size() + cap_drifts.size() + arch_drifts.size(); - // Build drifts JSON array. - std::ostringstream drifts_json; - drifts_json << "["; - bool drift_first = true; + util::JsonWriter json; + json.beginObject(); + json.key("claims_parsed").value(batch.claims_count); + json.key("results").raw(batch.results_json); + json.key("drifts").beginArray(); auto emit_drift = [&](const verify::DriftItem &d) { - if (!drift_first) - drifts_json << ","; - drift_first = false; - drifts_json - << "{\"type\":\"" << jsonEscape(d.type) << "\"" - << ",\"severity\":" << d.severity - << ",\"subject\":\"" << jsonEscape(d.subject) - << "\"" - << ",\"detail\":\"" << jsonEscape(d.detail) - << "\"}"; + json.beginObject(); + json.key("type").value(d.type); + json.key("severity").value(d.severity); + json.key("subject").value(d.subject); + json.key("detail").value(d.detail); + json.endObject(); }; for (const auto &d : doc_drifts) emit_drift(d); @@ -705,435 +788,44 @@ extern "C" char *engine_verify_summary(uint64_t project_id, const char *text) emit_drift(d); for (const auto &d : arch_drifts) emit_drift(d); - drifts_json << "]"; - - std::ostringstream json; - json << "{\"claims_parsed\":" << batch.claims_count - << ",\"results\":" << batch.results_json - << ",\"drifts\":" << drifts_json.str() - << ",\"summary\":{\"supported\":" << batch.supported - << ",\"contradicted\":" << batch.contradicted - << ",\"unknown\":" << batch.unknown - << ",\"drifts_found\":" << total_drifts - << ",\"trust_score\":"; + json.endArray(); + + json.key("summary").beginObject(); + json.key("supported").value(batch.supported); + json.key("contradicted").value(batch.contradicted); + json.key("unknown").value(batch.unknown); + json.key("drifts_found").value(total_drifts); + // How much was judgeable, and how many claims reached a decision — + // without these, `trust_score` alone is unreadable: 1.0 means + // "nothing was checkable" here (see BatchResult::status). + json.key("status").value(batch.status); + json.key("verdicts_decided") + .value(batch.supported + batch.contradicted); + json.key("trust_score"); size_t denom = batch.supported + batch.contradicted + total_drifts; - if (denom > 0) - json << (static_cast(batch.supported) / - static_cast(denom)); - else if (batch.claims_count > 0) - json << "0.0"; // all claims unknown — not trustworthy - else - json << "1.0"; // no claims and no drifts — nothing to dispute - json << "}}"; - return dupString(json.str()); - } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_verify_summary] ") + - e.what() + "\"}"); - } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_verify_summary] unknown exception\"}"); - } -} - -// engine_explain_module — return a Knowledge Card for a named module. -// -// Looks up the module by name (case-insensitive) in the modules table. If no -// row exists, falls back to deriving module info from file paths so the tool -// works on freshly-indexed projects where the modules table has not yet been -// populated. -// -// Output JSON (Knowledge Card): -// {"module":"engine","summary":"...","entities":{"count":N,"sample":[...]}, -// "capabilities":[...],"contracts":[...],"findings":[...], -// "integrity":92} -// On error: {"error":"module not found","module":""} -// -// MEMORY: caller MUST free the returned char* via engine_free_string(). -// THREAD SAFETY: single-threaded (GraphStore writer invariant). -extern "C" char *engine_explain_module(uint64_t project_id, - const char *module_name) -{ - try { - if (!g_store) - return dupString( - "{\"error\":\"not initialized " - "[module=ffi, method=engine_explain_module]\"}"); - if (!module_name || !*module_name) - return dupString( - "{\"error\":\"module_name is empty " - "[module=ffi, method=engine_explain_module]\"}"); - - std::string name(module_name); - std::string name_lower; - name_lower.reserve(name.size()); - for (char c : name) - name_lower += static_cast( - std::tolower(static_cast(c))); - - sqlite3 *db = g_store->handle(); - if (!db) - return dupString( - "{\"error\":\"db not open " - "[module=ffi, method=engine_explain_module]\"}"); - - // Resolve module row (case-insensitive name match). - // No project_id filter: module names are globally unique in - // both serial (single project) and parallel (merged) products, - // and the MCP layer's restored project_id may differ from the - // owning module's project_id in parallel products. - std::string summary; - bool found = false; - { - const char *sql = "SELECT name FROM modules " - "WHERE LOWER(name)=? " - "LIMIT 1"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - sqlite3_bind_text(stmt, 1, name_lower.c_str(), - -1, SQLITE_STATIC); - if (sqlite3_step(stmt) == SQLITE_ROW) { - found = true; - const char *n = - reinterpret_cast( - sqlite3_column_text( - stmt, 0)); - if (n) - summary = "Module: " + - std::string(n); - } - sqlite3_finalize(stmt); - } - } - - // Fallback: derive module existence from file paths so the tool works - // even when the modules table is empty (e.g. freshly indexed project). - // Use "%//%" so paths with a leading "./" or a parent directory - // still match. The slashes prevent partial segment matches (e.g. a - // query for "engine" won't match "./my_engine/foo"). - if (!found) { - // Build the LIKE pattern. The fallback targets files under - // `name`, so an absolute path (leading '/') must not gain a - // second slash: "%/" + "/Users/..." would produce - // "%//Users/..." which never matches a single-slash path. - // Relative module names keep the leading "/" to avoid - // partial-segment matches ("engine" vs "./my_engine"). - std::string like = name[0] == '/' ? "%" + name + "/%" : - "%/" + name + "/%"; - const char *sql = "SELECT COUNT(*) FROM files " - "WHERE path LIKE ?"; - sqlite3_stmt *stmt = nullptr; - int count = 0; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - sqlite3_bind_text(stmt, 1, like.c_str(), -1, - SQLITE_STATIC); - if (sqlite3_step(stmt) == SQLITE_ROW) - count = sqlite3_column_int(stmt, 0); - sqlite3_finalize(stmt); - } - if (count == 0) { - std::ostringstream j; - j << "{\"error\":\"module not found\",\"module\":\"" - << jsonEscape(name) << "\"}"; - return dupString(j.str()); - } - found = true; - summary = "Module derived from " + - std::to_string(count) + " files under " + - name + "/"; - } - - std::ostringstream json; - json << "{\"module\":\"" << jsonEscape(name) << "\"," - << "\"summary\":\"" << jsonEscape(summary) << "\","; - - // Entities: count + sample (up to 10) for this module's files. - // Use the same "%//%" pattern as the fallback check above so - // paths with a leading "./" match consistently. - { - std::string like = "%/" + name + "/%"; - // v0.2.5: read from the canonical `entity` table (the legacy - // graph_nodes table is empty in the canonical schema, so this - // previously always returned zero entities). entity.id - // preserves the legacy graph node identity. - std::string sql_str = - "SELECT name, kind, file_path FROM entity " - "WHERE file_path LIKE ? " - "ORDER BY id LIMIT " + - std::to_string(kEntitySampleLimit); - sqlite3_stmt *stmt = nullptr; - int total = 0; - // Count first - const char *csql = "SELECT COUNT(*) FROM entity " - "WHERE file_path LIKE ?"; - if (sqlite3_prepare_v2(db, csql, -1, &stmt, nullptr) == - SQLITE_OK) { - sqlite3_bind_text(stmt, 1, like.c_str(), -1, - SQLITE_STATIC); - if (sqlite3_step(stmt) == SQLITE_ROW) - total = sqlite3_column_int(stmt, 0); - sqlite3_finalize(stmt); - } - json << "\"entities\":{\"count\":" << total - << ",\"sample\":["; - if (sqlite3_prepare_v2(db, sql_str.c_str(), -1, &stmt, - nullptr) == SQLITE_OK) { - sqlite3_bind_text(stmt, 1, like.c_str(), -1, - SQLITE_STATIC); - bool first = true; - while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - const char *n = - reinterpret_cast( - sqlite3_column_text( - stmt, 0)); - int kind = sqlite3_column_int(stmt, 1); - const char *fp = - reinterpret_cast( - sqlite3_column_text( - stmt, 2)); - json << "{\"name\":\"" - << jsonEscape(n ? n : "") << "\"," - << "\"kind\":" << kind << "," - << "\"file_path\":\"" - << jsonEscape(fp ? fp : "") - << "\"}"; - } - sqlite3_finalize(stmt); - } - json << "]},"; - } - - // Capabilities - { - json << "\"capabilities\":["; - const char *sql = - "SELECT id, name, summary FROM capability " - "ORDER BY id"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - bool first = true; - while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - int64_t id = - sqlite3_column_int64(stmt, 0); - const char *n = - reinterpret_cast( - sqlite3_column_text( - stmt, 1)); - const char *s = - reinterpret_cast( - sqlite3_column_text( - stmt, 2)); - json << "{\"id\":" << id << "," - << "\"name\":\"" - << jsonEscape(n ? n : "") << "\"," - << "\"summary\":\"" - << jsonEscape(s ? s : "") << "\"}"; - } - sqlite3_finalize(stmt); - } - json << "],"; - } - - // Contracts - { - json << "\"contracts\":["; - const char *sql = - "SELECT id, name, origin, claim_text FROM contract " - "ORDER BY id"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - bool first = true; - while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - int64_t id = - sqlite3_column_int64(stmt, 0); - const char *n = - reinterpret_cast( - sqlite3_column_text( - stmt, 1)); - const char *o = - reinterpret_cast( - sqlite3_column_text( - stmt, 2)); - const char *c = - reinterpret_cast( - sqlite3_column_text( - stmt, 3)); - json << "{\"id\":" << id << "," - << "\"name\":\"" - << jsonEscape(n ? n : "") << "\"," - << "\"origin\":\"" - << jsonEscape(o ? o : "") << "\"," - << "\"claim_text\":\"" - << jsonEscape(c ? c : "") << "\"}"; - } - sqlite3_finalize(stmt); - } - json << "],"; - } - - // Findings + integrity score - { - json << "\"findings\":["; - const char *sql = - "SELECT id, rule, severity, description, confidence " - "FROM finding ORDER BY id"; - sqlite3_stmt *stmt = nullptr; - int sev2 = 0, sev1 = 0; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - bool first = true; - while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - int64_t id = - sqlite3_column_int64(stmt, 0); - const char *r = - reinterpret_cast( - sqlite3_column_text( - stmt, 1)); - int sev = sqlite3_column_int(stmt, 2); - const char *d = - reinterpret_cast( - sqlite3_column_text( - stmt, 3)); - double conf = - sqlite3_column_double(stmt, 4); - json << "{\"id\":" << id << "," - << "\"rule\":\"" - << jsonEscape(r ? r : "") << "\"," - << "\"severity\":" << sev << "," - << "\"description\":\"" - << jsonEscape(d ? d : "") << "\"," - << "\"confidence\":" << conf - << "}"; - if (sev == 2) - sev2++; - else if (sev == 1) - sev1++; - } - sqlite3_finalize(stmt); - } - json << "],"; - int integrity = kIntegrityMax - - kIntegritySev2Penalty * sev2 - - kIntegritySev1Penalty * sev1; - if (integrity < 0) - integrity = 0; - if (integrity > kIntegrityMax) - integrity = kIntegrityMax; - json << "\"integrity\":" << integrity << ","; - } - - // Cross-module dependencies from the pre-computed module_edge table. - // Populated by the async knowledge builder after indexing. When the - // table is empty (e.g. async build not yet complete), both arrays are - // empty — the card still returns successfully. - json << "\"cross_module\":{"; - - // depends_on: modules that this module calls (outgoing edges). - { - json << "\"depends_on\":["; - const char *sql = - "SELECT tgt_module, edge_count " - "FROM module_edge " - "WHERE project_id=? AND src_module LIKE ? " - "ORDER BY edge_count DESC LIMIT ?"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - std::string like = "%/" + name + "/%"; - sqlite3_bind_int64( - stmt, 1, - static_cast(project_id)); - sqlite3_bind_text(stmt, 2, like.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_int(stmt, 3, - kCrossModuleEdgeLimit); - bool first = true; - while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - const char *m = - reinterpret_cast( - sqlite3_column_text( - stmt, 0)); - int64_t ec = - sqlite3_column_int64(stmt, 1); - json << "{\"module\":\"" - << jsonEscape(m ? m : "") << "\"" - << ",\"edge_count\":" << ec << "}"; - } - sqlite3_finalize(stmt); - } - json << "],"; - } - - // depended_by: modules that call this module (incoming edges). - { - json << "\"depended_by\":["; - const char *sql = - "SELECT src_module, edge_count " - "FROM module_edge " - "WHERE project_id=? AND tgt_module LIKE ? " - "ORDER BY edge_count DESC LIMIT ?"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - std::string like = "%/" + name + "/%"; - sqlite3_bind_int64( - stmt, 1, - static_cast(project_id)); - sqlite3_bind_text(stmt, 2, like.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_int(stmt, 3, - kCrossModuleEdgeLimit); - bool first = true; - while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - const char *m = - reinterpret_cast( - sqlite3_column_text( - stmt, 0)); - int64_t ec = - sqlite3_column_int64(stmt, 1); - json << "{\"module\":\"" - << jsonEscape(m ? m : "") << "\"" - << ",\"edge_count\":" << ec << "}"; - } - sqlite3_finalize(stmt); - } - json << "]"; + if (denom > 0) { + json.value(static_cast(batch.supported) / + static_cast(denom)); + } else if (batch.claims_count > 0) { + // Literal "0.0": all claims unknown, not trustworthy. + json.raw("0.0"); + } else { + // No claims and no drifts: nothing was checked, so the score + // carries no information. Reported as 1.0 for shape stability — + // `status: "no_claims"` above is what says so, and a reader must + // take the two together. + json.raw("1.0"); } - - json << "}}"; + json.endObject(); + json.endObject(); return dupString(json.str()); } catch (const std::exception &e) { - return dupString( - std::string( - "{\"error\":\"[module=ffi, method=engine_explain_module] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_verify_summary", e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=engine_explain_module] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_verify_summary", "unknown exception")); } } @@ -1160,9 +852,13 @@ extern "C" char *engine_explain_module(uint64_t project_id, // // MEMORY: caller MUST free the returned char* via engine_free_string(). // THREAD SAFETY: single-threaded (GraphStore writer invariant). -extern "C" char *engine_get_verifier_registry_status(uint64_t project_id) +extern "C" char *engine_get_verifier_registry_status(engine_t handle, + uint64_t project_id) { + EngineContext *ctx = engineInstance(handle); + try { + auto _store_guard = waitForKnowledgeBuilder(); // Idempotent: arms the registry if empty without relying on a // static flag (Step 9.1 fix for lifecycle bug A15). ensureVerifiersRegistered(); @@ -1189,47 +885,40 @@ extern "C" char *engine_get_verifier_registry_status(uint64_t project_id) int64_t entity_count = 0; int64_t relation_count = 0; bool backend_ready = false; - if (g_store && project_id != 0) { + if (ctx->store && project_id != 0) { backend_ready = verify::evidence_backend_ready( - g_store.get(), project_id, &entity_count, + ctx->store.get(), project_id, &entity_count, &relation_count); } - std::ostringstream j; - j << "{\"registry_empty\":" << (count == 0 ? "true" : "false") - << ",\"verifier_count\":" << count << ",\"verifier_names\":["; - for (size_t i = 0; i < names.size(); ++i) { - if (i > 0) - j << ","; - j << "\"" << jsonEscape(names[i]) << "\""; - } - j << "],\"supported_claim_types\":["; - for (size_t i = 0; i < supported.size(); ++i) { - if (i > 0) - j << ","; - j << "\"" << verify::claimTypeWireName(supported[i]) - << "\""; - } - j << "],\"unsupported_claim_types\":["; - for (size_t i = 0; i < unsupported.size(); ++i) { - if (i > 0) - j << ","; - j << "\"" << verify::claimTypeWireName(unsupported[i]) - << "\""; - } - j << "],\"evidence_backend_ready\":" - << (backend_ready ? "true" : "false") - << ",\"entity_count\":" << entity_count - << ",\"relation_count\":" << relation_count << "}"; + util::JsonWriter j; + j.beginObject(); + j.key("registry_empty").value(count == 0); + j.key("verifier_count").value(count); + j.key("verifier_names").beginArray(); + for (const auto &name : names) + j.value(name); + j.endArray(); + j.key("supported_claim_types").beginArray(); + for (auto t : supported) + j.value(verify::claimTypeWireName(t)); + j.endArray(); + j.key("unsupported_claim_types").beginArray(); + for (auto t : unsupported) + j.value(verify::claimTypeWireName(t)); + j.endArray(); + j.key("evidence_backend_ready").value(backend_ready); + j.key("entity_count").value(entity_count); + j.key("relation_count").value(relation_count); + j.endObject(); return dupString(j.str()); } catch (const std::exception &e) { - return dupString( - std::string("{\"error\":\"[module=ffi, method=" - "engine_get_verifier_registry_status] ") + - e.what() + "\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_verifier_registry_status", + e.what())); } catch (...) { - return dupString( - "{\"error\":\"[module=ffi, method=" - "engine_get_verifier_registry_status] unknown exception\"}"); + return dupString(util::errorEnvelope( + "ffi", "engine_get_verifier_registry_status", + "unknown exception")); } } diff --git a/engine/src/engine_verify_planner_ffi.cpp b/engine/src/engine_verify_planner_ffi.cpp deleted file mode 100644 index a9bc645..0000000 --- a/engine/src/engine_verify_planner_ffi.cpp +++ /dev/null @@ -1,105 +0,0 @@ -// engine_verify_planner_ffi.cpp — Verification Planner FFI (VP4). -// -// Exposes the Phase 3 Verification Planner to the Rust MCP server via -// a single extern "C" entry point: -// -// char *engine_verify_statement(uint64_t project_id, -// const char *claim_text); -// -// The function parses a natural-language claim into an Intent, plans -// the evidence rules to execute, runs those rules via the -// EvidenceBuilder, and combines the results into a Verdict. The -// returned JSON has the shape: -// -// { -// "verdict": "Supported|Contradicted|PartiallyVerified|Unknown", -// "confidence": 0.0..1.0, -// "requirements": [ -// {"id":"...","weight":N,"satisfied":bool,"confidence":N}, ... -// ], -// "evidence": [ -// {"category":"...","title":"...","confidence":N, -// "item_count":N}, ... -// ] -// } -// -// All errors return a JSON object with an "error" field instead of -// crashing. Null `g_store` returns {"error":"engine not initialized"}. -// The caller MUST release the returned pointer via engine_free_string(). - -#include "engine_internal.h" -#include "platform_win.h" -#include "verify/intent_parser.h" -#include "verify/planner.h" -#include "verify/verdict_builder.h" -#include "verify/claim.h" -#include "verify/ffi_internal.h" -#include "evidence/evidence_builder.h" - -#include -#include -#include -#include -#include - -// ─── FFI entry point ──────────────────────────────────────────── - -// Verify a natural-language claim against the project's indexed -// evidence. Parses the claim into an Intent, plans and executes the -// evidence rules, and returns the verdict as JSON. -// -// @param project_id The project whose semantic_fact rows to query. -// @param claim_text The natural-language claim (e.g. "does this -// project safely handle CString?"). -// @return Heap-allocated JSON string (caller frees via -// engine_free_string). On error, returns a JSON object with -// an "error" or "verdict":"Unknown" field. -char *engine_verify_statement(uint64_t project_id, const char *claim_text) -{ - if (!g_store) - return dupString("{\"error\":\"engine not initialized\"}"); - if (!claim_text || !*claim_text) - return dupString("{\"verdict\":\"Unknown\",\"confidence\":0" - ",\"error\":\"empty claim\"}"); - - // Thin wrapper over the structured verify_claim path (VP4 → Step 9): - // parse the natural-language intent, map it to a structured Claim, - // and dispatch through verify_one_claim — the SAME core used by - // verify_claim. This replaces the old IntentParser → Planner → - // EvidenceBuilder → VerdictBuilder chain, which silently returned - // Unknown for any intent that did not map to a known evidence rule - // (no way to distinguish "unrecognized question" from "evidence - // insufficient"). Unrecognized intents now return a machine-readable - // error_code so MCP clients can tell the two apart. - verify::planner::IntentParser parser; - verify::planner::Intent intent = parser.parse(claim_text); - - verify::Claim claim; - if (intent.type == "capability_question") { - claim.type = verify::ClaimType::CapabilityExists; - } else if (intent.type == "safety_question" || - intent.type == "pattern_question") { - claim.type = verify::ClaimType::ContractHolds; - } else { - return dupString("{\"verdict\":\"Unknown\",\"confidence\":0," - "\"error_code\":\"intent_unrecognized\"," - "\"error\":\"claim intent not recognized; use " - "verify_claim with type capability_exists|" - "contract_holds|architecture_follows|" - "function_implements [module=ffi, " - "method=engine_verify_statement]\"}"); - } - claim.subject = intent.subject.empty() ? claim_text : intent.subject; - claim.predicate = "implemented_by"; - claim.scope = "repository"; - claim.source_kind = "manual"; - - verify_ffi::VerifyResult result = - verify_ffi::verify_one_claim(project_id, claim); - char *json = result.json; - // result.json is heap-allocated and owned by us (MUST NOT free twice: - // dupString copies, so the caller's free on our return value is the - // only free). verify_one_claim's caller contract: caller frees the - // returned pointer. We return it directly. - return json; -} diff --git a/engine/src/evidence/evidence_builder.cpp b/engine/src/evidence/evidence_builder.cpp index 11f3109..0d0fafb 100644 --- a/engine/src/evidence/evidence_builder.cpp +++ b/engine/src/evidence/evidence_builder.cpp @@ -237,6 +237,9 @@ struct MatchedFact { std::string symbol; double confidence = 1.0; std::string detail_json; + /// Name of the entity the fact is attached to (semantic_fact.function_id). + /// Empty when the join finds nothing. + std::string enclosing_symbol; }; // SELECT semantic_fact rows matching a single FactNeed for one @@ -253,10 +256,18 @@ std::vector queryFactsForNeed(store::GraphStore *store, sqlite3 *db = store->handle(); if (!db) return out; - const char *sql = "SELECT id, function_id, category, primitive, kind, " - "symbol, confidence, IFNULL(detail_json, '') " - "FROM semantic_fact WHERE project_id = ? " - "AND category = ? AND primitive = ? AND kind = ?"; + // Every column is qualified: `entity` has an `id` (and a `name`) of its + // own, so an unqualified `id` in the select list makes the whole statement + // fail to prepare — and this function then answers with ZERO facts, i.e. + // every rule silently stops producing evidence. + const char *sql = + "SELECT f.id, f.function_id, f.category, f.primitive, f.kind, " + "f.symbol, f.confidence, IFNULL(f.detail_json, ''), " + "IFNULL(e.name, '') " + "FROM semantic_fact f " + "LEFT JOIN entity e ON e.id = f.function_id " + "WHERE f.project_id = ? " + "AND f.category = ? AND f.primitive = ? AND f.kind = ?"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) != SQLITE_OK) { fprintf(stderr, @@ -283,6 +294,7 @@ std::vector queryFactsForNeed(store::GraphStore *store, mf.symbol = colText(stmt, 5); mf.confidence = sqlite3_column_double(stmt, 6); mf.detail_json = colText(stmt, 7); + mf.enclosing_symbol = colText(stmt, 8); out.push_back(std::move(mf)); } return out; @@ -297,7 +309,17 @@ EvidenceItem toEvidenceItem(const MatchedFact &mf) item.category = mf.category; item.primitive = mf.primitive; item.kind = mf.kind; - item.symbol = mf.symbol; + // A fact's own `symbol` is not always a symbol name: a TODO/FIXME marker is + // a *comment* record, whose "name" is the comment text, and the rule + // templates print {symbol} as the subject of the message — which produced + // "5 TODO marker(s) may indicate plan-code drift" listing items whose + // symbol was "// Comment records (kind=14) whose name contains TODO/FIXME" + // (the extractor's own source comment). The enclosing function the fact is + // attached to is the actionable subject, and the comment text still travels + // in the item's `snippet` (parsed from detail_json). Facts about a real + // symbol keep that symbol. + item.symbol = mf.enclosing_symbol.empty() ? mf.symbol : + mf.enclosing_symbol; applyDetailJson(mf.detail_json, item); return item; } @@ -473,6 +495,13 @@ std::vector combineCount(store::GraphStore *store, if (rule.needs.empty()) return result; auto primary = queryFactsForNeed(store, project_id, rule.needs[0]); + // No matches → no finding. Every other combine mode returns no Evidence in + // that case and the dispatcher's contract is "0 = no matches", but this one + // emitted a row with count=0 — making a rule that found nothing look like an + // inspector that produced a result, which inflated + // overall.inspectors_ran (and the confidence derived from it). + if (primary.empty()) + return result; Evidence ev; ev.category = rule.category; ev.confidence = 1.0; diff --git a/engine/src/evidence/rule.cpp b/engine/src/evidence/rule.cpp index 8ed2ad9..8bcc6f7 100644 --- a/engine/src/evidence/rule.cpp +++ b/engine/src/evidence/rule.cpp @@ -663,4 +663,51 @@ RuleLoader::loadFromDirectory(const std::string &dir_path) const return result; } +// ─── resolveRulesDir ───────────────────────────────────────────── +// +// Single authority for locating the rule-file directory. The three +// candidates are tried in order; the first one that is a directory +// containing at least one *.json file wins. See rule.h for the +// rationale (CWD-relative lookup silently produced zero rules). + +std::string resolveRulesDir() +{ + const char *env_dir = std::getenv("CODESCOPE_RULES_DIR"); + const char *build_default = +#ifdef CODESCOPE_RULES_DIR_DEFAULT + CODESCOPE_RULES_DIR_DEFAULT; +#else + nullptr; +#endif + const char *candidates[] = { + env_dir, + build_default, + "engine/src/evidence/rules", + }; + for (const char *cand : candidates) { + if (!cand || !*cand) + continue; + std::error_code ec; + if (!std::filesystem::is_directory(cand, ec)) + continue; + bool has_json = false; + for (const auto &entry : + std::filesystem::directory_iterator(cand, ec)) { + if (ec) + break; + if (entry.is_regular_file() && + entry.path().extension() == ".json") { + has_json = true; + break; + } + } + if (has_json) + return std::string(cand); + } + fprintf(stderr, "[module=evidence, method=resolveRulesDir] no rule " + "directory found (checked $CODESCOPE_RULES_DIR, build " + "default, and ./engine/src/evidence/rules)\n"); + return std::string(); +} + } // namespace evidence diff --git a/engine/src/evidence/rule.h b/engine/src/evidence/rule.h index f6dfa71..d1d0075 100644 --- a/engine/src/evidence/rule.h +++ b/engine/src/evidence/rule.h @@ -91,6 +91,23 @@ CombineMode combineModeFromString(const std::string &s); /// string for a CombineMode. Used by RuleLoader diagnostics and tests. const char *combineModeToString(CombineMode mode); +/// Resolve the rule-file directory independently of the process CWD. +/// Callers must not hard-code "engine/src/evidence/rules": that path +/// only exists when running from the repository root, so the MCP/CLI +/// vehicles (which run inside the target project) silently load zero +/// rules and `build_evidence` returns []. +/// +/// Resolution order: +/// 1. $CODESCOPE_RULES_DIR — explicit operator override. +/// 2. CODESCOPE_RULES_DIR_DEFAULT — absolute path baked in at build +/// time by CMake (CMAKE_CURRENT_SOURCE_DIR/src/evidence/rules). +/// 3. "engine/src/evidence/rules" relative to CWD — back-compat for +/// in-tree runs and tests that set the working directory. +/// Each candidate must be a directory containing at least one *.json +/// file; otherwise it is skipped. Returns "" when no candidate works +/// (the caller logs and treats it as "no rules", not as a crash). +std::string resolveRulesDir(); + /// RuleLoader reads all *.json rule files from a directory and /// returns a vector of RuleSet. The JSON schema is intentionally /// flat (see plan section 4.3) so a small string-based parser is diff --git a/engine/src/filter_policy.cpp b/engine/src/filter_policy.cpp index 782b98d..67ceb8a 100644 --- a/engine/src/filter_policy.cpp +++ b/engine/src/filter_policy.cpp @@ -2,9 +2,9 @@ #include #include #include -#include -#include +#include #include +#include FilterPolicy::FilterPolicy() { @@ -218,21 +218,20 @@ FilterPolicy::FilterPolicy() "example", "samples", "sample", - "scripts", - "hack", - "migrations", - "seeds", "e2e", - "integration", - "locale", - "locales", - "i18n", - "l10n", - "assets", - "static", - "public", - "media", - "external", + // NOTE: scripts / hack / migrations / seeds / integration / + // locale(s) / i18n / l10n / assets / static / public / media / + // external were MOVED OUT of this any-depth set into + // top_only_skip_dirs_ below. They are ordinary business + // directories outside the web-frontend convention + // (src/integration/, pkg/scripts/, app/static/, db/migrations/ + // all hold real, first-party code), and the generated/asset + // noise they were meant to exclude is already rejected by the + // source-extension filter. Matching them at any depth silently + // dropped source files — and therefore symbols and call edges + // (a false negative the index cannot distinguish from code that + // does not exist). They are still skipped at the project root + // and the next two levels (depth <= 3). }; // FAST mode skips even more — build/test artifacts that NORMAL @@ -622,14 +621,27 @@ FilterPolicy::FilterPolicy() lowercaseAll(skip_filename_prefixes_); lowercaseAll(skip_dir_prefixes_); - // Source-bearing dirs that were "rarely the focus of analysis" - // (test/, docs/, vendor/, bench/, samples/, examples/, ...) are - // now in normal_skip_dirs_ (matched at ANY depth). This set is - // kept empty intentionally — the prior top-only (depth ≤ 3) - // matching leaked deep-nested test dirs on monorepos. See the - // comment in normal_skip_dirs_ for the full rationale and the - // Java user override path (CODESCOPE_EXCLUDE_PATHS + .codescopeignore). - top_only_skip_dirs_ = {}; + // Top-only (depth <= kTopOnlyDepth) skip dirs — business-plausible + // names that were previously matched at ANY depth. They are skipped + // only within the first kTopOnlyDepth path components, which is where + // the conventions they target actually live: + // /public/ ... /assets/ ... + // /scripts/ ... /migrations/ ... + // /i18n/ ... /external/ ... + // Beyond that they are overwhelmingly first-party code, and a false + // negative from a silent skip is indistinguishable from code that does + // not exist — the worst failure mode for a code graph. The generated + // junk these names were meant to exclude (JSON/po/css/binary) is + // already rejected by the source-extension filter, so the relaxation + // costs little and restores symbols the index was dropping. + // Applied to EVERY language (see the step-1b check in shouldSkipPath); + // the Java carve-out below additionally protects Java package + // namespaces for the names that remain any-depth. + top_only_skip_dirs_ = { + "scripts", "hack", "migrations", "seeds", "integration", + "locale", "locales", "i18n", "l10n", "assets", + "static", "public", "media", "external", + }; lowercaseAll(top_only_skip_dirs_); // Java-protected skip dirs — the SAME source-bearing names that @@ -672,6 +684,45 @@ void FilterPolicy::buildActiveSets() } } +namespace +{ +// Fold language-label aliases onto one canonical member so a filter of +// "c" accepts files the path-based classifier labels "cpp" (and the +// reverse). The two classifiers disagree for `.c` (see +// query_analysis.cpp languagesCompatible), so exact-string matching +// silently indexes nothing when they do. Also trims and lowercases. +std::string canonicalLang(std::string s) +{ + // std::isspace also strips \r\n — a CRLF-tainted token ("c\r") must + // fold like "c" or the filter silently matches nothing. + while (!s.empty() && + std::isspace(static_cast(s.front()))) + s.erase(s.begin()); + while (!s.empty() && std::isspace(static_cast(s.back()))) + s.pop_back(); + for (auto &c : s) + c = static_cast( + std::tolower(static_cast(c))); + if (s == "c" || s == "c++" || s == "cpp") + return "cpp"; + if (s == "js" || s == "node" || s == "javascript") + return "javascript"; + if (s == "ts" || s == "typescript") + return "typescript"; + if (s == "py" || s == "python") + return "python"; + if (s == "golang" || s == "go") + return "go"; + if (s == "kt" || s == "kts" || s == "kotlin") + return "kotlin"; + if (s == "rb" || s == "ruby") + return "ruby"; + if (s == "rs" || s == "rust") + return "rust"; + return s; +} +} // namespace + void FilterPolicy::setLanguageFilter(const std::string &filter) { if (filter.empty()) { @@ -683,11 +734,9 @@ void FilterPolicy::setLanguageFilter(const std::string &filter) size_t start = 0, end; do { end = filter.find(',', start); - auto lang = filter.substr(start, end - start); - // Normalize to lowercase - for (auto &c : lang) - c = static_cast(std::tolower(c)); - lang_filter_set_.insert(lang); + auto lang = canonicalLang(filter.substr(start, end - start)); + if (!lang.empty()) + lang_filter_set_.insert(lang); start = end + 1; } while (end != std::string::npos); } @@ -696,7 +745,8 @@ bool FilterPolicy::isLanguageAccepted(const std::string &lang) const { if (!has_lang_filter_) return true; - return lang_filter_set_.find(lang) != lang_filter_set_.end(); + return lang_filter_set_.find(canonicalLang(lang)) != + lang_filter_set_.end(); } bool FilterPolicy::shouldSkipDir(const std::string &dir_name) const @@ -747,6 +797,30 @@ bool FilterPolicy::isJavaProtectedDir(const std::string &dir_name) const java_protected_skip_dirs_.end(); } +void applyProjectLanguageContext(FilterPolicy &filter, const std::string &dir) +{ + if (dir.empty() || filter.langContext() == "java") + return; + // Bounded by the first .java found, which is the common case for the Java + // projects this exists for; a project without one pays a full walk, once + // per root (engine_path_is_skipped caches its policy per project root, and + // each worker process sees only its own module). + std::error_code ec; + std::filesystem::recursive_directory_iterator it( + dir, std::filesystem::directory_options::skip_permission_denied, + ec); + const std::filesystem::recursive_directory_iterator end; + while (!ec && it != end) { + const std::filesystem::directory_entry &entry = *it; + if (entry.is_regular_file(ec) && + entry.path().extension() == ".java") { + filter.setLangContext("java"); + return; + } + it.increment(ec); + } +} + bool FilterPolicy::shouldSkipFile(const std::string &filename) const { // Case-insensitive: lowercase before lookup so .ENV.LOCAL matches. @@ -832,329 +906,3 @@ bool FilterPolicy::isSourceFile(const std::string &path) const return true; } - -const char *FilterPolicy::detectLanguage(const char *file_path) const -{ - // Guard against null / empty path — strrchr below would dereference it - if (!file_path || !*file_path) - return nullptr; - - const char *ext = strrchr(file_path, '.'); - if (!ext) { - // No extension: probe the shebang line to identify scripts - // (e.g. "#!/usr/bin/env python3", "#!/usr/bin/env node"). - std::ifstream f(file_path); - if (!f.is_open()) - return nullptr; - std::string first_line; - std::getline(f, first_line); - if (first_line.size() >= 2 && first_line[0] == '#' && - first_line[1] == '!') { - // Match the interpreter token after the last '/' on the - // shebang line so "/usr/bin/env python3" → "python3". - auto sp = first_line.rfind('/'); - std::string interp = (sp != std::string::npos) ? - first_line.substr(sp + 1) : - first_line.substr(2); - // Strip trailing whitespace/args - auto ws = interp.find_first_of(" \t\r\n"); - if (ws != std::string::npos) - interp.erase(ws); - // Lowercase for case-insensitive comparison - for (auto &c : interp) - c = static_cast(std::tolower(c)); - if (interp.rfind("python", 0) == 0) - return "python"; - if (interp == "node" || interp == "nodejs" || - interp == "deno") - return "javascript"; - if (interp == "bash" || interp == "sh" || - interp == "zsh" || interp == "ksh" || - interp == "fish") - return "bash"; - if (interp == "ruby" || interp == "rb") - return "ruby"; - if (interp == "perl" || interp == "perl5") - return "perl"; - if (interp == "php") - return "php"; - if (interp == "lua") - return "lua"; - if (interp == "rscript" || interp == "r") - return "r"; - if (interp == "awk" || interp == "gawk" || - interp == "mawk") - return "awk"; - } - return nullptr; - } - - // Lowercase the extension into a small buffer so that .PY / .Rs / - // .TSX on case-insensitive filesystems (Windows, default macOS) - // are recognized identically to their canonical lowercase forms. - std::string lext; - lext.reserve(16); - for (const char *p = ext; *p; ++p) - lext.push_back(static_cast(std::tolower(*p))); - - // Skip minified/bundled JS — generated code, expensive & low-value. - // Use both '/' and '\\' so basename extraction works on Windows too. - const char *slash_f = strrchr(file_path, '/'); - const char *slash_b = strrchr(file_path, '\\'); - const char *slash = (slash_b > slash_f) ? slash_b : slash_f; - const char *fname = slash ? slash + 1 : file_path; - size_t fname_len = strlen(fname); - // Lowercased basename tail checks for robustness on all platforms. - if (fname_len > 7) { - char tail[8] = { 0 }; - for (size_t i = 0; i < 7; i++) - tail[i] = static_cast( - std::tolower(fname[fname_len - 7 + i])); - if (strcmp(tail, ".min.js") == 0) - return nullptr; - } - if (fname_len > 10 && strstr(fname, ".bundle.js") != nullptr) - return nullptr; - if (strcmp(fname, "vendor.js") == 0) - return nullptr; - - if (lext == ".py") - return "python"; - if (lext == ".cpp" || lext == ".cc" || lext == ".cxx") - return "cpp"; - if (lext == ".c") - return "c"; - // C++ headers (.h/.hpp/.hxx/.hh) are parsed as C++ because the - // tree-sitter C++ grammar is a strict superset of the C grammar. - // Parsing .h as C would mis-handle templates, namespaces, classes, - // and other C++ constructs commonly found in .h files. - if (lext == ".h" || lext == ".hpp" || lext == ".hxx" || lext == ".hh") - return "cpp"; - if (lext == ".rs") - return "rust"; - if (lext == ".swift") - return "swift"; - if (lext == ".js" || lext == ".mjs" || lext == ".cjs") - return "javascript"; - if (lext == ".ts") - return "typescript"; - if (lext == ".tsx") - return "tsx"; - if (lext == ".go") - return "go"; - if (lext == ".java") - return "java"; - if (lext == ".kt" || lext == ".kts") - return "kotlin"; - if (lext == ".rb") - return "ruby"; - if (lext == ".scala") - return "scala"; - - return nullptr; -} - -bool FilterPolicy::shouldSkipPath(const std::string &rel_path, - bool is_dir) const -{ - if (rel_path.empty()) - return false; - - // 1. Check every path component against active_skip_dirs_ - // (this catches .venv, node_modules, etc. at any depth) - std::istringstream ss(rel_path); - std::string component; - while (std::getline(ss, component, '/')) { - if (shouldSkipDir(component)) - return true; - } - - // 1b. Shallow skip dirs — source-bearing but "rarely the focus - // of analysis" dirs (test/, tests/, docs/, vendor/, bench/, - // samples/, ...). For non-Java projects these are already - // caught at ANY depth by the active_skip_dirs_ check above - // (step 1), so this block is a no-op for them. For Java - // projects, the shouldSkipDir() Java carve-out defers these - // names HERE so nested package components (e.g. - // src/main/java/org/springframework/samples/petclinic) are - // NOT clobbered — only the first 3 path components are checked - // against java_protected_skip_dirs_, catching: - // /test/... (depth 1) - // /src/test/java/... (depth 2, Maven/Gradle) - // /packages//tests/... (depth 3, Lerna monorepo) - // without touching deep package namespaces. See README.md - // "Why Java is the (only) exception". - { - // No-op for non-Java: their test/docs/etc. are already - // skipped at any depth via active_skip_dirs_ in step 1. - if (lang_context_ != "java") - goto skip_top_only_check; - - // Extract up to 3 leading path components. - std::string head; - size_t off = 0; - for (int depth = 0; depth < 3; ++depth) { - auto slash = rel_path.find('/', off); - if (slash == std::string::npos) { - head = rel_path; - break; - } - off = slash + 1; - } - if (head.empty()) { - // rel_path has at least 3 components — take first 3. - head = rel_path.substr(0, off); - } - // Lowercase for case-insensitive comparison. - for (auto &c : head) - c = static_cast(std::tolower(c)); - // Check each of the first 3 components against - // java_protected_skip_dirs_. If any matches, skip. - std::istringstream ss2(head); - std::string comp; - while (std::getline(ss2, comp, '/')) { - if (comp.empty()) - continue; - if (java_protected_skip_dirs_.find(comp) != - java_protected_skip_dirs_.end()) - return true; - } - } -skip_top_only_check:; - - // 2. Check against .gitignore rules (whole-path matching) - if (!gitignore_rules_.empty() && - gitignoreMatches(gitignore_rules_, rel_path, is_dir)) - return true; - - // 3. Check against .codescopeignore raw patterns - // Supports: directory/ (trailing slash = dir only) - // filename (matches any component) - // /path/name (anchored from root) - for (const auto &pat : ignore_patterns_) { - bool dir_only = (!pat.empty() && pat.back() == '/'); - bool anchored = (!pat.empty() && pat[0] == '/'); - // For dir_only patterns applied to file entries: - // still check if any PATH COMPONENT matches the dir name - if (dir_only && pat.size() <= 1) - continue; - - // Normalize pattern: strip trailing / - std::string normalized = - dir_only ? pat.substr(0, pat.size() - 1) : pat; - - // Anchored: match from start - if (anchored) { - if (rel_path == normalized.substr(1) || - (rel_path.size() > normalized.size() - 1 && - rel_path.compare(0, normalized.size() - 1, - normalized.substr(1)) == 0 && - rel_path[normalized.size() - 1] == '/')) - return true; - continue; - } - - // Unanchored: check every path component - std::istringstream ss2(rel_path); - std::string comp; - while (std::getline(ss2, comp, '/')) { - if (comp == normalized) - return true; - } - } - - return false; -} - -bool FilterPolicy::shouldSkipEntry(const std::string &rel_path, - bool is_dir) const -{ - if (rel_path.empty()) - return false; - - // Normalize Windows backslashes to '/' so the rest of the pipeline - // (shouldSkipPath splits on '/', detectLanguage uses strrchr('/')) - // works uniformly across platforms. - std::string normalized = rel_path; - for (auto &c : normalized) - if (c == '\\') - c = '/'; - - // Extract the basename once — used by every per-entry check below. - // Use string_view into normalized to avoid copying (downstream calls - // that need std::string construct it from this view). - auto slash = normalized.rfind('/'); - size_t base_off = (slash == std::string::npos) ? 0 : slash + 1; - std::string_view base(normalized.data() + base_off, - normalized.size() - base_off); - - // ── Shared checks (both dirs & files) ── - // 1. Path-component skip_dirs (any depth) + gitignore + .codescopeignore. - // This alone catches node_modules/, .venv/, .git/ at any nesting. - if (shouldSkipPath(normalized, is_dir)) - return true; - - if (is_dir) { - // 2a. Bundle directory suffixes (.app, .framework, .xcodeproj ...) - // so we never recurse into binary payloads / IDE project bundles. - if (shouldSkipDirSuffix(std::string(base))) - return true; - // Dir-only checks done; fall through to the shared exclude-pattern - // check below so CODESCOPE_EXCLUDE_PATHS applies to directories too. - } else { - // ── File-only checks ── - // 2b. Exact filename + filename-prefix skip (.env, .env.local, ...) - if (shouldSkipFile(std::string(base))) - return true; - // 2c. Suffix skip — case-insensitive (.EXE == .exe). Catches - // binaries, archives, media, secrets, lock files, generated - // artifacts. - auto dot = base.rfind('.'); - if (dot != std::string_view::npos) { - if (shouldSkipSuffix(std::string(base.substr(dot)))) - return true; - } - - // 2d. STRICT mode: whitelist gate — only files that - // detectLanguage() recognizes as source code pass through. - // Catches config/docs/data files that slipped past the - // blacklist (.toml, .yaml, .json, .md). - if (mode_ == STRICT) { - if (detectLanguage(normalized.c_str()) == nullptr) - return true; - } - } - - // 3. User-specified exclude patterns (CODESCOPE_EXCLUDE_PATHS env var). - // Glob-matched against the full relative path so patterns like - // "test/*" or "vendor/**" skip both the directory and its contents. - // Applied LAST so it acts as a user override on top of all built-in - // filters. See loadExcludeEnv() for the env var format. - for (const auto &pat : exclude_patterns_) { - if (globMatch(pat, normalized)) - return true; - } - - return false; -} - -bool FilterPolicy::loadIgnoreFile(const std::string &project_root) -{ - std::string path = project_root + "/.codescopeignore"; - std::ifstream f(path); - if (!f.is_open()) - return false; - - std::string line; - ignore_patterns_.clear(); - while (std::getline(f, line)) { - // Trim - while (!line.empty() && - (line.back() == '\r' || line.back() == '\n')) - line.pop_back(); - if (line.empty() || line[0] == '#') - continue; - ignore_patterns_.push_back(line); - } - return !ignore_patterns_.empty(); -} diff --git a/engine/src/filter_policy.h b/engine/src/filter_policy.h index 8e44522..0f3044c 100644 --- a/engine/src/filter_policy.h +++ b/engine/src/filter_policy.h @@ -130,9 +130,36 @@ class FilterPolicy { // ── Ignore Files ───────────────────────────────────────────── // Load .codescopeignore patterns from project root. - bool loadIgnoreFile(const std::string &project_root); - // Load .gitignore patterns from project root. - bool loadGitignore(const std::string &project_root); + // `append` keeps the rules already loaded instead of replacing them, + // which is what a per-module worker needs: it scans + // `/`, whose own directory usually has no ignore + // file, while the rules live at the project root. Rules loaded first + // take precedence (the matcher stops at the first positive match), so + // the caller loads the more specific directory first and the project + // root second. + bool loadIgnoreFile(const std::string &project_root, + bool append = false); + // Load .gitignore patterns from project root. Same `append` contract. + bool loadGitignore(const std::string &project_root, + bool append = false); + + /// Tell the policy that paths handed to it are relative to a SUBDIRECTORY + /// of the project whose rules were loaded. + /// + /// `.gitignore` and CODESCOPE_EXCLUDE_PATHS are anchored at the project + /// root, but a per-module worker scans `/` and sees + /// module-relative paths, so an anchored rule could never match — a + /// `**/build-*/` rule pruned nothing in the parallel path and a build + /// tree's own sources were indexed. With the prefix set, the anchored + /// rules are matched against `` (the path as the project + /// root sees it) while the component-level checks stay on the relative + /// path, so naming a module `docs` does not silently drop it. + /// + /// Empty (the default) means "paths are already project-root relative". + void setScanPrefix(const std::string &prefix) + { + scan_prefix_ = prefix; + } // Load CODESCOPE_EXCLUDE_PATHS env var — comma-separated glob // patterns (e.g. "test/*,docs/*,vendor/*,third_party/*") that // extend the built-in skip list at index time. Useful for trimming @@ -234,6 +261,11 @@ class FilterPolicy { // .codescopeignore patterns (raw lines) std::vector ignore_patterns_; + + /// Prefix that turns this policy's scan-root-relative paths into + /// project-root-relative ones, for the rules that are anchored at the + /// project root. Empty for a whole-project scan. See setScanPrefix(). + std::string scan_prefix_; // .gitignore patterns (parsed rules) std::vector gitignore_rules_; // CODESCOPE_EXCLUDE_PATHS patterns (comma-separated globs from env). @@ -255,4 +287,23 @@ class FilterPolicy { std::string::const_iterator si, int depth = 0); }; +/// Flip `filter` into Java mode when the tree under `dir` contains any .java +/// file. No-op when `dir` is empty or the policy already knows the language. +/// +/// The test/docs/samples skip names collide with Java package components, so +/// for Java projects they are relaxed to a top-only (depth ≤ 3) check — but the +/// policy has to KNOW the project is Java, and the file that would tell it +/// (`.../samples/petclinic/Owner.java`) lives under a directory that is skipped +/// while the language is still unset: the chicken-and-egg that made Java +/// projects index 0 files. +/// +/// The indexer's own discovery used to open-code this scan. The server's module +/// discovery queries the same policy through `engine_path_is_skipped`, whose +/// policy was built with NO language context, so it applied the any-depth rule: +/// spring-petclinic reported `"modules":[]` — and `index-parallel` indexed +/// nothing while claiming "no source modules found" — whereas `codescope +/// discover` counted all 49 files. Both entry points now call this one +/// implementation. +void applyProjectLanguageContext(FilterPolicy &filter, const std::string &dir); + #endif // FILTER_POLICY_H diff --git a/engine/src/filter_policy_detect.cpp b/engine/src/filter_policy_detect.cpp new file mode 100644 index 0000000..762543f --- /dev/null +++ b/engine/src/filter_policy_detect.cpp @@ -0,0 +1,362 @@ +// filter_policy_detect.cpp — language detection and path/ignore matching. +// +// Split out of filter_policy.cpp (see plan/rules/code_rules.md 1000-line +// rule). Everything here answers "is this path or filename something we +// refuse to index": language detection (extension, then shebang probe), +// component-wise path matching, .gitignore-style rule matching, and the +// gitignore loader. The static skip/accept sets built by the constructor +// stay in filter_policy.cpp; this file only reads them. + +#include "filter_policy.h" + +#include +#include +#include +#include +#include +#include +#include + +const char *FilterPolicy::detectLanguage(const char *file_path) const +{ + // Guard against null / empty path — strrchr below would dereference it + if (!file_path || !*file_path) + return nullptr; + + const char *ext = strrchr(file_path, '.'); + if (!ext) { + // No extension: probe the shebang line to identify scripts + // (e.g. "#!/usr/bin/env python3", "#!/usr/bin/env node"). + std::ifstream f(file_path); + if (!f.is_open()) + return nullptr; + std::string first_line; + std::getline(f, first_line); + if (first_line.size() >= 2 && first_line[0] == '#' && + first_line[1] == '!') { + // Match the interpreter token after the last '/' on the + // shebang line so "/usr/bin/env python3" → "python3". + auto sp = first_line.rfind('/'); + std::string interp = (sp != std::string::npos) ? + first_line.substr(sp + 1) : + first_line.substr(2); + // Strip trailing whitespace/args + auto ws = interp.find_first_of(" \t\r\n"); + if (ws != std::string::npos) + interp.erase(ws); + // Lowercase for case-insensitive comparison + for (auto &c : interp) + c = static_cast(std::tolower(c)); + if (interp.rfind("python", 0) == 0) + return "python"; + if (interp == "node" || interp == "nodejs" || + interp == "deno") + return "javascript"; + if (interp == "bash" || interp == "sh" || + interp == "zsh" || interp == "ksh" || + interp == "fish") + return "bash"; + if (interp == "ruby" || interp == "rb") + return "ruby"; + if (interp == "perl" || interp == "perl5") + return "perl"; + if (interp == "php") + return "php"; + if (interp == "lua") + return "lua"; + if (interp == "rscript" || interp == "r") + return "r"; + if (interp == "awk" || interp == "gawk" || + interp == "mawk") + return "awk"; + } + return nullptr; + } + + // Lowercase the extension into a small buffer so that .PY / .Rs / + // .TSX on case-insensitive filesystems (Windows, default macOS) + // are recognized identically to their canonical lowercase forms. + std::string lext; + lext.reserve(16); + for (const char *p = ext; *p; ++p) + lext.push_back(static_cast(std::tolower(*p))); + + // Skip minified/bundled JS — generated code, expensive & low-value. + // Use both '/' and '\\' so basename extraction works on Windows too. + const char *slash_f = strrchr(file_path, '/'); + const char *slash_b = strrchr(file_path, '\\'); + const char *slash = (slash_b > slash_f) ? slash_b : slash_f; + const char *fname = slash ? slash + 1 : file_path; + size_t fname_len = strlen(fname); + // Lowercased basename tail checks for robustness on all platforms. + if (fname_len > 7) { + char tail[8] = { 0 }; + for (size_t i = 0; i < 7; i++) + tail[i] = static_cast( + std::tolower(fname[fname_len - 7 + i])); + if (strcmp(tail, ".min.js") == 0) + return nullptr; + } + if (fname_len > 10 && strstr(fname, ".bundle.js") != nullptr) + return nullptr; + if (strcmp(fname, "vendor.js") == 0) + return nullptr; + + if (lext == ".py") + return "python"; + if (lext == ".cpp" || lext == ".cc" || lext == ".cxx") + return "cpp"; + if (lext == ".c") + return "c"; + // C++ headers (.h/.hpp/.hxx/.hh) are parsed as C++ because the + // tree-sitter C++ grammar is a strict superset of the C grammar. + // Parsing .h as C would mis-handle templates, namespaces, classes, + // and other C++ constructs commonly found in .h files. + if (lext == ".h" || lext == ".hpp" || lext == ".hxx" || lext == ".hh") + return "cpp"; + if (lext == ".rs") + return "rust"; + if (lext == ".swift") + return "swift"; + if (lext == ".js" || lext == ".mjs" || lext == ".cjs") + return "javascript"; + if (lext == ".ts") + return "typescript"; + if (lext == ".tsx") + return "tsx"; + if (lext == ".go") + return "go"; + if (lext == ".java") + return "java"; + if (lext == ".kt" || lext == ".kts") + return "kotlin"; + if (lext == ".rb") + return "ruby"; + if (lext == ".scala") + return "scala"; + + return nullptr; +} + +bool FilterPolicy::shouldSkipPath(const std::string &rel_path, + bool is_dir) const +{ + if (rel_path.empty()) + return false; + + // 1. Check every path component against active_skip_dirs_ + // (this catches .venv, node_modules, etc. at any depth) + std::istringstream ss(rel_path); + std::string component; + while (std::getline(ss, component, '/')) { + if (shouldSkipDir(component)) + return true; + } + + // 1b. Shallow skip dirs — source-bearing but "rarely the focus + // of analysis" dirs (test/, tests/, docs/, vendor/, bench/, + // samples/, ...). For non-Java projects these are already + // caught at ANY depth by the active_skip_dirs_ check above + // (step 1), so this block is a no-op for them. For Java + // projects, the shouldSkipDir() Java carve-out defers these + // names HERE so nested package components (e.g. + // src/main/java/org/springframework/samples/petclinic) are + // NOT clobbered — only the first 3 path components are checked + // against java_protected_skip_dirs_, catching: + // /test/... (depth 1) + // /src/test/java/... (depth 2, Maven/Gradle) + // /packages//tests/... (depth 3, Lerna monorepo) + // without touching deep package namespaces. See README.md + // "Why Java is the (only) exception". + { + // Extract up to 3 leading path components (depth <= 3). + std::string head; + size_t off = 0; + for (int depth = 0; depth < 3; ++depth) { + auto slash = rel_path.find('/', off); + if (slash == std::string::npos) { + head = rel_path; + break; + } + off = slash + 1; + } + if (head.empty()) { + // rel_path has at least 3 components — take first 3. + head = rel_path.substr(0, off); + } + // Lowercase for case-insensitive comparison. + for (auto &c : head) + c = static_cast(std::tolower(c)); + std::istringstream ss2(head); + std::string comp; + while (std::getline(ss2, comp, '/')) { + if (comp.empty()) + continue; + // Depth-gated for EVERY language: the + // business-plausible dir names (public/, scripts/, + // integration/, static/, ...). Skipping these at any + // depth silently dropped real source files. + if (top_only_skip_dirs_.find(comp) != + top_only_skip_dirs_.end()) + return true; + // Java carve-out for the names that remain any-depth + // in active_skip_dirs_ (test/docs/samples/...). See + // the comment in java_protected_skip_dirs_. + if (lang_context_ == "java" && + java_protected_skip_dirs_.find(comp) != + java_protected_skip_dirs_.end()) + return true; + } + } + + // Rules anchored at the PROJECT root (.gitignore and the exclude list) are + // matched against the path as that root sees it: for a per-module worker + // scanning `/`, the module component has to be put + // back, or `**/build-*/` can never match a path that starts inside the + // module. Empty prefix = whole-project scan, i.e. the path as given. + std::string rule_path_storage; + const std::string &rule_path = + scan_prefix_.empty() ? + rel_path : + (rule_path_storage = scan_prefix_ + rel_path, + rule_path_storage); + + // 2. Check against .gitignore rules (whole-path matching) + if (!gitignore_rules_.empty() && + gitignoreMatches(gitignore_rules_, rule_path, is_dir)) + return true; + + // 3. Check against .codescopeignore raw patterns + // Supports: directory/ (trailing slash = dir only) + // filename (matches any component) + // /path/name (anchored from root) + for (const auto &pat : ignore_patterns_) { + bool dir_only = (!pat.empty() && pat.back() == '/'); + bool anchored = (!pat.empty() && pat[0] == '/'); + // For dir_only patterns applied to file entries: + // still check if any PATH COMPONENT matches the dir name + if (dir_only && pat.size() <= 1) + continue; + + // Normalize pattern: strip trailing / + std::string normalized = + dir_only ? pat.substr(0, pat.size() - 1) : pat; + + // Anchored: match from start. Anchored patterns are relative to the + // project root the file was loaded from, hence `rule_path`. + if (anchored) { + if (rule_path == normalized.substr(1) || + (rule_path.size() > normalized.size() - 1 && + rule_path.compare(0, normalized.size() - 1, + normalized.substr(1)) == 0 && + rule_path[normalized.size() - 1] == '/')) + return true; + continue; + } + + // Unanchored: check every path component + std::istringstream ss2(rule_path); + std::string comp; + while (std::getline(ss2, comp, '/')) { + if (comp == normalized) + return true; + } + } + + return false; +} + +bool FilterPolicy::shouldSkipEntry(const std::string &rel_path, + bool is_dir) const +{ + if (rel_path.empty()) + return false; + + // Normalize Windows backslashes to '/' so the rest of the pipeline + // (shouldSkipPath splits on '/', detectLanguage uses strrchr('/')) + // works uniformly across platforms. + std::string normalized = rel_path; + for (auto &c : normalized) + if (c == '\\') + c = '/'; + + // Extract the basename once — used by every per-entry check below. + // Use string_view into normalized to avoid copying (downstream calls + // that need std::string construct it from this view). + auto slash = normalized.rfind('/'); + size_t base_off = (slash == std::string::npos) ? 0 : slash + 1; + std::string_view base(normalized.data() + base_off, + normalized.size() - base_off); + + // ── Shared checks (both dirs & files) ── + // 1. Path-component skip_dirs (any depth) + gitignore + .codescopeignore. + // This alone catches node_modules/, .venv/, .git/ at any nesting. + if (shouldSkipPath(normalized, is_dir)) + return true; + + if (is_dir) { + // 2a. Bundle directory suffixes (.app, .framework, .xcodeproj ...) + // so we never recurse into binary payloads / IDE project bundles. + if (shouldSkipDirSuffix(std::string(base))) + return true; + // Dir-only checks done; fall through to the shared exclude-pattern + // check below so CODESCOPE_EXCLUDE_PATHS applies to directories too. + } else { + // ── File-only checks ── + // 2b. Exact filename + filename-prefix skip (.env, .env.local, ...) + if (shouldSkipFile(std::string(base))) + return true; + // 2c. Suffix skip — case-insensitive (.EXE == .exe). Catches + // binaries, archives, media, secrets, lock files, generated + // artifacts. + auto dot = base.rfind('.'); + if (dot != std::string_view::npos) { + if (shouldSkipSuffix(std::string(base.substr(dot)))) + return true; + } + + // 2d. STRICT mode: whitelist gate — only files that + // detectLanguage() recognizes as source code pass through. + // Catches config/docs/data files that slipped past the + // blacklist (.toml, .yaml, .json, .md). + if (mode_ == STRICT) { + if (detectLanguage(normalized.c_str()) == nullptr) + return true; + } + } + + // 3. User-specified exclude patterns (CODESCOPE_EXCLUDE_PATHS env var). + // Glob-matched against the FULL relative path, where `*` does not cross + // '/' and `**` does. Only "vendor/**" skips a directory AND its + // contents; "vendor/*" stops at the first level and a bare "vendor" + // matches nothing. Applied LAST so it acts as a user override on top of + // all built-in filters. See loadExcludeEnv() for the env var format. + for (const auto &pat : exclude_patterns_) { + if (globMatch(pat, normalized)) + return true; + } + + return false; +} + +bool FilterPolicy::loadIgnoreFile(const std::string &project_root, bool append) +{ + std::string path = project_root + "/.codescopeignore"; + std::ifstream f(path); + if (!f.is_open()) + return false; + + std::string line; + if (!append) + ignore_patterns_.clear(); + while (std::getline(f, line)) { + // Trim + while (!line.empty() && + (line.back() == '\r' || line.back() == '\n')) + line.pop_back(); + if (line.empty() || line[0] == '#') + continue; + ignore_patterns_.push_back(line); + } + return !ignore_patterns_.empty(); +} diff --git a/engine/src/filter_policy_ignore.cpp b/engine/src/filter_policy_ignore.cpp index 4c4ca5b..4c64283 100644 --- a/engine/src/filter_policy_ignore.cpp +++ b/engine/src/filter_policy_ignore.cpp @@ -8,9 +8,10 @@ // ─── Constants ───────────────────────────────────────────────── // Env var name for user-specified exclude paths. Comma-separated glob -// patterns (e.g. "test/*,docs/*,vendor/*,third_party/*") that extend +// patterns (e.g. "test/**,docs/**,vendor/**,third_party/**") that extend // the built-in skip list at index time to reduce graph_nodes count on -// very large projects. See FilterPolicy::loadExcludeEnv(). +// very large projects. Note `*` does not cross '/', so a subtree needs `**`. +// See FilterPolicy::loadExcludeEnv(). static constexpr const char *kExcludePathsEnv = "CODESCOPE_EXCLUDE_PATHS"; // ── Static gitignore matching helpers ──────────────────────── @@ -20,53 +21,83 @@ bool FilterPolicy::gitignoreMatches(const std::vector &rules, { bool ignored = false; for (const auto &r : rules) { - // Directory-only rule doesn't apply to files - if (r.dir_only && !is_dir) - continue; + // Does the rule match one candidate string? + auto matches_one = [&](const std::string &candidate) -> bool { + if (r.has_star) { + // Per gitignore spec: non-anchored pattern without '/' + // matches only the filename (last path component) + if (!r.anchored && + r.pattern.find('/') == std::string::npos) { + auto pos = candidate.rfind('/'); + auto basename = + (pos == std::string::npos) ? + candidate : + candidate.substr(pos + + 1); + return globMatch(r.pattern, basename); + } + return globMatch(r.pattern, candidate); + } + // Simple literal match — fast path + if (r.anchored) + return candidate == r.pattern; + // Literal match at path-component boundaries only, so + // "foo" matches "foo", "a/foo", "a/foo/b" but not + // "afoo" or "foobar". Iterate ALL occurrences (not just + // the last via rfind) so a pattern like "foo" matches + // even when "xfoo" appears later. + size_t search_from = 0; + while (true) { + auto pos = + candidate.find(r.pattern, search_from); + if (pos == std::string::npos) + break; + auto after = pos + r.pattern.size(); + bool left_boundary = + (pos == 0 || candidate[pos - 1] == '/'); + bool right_boundary = + (after == candidate.size() || + candidate[after] == '/'); + if (left_boundary && right_boundary) + return true; + search_from = pos + 1; + } + return false; + }; bool match = false; - if (r.has_star) { - // Per gitignore spec: non-anchored pattern without '/' - // matches only the filename (last path component) - if (!r.anchored && - r.pattern.find('/') == std::string::npos) { - auto pos = rel_path.rfind('/'); - auto basename = - (pos == std::string::npos) ? - rel_path : - rel_path.substr(pos + 1); - match = globMatch(r.pattern, basename); - } else { - match = globMatch(r.pattern, rel_path); + if (r.dir_only && !is_dir) { + // A directory-only rule names a DIRECTORY, so a file can + // never match it directly — but per gitignore semantics + // ignoring a directory ignores everything under it, so + // each ancestor directory is a candidate. Skipping this + // rule for files outright (the previous behaviour) meant + // `.gitignore`'s `**/build-*/` pruned the directory and + // nothing else, so a build tree's own sources — CMake's + // compiler-probe .c/.cpp files — were handed to the + // indexer and became entities, modules and call-edge + // candidates. See test_gitignore_build_dirs. + for (auto slash = rel_path.find('/'); + slash != std::string::npos; + slash = rel_path.find('/', slash + 1)) { + if (matches_one(rel_path.substr(0, slash))) { + match = true; + break; + } } } else { - // Simple literal match — fast path - if (r.anchored) { - match = (rel_path == r.pattern); - } else { - // Literal match at path-component boundaries only, - // so "foo" matches "foo", "a/foo", "a/foo/b" but - // not "afoo" or "foobar". Iterate ALL occurrences - // (not just the last via rfind) so a pattern like - // "foo" matches even when "xfoo" appears later. - size_t search_from = 0; - while (true) { - auto pos = rel_path.find(r.pattern, - search_from); - if (pos == std::string::npos) - break; - auto after = pos + r.pattern.size(); - bool left_boundary = - (pos == 0 || - rel_path[pos - 1] == '/'); - bool right_boundary = - (after == rel_path.size() || - rel_path[after] == '/'); - if (left_boundary && right_boundary) { + match = matches_one(rel_path); + // A directory under an ignored directory is ignored too, + // even when the rule's pattern ends at the parent. + if (!match && r.dir_only) { + for (auto slash = rel_path.find('/'); + slash != std::string::npos; + slash = rel_path.find('/', slash + 1)) { + if (matches_one(rel_path.substr( + 0, slash))) { match = true; break; } - search_from = pos + 1; } } } @@ -150,14 +181,15 @@ void FilterPolicy::printStats() const << " candidate_files=" << stats_.candidate_files << "\n"; } -bool FilterPolicy::loadGitignore(const std::string &project_root) +bool FilterPolicy::loadGitignore(const std::string &project_root, bool append) { std::string path = project_root + "/.gitignore"; std::ifstream f(path); if (!f.is_open()) return false; - gitignore_rules_.clear(); + if (!append) + gitignore_rules_.clear(); std::string line; while (std::getline(f, line)) { // Trim whitespace @@ -203,11 +235,21 @@ bool FilterPolicy::loadExcludeEnv() // under the FTS threshold. // // Default suggestions (NOT applied automatically — set explicitly): - // CODESCOPE_EXCLUDE_PATHS="test/*,docs/*,vendor/*,third_party/*" + // CODESCOPE_EXCLUDE_PATHS="test/**,docs/**,vendor/**,third_party/**" // - // Patterns are glob-matched against the relative path from the - // project root (e.g. "test/*" matches "test/foo.cpp" and - // "test/sub/bar.py" via the existing globMatch helper). + // Patterns are glob-matched against the FULL relative path from the project + // root, and `*` does not cross '/': "test/*" matches "test/foo.cpp" but NOT + // "test/sub/bar.py" — a subtree needs "test/**" ("**" spans separators). + // Directories are matched as well as files, but since the match is on the + // whole path only `dir/**` prunes a subtree; a bare "dir" or "dir/" matches + // nothing on its own. This comment used to claim "test/*" covered nested + // files, which made the suggested patterns exclude almost nothing. + // + // The list separator is a comma, and a pattern may contain one — a project + // with a directory named "a,b" produces `a,b/**`. Such a comma is escaped + // as `\,` by the producer (see join_exclude_patterns in + // server/src/scheduler/worker.rs); `\\` is a literal backslash. Any other + // backslash is kept verbatim so pre-existing patterns keep their meaning. const char *env = getenv(kExcludePathsEnv); if (!env || !*env) { // Env var not set or empty — nothing to load. @@ -215,10 +257,9 @@ bool FilterPolicy::loadExcludeEnv() return false; } std::string raw(env); - size_t start = 0, end; - do { - end = raw.find(',', start); - std::string pat = raw.substr(start, end - start); + std::string pat; + pat.reserve(raw.size()); + auto flush = [this, &pat]() { // Trim leading/trailing whitespace around each pattern. auto b = pat.find_first_not_of(" \t"); if (b != std::string::npos) { @@ -227,7 +268,22 @@ bool FilterPolicy::loadExcludeEnv() } if (!pat.empty()) exclude_patterns_.push_back(std::move(pat)); - start = end + 1; - } while (end != std::string::npos); + pat.clear(); + }; + for (size_t i = 0; i < raw.size(); ++i) { + char ch = raw[i]; + if (ch == '\\' && i + 1 < raw.size() && + (raw[i + 1] == ',' || raw[i + 1] == '\\')) { + pat.push_back(raw[i + 1]); + ++i; + continue; + } + if (ch == ',') { + flush(); + continue; + } + pat.push_back(ch); + } + flush(); return !exclude_patterns_.empty(); } diff --git a/engine/src/graph/graph_builder.cpp b/engine/src/graph/graph_builder.cpp index 202360a..7006fc9 100644 --- a/engine/src/graph/graph_builder.cpp +++ b/engine/src/graph/graph_builder.cpp @@ -324,10 +324,21 @@ CodeGraph GraphBuilder::buildSymbolGraph(const ir::SemanticUnit &unit) for (auto &rec : unit.allRecords()) { if (!isDeclarationKind(rec.kind)) continue; + // The anonymous root record — an empty-named Variable with no parent — + // is the container the visitors emit so every other record has a parent + // (e.g. JsVisitor::visit emits it as the TranslationUnit record). It is + // not a symbol: the bulk path never stores it (the entity table has no + // nameless row), so skipping it here is what makes a single-file graph + // match the same file in a bulk graph. + // + // The previous guard tested `recordKindToNodeType(kind) == + // NodeType::File`, which is never true for a Variable record + // (recordKindToNodeType maps Variable to NodeType::Variable), so the + // container stayed in the graph as a nameless ghost node. + if (rec.kind == ir::RecordKind::Variable && rec.name.empty() && + rec.parent_id == 0) + continue; NodeType nt = recordKindToNodeType(rec.kind); - if (nt == NodeType::File && - rec.kind == ir::RecordKind::Variable) - continue; // skip anonymous root Variable (used as container) addGraphNode(rec, nt); } diff --git a/engine/src/graph/graph_types.h b/engine/src/graph/graph_types.h index 2a965d8..b363189 100644 --- a/engine/src/graph/graph_types.h +++ b/engine/src/graph/graph_types.h @@ -92,8 +92,7 @@ const char *edgeTypeName(EdgeType t); // mapping. Production code MUST NOT branch on raw integer thresholds // such as `rtype >= 4`; it MUST call one of the helpers below. // -// See `plan/rules/relation_contract.md` and Step 0 of -// `ACCURACY_IMPROVEMENT_DEVELOPMENT_PLAN.md` for the full rationale. +// See `plan/rules/relation_contract.md` for the full rationale. /// Convert an integer `relation.type` value to a strongly-typed EdgeType. /// Out-of-range values map to `EdgeType::References` (a safe non-call diff --git a/engine/src/ir/builtin_registry.cpp b/engine/src/ir/builtin_registry.cpp index 19b8e61..591dfc7 100644 --- a/engine/src/ir/builtin_registry.cpp +++ b/engine/src/ir/builtin_registry.cpp @@ -1,4 +1,5 @@ #include "builtin_registry.h" +#include "builtin_registry_tables.h" #include #include @@ -7,7 +8,7 @@ namespace ir { // ── Python builtins (guaranteed by CPython) ───────────────────── -static const std::unordered_set &pyBuiltins() +const std::unordered_set &pyBuiltins() { static const std::unordered_set *s = new std::unordered_set{ @@ -115,7 +116,7 @@ static const std::unordered_set &pyBuiltins() return *s; } -static const std::unordered_set &pyStdlib() +const std::unordered_set &pyStdlib() { static const std::unordered_set< std::string> *s = new std::unordered_set{ @@ -178,7 +179,7 @@ static const std::unordered_set &pyStdlib() return *s; } -static const std::unordered_set &pyThirdParty() +const std::unordered_set &pyThirdParty() { static const std::unordered_set *s = new std::unordered_set{ @@ -542,1202 +543,6 @@ static const std::unordered_set &pyThirdParty() return *s; } -// ── C builtins ────────────────────────────────────────────────── -static const std::unordered_set &cBuiltins() -{ - static const std::unordered_set *s = - new std::unordered_set{ - // C standard library functions - "printf", - "sprintf", - "snprintf", - "fprintf", - "scanf", - "sscanf", - "fscanf", - "puts", - "gets", - "fputs", - "fgets", - "putchar", - "getchar", - "perror", - "strerror", - "malloc", - "calloc", - "realloc", - "free", - "aligned_alloc", - "memcpy", - "memmove", - "memset", - "memcmp", - "memchr", - "strlen", - "strcpy", - "strncpy", - "strcat", - "strncat", - "strcmp", - "strncmp", - "strchr", - "strrchr", - "strstr", - "strtok", - "strdup", - "strndup", - "strtol", - "strtoll", - "strtoul", - "strtoull", - "strtof", - "strtod", - "strtold", - "atoi", - "atol", - "atoll", - "atof", - "fopen", - "fclose", - "fread", - "fwrite", - "fseek", - "ftell", - "rewind", - "fflush", - "feof", - "ferror", - "clearerr", - "remove", - "rename", - "tmpfile", - "tmpnam", - "qsort", - "bsearch", - "abs", - "labs", - "llabs", - "div", - "ldiv", - "lldiv", - "rand", - "srand", - "exit", - "abort", - "atexit", - "quick_exit", - "_Exit", - "getenv", - "system", - "assert", - "signal", - "raise", - "setjmp", - "longjmp", - "va_start", - "va_end", - "va_arg", - "va_copy", - "offsetof", - "sizeof", - // C99 math - "sin", - "cos", - "tan", - "asin", - "acos", - "atan", - "atan2", - "sinh", - "cosh", - "tanh", - "asinh", - "acosh", - "atanh", - "exp", - "log", - "log10", - "log2", - "log1p", - "expm1", - "pow", - "sqrt", - "cbrt", - "hypot", - "ceil", - "floor", - "round", - "trunc", - "fabs", - "fmod", - "fmin", - "fmax", - "fdim", - "erf", - "erfc", - "tgamma", - "lgamma", - // Common POSIX / system calls - "open", - "close", - "read", - "write", - "lseek", - "stat", - "fstat", - "mmap", - "munmap", - "mprotect", - "mlock", - "munlock", - "fork", - "execvp", - "execve", - "wait", - "waitpid", - "waitid", - "pipe", - "dup", - "dup2", - "fcntl", - "ioctl", - "select", - "poll", - "epoll_create", - "epoll_ctl", - "epoll_wait", - "getpid", - "getppid", - "getuid", - "geteuid", - "getgid", - "getegid", - "chdir", - "getcwd", - "mkdir", - "rmdir", - "unlink", - "symlink", - "readlink", - "opendir", - "readdir", - "closedir", - "scandir", - "socket", - "bind", - "listen", - "accept", - "connect", - "send", - "recv", - "sendto", - "recvfrom", - "setsockopt", - "getsockopt", - "htons", - "htonl", - "ntohs", - "ntohl", - "inet_pton", - "inet_ntop", - "getaddrinfo", - "freeaddrinfo", - "gethostbyname", - // pthread - "pthread_create", - "pthread_join", - "pthread_detach", - "pthread_mutex_lock", - "pthread_mutex_unlock", - "pthread_mutex_init", - "pthread_mutex_destroy", - "pthread_cond_wait", - "pthread_cond_signal", - "pthread_cond_broadcast", - "pthread_self", - "pthread_exit", - "pthread_cancel", - // C11 threads - "thrd_create", - "thrd_join", - "thrd_detach", - "thrd_exit", - "mtx_init", - "mtx_lock", - "mtx_unlock", - "mtx_destroy", - "cnd_wait", - "cnd_signal", - "cnd_broadcast", - // Common type names (used as constructor calls in C++) - "int", - "long", - "float", - "double", - "char", - "short", - "unsigned", - "void", - "size_t", - "ssize_t", - "int8_t", - "int16_t", - "int32_t", - "int64_t", - "uint8_t", - "uint16_t", - "uint32_t", - "uint64_t", - "NULL", - "nullptr", - "true", - "false", - "bool", - }; - return *s; -} - -// ── C++ builtins ──────────────────────────────────────────────── -static const std::unordered_set &cppBuiltins() -{ - // C++ includes all C builtins, plus STL - static const std::unordered_set *s = - new std::unordered_set{ - // STL containers - "vector", - "map", - "unordered_map", - "set", - "unordered_set", - "list", - "deque", - "forward_list", - "array", - "span", - "string", - "string_view", - "wstring", - "pair", - "tuple", - "optional", - "variant", - "any", - "shared_ptr", - "unique_ptr", - "weak_ptr", - "function", - "bind", - "ref", - "cref", - // STL algorithms - "sort", - "stable_sort", - "partial_sort", - "nth_element", - "find", - "find_if", - "find_if_not", - "binary_search", - "lower_bound", - "upper_bound", - "equal_range", - "count", - "count_if", - "for_each", - "transform", - "copy", - "copy_if", - "move", - "fill", - "generate", - "replace", - "replace_if", - "remove", - "remove_if", - "unique", - "reverse", - "rotate", - "shuffle", - "random_shuffle", - "min", - "max", - "minmax", - "clamp", - "accumulate", - "inner_product", - "adjacent_difference", - "partial_sum", - // STL I/O - "cin", - "cout", - "cerr", - "clog", - "endl", - "flush", - // STL utilities - "make_shared", - "make_unique", - "make_pair", - "make_tuple", - "make_optional", - "make_variant", - "make_any", - "static_cast", - "dynamic_cast", - "const_cast", - "reinterpret_cast", - // C++17/20/23 filesystem - "filesystem", - "path", - "directory_iterator", - "recursive_directory_iterator", - "exists", - "is_directory", - "is_regular_file", - "file_size", - "create_directory", - "remove", - "rename", - "copy", - "current_path", - "temp_directory_path", - // C++20/23 ranges - "ranges", - "views", - "begin", - "end", - "size", - "empty", - "data", - "front", - "back", - "at", - "cbegin", - "cend", - // Threading - "thread", - "jthread", - "mutex", - "lock_guard", - "unique_lock", - "shared_lock", - "scoped_lock", - "condition_variable", - "future", - "promise", - "async", - "packaged_task", - "atomic", - "memory_order", - // Common macros expanded as identifiers - "assert", - "static_assert", - "sizeof", - "alignof", - "typeid", - "noexcept", - "override", - "final", - "default", - "delete", - "constexpr", - "consteval", - "constinit", - "co_await", - "co_yield", - "co_return", - }; - return *s; -} - -// ── Rust builtins ─────────────────────────────────────────────── -static const std::unordered_set &rustBuiltins() -{ - static const std::unordered_set *s = - new std::unordered_set{ - // Core language primitives - "Some", - "None", - "Ok", - "Err", - "Result", - "Option", - "Box", - "Vec", - "String", - "str", - "HashMap", - "HashSet", - "BTreeMap", - "BTreeSet", - "LinkedList", - "VecDeque", - "Rc", - "Arc", - "Cell", - "RefCell", - "Mutex", - "RwLock", - // Macros (most commonly used) - "println", - "print", - "eprintln", - "eprint", - "format", - "vec", - "vec_deque", - "btreemap", - "btreeset", - "hashmap", - "format_args", - "concat", - "stringify", - "include_str", - "include_bytes", - "include!", - "env", - "option_env", - "cfg", - "cfg_attr", - "column", - "file", - "line", - "module_path", - "todo", - "unimplemented", - "unreachable", - "panic", - "assert", - "assert_eq", - "assert_ne", - "debug_assert", - "debug_assert_eq", - "debug_assert_ne", - "matches", - "dbg", - "write", - "writeln", - // Core traits - "Clone", - "Copy", - "Debug", - "Display", - "Default", - "PartialEq", - "Eq", - "PartialOrd", - "Ord", - "Hash", - "Into", - "From", - "TryInto", - "TryFrom", - "IntoIterator", - "FromIterator", - "AsRef", - "AsMut", - "Deref", - "DerefMut", - "Drop", - "Send", - "Sync", - "Unpin", - "Sized", - // std library - "std", - "core", - "alloc", - "proc_macro", - // Common third-party crates - "serde", - "Serialize", - "Deserialize", - "serde_json", - "tokio", - "async_std", - "futures", - "rayon", - "reqwest", - "hyper", - "actix", - "axum", - "warp", - "rocket", - "clap", - "structopt", - "anyhow", - "thiserror", - "log", - "env_logger", - "tracing", - "tracing_subscriber", - "chrono", - "uuid", - "rand", - "regex", - "lazy_static", - "once_cell", - "parking_lot", - "crossbeam", - "diesel", - "sqlx", - "rusqlite", - "mongodb", - "tonic", - "prost", - "protobuf", - "tempfile", - "walkdir", - "glob", - "ignore", - }; - return *s; -} - -// ── Go builtins ───────────────────────────────────────────────── -static const std::unordered_set &goBuiltins() -{ - static const std::unordered_set *s = - new std::unordered_set{ - // Built-in functions - "append", - "cap", - "close", - "complex", - "copy", - "delete", - "imag", - "len", - "make", - "new", - "panic", - "print", - "println", - "real", - "recover", - // Built-in types - "bool", - "byte", - "complex64", - "complex128", - "error", - "float32", - "float64", - "int", - "int8", - "int16", - "int32", - "int64", - "rune", - "string", - "uint", - "uint8", - "uint16", - "uint32", - "uint64", - "uintptr", - "nil", - "true", - "false", - "iota", - // Standard library: common packages - "fmt", - "io", - "os", - "net", - "http", - "json", - "time", - "strings", - "strconv", - "bytes", - "bufio", - "sort", - "math", - "rand", - "crypto", - "encoding", - "compress", - "sync", - "atomic", - "context", - "reflect", - "regexp", - "log", - "flag", - "path", - "filepath", - "ioutil", - "template", - "html", - "testing", - "database/sql", - // Common third-party - "gin", - "echo", - "fiber", - "chi", - "gorilla", - "mux", - "gorm", - "sqlx", - "kafka", - "redis", - "mongodb", - "protobuf", - "grpc", - "cobra", - "viper", - "zap", - "prometheus", - "otel", - "opentracing", - "jaeger", - }; - return *s; -} - -// ── Java builtins ─────────────────────────────────────────────── -static const std::unordered_set &javaBuiltins() -{ - static const std::unordered_set *s = - new std::unordered_set{ - // java.lang — always imported - "Object", - "String", - "Integer", - "Long", - "Double", - "Float", - "Boolean", - "Character", - "Short", - "Byte", - "Void", - "Number", - "Math", - "System", - "Thread", - "Runnable", - "Class", - "Throwable", - "Exception", - "RuntimeException", - "Error", - "NullPointerException", - "IllegalArgumentException", - "IndexOutOfBoundsException", - "ArrayIndexOutOfBoundsException", - "StringBuilder", - "StringBuffer", - "Comparable", - "Cloneable", - "Enum", - "AutoCloseable", - "Iterable", - "Appendable", - "Runtime", - "Process", - "ProcessBuilder", - "SecurityManager", - // Common collections - "ArrayList", - "LinkedList", - "HashMap", - "HashSet", - "LinkedHashMap", - "TreeMap", - "TreeSet", - "PriorityQueue", - "ArrayDeque", - "Collections", - "Arrays", - "Comparator", - "Iterator", - "ListIterator", - "List", - "Set", - "Map", - "Queue", - "Deque", - "Collection", - // I/O - "File", - "Path", - "Paths", - "Files", - "InputStream", - "OutputStream", - "Reader", - "Writer", - "BufferedReader", - "BufferedWriter", - "FileReader", - "FileWriter", - "PrintWriter", - "Scanner", - // Common frameworks - "spring", - "SpringBoot", - "Autowired", - "Service", - "Component", - "Repository", - "Controller", - "RestController", - "RequestMapping", - "GetMapping", - "PostMapping", - "PutMapping", - "DeleteMapping", - "PathVariable", - "RequestParam", - "RequestBody", - "ResponseEntity", - "jpa", - "JpaRepository", - "CrudRepository", - "Entity", - "Table", - "Column", - "Id", - "GeneratedValue", - "OneToMany", - "ManyToOne", - "lombok", - "Data", - "Getter", - "Setter", - "NoArgsConstructor", - "AllArgsConstructor", - "Builder", - "Slf4j", - "Log4j", - "jackson", - "JsonIgnore", - "JsonProperty", - "JsonFormat", - "junit", - "Test", - "Before", - "After", - "BeforeEach", - "AfterEach", - }; - return *s; -} - -// ── JavaScript / TypeScript builtins ──────────────────────────── -static const std::unordered_set &jsBuiltins() -{ - static const std::unordered_set *s = - new std::unordered_set{ - // Global objects - "Object", - "Array", - "String", - "Number", - "Boolean", - "Symbol", - "Function", - "Date", - "RegExp", - "Error", - "TypeError", - "RangeError", - "ReferenceError", - "SyntaxError", - "EvalError", - "Promise", - "Proxy", - "Reflect", - "Map", - "Set", - "WeakMap", - "WeakSet", - "JSON", - "Math", - "Intl", - "Buffer", - "console", - "process", - "global", - "globalThis", - // Global functions - "parseInt", - "parseFloat", - "isNaN", - "isFinite", - "decodeURI", - "decodeURIComponent", - "encodeURI", - "encodeURIComponent", - "eval", - "uneval", - "escape", - "unescape", - // Array methods - "map", - "filter", - "reduce", - "reduceRight", - "forEach", - "find", - "findIndex", - "some", - "every", - "includes", - "indexOf", - "lastIndexOf", - "push", - "pop", - "shift", - "unshift", - "splice", - "slice", - "concat", - "join", - "reverse", - "sort", - "flat", - "flatMap", - "from", - "of", - "isArray", - "fill", - "copyWithin", - "entries", - "keys", - "values", - // Object methods - "keys", - "values", - "entries", - "assign", - "create", - "defineProperty", - "defineProperties", - "getOwnPropertyDescriptor", - "getOwnPropertyNames", - "getPrototypeOf", - "setPrototypeOf", - "freeze", - "seal", - "preventExtensions", - "isFrozen", - "isSealed", - "isExtensible", - "hasOwnProperty", - // String methods - "charAt", - "charCodeAt", - "concat", - "includes", - "indexOf", - "lastIndexOf", - "match", - "matchAll", - "replace", - "replaceAll", - "search", - "slice", - "split", - "substring", - "substr", - "toLowerCase", - "toUpperCase", - "trim", - "trimStart", - "trimEnd", - "padStart", - "padEnd", - "repeat", - "startsWith", - "endsWith", - "localeCompare", - "normalize", - "raw", - "fromCharCode", - "fromCodePoint", - // Promise - "then", - "catch", - "finally", - "resolve", - "reject", - "all", - "allSettled", - "race", - "any", - "withResolvers", - // Common Node.js - "require", - "module", - "exports", - "__dirname", - "__filename", - "setTimeout", - "setInterval", - "setImmediate", - "clearTimeout", - "clearInterval", - "clearImmediate", - "queueMicrotask", - "Buffer", - "fs", - "path", - "os", - "util", - "events", - "stream", - "http", - "https", - "net", - "crypto", - "url", - "querystring", - "child_process", - "cluster", - "worker_threads", - // Common Web APIs - "document", - "window", - "navigator", - "location", - "history", - "localStorage", - "sessionStorage", - "fetch", - "WebSocket", - "setTimeout", - "setInterval", - "requestAnimationFrame", - "addEventListener", - "removeEventListener", - "dispatchEvent", - "querySelector", - "querySelectorAll", - "getElementById", - "createElement", - "appendChild", - "removeChild", - // Common TS/JS frameworks - "react", - "React", - "useState", - "useEffect", - "useContext", - "useReducer", - "useCallback", - "useMemo", - "useRef", - "useImperativeHandle", - "useLayoutEffect", - "useDebugValue", - "useTransition", - "useDeferredValue", - "createContext", - "createRef", - "forwardRef", - "memo", - "Fragment", - "Component", - "PureComponent", - "createElement", - "cloneElement", - "isValidElement", - "Children", - "StrictMode", - "Suspense", - "lazy", - "startTransition", - "useId", - "useSyncExternalStore", - "vue", - "Vue", - "ref", - "computed", - "watch", - "watchEffect", - "onMounted", - "onUnmounted", - "onUpdated", - "nextTick", - "defineComponent", - "defineProps", - "defineEmits", - "defineExpose", - "angular", - "NgModule", - "Component", - "Injectable", - "Inject", - "Input", - "Output", - "EventEmitter", - "HostListener", - "Directive", - "Pipe", - "Service", - "Observable", - "Subject", - "BehaviorSubject", - "express", - "Router", - "next", - "Request", - "Response", - "lodash", - "_", - "axios", - "moment", - "dayjs", - "jest", - "describe", - "it", - "expect", - "beforeEach", - "afterEach", - "beforeAll", - "afterAll", - "test", - "mock", - "spyOn", - }; - return *s; -} - -// ── Swift builtins ────────────────────────────────────────────── -static const std::unordered_set &swiftBuiltins() -{ - static const std::unordered_set *s = - new std::unordered_set{ - "print", - "debugPrint", - "dump", - "fatalError", - "precondition", - "preconditionFailure", - "assert", - "assertionFailure", - "abs", - "min", - "max", - "stride", - "stride", - "sequence", - "zip", - "swap", - "type", - "unsafeBitCast", - "unsafeDowncast", - "Array", - "Dictionary", - "Set", - "String", - "Substring", - "Optional", - "Result", - "Bool", - "Int", - "UInt", - "Float", - "Double", - "CGFloat", - "Character", - "Data", - "Date", - "UUID", - "URL", - "Error", - "Codable", - "Encodable", - "Decodable", - "Equatable", - "Hashable", - "Comparable", - "Identifiable", - "CustomStringConvertible", - "CustomDebugStringConvertible", - "UIView", - "UIViewController", - "UITableView", - "UICollectionView", - "UIStackView", - "UILabel", - "UIButton", - "UIImageView", - "UITextField", - "NSObject", - "NSArray", - "NSDictionary", - "NSString", - "UserDefaults", - "NotificationCenter", - "FileManager", - "Bundle", - "DispatchQueue", - "DispatchGroup", - "DispatchWorkItem", - "OperationQueue", - "Timer", - "RunLoop", - "Thread", - "Lock", - "JSONEncoder", - "JSONDecoder", - "PropertyListEncoder", - "PropertyListDecoder", - "URLSession", - "URLRequest", - "URLResponse", - "HTTPURLResponse", - "DataTask", - "UploadTask", - "DownloadTask", - // SwiftUI - "View", - "Text", - "Image", - "Button", - "TextField", - "Toggle", - "Slider", - "Stepper", - "Picker", - "DatePicker", - "ColorPicker", - "List", - "ScrollView", - "VStack", - "HStack", - "ZStack", - "Spacer", - "Divider", - "Group", - "Section", - "Form", - "NavigationView", - "NavigationLink", - "TabView", - "Sheet", - "Alert", - "Popover", - "State", - "Binding", - "ObservedObject", - "StateObject", - "EnvironmentObject", - "Environment", - "Published", - "ObservableObject", - "@main", - "@UIApplicationMain", - "@main", - }; - return *s; -} - // ── Registry dispatch ─────────────────────────────────────────── // Map language → symbol set using SymbolTable = std::unordered_set; @@ -1757,7 +562,6 @@ static const RegistryMap ®istry() { "typescript", { &jsBuiltins() } }, { "js", { &jsBuiltins() } }, { "ts", { &jsBuiltins() } }, - { "swift", { &swiftBuiltins() } }, }; return *m; } @@ -1824,4 +628,4 @@ BuiltinRegistry::externalSymbols(const std::string &language) return result.first->second; } -} // namespace ir \ No newline at end of file +} // namespace ir diff --git a/engine/src/ir/builtin_registry.h b/engine/src/ir/builtin_registry.h index 1eb978e..acb44b3 100644 --- a/engine/src/ir/builtin_registry.h +++ b/engine/src/ir/builtin_registry.h @@ -53,25 +53,6 @@ class BuiltinRegistry { */ static const std::unordered_set & externalSymbols(const std::string &language); - - private: - // ── Language-specific symbol tables ──────────────────────── - // Each table contains builtin functions and high-frequency - // third-party library symbols. These are symbols that are NOT - // defined in user project code but are called from it. - - static const std::unordered_set &pythonBuiltins(); - static const std::unordered_set &pythonStdlib(); - static const std::unordered_set &pythonThirdParty(); - static const std::unordered_set &cBuiltins(); - static const std::unordered_set &cppBuiltins(); - static const std::unordered_set &cppStdlib(); - static const std::unordered_set &rustBuiltins(); - static const std::unordered_set &rustStdlib(); - static const std::unordered_set &goBuiltins(); - static const std::unordered_set &javaBuiltins(); - static const std::unordered_set &jsBuiltins(); - static const std::unordered_set &swiftBuiltins(); }; } // namespace ir diff --git a/engine/src/ir/builtin_registry_app.cpp b/engine/src/ir/builtin_registry_app.cpp new file mode 100644 index 0000000..9841e47 --- /dev/null +++ b/engine/src/ir/builtin_registry_app.cpp @@ -0,0 +1,428 @@ +// builtin_registry_app.cpp — Java / JavaScript / Swift symbol tables. +// +// Split out of builtin_registry.cpp (see plan/rules/code_rules.md +// 1000-line rule). Grouped as the managed/app-language tables: their +// builtins come from a runtime (JVM, browser, Foundation) rather than from +// the language spec, so the lists are long-tailed and change with the +// platform rather than with the compiler. + +#include "builtin_registry.h" +#include "builtin_registry_tables.h" + +#include +#include + +namespace ir +{ + +// ── Java builtins ─────────────────────────────────────────────── +const std::unordered_set &javaBuiltins() +{ + static const std::unordered_set *s = + new std::unordered_set{ + // java.lang — always imported + "Object", + "String", + "Integer", + "Long", + "Double", + "Float", + "Boolean", + "Character", + "Short", + "Byte", + "Void", + "Number", + "Math", + "System", + "Thread", + "Runnable", + "Class", + "Throwable", + "Exception", + "RuntimeException", + "Error", + "NullPointerException", + "IllegalArgumentException", + "IndexOutOfBoundsException", + "ArrayIndexOutOfBoundsException", + "StringBuilder", + "StringBuffer", + "Comparable", + "Cloneable", + "Enum", + "AutoCloseable", + "Iterable", + "Appendable", + "Runtime", + "Process", + "ProcessBuilder", + "SecurityManager", + // Common collections + "ArrayList", + "LinkedList", + "HashMap", + "HashSet", + "LinkedHashMap", + "TreeMap", + "TreeSet", + "PriorityQueue", + "ArrayDeque", + "Collections", + "Arrays", + "Comparator", + "Iterator", + "ListIterator", + "List", + "Set", + "Map", + "Queue", + "Deque", + "Collection", + // I/O + "File", + "Path", + "Paths", + "Files", + "InputStream", + "OutputStream", + "Reader", + "Writer", + "BufferedReader", + "BufferedWriter", + "FileReader", + "FileWriter", + "PrintWriter", + "Scanner", + // Common frameworks + "spring", + "SpringBoot", + "Autowired", + "Service", + "Component", + "Repository", + "Controller", + "RestController", + "RequestMapping", + "GetMapping", + "PostMapping", + "PutMapping", + "DeleteMapping", + "PathVariable", + "RequestParam", + "RequestBody", + "ResponseEntity", + "jpa", + "JpaRepository", + "CrudRepository", + "Entity", + "Table", + "Column", + "Id", + "GeneratedValue", + "OneToMany", + "ManyToOne", + "lombok", + "Data", + "Getter", + "Setter", + "NoArgsConstructor", + "AllArgsConstructor", + "Builder", + "Slf4j", + "Log4j", + "jackson", + "JsonIgnore", + "JsonProperty", + "JsonFormat", + "junit", + "Test", + "Before", + "After", + "BeforeEach", + "AfterEach", + }; + return *s; +} + +// ── JavaScript / TypeScript builtins ──────────────────────────── +const std::unordered_set &jsBuiltins() +{ + static const std::unordered_set *s = + new std::unordered_set{ + // Global objects + "Object", + "Array", + "String", + "Number", + "Boolean", + "Symbol", + "Function", + "Date", + "RegExp", + "Error", + "TypeError", + "RangeError", + "ReferenceError", + "SyntaxError", + "EvalError", + "Promise", + "Proxy", + "Reflect", + "Map", + "Set", + "WeakMap", + "WeakSet", + "JSON", + "Math", + "Intl", + "Buffer", + "console", + "process", + "global", + "globalThis", + // Global functions + "parseInt", + "parseFloat", + "isNaN", + "isFinite", + "decodeURI", + "decodeURIComponent", + "encodeURI", + "encodeURIComponent", + "eval", + "uneval", + "escape", + "unescape", + // Array methods + "map", + "filter", + "reduce", + "reduceRight", + "forEach", + "find", + "findIndex", + "some", + "every", + "includes", + "indexOf", + "lastIndexOf", + "push", + "pop", + "shift", + "unshift", + "splice", + "slice", + "concat", + "join", + "reverse", + "sort", + "flat", + "flatMap", + "from", + "of", + "isArray", + "fill", + "copyWithin", + "entries", + "keys", + "values", + // Object methods + "keys", + "values", + "entries", + "assign", + "create", + "defineProperty", + "defineProperties", + "getOwnPropertyDescriptor", + "getOwnPropertyNames", + "getPrototypeOf", + "setPrototypeOf", + "freeze", + "seal", + "preventExtensions", + "isFrozen", + "isSealed", + "isExtensible", + "hasOwnProperty", + // String methods + "charAt", + "charCodeAt", + "concat", + "includes", + "indexOf", + "lastIndexOf", + "match", + "matchAll", + "replace", + "replaceAll", + "search", + "slice", + "split", + "substring", + "substr", + "toLowerCase", + "toUpperCase", + "trim", + "trimStart", + "trimEnd", + "padStart", + "padEnd", + "repeat", + "startsWith", + "endsWith", + "localeCompare", + "normalize", + "raw", + "fromCharCode", + "fromCodePoint", + // Promise + "then", + "catch", + "finally", + "resolve", + "reject", + "all", + "allSettled", + "race", + "any", + "withResolvers", + // Common Node.js + "require", + "module", + "exports", + "__dirname", + "__filename", + "setTimeout", + "setInterval", + "setImmediate", + "clearTimeout", + "clearInterval", + "clearImmediate", + "queueMicrotask", + "Buffer", + "fs", + "path", + "os", + "util", + "events", + "stream", + "http", + "https", + "net", + "crypto", + "url", + "querystring", + "child_process", + "cluster", + "worker_threads", + // Common Web APIs + "document", + "window", + "navigator", + "location", + "history", + "localStorage", + "sessionStorage", + "fetch", + "WebSocket", + "setTimeout", + "setInterval", + "requestAnimationFrame", + "addEventListener", + "removeEventListener", + "dispatchEvent", + "querySelector", + "querySelectorAll", + "getElementById", + "createElement", + "appendChild", + "removeChild", + // Common TS/JS frameworks + "react", + "React", + "useState", + "useEffect", + "useContext", + "useReducer", + "useCallback", + "useMemo", + "useRef", + "useImperativeHandle", + "useLayoutEffect", + "useDebugValue", + "useTransition", + "useDeferredValue", + "createContext", + "createRef", + "forwardRef", + "memo", + "Fragment", + "Component", + "PureComponent", + "createElement", + "cloneElement", + "isValidElement", + "Children", + "StrictMode", + "Suspense", + "lazy", + "startTransition", + "useId", + "useSyncExternalStore", + "vue", + "Vue", + "ref", + "computed", + "watch", + "watchEffect", + "onMounted", + "onUnmounted", + "onUpdated", + "nextTick", + "defineComponent", + "defineProps", + "defineEmits", + "defineExpose", + "angular", + "NgModule", + "Component", + "Injectable", + "Inject", + "Input", + "Output", + "EventEmitter", + "HostListener", + "Directive", + "Pipe", + "Service", + "Observable", + "Subject", + "BehaviorSubject", + "express", + "Router", + "next", + "Request", + "Response", + "lodash", + "_", + "axios", + "moment", + "dayjs", + "jest", + "describe", + "it", + "expect", + "beforeEach", + "afterEach", + "beforeAll", + "afterAll", + "test", + "mock", + "spyOn", + }; + return *s; +} + +} // namespace ir diff --git a/engine/src/ir/builtin_registry_systems.cpp b/engine/src/ir/builtin_registry_systems.cpp new file mode 100644 index 0000000..f5ddf4d --- /dev/null +++ b/engine/src/ir/builtin_registry_systems.cpp @@ -0,0 +1,672 @@ +// builtin_registry_systems.cpp — C / C++ / Rust / Go symbol tables. +// +// Split out of builtin_registry.cpp (see plan/rules/code_rules.md +// 1000-line rule). These four languages share a table because they are the +// ones whose "builtin" sets overlap heavily with the standard library a +// project is assumed to have linked against. + +#include "builtin_registry.h" +#include "builtin_registry_tables.h" + +#include +#include + +namespace ir +{ + +// ── C builtins ────────────────────────────────────────────────── +const std::unordered_set &cBuiltins() +{ + static const std::unordered_set *s = + new std::unordered_set{ + // C standard library functions + "printf", + "sprintf", + "snprintf", + "fprintf", + "scanf", + "sscanf", + "fscanf", + "puts", + "gets", + "fputs", + "fgets", + "putchar", + "getchar", + "perror", + "strerror", + "malloc", + "calloc", + "realloc", + "free", + "aligned_alloc", + "memcpy", + "memmove", + "memset", + "memcmp", + "memchr", + "strlen", + "strcpy", + "strncpy", + "strcat", + "strncat", + "strcmp", + "strncmp", + "strchr", + "strrchr", + "strstr", + "strtok", + "strdup", + "strndup", + "strtol", + "strtoll", + "strtoul", + "strtoull", + "strtof", + "strtod", + "strtold", + "atoi", + "atol", + "atoll", + "atof", + "fopen", + "fclose", + "fread", + "fwrite", + "fseek", + "ftell", + "rewind", + "fflush", + "feof", + "ferror", + "clearerr", + "remove", + "rename", + "tmpfile", + "tmpnam", + "qsort", + "bsearch", + "abs", + "labs", + "llabs", + "div", + "ldiv", + "lldiv", + "rand", + "srand", + "exit", + "abort", + "atexit", + "quick_exit", + "_Exit", + "getenv", + "system", + "assert", + "signal", + "raise", + "setjmp", + "longjmp", + "va_start", + "va_end", + "va_arg", + "va_copy", + "offsetof", + "sizeof", + // C99 math + "sin", + "cos", + "tan", + "asin", + "acos", + "atan", + "atan2", + "sinh", + "cosh", + "tanh", + "asinh", + "acosh", + "atanh", + "exp", + "log", + "log10", + "log2", + "log1p", + "expm1", + "pow", + "sqrt", + "cbrt", + "hypot", + "ceil", + "floor", + "round", + "trunc", + "fabs", + "fmod", + "fmin", + "fmax", + "fdim", + "erf", + "erfc", + "tgamma", + "lgamma", + // Common POSIX / system calls + "open", + "close", + "read", + "write", + "lseek", + "stat", + "fstat", + "mmap", + "munmap", + "mprotect", + "mlock", + "munlock", + "fork", + "execvp", + "execve", + "wait", + "waitpid", + "waitid", + "pipe", + "dup", + "dup2", + "fcntl", + "ioctl", + "select", + "poll", + "epoll_create", + "epoll_ctl", + "epoll_wait", + "getpid", + "getppid", + "getuid", + "geteuid", + "getgid", + "getegid", + "chdir", + "getcwd", + "mkdir", + "rmdir", + "unlink", + "symlink", + "readlink", + "opendir", + "readdir", + "closedir", + "scandir", + "socket", + "bind", + "listen", + "accept", + "connect", + "send", + "recv", + "sendto", + "recvfrom", + "setsockopt", + "getsockopt", + "htons", + "htonl", + "ntohs", + "ntohl", + "inet_pton", + "inet_ntop", + "getaddrinfo", + "freeaddrinfo", + "gethostbyname", + // pthread + "pthread_create", + "pthread_join", + "pthread_detach", + "pthread_mutex_lock", + "pthread_mutex_unlock", + "pthread_mutex_init", + "pthread_mutex_destroy", + "pthread_cond_wait", + "pthread_cond_signal", + "pthread_cond_broadcast", + "pthread_self", + "pthread_exit", + "pthread_cancel", + // C11 threads + "thrd_create", + "thrd_join", + "thrd_detach", + "thrd_exit", + "mtx_init", + "mtx_lock", + "mtx_unlock", + "mtx_destroy", + "cnd_wait", + "cnd_signal", + "cnd_broadcast", + // Common type names (used as constructor calls in C++) + "int", + "long", + "float", + "double", + "char", + "short", + "unsigned", + "void", + "size_t", + "ssize_t", + "int8_t", + "int16_t", + "int32_t", + "int64_t", + "uint8_t", + "uint16_t", + "uint32_t", + "uint64_t", + "NULL", + "nullptr", + "true", + "false", + "bool", + }; + return *s; +} + +// ── C++ builtins ──────────────────────────────────────────────── +const std::unordered_set &cppBuiltins() +{ + // C++ includes all C builtins, plus STL + static const std::unordered_set *s = + new std::unordered_set{ + // STL containers + "vector", + "map", + "unordered_map", + "set", + "unordered_set", + "list", + "deque", + "forward_list", + "array", + "span", + "string", + "string_view", + "wstring", + "pair", + "tuple", + "optional", + "variant", + "any", + "shared_ptr", + "unique_ptr", + "weak_ptr", + "function", + "bind", + "ref", + "cref", + // STL algorithms + "sort", + "stable_sort", + "partial_sort", + "nth_element", + "find", + "find_if", + "find_if_not", + "binary_search", + "lower_bound", + "upper_bound", + "equal_range", + "count", + "count_if", + "for_each", + "transform", + "copy", + "copy_if", + "move", + "fill", + "generate", + "replace", + "replace_if", + "remove", + "remove_if", + "unique", + "reverse", + "rotate", + "shuffle", + "random_shuffle", + "min", + "max", + "minmax", + "clamp", + "accumulate", + "inner_product", + "adjacent_difference", + "partial_sum", + // STL I/O + "cin", + "cout", + "cerr", + "clog", + "endl", + "flush", + // STL utilities + "make_shared", + "make_unique", + "make_pair", + "make_tuple", + "make_optional", + "make_variant", + "make_any", + "static_cast", + "dynamic_cast", + "const_cast", + "reinterpret_cast", + // C++17/20/23 filesystem + "filesystem", + "path", + "directory_iterator", + "recursive_directory_iterator", + "exists", + "is_directory", + "is_regular_file", + "file_size", + "create_directory", + "remove", + "rename", + "copy", + "current_path", + "temp_directory_path", + // C++20/23 ranges + "ranges", + "views", + "begin", + "end", + "size", + "empty", + "data", + "front", + "back", + "at", + "cbegin", + "cend", + // Threading + "thread", + "jthread", + "mutex", + "lock_guard", + "unique_lock", + "shared_lock", + "scoped_lock", + "condition_variable", + "future", + "promise", + "async", + "packaged_task", + "atomic", + "memory_order", + // Common macros expanded as identifiers + "assert", + "static_assert", + "sizeof", + "alignof", + "typeid", + "noexcept", + "override", + "final", + "default", + "delete", + "constexpr", + "consteval", + "constinit", + "co_await", + "co_yield", + "co_return", + }; + return *s; +} + +// ── Rust builtins ─────────────────────────────────────────────── +const std::unordered_set &rustBuiltins() +{ + static const std::unordered_set *s = + new std::unordered_set{ + // Core language primitives + "Some", + "None", + "Ok", + "Err", + "Result", + "Option", + "Box", + "Vec", + "String", + "str", + "HashMap", + "HashSet", + "BTreeMap", + "BTreeSet", + "LinkedList", + "VecDeque", + "Rc", + "Arc", + "Cell", + "RefCell", + "Mutex", + "RwLock", + // Macros (most commonly used) + "println", + "print", + "eprintln", + "eprint", + "format", + "vec", + "vec_deque", + "btreemap", + "btreeset", + "hashmap", + "format_args", + "concat", + "stringify", + "include_str", + "include_bytes", + "include!", + "env", + "option_env", + "cfg", + "cfg_attr", + "column", + "file", + "line", + "module_path", + "todo", + "unimplemented", + "unreachable", + "panic", + "assert", + "assert_eq", + "assert_ne", + "debug_assert", + "debug_assert_eq", + "debug_assert_ne", + "matches", + "dbg", + "write", + "writeln", + // Core traits + "Clone", + "Copy", + "Debug", + "Display", + "Default", + "PartialEq", + "Eq", + "PartialOrd", + "Ord", + "Hash", + "Into", + "From", + "TryInto", + "TryFrom", + "IntoIterator", + "FromIterator", + "AsRef", + "AsMut", + "Deref", + "DerefMut", + "Drop", + "Send", + "Sync", + "Unpin", + "Sized", + // std library + "std", + "core", + "alloc", + "proc_macro", + // Common third-party crates + "serde", + "Serialize", + "Deserialize", + "serde_json", + "tokio", + "async_std", + "futures", + "rayon", + "reqwest", + "hyper", + "actix", + "axum", + "warp", + "rocket", + "clap", + "structopt", + "anyhow", + "thiserror", + "log", + "env_logger", + "tracing", + "tracing_subscriber", + "chrono", + "uuid", + "rand", + "regex", + "lazy_static", + "once_cell", + "parking_lot", + "crossbeam", + "diesel", + "sqlx", + "rusqlite", + "mongodb", + "tonic", + "prost", + "protobuf", + "tempfile", + "walkdir", + "glob", + "ignore", + }; + return *s; +} + +// ── Go builtins ───────────────────────────────────────────────── +const std::unordered_set &goBuiltins() +{ + static const std::unordered_set *s = + new std::unordered_set{ + // Built-in functions + "append", + "cap", + "close", + "complex", + "copy", + "delete", + "imag", + "len", + "make", + "new", + "panic", + "print", + "println", + "real", + "recover", + // Built-in types + "bool", + "byte", + "complex64", + "complex128", + "error", + "float32", + "float64", + "int", + "int8", + "int16", + "int32", + "int64", + "rune", + "string", + "uint", + "uint8", + "uint16", + "uint32", + "uint64", + "uintptr", + "nil", + "true", + "false", + "iota", + // Standard library: common packages + "fmt", + "io", + "os", + "net", + "http", + "json", + "time", + "strings", + "strconv", + "bytes", + "bufio", + "sort", + "math", + "rand", + "crypto", + "encoding", + "compress", + "sync", + "atomic", + "context", + "reflect", + "regexp", + "log", + "flag", + "path", + "filepath", + "ioutil", + "template", + "html", + "testing", + "database/sql", + // Common third-party + "gin", + "echo", + "fiber", + "chi", + "gorilla", + "mux", + "gorm", + "sqlx", + "kafka", + "redis", + "mongodb", + "protobuf", + "grpc", + "cobra", + "viper", + "zap", + "prometheus", + "otel", + "opentracing", + "jaeger", + }; + return *s; +} + +} // namespace ir diff --git a/engine/src/ir/builtin_registry_tables.h b/engine/src/ir/builtin_registry_tables.h new file mode 100644 index 0000000..d6eb8bd --- /dev/null +++ b/engine/src/ir/builtin_registry_tables.h @@ -0,0 +1,43 @@ +#ifndef BUILTIN_REGISTRY_TABLES_H +#define BUILTIN_REGISTRY_TABLES_H + +#include +#include + +// Language-specific builtin / third-party symbol tables, split across three +// translation units (see plan/rules/code_rules.md 1000-line rule): +// builtin_registry.cpp — Python +// builtin_registry_systems.cpp — C, C++, Rust, Go +// builtin_registry_app.cpp — Java, JavaScript/TypeScript, Swift +// +// They are declared here rather than kept `static` in one file so the +// registry dispatch table can reference every table from a single TU. +// Each table is a function-local `static` set built once on first call, so +// the split costs nothing at startup: a project only pays for the languages +// it actually contains. + +namespace ir +{ + +/** @return The pyBuiltins symbol table (lazily built, never mutated). */ +const std::unordered_set &pyBuiltins(); +/** @return The pyStdlib symbol table (lazily built, never mutated). */ +const std::unordered_set &pyStdlib(); +/** @return The pyThirdParty symbol table (lazily built, never mutated). */ +const std::unordered_set &pyThirdParty(); +/** @return The cBuiltins symbol table (lazily built, never mutated). */ +const std::unordered_set &cBuiltins(); +/** @return The cppBuiltins symbol table (lazily built, never mutated). */ +const std::unordered_set &cppBuiltins(); +/** @return The rustBuiltins symbol table (lazily built, never mutated). */ +const std::unordered_set &rustBuiltins(); +/** @return The goBuiltins symbol table (lazily built, never mutated). */ +const std::unordered_set &goBuiltins(); +/** @return The javaBuiltins symbol table (lazily built, never mutated). */ +const std::unordered_set &javaBuiltins(); +/** @return The jsBuiltins symbol table (lazily built, never mutated). */ +const std::unordered_set &jsBuiltins(); + +} // namespace ir + +#endif // BUILTIN_REGISTRY_TABLES_H diff --git a/engine/src/ir/ir_translator.cpp b/engine/src/ir/ir_translator.cpp index b5da91c..5634dcc 100644 --- a/engine/src/ir/ir_translator.cpp +++ b/engine/src/ir/ir_translator.cpp @@ -1,5 +1,7 @@ #include "ir_translator.h" +#include + #include "translators/js_visitor.h" #include "translators/ts_visitor.h" #include "translators/tsx_visitor.h" @@ -9,8 +11,10 @@ #include "translators/python_visitor.h" #include "translators/rust_visitor.h" #include "translators/java_visitor.h" -// Swift translator excluded: Swift grammar not in FetchContent build -// #include "translators/swift_visitor.h" +// Swift has no visitor here: its grammar is not vendored (see +// parser/parser.cpp for the ABI reason), so a .swift file never reaches this +// layer. The Swift visitor/translator sources were removed rather than left +// unreachable — restore them together with the grammar. // Forward-declare concrete translators (implemented in translators/ dir) // Each returns a new Translator* or nullptr if the grammar can't be loaded. @@ -26,7 +30,6 @@ std::unique_ptr createJavascriptTranslator(); std::unique_ptr createTypescriptTranslator(); std::unique_ptr createGoTranslator(); std::unique_ptr createJavaTranslator(); -std::unique_ptr createSwiftTranslator(); std::unique_ptr createTsxTranslator(); } // namespace ir @@ -59,18 +62,34 @@ std::unique_ptr createTranslator(const char *language) return createGoTranslator(); if (lang == "java") return createJavaTranslator(); - // Swift translator temporarily disabled: Swift grammar was removed - // from FetchContent due to ABI incompatibility with core v0.24.7. - // The Swift translator source files are excluded from the build - // (not in ENGINE_SOURCES in CMakeLists.txt). - // if (lang == "swift") - // return createSwiftTranslator(); + // No Swift branch: the grammar is not vendored (parser/parser.cpp) and + // the translator was removed with it. Swift files are detected by + // extension and reported as `language_missing` parse failures instead. if (lang == "tsx") return createTsxTranslator(); return nullptr; } +bool TranslateDepth::exceeded(const char *file_path, const char *method) +{ + if (depth_ < kMaxTranslateDepth) + return false; + if (!reported_) { + // Reported once per file: a truncated translation must be visible + // (plan/rules/code_rules.md, no silent handling), and the message + // names the module and method so it can be traced back here. + reported_ = true; + fprintf(stderr, + "[module=ir, method=%s] AST nesting exceeded " + "kMaxTranslateDepth=%d in '%s'; deeper nodes skipped to " + "avoid stack overflow (legacy translator)\n", + method ? method : "translateChildren", + kMaxTranslateDepth, file_path ? file_path : ""); + } + return true; +} + std::unique_ptr createJsVisitor(const char *language) { // Guard against null language pointer @@ -99,9 +118,7 @@ std::unique_ptr createJsVisitor(const char *language) return std::make_unique(); if (lang == "java") return std::make_unique(); - // Swift visitor temporarily disabled - // if (lang == "swift") - // return std::make_unique(); + // No Swift branch: see the note at the top of this file. return nullptr; } diff --git a/engine/src/ir/ir_translator.h b/engine/src/ir/ir_translator.h index 1880282..9ad4893 100644 --- a/engine/src/ir/ir_translator.h +++ b/engine/src/ir/ir_translator.h @@ -17,6 +17,80 @@ class JsVisitor; // Pure function: source tree → IR TranslationUnit. // No resolver, no graph, no DB — thread-safe, each call is independent. +// ─── Legacy-translator recursion bound ─────────────────────────── +// The LIVE IR pipeline is the visitors (`*_visitor.cpp`), which bound their +// recursion with kMaxVisitDepth and report the truncation. The translators +// (`*_translator.cpp`) are the fallback used when no visitor exists for a +// language, and their tree walk is just as recursive: a pathologically deep +// AST would overflow the indexer's 512 KB worker stack, and the FFI try/catch +// cannot recover a stack overflow. They share the visitors' bound so the two +// pipelines cannot drift apart. +constexpr int kMaxTranslateDepth = 250; + +/// Per-file recursion counter for a translator. +/// +/// Not thread-safe by design: one instance belongs to one translator, and a +/// translator instance is used by a single parse worker at a time. +class TranslateDepth { + public: + /// Report whether the walk must stop at the current depth. The first stop + /// in a file is logged to stderr, tagged with the module and method, so a + /// truncated translation is never silent; later stops are suppressed to + /// keep one line per file. + /// \param file_path File being translated (for the report). + /// \param method Caller name (for the report). + /// \return true when the caller must return without recursing further. + bool exceeded(const char *file_path, const char *method); + + /// Enter one level. Paired with pop() (DepthGuard does both). + void push() + { + ++depth_; + } + /// Leave one level. + void pop() + { + --depth_; + } + /// Reset for a new file. + void reset() + { + depth_ = 0; + reported_ = false; + } + /// Current depth (tests and diagnostics). + int depth() const + { + return depth_; + } + + private: + int depth_ = 0; + bool reported_ = false; +}; + +/// RAII bracket around one recursion level: push on construction, pop on +/// destruction, so an early return inside the walk cannot leave the counter +/// skewed. +class DepthGuard { + public: + /// \param depth Counter owned by the calling translator. + explicit DepthGuard(TranslateDepth &depth) + : depth_(depth) + { + depth_.push(); + } + ~DepthGuard() + { + depth_.pop(); + } + DepthGuard(const DepthGuard &) = delete; + DepthGuard &operator=(const DepthGuard &) = delete; + + private: + TranslateDepth &depth_; +}; + class Translator { public: virtual ~Translator() = default; diff --git a/engine/src/ir/semantic_emitter.cpp b/engine/src/ir/semantic_emitter.cpp index 336cd5f..39e235a 100644 --- a/engine/src/ir/semantic_emitter.cpp +++ b/engine/src/ir/semantic_emitter.cpp @@ -91,24 +91,24 @@ uint64_t SemanticEmitter::emitImport(const std::string &module_name, loc); } -uint64_t SemanticEmitter::emitExport(const std::string &name, SourceRange loc, - uint64_t parent_id) +uint64_t SemanticEmitter::emitImportBinding(const std::string &binding, + const std::string &module_spec, + SourceRange loc, uint64_t parent_id) { - return unit_->addRecord(RecordKind::Export, name, parent_id, loc); + // A distinct kind from emitImport, deliberately: `Import` records the whole + // statement and is what the `import` table and the visitor tests count, so + // reusing it inflated those counts (the JS/TS visitor tests failed exactly + // that way, 1 import → 2). The binding sits in `name` and the specifier in + // `type_name`, which is what lets the Resolver map a bare call name back to + // its module. + return unit_->addTypedRecord(RecordKind::ImportBinding, binding, + module_spec, parent_id, loc); } -// ── Scope Emitters ─────────────────────────────────────────── - -void SemanticEmitter::emitScope(int kind, int scope_kind, - const std::string &name, SourceRange loc) +uint64_t SemanticEmitter::emitExport(const std::string &name, SourceRange loc, + uint64_t parent_id) { - // Scope entries/exits are stored as records with special kinds. - // kind=0 means scope enter, kind=1 means scope exit. - // The scope_kind is encoded in the arity field. - // scope_kind: 0=Global, 1=Module, 2=Function, 3=Block, 4=Trait, 5=Impl - RecordKind rk = (kind == 0) ? RecordKind::TranslationUnit : - RecordKind::Comment; - unit_->addRecord(rk, name, 0, loc, scope_kind, false); + return unit_->addRecord(RecordKind::Export, name, parent_id, loc); } // ── Reference Emitter ──────────────────────────────────────── diff --git a/engine/src/ir/semantic_emitter.h b/engine/src/ir/semantic_emitter.h index c5c08d8..6a37356 100644 --- a/engine/src/ir/semantic_emitter.h +++ b/engine/src/ir/semantic_emitter.h @@ -58,16 +58,31 @@ class SemanticEmitter { uint64_t emitImport(const std::string &module_name, SourceRange loc, uint64_t parent_id = 0); + + /// Emit one imported local binding together with the module it was + /// imported FROM. + /// + /// `emitImport` records the raw import statement, and the `import` table + /// built from it derives its `alias` column from the module path's last + /// segment — so for `import { widgetHelper } from "../lib/Widget"` the + /// table says `alias="Widget"`, not `"widgetHelper"`. A bare call + /// (`widgetHelper()`) therefore has no key to look its module up by, which + /// is exactly the evidence a JS/TS cross-directory call needs: the + /// language has no receiver to match on and `import_alias` only records + /// names that ARE imported, not where from. This emits the pair + /// explicitly, as `name = binding, type_name = module_spec`. + /// + /// \param binding Local name the module was bound to. + /// \param module_spec Module specifier as written ("./helper"). + /// \param loc Source location of the import statement. + /// \param parent_id Parent record ID; 0 for a file-level import. + /// \return The new record's ID. + uint64_t emitImportBinding(const std::string &binding, + const std::string &module_spec, + SourceRange loc, uint64_t parent_id = 0); uint64_t emitExport(const std::string &name, SourceRange loc, uint64_t parent_id = 0); - // ── Scope Emitters ───────────────────────────────────────── - - /// Emit a scope entry/exit. kind: 0=enter, 1=exit. - /// scope_kind: 0=Global, 1=Module, 2=Function, 3=Block, 4=Trait, 5=Impl - void emitScope(int kind, int scope_kind, const std::string &name, - SourceRange loc); - // ── Reference Emitter ────────────────────────────────────── /// Emit a call reference (no resolution). diff --git a/engine/src/ir/semantic_unit.h b/engine/src/ir/semantic_unit.h index 41e7c50..70be857 100644 --- a/engine/src/ir/semantic_unit.h +++ b/engine/src/ir/semantic_unit.h @@ -72,6 +72,16 @@ enum class RecordKind : uint8_t { /// The `name` field stores the implementing type name. /// The `type_name` field stores the interface/trait name. InterfaceImpl, + /// Import binding: the local name a module was bound to, and the module. + /// The `name` field stores the local binding, `type_name` the module + /// specifier as written ("./helper"). Distinct from `Import`, which records + /// the whole import STATEMENT — that is what the `import` table is built + /// from and what the visitor tests count, and a statement with three + /// bindings produces one `Import` plus three `ImportBinding` records. The + /// store derives the `import` table's `alias` from the module path's last + /// segment, so it cannot express this pair; the Resolver reads it from here + /// to tie a bare call to the module it came from. + ImportBinding, }; /// Kind of a call expression — used to distinguish direct calls from @@ -84,6 +94,10 @@ enum class CallKind : uint8_t { Constructor = 3, // constructor/factory call: NewService() StaticMethod = 4, // static method call: ClassName.staticMethod() (Java/C++) + // Not emitted by any visitor yet: Java records `Class.m()` as + // Method through its receiver check (java_visitor.cpp), and that is + // the conservative direction — the resolver's fuzzy fallback only + // accepts CallKind::Direct, so a static call is never guessed. Virtual = 5, // virtual dispatch / trait method (C++/Rust) }; @@ -128,7 +142,7 @@ struct Record { /// Populated by Visitors per language; flows Record → SemanticUnit → /// entity.visibility column via the staging pipeline. The role classifier /// in state_builder.cpp fuses pub_count (visibility=1) with call-graph - /// counts — see docs/dev_plans/role_classifier_plan.md. + /// counts. int visibility = 0; // ── Call fact fields (Step 3, plan §3.1) ──────────────────────── diff --git a/engine/src/ir/translators/c_translator.cpp b/engine/src/ir/translators/c_translator.cpp index a3e7f74..ecf4173 100644 --- a/engine/src/ir/translators/c_translator.cpp +++ b/engine/src/ir/translators/c_translator.cpp @@ -23,6 +23,8 @@ class CTranslator : public Translator { TranslationUnit *unit_ = nullptr; const char *source_ = nullptr; std::string file_path_; + // Recursion counter for this file (see kMaxTranslateDepth). + TranslateDepth depth_; struct Scope { std::unordered_map symbols; @@ -62,7 +64,6 @@ class CTranslator : public Translator { Node *handleFuncDef(TSNode ts_node, Node *parent); Node *handleDeclaration(TSNode ts_node, Node *parent); - Node *handleAttributedDeclarator(TSNode ts_node, Node *parent); Node *handleStruct(TSNode ts_node, Node *parent, bool is_union); Node *handleCallExpr(TSNode ts_node, Node *parent); Node *handleIdentifier(TSNode ts_node, Node *parent); @@ -84,6 +85,7 @@ class CTranslator : public Translator { TranslationUnit *CTranslator::translate(TSTree *tree, const char *source, const char *file_path) { + depth_.reset(); unit_ = new TranslationUnit(); unit_->source_content = source; source_ = source; @@ -360,6 +362,13 @@ Node *CTranslator::translateNode(TSNode ts_node, Node *parent) void CTranslator::translateChildren(TSNode ts_node, Node *parent) { + // Bound native-stack recursion (see kMaxTranslateDepth): a pathologically + // deep AST would overflow the indexer's 512 KB worker stack, which the FFI + // try/catch cannot recover. The visitors apply the same bound. + if (depth_.exceeded(file_path_.c_str(), "translateChildren")) + return; + DepthGuard depth_guard(depth_); + uint32_t count = ts_node_child_count(ts_node); for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(ts_node, i); @@ -568,43 +577,6 @@ Node *CTranslator::handleParamDecl(TSNode ts_node, Node *parent) return param; } -Node *CTranslator::handleAttributedDeclarator(TSNode ts_node, Node *parent) -{ - // Attributed declarators wrap a function_declarator with GNU C - // __attribute__((...)). Walk children to find the actual declarator - // and, if present with a compound_statement sibling, treat as a - // function definition. - uint32_t count = ts_node_child_count(ts_node); - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(ts_node, i); - if (!ts_node_is_named(child)) - continue; - if (strcmp(ts_node_type(child), "function_declarator") == 0) { - // Check parent (the enclosing declaration) for a body - TSNode p = ts_node_parent(ts_node); - if (!ts_node_is_null(p)) { - uint32_t pc = ts_node_child_count(p); - for (uint32_t j = 0; j < pc; j++) { - TSNode sib = ts_node_child(p, j); - if (!ts_node_is_named(sib)) - continue; - if (strcmp(ts_node_type(sib), - "compound_statement") == 0) - return handleFuncDef(p, parent); - } - } - // No body found — treat as a declaration - auto *func = makeNode(NodeKind::FunctionDecl, child); - func->name = nodeText(child); - parent->children.push_back(func); - return func; - } - } - // Fallback: recurse into children - translateChildren(ts_node, parent); - return nullptr; -} - Node *CTranslator::handlePreprocDef(TSNode ts_node, Node *parent) { auto *macro = makeNode(NodeKind::MacroDecl, ts_node); diff --git a/engine/src/ir/translators/c_visitor.cpp b/engine/src/ir/translators/c_visitor.cpp index c07ebf1..b84ecf8 100644 --- a/engine/src/ir/translators/c_visitor.cpp +++ b/engine/src/ir/translators/c_visitor.cpp @@ -110,6 +110,15 @@ SemanticUnit *CVisitor::visit(TSTree *tree, const char *source, class_scope_stack_.clear(); TSNode root_node = ts_tree_root_node(tree); + // Names this file defines, so a locally declared `free`/`assert` ... is not + // dropped as a stdlib call. + defined_names_.clear(); + collectDefinedNames(root_node); + // Out-of-class definitions present in this file (see + // out_of_class_defs_): a class-body declaration for one of them must not + // be emitted a second time. + out_of_class_defs_.clear(); + collectOutOfClassDefs(root_node); pushScope(); SourceRange root_loc = location(root_node); uint64_t root_id = emitter_->emitVariable("", root_loc, 0); @@ -160,6 +169,7 @@ void CVisitor::handleFuncDef(TSNode node, uint64_t parent_id) } uint64_t id = emitter_->emitFunction(name, loc, parent_id, 0, false, detectVisibility(node)); + recordParameterTypes(node); defineSymbol(name, id); // Step 4/5 (plan §4C/§5): tag methods with a qualified name so the // Resolver's factorReceiverTypeMatch can match a call's receiver_type @@ -197,6 +207,45 @@ void CVisitor::handleFuncDef(TSNode node, uint64_t parent_id) popScope(); } +void CVisitor::recordParameterTypes(TSNode func_def) +{ + // The parameter list hangs off the declarator, one level below the + // definition — the same reason extractName() reads the `declarator` field: + // the first named child of a function_definition is the return type, so + // scanning the definition's children reaches the return type instead. + // + // Only named parameters are recorded (`void f(int)` binds nothing), and + // extractName() unwraps the declarator for the parameter, so `*o` and `&o` + // both yield `o`. + const uint32_t count = ts_node_child_count(func_def); + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(func_def, i); + if (!ts_node_is_named(child) || + strcmp(ts_node_type(child), "function_declarator") != 0) + continue; + TSNode params = + ts_node_child_by_field_name(child, "parameters", 10); + if (ts_node_is_null(params)) + continue; + const uint32_t param_count = ts_node_child_count(params); + for (uint32_t p = 0; p < param_count; p++) { + TSNode param = ts_node_child(params, p); + if (!ts_node_is_named(param) || + strcmp(ts_node_type(param), + "parameter_declaration") != 0) + continue; + TSNode type_node = + ts_node_child_by_field_name(param, "type", 4); + if (ts_node_is_null(type_node)) + continue; + const std::string param_name = extractName(param); + if (param_name.empty()) + continue; + recordVarType(param_name, nodeText(type_node)); + } + } +} + void CVisitor::handleDeclaration(TSNode node, uint64_t parent_id) { // Step 4 (plan §4C): extract variable → type bindings from @@ -354,8 +403,17 @@ void CVisitor::handleCall(TSNode node, uint64_t parent_id) // Skip C compiler builtins and common stdlib functions — they are NOT // user-defined calls and the Resolver Pipeline would generate // false-positive edges by matching them to entities with the same name. + // + // NOTE (deliberate, unlike the Go/Java/Python/JS visitors): this also + // filters QUALIFIED calls (`ops->free(x)`, `Util::clone()`), whose last + // segment is the bare `name`. The C list contains very common stdlib + // names (malloc/free/memcpy/printf/exit/assert) and C's idiomatic + // vtable-style structs name callback fields exactly that way, while the + // resolver has no receiver-type evidence for a field call. Emitting a + // record would therefore re-open the false-positive flood this filter + // exists to prevent; dropping the call is the conservative choice. // Reference: codebase-memory-mcp (MIT) c_lsp.c :: is_c_builtin_func() - if (!name.empty() && isCBuiltin(name)) { + if (!name.empty() && isCBuiltin(name) && !isLocallyDefined(name)) { // Still visit children to pick up nested calls/expressions for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(node, i); @@ -366,7 +424,7 @@ void CVisitor::handleCall(TSNode node, uint64_t parent_id) if (strcmp(ts_node_type(child), "argument_list") == 0) visitChildren(child, parent_id); else - visitNode(child, parent_id); + visitChild(child, parent_id); } return; } @@ -456,7 +514,7 @@ void CVisitor::handleCall(TSNode node, uint64_t parent_id) if (strcmp(ts_node_type(child), "argument_list") == 0) visitChildren(child, id); else - visitNode(child, id); + visitChild(child, id); } } @@ -486,7 +544,7 @@ void CVisitor::handleNewExpr(TSNode node, uint64_t parent_id) } // Skip C compiler builtins / common stdlib — NOT user-defined calls. - if (!name.empty() && isCBuiltin(name)) { + if (!name.empty() && isCBuiltin(name) && !isLocallyDefined(name)) { for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(node, i); if (!ts_node_is_named(child)) @@ -494,7 +552,7 @@ void CVisitor::handleNewExpr(TSNode node, uint64_t parent_id) if (strcmp(ts_node_type(child), "argument_list") == 0) visitChildren(child, parent_id); else - visitNode(child, parent_id); + visitChild(child, parent_id); } return; } @@ -538,7 +596,7 @@ void CVisitor::handleNewExpr(TSNode node, uint64_t parent_id) if (strcmp(t, "argument_list") == 0) visitChildren(child, id); else - visitNode(child, id); + visitChild(child, id); } } @@ -586,6 +644,43 @@ void CVisitor::handlePreprocDef(TSNode node, uint64_t parent_id) std::string CVisitor::extractName(TSNode node) { + // Read the `declarator` FIELD first instead of scanning children in order. + // + // In a function_definition the first named child is the return type, and for + // a qualified return type (`std::string`, `nlohmann::json`, `Status::Code`) + // tree-sitter-cpp parses that type as a qualified_identifier. The scan below + // then matched the TYPE before the declarator, recursed into it, found only + // namespace_identifier / type_identifier — neither of which any branch + // accepts — and returned "" for the whole definition. handleFuncDef treats + // an empty name as "no name" and drops the record, so every function + // returning a qualified type was invisible to the index: the entire + // QueryEngine surface (all of whose methods return std::string JSON) plus + // the std::string-returning store/graph accessors. Variables declared with a + // qualified type (`std::string name;`) were dropped the same way. + // + // The declarator field is the same node the old branches reached for the + // cases that did work, so nothing regresses. + TSNode declarator = ts_node_child_by_field_name(node, "declarator", 10); + if (!ts_node_is_null(declarator)) { + // The declarator is often the name node itself (a free function, or the + // innermost level of a pointer/function declarator chain). The scan + // below inspects CHILDREN, so recursing into a bare identifier would + // find no children and return "" — which silently dropped every free + // function definition. Read the text directly when the declarator + // already is the name. + // + // operator_name / destructor_name are name nodes too (`operator==`, + // `~Point`): they have no children, so recursing into them returned "" + // and the whole definition was dropped. + const char *dt = ts_node_type(declarator); + if (strcmp(dt, "identifier") == 0 || + strcmp(dt, "field_identifier") == 0 || + strcmp(dt, "operator_name") == 0 || + strcmp(dt, "destructor_name") == 0) + return nodeText(declarator); + return extractName(declarator); + } + uint32_t count = ts_node_child_count(node); for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(node, i); @@ -603,6 +698,15 @@ std::string CVisitor::extractName(TSNode node) // emitting a Function record or calling defineSymbol). if (strcmp(t, "field_identifier") == 0) return nodeText(child); + // operator_name / destructor_name: `bool operator==(...)`, + // `~Point()`. tree-sitter-cpp parses the overloaded operator as an + // operator_name node ("operator==") and a destructor as a + // destructor_name node ("~Point"); neither is an identifier or + // field_identifier, so both were previously skipped and the + // definition emitted with an empty name (dropped as a Variable). + if (strcmp(t, "operator_name") == 0 || + strcmp(t, "destructor_name") == 0) + return nodeText(child); // qualified_identifier: out-of-class member function // definitions like "int64_t GraphStore::buildCallEdgesSQL( // uint64_t pid) { ... }". tree-sitter-cpp parses the @@ -636,6 +740,30 @@ std::string CVisitor::extractName(TSNode node) return ""; } +void CVisitor::collectOutOfClassDefs(TSNode node, int depth) +{ + // Bound the recursion with the same cap as the other walks: this scan runs + // over the whole tree before the traversal, so an unbounded deep AST would + // overflow the stack here too. It does not report the truncation itself — + // collectDefinedNames walks the same tree with the same cap and runs first, + // so the file is already covered by the once-per-file report ([module=ir, + // method=collectDefinedNames]). + if (depth >= kMaxVisitDepth) + return; + uint32_t cnt = ts_node_child_count(node); + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(node, i); + if (!ts_node_is_named(c)) + continue; + if (strcmp(ts_node_type(c), "function_definition") == 0) { + std::string q = extractQualifiedName(c); + if (!q.empty()) + out_of_class_defs_.insert(std::move(q)); + } + collectOutOfClassDefs(c, depth + 1); + } +} + std::string CVisitor::extractQualifiedName(TSNode node) { // Walk the function_definition's declarator chain for a @@ -644,31 +772,72 @@ std::string CVisitor::extractQualifiedName(TSNode node) // identifier (scope), "::" (unnamed), field_identifier (name). // Returns "" when no qualified scope is present; the caller then falls // back to currentClassName() for in-class methods. + // + // Read the `declarator` field first, for the same reason as extractName(): + // the first named child of a function_definition is the return type, and a + // qualified return type (`std::string`) is itself a qualified_identifier. + // Scanning children in order used to hit that type first and return "" — so + // `Scope::method` definitions were registered without their scope whenever + // the return type happened to be qualified. The declarator never contains + // the return type. + // Collect every name segment of the qualified_identifier in source + // order; the last one is the method, the rest is its scope. + // + // tree-sitter-cpp mixes node types inside one qualified_identifier: + // `GraphStore::buildCallEdgesSQL` is namespace_identifier + identifier, + // `Point::operator==` is type_identifier + operator_name, and a nested + // `a::b::c` adds a namespace_identifier per level. The scope is NOT + // always an `identifier` — matching only `identifier` (the original + // code) returned an empty qualified name for every out-of-class + // definition. Taking the trailing segment handles all of those shapes. + auto nameOf = [this](TSNode qualified) -> std::string { + std::vector parts; + uint32_t qc = ts_node_child_count(qualified); + for (uint32_t j = 0; j < qc; j++) { + TSNode q = ts_node_child(qualified, j); + if (!ts_node_is_named(q)) + continue; + const char *qt = ts_node_type(q); + if (strcmp(qt, "identifier") == 0 || + strcmp(qt, "type_identifier") == 0 || + strcmp(qt, "namespace_identifier") == 0 || + strcmp(qt, "field_identifier") == 0 || + strcmp(qt, "operator_name") == 0 || + strcmp(qt, "destructor_name") == 0) + parts.push_back(nodeText(q)); + } + if (parts.size() < 2) + return std::string(); + std::string out; + for (size_t i = 0; i + 1 < parts.size(); i++) { + if (!out.empty()) + out += "::"; + out += parts[i]; + } + out += "::"; + out += parts.back(); + return out; + }; + + // The node reached through the declarator chain may itself BE the + // qualified_identifier. It has no declarator field, so recursing past it + // would return "" and silently drop the scope of every out-of-class + // definition — check for it before descending. + if (strcmp(ts_node_type(node), "qualified_identifier") == 0) + return nameOf(node); + + TSNode declarator = ts_node_child_by_field_name(node, "declarator", 10); + if (!ts_node_is_null(declarator)) + return extractQualifiedName(declarator); + uint32_t count = ts_node_child_count(node); for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(node, i); if (!ts_node_is_named(child)) continue; const char *t = ts_node_type(child); - if (strcmp(t, "qualified_identifier") == 0) { - std::string scope; - std::string method; - uint32_t qc = ts_node_child_count(child); - for (uint32_t j = 0; j < qc; j++) { - TSNode q = ts_node_child(child, j); - if (!ts_node_is_named(q)) - continue; - const char *qt = ts_node_type(q); - if (strcmp(qt, "identifier") == 0 && - scope.empty()) - scope = nodeText(q); - else if (strcmp(qt, "field_identifier") == 0) - method = nodeText(q); - } - if (!scope.empty() && !method.empty()) - return scope + "::" + method; - return ""; - } + if (strcmp(t, "qualified_identifier") == 0) + return nameOf(child); // Recurse into declarator wrappers that may contain the // qualified_identifier (function_declarator, pointer_declarator). if (strcmp(t, "function_declarator") == 0 || diff --git a/engine/src/ir/translators/c_visitor.h b/engine/src/ir/translators/c_visitor.h index 0516995..b3724fb 100644 --- a/engine/src/ir/translators/c_visitor.h +++ b/engine/src/ir/translators/c_visitor.h @@ -4,6 +4,7 @@ #include "js_visitor.h" #include #include +#include #include namespace ir @@ -36,6 +37,18 @@ class CVisitor : public JsVisitor { var_types_[name] = type; } + /// Record every named parameter of a function definition into var_types_. + /// + /// A parameter is a declaration too, and handleCall fills receiver_type for + /// `o->method()` from var_types_. Only `declaration` nodes fed it, so a + /// parameter's type was never known: the call fact went in with + /// receiver_text but an EMPTY receiver_type, and when two types declare the + /// same method name (the common `ToString`/`begin`/`clone` case) the + /// resolver had nothing to choose with and abstained — a false negative that + /// Go, Java and Python do not produce for the same call shape (measured in + /// test_resolver_language_consistency). + void recordParameterTypes(TSNode func_def); + /// Push/pop the enclosing class name for this->method() inference. void pushClassScope(const std::string &class_name) { @@ -54,6 +67,48 @@ class CVisitor : public JsVisitor { return class_scope_stack_.back(); } + /// Extract the declared name from a declaration/definition node + /// (function_definition, field_declaration, init_declarator, ...). + /// Walks the `declarator` field so a qualified return type does not + /// shadow the name, and handles function/pointer/array declarators, + /// qualified names, operator overloads and destructors. + /// \param node The declaration node. + /// \return The declared name, or "" when the node declares none. + /// Protected rather than private so CppVisitor can reuse it for + /// in-class method DECLARATIONS (`field_declaration`). + std::string extractName(TSNode node); + + /// Extract a scope-qualified name from an out-of-class member function + /// definition (e.g. `int64_t GraphStore::buildCallEdgesSQL(...)` → + /// "GraphStore::buildCallEdgesSQL"). Walks the function_definition's + /// declarator chain to find a `qualified_identifier` and concatenates + /// its scope identifier and field identifier. Returns "" when the + /// definition is not scope-qualified (in-class methods use + /// currentClassName() at the call site instead). + /// \param node The function_definition node. + /// \return "Scope::name" or "" if no qualified scope is present. + std::string extractQualifiedName(TSNode node); + + /// Qualified names ("Scope::name") of every out-of-class member function + /// DEFINED in the file being visited, filled once per visit(). + /// + /// A class-body `field_declaration` (a method DECLARATION such as + /// `void foo();`) and the out-of-class definition of the same method are + /// two records with the same bare name and arity when they sit in one + /// file. Emitting both created a homonym pair that ties on every Resolver + /// factor, so the Step-5 ambiguity gate abstained and the CALLS edge was + /// LOST — verified by test_qualified_id_ast (buildGraph → + /// buildCallEdgesSQL). CppVisitor therefore skips a declaration whose + /// definition is already in this file; the definition represents it. + std::unordered_set out_of_class_defs_; + /// Collect out_of_class_defs_ from a subtree (see the member comment). + /// Recursion is bounded by kMaxVisitDepth: this scan runs over the whole + /// tree before the traversal, so a pathologically deep AST would otherwise + /// overflow the stack here. + /// \param node Subtree to scan. + /// \param depth Current recursion depth (0 at the root). + void collectOutOfClassDefs(TSNode node, int depth = 0); + private: void handleFuncDef(TSNode node, uint64_t parent_id); void handleDeclaration(TSNode node, uint64_t parent_id); @@ -67,18 +122,6 @@ class CVisitor : public JsVisitor { void handleInclude(TSNode node, uint64_t parent_id); void handleTypeDef(TSNode node, uint64_t parent_id); void handlePreprocDef(TSNode node, uint64_t parent_id); - std::string extractName(TSNode node); - - /// Extract a scope-qualified name from an out-of-class member function - /// definition (e.g. `int64_t GraphStore::buildCallEdgesSQL(...)` → - /// "GraphStore::buildCallEdgesSQL"). Walks the function_definition's - /// declarator chain to find a `qualified_identifier` and concatenates - /// its scope identifier and field identifier. Returns "" when the - /// definition is not scope-qualified (in-class methods use - /// currentClassName() at the call site instead). - /// \param node The function_definition node. - /// \return "Scope::name" or "" if no qualified scope is present. - std::string extractQualifiedName(TSNode node); /// Extract the method name from a field_expression callee. /// For "a.adder(...)" the field_expression's children are: diff --git a/engine/src/ir/translators/cpp_translator.cpp b/engine/src/ir/translators/cpp_translator.cpp index 8b734ad..1213e29 100644 --- a/engine/src/ir/translators/cpp_translator.cpp +++ b/engine/src/ir/translators/cpp_translator.cpp @@ -23,6 +23,8 @@ class CppTranslator : public Translator { TranslationUnit *unit_ = nullptr; const char *source_ = nullptr; std::string file_path_; + // Recursion counter for this file (see kMaxTranslateDepth). + TranslateDepth depth_; struct Scope { std::unordered_map symbols; @@ -82,6 +84,7 @@ class CppTranslator : public Translator { TranslationUnit *CppTranslator::translate(TSTree *tree, const char *source, const char *file_path) { + depth_.reset(); unit_ = new TranslationUnit(); unit_->source_content = source; source_ = source; @@ -397,6 +400,13 @@ Node *CppTranslator::translateNode(TSNode ts_node, Node *parent) void CppTranslator::translateChildren(TSNode ts_node, Node *parent) { + // Bound native-stack recursion (see kMaxTranslateDepth): a pathologically + // deep AST would overflow the indexer's 512 KB worker stack, which the FFI + // try/catch cannot recover. The visitors apply the same bound. + if (depth_.exceeded(file_path_.c_str(), "translateChildren")) + return; + DepthGuard depth_guard(depth_); + uint32_t count = ts_node_child_count(ts_node); for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(ts_node, i); diff --git a/engine/src/ir/translators/cpp_visitor.cpp b/engine/src/ir/translators/cpp_visitor.cpp index 76d9ae5..ca2864e 100644 --- a/engine/src/ir/translators/cpp_visitor.cpp +++ b/engine/src/ir/translators/cpp_visitor.cpp @@ -15,8 +15,24 @@ SemanticUnit *CppVisitor::visit(TSTree *tree, const char *source, unit_->setFilePath(fp); unit_->setLanguage("cpp"); source_ = source; + // Step 4: reset per-file tracking so the visitor arena can reuse this + // visitor across files without leaking the previous file's bindings. + var_types_.clear(); + class_scope_stack_.clear(); TSNode root_node = ts_tree_root_node(tree); + // Names this file defines, so a user function that happens to share a + // C/C++ builtin name (`void free(void*)`, a local `format`) is not dropped + // by handleCall's builtin filter. CVisitor::visit does this too, but this + // override never called through it, so .cpp files had an empty + // defined_names_ and the exemption never applied. + defined_names_.clear(); + collectDefinedNames(root_node); + // Out-of-class definitions present in this file — a class-body + // declaration of one of them must not be emitted twice (see + // CVisitor::out_of_class_defs_). + out_of_class_defs_.clear(); + collectOutOfClassDefs(root_node); pushScope(); SourceRange root_loc = location(root_node); uint64_t root_id = emitter_->emitVariable("", root_loc, 0); @@ -36,8 +52,102 @@ void CppVisitor::visitNode(TSNode node, uint64_t parent_id) return handleNamespace(node, parent_id); if (strcmp(type, "template_declaration") == 0) return handleTemplate(node, parent_id); + if (strcmp(type, "field_declaration") == 0) + return handleMemberFunctionDecl(node, parent_id); + // NOTE: a class-body destructor/constructor DECLARATION (`~Point();`) + // parses as a `declaration`, not a field_declaration, and is routed to + // handleMemberFunctionDecl by visitClassBody() — which is the only place + // that can tell a class member from a declaration inside a member + // function's body. Handling it here on `currentClassName()` alone (as an + // earlier revision did) also matched local function declarations inside + // methods and emitted phantom `Class::local` entities. CVisitor::visitNode(node, parent_id); } + +void CppVisitor::visitClassBody(TSNode list, uint64_t parent_id) +{ + // Walks a class body's field_declaration_list. This is the ONLY place + // where a bare `declaration` node belongs to the class rather than to a + // function body nested inside it: `~Point();` and `Point();` parse that + // way (declarator + parameter_list, no body), while a local + // `int helper(int);` inside a method is a declaration nested deeper. + uint32_t cnt = ts_node_child_count(list); + for (uint32_t i = 0; i < cnt; i++) { + TSNode child = ts_node_child(list, i); + if (!ts_node_is_named(child)) + continue; + if (strcmp(ts_node_type(child), "declaration") == 0 && + declaresFunction(child)) { + handleMemberFunctionDecl(child, parent_id); + continue; + } + visitNode(child, parent_id); + } +} + +bool CppVisitor::declaresFunction(TSNode node) +{ + // A member that DECLARES a function carries a parameter list. Most forms + // wrap the name in a function_declarator (`void foo();`), but the + // special member forms do not — `~Point();` / `Point();` parse as a + // declaration whose function_declarator holds a bare destructor_name / + // identifier plus the parameter list. A member without either + // (`int x;`, `std::string name;`) is plain data. + uint32_t cnt = ts_node_child_count(node); + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(node, i); + if (!ts_node_is_named(c)) + continue; + const char *t = ts_node_type(c); + if (strcmp(t, "function_declarator") == 0 || + strcmp(t, "parameter_list") == 0) + return true; + } + return false; +} + +void CppVisitor::handleMemberFunctionDecl(TSNode node, uint64_t parent_id) +{ + // In-class method DECLARATIONS parse as field_declaration members of the + // class body's field_declaration_list — `void foo();`, + // `bool operator==(const Point&) const;`, `~Point();`. Without a handler + // they fell through to the generic recursion and were recorded as + // Variables (the declarator's field_identifier), so the class's method + // surface was missing from the graph and calls to those methods could + // never resolve to a declaration in this file. + // + // A data member (`int x;`, `std::string name;`) keeps the old behaviour + // of being recursed into for its variable/type records. + if (!declaresFunction(node)) { + visitChildren(node, parent_id); + return; + } + + std::string name = extractName(node); + if (name.empty()) { + visitChildren(node, parent_id); + return; + } + // When this file also holds the out-of-class definition of the same + // method, emitting the declaration too would create two same-name, + // same-arity candidates in one file. The Resolver's Step-5 ambiguity + // gate abstains on such a tie, so the CALLS edge to the method would be + // LOST (test_qualified_id_ast: buildGraph → buildCallEdgesSQL). The + // definition already represents the method — skip the duplicate. + std::string cls = currentClassName(); + if (!cls.empty() && out_of_class_defs_.count(cls + "::" + name) != 0) { + visitChildren(node, parent_id); + return; + } + SourceRange loc = location(node); + uint64_t id = emitter_->emitMethod(name, loc, parent_id); + defineSymbol(name, id); + if (!cls.empty()) + unit_->setQualifiedName(id, cls + "::" + name); + // No body — nothing to enter a function scope for. Still recurse so + // parameter type references and default arguments are visited. + visitChildren(node, parent_id); +} void CppVisitor::handleClassSpec(TSNode node, uint64_t parent_id) { SourceRange loc = location(node); @@ -72,11 +182,10 @@ void CppVisitor::handleClassSpec(TSNode node, uint64_t parent_id) // `field_declaration_list`, NOT `class_body` (the latter is // the Java grammar). The previous `class_body` check never // matched, so all C++ class fields were silently dropped. - // See CODE_REVIEW_FINDINGS_2026-07-19.md H8. if (strcmp(t, "field_declaration_list") == 0) - visitChildren(c, id); + visitClassBody(c, id); else - visitNode(c, id); + visitChild(c, id); } popClassScope(); popScope(); @@ -92,7 +201,7 @@ void CppVisitor::handleNamespace(TSNode node, uint64_t parent_id) strcmp(ts_node_type(c), "identifier") == 0) visitChildren(c, parent_id); else - visitNode(c, parent_id); + visitChild(c, parent_id); } } void CppVisitor::handleTemplate(TSNode node, uint64_t parent_id) @@ -104,7 +213,7 @@ void CppVisitor::handleTemplate(TSNode node, uint64_t parent_id) continue; if (strcmp(ts_node_type(c), "template_parameter_list") == 0) continue; - visitNode(c, parent_id); + visitChild(c, parent_id); } } } // namespace ir diff --git a/engine/src/ir/translators/cpp_visitor.h b/engine/src/ir/translators/cpp_visitor.h index 02d2fdb..74c7a58 100644 --- a/engine/src/ir/translators/cpp_visitor.h +++ b/engine/src/ir/translators/cpp_visitor.h @@ -16,6 +16,31 @@ class CppVisitor : public CVisitor { void handleClassSpec(TSNode node, uint64_t parent_id); void handleNamespace(TSNode node, uint64_t parent_id); void handleTemplate(TSNode node, uint64_t parent_id); + /// Emit a method DECLARATION/definition-less member from a class-body + /// `field_declaration` (`void foo();`, `bool operator==(...) const;`) + /// or from the `declaration` a destructor/constructor declaration + /// (`~Point();`, `Point();`) parses as. A member that declares no + /// function (a plain data member such as `int x;`) is only recursed + /// into, as before. + void handleMemberFunctionDecl(TSNode node, uint64_t parent_id); + + /// Walk one class body (`field_declaration_list`). + /// + /// Exists so a bare `declaration` node is only treated as a member + /// declaration when it really is a DIRECT member of the class: a local + /// function declaration inside a method body is a declaration too, and + /// dispatching on the enclosing class name alone turned it into a + /// phantom `Class::local` entity. + /// + /// \param list The class body node. + /// \param parent_id Record id of the class. + void visitClassBody(TSNode list, uint64_t parent_id); + + /// True when `node` declares a function: a direct named child is a + /// `function_declarator` (most forms) or a `parameter_list` (the + /// special member forms). False for a plain data member. + /// \param node A `field_declaration` or `declaration` node. + bool declaresFunction(TSNode node); }; } // namespace ir #endif diff --git a/engine/src/ir/translators/go_translator.cpp b/engine/src/ir/translators/go_translator.cpp index ade4637..9aebc5f 100644 --- a/engine/src/ir/translators/go_translator.cpp +++ b/engine/src/ir/translators/go_translator.cpp @@ -24,6 +24,8 @@ class GoTranslator : public Translator { TranslationUnit *unit_ = nullptr; const char *source_ = nullptr; std::string file_path_; + // Recursion counter for this file (see kMaxTranslateDepth). + TranslateDepth depth_; struct Scope { std::unordered_map symbols; @@ -81,6 +83,7 @@ class GoTranslator : public Translator { TranslationUnit *GoTranslator::translate(TSTree *tree, const char *source, const char *file_path) { + depth_.reset(); unit_ = new TranslationUnit(); unit_->source_content = source; source_ = source; @@ -217,6 +220,13 @@ Node *GoTranslator::translateNode(TSNode ts_node, Node *parent) void GoTranslator::translateChildren(TSNode ts_node, Node *parent) { + // Bound native-stack recursion (see kMaxTranslateDepth): a pathologically + // deep AST would overflow the indexer's 512 KB worker stack, which the FFI + // try/catch cannot recover. The visitors apply the same bound. + if (depth_.exceeded(file_path_.c_str(), "translateChildren")) + return; + DepthGuard depth_guard(depth_); + uint32_t count = ts_node_child_count(ts_node); for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(ts_node, i); @@ -292,8 +302,7 @@ Node *GoTranslator::handleMethodDecl(TSNode ts_node, Node *parent) // — otherwise the second parameter_list's parameter types (e.g. // `int` in `func (r *MyType) Method(a int)`) overwrite receiver_type // → receiver edge points to `int` instead of `MyType`, breaking - // every Go method-to-type association. See - // CODE_REVIEW_FINDINGS_2026-07-19.md C4. + // every Go method-to-type association. if (strcmp(t, "parameter_list") == 0) { // First parameter_list is receiver uint32_t pc = ts_node_child_count(child); diff --git a/engine/src/ir/translators/go_visitor.cpp b/engine/src/ir/translators/go_visitor.cpp index 266d177..e5a2356 100644 --- a/engine/src/ir/translators/go_visitor.cpp +++ b/engine/src/ir/translators/go_visitor.cpp @@ -1,71 +1,38 @@ #include "go_visitor.h" #include -#include #include #include #include "../builtin_registry.h" namespace ir { -// ─── HTTP method constants for route detection ────────────────── -// Used to identify route registrations like r.GET("/path", handler). -// Reference: codebase-memory-mcp (MIT) service_patterns.c -static const char *kHttpMethods[] = { - "GET", "POST", "PUT", "DELETE", "PATCH", - "HEAD", "OPTIONS", "HandleFunc", "Handle", // net/http - nullptr, -}; - -// ─── Language-specific builtin detection ─────────────────────────── -// -// Reference: codebase-memory-mcp (MIT, https://github.com/DeusData/codebase-memory-mcp) -// internal/cbm/lsp/go_lsp.c :: is_go_builtin_func() -// -// Go built-in functions are language keywords that should NOT create -// reference entries — they are not user-defined functions and cannot be -// resolved to any project entity. Without this filter, every call to -// len(), append(), copy(), etc. generates a "Len"/"Append"/"Copy" -// reference that the Resolver Pipeline incorrectly matches to any -// project entity with the same name, producing massive false-positive -// cross-module call edges. -// -// The same pattern should be applied to every language visitor: -// filter out language builtins / keywords at the parser level so the -// resolver never sees them. - -static bool isGoBuiltin(const std::string &name) +namespace { - // Complete list of Go built-in functions (per language spec). - // All of these are predeclared identifiers that cannot be - // redefined and never resolve to user-defined functions. - static const char *kBuiltins[] = { - "append", - "cap", - "clear", - "close", - "copy", - "delete", - "len", - "make", - "max", - "min", - "new", - "panic", - "print", - "println", - "recover", - // Built-in types used as conversion functions - "complex", - "imag", - "real", - nullptr, - }; - for (const char **b = kBuiltins; *b; b++) { - if (name == *b) - return true; +/// Collect the identifier nodes of one side of a Go `:=` / `=`. +/// +/// tree-sitter-go wraps BOTH sides of `a, b := x, y` in expression_list nodes +/// (the rule reduces with three children: left, ":=", right), so the names sit +/// one level below the declaration. Scanning the declaration's own children +/// for `identifier` therefore matched nothing and silently dropped every short +/// variable declaration. +/// \param list An expression_list node (or the declaration itself). +/// \param out Appended identifier nodes, in source order. +void collectGoListIdentifiers(TSNode list, std::vector &out) +{ + uint32_t cnt = ts_node_child_count(list); + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(list, i); + if (!ts_node_is_named(c)) + continue; + const char *t = ts_node_type(c); + if (strcmp(t, "identifier") == 0) + out.push_back(c); + else if (strcmp(t, "expression_list") == 0) + collectGoListIdentifiers(c, out); } - return false; } +} // namespace + GoVisitor::GoVisitor() { } @@ -96,6 +63,11 @@ SemanticUnit *GoVisitor::visit(TSTree *tree, const char *source, const char *fp) interface_embeds_.clear(); TSNode root_node = ts_tree_root_node(tree); + // Names this file defines, so a bare call to a user function whose name + // collides with a Go builtin (`func len(...)`, `func cap(...)`) is not + // dropped by the builtin filter in handleCall (see collectDefinedNames). + defined_names_.clear(); + collectDefinedNames(root_node); pushScope(); SourceRange root_loc = location(root_node); uint64_t root_id = emitter_->emitVariable("", root_loc, 0); @@ -226,8 +198,14 @@ void GoVisitor::visitNode(TSNode node, uint64_t parent_id) return handleRange(node, parent_id); if (strcmp(type, "parameter_declaration") == 0) return handleParameterDecl(node, parent_id); - if (strcmp(type, "method_spec") == 0) - return handleInterfaceMethod(node, parent_id); + // The blank identifier binds nothing: `_` is a discard, not a symbol. The + // short-variable path already skips it (handleShortVar) but every other + // context reaches visitIdentifier, which emits a Variable for any + // identifier — so `_, x = f()` used to produce a `_` entity while + // `_, x := f()` did not. Dropping it here keeps the two assignment forms + // consistent and keeps `_` out of the graph. + if (strcmp(type, "identifier") == 0 && nodeText(node) == "_") + return; JsVisitor::visitNode(node, parent_id); } void GoVisitor::handleFuncDecl(TSNode node, uint64_t parent_id) @@ -408,523 +386,6 @@ void GoVisitor::handleMethodDecl(TSNode node, uint64_t parent_id) popFunctionScope(); popScope(); } -void GoVisitor::handleTypeDecl(TSNode node, uint64_t parent_id) -{ - uint32_t cnt = ts_node_child_count(node); - for (uint32_t i = 0; i < cnt; i++) { - TSNode c = ts_node_child(node, i); - if (!ts_node_is_named(c)) - continue; - if (strcmp(ts_node_type(c), "type_spec") == 0) { - SourceRange loc = location(c); - std::string name = extractName(c); - if (name.empty()) - continue; - // Determine type kind from the definition child - uint32_t sc = ts_node_child_count(c); - bool is_struct = false, is_interface = false; - for (uint32_t j = 0; j < sc; j++) { - TSNode def = ts_node_child(c, j); - if (!ts_node_is_named(def)) - continue; - const char *dt = ts_node_type(def); - if (strcmp(dt, "struct_type") == 0) { - is_struct = true; - break; - } - if (strcmp(dt, "interface_type") == 0) { - is_interface = true; - break; - } - } - uint64_t id; - if (is_struct) - id = emitter_->emitClass( - name, loc, parent_id, - isupper(static_cast( - name[0])) ? - 1 : - 0); - else if (is_interface) - id = emitter_->emitInterface( - name, loc, parent_id, - isupper(static_cast( - name[0])) ? - 1 : - 0); - else - id = emitter_->emitTypeAlias( - name, loc, parent_id, - isupper(static_cast( - name[0])) ? - 1 : - 0); - defineSymbol(name, id); - if (is_interface) { - // Step 8: walk the interface body with - // current_interface_ set so handleInterfaceMethod - // collects the interface's method set. - current_interface_ = name; - // v0.2.5: capture embedded interfaces. An interface - // body may embed another interface by naming its type - // (type A interface { B; foo() }). That embedded - // name is a type_identifier child of interface_type - // (not a method_elem), so handleInterfaceMethod never - // sees it. Scan the body's named children for - // type_identifier / embedded_interface entries whose - // text names a known interface and record the embed so - // the end-of-file method-set check can expand A with B's - // methods (transitively). - { - uint32_t body_count = - ts_node_child_count(c); - for (uint32_t bi = 0; bi < body_count; - bi++) { - TSNode child = - ts_node_child(c, bi); - if (!ts_node_is_named(child)) - continue; - const char *ct = - ts_node_type(child); - // A Go interface body embeds other - // interfaces by naming them as a plain - // type (type A interface { B; foo() }). - // Record the name unconditionally — the - // end-of-file expansion skips embedded - // interfaces that were declared in - // another file (not in interface_methods_). - if (strcmp(ct, - "type_identifier") == - 0 || - strcmp(ct, - "embedded_interface") == - 0) { - std::string emb = - nodeText(child); - if (!emb.empty()) - interface_embeds_[name] - .push_back( - emb); - } - } - } - visitChildren(c, id); - current_interface_.clear(); - } else if (is_struct) { - // Visit type body (struct fields). - visitChildren(c, id); - // Step 8.1c: collect struct field -> type so - // handleCall can resolve field-chain receivers - // (r.pluginBus.AfterStep): first segment from - // var_types_, then walk fields via this table. - uint32_t sc = ts_node_child_count(c); - for (uint32_t j = 0; j < sc; j++) { - TSNode def = ts_node_child(c, j); - if (!ts_node_is_named(def)) - continue; - if (strcmp(ts_node_type(def), - "struct_type") != 0) - continue; - // struct_type -> field_declaration_list - uint32_t dc = ts_node_child_count(def); - for (uint32_t k = 0; k < dc; k++) { - TSNode dl = - ts_node_child(def, k); - if (!ts_node_is_named(dl)) - continue; - if (strcmp(ts_node_type(dl), - "field_declaration_list") != - 0) - continue; - uint32_t fc = - ts_node_child_count(dl); - for (uint32_t m = 0; m < fc; - m++) { - TSNode fd = - ts_node_child( - dl, m); - if (!ts_node_is_named( - fd)) - continue; - if (strcmp(ts_node_type( - fd), - "field_declaration") != - 0) - continue; - TSNode fname = - ts_node_child_by_field_name( - fd, - "name", - 4); - TSNode ftype = - ts_node_child_by_field_name( - fd, - "type", - 4); - if (ts_node_is_null( - fname) || - ts_node_is_null( - ftype)) - continue; - std::string fname_txt = - nodeText(fname); - std::string ftype_txt = - nodeText(ftype); - // Unwrap pointer_type - // (`*PluginBus` → `PluginBus`). - if (strcmp(ts_node_type( - ftype), - "pointer_type") == - 0) { - uint32_t pc = ts_node_child_count( - ftype); - for (uint32_t n = - 0; - n < pc; - n++) { - TSNode inner = ts_node_child( - ftype, - n); - if (ts_node_is_named( - inner) && - std::string(ts_node_type( - inner)) == - "type_identifier") - ftype_txt = nodeText( - inner); - } - } - if (!fname_txt.empty() && - !ftype_txt.empty()) { - struct_fields_ - [name] - [fname_txt] = - ftype_txt; - // Persist field -> type as - // a TypeRef record under the - // struct entity so the - // Resolver can rebuild the - // field table GLOBALLY - // (cross-file) for field - // chain receivers. - emitter_->emitTypeRef( - fname_txt, - ftype_txt, - location( - fd), - id); - } - } - } - } - } else { - // Visit type body (type aliases, etc.) - visitChildren(c, id); - } - } - } -} -void GoVisitor::handleCall(TSNode node, uint64_t parent_id) -{ - SourceRange loc = location(node); - std::string name; - std::string selector_name; - uint32_t cnt = ts_node_child_count(node); - for (uint32_t i = 0; i < cnt; i++) { - TSNode c = ts_node_child(node, i); - if (!ts_node_is_named(c)) - continue; - if (strcmp(ts_node_type(c), "selector_expression") == 0) { - selector_name = nodeText(c); - // Extract just the method name after the last dot - size_t dot = selector_name.rfind('.'); - name = (dot != std::string::npos) ? - selector_name.substr(dot + 1) : - selector_name; - } else if (strcmp(ts_node_type(c), "identifier") == 0) { - name = nodeText(c); - } - } - - if (!name.empty() && isGoBuiltin(name)) { - visitChildren(node, parent_id); - return; - } - - // ── Route detection ─────────────────────────────────────────── - // Detect HTTP route registrations (Gin/Echo/Chi/net/http) - // Reference: codebase-memory-mcp (MIT) service_patterns.c - { - // Check if this is a selector expression call (method call) - for (uint32_t i = 0; i < cnt; i++) { - TSNode c = ts_node_child(node, i); - if (!ts_node_is_named(c)) - continue; - if (strcmp(ts_node_type(c), "selector_expression") == - 0) { - std::string sel_text = nodeText(c); - size_t dot_pos = sel_text.rfind('.'); - if (dot_pos == std::string::npos) - break; - std::string method = - sel_text.substr(dot_pos + 1); - bool is_http_method = false; - for (int hi = 0; kHttpMethods[hi] != nullptr; - hi++) { - if (method == kHttpMethods[hi]) { - is_http_method = true; - break; - } - } - if (!is_http_method) - break; - - // Find argument_list child — arguments are NOT direct children - std::string route_path; - std::string handler_name; - bool route_found = false; - for (uint32_t j = 0; j < cnt && !route_found; - j++) { - TSNode arg_node = - ts_node_child(node, j); - if (!ts_node_is_named(arg_node)) - continue; - if (strcmp(ts_node_type(arg_node), - "argument_list") != 0) - continue; - uint32_t ac = - ts_node_child_count(arg_node); - bool found_path = false; - for (uint32_t k = 0; k < ac; k++) { - TSNode arg = ts_node_child( - arg_node, k); - if (!ts_node_is_named(arg)) - continue; - const char *arg_type = - ts_node_type(arg); - if (!found_path) { - if (strcmp(arg_type, - "interpreted_string_literal") == - 0 || - strcmp(arg_type, - "raw_string_literal") == - 0) { - route_path = nodeText( - arg); - if (route_path.size() >= - 2 && - route_path.front() == - '"' && - route_path.back() == - '"') - route_path = - route_path - .substr(1, - route_path.size() - - 2); - found_path = - true; - } - } else { - if (strcmp(arg_type, - "identifier") == - 0 || - strcmp(arg_type, - "selector_expression") == - 0) { - handler_name = - nodeText( - arg); - break; - } - } - } - break; - } - if (!route_path.empty()) - emitter_->emitRoute( - method + " " + route_path, - handler_name, loc, parent_id); - route_found = true; - break; - } - } - } - - // ── Classify call kind ───────────────────────────────────────── - CallKind call_kind = CallKind::Direct; - if (!selector_name.empty()) { - // Method call: obj.Method() or pkg.Func() - call_kind = CallKind::Method; - // Step 8 (plan §8): interface dispatch. If the receiver's - // static type is a known interface (declared in this file), - // classify the call as Interface so the Resolver's dispatch - // expansion builds a bounded candidate set from - // interface_impl_index_ instead of guessing one method. The - // receiver_type field (filled in setCallFacts below from - // var_types_) carries the interface name — required by - // pipeline.cpp's `!ref.receiver_type.empty()` gate. - size_t dot = selector_name.rfind('.'); - std::string recv_text = (dot != std::string::npos) ? - selector_name.substr(0, dot) : - std::string(); - if (!recv_text.empty() && - import_aliases_.count(recv_text) == 0) { - auto vt = var_types_.find(recv_text); - if (vt != var_types_.end() && - interface_methods_.count(vt->second) > 0) - call_kind = CallKind::Interface; - } - // Check for constructor pattern: NewType(). The previous - // `name.size() > 3` threshold excluded exactly "New" (3 chars), - // so a bare `New()` call was misclassified as Direct and never - // got the constructor boost in the Resolver Pipeline. - // See CODE_REVIEW_FINDINGS_2026-07-19.md H6. - if (call_kind == CallKind::Method && name.size() >= 3 && - name[0] == 'N' && name[1] == 'e' && name[2] == 'w') - call_kind = CallKind::Constructor; - } else { - // Bare function call: check if it's a constructor - // Same threshold fix as above (>= 3 includes "New" itself). - if (name.size() >= 3 && name[0] == 'N' && name[1] == 'e' && - name[2] == 'w') - call_kind = CallKind::Constructor; - } - - // Use the containing function as parent_id (not the immediate - // syntactic parent, which may be another call record). Without - // this, nested calls inside another call's argument_list would - // have their parent_id set to the outer call record, which is - // NOT in _r2n (only declarations are). The reference-table JOIN - // would fail and the nested call would be dropped. - uint64_t func_id = currentFunctionId(); - uint64_t call_parent = (func_id != 0) ? func_id : parent_id; - - // Compute arity from the `argument_list` child node's named children. - // Previously hardcoded to 0, which degraded overload disambiguation - // by arity in the Resolver Pipeline. Mirrors CVisitor::countArguments. - int arity = 0; - for (uint32_t i = 0; i < cnt; i++) { - TSNode c = ts_node_child(node, i); - if (strcmp(ts_node_type(c), "argument_list") != 0) - continue; - uint32_t ac = ts_node_child_count(c); - for (uint32_t j = 0; j < ac; j++) { - TSNode arg = ts_node_child(c, j); - if (ts_node_is_named(arg)) - ++arity; - } - break; - } - - uint64_t id = emitter_->emitCall(name, loc, call_parent, arity, false, - static_cast(call_kind)); - - // ── Step 4 (plan §4A): structured call facts ────────────────── - // For selector calls (obj.Method() or pkg.Func()), record the full - // qualified target, the receiver expression, the inferred receiver - // type, and the import alias (if the receiver is an imported package - // alias). Bare calls leave all fields empty — an empty receiver_text - // is the meaningful "no receiver" signal for the Resolver. - if (!selector_name.empty()) { - std::string qualified_target = selector_name; - std::string receiver_text; - std::string receiver_type; - std::string import_alias; - size_t dot = selector_name.rfind('.'); - if (dot != std::string::npos) - receiver_text = selector_name.substr(0, dot); - // If the receiver is a known import alias, this is a - // package-qualified call (e.g. fmt.Println). Otherwise, if the - // receiver is a local variable with a known type, record the - // type so the Resolver can match the method by receiver type. - if (!receiver_text.empty()) { - if (import_aliases_.count(receiver_text) > 0) { - import_alias = receiver_text; - } else { - // Step 8.1c: resolve field-chain receivers - // (`r.pluginBus.AfterStep`). Resolve the first - // segment via var_types_, then walk each - // subsequent field through struct_fields_ so - // receiver_type becomes the field's type (e.g. - // an interface) instead of empty. A field-chain - // must resolve EVERY segment; if any lookup fails - // the whole chain is treated as unknown (empty - // receiver_type) rather than falling back to the - // first segment's type — an incorrect concrete - // type would misroute the Resolver into a false - // positive edge. - auto vt = var_types_.find(receiver_text); - if (vt != var_types_.end()) { - receiver_type = vt->second; - } else if (receiver_text.find('.') != - std::string::npos) { - std::string cur = receiver_text; - std::string cur_type; - bool chain_ok = false; - // First segment: variable type. - size_t first_dot = cur.find('.'); - std::string first = - cur.substr(0, first_dot); - auto fv = var_types_.find(first); - if (fv != var_types_.end()) { - cur_type = fv->second; - chain_ok = true; - } - // Remaining segments: struct fields. - size_t pos = first_dot; - while (chain_ok && - pos != std::string::npos) { - size_t next = - cur.find('.', pos + 1); - std::string field = cur.substr( - pos + 1, - (next == - std::string::npos) ? - std::string::npos : - next - pos - 1); - auto ft = struct_fields_.find( - cur_type); - if (ft == - struct_fields_.end()) { - chain_ok = false; - break; - } - auto fld = - ft->second.find(field); - if (fld == ft->second.end()) { - chain_ok = false; - break; - } - cur_type = fld->second; - pos = next; - } - if (chain_ok && !cur_type.empty()) - receiver_type = cur_type; - } - } - } - emitter_->setCallFacts(id, qualified_target, receiver_text, - receiver_type, import_alias); - } - - // ── Intra-file callee resolution ─────────────────────────── - // Store the resolved callee's record ID as ref_original_id. - // Enables P1 call-edge construction in buildCallEdgesSQL. - if (!name.empty()) { - uint64_t target = resolveSymbol(name); - if (target) { - unit_->setCallReference(id, target); - unit_->setCallStrategy(id, "p1_intra"); - } else { - unit_->setCallStrategy( - id, BuiltinRegistry::resolve(unit_->language(), - name)); - } - } - - visitChildren(node, id); -} void GoVisitor::handleImport(TSNode node, uint64_t parent_id) { emitter_->emitImport(nodeText(node), location(node), parent_id); @@ -1005,7 +466,9 @@ void GoVisitor::handleVarDecl(TSNode node, uint64_t parent_id) if (strcmp(ts_node_type(c), "var_spec") == 0 || strcmp(ts_node_type(c), "const_spec") == 0) { std::string name = extractName(c); - if (!name.empty()) { + // The blank identifier (`var _ = f()`) discards the value and + // binds no symbol — its initializer is still visited below. + if (!name.empty() && name != "_") { uint64_t id = emitter_->emitVariable( name, location(c), parent_id, isupper(static_cast( @@ -1046,58 +509,72 @@ void GoVisitor::handleVarDecl(TSNode node, uint64_t parent_id) } } } + // Walk the value expression: a call in `var x = f()` (or + // `var _ = f()`) belongs to the enclosing function. This handler + // used to drop the value entirely, losing those call edges. Only + // the value field is followed — the name and type nodes are + // consumed above, and re-visiting the type would emit stray + // identifier records. + TSNode value = + ts_node_child_by_field_name(c, "value", 5); + if (!ts_node_is_null(value)) + visitChild(value, parent_id); } } } void GoVisitor::handleShortVar(TSNode node, uint64_t parent_id) { // Step 4 (plan §4A): short_var_declaration has the form - // `name := expr` or `name, name2 := expr1, expr2`. tree-sitter - // exposes the left-hand identifiers and the right-hand expressions - // as siblings. We pair them positionally to infer variable types - // from composite literals (e.g. `b := Box{...}` → type "Box"). - uint32_t cnt = ts_node_child_count(node); - // First pass: collect LHS identifier names in order. + // `name := expr` or `name, name2 := expr1, expr2`. + // + // tree-sitter-go reduces this rule with three children — + // short_var_declaration(left: expression_list, ":=", right: expression_list) + // — so the LHS names live one level BELOW the declaration. The previous + // implementation scanned the declaration's direct children for + // `identifier`, matched none, and therefore never emitted or defined a + // single short variable: receiver-type inference lost every `:=` binding + // and the RHS was visited as one opaque expression_list. Read the `left` + // and `right` fields and walk one level into each list. + std::vector lhs_nodes; + TSNode left = ts_node_child_by_field_name(node, "left", 4); + if (!ts_node_is_null(left)) + collectGoListIdentifiers(left, lhs_nodes); + + // Emit one Variable per LHS name, preserving declaration order so the + // Nth name pairs with the Nth RHS expression below (Go requires the + // counts to match for `:=`). The blank identifier binds no symbol. std::vector lhs_names; - for (uint32_t i = 0; i < cnt; i++) { - TSNode c = ts_node_child(node, i); - if (!ts_node_is_named(c)) + lhs_names.reserve(lhs_nodes.size()); + for (TSNode n : lhs_nodes) { + std::string name = nodeText(n); + if (name.empty() || name == "_") continue; - if (strcmp(ts_node_type(c), "identifier") == 0) - lhs_names.push_back(nodeText(c)); + uint64_t id = emitter_->emitVariable( + name, location(n), parent_id, + isupper(static_cast(name[0])) ? 1 : 0); + defineSymbol(name, id); + lhs_names.push_back(std::move(name)); } - // Second pass: emit variables, recurse into RHS, and infer types. - // rhs_idx tracks the Nth RHS expression so it pairs with - // lhs_names[N] (Go requires LHS and RHS counts to match for `:=`). + + // RHS: visit each expression so `r := foo()` still produces its call + // edge, and infer the variable type from a composite-literal RHS + // (e.g. `b := Box{val: 5}` → recordVarType("b", "Box")). + TSNode right = ts_node_child_by_field_name(node, "right", 5); + if (ts_node_is_null(right)) + return; + uint32_t rc = ts_node_child_count(right); size_t rhs_idx = 0; - for (uint32_t i = 0; i < cnt; i++) { - TSNode c = ts_node_child(node, i); + for (uint32_t i = 0; i < rc; i++) { + TSNode c = ts_node_child(right, i); if (!ts_node_is_named(c)) continue; - if (strcmp(ts_node_type(c), "identifier") == 0) { - std::string name = nodeText(c); - uint64_t id = emitter_->emitVariable( - name, location(c), parent_id, - isupper(static_cast(name[0])) ? - 1 : - 0); - defineSymbol(name, id); - } else { - // Recurse into the RHS expression so call expressions such - // as `r := foo()` are visited and emitted as call edges. - // Without this, intra-file calls inside `:=` assignments - // were silently dropped (only `=` assignments recursed). - visitNode(c, parent_id); - // Step 4: infer type from composite literal RHS - // (e.g. `b := Box{val: 5}` → recordVarType("b","Box")). - if (rhs_idx < lhs_names.size()) { - std::string inferred = inferCompositeType(c); - if (!inferred.empty()) - recordVarType(lhs_names[rhs_idx], - inferred); - } - ++rhs_idx; + visitChild(c, parent_id); + if (rhs_idx < lhs_names.size()) { + std::string inferred = inferCompositeType(c); + if (!inferred.empty()) + recordVarType(lhs_names[rhs_idx], inferred); } + ++rhs_idx; } } diff --git a/engine/src/ir/translators/go_visitor_calls.cpp b/engine/src/ir/translators/go_visitor_calls.cpp new file mode 100644 index 0000000..54b76bd --- /dev/null +++ b/engine/src/ir/translators/go_visitor_calls.cpp @@ -0,0 +1,644 @@ +// go_visitor_calls.cpp — type declarations and call-site analysis. +// +// Split out of go_visitor.cpp (see plan/rules/code_rules.md 1000-line +// rule). handleTypeDecl and handleCall are the two largest handlers and +// the only users of the Go builtin filter and the HTTP method table, so +// they moved together: handleCall needs both to decide whether a call is +// a route registration (r.GET("/x", h)) or a language builtin (len/append) +// that must not become a project-internal reference edge. + +#include "go_visitor.h" + +#include +#include +#include +#include +#include +#include "../builtin_registry.h" + +namespace ir +{ + +namespace +{ +/// Remove every ASCII whitespace byte from `s`. +/// +/// `nodeText()` returns the source slice verbatim, and a method call on a +/// continuation line ends its receiver at the previous line: +/// +/// rows := stubRows("x"). +/// AddRow("y") // selector text: stubRows("x").\n\t\tAddRow +/// +/// so the slice taken after the last '.' starts with the newline and the +/// indentation. Those bytes used to become part of the callee NAME — entities +/// were literally named "\n\t\t\t\tAddRow" (visible in the +/// detect_ffi_boundaries external_symbols list) — while every receiver lookup +/// (`var_types_`, `import_aliases_`) and the route-method comparison silently +/// failed, because "\n\tHandleFunc" never equals "HandleFunc" and a receiver +/// spelled "scheduler\n\t.Run" is in no map. A Go selector chain carries no +/// significant whitespace, so the bytes are removed rather than merely trimmed +/// at the ends (a multi-line chain can carry them in the middle too). +std::string stripWhitespace(const std::string &s) +{ + std::string out; + out.reserve(s.size()); + for (char c : s) { + if (!std::isspace(static_cast(c))) + out.push_back(c); + } + return out; +} +} // namespace + +// ─── HTTP method constants for route detection ────────────────── +// Used to identify route registrations like r.GET("/path", handler). +// Reference: codebase-memory-mcp (MIT) service_patterns.c +static const char *kHttpMethods[] = { + "GET", "POST", "PUT", "DELETE", "PATCH", + "HEAD", "OPTIONS", "HandleFunc", "Handle", // net/http + nullptr, +}; + +// ─── Language-specific builtin detection ─────────────────────────── +// +// Reference: codebase-memory-mcp (MIT, https://github.com/DeusData/codebase-memory-mcp) +// internal/cbm/lsp/go_lsp.c :: is_go_builtin_func() +// +// Go built-in functions are language keywords that should NOT create +// reference entries — they are not user-defined functions and cannot be +// resolved to any project entity. Without this filter, every call to +// len(), append(), copy(), etc. generates a "Len"/"Append"/"Copy" +// reference that the Resolver Pipeline incorrectly matches to any +// project entity with the same name, producing massive false-positive +// cross-module call edges. +// +// The same pattern should be applied to every language visitor: +// filter out language builtins / keywords at the parser level so the +// resolver never sees them. + +static bool isGoBuiltin(const std::string &name) +{ + // Complete list of Go built-in functions (per language spec). + // All of these are predeclared identifiers that cannot be + // redefined and never resolve to user-defined functions. + static const char *kBuiltins[] = { + "append", + "cap", + "clear", + "close", + "copy", + "delete", + "len", + "make", + "max", + "min", + "new", + "panic", + "print", + "println", + "recover", + // Built-in types used as conversion functions + "complex", + "imag", + "real", + nullptr, + }; + for (const char **b = kBuiltins; *b; b++) { + if (name == *b) + return true; + } + return false; +} + +void GoVisitor::handleTypeDecl(TSNode node, uint64_t parent_id) +{ + uint32_t cnt = ts_node_child_count(node); + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(node, i); + if (!ts_node_is_named(c)) + continue; + if (strcmp(ts_node_type(c), "type_spec") == 0) { + SourceRange loc = location(c); + std::string name = extractName(c); + if (name.empty()) + continue; + // Determine type kind from the definition child + uint32_t sc = ts_node_child_count(c); + bool is_struct = false, is_interface = false; + for (uint32_t j = 0; j < sc; j++) { + TSNode def = ts_node_child(c, j); + if (!ts_node_is_named(def)) + continue; + const char *dt = ts_node_type(def); + if (strcmp(dt, "struct_type") == 0) { + is_struct = true; + break; + } + if (strcmp(dt, "interface_type") == 0) { + is_interface = true; + break; + } + } + uint64_t id; + if (is_struct) + id = emitter_->emitClass( + name, loc, parent_id, + isupper(static_cast( + name[0])) ? + 1 : + 0); + else if (is_interface) + id = emitter_->emitInterface( + name, loc, parent_id, + isupper(static_cast( + name[0])) ? + 1 : + 0); + else + id = emitter_->emitTypeAlias( + name, loc, parent_id, + isupper(static_cast( + name[0])) ? + 1 : + 0); + defineSymbol(name, id); + if (is_interface) { + // Step 8: walk the interface body with + // current_interface_ set so handleInterfaceMethod + // collects the interface's method set. + current_interface_ = name; + // v0.2.5: capture embedded interfaces. An interface + // body may embed another interface by naming its type + // (type A interface { B; foo() }). That embedded + // name is a type_identifier child of interface_type + // (not a method_elem), so handleInterfaceMethod never + // sees it. Scan the body's named children for + // type_identifier / embedded_interface entries whose + // text names a known interface and record the embed so + // the end-of-file method-set check can expand A with B's + // methods (transitively). + { + uint32_t body_count = + ts_node_child_count(c); + for (uint32_t bi = 0; bi < body_count; + bi++) { + TSNode child = + ts_node_child(c, bi); + if (!ts_node_is_named(child)) + continue; + const char *ct = + ts_node_type(child); + // A Go interface body embeds other + // interfaces by naming them as a plain + // type (type A interface { B; foo() }). + // Record the name unconditionally — the + // end-of-file expansion skips embedded + // interfaces that were declared in + // another file (not in interface_methods_). + if (strcmp(ct, + "type_identifier") == + 0 || + strcmp(ct, + "embedded_interface") == + 0) { + std::string emb = + nodeText(child); + if (!emb.empty()) + interface_embeds_[name] + .push_back( + emb); + } + } + } + visitChildren(c, id); + current_interface_.clear(); + } else if (is_struct) { + // Visit type body (struct fields). + visitChildren(c, id); + // Step 8.1c: collect struct field -> type so + // handleCall can resolve field-chain receivers + // (r.pluginBus.AfterStep): first segment from + // var_types_, then walk fields via this table. + uint32_t sc = ts_node_child_count(c); + for (uint32_t j = 0; j < sc; j++) { + TSNode def = ts_node_child(c, j); + if (!ts_node_is_named(def)) + continue; + if (strcmp(ts_node_type(def), + "struct_type") != 0) + continue; + // struct_type -> field_declaration_list + uint32_t dc = ts_node_child_count(def); + for (uint32_t k = 0; k < dc; k++) { + TSNode dl = + ts_node_child(def, k); + if (!ts_node_is_named(dl)) + continue; + if (strcmp(ts_node_type(dl), + "field_declaration_list") != + 0) + continue; + uint32_t fc = + ts_node_child_count(dl); + for (uint32_t m = 0; m < fc; + m++) { + TSNode fd = + ts_node_child( + dl, m); + if (!ts_node_is_named( + fd)) + continue; + if (strcmp(ts_node_type( + fd), + "field_declaration") != + 0) + continue; + TSNode fname = + ts_node_child_by_field_name( + fd, + "name", + 4); + TSNode ftype = + ts_node_child_by_field_name( + fd, + "type", + 4); + if (ts_node_is_null( + fname) || + ts_node_is_null( + ftype)) + continue; + std::string fname_txt = + nodeText(fname); + std::string ftype_txt = + nodeText(ftype); + // Unwrap pointer_type + // (`*PluginBus` → `PluginBus`). + if (strcmp(ts_node_type( + ftype), + "pointer_type") == + 0) { + uint32_t pc = ts_node_child_count( + ftype); + for (uint32_t n = + 0; + n < pc; + n++) { + TSNode inner = ts_node_child( + ftype, + n); + if (ts_node_is_named( + inner) && + std::string(ts_node_type( + inner)) == + "type_identifier") + ftype_txt = nodeText( + inner); + } + } + if (!fname_txt.empty() && + !ftype_txt.empty()) { + struct_fields_ + [name] + [fname_txt] = + ftype_txt; + // Persist field -> type as + // a TypeRef record under the + // struct entity so the + // Resolver can rebuild the + // field table GLOBALLY + // (cross-file) for field + // chain receivers. + emitter_->emitTypeRef( + fname_txt, + ftype_txt, + location( + fd), + id); + } + } + } + } + } else { + // Visit type body (type aliases, etc.) + visitChildren(c, id); + } + } + } +} +void GoVisitor::handleCall(TSNode node, uint64_t parent_id) +{ + SourceRange loc = location(node); + std::string name; + std::string selector_name; + uint32_t cnt = ts_node_child_count(node); + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(node, i); + if (!ts_node_is_named(c)) + continue; + if (strcmp(ts_node_type(c), "selector_expression") == 0) { + selector_name = nodeText(c); + // Extract just the method name after the last dot + size_t dot = selector_name.rfind('.'); + name = (dot != std::string::npos) ? + stripWhitespace( + selector_name.substr(dot + 1)) : + stripWhitespace(selector_name); + } else if (strcmp(ts_node_type(c), "identifier") == 0) { + name = nodeText(c); + } + } + + // Only BARE calls hit the builtin filter. `selector_name` holds the + // full receiver expression ("b.copy"), so a non-empty value means this + // is a method call on a value — `copy` there is an ordinary user method + // that happens to share a builtin's name, and dropping its record would + // hide a real project-internal call edge. `len(x)` has no receiver and + // is still filtered. + // + // A name THIS FILE declares is user code whatever the builtin list says + // (`func len(x []int) int`), so it survives the filter — the same + // exemption the JS/Rust/Java/Python/C visitors apply. + if (selector_name.empty() && !name.empty() && isGoBuiltin(name) && + !isLocallyDefined(name)) { + visitChildren(node, parent_id); + return; + } + + // ── Route detection ─────────────────────────────────────────── + // Detect HTTP route registrations (Gin/Echo/Chi/net/http) + // Reference: codebase-memory-mcp (MIT) service_patterns.c + { + // Check if this is a selector expression call (method call) + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(node, i); + if (!ts_node_is_named(c)) + continue; + if (strcmp(ts_node_type(c), "selector_expression") == + 0) { + std::string sel_text = nodeText(c); + size_t dot_pos = sel_text.rfind('.'); + if (dot_pos == std::string::npos) + break; + std::string method = stripWhitespace( + sel_text.substr(dot_pos + 1)); + bool is_http_method = false; + for (int hi = 0; kHttpMethods[hi] != nullptr; + hi++) { + if (method == kHttpMethods[hi]) { + is_http_method = true; + break; + } + } + if (!is_http_method) + break; + + // Find argument_list child — arguments are NOT direct children + std::string route_path; + std::string handler_name; + bool route_found = false; + for (uint32_t j = 0; j < cnt && !route_found; + j++) { + TSNode arg_node = + ts_node_child(node, j); + if (!ts_node_is_named(arg_node)) + continue; + if (strcmp(ts_node_type(arg_node), + "argument_list") != 0) + continue; + uint32_t ac = + ts_node_child_count(arg_node); + bool found_path = false; + for (uint32_t k = 0; k < ac; k++) { + TSNode arg = ts_node_child( + arg_node, k); + if (!ts_node_is_named(arg)) + continue; + const char *arg_type = + ts_node_type(arg); + if (!found_path) { + if (strcmp(arg_type, + "interpreted_string_literal") == + 0 || + strcmp(arg_type, + "raw_string_literal") == + 0) { + route_path = nodeText( + arg); + if (route_path.size() >= + 2 && + route_path.front() == + '"' && + route_path.back() == + '"') + route_path = + route_path + .substr(1, + route_path.size() - + 2); + found_path = + true; + } + } else { + if (strcmp(arg_type, + "identifier") == + 0 || + strcmp(arg_type, + "selector_expression") == + 0) { + handler_name = + nodeText( + arg); + break; + } + } + } + break; + } + if (!route_path.empty()) + emitter_->emitRoute( + method + " " + route_path, + handler_name, loc, parent_id); + route_found = true; + break; + } + } + } + + // ── Classify call kind ───────────────────────────────────────── + CallKind call_kind = CallKind::Direct; + if (!selector_name.empty()) { + // Method call: obj.Method() or pkg.Func() + call_kind = CallKind::Method; + // Step 8 (plan §8): interface dispatch. If the receiver's + // static type is a known interface (declared in this file), + // classify the call as Interface so the Resolver's dispatch + // expansion builds a bounded candidate set from + // interface_impl_index_ instead of guessing one method. The + // receiver_type field (filled in setCallFacts below from + // var_types_) carries the interface name — required by + // pipeline.cpp's `!ref.receiver_type.empty()` gate. + size_t dot = selector_name.rfind('.'); + std::string recv_text = + (dot != std::string::npos) ? + stripWhitespace(selector_name.substr(0, dot)) : + std::string(); + if (!recv_text.empty() && + import_aliases_.count(recv_text) == 0) { + auto vt = var_types_.find(recv_text); + if (vt != var_types_.end() && + interface_methods_.count(vt->second) > 0) + call_kind = CallKind::Interface; + } + // Check for constructor pattern: NewType(). The previous + // `name.size() > 3` threshold excluded exactly "New" (3 chars), + // so a bare `New()` call was misclassified as Direct and never + // got the constructor boost in the Resolver Pipeline. + if (call_kind == CallKind::Method && name.size() >= 3 && + name[0] == 'N' && name[1] == 'e' && name[2] == 'w') + call_kind = CallKind::Constructor; + } else { + // Bare function call: check if it's a constructor + // Same threshold fix as above (>= 3 includes "New" itself). + if (name.size() >= 3 && name[0] == 'N' && name[1] == 'e' && + name[2] == 'w') + call_kind = CallKind::Constructor; + } + + // Use the containing function as parent_id (not the immediate + // syntactic parent, which may be another call record). Without + // this, nested calls inside another call's argument_list would + // have their parent_id set to the outer call record, which is + // NOT in _r2n (only declarations are). The reference-table JOIN + // would fail and the nested call would be dropped. + uint64_t func_id = currentFunctionId(); + uint64_t call_parent = (func_id != 0) ? func_id : parent_id; + + // Compute arity from the `argument_list` child node's named children. + // Previously hardcoded to 0, which degraded overload disambiguation + // by arity in the Resolver Pipeline. Mirrors CVisitor::countArguments. + int arity = 0; + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(node, i); + if (strcmp(ts_node_type(c), "argument_list") != 0) + continue; + uint32_t ac = ts_node_child_count(c); + for (uint32_t j = 0; j < ac; j++) { + TSNode arg = ts_node_child(c, j); + if (ts_node_is_named(arg)) + ++arity; + } + break; + } + + uint64_t id = emitter_->emitCall(name, loc, call_parent, arity, false, + static_cast(call_kind)); + + // ── Step 4 (plan §4A): structured call facts ────────────────── + // For selector calls (obj.Method() or pkg.Func()), record the full + // qualified target, the receiver expression, the inferred receiver + // type, and the import alias (if the receiver is an imported package + // alias). Bare calls leave all fields empty — an empty receiver_text + // is the meaningful "no receiver" signal for the Resolver. + if (!selector_name.empty()) { + std::string qualified_target = selector_name; + std::string receiver_text; + std::string receiver_type; + std::string import_alias; + size_t dot = selector_name.rfind('.'); + if (dot != std::string::npos) + receiver_text = + stripWhitespace(selector_name.substr(0, dot)); + // If the receiver is a known import alias, this is a + // package-qualified call (e.g. fmt.Println). Otherwise, if the + // receiver is a local variable with a known type, record the + // type so the Resolver can match the method by receiver type. + if (!receiver_text.empty()) { + if (import_aliases_.count(receiver_text) > 0) { + import_alias = receiver_text; + } else { + // Step 8.1c: resolve field-chain receivers + // (`r.pluginBus.AfterStep`). Resolve the first + // segment via var_types_, then walk each + // subsequent field through struct_fields_ so + // receiver_type becomes the field's type (e.g. + // an interface) instead of empty. A field-chain + // must resolve EVERY segment; if any lookup fails + // the whole chain is treated as unknown (empty + // receiver_type) rather than falling back to the + // first segment's type — an incorrect concrete + // type would misroute the Resolver into a false + // positive edge. + auto vt = var_types_.find(receiver_text); + if (vt != var_types_.end()) { + receiver_type = vt->second; + } else if (receiver_text.find('.') != + std::string::npos) { + std::string cur = receiver_text; + std::string cur_type; + bool chain_ok = false; + // First segment: variable type. + size_t first_dot = cur.find('.'); + std::string first = + cur.substr(0, first_dot); + auto fv = var_types_.find(first); + if (fv != var_types_.end()) { + cur_type = fv->second; + chain_ok = true; + } + // Remaining segments: struct fields. + size_t pos = first_dot; + while (chain_ok && + pos != std::string::npos) { + size_t next = + cur.find('.', pos + 1); + std::string field = cur.substr( + pos + 1, + (next == + std::string::npos) ? + std::string::npos : + next - pos - 1); + auto ft = struct_fields_.find( + cur_type); + if (ft == + struct_fields_.end()) { + chain_ok = false; + break; + } + auto fld = + ft->second.find(field); + if (fld == ft->second.end()) { + chain_ok = false; + break; + } + cur_type = fld->second; + pos = next; + } + if (chain_ok && !cur_type.empty()) + receiver_type = cur_type; + } + } + } + emitter_->setCallFacts(id, qualified_target, receiver_text, + receiver_type, import_alias); + } + + // ── Intra-file callee resolution ─────────────────────────── + // Store the resolved callee's record ID as ref_original_id. + // Enables P1 call-edge construction in buildCallEdgesSQL. + if (!name.empty()) { + uint64_t target = resolveSymbol(name); + if (target) { + unit_->setCallReference(id, target); + unit_->setCallStrategy(id, "p1_intra"); + } else { + unit_->setCallStrategy( + id, BuiltinRegistry::resolve(unit_->language(), + name)); + } + } + + visitChildren(node, id); +} + +} // namespace ir diff --git a/engine/src/ir/translators/java_translator.cpp b/engine/src/ir/translators/java_translator.cpp index c6203ae..c87f434 100644 --- a/engine/src/ir/translators/java_translator.cpp +++ b/engine/src/ir/translators/java_translator.cpp @@ -23,6 +23,8 @@ class JavaTranslator : public Translator { TranslationUnit *unit_ = nullptr; const char *source_ = nullptr; std::string file_path_; + // Recursion counter for this file (see kMaxTranslateDepth). + TranslateDepth depth_; struct Scope { std::unordered_map symbols; @@ -33,6 +35,12 @@ class JavaTranslator : public Translator { Node *makeNode(NodeKind kind, TSNode ts_node); void setLocation(Node *node, TSNode ts_node); std::string nodeText(TSNode ts_node); + /// The declared name of a Java declaration node. + /// + /// Prefers the grammar's own `name` field and falls back to the LAST + /// top-level identifier. See the definition for why a first-match scan is + /// wrong here. + std::string declaredName(TSNode ts_node); void pushScope() { @@ -77,6 +85,7 @@ class JavaTranslator : public Translator { TranslationUnit *JavaTranslator::translate(TSTree *tree, const char *source, const char *file_path) { + depth_.reset(); unit_ = new TranslationUnit(); unit_->source_content = source; source_ = source; @@ -122,6 +131,36 @@ void JavaTranslator::setLocation(Node *node, TSNode ts_node) node->loc.end_col = end.column; } +std::string JavaTranslator::declaredName(TSNode ts_node) +{ + // The grammar's `name` field is authoritative: class_declaration, + // interface_declaration, method_declaration, constructor_declaration and + // enum_declaration all define one. + TSNode named = ts_node_child_by_field_name(ts_node, "name", 4); + if (!ts_node_is_null(named)) + return nodeText(named); + + // Fallback for node shapes that do not expose the field: take the LAST + // top-level identifier, because a Java declaration writes the type before + // the name. The previous first-match scan is what named every non-void + // method after its RETURN TYPE — this grammar spells a class type as + // `identifier` as well, so `public String processUpdateForm(...)` became an + // entity called `String` (measured on spring-petclinic: 27 entities named + // `String`, 4 `Pet`, 3 `LocalDate`, 2 `PetType`, 2 `Owner`, while every + // `void` method kept its real name — its type node is `void_type`, not + // `identifier`). Those type-named entities then received edges under the + // single-same-module-candidate heuristic, so the damage was not only + // cosmetic. + std::string last; + const uint32_t count = ts_node_child_count(ts_node); + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(ts_node, i); + if (strcmp(ts_node_type(child), "identifier") == 0) + last = nodeText(child); + } + return last; +} + std::string JavaTranslator::nodeText(TSNode ts_node) { uint32_t start = ts_node_start_byte(ts_node); @@ -325,6 +364,13 @@ Node *JavaTranslator::translateNode(TSNode ts_node, Node *parent) void JavaTranslator::translateChildren(TSNode ts_node, Node *parent) { + // Bound native-stack recursion (see kMaxTranslateDepth): a pathologically + // deep AST would overflow the indexer's 512 KB worker stack, which the FFI + // try/catch cannot recover. The visitors apply the same bound. + if (depth_.exceeded(file_path_.c_str(), "translateChildren")) + return; + DepthGuard depth_guard(depth_); + uint32_t count = ts_node_child_count(ts_node); for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(ts_node, i); @@ -337,15 +383,7 @@ void JavaTranslator::translateChildren(TSNode ts_node, Node *parent) Node *JavaTranslator::handleClass(TSNode ts_node, Node *parent) { auto *cls = makeNode(NodeKind::ClassDecl, ts_node); - - uint32_t count = ts_node_child_count(ts_node); - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(ts_node, i); - if (strcmp(ts_node_type(child), "identifier") == 0) { - cls->name = nodeText(child); - break; - } - } + cls->name = declaredName(ts_node); defineSymbol(cls->name, cls); parent->children.push_back(cls); @@ -361,15 +399,7 @@ Node *JavaTranslator::handleClass(TSNode ts_node, Node *parent) Node *JavaTranslator::handleInterface(TSNode ts_node, Node *parent) { auto *iface = makeNode(NodeKind::ClassDecl, ts_node); - - uint32_t count = ts_node_child_count(ts_node); - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(ts_node, i); - if (strcmp(ts_node_type(child), "identifier") == 0) { - iface->name = nodeText(child); - break; - } - } + iface->name = declaredName(ts_node); defineSymbol(iface->name, iface); parent->children.push_back(iface); @@ -384,14 +414,11 @@ Node *JavaTranslator::handleMethod(TSNode ts_node, Node *parent) { auto *method = makeNode(NodeKind::MethodDecl, ts_node); + // See declaredName(): in this grammar a class return type is spelled + // `identifier`, so the previous first-match scan named a method after its + // type whenever the type was not a primitive or `void`. + method->name = declaredName(ts_node); uint32_t count = ts_node_child_count(ts_node); - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(ts_node, i); - if (strcmp(ts_node_type(child), "identifier") == 0) { - method->name = nodeText(child); - break; - } - } defineSymbol(method->name, method); parent->children.push_back(method); @@ -424,14 +451,11 @@ Node *JavaTranslator::handleConstructor(TSNode ts_node, Node *parent) { auto *method = makeNode(NodeKind::MethodDecl, ts_node); + // See declaredName(): in this grammar a class return type is spelled + // `identifier`, so the previous first-match scan named a method after its + // type whenever the type was not a primitive or `void`. + method->name = declaredName(ts_node); uint32_t count = ts_node_child_count(ts_node); - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(ts_node, i); - if (strcmp(ts_node_type(child), "identifier") == 0) { - method->name = nodeText(child); - break; - } - } defineSymbol(method->name, method); parent->children.push_back(method); @@ -603,15 +627,7 @@ Node *JavaTranslator::handleNew(TSNode ts_node, Node *parent) Node *JavaTranslator::handleEnum(TSNode ts_node, Node *parent) { auto *en = makeNode(NodeKind::EnumDecl, ts_node); - - uint32_t count = ts_node_child_count(ts_node); - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(ts_node, i); - if (strcmp(ts_node_type(child), "identifier") == 0) { - en->name = nodeText(child); - break; - } - } + en->name = declaredName(ts_node); defineSymbol(en->name, en); parent->children.push_back(en); diff --git a/engine/src/ir/translators/java_visitor.cpp b/engine/src/ir/translators/java_visitor.cpp index ff396ef..727eb5b 100644 --- a/engine/src/ir/translators/java_visitor.cpp +++ b/engine/src/ir/translators/java_visitor.cpp @@ -86,6 +86,10 @@ SemanticUnit *JavaVisitor::visit(TSTree *tree, const char *source, import_aliases_.clear(); TSNode root_node = ts_tree_root_node(tree); + // Names this file defines, so a call to a locally declared `valueOf` / + // `format` / `map` ... is not mistaken for the JDK static of that name. + defined_names_.clear(); + collectDefinedNames(root_node); pushScope(); SourceRange root_loc = location(root_node); uint64_t root_id = emitter_->emitVariable("", root_loc, 0); @@ -113,10 +117,57 @@ void JavaVisitor::visitNode(TSNode node, uint64_t parent_id) return handleObjectCreation(node, parent_id); if (strcmp(type, "variable_declarator") == 0) return handleVariableDecl(node, parent_id); + if (strcmp(type, "formal_parameter") == 0 || + strcmp(type, "spread_parameter") == 0) + return handleFormalParameter(node, parent_id); if (strcmp(type, "import_declaration") == 0) return handleImport(node, parent_id); JsVisitor::visitNode(node, parent_id); } + +void JavaVisitor::handleFormalParameter(TSNode node, uint64_t parent_id) +{ + // `void use(Drawable d)`: the parameter's TYPE has to be recorded, not just + // its name. handleMethodInvocation resolves `d.draw()`'s receiver_type from + // var_types_, so a parameter that was never recorded left the receiver + // factor empty — the interface method and every implementation then tied on + // every factor, the Step-5 ambiguity gate abstained, and the call produced + // no edge at all. + std::string name; + std::string type_name; + uint32_t cnt = ts_node_child_count(node); + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(node, i); + if (!ts_node_is_named(c)) + continue; + const char *t = ts_node_type(c); + if (strcmp(t, "identifier") == 0) { + name = nodeText(c); + continue; + } + if (strcmp(t, "type_identifier") == 0) { + type_name = nodeText(c); + continue; + } + // `List items` — the base type is the receiver's type, the type + // arguments are not (mirrors handleVariableDecl). + if (strcmp(t, "generic_type") == 0) { + TSNode base = ts_node_child(c, 0); + if (!ts_node_is_null(base)) + type_name = nodeText(base); + continue; + } + } + if (name.empty()) { + visitChildren(node, parent_id); + return; + } + uint64_t id = + emitter_->emitVariable(name, location(node), parent_id, 0); + defineSymbol(name, id); + if (!type_name.empty()) + recordVarType(name, type_name); +} void JavaVisitor::handleMethodDecl(TSNode node, uint64_t parent_id) { SourceRange loc = location(node); @@ -128,6 +179,16 @@ void JavaVisitor::handleMethodDecl(TSNode node, uint64_t parent_id) uint64_t id = emitter_->emitMethod(name, loc, parent_id, 0, false, detectVisibility(node)); defineSymbol(name, id); + // Tag the method with its declaring type (`Circle::draw`). The Resolver's + // factorReceiverTypeMatch and the interface dispatch index both match on + // this prefix: without it, an interface method and each of its + // implementations are candidates with identical names and arity, tie on + // every factor, and the Step-5 ambiguity gate abstains — so a call through + // the interface produced no edge at all (the same reasoning as + // CVisitor::handleFuncDef). + std::string cls = currentClassName(); + if (!cls.empty()) + unit_->setQualifiedName(id, cls + "::" + name); pushScope(); pushFunctionScope(id); uint32_t cnt = ts_node_child_count(node); @@ -164,23 +225,8 @@ void JavaVisitor::handleClassDecl(TSNode node, uint64_t parent_id) TSNode c = ts_node_child(node, i); if (!ts_node_is_named(c)) continue; - if (strcmp(ts_node_type(c), "super_interfaces") == 0) { - uint32_t sc = ts_node_child_count(c); - for (uint32_t j = 0; j < sc; j++) { - TSNode iface = ts_node_child(c, j); - if (!ts_node_is_named(iface)) - continue; - if (strcmp(ts_node_type(iface), - "type_identifier") == 0) { - std::string iface_name = - nodeText(iface); - if (!iface_name.empty()) - emitter_->emitInterfaceImpl( - name, iface_name, - location(iface), id); - } - } - } + if (strcmp(ts_node_type(c), "super_interfaces") == 0) + emitInterfaceImpls(c, name, id); } for (uint32_t i = 0; i < cnt; i++) { TSNode c = ts_node_child(node, i); @@ -191,11 +237,52 @@ void JavaVisitor::handleClassDecl(TSNode node, uint64_t parent_id) if (strcmp(ts_node_type(c), "class_body") == 0) visitChildren(c, id); else - visitNode(c, id); + visitChild(c, id); } popClassScope(); popScope(); } +void JavaVisitor::emitInterfaceImpls(TSNode node, const std::string &impl_type, + uint64_t parent_id) +{ + uint32_t cnt = ts_node_child_count(node); + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(node, i); + if (!ts_node_is_named(c)) + continue; + const char *t = ts_node_type(c); + if (strcmp(t, "type_identifier") == 0) { + std::string iface_name = nodeText(c); + if (!iface_name.empty()) + emitter_->emitInterfaceImpl(impl_type, + iface_name, + location(c), + parent_id); + continue; + } + // `implements Comparable` — record the base type only; the + // type arguments are not implemented interfaces. + if (strcmp(t, "generic_type") == 0) { + uint32_t gc = ts_node_child_count(c); + for (uint32_t j = 0; j < gc; j++) { + TSNode g = ts_node_child(c, j); + if (ts_node_is_named(g) && + strcmp(ts_node_type(g), + "type_identifier") == 0) { + std::string iface_name = nodeText(g); + if (!iface_name.empty()) + emitter_->emitInterfaceImpl( + impl_type, iface_name, + location(g), parent_id); + break; + } + } + continue; + } + emitInterfaceImpls(c, impl_type, parent_id); + } +} + void JavaVisitor::handleInterfaceDecl(TSNode node, uint64_t parent_id) { SourceRange loc = location(node); @@ -207,7 +294,15 @@ void JavaVisitor::handleInterfaceDecl(TSNode node, uint64_t parent_id) uint64_t id = emitter_->emitInterface(name, loc, parent_id, detectVisibility(node)); defineSymbol(name, id); + // Push an interface scope so its methods are qualified + // (`Drawable::draw`) exactly like class methods: the resolver's dispatch + // index reconstructs method sets from those qualified names to find the + // implementations of an interface. + pushScope(); + pushClassScope(name); visitChildren(node, id); + popClassScope(); + popScope(); } void JavaVisitor::handleEnumDecl(TSNode node, uint64_t parent_id) { @@ -233,7 +328,7 @@ void JavaVisitor::handleMethodInvocation(TSNode node, uint64_t parent_id) // callee names were always the receiver → resolveSymbol never // matched → ref_original_id=0 → all Java call-edges were lost. // ts_node_child_by_field_name fetches the `name` field directly - // regardless of child order. See CODE_REVIEW_FINDINGS_2026-07-19.md C3. + // regardless of child order. std::string name; TSNode name_node = ts_node_child_by_field_name(node, "name", 4); if (!ts_node_is_null(name_node)) @@ -243,8 +338,28 @@ void JavaVisitor::handleMethodInvocation(TSNode node, uint64_t parent_id) // path and the post-emit recursion below. uint32_t cnt = ts_node_child_count(node); - // Skip Java common JDK methods — they are NOT user-defined calls - if (!name.empty() && isJavaBuiltin(name)) { + // obj.method() / Class.method() — the method_invocation node carries an + // optional `object` field (the receiver). Fetched before the builtin + // check below, which needs it to tell a bare call from a method call. + TSNode obj_node = ts_node_child_by_field_name(node, "object", 6); + bool has_receiver = !ts_node_is_null(obj_node); + + // Skip Java JDK builtins — but ONLY for unqualified calls. + // method_invocation covers both `println(...)` and `obj.method()`, and + // `name` is the bare method name either way, so the previous check + // dropped every call whose method name collided with the JDK list: + // `map`, `filter`, `forEach`, `collect`, `reduce`, `indexOf`, `replace`, + // `format`, `compareTo`, `startsWith`, `equals`, `toString`, `hashCode`, + // `clone` … are all extremely common user-defined method names, and none + // of them produced a call record — a systematic false negative. A call + // with a receiver can never be a JDK static, so it keeps its record and + // the receiver/interface evidence the Resolver needs. + // Same exemption as Python, and it covers static imports: `import static + // foo.Bar.valueOf;` inserts `valueOf` into import_aliases_ (the last + // segment), so a statically imported user method is no longer dropped as + // the JDK's Integer.valueOf. + if (!has_receiver && !name.empty() && isJavaBuiltin(name) && + !isLocallyDefined(name) && import_aliases_.count(name) == 0) { for (uint32_t i = 0; i < cnt; i++) { TSNode c = ts_node_child(node, i); if (!ts_node_is_named(c)) @@ -254,7 +369,7 @@ void JavaVisitor::handleMethodInvocation(TSNode node, uint64_t parent_id) strcmp(t, "scoped_identifier") == 0 || strcmp(t, "field_access") == 0) continue; - visitNode(c, parent_id); + visitChild(c, parent_id); } return; } @@ -265,8 +380,6 @@ void JavaVisitor::handleMethodInvocation(TSNode node, uint64_t parent_id) // old find('.') check never matched and every method call was // mislabeled Direct, skipping the Resolver's CallKindMatch factor // and receiver evidence. Detect the receiver to mark Method. - TSNode obj_node = ts_node_child_by_field_name(node, "object", 6); - bool has_receiver = !ts_node_is_null(obj_node); CallKind call_kind = CallKind::Direct; if (has_receiver) { call_kind = CallKind::Method; @@ -358,7 +471,7 @@ void JavaVisitor::handleMethodInvocation(TSNode node, uint64_t parent_id) strcmp(t, "scoped_identifier") == 0 || strcmp(t, "field_access") == 0) continue; - visitNode(c, id); + visitChild(c, id); } } void JavaVisitor::handleObjectCreation(TSNode node, uint64_t parent_id) @@ -369,7 +482,7 @@ void JavaVisitor::handleObjectCreation(TSNode node, uint64_t parent_id) // via child_by_field_name("type"), which may be a type_identifier // (`Foo`) or a generic_type (`Foo`). Mirrors handleMethodInvocation's // use of child_by_field_name for robust name extraction regardless of - // child order. See CODE_REVIEW_FINDINGS_2026-07-19.md C3 (same pattern). + // child order. std::string name; TSNode type_node = ts_node_child_by_field_name(node, "type", 4); if (!ts_node_is_null(type_node)) { @@ -434,7 +547,7 @@ void JavaVisitor::handleObjectCreation(TSNode node, uint64_t parent_id) strcmp(t, "scoped_type_identifier") == 0 || strcmp(t, "array_type") == 0) continue; - visitNode(c, id); + visitChild(c, id); } } void JavaVisitor::handleVariableDecl(TSNode node, uint64_t parent_id) @@ -543,7 +656,7 @@ void JavaVisitor::handleVariableDecl(TSNode node, uint64_t parent_id) continue; if (strcmp(ts_node_type(c), "identifier") == 0) continue; - visitNode(c, parent_id); + visitChild(c, parent_id); } } void JavaVisitor::handleImport(TSNode node, uint64_t parent_id) @@ -580,6 +693,26 @@ void JavaVisitor::handleImport(TSNode node, uint64_t parent_id) } std::string JavaVisitor::extractName(TSNode node) { + // The grammar's `name` field is authoritative for every declaration this is + // reached for (class_declaration, interface_declaration, + // method_declaration, constructor_declaration, enum_declaration). + TSNode named = ts_node_child_by_field_name(node, "name", 4); + if (!ts_node_is_null(named)) + return nodeText(named); + + // Fallback for node shapes that do not expose the field: the LAST + // identifier, because a Java declaration writes the type before the name. + // The previous first-match scan accepted `type_identifier` as well, and for + // `public String describe(...)` the return type IS a type_identifier — so + // every method whose return type was a class (not a primitive or `void`) + // was recorded under that type: measured on spring-petclinic, 27 entities + // called `String`, 4 `Pet`, 3 `LocalDate`, 2 `PetType`, 2 `Owner`, while + // every `void` method kept its real name (`void_type` is not matched here). + // Those type-named entities then collected edges under the + // single-same-module-candidate heuristic, so the damage was not cosmetic: + // `find_definition("processUpdateForm")` answered nothing and a call to + // `updatePetDetails` was attributed to an entity called `String`. + std::string last; uint32_t cnt = ts_node_child_count(node); for (uint32_t i = 0; i < cnt; i++) { TSNode c = ts_node_child(node, i); @@ -587,9 +720,9 @@ std::string JavaVisitor::extractName(TSNode node) continue; if (strcmp(ts_node_type(c), "identifier") == 0 || strcmp(ts_node_type(c), "type_identifier") == 0) - return nodeText(c); + last = nodeText(c); } - return ""; + return last; } int JavaVisitor::detectVisibility(TSNode node) diff --git a/engine/src/ir/translators/java_visitor.h b/engine/src/ir/translators/java_visitor.h index 612d28f..10f396c 100644 --- a/engine/src/ir/translators/java_visitor.h +++ b/engine/src/ir/translators/java_visitor.h @@ -16,11 +16,32 @@ class JavaVisitor : public JsVisitor { protected: void visitNode(TSNode node, uint64_t parent_id) override; + /// Emit one method/constructor parameter and record its type. + /// + /// The type binding is what lets `handleMethodInvocation` fill in + /// `receiver_type` for a call through a parameter + /// (`void use(Drawable d) { d.draw(); }`); without it the receiver + /// evidence is empty and the resolver's ambiguity gate abstains. + /// + /// \param node A `formal_parameter` / `spread_parameter` node. + /// \param parent_id Record id of the declaring method/class. + void handleFormalParameter(TSNode node, uint64_t parent_id); + private: void handleMethodDecl(TSNode node, uint64_t parent_id); void handleClassDecl(TSNode node, uint64_t parent_id); void handleInterfaceDecl(TSNode node, uint64_t parent_id); void handleEnumDecl(TSNode node, uint64_t parent_id); + /// Emit one InterfaceImpl record per type named in an `implements` + /// clause subtree. tree-sitter-java nests the interface names in a + /// `type_list` (and generic arguments in `generic_type`), so the previous + /// direct-children scan of `super_interfaces` never matched and every + /// `implements` edge was silently dropped. + /// \param node A `super_interfaces` / `type_list` subtree. + /// \param impl_type Name of the implementing class. + /// \param parent_id Record to attach the impl records to (the class). + void emitInterfaceImpls(TSNode node, const std::string &impl_type, + uint64_t parent_id); void handleMethodInvocation(TSNode node, uint64_t parent_id); /// Emit a Call record for a `new` constructor expression /// (object_creation_expression), mirroring handleMethodInvocation diff --git a/engine/src/ir/translators/javascript_translator.cpp b/engine/src/ir/translators/javascript_translator.cpp index 03e08bc..b7d7a82 100644 --- a/engine/src/ir/translators/javascript_translator.cpp +++ b/engine/src/ir/translators/javascript_translator.cpp @@ -23,6 +23,8 @@ class JavascriptTranslator : public Translator { TranslationUnit *unit_ = nullptr; const char *source_ = nullptr; std::string file_path_; + // Recursion counter for this file (see kMaxTranslateDepth). + TranslateDepth depth_; struct Scope { std::unordered_map symbols; @@ -79,6 +81,7 @@ TranslationUnit *JavascriptTranslator::translate(TSTree *tree, const char *source, const char *file_path) { + depth_.reset(); unit_ = new TranslationUnit(); unit_->source_content = source; source_ = source; @@ -327,6 +330,13 @@ Node *JavascriptTranslator::translateNode(TSNode ts_node, Node *parent) void JavascriptTranslator::translateChildren(TSNode ts_node, Node *parent) { + // Bound native-stack recursion (see kMaxTranslateDepth): a pathologically + // deep AST would overflow the indexer's 512 KB worker stack, which the FFI + // try/catch cannot recover. The visitors apply the same bound. + if (depth_.exceeded(file_path_.c_str(), "translateChildren")) + return; + DepthGuard depth_guard(depth_); + uint32_t count = ts_node_child_count(ts_node); for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(ts_node, i); diff --git a/engine/src/ir/translators/js_visitor.cpp b/engine/src/ir/translators/js_visitor.cpp index b22324d..a2f7921 100644 --- a/engine/src/ir/translators/js_visitor.cpp +++ b/engine/src/ir/translators/js_visitor.cpp @@ -1,8 +1,8 @@ #include "js_visitor.h" +#include #include #include -#include "../builtin_registry.h" #include "ahocorasick.h" namespace ir @@ -14,62 +14,31 @@ namespace ir // internal/cbm/lsp/ts_lsp.c :: builtins[] (type names) // internal/cbm/lsp/c_lsp.c :: is_c_builtin_func() (pattern) // -// JS/TS global built-in functions and constructors. These are NOT -// user-defined functions and should not create reference entries. -// The Resolver Pipeline would otherwise match them by name to any -// project entity with the same name, producing false positives. -static bool isJsBuiltin(const std::string &name) -{ - static const char *kBuiltins[] = { - // Global built-in functions - "eval", - "parseInt", - "parseFloat", - "isNaN", - "isFinite", - "decodeURI", - "decodeURIComponent", - "encodeURI", - "encodeURIComponent", - "escape", - "unescape", - // Built-in constructors (used as functions) - "Array", - "Boolean", - "Date", - "Error", - "Function", - "Map", - "Number", - "Object", - "Promise", - "RegExp", - "Set", - "String", - "Symbol", - "WeakMap", - "WeakSet", - "BigInt", - "Infinity", - "NaN", - "undefined", - "null", - nullptr, - }; - for (const char **b = kBuiltins; *b; b++) { - if (name == *b) - return true; - } - return false; -} - // ── Aho-Corasick dispatch table for visitNode ────────────────── -static const ACAutomaton &getJsAC() -{ - static ACAutomaton ac; - static bool built = false; - if (!built) { +// Builds the dispatch automaton in its constructor so the shared instance can +// be a function-local static object: C++11 guarantees such an object is +// initialised exactly once, with other threads blocking until the first +// finishes. +// +// The previous form, +// +// static ACAutomaton ac; +// static bool built = false; +// if (!built) { ...addPattern()...; ac.build(); built = true; } +// +// was an unsynchronised lazy init. Parsing runs on std::thread workers +// (engine_index_files.cpp), so two threads reaching their first JavaScript file +// at the same moment both ran addPattern()/build() on the SAME automaton, each +// writing next[]/fail/out_link while the other read them. +// +// The automaton is built in place rather than returned by value because it owns +// raw nodes — copying it would double-free them (see ahocorasick.h). +struct JsACHolder { + ACAutomaton ac; + + JsACHolder() + { // Handlers (produce semantic records) ac.addPattern("function_declaration", 100); ac.addPattern("generator_function_declaration", 100); @@ -107,7 +76,7 @@ static const ACAutomaton &getJsAC() ac.addPattern("switch_statement", 300); ac.addPattern("switch_case", 300); ac.addPattern("try_statement", 300); - ac.addPattern("catch_clause", 300); + ac.addPattern("catch_clause", 111); ac.addPattern("throw_statement", 300); ac.addPattern("binary_expression", 300); ac.addPattern("unary_expression", 300); @@ -118,15 +87,23 @@ static const ACAutomaton &getJsAC() ac.addPattern("await_expression", 300); ac.addPattern("yield_expression", 300); ac.build(); - built = true; } - return ac; +}; + +static const ACAutomaton &getJsAC() +{ + static const JsACHolder holder; + return holder.ac; } JsVisitor::JsVisitor() { } +// The definition-node tables and JsVisitor::collectDefinedNames live in +// js_visitor_defined_names.cpp (1000-line rule). visit() below calls it to +// fill defined_names_, which the builtin-name filters consult. + SemanticUnit *JsVisitor::visit(TSTree *tree, const char *source, const char *file_path) { @@ -139,7 +116,11 @@ SemanticUnit *JsVisitor::visit(TSTree *tree, const char *source, source_ = source; TSNode root_node = ts_tree_root_node(tree); + defined_names_.clear(); + collectDefinedNames(root_node); + visit_depth_ = 0; + depth_truncated_ = false; pushScope(); // Emit TranslationUnit as root record (parent_id = 0) SourceRange root_loc = location(root_node); @@ -163,6 +144,10 @@ void JsVisitor::reset() // Step 4: reset per-file tracking. var_types_.clear(); class_scope_stack_.clear(); + import_aliases_.clear(); + import_symbol_aliases_.clear(); + visit_depth_ = 0; + depth_truncated_ = false; unit_ = nullptr; emitter_ = nullptr; source_ = nullptr; @@ -250,8 +235,31 @@ void JsVisitor::visitChildren(TSNode node, uint64_t parent_id) TSNode child = ts_node_child(node, i); if (!ts_node_is_named(child)) continue; - visitNode(child, parent_id); + visitChild(child, parent_id); + } +} + +void JsVisitor::visitChild(TSNode node, uint64_t parent_id) +{ + // Bound native-stack recursion so a pathologically deep AST cannot + // SIGSEGV the indexer (the FFI try/catch cannot recover a stack + // overflow). Report the truncation once per file rather than dropping + // nodes silently. [module=ir, method=visitChild] + if (visit_depth_ >= kMaxVisitDepth) { + if (!depth_truncated_) { + depth_truncated_ = true; + const char *fp = unit_ ? unit_->filePath().c_str() : ""; + fprintf(stderr, + "[module=ir, method=visitChild] AST nesting " + "exceeded kMaxVisitDepth=%d in '%s'; deeper nodes " + "skipped to avoid stack overflow\n", + kMaxVisitDepth, fp); + } + return; } + ++visit_depth_; + visitNode(node, parent_id); + --visit_depth_; } // ── Node dispatcher ─────────────────────────────────────────── @@ -285,6 +293,8 @@ void JsVisitor::visitNode(TSNode node, uint64_t parent_id) return visitMemberExpr(node, parent_id); case 110: return visitNewExpr(node, parent_id); + case 111: + return visitCatchClause(node, parent_id); // ── Literals ────────────────────────────────────────── case 200: @@ -361,6 +371,48 @@ void JsVisitor::visitArrowFunction(TSNode node, uint64_t parent_id) popScope(); } +// ── Catch clause (empty-catch evidence) ───────────────────────── +// +// Emits one Comment-kind record named "catch" with empty qualified_name +// when the catch body holds no named statements. extractErrorFacts +// (semantic_fact_extractor.cpp) selects exactly that shape — +// name='catch', qualified_name='' — to produce empty_catch facts. +// A non-empty body is evidence the error was handled, so no record is +// emitted and the pass-through recursion below still visits the body. +void JsVisitor::visitCatchClause(TSNode node, uint64_t parent_id) +{ + SourceRange loc = location(node); + bool body_empty = true; + uint32_t count = ts_node_child_count(node); + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(node, i); + const char *t = ts_node_type(child); + if (strcmp(t, "statement_block") != 0) + continue; + uint32_t body_count = ts_node_child_count(child); + for (uint32_t j = 0; j < body_count; j++) { + TSNode stmt = ts_node_child(child, j); + if (!ts_node_is_named(stmt)) + continue; + // Comments are named nodes but are not statements — + // a comment-only body still counts as empty. + if (strcmp(ts_node_type(stmt), "comment") == 0) + continue; + body_empty = false; + break; + } + break; + } + if (body_empty) { + // Comment kind is persisted by insertFileResultBatch (unlike + // Variable/Literal) and extractErrorFacts matches on name + + // qualified_name only, so this is the cheapest record shape + // that reaches the evidence layer. + emitter_->emitComment("catch", loc, parent_id); + } + visitChildren(node, parent_id); +} + void JsVisitor::visitClassDecl(TSNode node, uint64_t parent_id) { SourceRange loc = location(node); @@ -406,6 +458,19 @@ void JsVisitor::visitMethodDef(TSNode node, uint64_t parent_id) uint64_t method_id = emitter_->emitMethod( name, loc, parent_id, 0, false, detectVisibility(node)); defineSymbol(name, method_id); + // The resolver matches a call's receiver_type against the candidate's + // DECLARING type, which it reads from qualified_name (the C++ and Rust + // visitors set it for exactly this reason). With the bare method name + // there, `TsOwner::method` and `TsDecoy::method` look identical, so a call + // whose receiver type IS known still had no candidate to prefer and the + // ambiguity gate abstained — measured for TypeScript in + // test_resolver_language_consistency. JavaScript class methods get the same + // treatment, which is why this lives in the shared visitor. + { + const std::string owner = currentClassName(); + if (!owner.empty()) + unit_->setQualifiedName(method_id, owner + "::" + name); + } pushScope(); pushFunctionScope(method_id); @@ -429,154 +494,6 @@ void JsVisitor::visitMethodDef(TSNode node, uint64_t parent_id) popScope(); } -void JsVisitor::visitCallExpr(TSNode node, uint64_t parent_id) -{ - SourceRange loc = location(node); - std::string callee_name; - bool has_member_expr = false; // obj.method() — member expression call - - uint32_t count = ts_node_child_count(node); - - // Extract callee name from first identifier child - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(node, i); - if (!ts_node_is_named(child)) - continue; - const char *t = ts_node_type(child); - // Skip property_identifier (member expression targets - // like obj.method) — they are NOT standalone function calls. - if (strcmp(t, "property_identifier") == 0) - continue; - if (strcmp(t, "identifier") == 0) { - callee_name = nodeText(child); - break; - } - // Handle member_expression: obj.method() produces a - // member_expression as the first named child (not an - // identifier). Previously this fell through, leaving - // callee_name empty and dropping the majority of JS/TS - // call edges. Extract the trailing property_identifier - // (the method name) from the member_expression, mirroring - // CVisitor::extractFieldMethodName for field_expression. - if (strcmp(t, "member_expression") == 0) { - // obj.method() — mark as a method call so the Resolver's - // CallKindMatch factor and receiver evidence apply. The - // bare method name below has no '.', so the - // callee_name.find('.') classification below would - // otherwise mislabel every method call as Direct. - has_member_expr = true; - uint32_t mc = ts_node_child_count(child); - for (uint32_t j = 0; j < mc; j++) { - TSNode mchild = ts_node_child(child, j); - if (!ts_node_is_named(mchild)) - continue; - const char *mt = ts_node_type(mchild); - if (strcmp(mt, "property_identifier") == 0 || - strcmp(mt, - "shorthand_property_identifier") == - 0) { - callee_name = nodeText(mchild); - } - } - break; - } - } - - // Skip JS/TS built-in global functions — they are NOT user-defined - // calls and the Resolver Pipeline would generate false-positive edges. - // Reference: codebase-memory-mcp (MIT) ts_lsp.c :: builtins[] - if (!callee_name.empty() && isJsBuiltin(callee_name)) { - // Still visit children to pick up nested calls/expressions - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(node, i); - if (!ts_node_is_named(child)) - continue; - const char *t = ts_node_type(child); - if (strcmp(t, "identifier") == 0) - continue; - visitNode(child, parent_id); - } - return; - } - - // Classify call kind. obj.method() member-expression calls carry only - // the bare method name (property_identifier), so callee_name has no - // '.' — without has_member_expr every method call was mislabeled - // Direct, skipping the Resolver's CallKindMatch factor and receiver - // evidence. Mark them Method explicitly. - CallKind call_kind = CallKind::Direct; - if (has_member_expr || callee_name.find('.') != std::string::npos) - call_kind = CallKind::Method; - // Constructor detection: any non-empty capitalized name. The previous - // `callee_name.size() > 3` threshold skipped short class names like - // `Foo()`, `Url()`, `Db()` → all were misclassified as Direct calls, - // so the Resolver Pipeline never applied the constructor boost factor - // and cross-module constructor resolution silently failed. - // See CODE_REVIEW_FINDINGS_2026-07-19.md H6. - else if (callee_name.size() >= 1 && callee_name[0] >= 'A' && - callee_name[0] <= 'Z') - call_kind = CallKind::Constructor; - - // Use the containing function as parent_id (not the immediate - // syntactic parent, which may be another call record). This - // ensures nested calls' parent_id points to a declaration - // record present in _r2n, so the reference-table JOIN succeeds. - uint64_t func_id = currentFunctionId(); - uint64_t call_parent = (func_id != 0) ? func_id : parent_id; - - // Compute arity from the `arguments` child node's named children. - // Previously hardcoded to 0, which degraded overload disambiguation - // by arity in the Resolver Pipeline. Mirrors CVisitor::countArguments. - int arity = 0; - for (uint32_t i = 0; i < count; i++) { - TSNode c = ts_node_child(node, i); - if (strcmp(ts_node_type(c), "arguments") != 0) - continue; - uint32_t ac = ts_node_child_count(c); - for (uint32_t j = 0; j < ac; j++) { - TSNode arg = ts_node_child(c, j); - if (ts_node_is_named(arg)) - ++arity; - } - break; - } - - uint64_t call_id = emitter_->emitCall(callee_name, loc, call_parent, - arity, false, - static_cast(call_kind)); - - // ── Intra-file callee resolution ─────────────────────────── - // Store the resolved callee's record ID as ref_original_id on - // the CallExpr. Enables P1 call-edge construction in - // buildCallEdgesSQL (JOIN on ref_original_id > 0). - if (!callee_name.empty()) { - uint64_t target = resolveSymbol(callee_name); - if (target) { - unit_->setCallReference(call_id, target); - unit_->setCallStrategy(call_id, "p1_intra"); - } else { - unit_->setCallStrategy( - call_id, - BuiltinRegistry::resolve(unit_->language(), - callee_name)); - } - } - - // Recurse into children (arguments, member expressions, etc.) - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(node, i); - if (!ts_node_is_named(child)) - continue; - const char *t = ts_node_type(child); - if (strcmp(t, "identifier") == 0) { - // Already extracted above — skip to avoid - // creating an extra identifier record. - continue; - } - visitNode(child, call_id); - } -} - void JsVisitor::visitIdentifier(TSNode node, uint64_t parent_id) { // Identifiers are only emitted when they carry semantic value @@ -618,13 +535,13 @@ void JsVisitor::visitVariableDecl(TSNode node, uint64_t parent_id) "identifier") == 0) continue; if (ts_node_is_named(decl)) - visitNode(decl, parent_id); + visitChild(decl, parent_id); } } else { visitChildren(child, parent_id); } } else if (ts_node_is_named(child)) { - visitNode(child, parent_id); + visitChild(child, parent_id); } } } @@ -634,10 +551,94 @@ void JsVisitor::visitImportStmt(TSNode node, uint64_t parent_id) SourceRange loc = location(node); std::string module_name = nodeText(node); emitter_->emitImport(module_name, loc, parent_id); + + // Record the locally-bound import names so visitCallExpr can emit + // import_alias evidence for `ns.fn()` / `Foo.bar()` calls. The + // module specifier (the quoted source string) is kept as the map + // value for future scope checks; today only membership matters. + std::string module_spec; + uint32_t count = ts_node_child_count(node); + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(node, i); + if (strcmp(ts_node_type(child), "string") == 0) { + module_spec = nodeText(child); + // The `string` node text keeps its quotes, and a specifier is + // only useful to a path comparison without them ("../lib/Widget"). + if (module_spec.size() >= 2 && + (module_spec.front() == '\'' || + module_spec.front() == '"') && + module_spec.back() == module_spec.front()) + module_spec = module_spec.substr( + 1, module_spec.size() - 2); + break; + } + } + collectImportBindings(node, module_spec, parent_id); // Import children (import_clause, from_clause) are structural — // no need to emit records for them. } +void JsVisitor::collectImportBindings(TSNode node, + const std::string &module_spec, + uint64_t parent_id) +{ + const char *t = ts_node_type(node); + + // `name` or `name as alias`: the bound name is the LAST identifier + // child (the alias when present, otherwise the name itself). + if (strcmp(t, "import_specifier") == 0) { + std::string imported; + std::string bound; + uint32_t cc = ts_node_child_count(node); + for (uint32_t i = 0; i < cc; i++) { + TSNode c = ts_node_child(node, i); + if (strcmp(ts_node_type(c), "identifier") != 0) + continue; + // `name` or `name as alias`: the FIRST identifier is the + // symbol the module exports, the LAST is the local name the call + // site uses. They differ only in the aliased form. + if (imported.empty()) + imported = nodeText(c); + bound = nodeText(c); + } + if (!bound.empty()) { + import_aliases_[bound] = module_spec; + if (!imported.empty() && imported != bound) + import_symbol_aliases_[bound] = imported; + // Also record the binding -> module pair for the Resolver: the + // `import` table cannot supply it, because its alias column is the + // module path's last segment rather than the name the module was + // bound to, so a bare call has no key to look its module up by. + emitter_->emitImportBinding(bound, module_spec, + location(node), parent_id); + } + return; + } + + // A bare identifier inside the import clause is either the default + // import (`import Foo from ...`) or the namespace binding + // (`import * as ns from ...`). Named-import identifiers are handled + // by the import_specifier branch above and never reach here. + if (strcmp(t, "identifier") == 0) { + std::string bound = nodeText(node); + if (!bound.empty()) { + import_aliases_[bound] = module_spec; + // Also record the binding -> module pair for the Resolver: the + // `import` table cannot supply it, because its alias column is the + // module path's last segment rather than the name the module was + // bound to, so a bare call has no key to look its module up by. + emitter_->emitImportBinding(bound, module_spec, + location(node), parent_id); + } + return; + } + + uint32_t cc = ts_node_child_count(node); + for (uint32_t i = 0; i < cc; i++) + collectImportBindings(ts_node_child(node, i), module_spec, + parent_id); +} + void JsVisitor::visitExportStmt(TSNode node, uint64_t parent_id) { SourceRange loc = location(node); @@ -655,7 +656,7 @@ void JsVisitor::visitExportStmt(TSNode node, uint64_t parent_id) strcmp(t, "class_declaration") == 0 || strcmp(t, "variable_declaration") == 0 || strcmp(t, "lexical_declaration") == 0) { - visitNode(child, export_id); + visitChild(child, export_id); } } } @@ -687,116 +688,4 @@ void JsVisitor::visitMemberExpr(TSNode node, uint64_t parent_id) visitChildren(node, member_id); } -void JsVisitor::visitNewExpr(TSNode node, uint64_t parent_id) -{ - SourceRange loc = location(node); - std::string callee_name; - uint32_t count = ts_node_child_count(node); - - // Extract the constructor name from the callee child. For `new Foo()` - // it is an identifier; for `new Foo.Bar()` it is a member_expression - // whose trailing property_identifier is the constructor. Mirrors - // visitCallExpr so the constructor resolves by bare name. - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(node, i); - if (!ts_node_is_named(child)) - continue; - const char *t = ts_node_type(child); - if (strcmp(t, "member_expression") == 0) { - uint32_t mc = ts_node_child_count(child); - for (uint32_t j = 0; j < mc; j++) { - TSNode mchild = ts_node_child(child, j); - if (!ts_node_is_named(mchild)) - continue; - const char *mt = ts_node_type(mchild); - if (strcmp(mt, "property_identifier") == 0 || - strcmp(mt, - "shorthand_property_identifier") == - 0) { - callee_name = nodeText(mchild); - } - } - break; - } - if (strcmp(t, "identifier") == 0) { - callee_name = nodeText(child); - break; - } - } - - // Unknown constructor shape — still recurse to capture nested calls. - if (callee_name.empty()) { - visitChildren(node, parent_id); - return; - } - - // Skip JS/TS built-in constructors (Array, Map, ...) — they are NOT - // user-defined calls; the Resolver Pipeline would generate FPs. - if (isJsBuiltin(callee_name)) { - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(node, i); - if (!ts_node_is_named(child)) - continue; - const char *t = ts_node_type(child); - if (strcmp(t, "identifier") == 0 || - strcmp(t, "member_expression") == 0) - continue; - visitNode(child, parent_id); - } - return; - } - - // Constructor calls are always Constructor kind. - CallKind call_kind = CallKind::Constructor; - - uint64_t func_id = currentFunctionId(); - uint64_t call_parent = (func_id != 0) ? func_id : parent_id; - - // Compute arity from the `arguments` child node (M-10 mirror). - int arity = 0; - for (uint32_t i = 0; i < count; i++) { - TSNode c = ts_node_child(node, i); - if (strcmp(ts_node_type(c), "arguments") != 0) - continue; - uint32_t ac = ts_node_child_count(c); - for (uint32_t j = 0; j < ac; j++) { - TSNode arg = ts_node_child(c, j); - if (ts_node_is_named(arg)) - ++arity; - } - break; - } - - uint64_t call_id = emitter_->emitCall(callee_name, loc, call_parent, - arity, false, - static_cast(call_kind)); - - // ── Intra-file callee resolution ─────────────────────────── - if (!callee_name.empty()) { - uint64_t target = resolveSymbol(callee_name); - if (target) { - unit_->setCallReference(call_id, target); - unit_->setCallStrategy(call_id, "p1_intra"); - } else { - unit_->setCallStrategy( - call_id, - BuiltinRegistry::resolve(unit_->language(), - callee_name)); - } - } - - // Recurse into children (arguments, nested expressions), skipping the - // already-extracted constructor identifier / member_expression. - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(node, i); - if (!ts_node_is_named(child)) - continue; - const char *t = ts_node_type(child); - if (strcmp(t, "identifier") == 0 || - strcmp(t, "member_expression") == 0) - continue; - visitNode(child, call_id); - } -} - } // namespace ir diff --git a/engine/src/ir/translators/js_visitor.h b/engine/src/ir/translators/js_visitor.h index 1f4bdf7..57cb428 100644 --- a/engine/src/ir/translators/js_visitor.h +++ b/engine/src/ir/translators/js_visitor.h @@ -3,6 +3,7 @@ #include #include +#include #include #include "../semantic_emitter.h" @@ -55,6 +56,33 @@ class JsVisitor { SemanticEmitter *emitter_; const char *source_; + // ── Recursion depth guard ─────────────────────────────── + // tree-sitter accepts pathologically deep ASTs (tens of thousands of + // nested nodes, e.g. 20k nested calls). The visitor walk is recursive + // (visitChildren → visitNode → handleX → visitChildren …), so such input + // overflows the native stack and crashes (SIGBUS/SIGSEGV) — a fault the + // FFI try/catch boundary cannot recover from. visitChildren and + // collectDefinedNames are the two choke points every descent passes + // through, so bounding their nesting bounds the whole walk. + // + // The cap MUST fit the smallest stack the walk can run on. The parallel + // indexer parses on spawned std::threads, whose default stack is 512 KB + // on macOS (vs. 8 MB for the main thread); one visit level costs up to + // ~1.5 KB (visitChild + visitNode + a language handleX frame with its + // std::string locals). 250 levels ≈ 375 KB leaves a safe margin under + // 512 KB while still being far deeper than any hand-written source. + // Shared by all language visitors via inheritance. + static constexpr int kMaxVisitDepth = 250; + int visit_depth_ = 0; + // Set once per file when a capped walk first truncates a subtree, so the + // truncation is reported exactly once instead of silently or per node. + // + // Shared by every capped walk (visitChild, collectDefinedNames, + // collectOutOfClassDefs): the first one to reach the limit reports the + // file and the others stay quiet, which keeps a deep file to a single + // diagnostic. Reset in reset() and at the top of visit(). + bool depth_truncated_ = false; + // ── Scope tracking ────────────────────────────────────── struct Scope { std::unordered_map symbols; @@ -101,6 +129,50 @@ class JsVisitor { std::unordered_map var_types_; std::vector class_scope_stack_; + // import_aliases_ maps a locally-bound import name to its module + // specifier, e.g.: + // import * as ns from './x' → "ns" → "./x" + // import Foo from './x' → "Foo" → "./x" + // import { A as B } from './x' → "B" → "./x" + // visitCallExpr uses it to emit import_alias evidence for + // `ns.fn()` / `Foo.bar()` calls, mirroring JavaVisitor::import_aliases_ + // and RustVisitor::use_aliases_. Without it JS/TS member calls + // carried no structured evidence at all. + std::unordered_map import_aliases_; + + // import_symbol_aliases_ maps a LOCAL alias to the symbol it stands for: + // `import { pick as pickLib }` binds `pickLib` to `pick`. Only the local + // name exists at the call site, so a bare `pickLib()` named no declaration + // and resolved to nothing — the module binding was recorded + // (emitImportBinding) but which symbol to look for was lost. The call is + // emitted under the imported name instead, so it resolves like any other + // call to that symbol. Namespace/default bindings are untouched: `ns.fn()` + // is a receiver call, and `import Foo from ...` has no second name. + std::unordered_map import_symbol_aliases_; + + /// The name a call is emitted under: the imported symbol when the call + /// uses a local alias, the call's own name otherwise. + std::string emittedCallName(const std::string &callee_name) const + { + auto it = import_symbol_aliases_.find(callee_name); + return it == import_symbol_aliases_.end() ? callee_name : + it->second; + } + + /** + * Record the names introduced by one import clause into + * import_aliases_. Handles default imports, namespace imports + * (`* as ns`) and named specifiers (including `name as alias`). + * \param node An import_clause / namespace_import / + * named_imports / import_specifier node. + * \param module_spec The module specifier string of the import. + * \param parent_id Record to attach the binding records to (the import + * statement's parent), so emitImportBinding lands in + * the same scope as the statement itself. + */ + void collectImportBindings(TSNode node, const std::string &module_spec, + uint64_t parent_id); + void recordVarType(const std::string &name, const std::string &type) { if (!name.empty() && !type.empty()) @@ -126,6 +198,35 @@ class JsVisitor { // ── Helpers ───────────────────────────────────────────── SourceRange location(TSNode node); std::string nodeText(TSNode node); + + /** + * Names the file being visited defines itself (functions, methods, classes, + * …), collected once per visit() by each language's visitor. + * + * The builtin-name filters below exist to keep pre-declared library + * symbols out of the graph, but they match on NAME only, so a user + * function that happens to share a builtin name — `def format()`, + * `void free()`, `format` from `import static` — had its call dropped + * entirely: a systematic false negative in the call graph. A name this file + * defines is always user code, so it survives the filter. + */ + std::unordered_set defined_names_; + + /// Collect every name this file defines, starting at the tree root. + /// Node types that introduce a name are per-language (see + /// kDefNodeTypes in js_visitor.cpp). `depth` bounds the recursive walk + /// with the same kMaxVisitDepth cap as visitChildren so a pathologically + /// deep AST cannot overflow the stack here (this walk runs before the + /// main visit). + void collectDefinedNames(TSNode node, int depth = 0); + + /// True when `name` is defined by the file being visited, i.e. a call to it + /// is a call to user code and must not be filtered as a builtin. + bool isLocallyDefined(const std::string &name) const + { + return defined_names_.count(name) != 0; + } + /** * Zero-copy node text view — borrows from source_, no heap alloc. * Use for temporary lookups (comparisons, name extraction). @@ -140,6 +241,20 @@ class JsVisitor { virtual void visitChildren(TSNode node, uint64_t parent_id); + /// Guarded descent into one node: applies the `kMaxVisitDepth` cap and + /// then dispatches to `visitNode`. + /// + /// Every recursive step must come through here instead of calling + /// `visitNode` directly — including the language handlers that recurse on + /// their own (class bodies, declaration lists, ...). Those direct calls sat + /// outside `visitChildren`'s accounting, so a deep chain of them could + /// still overflow the stack. Counting per descent makes the maximum + /// concurrent depth the AST depth. + /// + /// \param node Node to visit. + /// \param parent_id Record id to use as the node's parent. + void visitChild(TSNode node, uint64_t parent_id); + virtual void visitNode(TSNode node, uint64_t parent_id); virtual void visitFunctionDecl(TSNode node, uint64_t parent_id); @@ -158,6 +273,10 @@ class JsVisitor { /// (e.g. `new Foo(x)`), mirroring visitCallExpr. Captures the /// constructor name so constructor call-edges are not dropped (M-9). void visitNewExpr(TSNode node, uint64_t parent_id); + /// Emit an empty-catch evidence record (name='catch', empty + /// qualified_name) when the catch body has no statements, then + /// recurse. Non-empty bodies are visited without emitting. + void visitCatchClause(TSNode node, uint64_t parent_id); /// Detect JS/TS visibility: returns 1 if node is exported (wrapped in /// export_statement or marked export), else 0. v0.2.2 role classifier. int detectVisibility(TSNode node); diff --git a/engine/src/ir/translators/js_visitor_calls.cpp b/engine/src/ir/translators/js_visitor_calls.cpp new file mode 100644 index 0000000..7443790 --- /dev/null +++ b/engine/src/ir/translators/js_visitor_calls.cpp @@ -0,0 +1,400 @@ +// js_visitor_calls.cpp — call-site and constructor-call analysis. +// +// Split out of js_visitor.cpp (see plan/rules/code_rules.md 1000-line +// rule), following go_visitor.cpp / go_visitor_calls.cpp. visitCallExpr +// and visitNewExpr are the two largest handlers and the only users of +// the JS builtin filter, so they moved together with isJsBuiltin. + +#include "js_visitor.h" + +#include +#include +#include +#include "../builtin_registry.h" + +namespace ir +{ + +namespace +{ + +// JS/TS global built-in functions and constructors. These are NOT +// user-defined functions and should not create reference entries. +// The Resolver Pipeline would otherwise match them by name to any +// project entity with the same name, producing false positives. +// Reference: codebase-memory-mcp (MIT) ts_lsp.c :: builtins[] +bool isJsBuiltin(const std::string &name) +{ + static const char *kBuiltins[] = { + // Global built-in functions + "eval", + "parseInt", + "parseFloat", + "isNaN", + "isFinite", + "decodeURI", + "decodeURIComponent", + "encodeURI", + "encodeURIComponent", + "escape", + "unescape", + // Built-in constructors (used as functions) + "Array", + "Boolean", + "Date", + "Error", + "Function", + "Map", + "Number", + "Object", + "Promise", + "RegExp", + "Set", + "String", + "Symbol", + "WeakMap", + "WeakSet", + "BigInt", + "Infinity", + "NaN", + "undefined", + "null", + nullptr, + }; + for (const char **b = kBuiltins; *b; b++) { + if (name == *b) + return true; + } + return false; +} + +} // namespace + +void JsVisitor::visitCallExpr(TSNode node, uint64_t parent_id) +{ + SourceRange loc = location(node); + std::string callee_name; + std::string + receiver_text; // "obj" in obj.method(), "this" in this.method() + bool has_member_expr = false; // obj.method() — member expression call + + uint32_t count = ts_node_child_count(node); + + // Extract callee name from first identifier child + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(node, i); + if (!ts_node_is_named(child)) + continue; + const char *t = ts_node_type(child); + // Skip property_identifier (member expression targets + // like obj.method) — they are NOT standalone function calls. + if (strcmp(t, "property_identifier") == 0) + continue; + if (strcmp(t, "identifier") == 0) { + callee_name = nodeText(child); + break; + } + // Handle member_expression: obj.method() produces a + // member_expression as the first named child (not an + // identifier). Previously this fell through, leaving + // callee_name empty and dropping the majority of JS/TS + // call edges. Extract the trailing property_identifier + // (the method name) from the member_expression, mirroring + // CVisitor::extractFieldMethodName for field_expression. + if (strcmp(t, "member_expression") == 0) { + // obj.method() — mark as a method call so the Resolver's + // CallKindMatch factor and receiver evidence apply. The + // bare method name below has no '.', so the + // callee_name.find('.') classification below would + // otherwise mislabel every method call as Direct. + has_member_expr = true; + uint32_t mc = ts_node_child_count(child); + bool receiver_found = false; + for (uint32_t j = 0; j < mc; j++) { + TSNode mchild = ts_node_child(child, j); + if (!ts_node_is_named(mchild)) + continue; + const char *mt = ts_node_type(mchild); + if (strcmp(mt, "property_identifier") == 0 || + strcmp(mt, + "shorthand_property_identifier") == + 0) { + callee_name = nodeText(mchild); + } else if (!receiver_found) { + // The first named child of a member + // expression is the receiver: an identifier + // (`r`), `this`, or a nested member_expression + // for chained access (`a.b.c()` → "a.b"). + receiver_text = nodeText(mchild); + receiver_found = true; + } + } + break; + } + } + + // Skip JS/TS built-in global functions — but ONLY for bare calls. For + // `obj.method()` the extracted callee_name is the property identifier, so + // filtering it here would drop any method named `Map`, `Set`, `String`, + // `Number`, `Symbol`, `Date` … with no call record. A member call cannot + // be a global builtin, so it keeps its record and receiver evidence. + // A name this file defines or imports is user code even when it matches + // a global builtin (`function Map() {}`, `import {Map} from './m'`). + if (!has_member_expr && !callee_name.empty() && + isJsBuiltin(callee_name) && !isLocallyDefined(callee_name) && + import_aliases_.count(callee_name) == 0) { + // Still visit children to pick up nested calls/expressions + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(node, i); + if (!ts_node_is_named(child)) + continue; + const char *t = ts_node_type(child); + if (strcmp(t, "identifier") == 0) + continue; + visitChild(child, parent_id); + } + return; + } + + // Classify call kind. obj.method() member-expression calls carry only + // the bare method name (property_identifier), so callee_name has no + // '.' — without has_member_expr every method call was mislabeled + // Direct, skipping the Resolver's CallKindMatch factor and receiver + // evidence. Mark them Method explicitly. + CallKind call_kind = CallKind::Direct; + if (has_member_expr || callee_name.find('.') != std::string::npos) + call_kind = CallKind::Method; + // Constructor detection: any non-empty capitalized name. The previous + // `callee_name.size() > 3` threshold skipped short class names like + // `Foo()`, `Url()`, `Db()` → all were misclassified as Direct calls, + // so the Resolver Pipeline never applied the constructor boost factor + // and cross-module constructor resolution silently failed. + else if (callee_name.size() >= 1 && callee_name[0] >= 'A' && + callee_name[0] <= 'Z') + call_kind = CallKind::Constructor; + + // Use the containing function as parent_id (not the immediate + // syntactic parent, which may be another call record). This + // ensures nested calls' parent_id points to a declaration + // record present in _r2n, so the reference-table JOIN succeeds. + uint64_t func_id = currentFunctionId(); + uint64_t call_parent = (func_id != 0) ? func_id : parent_id; + + // Compute arity from the `arguments` child node's named children. + // Previously hardcoded to 0, which degraded overload disambiguation + // by arity in the Resolver Pipeline. Mirrors CVisitor::countArguments. + int arity = 0; + for (uint32_t i = 0; i < count; i++) { + TSNode c = ts_node_child(node, i); + if (strcmp(ts_node_type(c), "arguments") != 0) + continue; + uint32_t ac = ts_node_child_count(c); + for (uint32_t j = 0; j < ac; j++) { + TSNode arg = ts_node_child(c, j); + if (ts_node_is_named(arg)) + ++arity; + } + break; + } + + uint64_t call_id = emitter_->emitCall(emittedCallName(callee_name), loc, + call_parent, arity, false, + static_cast(call_kind)); + + // ── Step 3 (plan §3.1): structured call facts ────────────── + // Mirror the Go/Python/Java/Rust/C visitors: record the receiver, + // qualified target and import alias so the Resolver can + // disambiguate same-name methods instead of guessing. Without this, + // JS/TS reference rows carried empty evidence, which both disabled + // the fuzzy fallback (its `has_evidence` gate requires receiver_type + // / qualified_target / import_alias) and left + // factorReceiverTypeMatch neutral for every candidate. + if (!callee_name.empty() && !receiver_text.empty()) { + std::string receiver_type; + std::string import_alias; + if (receiver_text == "this" || receiver_text == "super") { + std::string cls = currentClassName(); + if (!cls.empty()) + receiver_type = cls; + } else if (import_aliases_.count(receiver_text) > 0) { + import_alias = receiver_text; + } else { + auto vt = var_types_.find(receiver_text); + if (vt != var_types_.end()) + receiver_type = vt->second; + } + std::string qualified_target = + receiver_text + "." + callee_name; + emitter_->setCallFacts(call_id, qualified_target, receiver_text, + receiver_type, import_alias); + } + + // ── Intra-file callee resolution ─────────────────────────── + // Store the resolved callee's record ID as ref_original_id on + // the CallExpr. Enables P1 call-edge construction in + // buildCallEdgesSQL (JOIN on ref_original_id > 0). + if (!callee_name.empty()) { + uint64_t target = resolveSymbol(callee_name); + if (target) { + unit_->setCallReference(call_id, target); + unit_->setCallStrategy(call_id, "p1_intra"); + } else { + unit_->setCallStrategy( + call_id, + BuiltinRegistry::resolve(unit_->language(), + callee_name)); + } + } + + // Recurse into children (arguments, member expressions, etc.) + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(node, i); + if (!ts_node_is_named(child)) + continue; + const char *t = ts_node_type(child); + if (strcmp(t, "identifier") == 0) { + // Already extracted above — skip to avoid + // creating an extra identifier record. + continue; + } + visitChild(child, call_id); + } +} + +void JsVisitor::visitNewExpr(TSNode node, uint64_t parent_id) +{ + SourceRange loc = location(node); + std::string callee_name; + std::string receiver_text; // "ns" in new ns.Foo() + uint32_t count = ts_node_child_count(node); + + // Extract the constructor name from the callee child. For `new Foo()` + // it is an identifier; for `new Foo.Bar()` it is a member_expression + // whose trailing property_identifier is the constructor. Mirrors + // visitCallExpr so the constructor resolves by bare name. + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(node, i); + if (!ts_node_is_named(child)) + continue; + const char *t = ts_node_type(child); + if (strcmp(t, "member_expression") == 0) { + uint32_t mc = ts_node_child_count(child); + bool receiver_found = false; + for (uint32_t j = 0; j < mc; j++) { + TSNode mchild = ts_node_child(child, j); + if (!ts_node_is_named(mchild)) + continue; + const char *mt = ts_node_type(mchild); + if (strcmp(mt, "property_identifier") == 0 || + strcmp(mt, + "shorthand_property_identifier") == + 0) { + callee_name = nodeText(mchild); + } else if (!receiver_found) { + // Namespace receiver of a qualified + // constructor: `new ns.Foo()` → "ns". + receiver_text = nodeText(mchild); + receiver_found = true; + } + } + break; + } + if (strcmp(t, "identifier") == 0) { + callee_name = nodeText(child); + break; + } + } + + // Unknown constructor shape — still recurse to capture nested calls. + if (callee_name.empty()) { + visitChildren(node, parent_id); + return; + } + + // Skip JS/TS built-in constructors (Array, Map, ...) — they are NOT + // user-defined calls; the Resolver Pipeline would generate FPs. Only + // apply this to the unqualified form: `new ns.Array()` names a user type + // in a namespace, not the builtin. + if (receiver_text.empty() && isJsBuiltin(callee_name) && + !isLocallyDefined(callee_name) && + import_aliases_.count(callee_name) == 0) { + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(node, i); + if (!ts_node_is_named(child)) + continue; + const char *t = ts_node_type(child); + if (strcmp(t, "identifier") == 0 || + strcmp(t, "member_expression") == 0) + continue; + visitChild(child, parent_id); + } + return; + } + + // Constructor calls are always Constructor kind. + CallKind call_kind = CallKind::Constructor; + + uint64_t func_id = currentFunctionId(); + uint64_t call_parent = (func_id != 0) ? func_id : parent_id; + + // Compute arity from the `arguments` child node (M-10 mirror). + int arity = 0; + for (uint32_t i = 0; i < count; i++) { + TSNode c = ts_node_child(node, i); + if (strcmp(ts_node_type(c), "arguments") != 0) + continue; + uint32_t ac = ts_node_child_count(c); + for (uint32_t j = 0; j < ac; j++) { + TSNode arg = ts_node_child(c, j); + if (ts_node_is_named(arg)) + ++arity; + } + break; + } + + uint64_t call_id = emitter_->emitCall(emittedCallName(callee_name), loc, + call_parent, arity, false, + static_cast(call_kind)); + + // Step 3 (plan §3.1): a namespace-qualified constructor + // (`new ns.Foo()`) carries import-alias evidence; a bare + // `new Foo()` has no receiver and correctly records nothing. + if (!callee_name.empty() && !receiver_text.empty()) { + std::string import_alias; + if (import_aliases_.count(receiver_text) > 0) + import_alias = receiver_text; + std::string qualified_target = + receiver_text + "." + callee_name; + emitter_->setCallFacts(call_id, qualified_target, receiver_text, + "", import_alias); + } + + // ── Intra-file callee resolution ─────────────────────────── + if (!callee_name.empty()) { + uint64_t target = resolveSymbol(callee_name); + if (target) { + unit_->setCallReference(call_id, target); + unit_->setCallStrategy(call_id, "p1_intra"); + } else { + unit_->setCallStrategy( + call_id, + BuiltinRegistry::resolve(unit_->language(), + callee_name)); + } + } + + // Recurse into children (arguments, nested expressions), skipping the + // already-extracted constructor identifier / member_expression. + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(node, i); + if (!ts_node_is_named(child)) + continue; + const char *t = ts_node_type(child); + if (strcmp(t, "identifier") == 0 || + strcmp(t, "member_expression") == 0) + continue; + visitChild(child, call_id); + } +} + +} // namespace ir diff --git a/engine/src/ir/translators/js_visitor_defined_names.cpp b/engine/src/ir/translators/js_visitor_defined_names.cpp new file mode 100644 index 0000000..6e8c823 --- /dev/null +++ b/engine/src/ir/translators/js_visitor_defined_names.cpp @@ -0,0 +1,160 @@ +// js_visitor_defined_names.cpp — names the file being visited declares. +// +// Split out of js_visitor.cpp (plan/rules/code_rules.md: no source file may +// exceed 1000 lines), following the go_visitor.cpp / go_visitor_calls.cpp +// precedent. +// +// Fills JsVisitor::defined_names_, which the builtin-name filters consult. Those +// filters match on NAME only, so without this set a call to a user function whose +// name collides with a builtin (`function Map() {}`, `def format()`) was dropped +// from the graph — a systematic false negative in the call graph. + +#include "js_visitor.h" + +#include +#include +#include +#include + +namespace ir +{ + +namespace +{ +/// Node types that DECLARE a name, per language, for collectDefinedNames(). +/// A call to a name declared in the same file is a call to user code even when +/// the name also appears in the language's builtin list (`def format()`, +/// `class Map`, `free()`), so the builtin filters must not drop it. +/// +/// Only types whose `name` field holds the declared name belong here; C/C++ +/// function definitions are handled by the declarator fallback in +/// collectDefinedNames(). +struct DefNodeTypes { + const char *language; + const char *const *types; +}; + +const char *const kPythonDefNodes[] = { "function_definition", + "class_definition", nullptr }; + +const char *const kJsDefNodes[] = { + "function_declaration", "generator_function_declaration", + "class_declaration", "method_definition", + "variable_declarator", nullptr +}; + +const char *const kJavaDefNodes[] = { "method_declaration", + "constructor_declaration", + "class_declaration", + "interface_declaration", + "enum_declaration", + "record_declaration", + nullptr }; + +const char *const kRustDefNodes[] = { + "function_item", "struct_item", "enum_item", "trait_item", "type_item", + "const_item", "static_item", "mod_item", nullptr +}; + +const char *const kCDefNodes[] = { "function_definition", "struct_specifier", + "enum_specifier", "type_definition", + "class_specifier", nullptr }; + +const char *const kGoDefNodes[] = { "function_declaration", + "method_declaration", "type_declaration", + nullptr }; + +/// True for node types that can hold a declared NAME on the declarator +/// fallback path below. C/C++ method definitions carry a field_identifier, +/// operator overloads an operator_name, destructors a destructor_name — +/// none of which is a plain `identifier`. +bool isDeclaredNameNode(const char *type) +{ + return strcmp(type, "identifier") == 0 || + strcmp(type, "field_identifier") == 0 || + strcmp(type, "operator_name") == 0 || + strcmp(type, "destructor_name") == 0; +} + +const DefNodeTypes kDefNodeTypes[] = { + { "python", kPythonDefNodes }, { "javascript", kJsDefNodes }, + { "typescript", kJsDefNodes }, { "tsx", kJsDefNodes }, + { "java", kJavaDefNodes }, { "rust", kRustDefNodes }, + { "c", kCDefNodes }, { "cpp", kCDefNodes }, + { "objective-c", kCDefNodes }, { "go", kGoDefNodes }, +}; +} // namespace + +void JsVisitor::collectDefinedNames(TSNode node, int depth) +{ + // Bound the recursion with the same cap as visitChildren. This walk runs + // before the main visit, so an unbounded deep AST would SIGSEGV here + // first. Report the truncation once per file. [module=ir, + // method=collectDefinedNames] + if (depth >= kMaxVisitDepth) { + if (!depth_truncated_) { + depth_truncated_ = true; + const char *fp = unit_ ? unit_->filePath().c_str() : ""; + fprintf(stderr, + "[module=ir, method=collectDefinedNames] AST " + "nesting exceeded kMaxVisitDepth=%d in '%s'; " + "deeper nodes skipped to avoid stack overflow\n", + kMaxVisitDepth, fp); + } + return; + } + const char *language = unit_ ? unit_->language().c_str() : ""; + const char *const *types = nullptr; + for (const auto &entry : kDefNodeTypes) { + if (strcmp(entry.language, language) == 0) { + types = entry.types; + break; + } + } + if (!types) + return; + + uint32_t count = ts_node_child_count(node); + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(node, i); + if (!ts_node_is_named(child)) + continue; + const char *type = ts_node_type(child); + bool declares = false; + for (const char *const *t = types; *t; t++) { + if (strcmp(type, *t) == 0) { + declares = true; + break; + } + } + if (declares) { + TSNode name_node = + ts_node_child_by_field_name(child, "name", 4); + if (ts_node_is_null(name_node)) { + // C/C++ carry the declared name inside the + // declarator chain: + // function_definition → function_declarator → + // identifier (or pointer_declarator → + // function_declarator → identifier for + // `void *free()`). + name_node = ts_node_child_by_field_name( + child, "declarator", 10); + while (!ts_node_is_null(name_node) && + !isDeclaredNameNode( + ts_node_type(name_node)) && + ts_node_named_child_count(name_node) > 0) + name_node = ts_node_named_child( + name_node, 0); + } + if (!ts_node_is_null(name_node) && + isDeclaredNameNode(ts_node_type(name_node))) { + std::string name = nodeText(name_node); + if (!name.empty()) + defined_names_.insert(name); + } + } + collectDefinedNames(child, depth + 1); + } +} + +} // namespace ir diff --git a/engine/src/ir/translators/python_translator.cpp b/engine/src/ir/translators/python_translator.cpp index cb9e2c8..504ccf3 100644 --- a/engine/src/ir/translators/python_translator.cpp +++ b/engine/src/ir/translators/python_translator.cpp @@ -25,6 +25,8 @@ class PythonTranslator : public Translator { TranslationUnit *unit_ = nullptr; const char *source_ = nullptr; std::string file_path_; + // Recursion counter for this file (see kMaxTranslateDepth). + TranslateDepth depth_; // Scope stack for name resolution struct Scope { @@ -61,6 +63,7 @@ class PythonTranslator : public Translator { TranslationUnit *PythonTranslator::translate(TSTree *tree, const char *source, const char *file_path) { + depth_.reset(); unit_ = new TranslationUnit(); unit_->source_content = source; source_ = source; @@ -220,6 +223,13 @@ Node *PythonTranslator::translateNode(TSNode ts_node, Node *parent) void PythonTranslator::translateChildren(TSNode ts_node, Node *parent) { + // Bound native-stack recursion (see kMaxTranslateDepth): a pathologically + // deep AST would overflow the indexer's 512 KB worker stack, which the FFI + // try/catch cannot recover. The visitors apply the same bound. + if (depth_.exceeded(file_path_.c_str(), "translateChildren")) + return; + DepthGuard depth_guard(depth_); + uint32_t count = ts_node_child_count(ts_node); for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(ts_node, i); diff --git a/engine/src/ir/translators/python_visitor.cpp b/engine/src/ir/translators/python_visitor.cpp index 3d3002f..dd7f094 100644 --- a/engine/src/ir/translators/python_visitor.cpp +++ b/engine/src/ir/translators/python_visitor.cpp @@ -71,6 +71,10 @@ SemanticUnit *PythonVisitor::visit(TSTree *tree, const char *source, class_scope_stack_.clear(); TSNode root_node = ts_tree_root_node(tree); + // Names this file defines, so a call to `format`/`next`/`set` ... that the + // file itself declares is not mistaken for the Python builtin. + defined_names_.clear(); + collectDefinedNames(root_node); pushScope(); SourceRange root_loc = location(root_node); uint64_t root_id = emitter_->emitVariable("", root_loc, 0); @@ -95,8 +99,44 @@ void PythonVisitor::visitNode(TSNode node, uint64_t parent_id) return handleImport(node, parent_id); if (strcmp(type, "assignment") == 0) return handleAssignment(node, parent_id); + if (strcmp(type, "except_clause") == 0) + return handleExceptClause(node, parent_id); JsVisitor::visitNode(node, parent_id); } + +// ── Except clause (bare-except evidence) ───────────────────────── +// +// Emits one Comment-kind record named "except" with empty qualified_name +// when the clause catches nothing (`except:`). extractErrorFacts selects +// exactly that shape — name='except', qualified_name='' — to produce +// bare_except facts. A typed handler (`except ValueError:`) is evidence +// the error was classified, so no record is emitted. +void PythonVisitor::handleExceptClause(TSNode node, uint64_t parent_id) +{ + // tree-sitter-python except_clause children: "except" keyword, + // an optional type expression, an optional "as" pattern, then the + // block. A bare `except:` has no named child before the block. + bool bare = true; + uint32_t count = ts_node_child_count(node); + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(node, i); + const char *t = ts_node_type(child); + if (strcmp(t, "block") == 0) + break; + // "except" keyword is anonymous; any named node before the + // block is the caught type or the `as` pattern. + if (ts_node_is_named(child)) { + bare = false; + break; + } + } + if (bare) { + // Comment kind is persisted by insertFileResultBatch and + // extractErrorFacts matches on name + qualified_name only. + emitter_->emitComment("except", location(node), parent_id); + } + visitChildren(node, parent_id); +} void PythonVisitor::handleFuncDef(TSNode node, uint64_t parent_id) { SourceRange loc = location(node); @@ -313,10 +353,19 @@ void PythonVisitor::handleCall(TSNode node, uint64_t parent_id) } } - // Skip Python built-in functions — they are NOT user-defined calls - // and the Resolver Pipeline would generate false-positive edges. + // Skip Python built-in functions — but ONLY for bare calls. For + // `obj.method()` the extracted `name` is the attribute (method) name, so + // matching it against the builtin list dropped every user-defined method + // named `format` / `next` / `set` / `map` / `filter` / `min` / `max` / + // `type` / `sum` with no call record at all. An attribute call can never + // be a pre-declared builtin, so it keeps its record (and its + // receiver/qualified_target evidence) for the Resolver. // Reference: codebase-memory-mcp (MIT) helpers.c :: python_resolvable_builtins - if (!name.empty() && isPythonBuiltin(name)) { + // A name this file defines, or one it imported, is user code even when it + // matches a builtin: `def format()` then `format(x)` is a real call edge + // that this filter used to delete. + if (!is_attribute_call && !name.empty() && isPythonBuiltin(name) && + !isLocallyDefined(name) && import_aliases_.count(name) == 0) { visitChildren(node, parent_id); return; } @@ -330,7 +379,6 @@ void PythonVisitor::handleCall(TSNode node, uint64_t parent_id) // `Url()`, `Db()` → all were misclassified as Direct calls, so the // Resolver Pipeline never applied the constructor boost factor and // cross-module constructor resolution silently failed. - // See CODE_REVIEW_FINDINGS_2026-07-19.md H6. else if (name.size() >= 1 && name[0] >= 'A' && name[0] <= 'Z') call_kind = CallKind::Constructor; @@ -477,7 +525,7 @@ void PythonVisitor::handleAssignment(TSNode node, uint64_t parent_id) // "self.data = self._load_data()" are detected. // Previously, handleAssignment only looked for identifier // children, so calls in the RHS were silently skipped. - visitNode(c, parent_id); + visitChild(c, parent_id); } } } @@ -499,36 +547,35 @@ std::string PythonVisitor::extractAttributeName(TSNode attr) { // tree-sitter-python attribute children for "obj.method": // identifier (obj), identifier (method) - // For chained access "self.fig.add_trace()" the object is - // itself a nested attribute, so the attribute's first child - // is another attribute. We recurse into any attribute child - // first, and fall back to the LAST named identifier on the - // current level. This ensures the resolved name is the - // method being invoked ("add_trace"), not the receiver - // ("self.fig" or "fig"). Without recursion, "self.fig.add_trace" - // yielded name "fig" (or worse, "self"), so resolveSymbol() - // never matched the method definition → ref_original_id = 0 - // → P1 call-edge construction skipped. + // For a chained access the receiver is itself an attribute: + // attribute(attribute(self, helper), compute) + // + // The name being accessed is always the LAST identifier of the OUTERMOST + // attribute, so read the current level first and use the receiver's name + // only when this level has no identifier (e.g. a subscript-style callee). + // + // The previous version recursed into the nested attribute FIRST and + // returned that result, so `self.helper.compute()` resolved to "helper" — + // the receiver of the call — instead of "compute". resolveSymbol() then + // never matched the method definition, leaving ref_original_id = 0 and + // skipping P1 call-edge construction for every chained call. std::string last; + std::string inner; uint32_t cnt = ts_node_child_count(attr); for (uint32_t i = 0; i < cnt; i++) { TSNode c = ts_node_child(attr, i); if (!ts_node_is_named(c)) continue; const char *t = ts_node_type(c); - if (strcmp(t, "attribute") == 0) { - // Recurse into nested attribute first — its - // result is more specific than any identifier - // at the current level. - std::string inner = extractAttributeName(c); - if (!inner.empty()) - return inner; - } if (strcmp(t, "identifier") == 0) last = nodeText(c); + else if (inner.empty() && strcmp(t, "attribute") == 0) + inner = extractAttributeName(c); } if (!last.empty()) return last; + if (!inner.empty()) + return inner; // Fallback: no identifier child (e.g. subscript-style callee). // Return the full attribute text so downstream resolution can // still attempt a name-only match. diff --git a/engine/src/ir/translators/python_visitor.h b/engine/src/ir/translators/python_visitor.h index 16dfa8d..72eaa6e 100644 --- a/engine/src/ir/translators/python_visitor.h +++ b/engine/src/ir/translators/python_visitor.h @@ -22,15 +22,21 @@ class PythonVisitor : public JsVisitor { void handleCall(TSNode node, uint64_t parent_id); void handleImport(TSNode node, uint64_t parent_id); void handleAssignment(TSNode node, uint64_t parent_id); + /// Emit a bare-except evidence record (name='except', empty + /// qualified_name) when the clause catches nothing, then recurse. + void handleExceptClause(TSNode node, uint64_t parent_id); std::string extractName(TSNode node); /// Extract the method name from an attribute callee. /// For "obj.method(...)" the attribute's named children are: /// identifier (obj), identifier (method). + /// For a chained callee (`self.helper.compute()`) the receiver is a nested + /// attribute, so the name taken is the LAST identifier of the OUTERMOST + /// attribute ("compute"); the receiver's name is only used when the + /// current level has no identifier (e.g. a subscript-style callee). /// Returns the LAST identifier text ("method") so resolveSymbol() - /// can match the method definition. Returns "" if no identifier - /// child is found. Falls back to the full attribute text for - /// non-identifier attribute children (e.g. subscript expressions). + /// can match the method definition. Falls back to the full attribute text + /// for non-identifier attribute children (e.g. subscript expressions). /// \param attr The attribute node (callee of a call). std::string extractAttributeName(TSNode attr); diff --git a/engine/src/ir/translators/rust_translator.cpp b/engine/src/ir/translators/rust_translator.cpp index c77cb83..1e83326 100644 --- a/engine/src/ir/translators/rust_translator.cpp +++ b/engine/src/ir/translators/rust_translator.cpp @@ -23,6 +23,8 @@ class RustTranslator : public Translator { TranslationUnit *unit_ = nullptr; const char *source_ = nullptr; std::string file_path_; + // Recursion counter for this file (see kMaxTranslateDepth). + TranslateDepth depth_; struct Scope { std::unordered_map symbols; @@ -79,6 +81,7 @@ class RustTranslator : public Translator { TranslationUnit *RustTranslator::translate(TSTree *tree, const char *source, const char *file_path) { + depth_.reset(); unit_ = new TranslationUnit(); unit_->source_content = source; source_ = source; @@ -301,6 +304,13 @@ Node *RustTranslator::translateNode(TSNode ts_node, Node *parent) void RustTranslator::translateChildren(TSNode ts_node, Node *parent) { + // Bound native-stack recursion (see kMaxTranslateDepth): a pathologically + // deep AST would overflow the indexer's 512 KB worker stack, which the FFI + // try/catch cannot recover. The visitors apply the same bound. + if (depth_.exceeded(file_path_.c_str(), "translateChildren")) + return; + DepthGuard depth_guard(depth_); + uint32_t count = ts_node_child_count(ts_node); for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(ts_node, i); @@ -418,7 +428,7 @@ Node *RustTranslator::handleImpl(TSNode ts_node, Node *parent) // the first type_identifier IS the implementing type. We scan all // children: if a `for` keyword appears, the type_identifier AFTER // it is the implementing type; otherwise we use the first - // type_identifier. See CODE_REVIEW_FINDINGS_2026-07-19.md H7. + // type_identifier. std::string impl_type; std::string first_type; bool seen_for_keyword = false; diff --git a/engine/src/ir/translators/rust_visitor.cpp b/engine/src/ir/translators/rust_visitor.cpp index 2af4502..a4369a7 100644 --- a/engine/src/ir/translators/rust_visitor.cpp +++ b/engine/src/ir/translators/rust_visitor.cpp @@ -86,6 +86,9 @@ SemanticUnit *RustVisitor::visit(TSTree *tree, const char *source, use_aliases_.clear(); TSNode root_node = ts_tree_root_node(tree); + // Names this file defines (see collectDefinedNames). + defined_names_.clear(); + collectDefinedNames(root_node); pushScope(); SourceRange root_loc = location(root_node); uint64_t root_id = emitter_->emitVariable("", root_loc, 0); @@ -111,6 +114,8 @@ void RustVisitor::visitNode(TSNode node, uint64_t parent_id) return handleImpl(node, parent_id); if (strcmp(type, "call_expression") == 0) return handleCall(node, parent_id); + if (strcmp(type, "macro_invocation") == 0) + return handleMacro(node, parent_id); if (strcmp(type, "let_declaration") == 0) return handleLet(node, parent_id); if (strcmp(type, "use_declaration") == 0) @@ -138,12 +143,61 @@ void RustVisitor::handleFunction(TSNode node, uint64_t parent_id) const char *t = ts_node_type(c); if (strcmp(t, "identifier") == 0) continue; - if (strcmp(t, "parameters") == 0 || strcmp(t, "block") == 0) + if (strcmp(t, "parameters") == 0) { + recordParameterTypes(c, id); visitChildren(c, id); + } else if (strcmp(t, "block") == 0) { + visitChildren(c, id); + } } popFunctionScope(); popScope(); } +std::string RustVisitor::firstIdentifier(TSNode node, int depth) +{ + if (ts_node_is_null(node) || depth > 4) + return ""; + if (strcmp(ts_node_type(node), "identifier") == 0) + return nodeText(node); + const uint32_t count = ts_node_child_count(node); + for (uint32_t i = 0; i < count; ++i) { + TSNode child = ts_node_child(node, i); + if (!ts_node_is_named(child)) + continue; + const std::string got = firstIdentifier(child, depth + 1); + if (!got.empty()) + return got; + } + return ""; +} + +void RustVisitor::recordParameterTypes(TSNode params, uint64_t fn_id) +{ + const uint32_t count = ts_node_child_count(params); + for (uint32_t i = 0; i < count; ++i) { + TSNode param = ts_node_child(params, i); + if (!ts_node_is_named(param) || + strcmp(ts_node_type(param), "parameter") != 0) + continue; + TSNode type_node = + ts_node_child_by_field_name(param, "type", 4); + TSNode pattern = + ts_node_child_by_field_name(param, "pattern", 7); + if (ts_node_is_null(type_node) || ts_node_is_null(pattern)) + continue; + const std::string pin_name = firstIdentifier(pattern, 0); + if (pin_name.empty()) + continue; + const std::string type_text = nodeText(type_node); + // Same pair as the local-declaration path: the TypeRef puts the binding + // in the resolver's variable-type table, and var_types_ feeds the + // receiver_type of a call on this parameter. + emitter_->emitTypeRef(pin_name, type_text, location(pattern), + fn_id); + recordVarType(pin_name, normalizeTypeName(type_text)); + } +} + void RustVisitor::handleStruct(TSNode node, uint64_t parent_id) { SourceRange loc = location(node); @@ -260,6 +314,16 @@ void RustVisitor::handleImpl(TSNode node, uint64_t parent_id) false, detectVisibility(fn_node)); defineSymbol(name, id); + // Same purpose as the C++ visitor's qualified_name: the resolver matches + // a call's receiver_type against the candidate's DECLARING type + // (factorReceiverTypeMatch). With the bare method name as the whole + // qualified_name, `RsOwner::method` and `RsDecoy::method` look + // identical, so a call whose receiver type is `RsOwner` had no candidate + // to prefer and the ambiguity gate abstained — even though the call site + // did record receiver_type (measured in + // test_resolver_language_consistency). + if (!self_type.empty()) + unit_->setQualifiedName(id, self_type + "::" + name); pushScope(); pushFunctionScope(id); uint32_t cc = ts_node_child_count(fn_node); @@ -270,9 +334,12 @@ void RustVisitor::handleImpl(TSNode node, uint64_t parent_id) const char *t = ts_node_type(gc); if (strcmp(t, "identifier") == 0) continue; - if (strcmp(t, "parameters") == 0 || - strcmp(t, "block") == 0) + if (strcmp(t, "parameters") == 0) { + recordParameterTypes(gc, id); + visitChildren(gc, id); + } else if (strcmp(t, "block") == 0) { visitChildren(gc, id); + } } popFunctionScope(); popScope(); @@ -363,7 +430,14 @@ void RustVisitor::handleCall(TSNode node, uint64_t parent_id) // Reference: codebase-memory-mcp (MIT) c_lsp.c :: is_c_builtin_func() (pattern) // Match against the full qualified text so `Type::new` (whose bare name // `new` is in the builtin list) is NOT wrongly filtered out. - if (!qualified.empty() && isRustBuiltin(qualified)) { + // + // A name THIS FILE defines is user code whatever the builtin list says — + // `fn drop()`, `macro_rules! println` — so it survives the filter. Same + // exemption as the other four translators (see collectDefinedNames); the + // bare name is what the file declares, so the check uses `name`, while the + // filter itself keeps matching the qualified text. + if (!qualified.empty() && isRustBuiltin(qualified) && + !isLocallyDefined(name)) { visitChildren(node, parent_id); return; } @@ -467,7 +541,92 @@ void RustVisitor::handleCall(TSNode node, uint64_t parent_id) strcmp(t, "field_expression") == 0 || strcmp(t, "scoped_identifier") == 0) continue; - visitNode(c, id); + visitChild(c, id); + } +} +void RustVisitor::handleMacro(TSNode node, uint64_t parent_id) +{ + // tree-sitter-rust macro_invocation children: + // identifier (or scoped_identifier) — the macro name + // token_tree — the delimiter group + arguments + SourceRange loc = location(node); + std::string qualified; + std::string name; + uint32_t cnt = ts_node_child_count(node); + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(node, i); + if (!ts_node_is_named(c)) + continue; + const char *t = ts_node_type(c); + if (strcmp(t, "identifier") == 0 || + strcmp(t, "scoped_identifier") == 0) { + qualified = nodeText(c); + name = bareCalleeName(qualified); + break; + } + } + if (name.empty()) { + visitChildren(node, parent_id); + return; + } + + // Same builtin filter as handleCall: the standard-library macros + // (println!, vec!, assert_eq!, ...) must not fabricate edges, but a macro + // this file declares itself is user code and survives. + if (isRustBuiltin(qualified) && !isLocallyDefined(name)) { + visitChildren(node, parent_id); + return; + } + + CallKind call_kind = CallKind::Direct; + if (qualified.find("::") != std::string::npos) + call_kind = CallKind::Method; + + uint64_t func_id = currentFunctionId(); + uint64_t call_parent = (func_id != 0) ? func_id : parent_id; + + uint64_t id = emitter_->emitCall(name, loc, call_parent, 0, false, + static_cast(call_kind)); + + // Structured call facts for a path-qualified macro (`crate::log!(...)`). + if (!qualified.empty() && qualified != name) { + std::string receiver_text = + extractReceiverText(node, qualified); + std::string receiver_type; + std::string import_alias; + if (!receiver_text.empty()) { + if (use_aliases_.count(receiver_text) > 0) + import_alias = receiver_text; + else { + auto vt = var_types_.find(receiver_text); + if (vt != var_types_.end()) + receiver_type = vt->second; + } + } + emitter_->setCallFacts(id, qualified, receiver_text, + receiver_type, import_alias); + } + + uint64_t target = resolveSymbol(name); + if (target) { + unit_->setCallReference(id, target); + unit_->setCallStrategy(id, "p1_intra"); + } else { + unit_->setCallStrategy( + id, BuiltinRegistry::resolve(unit_->language(), name)); + } + + // Recurse into the token tree so nested invocations and calls in the + // macro arguments are still visited. + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(node, i); + if (!ts_node_is_named(c)) + continue; + const char *t = ts_node_type(c); + if (strcmp(t, "identifier") == 0 || + strcmp(t, "scoped_identifier") == 0) + continue; + visitChild(c, id); } } void RustVisitor::handleLet(TSNode node, uint64_t parent_id) @@ -529,7 +688,7 @@ void RustVisitor::handleLet(TSNode node, uint64_t parent_id) strcmp(t, "array_type") == 0 || strcmp(t, "tuple_type") == 0) continue; - visitNode(c, parent_id); + visitChild(c, parent_id); } } void RustVisitor::handleUse(TSNode node, uint64_t parent_id) diff --git a/engine/src/ir/translators/rust_visitor.h b/engine/src/ir/translators/rust_visitor.h index 8738bc5..9d3725e 100644 --- a/engine/src/ir/translators/rust_visitor.h +++ b/engine/src/ir/translators/rust_visitor.h @@ -23,6 +23,13 @@ class RustVisitor : public JsVisitor { void handleTrait(TSNode node, uint64_t parent_id); void handleImpl(TSNode node, uint64_t parent_id); void handleCall(TSNode node, uint64_t parent_id); + /// Emit a Call record for a `macro_invocation` (`println!(...)`, + /// `vec![...]`, `foo!()`). Rust macros are a first-class call construct: + /// without this branch every macro invocation produced no record at all, + /// so calls made *inside* a macro body were invisible to the call graph. + /// Built-in macros are still filtered by isRustBuiltin(); a macro the file + /// declares itself survives via isLocallyDefined(). + void handleMacro(TSNode node, uint64_t parent_id); void handleLet(TSNode node, uint64_t parent_id); void handleUse(TSNode node, uint64_t parent_id); std::string extractName(TSNode node); @@ -50,6 +57,20 @@ class RustVisitor : public JsVisitor { var_types_[name] = type; } + /// Record every named parameter of a function or method into var_types_ and + /// emit its TypeRef, exactly as the local-declaration path does. + /// + /// Parameters were visited for their bodies only, so the declared type of a + /// parameter was never recorded: `o.method()` carried receiver_text with an + /// EMPTY receiver_type, and when two types declare the same method name the + /// resolver had nothing to choose with and abstained — a false negative Go, + /// Java, Python and C++ do not produce for the same call shape (measured in + /// test_resolver_language_consistency). + void recordParameterTypes(TSNode params, uint64_t fn_id); + + /// The first identifier of a pattern (`o`, `mut o`, `(a, b)` → `a`), or "". + std::string firstIdentifier(TSNode node, int depth); + void pushImplScope(const std::string &type_name) { if (!type_name.empty()) diff --git a/engine/src/ir/translators/scanner_visitor.cpp b/engine/src/ir/translators/scanner_visitor.cpp deleted file mode 100644 index 8530971..0000000 --- a/engine/src/ir/translators/scanner_visitor.cpp +++ /dev/null @@ -1,447 +0,0 @@ -#include "scanner_visitor.h" -#include -#include - -namespace ir -{ - -ScannerVisitor::ScannerVisitor() -{ -} - -SemanticUnit *ScannerVisitor::scan(const std::string &source, - const char *file_path) -{ - unit_ = new SemanticUnit(); - SemanticEmitter emitter(unit_); - emitter_ = &emitter; - unit_->setFilePath(file_path); - unit_->setLanguage(language_); - - std::istringstream stream(source); - std::string line; - int line_num = 0; - - while (std::getline(stream, line)) { - line_num++; - std::string_view sv(line); - sv = trimLeft(sv); - if (sv.empty() || sv[0] == '/' || sv[0] == '#' || sv[0] == '*') - continue; - - std::string kind = detectDeclLine(sv); - if (kind.empty()) - continue; - - std::string name = extractDeclName(sv, kind); - if (name.empty()) - continue; - - // Skip false positives - if (name == "if" || name == "for" || name == "while" || - name == "switch" || name == "catch" || name == "return" || - name == "else") - continue; - - // Find end column from the name + signature length - uint32_t end_col = - static_cast(sv.size() > 80 ? 80 : sv.size()); - SourceRange loc{ static_cast(line_num - 1), 0, - static_cast(line_num - 1), end_col }; - - emitDecl(kind, name, loc, 0); - } - - emitter_ = nullptr; - return unit_; -} - -void ScannerVisitor::emitDecl(const std::string &kind, const std::string &name, - SourceRange loc, uint64_t parent_id) -{ - if (kind == "function" || kind == "method") - emitter_->emitFunction(name, loc, parent_id); - else if (kind == "class") - emitter_->emitClass(name, loc, parent_id); - else if (kind == "struct") - emitter_->emitClass(name, loc, parent_id); - else if (kind == "interface") - emitter_->emitInterface(name, loc, parent_id); - else if (kind == "enum") - emitter_->emitEnum(name, loc, parent_id); - else if (kind == "type_alias") - emitter_->emitTypeAlias(name, loc, parent_id); - else if (kind == "const" || kind == "variable") - emitter_->emitVariable(name, loc, parent_id); -} - -// ── Line detection helpers ────────────────────────────────────── - -std::string_view ScannerVisitor::trimLeft(std::string_view s) -{ - while (!s.empty() && (s[0] == ' ' || s[0] == '\t')) - s.remove_prefix(1); - return s; -} - -bool ScannerVisitor::startsWithKW(std::string_view line, const char *kw) -{ - line = trimLeft(line); - auto klen = std::strlen(kw); - if (line.size() < klen) - return false; - if (line.substr(0, klen) != kw) - return false; - // kw ending with space: space was delimiter - if (klen > 0 && kw[klen - 1] == ' ') - return true; - // Otherwise must be followed by non-word char - if (line.size() > klen) { - char c = line[klen]; - return (c == ' ' || c == '(' || c == '<' || c == '\t' || - c == '{' || c == '[' || c == ':' || c == ';'); - } - return true; -} - -std::string ScannerVisitor::detectDeclLine(std::string_view line) -{ - line = trimLeft(line); - if (line.empty() || line[0] == '/' || line[0] == '#' || line[0] == '*') - return ""; - - // C/C++ detection - if (language_ == "c" || language_ == "cpp") { - if (startsWithKW(line, "class ")) - return "class"; - if (startsWithKW(line, "struct ")) - return "struct"; - if (startsWithKW(line, "enum ")) - return "enum"; - if (startsWithKW(line, "union ")) - return "struct"; - if (startsWithKW(line, "namespace ")) - return ""; - if (startsWithKW(line, "using namespace ")) - return ""; - if (startsWithKW(line, "template ")) - return ""; - // C function detection via looksLikeCFunction logic - if (line.find('(') != std::string_view::npos && - line.find('(') > 0 && line.back() != ';' && - line.back() != ',') { - // More careful: skip if/while/for/switch - static const char *skip[] = { - "if", "while", "for", "switch", "catch", - "return", "sizeof", "typeof", nullptr - }; - auto name_end = line.find('('); - auto name_start = name_end; - while (name_start > 0 && - (isalnum(line[name_start - 1]) || - line[name_start - 1] == '_' || - line[name_start - 1] == ':')) - name_start--; - if (name_start == name_end) - return ""; - auto name_tok = - line.substr(name_start, name_end - name_start); - for (const char **s = skip; *s; s++) - if (name_tok == *s) - return ""; - // Must have a known type keyword before the name - auto before = trimLeft(line.substr(0, name_start)); - static const char *types[] = { - "int ", "void ", "char ", "long ", - "short ", "float ", "double ", "bool ", - "signed ", "unsigned ", "const ", "struct ", - "union ", "enum ", "class ", "size_t ", - "ssize_t ", nullptr - }; - for (const char **t = types; *t; t++) - if (before.find(*t) != std::string_view::npos) - return "function"; - } - } - - // Rust - if (language_ == "rust") { - if (startsWithKW(line, "pub unsafe fn ") || - startsWithKW(line, "pub async fn ") || - startsWithKW(line, "pub fn ") || startsWithKW(line, "fn ")) - return "function"; - if (startsWithKW(line, "pub struct ") || - startsWithKW(line, "struct ")) - return "struct"; - if (startsWithKW(line, "pub enum ") || - startsWithKW(line, "enum ")) - return "enum"; - if (startsWithKW(line, "pub trait ") || - startsWithKW(line, "trait ")) - return "interface"; - if (startsWithKW(line, "pub type ") || - startsWithKW(line, "type ")) - return "type_alias"; - } - - // Python - if (language_ == "python") { - if (startsWithKW(line, "async def ") || - startsWithKW(line, "def ")) - return "function"; - if (startsWithKW(line, "class ")) - return "class"; - } - - // JS/TS - if (language_ == "javascript" || language_ == "typescript") { - if (startsWithKW(line, "export async function ") || - startsWithKW(line, "export function ") || - startsWithKW(line, "async function ") || - startsWithKW(line, "function ")) - return "function"; - if (startsWithKW(line, "export class ") || - startsWithKW(line, "class ")) - return "class"; - if (startsWithKW(line, "export interface ") || - startsWithKW(line, "interface ")) - return "interface"; - if (startsWithKW(line, "export enum ") || - startsWithKW(line, "enum ")) - return "enum"; - } - - // Go - if (language_ == "go") { - if (startsWithKW(line, "func ")) - return "function"; - if (startsWithKW(line, "type ")) { - if (line.find("struct") != std::string_view::npos) - return "struct"; - if (line.find("interface") != std::string_view::npos) - return "interface"; - return "type_alias"; - } - } - - // Java - if (language_ == "java") { - if (startsWithKW(line, "class ") || - startsWithKW(line, "public class ") || - startsWithKW(line, "private class ") || - startsWithKW(line, "protected class ")) - return "class"; - if (startsWithKW(line, "interface ") || - startsWithKW(line, "public interface ")) - return "interface"; - if (startsWithKW(line, "enum ") || - startsWithKW(line, "public enum ")) - return "enum"; - } - - // Swift - if (language_ == "swift") { - if (startsWithKW(line, "func ") || - startsWithKW(line, "public func ") || - startsWithKW(line, "open func ")) - return "function"; - if (startsWithKW(line, "class ") || - startsWithKW(line, "public class ") || - startsWithKW(line, "open class ")) - return "class"; - if (startsWithKW(line, "struct ") || - startsWithKW(line, "public struct ")) - return "struct"; - if (startsWithKW(line, "enum ") || - startsWithKW(line, "public enum ")) - return "enum"; - if (startsWithKW(line, "protocol ") || - startsWithKW(line, "public protocol ")) - return "interface"; - } - - return ""; -} - -std::string ScannerVisitor::extractDeclName(std::string_view line, - const std::string &kind) -{ - line = trimLeft(line); - - // Skip language-specific keyword prefixes to find the name token - auto skip_prefix = [&](const char **keywords) { - for (const char **kw = keywords; *kw; kw++) { - if (startsWithKW(line, *kw)) { - line = trimLeft(line.substr(std::strlen(*kw))); - return true; - } - } - return false; - }; - - // For struct/class/enum declarations, the name is the next identifier - if (kind == "struct" || kind == "class" || kind == "enum" || - kind == "interface" || kind == "type_alias") { - if (language_ == "rust") { - static const char *rk[] = { - "pub struct", "struct", "pub enum", "enum", - "pub trait", "trait", "pub type", "type", - "pub union", "union", nullptr - }; - skip_prefix(rk); - } else if (language_ == "c" || language_ == "cpp") { - static const char *ck[] = { "class ", "struct ", - "enum ", "union ", - "enum class ", nullptr }; - skip_prefix(ck); - } else if (language_ == "java") { - static const char *jk[] = { "public class", - "private class", - "protected class", - "class", - "public interface", - "interface", - "public enum", - "enum", - nullptr }; - skip_prefix(jk); - } else if (language_ == "swift") { - static const char *sk[] = { - "open class", "public class", - "class", "public struct", - "struct", "public enum", - "enum", "public protocol", - "protocol", nullptr - }; - skip_prefix(sk); - } else if (language_ == "go") { - if (line.substr(0, 5) == "type ") - line = trimLeft(line.substr(5)); - } else if (language_ == "python") { - static const char *pk[] = { "class", nullptr }; - skip_prefix(pk); - } else if (language_ == "javascript" || - language_ == "typescript") { - static const char *jk2[] = { "export class", - "class", - "export interface", - "interface", - "export enum", - "enum", - nullptr }; - skip_prefix(jk2); - } - // Take first word as name - size_t pos = 0; - while (pos < line.size() && - (isalnum(line[pos]) || line[pos] == '_')) - pos++; - if (pos == 0) - return ""; - return std::string(line.substr(0, pos)); - } - - // For function/method declarations, find name before '(' - if (language_ == "rust") { - static const char *rk[] = { - "pub unsafe fn", "pub async fn", "pub fn", "fn", - "pub struct", "struct", "pub enum", "enum", - "pub trait", "trait", "pub type", "type", - nullptr - }; - skip_prefix(rk); - } else if (language_ == "python") { - static const char *pk[] = { "async def", "def", "class", - nullptr }; - skip_prefix(pk); - } else if (language_ == "javascript" || language_ == "typescript") { - static const char *jk[] = { "export async function", - "export function", - "async function", - "function", - "export class", - "class", - "export interface", - "interface", - "export enum", - "enum", - nullptr }; - skip_prefix(jk); - } else if (language_ == "go") { - if (line.substr(0, 5) == "func ") - line = trimLeft(line.substr(5)); - else if (line.substr(0, 5) == "type ") - line = trimLeft(line.substr(5)); - } else if (language_ == "swift") { - static const char *sk[] = { - "open func", "public func", "func", - "open class", "public class", "class", - "public struct", "struct", "public enum", - "enum", "public protocol", "protocol", - nullptr - }; - skip_prefix(sk); - } else if (language_ == "java") { - static const char *jk2[] = { "public class", - "private class", - "protected class", - "class", - "public interface", - "interface", - "public enum", - "enum", - nullptr }; - skip_prefix(jk2); - } else if (language_ == "c" || language_ == "cpp") { - // For C/C++, skip storage class or type prefix, then find name before '(' - static const char *ck[] = { "static ", "extern ", - "inline ", "virtual ", - "constexpr ", "const ", - "volatile ", nullptr }; - skip_prefix(ck); - // If there's a '(' in the line, the name is the token just before it - auto paren = line.find('('); - if (paren != std::string_view::npos && paren > 0) { - auto end = paren; - auto start = end; - while (start > 0 && (isalnum(line[start - 1]) || - line[start - 1] == '_' || - line[start - 1] == ':')) - start--; - if (start < end) { - // Skip backwards over * and & (pointer/reference) - line = line.substr(start, end - start); - // Strip leading type keywords - static const char *tk[] = { "unsigned ", - "signed ", "long ", - "short ", nullptr }; - skip_prefix(tk); - // Now find the last word (the actual name) - auto last_space = line.rfind(' '); - if (last_space != std::string_view::npos) - line = trimLeft( - line.substr(last_space + 1)); - // Clean up any trailing * - while (!line.empty() && line.back() == '*') - line.remove_suffix(1); - line = trimLeft(line); - } - } - } - - // Extract identifier: alphanumeric + underscore + scope - size_t pos = 0; - while (pos < line.size() && - (isalnum(line[pos]) || line[pos] == '_' || line[pos] == ':')) - pos++; - if (pos == 0) - return ""; - - std::string name(line.substr(0, pos)); - // Remove trailing ':' from scope operators - while (!name.empty() && name.back() == ':') - name.pop_back(); - return name; -} - -} // namespace ir diff --git a/engine/src/ir/translators/scanner_visitor.h b/engine/src/ir/translators/scanner_visitor.h deleted file mode 100644 index c37f852..0000000 --- a/engine/src/ir/translators/scanner_visitor.h +++ /dev/null @@ -1,63 +0,0 @@ -#ifndef SCANNER_VISITOR_H -#define SCANNER_VISITOR_H - -#include -#include -#include "../semantic_emitter.h" -#include "../semantic_unit.h" - -namespace ir -{ - -/** - * Fast scanner visitor — line-based declaration extraction without tree-sitter. - * - * For each source file, reads lines and detects function/class/struct/enum/etc. - * declarations by keyword matching (no parse tree). ~10x faster than tree-sitter. - * - * Emits SemanticUnit records compatible with the pipeline, so the same - * GraphBuilder → Store path works for both fast and full mode. - * - * Supported languages: C, C++, Go, Rust, Python, Java, JavaScript, TypeScript, - * Swift. - */ -class ScannerVisitor { - public: - ScannerVisitor(); - - /** - * Scan a source file and produce a SemanticUnit with declaration records. - * Only top-level declarations are extracted (no function bodies). - * Returns nullptr on empty/error. - */ - SemanticUnit *scan(const std::string &source, const char *file_path); - - /** Set the language for interpretation. */ - void setLanguage(const std::string &lang) - { - language_ = lang; - } - - private: - SemanticUnit *unit_ = nullptr; - SemanticEmitter *emitter_ = nullptr; - std::string language_; - - // Helpers - static std::string_view trimLeft(std::string_view s); - static bool startsWithKW(std::string_view line, const char *kw); - - /** Try to detect declaration on a line. Returns empty string if none. */ - std::string detectDeclLine(std::string_view line); - - /** Extract name from a declaration line. */ - std::string extractDeclName(std::string_view line, - const std::string &kind); - - /** Emit a record for the detected declaration. */ - void emitDecl(const std::string &kind, const std::string &name, - SourceRange loc, uint64_t parent_id); -}; - -} // namespace ir -#endif diff --git a/engine/src/ir/translators/swift_translator.cpp b/engine/src/ir/translators/swift_translator.cpp deleted file mode 100644 index 92c2eca..0000000 --- a/engine/src/ir/translators/swift_translator.cpp +++ /dev/null @@ -1,392 +0,0 @@ -#include "../ir_translator.h" - -#include -#include -#include -#include -#include - -namespace ir -{ - -// ─── Swift CST → IR Translator ────────────────────────────────── - -class SwiftTranslator : public Translator { - public: - const char *language() const override - { - return "swift"; - } - - TranslationUnit *translate(TSTree *tree, const char *source, - const char *file_path) override; - - private: - TranslationUnit *unit_ = nullptr; - const char *source_ = nullptr; - std::string file_path_; - - struct Scope { - std::unordered_map symbols; - }; - std::vector scopes_; - std::unordered_map definitions_; - std::vector class_stack_; - - Node *makeNode(NodeKind kind, TSNode ts_node); - void setLocation(Node *node, TSNode ts_node); - std::string nodeText(TSNode ts_node); - - void pushScope(); - void popScope(); - void defineSymbol(const std::string &name, Node *node); - Node *resolveSymbol(const std::string &name); - - Node *translateNode(TSNode ts_node, Node *parent); - void translateChildren(TSNode ts_node, Node *parent); - - Node *handleFuncDecl(TSNode ts_node, Node *parent); - Node *handleClassDecl(TSNode ts_node, Node *parent); - Node *handleStructDecl(TSNode ts_node, Node *parent); - Node *handleCall(TSNode ts_node, Node *parent); - Node *handleIdentifier(TSNode ts_node, Node *parent); - Node *handleImport(TSNode ts_node, Node *parent); - std::string extractName(TSNode ts_node); -}; - -// ─── Helpers ──────────────────────────────────────────────────── - -Node *SwiftTranslator::makeNode(NodeKind kind, TSNode ts_node) -{ - auto *node = new Node(); - node->kind = kind; - node->language = "swift"; - node->file_path = file_path_; - setLocation(node, ts_node); - return node; -} - -void SwiftTranslator::setLocation(Node *node, TSNode ts_node) -{ - TSPoint start = ts_node_start_point(ts_node); - TSPoint end = ts_node_end_point(ts_node); - node->loc.start_row = start.row; - node->loc.start_col = start.column; - node->loc.end_row = end.row; - node->loc.end_col = end.column; -} - -std::string SwiftTranslator::nodeText(TSNode ts_node) -{ - uint32_t start = ts_node_start_byte(ts_node); - uint32_t end = ts_node_end_byte(ts_node); - return std::string(source_ + start, end - start); -} - -void SwiftTranslator::pushScope() -{ - scopes_.push_back(Scope{}); -} - -void SwiftTranslator::popScope() -{ - if (!scopes_.empty()) - scopes_.pop_back(); -} - -void SwiftTranslator::defineSymbol(const std::string &name, Node *node) -{ - if (!scopes_.empty()) - scopes_.back().symbols[name] = node; - definitions_[name] = node; -} - -Node *SwiftTranslator::resolveSymbol(const std::string &name) -{ - for (auto it = scopes_.rbegin(); it != scopes_.rend(); ++it) { - auto found = it->symbols.find(name); - if (found != it->symbols.end()) - return found->second; - } - auto def = definitions_.find(name); - if (def != definitions_.end()) - return def->second; - return nullptr; -} - -std::string SwiftTranslator::extractName(TSNode ts_node) -{ - uint32_t count = ts_node_child_count(ts_node); - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(ts_node, i); - if (!ts_node_is_named(child)) - continue; - const char *t = ts_node_type(child); - if (strcmp(t, "identifier") == 0) - return nodeText(child); - if (strcmp(t, "simple_identifier") == 0) - return nodeText(child); - if (strcmp(t, "type_identifier") == 0) - return nodeText(child); - std::string sub = extractName(child); - if (!sub.empty()) - return sub; - } - return ""; -} - -// ─── Handlers ────────────────────────────────────────────────── - -Node *SwiftTranslator::handleFuncDecl(TSNode ts_node, Node *parent) -{ - auto *func = makeNode(NodeKind::FunctionDecl, ts_node); - func->name = extractName(ts_node); - defineSymbol(func->name, func); - parent->children.push_back(func); - - pushScope(); - uint32_t count = ts_node_child_count(ts_node); - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(ts_node, i); - if (!ts_node_is_named(child)) - continue; - const char *t = ts_node_type(child); - if (strcmp(t, "function_declaration") == 0) - continue; - if (strcmp(t, "parameter") == 0 || - strcmp(t, "parameter_list") == 0) { - translateChildren(child, func); - } else if (strcmp(t, "body") == 0 || - strcmp(t, "statement") == 0 || - strcmp(t, "function_body") == 0) { - translateChildren(child, func); - } else if (strcmp(t, "mutating") == 0 || - strcmp(t, "nonmutating") == 0 || - strcmp(t, "static") == 0 || - strcmp(t, "public") == 0 || - strcmp(t, "private") == 0 || - strcmp(t, "internal") == 0) { - // modifiers — skip, process real children - } else { - translateChildren(child, func); - } - } - popScope(); - return func; -} - -Node *SwiftTranslator::handleClassDecl(TSNode ts_node, Node *parent) -{ - bool is_class = true; - uint32_t cc = ts_node_child_count(ts_node); - for (uint32_t i = 0; i < cc; i++) { - TSNode c = ts_node_child(ts_node, i); - if (!ts_node_is_named(c)) - continue; - if (strcmp(ts_node_type(c), "struct") == 0) { - is_class = false; - break; - } - } - - auto *cls = makeNode( - is_class ? NodeKind::ClassDecl : NodeKind::ClassDecl, ts_node); - cls->name = extractName(ts_node); - if (!cls->name.empty()) - defineSymbol(cls->name, cls); - parent->children.push_back(cls); - class_stack_.push_back(cls); - - pushScope(); - translateChildren(ts_node, cls); - popScope(); - - class_stack_.pop_back(); - return cls; -} - -Node *SwiftTranslator::handleCall(TSNode ts_node, Node *parent) -{ - auto *call = makeNode(NodeKind::CallExpr, ts_node); - parent->children.push_back(call); - - uint32_t count = ts_node_child_count(ts_node); - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(ts_node, i); - if (!ts_node_is_named(child)) - continue; - const char *t = ts_node_type(child); - if (strcmp(t, "identifier") == 0 || - strcmp(t, "simple_identifier") == 0) { - std::string fname = nodeText(child); - call->name = fname; - Node *target = resolveSymbol(fname); - if (target) { - call->semantic_edges.push_back( - { target, Relation::CallTarget }); - } - } - // Recurse into argument list - translateChildren(child, call); - } - return call; -} - -Node *SwiftTranslator::handleIdentifier(TSNode ts_node, Node *parent) -{ - auto *id_expr = makeNode(NodeKind::IdentifierExpr, ts_node); - id_expr->name = nodeText(ts_node); - parent->children.push_back(id_expr); - - Node *target = resolveSymbol(id_expr->name); - if (target) { - id_expr->semantic_edges.push_back( - { target, Relation::SymbolRef }); - } - return id_expr; -} - -Node *SwiftTranslator::handleImport(TSNode ts_node, Node *parent) -{ - auto *imp = makeNode(NodeKind::ImportDecl, ts_node); - imp->name = extractName(ts_node); - parent->children.push_back(imp); - return imp; -} - -void SwiftTranslator::translateChildren(TSNode ts_node, Node *parent) -{ - uint32_t count = ts_node_child_count(ts_node); - for (uint32_t i = 0; i < count; i++) { - TSNode child = ts_node_child(ts_node, i); - if (!ts_node_is_named(child)) - continue; - translateNode(child, parent); - } -} - -Node *SwiftTranslator::translateNode(TSNode ts_node, Node *parent) -{ - const char *type = ts_node_type(ts_node); - - if (strcmp(type, "function_declaration") == 0) - return handleFuncDecl(ts_node, parent); - if (strcmp(type, "class_declaration") == 0 || - strcmp(type, "struct_declaration") == 0) - return handleClassDecl(ts_node, parent); - if (strcmp(type, "call_expression") == 0) - return handleCall(ts_node, parent); - if (strcmp(type, "identifier") == 0 || - strcmp(type, "simple_identifier") == 0) - return handleIdentifier(ts_node, parent); - if (strcmp(type, "import_declaration") == 0) - return handleImport(ts_node, parent); - - // Control flow - if (strcmp(type, "return_statement") == 0) { - auto *n = makeNode(NodeKind::ReturnStmt, ts_node); - parent->children.push_back(n); - translateChildren(ts_node, n); - return n; - } - if (strcmp(type, "if_statement") == 0) { - auto *n = makeNode(NodeKind::IfStmt, ts_node); - parent->children.push_back(n); - translateChildren(ts_node, n); - return n; - } - if (strcmp(type, "for_statement") == 0 || - strcmp(type, "for_in_statement") == 0) { - auto *n = makeNode(NodeKind::ForStmt, ts_node); - parent->children.push_back(n); - translateChildren(ts_node, n); - return n; - } - if (strcmp(type, "while_statement") == 0) { - auto *n = makeNode(NodeKind::WhileStmt, ts_node); - parent->children.push_back(n); - translateChildren(ts_node, n); - return n; - } - if (strcmp(type, "switch_expression") == 0 || - strcmp(type, "switch_statement") == 0) { - auto *n = makeNode(NodeKind::SwitchStmt, ts_node); - parent->children.push_back(n); - translateChildren(ts_node, n); - return n; - } - - // Literals - if (strcmp(type, "integer_literal") == 0 || - strcmp(type, "float_literal") == 0 || - strcmp(type, "string_literal") == 0 || - strcmp(type, "boolean_literal") == 0) { - auto *n = makeNode(NodeKind::LiteralExpr, ts_node); - parent->children.push_back(n); - return n; - } - - // Comments - if (strcmp(type, "comment") == 0) { - auto *n = makeNode(NodeKind::Comment, ts_node); - n->name = nodeText(ts_node); - parent->children.push_back(n); - return n; - } - - // Recursive containers - if (strcmp(type, "expression") == 0 || strcmp(type, "statement") == 0 || - strcmp(type, "body") == 0 || strcmp(type, "function_body") == 0 || - strcmp(type, "class_body") == 0 || - strcmp(type, "parameter_list") == 0 || - strcmp(type, "parameter") == 0 || strcmp(type, "argument") == 0 || - strcmp(type, "tuple") == 0 || - strcmp(type, "member_access_expression") == 0 || - strcmp(type, "binary_expression") == 0 || - strcmp(type, "assignment") == 0 || - strcmp(type, "optional_chaining_expression") == 0) { - translateChildren(ts_node, parent); - return nullptr; - } - - return nullptr; -} - -// ─── Translation Entry ───────────────────────────────────────── - -TranslationUnit *SwiftTranslator::translate(TSTree *tree, const char *source, - const char *file_path) -{ - unit_ = new TranslationUnit(); - unit_->source_content = source; - source_ = source; - file_path_ = file_path; - - TSNode root_node = ts_tree_root_node(tree); - auto *root = makeNode(NodeKind::TranslationUnit, root_node); - root->file_path = file_path; - root->language = "swift"; - unit_->root = root; - - pushScope(); - translateChildren(root_node, root); - popScope(); - - std::function collect = [&](Node *n) { - unit_->all_nodes.push_back(n); - for (auto *c : n->children) - collect(c); - }; - collect(root); - unit_->assignIds(); - return unit_; -} - -// ─── Factory ─────────────────────────────────────────────────── - -std::unique_ptr createSwiftTranslator() -{ - return std::make_unique(); -} - -} // namespace ir diff --git a/engine/src/ir/translators/swift_visitor.cpp b/engine/src/ir/translators/swift_visitor.cpp deleted file mode 100644 index e6596da..0000000 --- a/engine/src/ir/translators/swift_visitor.cpp +++ /dev/null @@ -1,216 +0,0 @@ -#include "swift_visitor.h" -#include -#include -#include "../builtin_registry.h" -namespace ir -{ -SwiftVisitor::SwiftVisitor() -{ -} -SemanticUnit *SwiftVisitor::visit(TSTree *tree, const char *source, - const char *fp) -{ - unit_ = new SemanticUnit(); - SemanticEmitter emitter(unit_); - emitter_ = &emitter; - unit_->setFilePath(fp); - unit_->setLanguage("swift"); - source_ = source; - - TSNode root_node = ts_tree_root_node(tree); - pushScope(); - SourceRange root_loc = location(root_node); - uint64_t root_id = emitter_->emitVariable("", root_loc, 0); - (void)root_id; - visitChildren(root_node, 0); - popScope(); - - emitter_ = nullptr; - return unit_; -} -void SwiftVisitor::visitNode(TSNode node, uint64_t parent_id) -{ - const char *type = ts_node_type(node); - if (strcmp(type, "function_declaration") == 0) - return handleFuncDecl(node, parent_id); - if (strcmp(type, "class_declaration") == 0) - return handleClassDecl(node, parent_id); - if (strcmp(type, "struct_declaration") == 0) - return handleStructDecl(node, parent_id); - if (strcmp(type, "enum_declaration") == 0) - return handleEnumDecl(node, parent_id); - if (strcmp(type, "protocol_declaration") == 0) - return handleProtocolDecl(node, parent_id); - if (strcmp(type, "call_expression") == 0) - return handleCall(node, parent_id); - if (strcmp(type, "variable_declaration") == 0) - return handleVarDecl(node, parent_id); - if (strcmp(type, "import_declaration") == 0) - return handleImport(node, parent_id); - JsVisitor::visitNode(node, parent_id); -} -void SwiftVisitor::handleFuncDecl(TSNode node, uint64_t parent_id) -{ - SourceRange loc = location(node); - std::string name = extractName(node); - if (name.empty()) { - visitChildren(node, parent_id); - return; - } - uint64_t id = emitter_->emitFunction(name, loc, parent_id, 0, false, - detectVisibility(node)); - defineSymbol(name, id); - pushScope(); - pushFunctionScope(id); - uint32_t cnt = ts_node_child_count(node); - for (uint32_t i = 0; i < cnt; i++) { - TSNode c = ts_node_child(node, i); - if (!ts_node_is_named(c)) - continue; - if (strcmp(ts_node_type(c), "identifier") == 0) - continue; - if (strcmp(ts_node_type(c), "body") == 0) - visitChildren(c, id); - } - popFunctionScope(); - popScope(); -} -void SwiftVisitor::handleClassDecl(TSNode node, uint64_t parent_id) -{ - SourceRange loc = location(node); - std::string name = extractName(node); - if (name.empty()) { - visitChildren(node, parent_id); - return; - } - uint64_t id = emitter_->emitClass(name, loc, parent_id, - detectVisibility(node)); - defineSymbol(name, id); - visitChildren(node, id); -} -void SwiftVisitor::handleStructDecl(TSNode node, uint64_t parent_id) -{ - SourceRange loc = location(node); - std::string name = extractName(node); - if (name.empty()) { - visitChildren(node, parent_id); - return; - } - uint64_t id = emitter_->emitClass(name, loc, parent_id, - detectVisibility(node)); - defineSymbol(name, id); - visitChildren(node, id); -} -void SwiftVisitor::handleEnumDecl(TSNode node, uint64_t parent_id) -{ - SourceRange loc = location(node); - std::string name = extractName(node); - if (name.empty()) { - visitChildren(node, parent_id); - return; - } - uint64_t id = emitter_->emitEnum(name, loc, parent_id, - detectVisibility(node)); - defineSymbol(name, id); - visitChildren(node, id); -} -void SwiftVisitor::handleProtocolDecl(TSNode node, uint64_t parent_id) -{ - SourceRange loc = location(node); - std::string name = extractName(node); - if (name.empty()) { - visitChildren(node, parent_id); - return; - } - uint64_t id = emitter_->emitInterface(name, loc, parent_id, - detectVisibility(node)); - defineSymbol(name, id); - visitChildren(node, id); -} -void SwiftVisitor::handleCall(TSNode node, uint64_t parent_id) -{ - std::string name = nodeText(node); - // Use the containing function as parent_id (not the immediate - // syntactic parent, which may be another call record). Without - // this, nested calls would have their parent_id set to the - // outer call record, which is NOT in _r2n (only declarations - // are). The reference-table JOIN would fail and the nested - // call would be dropped. - uint64_t func_id = currentFunctionId(); - uint64_t call_parent = (func_id != 0) ? func_id : parent_id; - uint64_t id = emitter_->emitCall(name, location(node), call_parent); - - // ── Intra-file callee resolution ─────────────────────────── - // Store the resolved callee's record ID as ref_original_id. - // Enables P1 call-edge construction in buildCallEdgesSQL. - if (!name.empty()) { - uint64_t target = resolveSymbol(name); - if (target) { - unit_->setCallReference(id, target); - unit_->setCallStrategy(id, "p1_intra"); - } else { - unit_->setCallStrategy( - id, BuiltinRegistry::resolve(unit_->language(), - name)); - } - } - - visitChildren(node, parent_id); -} -void SwiftVisitor::handleVarDecl(TSNode node, uint64_t parent_id) -{ - uint32_t cnt = ts_node_child_count(node); - for (uint32_t i = 0; i < cnt; i++) { - TSNode c = ts_node_child(node, i); - if (!ts_node_is_named(c)) - continue; - if (strcmp(ts_node_type(c), "identifier") == 0) { - std::string name = nodeText(c); - if (!name.empty()) { - uint64_t id = emitter_->emitVariable( - name, location(c), parent_id, - detectVisibility(c)); - defineSymbol(name, id); - } - } - } -} -void SwiftVisitor::handleImport(TSNode node, uint64_t parent_id) -{ - emitter_->emitImport(nodeText(node), location(node), parent_id); -} -std::string SwiftVisitor::extractName(TSNode node) -{ - uint32_t cnt = ts_node_child_count(node); - for (uint32_t i = 0; i < cnt; i++) { - TSNode c = ts_node_child(node, i); - if (!ts_node_is_named(c)) - continue; - if (strcmp(ts_node_type(c), "identifier") == 0 || - strcmp(ts_node_type(c), "type_identifier") == 0) - return nodeText(c); - } - return ""; -} - -int SwiftVisitor::detectVisibility(TSNode node) -{ - // Swift access-level modifiers appear as named children of the declaration - // node (function_declaration/class_declaration/etc). Scan for: - // public/open → 1 (exported) - // internal/fileprivate/private (default) → 0 - uint32_t cnt = ts_node_child_count(node); - for (uint32_t i = 0; i < cnt; i++) { - TSNode c = ts_node_child(node, i); - if (!ts_node_is_named(c)) - continue; - std::string txt = nodeText(c); - if (txt == "public" || txt == "open") - return 1; - if (txt == "internal" || txt == "fileprivate" || - txt == "private") - return 0; - } - return 0; // Swift default = internal -} -} // namespace ir diff --git a/engine/src/ir/translators/swift_visitor.h b/engine/src/ir/translators/swift_visitor.h deleted file mode 100644 index b95623f..0000000 --- a/engine/src/ir/translators/swift_visitor.h +++ /dev/null @@ -1,30 +0,0 @@ -#ifndef SWIFT_VISITOR_H -#define SWIFT_VISITOR_H -#include "js_visitor.h" -namespace ir -{ -class SwiftVisitor : public JsVisitor { - public: - SwiftVisitor(); - SemanticUnit *visit(TSTree *tree, const char *source, - const char *file_path) override; - - protected: - void visitNode(TSNode node, uint64_t parent_id) override; - - private: - void handleFuncDecl(TSNode node, uint64_t parent_id); - void handleClassDecl(TSNode node, uint64_t parent_id); - void handleStructDecl(TSNode node, uint64_t parent_id); - void handleEnumDecl(TSNode node, uint64_t parent_id); - void handleProtocolDecl(TSNode node, uint64_t parent_id); - void handleCall(TSNode node, uint64_t parent_id); - void handleVarDecl(TSNode node, uint64_t parent_id); - void handleImport(TSNode node, uint64_t parent_id); - std::string extractName(TSNode node); - /// Detect Swift visibility: 1=public/open, 0=internal/fileprivate/private. - /// v0.2.2 role classifier signal. - int detectVisibility(TSNode node); -}; -} // namespace ir -#endif diff --git a/engine/src/ir/translators/ts_visitor.cpp b/engine/src/ir/translators/ts_visitor.cpp index b48107b..b276243 100644 --- a/engine/src/ir/translators/ts_visitor.cpp +++ b/engine/src/ir/translators/ts_visitor.cpp @@ -1,5 +1,6 @@ #include "ts_visitor.h" +#include #include #include @@ -24,8 +25,14 @@ SemanticUnit *TsVisitor::visit(TSTree *tree, const char *source, // file do not leak into the current file's receiver inference. var_types_.clear(); class_scope_stack_.clear(); + import_aliases_.clear(); TSNode root_node = ts_tree_root_node(tree); + // Names this file defines, so a user function whose name collides with a + // JS/TS builtin (`function map() {}`, `function format() {}`) is not + // dropped by visitCallExpr's builtin filter. + defined_names_.clear(); + collectDefinedNames(root_node); pushScope(); SourceRange root_loc = location(root_node); uint64_t root_id = emitter_->emitVariable("", root_loc, 0); @@ -48,11 +55,77 @@ void TsVisitor::visitNode(TSNode node, uint64_t parent_id) return visitTypeAliasDecl(node, parent_id); if (strcmp(type, "enum_declaration") == 0) return visitEnumDecl(node, parent_id); + // `abstract class Foo ...` is a distinct node type in the TS grammar but + // introduces a class exactly like class_declaration (including its + // optional `implements` clause) — route it through the same handler. + if (strcmp(type, "abstract_class_declaration") == 0) + return visitClassDecl(node, parent_id); + + // Function-like nodes carry their parameter types in the annotation, which + // the shared JavaScript handlers never read: they walk `formal_parameters` + // only for the parameter bodies (default values), so `o.method()` went in + // with receiver_text but an EMPTY receiver_type and the resolver abstained + // whenever two types declare the same method name. Same defect and same fix + // as the C/C++ and Rust visitors (see + // test_resolver_language_consistency); JavaScript has no annotations, which + // is why only this override does it. + if (strcmp(type, "function_declaration") == 0 || + strcmp(type, "function_expression") == 0 || + strcmp(type, "generator_function_declaration") == 0 || + strcmp(type, "method_definition") == 0 || + strcmp(type, "arrow_function") == 0) + recordParameterTypes(node); // ── Fall back to JavaScript handling for all shared types ──── JsVisitor::visitNode(node, parent_id); } +void TsVisitor::recordParameterTypes(TSNode fn_node) +{ + // `o: Owner` is a required_parameter whose `type` field wraps the annotation + // in a type_annotation node, which extractTsTypeAnnotation unwraps (and + // which already handles type_identifier, predefined_type and generic_type). + // An untyped parameter (`o`) has no annotation and binds nothing, and an + // arrow function's bare `x => ...` has no formal_parameters at all. + TSNode params = ts_node_child_by_field_name(fn_node, "parameters", 10); + if (ts_node_is_null(params)) + return; + const uint32_t count = ts_node_child_count(params); + for (uint32_t i = 0; i < count; ++i) { + TSNode param = ts_node_child(params, i); + if (!ts_node_is_named(param)) + continue; + const char *pt = ts_node_type(param); + if (strcmp(pt, "required_parameter") != 0 && + strcmp(pt, "optional_parameter") != 0) + continue; + TSNode type_node = + ts_node_child_by_field_name(param, "type", 4); + TSNode pattern = + ts_node_child_by_field_name(param, "pattern", 7); + if (ts_node_is_null(pattern)) { + // Older grammar shapes hang the name directly off the parameter. + for (uint32_t k = 0; k < ts_node_child_count(param); + ++k) { + TSNode child = ts_node_child(param, k); + if (ts_node_is_named(child) && + strcmp(ts_node_type(child), "identifier") == + 0) { + pattern = child; + break; + } + } + } + if (ts_node_is_null(type_node) || ts_node_is_null(pattern)) + continue; + const std::string param_name = nodeText(pattern); + const std::string param_type = + extractTsTypeAnnotation(type_node); + if (!param_name.empty() && !param_type.empty()) + recordVarType(param_name, param_type); + } +} + // ── Class Declaration (TS override: check type_identifier, push scope) ──── void TsVisitor::visitClassDecl(TSNode node, uint64_t parent_id) @@ -73,6 +146,25 @@ void TsVisitor::visitClassDecl(TSNode node, uint64_t parent_id) uint64_t cls_id = emitter_->emitClass(name, loc, parent_id); defineSymbol(name, cls_id); + // `class Foo implements Bar, Baz` — the TS grammar wraps the clause in + // class_heritage. Without this scan, no InterfaceImpl record was ever + // emitted for a TypeScript (or TSX) class. + for (uint32_t i = 0; i < count; i++) { + TSNode child = ts_node_child(node, i); + if (!ts_node_is_named(child)) + continue; + if (strcmp(ts_node_type(child), "class_heritage") != 0) + continue; + uint32_t hc = ts_node_child_count(child); + for (uint32_t j = 0; j < hc; j++) { + TSNode clause = ts_node_child(child, j); + if (ts_node_is_named(clause) && + strcmp(ts_node_type(clause), "implements_clause") == + 0) + emitImplementClause(clause, name, cls_id); + } + } + pushScope(); // Step 4: push class scope so this.method() resolves receiver_type // to this enclosing class name. @@ -82,6 +174,38 @@ void TsVisitor::visitClassDecl(TSNode node, uint64_t parent_id) popScope(); } +void TsVisitor::emitImplementClause(TSNode node, const std::string &impl_type, + uint64_t parent_id) +{ + uint32_t cnt = ts_node_child_count(node); + for (uint32_t i = 0; i < cnt; i++) { + TSNode c = ts_node_child(node, i); + if (!ts_node_is_named(c)) + continue; + const char *t = ts_node_type(c); + std::string iface; + if (strcmp(t, "type_identifier") == 0) { + iface = nodeText(c); + } else { + // generic_type (`Bar`) and nested_type_identifier + // (`ns.Bar`): record the base/last type_identifier and + // ignore the type arguments, which are not interfaces. + uint32_t cc = ts_node_child_count(c); + for (uint32_t j = 0; j < cc; j++) { + TSNode g = ts_node_child(c, j); + if (!ts_node_is_named(g)) + continue; + if (strcmp(ts_node_type(g), + "type_identifier") == 0) + iface = nodeText(g); + } + } + if (!iface.empty()) + emitter_->emitInterfaceImpl(impl_type, iface, + location(c), parent_id); + } +} + // ── Interface Declaration ──────────────────────────────────── void TsVisitor::visitInterfaceDecl(TSNode node, uint64_t parent_id) @@ -252,7 +376,7 @@ void TsVisitor::visitVariableDecl(TSNode node, uint64_t parent_id) if (strcmp(dt, "type_annotation") == 0) continue; if (ts_node_is_named(decl)) - visitNode(decl, parent_id); + visitChild(decl, parent_id); } } } @@ -275,20 +399,28 @@ std::string TsVisitor::extractTsTypeAnnotation(TSNode type_node) if (ts_node_is_null(type_node)) return ""; - // type_annotation wraps the actual type as its first named child. - uint32_t tc = ts_node_child_count(type_node); - TSNode inner = {}; - bool found_inner = false; - for (uint32_t i = 0; i < tc; i++) { - TSNode child = ts_node_child(type_node, i); - if (ts_node_is_named(child)) { - inner = child; - found_inner = true; - break; + // `type_annotation` is the wrapper (`: Foo`) and holds the type as its + // first named child. Every other node this is called with IS the type — + // the array/union/parenthesized branches below recurse with an element or + // member type, and unwrapping those as if they were annotations discarded + // the type entirely: `Foo[]` reached `type_identifier Foo`, looked for its + // (non-existent) first named child and returned "" — the element type of + // every array annotation was lost. + TSNode inner = type_node; + if (strcmp(ts_node_type(type_node), "type_annotation") == 0) { + const uint32_t tc = ts_node_child_count(type_node); + bool found_inner = false; + for (uint32_t i = 0; i < tc; i++) { + TSNode child = ts_node_child(type_node, i); + if (ts_node_is_named(child)) { + inner = child; + found_inner = true; + break; + } } + if (!found_inner) + return ""; } - if (!found_inner) - return ""; const char *it = ts_node_type(inner); @@ -373,15 +505,29 @@ std::string TsVisitor::extractTsTypeAnnotation(TSNode type_node) return ""; } - // Fallback: use the text of the inner node, stripping generics. + // Fallback: use the text of the inner node, stripping generics and array + // brackets — but only when what is left is a plain type name. + // + // The fallback used to return that text unconditionally, and for a shape + // this extractor does not model it therefore returned a fragment of the + // annotation instead of a type: `const tabs: { id: TabId; label: string }[]` + // recorded the receiver type of `tabs.map(...)` as `id: TabId` (measured on + // codebase-memory-mcp). No declaration is keyed by a name containing a + // space, so such a value can never match a candidate — it is wrong data in + // the database and in every tool that prints it, and "" (unknown) is the + // honest answer for an object, tuple or function type. std::string txt = nodeText(inner); size_t lt = txt.find('<'); if (lt != std::string::npos) - return txt.substr(0, lt); - // Strip array brackets. + txt = txt.substr(0, lt); size_t lb = txt.find('['); if (lb != std::string::npos) - return txt.substr(0, lb); + txt = txt.substr(0, lb); + for (char ch : txt) { + if (!std::isalnum(static_cast(ch)) && + ch != '_' && ch != '.' && ch != '$') + return ""; + } return txt; } diff --git a/engine/src/ir/translators/ts_visitor.h b/engine/src/ir/translators/ts_visitor.h index 25d6510..ca47eb6 100644 --- a/engine/src/ir/translators/ts_visitor.h +++ b/engine/src/ir/translators/ts_visitor.h @@ -50,6 +50,20 @@ class TsVisitor : public JsVisitor { // for `r.render()`. void visitVariableDecl(TSNode node, uint64_t parent_id) override; + /// Record every annotated parameter of a function-like node into var_types_, + /// so visitCallExpr can fill receiver_type for `o.method()` when `o` is a + /// parameter. + /// + /// The shared JavaScript handlers walk `formal_parameters` for the parameter + /// bodies only, so a parameter's declared type was never recorded: the call + /// fact carried receiver_text with an EMPTY receiver_type, and when two + /// types declare the same method name the resolver had nothing to choose + /// with and abstained — a false negative Go, Java, Python, C++ and Rust do + /// not produce for the same call shape (measured in + /// test_resolver_language_consistency). JavaScript has no annotations, so + /// this lives in the TypeScript visitor. + void recordParameterTypes(TSNode fn_node); + // ── TypeScript-specific handlers ─────────────────────────── /** * Handle interface_declaration. @@ -71,11 +85,30 @@ class TsVisitor : public JsVisitor { */ void visitEnumDecl(TSNode node, uint64_t parent_id); - private: + /** + * Emit one InterfaceImpl record per type named in a TS + * `implements_clause` subtree (`class Foo implements Bar, Baz`). + * The clause lists the type nodes directly, so this walks them and + * records each base type name; without it every `implements` edge was + * silently dropped for TypeScript (and therefore TSX). + * \param node The implements_clause node. + * \param impl_type Name of the implementing class. + * \param parent_id Record to attach the impl records to (the class). + */ + void emitImplementClause(TSNode node, const std::string &impl_type, + uint64_t parent_id); + + protected: /// Extract the bare type name from a TS type annotation node, - /// stripping generics (`Array` → "Array"), union/intersection - /// (takes first member), and array brackets (`T[]` → "T"). - /// Returns empty string if the type cannot be determined. + /// stripping generics (`Array` → "Array"), union/intersection (takes + /// the first member) and array brackets (`T[]` → "T"). + /// + /// Returns "" — UNKNOWN — when the annotation is not a plain (optionally + /// qualified) type name. An object, tuple or function type is not something + /// a declaration can be keyed by, and returning a fragment of it put wrong + /// data in the database and in every tool that printed it (see the fallback + /// in the definition). test_ts_visitor reaches it through a derived probe + /// rather than by widening this contract. std::string extractTsTypeAnnotation(TSNode type_node); }; diff --git a/engine/src/ir/translators/tsx_visitor.cpp b/engine/src/ir/translators/tsx_visitor.cpp index 54a31a3..a34aae8 100644 --- a/engine/src/ir/translators/tsx_visitor.cpp +++ b/engine/src/ir/translators/tsx_visitor.cpp @@ -21,6 +21,12 @@ SemanticUnit *TsxVisitor::visit(TSTree *tree, const char *source, source_ = source; TSNode root_node = ts_tree_root_node(tree); + // Names this file defines, so a user function whose name collides with a + // JS/TS builtin is not dropped by visitCallExpr's builtin filter. This + // override of visit() never called JsVisitor::visit(), so defined_names_ + // stayed empty for .tsx files. + defined_names_.clear(); + collectDefinedNames(root_node); pushScope(); SourceRange root_loc = location(root_node); uint64_t root_id = emitter_->emitVariable("", root_loc, 0); @@ -36,9 +42,38 @@ void TsxVisitor::visitNode(TSNode node, uint64_t parent_id) { const char *type = ts_node_type(node); - // ── JSX: skip self-closing elements (no semantic value) ──── - if (strcmp(type, "jsx_self_closing_element") == 0) - return; // Skip — no children with JS code + // ── JSX: visit JS inside self-closing element attributes ─── + // `` carries real JS in the jsx_expression of its + // attributes. Returning immediately (the previous behaviour) dropped every + // call inside a self-closing element. Only the jsx_expression subtrees are + // visited: the tag name and literal attribute values carry no code, and + // recursing into them would add Variable/Literal noise. + if (strcmp(type, "jsx_self_closing_element") == 0) { + uint32_t cnt = ts_node_child_count(node); + for (uint32_t i = 0; i < cnt; i++) { + TSNode child = ts_node_child(node, i); + if (!ts_node_is_named(child)) + continue; + const char *ct = ts_node_type(child); + // Spread attribute: ``. + if (strcmp(ct, "jsx_expression") == 0) { + visitChildren(child, parent_id); + continue; + } + if (strcmp(ct, "jsx_attribute") != 0) + continue; + // jsx_attribute → property_identifier + (jsx_expression | string) + uint32_t ac = ts_node_child_count(child); + for (uint32_t j = 0; j < ac; j++) { + TSNode entry = ts_node_child(child, j); + if (ts_node_is_named(entry) && + strcmp(ts_node_type(entry), + "jsx_expression") == 0) + visitChildren(entry, parent_id); + } + } + return; + } // ── JSX elements: recurse into children to find JSX // expressions that may contain meaningful JS code ───── diff --git a/engine/src/ir/translators/typescript_translator.cpp b/engine/src/ir/translators/typescript_translator.cpp index 96e3358..0580118 100644 --- a/engine/src/ir/translators/typescript_translator.cpp +++ b/engine/src/ir/translators/typescript_translator.cpp @@ -23,6 +23,8 @@ class TypescriptTranslator : public Translator { TranslationUnit *unit_ = nullptr; const char *source_ = nullptr; std::string file_path_; + // Recursion counter for this file (see kMaxTranslateDepth). + TranslateDepth depth_; struct Scope { std::unordered_map symbols; @@ -83,6 +85,7 @@ TranslationUnit *TypescriptTranslator::translate(TSTree *tree, const char *source, const char *file_path) { + depth_.reset(); unit_ = new TranslationUnit(); unit_->source_content = source; source_ = source; @@ -373,6 +376,13 @@ Node *TypescriptTranslator::translateNode(TSNode ts_node, Node *parent) void TypescriptTranslator::translateChildren(TSNode ts_node, Node *parent) { + // Bound native-stack recursion (see kMaxTranslateDepth): a pathologically + // deep AST would overflow the indexer's 512 KB worker stack, which the FFI + // try/catch cannot recover. The visitors apply the same bound. + if (depth_.exceeded(file_path_.c_str(), "translateChildren")) + return; + DepthGuard depth_guard(depth_); + uint32_t count = ts_node_child_count(ts_node); for (uint32_t i = 0; i < count; i++) { TSNode child = ts_node_child(ts_node, i); diff --git a/engine/src/lsp/lsp_client.cpp b/engine/src/lsp/lsp_client.cpp index c9c197c..f6ca6cc 100644 --- a/engine/src/lsp/lsp_client.cpp +++ b/engine/src/lsp/lsp_client.cpp @@ -1,4 +1,6 @@ #include "lsp_client.h" +#include "util/json_writer.h" +#include "lsp_framing.h" #include "platform_win.h" #include @@ -21,6 +23,12 @@ static constexpr int kSpawnWaitUs = 100000; // 100 ms grace before checking child static constexpr size_t kMaxResponseBytes = 1 << 20; // 1 MiB cap on LSP response +// Deadline for one write to the server's stdin. The pipe buffer holds ~64KB, so +// a live server drains this well before the deadline; only a server that +// stopped reading its stdin can reach it. +static constexpr int kWriteTimeoutMs = 5000; +// Grace period for the server to exit after the `exit` notification. +static constexpr int kExitGraceMs = 3000; // ─── JSON-RPC helpers (minimal, no external deps) ───────────── @@ -55,6 +63,12 @@ static std::string wrapLspMessage(const std::string &body) return msg.str(); } +// Message framing (takeNextLspMessage) and response matching (isResponseFor) +// live in lsp_framing.h: they are pure string handling, and the defects they +// fix only show up when a server interleaves notifications with responses, so +// they are tested directly (tests/test_lsp_framing.cpp) instead of through a +// live server. + // ─── LspClient implementation ───────────────────────────────── LspClient::~LspClient() @@ -103,6 +117,21 @@ bool LspClient::start(const char *command, const char *root_uri) return true; } +/// Escape a string for embedding inside a JSON string literal. +/// Handles `"`, `\`, control characters and newlines so a hostile source +/// buffer cannot break the JSON-RPC frame for textDocument/didOpen. +/// @param raw NUL-terminated C string (may be nullptr → empty). +/// @return Escaped text safe to concatenate into a JSON string value. +/// +/// Delegates to util::jsonEscapeString, so the engine has one escaper. +/// That changes two bytes for backspace/form-feed: this function used to +/// emit \b and \f, the shared one emits \u0008 / \u000c. Both are valid +/// JSON and decode to the same character. +static std::string jsonEscapeLsp(const char *raw) +{ + return raw ? util::jsonEscapeString(raw) : std::string(); +} + bool LspClient::openDocument(const char *file_uri, const char *source_text) { if (!isRunning()) { @@ -113,10 +142,10 @@ bool LspClient::openDocument(const char *file_uri, const char *source_text) std::ostringstream params; params << "{" << "\"textDocument\":{" - << "\"uri\":\"" << file_uri << "\"," + << "\"uri\":\"" << jsonEscapeLsp(file_uri) << "\"," << "\"languageId\":\"python\"," << "\"version\":1," - << "\"text\":\"" << source_text << "\"" + << "\"text\":\"" << jsonEscapeLsp(source_text) << "\"" << "}}"; std::string notif = @@ -132,7 +161,8 @@ std::string LspClient::queryDefinition(const char *file_uri, int line, std::ostringstream params; params << "{" - << "\"textDocument\":{\"uri\":\"" << file_uri << "\"}," + << "\"textDocument\":{\"uri\":\"" << jsonEscapeLsp(file_uri) + << "\"}," << "\"position\":{\"line\":" << line << ",\"character\":" << column << "}" << "}"; @@ -154,7 +184,8 @@ std::string LspClient::queryHover(const char *file_uri, int line, int column) std::ostringstream params; params << "{" - << "\"textDocument\":{\"uri\":\"" << file_uri << "\"}," + << "\"textDocument\":{\"uri\":\"" << jsonEscapeLsp(file_uri) + << "\"}," << "\"position\":{\"line\":" << line << ",\"character\":" << column << "}" << "}"; @@ -176,7 +207,8 @@ std::string LspClient::queryDocumentSymbols(const char *file_uri) std::ostringstream params; params << "{" - << "\"textDocument\":{\"uri\":\"" << file_uri << "\"}" + << "\"textDocument\":{\"uri\":\"" << jsonEscapeLsp(file_uri) + << "\"}" << "}"; std::string req = buildJsonRpc("textDocument/documentSymbol", @@ -285,10 +317,36 @@ void LspClient::stop() stdin_fd_ = -1; stdout_fd_ = -1; - // Wait for process to exit + // Reap the child. WNOHANG alone returned immediately even when the server + // was still running, and because pid_ was cleared straight afterwards + // nothing ever waited for it again: the process stayed a zombie for the + // lifetime of the indexer. Wait for it within a grace period, force-kill it + // if it declines, and report the exit status so a server that failed is not + // silently treated as a clean shutdown. if (pid_ > 0) { - int status; - waitpid(pid_, &status, WNOHANG); + int status = 0; + bool reaped = false; + for (int waited = 0; waited < kExitGraceMs; waited += 50) { + const pid_t r = waitpid(pid_, &status, WNOHANG); + if (r == pid_) { + reaped = true; + break; + } + if (r < 0) + break; // ECHILD — already reaped elsewhere + usleep(50 * 1000); + } + if (!reaped) { + kill(pid_, SIGKILL); + waitpid(pid_, &status, 0); // reap the corpse + if (error_.empty()) + error_ = + "LSP server ignored the exit notification; killed"; + } else if (WIFEXITED(status) && WEXITSTATUS(status) != 0) { + if (error_.empty()) + error_ = "LSP server exited with status " + + std::to_string(WEXITSTATUS(status)); + } } pid_ = 0; #else @@ -476,6 +534,15 @@ bool LspClient::sendMessage(const std::string &body) } std::string msg = wrapLspMessage(body); +#ifndef _WIN32 + // A blocking write() to a server that stopped reading its stdin never + // returns, hanging the caller inside what should be a bounded query. Wait + // for writability with a deadline first. + if (!waitWritable(kWriteTimeoutMs)) { + error_ = "timeout writing to LSP server"; + return false; + } +#endif #ifndef _WIN32 ssize_t written = write(stdin_fd_, msg.c_str(), msg.size()); #else @@ -490,6 +557,14 @@ bool LspClient::sendMessage(const std::string &body) size_t offset = (written > 0) ? static_cast(written) : 0; while (offset < msg.size()) { +#ifndef _WIN32 + // Same deadline inside the retry loop: a partial write means the + // pipe is full, i.e. the server is not draining it. + if (!waitWritable(kWriteTimeoutMs)) { + error_ = "timeout writing to LSP server"; + return false; + } +#endif #ifndef _WIN32 ssize_t n = write(stdin_fd_, msg.data() + offset, msg.size() - offset); @@ -511,14 +586,25 @@ bool LspClient::sendMessage(const std::string &body) return true; } +#ifndef _WIN32 +bool LspClient::waitWritable(int timeout_ms) +{ + struct pollfd pfd; + pfd.fd = stdin_fd_; + pfd.events = POLLOUT; + return poll(&pfd, 1, timeout_ms) > 0; +} +#endif + std::string LspClient::readResponse(int expected_id, int timeout_ms) { #ifndef _WIN32 if (stdout_fd_ < 0) return ""; - // Read until we find Content-Length header + body - std::string buffer; + // Reads accumulate in the persistent read_buffer_: one read can deliver + // several messages, and one message can span several reads. + std::string &buffer = read_buffer_; auto start_time = time(nullptr); while (true) { @@ -558,39 +644,23 @@ std::string LspClient::readResponse(int expected_id, int timeout_ms) buf[n] = '\0'; buffer += buf; - // Check if we have a complete message (Content-Length header + body) - auto header_end = buffer.find("\r\n\r\n"); - if (header_end != std::string::npos) { - // Parse Content-Length - auto cl_pos = buffer.find("Content-Length:"); - if (cl_pos == std::string::npos) - continue; - - auto val_start = cl_pos + 15; // after "Content-Length:" - while (val_start < buffer.size() && - buffer[val_start] == ' ') - val_start++; - auto val_end = buffer.find_first_of("\r\n", val_start); - if (val_end == std::string::npos) - continue; - - int content_length = std::atoi( - buffer.substr(val_start, val_end - val_start) - .c_str()); - if (content_length <= 0) - continue; - - size_t body_start = header_end + 4; // after \r\n\r\n - if (buffer.size() >= - body_start + static_cast(content_length)) { - std::string body = buffer.substr( - body_start, content_length); - - // In a real implementation, we'd match the id. - // For our minimal case, just return the body. - (void)expected_id; - return body; + // Take every message already buffered. Only a response whose id + // matches this request answers the call: a notification (server + // logging, diagnostics) or an earlier request's response is skipped + // and the loop keeps looking, rather than being returned as the + // answer. + std::string body; + for (;;) { + const LspFraming framed = + takeNextLspMessage(buffer, body); + if (framed == LspFraming::Message) { + if (isResponseFor(body, expected_id)) + return body; + continue; // notification or another request's response } + if (framed == LspFraming::Skipped) + continue; // unusable frame dropped, try the next + break; // NeedMore — read more bytes } // Prevent infinite loop on malformed data @@ -610,7 +680,8 @@ std::string LspClient::readResponse(int expected_id, int timeout_ms) if (stdout_fd_ < 0) return ""; - std::string buffer; + // Same persistent buffer as the POSIX path (see read_buffer_). + std::string &buffer = read_buffer_; auto start_time = GetTickCount64(); while (true) { @@ -637,39 +708,20 @@ std::string LspClient::readResponse(int expected_id, int timeout_ms) buf[bytes_read] = '\0'; buffer += buf; - // Check for complete message (Content-Length header + body) - auto header_end = buffer.find("\r\n\r\n"); - if (header_end != std::string::npos) { - auto cl_pos = buffer.find("Content-Length:"); - if (cl_pos != std::string::npos) { - auto val_start = cl_pos + 15; - while (val_start < buffer.size() && - buffer[val_start] == ' ') - val_start++; - auto val_end = buffer.find_first_of( - "\r\n", val_start); - if (val_end != std::string::npos) { - int content_length = std::atoi( - buffer.substr(val_start, - val_end - - val_start) - .c_str()); - if (content_length > 0) { - size_t body_start = - header_end + 4; - if (buffer.size() >= - body_start + - static_cast< - size_t>( - content_length)) { - (void)expected_id; - return buffer.substr( - body_start, - content_length); - } - } - } + // Same message framing and id matching as the POSIX path: a + // notification is not the answer to this request. + std::string body; + for (;;) { + const LspFraming framed = + takeNextLspMessage(buffer, body); + if (framed == LspFraming::Message) { + if (isResponseFor(body, expected_id)) + return body; + continue; } + if (framed == LspFraming::Skipped) + continue; + break; } } diff --git a/engine/src/lsp/lsp_client.h b/engine/src/lsp/lsp_client.h index dd883c8..b9c622d 100644 --- a/engine/src/lsp/lsp_client.h +++ b/engine/src/lsp/lsp_client.h @@ -154,6 +154,25 @@ class LspClient { // Read a JSON-RPC response from the server (blocks until received). // Returns the body of the response, or empty on timeout/failure. std::string readResponse(int expected_id, int timeout_ms = 5000); + + /** + * Bytes received from the server but not yet consumed. + * + * The LSP stream is framed, not one-message-per-read: a read can deliver a + * `window/logMessage` notification and the response in the same chunk, and + * a single message can be split across reads. A buffer local to one + * readResponse() call threw away whatever followed the message it + * returned, and returned the notification when one arrived first. + */ + std::string read_buffer_; + +#ifndef _WIN32 + /** + * Wait until the server's stdin accepts more bytes, up to `timeout_ms`. + * False on timeout or error. See sendMessage(). + */ + bool waitWritable(int timeout_ms); +#endif }; #endif // LSP_CLIENT_H diff --git a/engine/src/lsp/lsp_framing.h b/engine/src/lsp/lsp_framing.h new file mode 100644 index 0000000..d833c10 --- /dev/null +++ b/engine/src/lsp/lsp_framing.h @@ -0,0 +1,88 @@ +#ifndef LSP_FRAMING_H +#define LSP_FRAMING_H + +// LSP message framing and response matching. +// +// Split out of lsp_client.cpp so the parsing rules can be tested directly: they +// are pure string handling, and the bugs they fix are invisible from the +// public API (they only show up when a real server interleaves notifications +// with responses). + +#include +#include + +/// Result of one framing attempt. +enum class LspFraming { + Message, // out_body holds a complete message; its bytes were consumed + NeedMore, // the buffer holds no complete message yet + Skipped, // an unusable frame was discarded; try the next one +}; + +// Try to take one complete LSP-framed message out of `buffer`. +// +// LSP frames each message as `Content-Length: N\r\n\r\n`; a stream read +// can deliver several messages, part of one, or a whole message plus the start +// of the next. +// +// The third result (Skipped) is not decoration: when junk framing is discarded +// the caller must come back for the message behind it rather than wait for more +// bytes. Reporting that as NeedMore stalls a caller whose buffer already holds +// a complete answer. +inline LspFraming takeNextLspMessage(std::string &buffer, std::string &out_body) +{ + auto header_end = buffer.find("\r\n\r\n"); + if (header_end == std::string::npos) + return LspFraming::NeedMore; + auto cl_pos = buffer.find("Content-Length:"); + if (cl_pos == std::string::npos || cl_pos > header_end) + return LspFraming::NeedMore; + auto val_start = cl_pos + 15; // after "Content-Length:" + while (val_start < buffer.size() && buffer[val_start] == ' ') + val_start++; + auto val_end = buffer.find_first_of("\r\n", val_start); + if (val_end == std::string::npos) + return LspFraming::NeedMore; + int content_length = std::atoi( + buffer.substr(val_start, val_end - val_start).c_str()); + if (content_length <= 0) { + // A header without a usable length can never be framed. Drop it so + // the caller does not spin on the same bytes forever. + buffer.erase(0, header_end + 4); + return LspFraming::Skipped; + } + const size_t body_start = header_end + 4; + if (buffer.size() < body_start + static_cast(content_length)) + return LspFraming::NeedMore; + out_body = + buffer.substr(body_start, static_cast(content_length)); + buffer.erase(0, body_start + static_cast(content_length)); + return LspFraming::Message; +} + +// Wrap a JSON-RPC body in LSP framing. +inline std::string wrapLspFramed(const std::string &body) +{ + return "Content-Length: " + std::to_string(body.size()) + "\r\n\r\n" + + body; +} + +// True when `body` is a JSON-RPC RESPONSE carrying `expected_id`. +// +// A notification carries no id, and a response to an earlier request carries a +// different one: neither answers the call in progress. Returning whichever +// message arrived first handed the caller a `window/logMessage` notification +// where it expected a definition result. +inline bool isResponseFor(const std::string &body, int expected_id) +{ + auto pos = body.find("\"id\""); + if (pos == std::string::npos) + return false; + pos += 4; + while (pos < body.size() && (body[pos] == ' ' || body[pos] == ':')) + pos++; + if (pos >= body.size() || body[pos] < '0' || body[pos] > '9') + return false; + return std::atoi(body.c_str() + pos) == expected_id; +} + +#endif // LSP_FRAMING_H diff --git a/engine/src/model/plugins/architecture.cpp b/engine/src/model/plugins/architecture.cpp index 00288aa..e645b03 100644 --- a/engine/src/model/plugins/architecture.cpp +++ b/engine/src/model/plugins/architecture.cpp @@ -32,6 +32,19 @@ ModelResult ArchitecturePlugin::build(uint64_t project_id, ModelResult r; r.plugin_name = "Architecture"; + // Clear previous rows so a rebuild is idempotent. Without this every + // enhance/async rebuild APPENDs a second copy of each cross-module + // edge and StateBuilder's COUNT(*) doubles, triples, … (the unfixed + // twin of the old architecture_state accumulation bug). + if (!store_->exec("DELETE FROM architecture_edge")) { + // Surface the failure (code_rules: no silent error handling) + // and refuse to append more rows on top of a half-cleared table. + r.error = "[module=model, method=ArchitecturePlugin::build] " + "DELETE architecture_edge failed: " + + store_->error(); + return r; + } + // Lazily compute the matching module scopes for each entity, cached // so that entities appearing in multiple call edges are scanned once. std::unordered_map> @@ -52,7 +65,10 @@ ModelResult ArchitecturePlugin::build(uint64_t project_id, // For each call edge, emit an architecture_edge for each // cross-module (src_mod != tgt_mod) scope pair. The entity_id // stored is the target (callee), matching the original SQL's - // `r.target_id`. + // `r.target_id`. The module columns follow the call direction: + // caller_module = the source module, callee_module = the target module. + // (They were named layer_upper/layer_lower until v0.7, which is what made + // the model layer read a plain cross-module call as a layer violation.) int64_t edges = 0; for (const auto &edge : ctx.call_edges) { auto src_it = ctx.entities_by_id.find(edge.source_id); diff --git a/engine/src/model/plugins/capability.cpp b/engine/src/model/plugins/capability.cpp index 3d6cf19..87f5b23 100644 --- a/engine/src/model/plugins/capability.cpp +++ b/engine/src/model/plugins/capability.cpp @@ -67,9 +67,16 @@ ModelResult CapabilityPlugin::build(uint64_t project_id, for (const auto &doc : ctx.documents) { const std::string &text = doc.content; size_t pos = 0; - while ((pos = text.find_first_of("-\n*", pos)) != - std::string::npos) { - size_t end = text.find('\n', pos + 1); + // Capabilities are LINES that mention a supported feature. The scan + // used to advance on '-' and '*' as well as '\n', so it restarted + // mid-line: "A thread-safe design - Supports fast lookup" yielded a + // capability named after the fragment that follows the hyphen + // ("FastLookup") — half a sentence, a name no code can match, which + // the capability-drift detector then reported as undocumented. The + // bullet markers themselves are still handled: normalizeCapabilityName + // strips leading " \t-*". + while (pos < text.size()) { + size_t end = text.find('\n', pos); if (end == std::string::npos) end = text.size(); std::string line = text.substr(pos, end - pos); @@ -87,6 +94,8 @@ ModelResult CapabilityPlugin::build(uint64_t project_id, doc.file_path); ++caps; } + if (end >= text.size()) + break; pos = end + 1; } } diff --git a/engine/src/model/plugins/contract.cpp b/engine/src/model/plugins/contract.cpp index aded3ff..a4d1aea 100644 --- a/engine/src/model/plugins/contract.cpp +++ b/engine/src/model/plugins/contract.cpp @@ -1,4 +1,5 @@ #include "contract.h" +#include #include #include @@ -82,6 +83,16 @@ ModelResult ContractPlugin::build(uint64_t project_id, const ModelContext &ctx) "lock free", "not safe", "unsafe" }; for (auto *kw : keywords) { + // Negative forms must not become positive contracts: + // "not safe" / "unsafe" in a README are denials. + std::string kw_lower = kw; + for (auto &c : kw_lower) + c = static_cast(std::tolower( + static_cast(c))); + if (kw_lower.find("not") != std::string::npos || + kw_lower.find("unsafe") != std::string::npos) { + continue; + } if (containsCI(text, kw)) { // Canonicalise the contract name before insert so // ContractVerifier's lowercase-exact router matches. @@ -92,9 +103,63 @@ ModelResult ContractPlugin::build(uint64_t project_id, const ModelContext &ctx) // lowercase to "thread-safe" / "memory safe" (still // with separators) and NEVER match the router, so // "thread-safe" claims silently returned Unknown and - // trust_score was penalised. See - // CODE_REVIEW_FINDINGS_2026-07-19.md H1. + // trust_score was penalised. // Canonical form: lowercase + strip '-' and spaces. + // + // Negation guard (same polarity rule as + // ClaimParser): skip when the occurrence is + // preceded by a denial ("not thread-safe"). + // Suffix-only checks: a denial earlier in the + // sentence ("don't touch X … thread-safe") must + // not suppress an unrelated positive claim. + std::string text_lower = text; + for (auto &c : text_lower) + c = static_cast(std::tolower( + static_cast(c))); + size_t pos = text_lower.find(kw_lower); + if (pos != std::string::npos) { + std::string before = + text_lower.substr(0, pos); + // Suffix checks with a word-boundary guard + // only on the standalone negators ("not ", + // "never "): "cannot thread-safe" ends in + // "not " but is NOT a denial (same rule as + // ClaimParser's `\bnot`). Contractions + // ("isn't", "don't", "won't") always carry + // a letter before "n't", so a boundary check + // on that suffix would reject every real + // contraction — do not apply one there. + auto ends_suffix = [&](const char *sfx) { + size_t n = std::strlen(sfx); + if (before.size() < n) + return false; + return before.compare( + before.size() - + n, + n, sfx) == 0; + }; + auto ends_word = [&](const char *sfx) { + size_t n = std::strlen(sfx); + if (!ends_suffix(sfx)) + return false; + size_t start = + before.size() - n; + if (start == 0) + return true; + unsigned char prev = static_cast< + unsigned char>( + before[start - 1]); + return !std::isalnum(prev); + }; + // "not thread-safe", "never thread-safe", + // "isn't thread-safe", "is not thread-safe". + if (ends_word("not ") || + ends_word("never ") || + ends_suffix("n't ") || + ends_suffix("n't")) { + continue; + } + } std::string canonical = normalizeContractName(kw); store_->insertContract(project_id, canonical, diff --git a/engine/src/model/plugins/workflow.cpp b/engine/src/model/plugins/workflow.cpp index 5c30764..cca8119 100644 --- a/engine/src/model/plugins/workflow.cpp +++ b/engine/src/model/plugins/workflow.cpp @@ -1,5 +1,6 @@ #include "workflow.h" #include +#include #include #include @@ -31,6 +32,26 @@ ModelResult WorkflowPlugin::build(uint64_t project_id, const ModelContext &ctx) ModelResult r; r.plugin_name = "Workflow"; + // Rebuild idempotently. insertWorkflow is guarded (it returns the + // existing row id), but insertWorkflowStep is a plain INSERT with no + // dedup and workflow_step has no UNIQUE, so each rebuild (every enhance + // / model build) appended a second copy of every step — inflating + // workflow_state.steps_done / steps_total. Clear this project's steps + // first so a rebuild replaces them instead of accumulating. + { + const std::string del = + "DELETE FROM workflow_step WHERE workflow_id IN " + "(SELECT id FROM workflow WHERE project_id=" + + std::to_string(project_id) + ")"; + if (!store_->exec(del.c_str())) { + r.error = std::string("[module=workflow, method=build] " + "delete workflow_step failed: ") + + store_->error(); + fprintf(stderr, "%s\n", r.error.c_str()); + return r; + } + } + // Build the set of entity ids that are the source of at least one // relation. This replicates the original `EXISTS (SELECT 1 FROM // relation r WHERE r.source_id = e.id)` filter. diff --git a/engine/src/model/project_state_builder.cpp b/engine/src/model/project_state_builder.cpp index 7a9eb73..c7d672b 100644 --- a/engine/src/model/project_state_builder.cpp +++ b/engine/src/model/project_state_builder.cpp @@ -24,6 +24,7 @@ // omits the corresponding key (or sets its score to a default). #include "project_state_builder.h" +#include "util/json_writer.h" #include #include @@ -38,6 +39,7 @@ #include #include "../evidence/evidence_builder.h" +#include "../evidence/rule.h" #include "../store/store.h" namespace model @@ -93,37 +95,7 @@ std::string colText(sqlite3_stmt *stmt, int col) // Mirrors the jsonEscape helper used across the engine. std::string escapeJson(const std::string &s) { - std::string out; - out.reserve(s.size() + 8); - for (char c : s) { - switch (c) { - case '"': - out += "\\\""; - break; - case '\\': - out += "\\\\"; - break; - case '\n': - out += "\\n"; - break; - case '\r': - out += "\\r"; - break; - case '\t': - out += "\\t"; - break; - default: - if (static_cast(c) < 0x20) { - char buf[8]; - std::snprintf(buf, sizeof(buf), "\\u%04x", - static_cast(c)); - out += buf; - } else { - out += c; - } - } - } - return out; + return util::jsonEscapeString(s); } // Format a double as a JSON number with up to 4 decimal places. @@ -321,22 +293,35 @@ int64_t countVerifiedCapabilities(store::GraphStore *store, uint64_t project_id) return count; } -// Sum architecture_state.violations for a project. Returns 0 on -// error or empty table. -int64_t sumArchitectureViolations(store::GraphStore *store, uint64_t project_id) +// Sum one architecture_state column for a project. `column` is chosen by +// the callers below (never client input) AND checked against an allowlist, +// because it is interpolated into the SQL — together the two keep this a +// constant query. Returns 0 on error or empty table. +static int64_t sumArchitectureColumn(store::GraphStore *store, + uint64_t project_id, const char *column) { + if (std::strcmp(column, "violations") != 0 && + std::strcmp(column, "cross_module_edges") != 0) { + std::fprintf( + stderr, + "[module=project_state, method=sumArchitectureColumn] " + "refusing non-allowlisted column '%s'\n", + column); + return 0; + } if (!store || !store->handle()) return 0; - const char *sql = "SELECT COALESCE(SUM(violations), 0) " - "FROM architecture_state WHERE project_id = ?"; + const std::string sql = std::string("SELECT COALESCE(SUM(") + column + + "), 0) FROM architecture_state " + "WHERE project_id = ?"; sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(store->handle(), sql, -1, &stmt, nullptr) != - SQLITE_OK) { + if (sqlite3_prepare_v2(store->handle(), sql.c_str(), -1, &stmt, + nullptr) != SQLITE_OK) { std::fprintf( stderr, - "[module=project_state, method=sumArchitectureViolations] " - "prepare failed: %s\n", - sqlite3_errmsg(store->handle())); + "[module=project_state, method=sumArchitectureColumn] " + "prepare failed for column '%s': %s\n", + column, sqlite3_errmsg(store->handle())); return 0; } StmtGuard guard(stmt); @@ -348,6 +333,24 @@ int64_t sumArchitectureViolations(store::GraphStore *store, uint64_t project_id) return count; } +// Sum architecture_state.violations for a project. Returns 0 on error or +// empty table. Always 0 today: calling a cross-module dependency a violation +// needs a layer model, which this graph does not carry — see +// StateBuilder::buildArchitectureState. +int64_t sumArchitectureViolations(store::GraphStore *store, uint64_t project_id) +{ + return sumArchitectureColumn(store, project_id, "violations"); +} + +// Sum architecture_state.cross_module_edges: how many call edges cross a +// module boundary, summed over the tracked module pairs. This is a +// dependency count, NOT a violation count. +int64_t sumArchitectureCrossModuleEdges(store::GraphStore *store, + uint64_t project_id) +{ + return sumArchitectureColumn(store, project_id, "cross_module_edges"); +} + // Sum workflow_state.steps_done and steps_total across all workflows // for a project. Returns (steps_done, steps_total). Used for the // workflow score = done / total. @@ -521,17 +524,6 @@ bool upsertProjectState(store::GraphStore *store, uint64_t project_id, return true; } -// Resolve the rules directory from CODESCOPE_RULES_DIR env var, or -// fall back to the default relative path "engine/src/evidence/rules". -// Mirrors the fallback used in engine_evidence_ffi.cpp. -std::string resolveRulesDir() -{ - const char *env_dir = std::getenv("CODESCOPE_RULES_DIR"); - if (env_dir && *env_dir) - return std::string(env_dir); - return "engine/src/evidence/rules"; -} - } // namespace // ─── ProjectStateBuilder public API ────────────────────────────── @@ -546,7 +538,7 @@ bool ProjectStateBuilder::build(uint64_t project_id) // 1. Run evidence::EvidenceBuilder::buildAll to get all evidence. evidence::EvidenceBuilder ev_builder(store_); - ev_builder.loadRules(resolveRulesDir()); + ev_builder.loadRules(evidence::resolveRulesDir()); auto evidences = ev_builder.buildAll(project_id); // 2. Aggregate evidence by category (count + titles). @@ -594,6 +586,8 @@ bool ProjectStateBuilder::build(uint64_t project_id) int64_t capability_verified = countVerifiedCapabilities(store_, project_id); int64_t arch_violations = sumArchitectureViolations(store_, project_id); + int64_t arch_cross_module = + sumArchitectureCrossModuleEdges(store_, project_id); int64_t workflow_total = countRows(store_, "SELECT COUNT(*) FROM workflow_state " @@ -607,7 +601,10 @@ bool ProjectStateBuilder::build(uint64_t project_id) double confidence = computeOverallConfidence(agg, arch_violations, dead_entities); - // Architecture score = 1 - violations * penalty (clamped). + // Architecture score = 1 - violations * penalty (clamped). `violations` + // counts only real layer violations, so a project is no longer penalised + // for having cross-module dependencies; those are reported as + // cross_module_edges alongside the score. double arch_score = 1.0 - kPenaltyPerArchitectureViolation * static_cast(arch_violations); if (arch_score < 0.0) @@ -689,7 +686,8 @@ bool ProjectStateBuilder::build(uint64_t project_id) // architecture ss << ",\"architecture\":{\"score\":" << fmtDouble(arch_score) - << ",\"violations\":" << arch_violations << "}"; + << ",\"violations\":" << arch_violations + << ",\"cross_module_edges\":" << arch_cross_module << "}"; // workflow ss << ",\"workflow\":{\"score\":" << fmtDouble(workflow_score) diff --git a/engine/src/model/semantic_fact_extractor.cpp b/engine/src/model/semantic_fact_extractor.cpp index bcb0c0b..db2fbdc 100644 --- a/engine/src/model/semantic_fact_extractor.cpp +++ b/engine/src/model/semantic_fact_extractor.cpp @@ -14,6 +14,7 @@ // by extractFrameworkFacts. #include "semantic_fact_extractor.h" +#include "util/json_writer.h" #include #include @@ -41,7 +42,11 @@ static constexpr int kNodeTypeMethod = 1; // NodeType::Method static constexpr double kConfidenceDefault = 1.0; static constexpr double kConfidenceIgnoredReturn = 0.8; static constexpr double kConfidenceUncheckedError = 0.7; -static constexpr double kConfidenceCgoCallback = 0.6; +// The confidence the `cgo_callback` rule will use once that rule exists. It is +// deliberately kept rather than deleted — it is the only remaining reference to +// the intended rule — and marked maybe_unused so the build stays warning-free +// (2026-09-18 review #9b). +[[maybe_unused]] static constexpr double kConfidenceCgoCallback = 0.6; // ─── Local helpers ──────────────────────────────────────────────── @@ -57,64 +62,11 @@ static std::string buildDetailJson(int line, const std::string &snippet, out += "{\"line\":"; out += std::to_string(line); out += ",\"snippet\":\""; - for (char c : snippet) { - switch (c) { - case '"': - out += "\\\""; - break; - case '\\': - out += "\\\\"; - break; - case '\n': - out += "\\n"; - break; - case '\r': - out += "\\r"; - break; - case '\t': - out += "\\t"; - break; - default: - if (static_cast(c) < 0x20) { - // Control char: emit \u00XX for safety. - char buf[8]; - snprintf(buf, sizeof(buf), "\\u%04x", - static_cast(c)); - out += buf; - } else { - out += c; - } - } - } + out += util::jsonEscapeString(snippet); + out += "\",\"related_symbol\":\""; - for (char c : related_symbol) { - switch (c) { - case '"': - out += "\\\""; - break; - case '\\': - out += "\\\\"; - break; - case '\n': - out += "\\n"; - break; - case '\r': - out += "\\r"; - break; - case '\t': - out += "\\t"; - break; - default: - if (static_cast(c) < 0x20) { - char buf[8]; - snprintf(buf, sizeof(buf), "\\u%04x", - static_cast(c)); - out += buf; - } else { - out += c; - } - } - } + out += util::jsonEscapeString(related_symbol); + out += "\"}"; return out; } @@ -182,6 +134,73 @@ int64_t SemanticFactExtractor::extractAll(uint64_t project_id) // .Add + WaitGroup qn → sync/waitgroup/add // (defer) → sync/defer (detected via name='defer' in Go) +/// Classify a Go synchronisation call name into the `primitive` / `kind` +/// columns of the sync facts. +/// +/// The name is the call's qualified Go selector as recorded by the visitor, +/// e.g. `m.Lock`, `rw.RLock`, `rw.RUnlock`, `wg.Add`, `counter.Load`, `defer`. +/// +/// The RWMutex forms are matched before the plain ones because ".RLock" also +/// ends in "Lock" and ".RUnlock" also ends in "Unlock". The previous inline +/// guard required a '.' immediately before "Lock", so `.RLock` / `.WLock` never +/// entered the lock branch at all — their facts were dropped entirely and the +/// `rwmutex_usage` rule was permanently empty, with the inner R/W checks left +/// unreachable. +/// +/// @param name Call name as recorded (may be unqualified). +/// @param primitive [out] "defer" / "mutex" / "rwmutex" / "atomic" / +/// "waitgroup" on success. +/// @param kind [out] "defer" / "lock" / "defer_unlock" / "load" / "add". +/// @return false when the name matches none of the modelled forms, so the +/// caller skips it instead of writing a fact with no meaning. +static bool classifySyncCall(const std::string &name, std::string &primitive, + std::string &kind) +{ + auto ends_with = [&name](const char *suffix) { + const size_t n = std::strlen(suffix); + return name.size() >= n && + name.compare(name.size() - n, n, suffix) == 0; + }; + if (name == "defer") { + primitive = "defer"; + kind = "defer"; + return true; + } + if (ends_with(".RLock") || ends_with(".WLock")) { + primitive = "rwmutex"; + kind = "lock"; + return true; + } + if (ends_with(".RUnlock") || ends_with(".WUnlock")) { + primitive = "rwmutex"; + kind = "defer_unlock"; + return true; + } + if (ends_with(".Lock")) { + primitive = "mutex"; + kind = "lock"; + return true; + } + if (ends_with(".Unlock")) { + // The Evidence Builder's mutex_without_defer_unlock rule treats + // this as the optional match (kind="defer_unlock"). + primitive = "mutex"; + kind = "defer_unlock"; + return true; + } + if (ends_with(".Load")) { + primitive = "atomic"; + kind = "load"; + return true; + } + if (ends_with(".Add")) { + primitive = "waitgroup"; + kind = "add"; + return true; + } + return false; +} + int64_t SemanticFactExtractor::extractSyncFacts(uint64_t project_id) { // One SELECT that scans CallExpr records and tags each match with @@ -192,10 +211,10 @@ int64_t SemanticFactExtractor::extractSyncFacts(uint64_t project_id) const char *sql = "SELECT sr.name, sr.qualified_name, sr.language, " " sr.start_row, sr.file_path, " - " (SELECT gn.id FROM graph_nodes gn " + " (SELECT gn.id FROM entity gn " " WHERE gn.project_id = sr.project_id " " AND gn.file_path = sr.file_path " - " AND gn.node_type IN (?, ?) " + " AND gn.kind IN (?, ?) " " AND gn.start_row <= sr.start_row " " AND (gn.end_row >= sr.start_row OR gn.end_row = 0) " " ORDER BY gn.start_row DESC LIMIT 1) AS fn_id " @@ -203,6 +222,7 @@ int64_t SemanticFactExtractor::extractSyncFacts(uint64_t project_id) "WHERE sr.project_id = ? AND sr.kind = ? " " AND (sr.name LIKE '%.Lock' OR sr.name LIKE '%.Unlock' " " OR sr.name LIKE '%.RLock' OR sr.name LIKE '%.WLock' " + " OR sr.name LIKE '%.RUnlock' OR sr.name LIKE '%.WUnlock' " " OR (sr.name LIKE '%.Load' AND sr.qualified_name LIKE '%atomic%') " " OR (sr.name LIKE '%.Add' AND sr.qualified_name LIKE '%WaitGroup%') " " OR (sr.language = 'go' AND sr.name = 'defer'))"; @@ -233,46 +253,8 @@ int64_t SemanticFactExtractor::extractSyncFacts(uint64_t project_id) std::string primitive; std::string kind; - if (name == "defer") { - primitive = "defer"; - kind = "defer"; - } else if (name.size() >= 5 && - name.compare(name.size() - 4, 4, "Lock") == 0 && - name.size() >= 6 && name[name.size() - 5] == '.') { - // .Lock or .RLock or .WLock — per v0.3 plan section 3.2, - // all lock acquisitions use kind="lock" regardless of - // mutex flavor (mutex/rwmutex). The primitive column - // disambiguates the lock type. - if (name.size() >= 6 && name[name.size() - 6] == 'R') { - primitive = "rwmutex"; - kind = "lock"; - } else if (name.size() >= 6 && - name[name.size() - 6] == 'W') { - primitive = "rwmutex"; - kind = "lock"; - } else { - primitive = "mutex"; - kind = "lock"; - } - } else if (name.size() >= 7 && - name.compare(name.size() - 6, 6, "Unlock") == 0) { - // Any .Unlock() call — the Evidence Builder's - // mutex_without_defer_unlock rule treats this as the - // optional match (defer_unlock kind). Per v0.3 plan - // section 3.2 this is kind="defer_unlock". - primitive = "mutex"; - kind = "defer_unlock"; - } else if (name.size() >= 6 && - name.compare(name.size() - 5, 5, ".Load") == 0) { - primitive = "atomic"; - kind = "load"; - } else if (name.size() >= 5 && - name.compare(name.size() - 4, 4, ".Add") == 0) { - primitive = "waitgroup"; - kind = "add"; - } else { - continue; // safety net — should not happen - } + if (!classifySyncCall(name, primitive, kind)) + continue; // not a synchronisation call we model std::string detail = buildDetailJson( start_row, name + " (" + file_path + ")", @@ -301,10 +283,10 @@ int64_t SemanticFactExtractor::extractMemoryFacts(uint64_t project_id) // C.CString / C.CBytes / C.free (Go cgo) + malloc/calloc/realloc/free. const char *sql = "SELECT sr.name, sr.start_row, sr.file_path, " - " (SELECT gn.id FROM graph_nodes gn " + " (SELECT gn.id FROM entity gn " " WHERE gn.project_id = sr.project_id " " AND gn.file_path = sr.file_path " - " AND gn.node_type IN (?, ?) " + " AND gn.kind IN (?, ?) " " AND gn.start_row <= sr.start_row " " AND (gn.end_row >= sr.start_row OR gn.end_row = 0) " " ORDER BY gn.start_row DESC LIMIT 1) AS fn_id " @@ -392,10 +374,10 @@ int64_t SemanticFactExtractor::extractErrorFacts(uint64_t project_id) // the Python/JS visitors. const char *sql = "SELECT sr.name, sr.language, sr.start_row, sr.file_path, " - " (SELECT gn.id FROM graph_nodes gn " + " (SELECT gn.id FROM entity gn " " WHERE gn.project_id = sr.project_id " " AND gn.file_path = sr.file_path " - " AND gn.node_type IN (?, ?) " + " AND gn.kind IN (?, ?) " " AND gn.start_row <= sr.start_row " " AND (gn.end_row >= sr.start_row OR gn.end_row = 0) " " ORDER BY gn.start_row DESC LIMIT 1) AS fn_id " @@ -483,10 +465,10 @@ int64_t SemanticFactExtractor::extractPatternFacts(uint64_t project_id) const char *sql = "SELECT sr.name, sr.kind, sr.language, sr.start_row, " " sr.file_path, " - " (SELECT gn.id FROM graph_nodes gn " + " (SELECT gn.id FROM entity gn " " WHERE gn.project_id = sr.project_id " " AND gn.file_path = sr.file_path " - " AND gn.node_type IN (?, ?) " + " AND gn.kind IN (?, ?) " " AND gn.start_row <= sr.start_row " " AND (gn.end_row >= sr.start_row OR gn.end_row = 0) " " ORDER BY gn.start_row DESC LIMIT 1) AS fn_id " @@ -601,10 +583,10 @@ int64_t SemanticFactExtractor::extractFrameworkFacts(uint64_t project_id) // fact to. import.file_path is the source file (populated by the // Python/JS/Go/Rust visitors when emitting Import records). const char *sql = "SELECT i.target_path, i.file_path, " - " (SELECT gn.id FROM graph_nodes gn " + " (SELECT gn.id FROM entity gn " " WHERE gn.project_id = i.project_id " " AND gn.file_path = i.file_path " - " AND gn.node_type IN (?, ?) " + " AND gn.kind IN (?, ?) " " ORDER BY gn.start_row LIMIT 1) AS fn_id " "FROM import i " "WHERE i.project_id = ? " @@ -698,10 +680,10 @@ int64_t SemanticFactExtractor::extractFfiFacts(uint64_t project_id) const char *sql = "SELECT sr.name, sr.qualified_name, sr.language, " " sr.start_row, sr.file_path, sr.kind, " - " (SELECT gn.id FROM graph_nodes gn " + " (SELECT gn.id FROM entity gn " " WHERE gn.project_id = sr.project_id " " AND gn.file_path = sr.file_path " - " AND gn.node_type IN (?, ?) " + " AND gn.kind IN (?, ?) " " AND gn.start_row <= sr.start_row " " AND (gn.end_row >= sr.start_row OR gn.end_row = 0) " " ORDER BY gn.start_row DESC LIMIT 1) AS fn_id " @@ -784,26 +766,31 @@ int64_t SemanticFactExtractor::extractFfiFacts(uint64_t project_id) } sqlite3_finalize(stmt); - // ── Query 2: Cross-language call edges (graph_edges) ────────── + // ── Query 2: Cross-language call edges (relation) ───────────── // Detects Rust extern "C" functions and C functions declared in // extern "C" blocks by finding functions whose callers come from a // different language (e.g. Rust function called from C code). + // Reads the canonical `entity` / `relation` tables: `graph_nodes` / + // `graph_edges` are deprecated and are never populated by the + // production indexing path (engine_index_project → buildGraph), so a + // query against them silently yields zero facts. `relation.type = 1` + // is Calls (kRelationTypeCall). const char *sql_xl = - "SELECT gn.name, gn.qualified_name, gn.language, " - " gn.start_row, gn.file_path, gn.id AS fn_id " - "FROM graph_nodes gn " - "WHERE gn.project_id = ? " - " AND gn.node_type IN (?, ?) " - " AND gn.id IN ( " - " SELECT ge.target_node_id " - " FROM graph_edges ge " - " JOIN graph_nodes gn_src ON ge.source_node_id = gn_src.id " - " WHERE ge.project_id = ? " - " AND gn_src.language != gn.language " - " AND gn.language IN ('rust', 'c', 'cpp', 'zig')" + "SELECT e.name, e.qualified_name, e.language, " + " e.start_row, e.file_path, e.id AS fn_id " + "FROM entity e " + "WHERE e.project_id = ? " + " AND e.kind IN (?, ?) " + " AND e.id IN ( " + " SELECT r.target_id FROM relation r " + " JOIN entity src ON r.source_id = src.id " + " WHERE r.project_id = ? " + " AND r.type = 1 " + " AND src.language != e.language " + " AND e.language IN ('rust', 'c', 'cpp', 'zig')" " ) " " -- Exclude functions already detected by Query 1 " - " AND gn.id NOT IN (SELECT function_id FROM semantic_fact " + " AND e.id NOT IN (SELECT function_id FROM semantic_fact " " WHERE project_id = ? AND category = 'ffi')"; sqlite3_stmt *stmt_xl = nullptr; if (sqlite3_prepare_v2(store_->handle(), sql_xl, -1, &stmt_xl, diff --git a/engine/src/model/state_builder.cpp b/engine/src/model/state_builder.cpp index 9a3f0a6..2a36142 100644 --- a/engine/src/model/state_builder.cpp +++ b/engine/src/model/state_builder.cpp @@ -1,6 +1,9 @@ #include "state_builder.h" +#include "../verify/capability_verifier.h" #include #include +#include +#include #include namespace model @@ -24,14 +27,15 @@ int64_t StateBuilder::buildModuleSummaries() // - pub_count: COUNT of entity.visibility=1 (pub/public/export) // in the module — distinguishes "对外接口层" from // "内部实现层", a signal the call graph cannot give. - // - entry_reachable: MAX(graph_nodes.is_entry_point) — does this - // module contain a main/init/setup/run/handler? - // Rules match by PRIORITY (first hit stops, see role_classifier_plan.md). - // Split the aggregate and the graph_nodes entry_reachable scan into - // two CTEs. A single 6-table LEFT JOIN that combined the three - // relation joins with graph_nodes made SQLite build four COUNT(DISTINCT) + // - entry_reachable: does this module contain a main/init entity? + // Derived from the canonical entity.name/language + // columns (see the entry CTE below). + // Rules match by PRIORITY (first hit stops). + // Split the aggregate and the entry_reachable scan into two CTEs. A + // single 6-table LEFT JOIN that combined the three relation joins + // with the entry scan made SQLite build four COUNT(DISTINCT) // temp B-trees over a blown-up intermediate result (~29s on a 26k-node - // Go tree). Isolating the graph_nodes join into its own CTE — joined + // Go tree). Isolating the entry scan into its own CTE — joined // only on module_id after both aggregates finish — drops the cost to // <0.2s with identical results. INDEXED BY forces the right index for // each relation join; SQLite otherwise picks idx_relation_unique_typed @@ -71,19 +75,52 @@ int64_t StateBuilder::buildModuleSummaries() " WHERE s.kind = 1 AND s.project_id = ? " " GROUP BY s.id, s.name " "), entry AS (" + // Entry-point reachability is derived from the canonical + // entity.name + entity.language columns. The previous version + // read graph_nodes.is_entry_point, but graph_nodes has not been + // written since the canonical-schema migration, so + // entry_reachable was always 0 and the 'entry' role in the CASE + // below could never fire. The name/language predicate mirrors + // graph::isEntryPointName (graph_builder.cpp) so the SQL path and + // the in-memory graph path agree on what an entry point is. " SELECT s.id AS module_id, " - " MAX(COALESCE(gn.is_entry_point, 0)) AS entry_reachable " + " MAX(CASE WHEN " + " (e.name = 'main' AND e.language IN " + " ('c', 'cpp', 'c++', 'go', 'rust')) " + " OR (e.name = 'init' AND e.language = 'go') " + " THEN 1 ELSE 0 END) AS entry_reachable " " FROM scope s " " JOIN entity e ON e.project_id = ? AND e.module_path = s.name " - " LEFT JOIN graph_nodes gn ON gn.project_id = ? " - " AND gn.name = e.name AND gn.file_path = e.file_path " " WHERE s.kind = 1 AND s.project_id = ? " " GROUP BY s.id " + "), intra AS (" + // Internal coupling: call edges whose source AND target entity both + // live in this module. Keyed by module_path — the same column the agg + // CTE joins scope.name on — so it is one grouped scan instead of a + // correlated subquery per module. Self-loops are excluded for the + // same reason the incoming/outgoing counts above exclude them: an + // entity calling itself is not a dependency between two entities. + // The column has always existed (NOT NULL DEFAULT 0) but this INSERT + // passed a literal 0, so every module reported zero internal coupling + // whatever the call graph said — measured on goagent (131 modules, + // 24206 call edges) and on this repository. + " SELECT se.module_path AS module_path, " + " COUNT(*) AS internal " + " FROM relation r2 " + " JOIN entity se ON se.id = r2.source_id " + " JOIN entity te ON te.id = r2.target_id " + " WHERE r2.project_id = ? AND se.project_id = ? " + " AND te.project_id = ? " + " AND se.module_path = te.module_path " + " AND se.module_path != '' " + " AND r2.source_id != r2.target_id " + " GROUP BY se.module_path " ") " "INSERT OR REPLACE INTO module_summary " "(project_id, module_id, state, incoming_count, outgoing_count, " " internal_edges, dead_entities, utilization, confidence, role) " - "SELECT ?, agg.module_id, 0, incoming, outgoing, 0, dead, " + "SELECT ?, agg.module_id, 0, incoming, outgoing, " + " COALESCE(intra.internal, 0), dead, " " CASE WHEN total > 0 " " THEN 1.0 - CAST(dead AS REAL) / total ELSE 0.0 END, " " 0.85, " @@ -137,6 +174,7 @@ int64_t StateBuilder::buildModuleSummaries() // Priority 8: infra — true fallback " ELSE 'infra' END " "FROM agg LEFT JOIN entry ON entry.module_id = agg.module_id " + "LEFT JOIN intra ON intra.module_path = agg.module_name " "WHERE agg.total >= 3"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(store_->handle(), sql.c_str(), -1, &stmt, @@ -147,8 +185,12 @@ int64_t StateBuilder::buildModuleSummaries() sqlite3_errmsg(store_->handle())); return -1; } - // Bind order: agg (4) + entry (3) + SELECT (1) = 8 ? params. - for (int i = 1; i <= 8; i++) + // Bind order: agg (4) + entry (2) + intra (3) + SELECT (1) = 10 params. + // entry previously contributed 3 (one per graph_nodes/scope/entity + // project filter); the deprecated graph_nodes join is gone, so the + // entry CTE now filters project_id twice. intra filters the relation + // plus its source and target entities. + for (int i = 1; i <= 10; i++) sqlite3_bind_int64(stmt, i, static_cast(project_id_)); int rc = sqlite3_step(stmt); @@ -165,97 +207,176 @@ int64_t StateBuilder::buildModuleSummaries() int64_t StateBuilder::buildCapabilityState() { - // Single INSERT...SELECT with a recursive CTE to derive the - // capability name. The CTE scans each matched entity name and finds - // the first position >= 2 where an uppercase letter is followed by - // a lowercase letter — i.e. the start of a new CamelCase word. The - // capability name is the prefix before that position. + // capability_state is the snapshot build_project_state reports as + // project_state.capability (total / verified). It answers the same question + // detect_capability_drift and verify_claim answer, so it asks the same + // implementation: verify::implementingEntitiesFor, the one place the rule + // lives. This builder used to restate the rule in SQL ("model/ cannot + // include verify/") and the two copies drifted — the restated arm floored + // the raw entity name where the verifier floors the normalised one, so one + // capability read 'Implemented' here and 'Contradicted' there — and before + // that the rows came from a naming heuristic (Auth%/Login%/JWT%/…, every + // hit 'Implemented' without consulting the call graph), which is why goagent + // reported 19 capabilities against the 7 its README declares. // - // Acronym handling: a run of uppercase letters like "JWT" in - // "JWTValidator" must be treated as ONE word, not three. The - // previous CTE matched ANY uppercase letter at pos >= 2, so for - // "JWTValidator" it found pos=2 ('W') and returned substr(name,1,1) - // = "J" — truncating the acronym to its first letter. Requiring - // next_ch GLOB '[a-z]' skips the W and T (followed by uppercase), - // finds 'V' at pos=4 (followed by 'a'), and returns "JWT". - // For "AuthValidator" the rule still finds 'V' at pos=5 and - // returns "Auth". - std::string sql = - "INSERT OR IGNORE INTO capability_state " - "(project_id, name, state) " - "WITH RECURSIVE " - "matched(name) AS (" - " SELECT name FROM entity " - " WHERE project_id = ? AND kind = 0" - " AND (name LIKE 'Auth%' OR name LIKE 'Login%'" - " OR name LIKE 'JWT%' OR name LIKE 'Token%'" - " OR name LIKE 'Rate%' OR name LIKE 'Cache%'" - " OR name LIKE 'Log%' OR name LIKE 'Metric%'" - " OR name LIKE 'Health%' OR name LIKE 'Config%')" - " LIMIT 50" - "), " - // scan carries the current char (ch) and the next char - // (next_ch) so the word-boundary filter can require an - // uppercase letter followed by a lowercase letter. substr - // returns '' past end-of-string, and '' GLOB '[a-z]' is 0, - // so a trailing uppercase letter is correctly rejected. - "scan(name, pos, ch, next_ch) AS (" - " SELECT name, 2, substr(name, 2, 1), substr(name, 3, 1) " - " FROM matched " - " WHERE length(name) >= 2 " - " UNION ALL " - " SELECT name, pos + 1, " - " substr(name, pos + 1, 1), substr(name, pos + 2, 1) " - " FROM scan " - " WHERE pos < length(name)" - ") " - "SELECT ?, " - " CASE " - " WHEN fu.pos IS NOT NULL " - " THEN substr(m.name, 1, fu.pos - 1) " - " ELSE m.name END, " - " 'Implemented' " - "FROM matched m " - "LEFT JOIN (" - " SELECT name, MIN(pos) AS pos FROM scan " - // Word boundary: uppercase followed by lowercase. This skips - // intra-acronym uppercase letters (e.g. 'W','T' in "JWT") - // because they are followed by another uppercase letter. - " WHERE ch GLOB '[A-Z]' AND next_ch GLOB '[a-z]' " - " GROUP BY name" - ") fu ON fu.name = m.name"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(store_->handle(), sql.c_str(), -1, &stmt, + // Including verify/ from here is not a layering violation: + // engine/CMakeLists.txt builds ONE static library (astgraph_engine) from all + // of engine/src, and verify/ includes only store/, so there is no cycle. The + // old note that treated this include as impossible is what forced the second + // copy, and a six-arm SQL predicate maintained twice is how the drift got in. + // capability_drift.cpp delegates to the same function for the same reason. + // + // Rows come from the DECLARED capabilities (the `capability` table, filled + // from the project README by the knowledge builder). 'Declared' means the + // evidence chain is incomplete, which is exactly what the verifier reports + // as Contradicted. + // + // Rebuild idempotently: capability_state has no UNIQUE(project_id, name), so + // an INSERT without this DELETE appended a second copy of every row on each + // rebuild (each enhance / build_project_state) and inflated + // project_state.capability.total by the number of runs. + { + const std::string del = + "DELETE FROM capability_state WHERE project_id=" + + std::to_string(project_id_); + if (!store_->exec(del.c_str())) { + fprintf(stderr, + "[module=state_builder, method=buildCapabilityState] " + "delete failed: %s\n", + store_->error().c_str()); + return -1; + } + } + + // The declared capability names. + std::vector declared; + { + const char *names_sql = "SELECT name FROM capability " + "WHERE project_id = ? ORDER BY name"; + sqlite3_stmt *names_st = nullptr; + if (sqlite3_prepare_v2(store_->handle(), names_sql, -1, + &names_st, nullptr) != SQLITE_OK) { + fprintf(stderr, + "[module=state_builder, method=buildCapabilityState] " + "prepare capability names failed: %s\n", + sqlite3_errmsg(store_->handle())); + return -1; + } + sqlite3_bind_int64(names_st, 1, + static_cast(project_id_)); + while (sqlite3_step(names_st) == SQLITE_ROW) { + const char *name = reinterpret_cast( + sqlite3_column_text(names_st, 0)); + if (name) + declared.emplace_back(name); + } + sqlite3_finalize(names_st); + } + + const char *insert_sql = + "INSERT INTO capability_state (project_id, name, state) " + "VALUES (?,?,?)"; + sqlite3_stmt *insert_st = nullptr; + if (sqlite3_prepare_v2(store_->handle(), insert_sql, -1, &insert_st, nullptr) != SQLITE_OK) { fprintf(stderr, "[module=state_builder, method=buildCapabilityState] " - "prepare failed: %s\n", + "prepare insert failed: %s\n", sqlite3_errmsg(store_->handle())); return -1; } - sqlite3_bind_int64(stmt, 1, static_cast(project_id_)); - sqlite3_bind_int64(stmt, 2, static_cast(project_id_)); - int rc = sqlite3_step(stmt); - sqlite3_finalize(stmt); - if (rc != SQLITE_DONE) { - fprintf(stderr, - "[module=state_builder, method=buildCapabilityState] " - "step failed (rc=%d): %s\n", - rc, sqlite3_errmsg(store_->handle())); - return -1; + int64_t written = 0; + for (const std::string &name : declared) { + bool ok = false; + const std::vector implementing = + verify::implementingEntitiesFor(store_, project_id_, + name, ok); + if (!ok) { + // The shared rule's query failed. Writing 'Declared' here would turn + // an infrastructure failure into a product verdict, which is the + // silent error its contract forbids (code_rules §1). + fprintf(stderr, + "[module=state_builder, method=buildCapabilityState] " + "implementingEntitiesFor failed for \"%s\"\n", + name.c_str()); + sqlite3_finalize(insert_st); + return -1; + } + sqlite3_bind_int64(insert_st, 1, + static_cast(project_id_)); + sqlite3_bind_text(insert_st, 2, name.c_str(), -1, + SQLITE_TRANSIENT); + sqlite3_bind_text(insert_st, 3, + implementing.empty() ? "Declared" : + "Implemented", + -1, SQLITE_STATIC); + const int rc = sqlite3_step(insert_st); + if (rc != SQLITE_DONE) { + fprintf(stderr, + "[module=state_builder, method=buildCapabilityState] " + "insert failed for \"%s\" (rc=%d): %s\n", + name.c_str(), rc, + sqlite3_errmsg(store_->handle())); + sqlite3_finalize(insert_st); + return -1; + } + sqlite3_reset(insert_st); + ++written; } - return sqlite3_changes(store_->handle()); + sqlite3_finalize(insert_st); + return written; } int64_t StateBuilder::buildWorkflowState() { - // Single INSERT...SELECT replaces the per-row prepare/step/finalize - // loop. + // Clear previous rows so a rebuild is idempotent (rename-safe). + // INSERT OR IGNORE without a DELETE left stale workflow names behind + // after a rename and double-counted on every enhance. + if (!store_->exec( + (std::string( + "DELETE FROM workflow_state WHERE project_id=") + + std::to_string(project_id_)) + .c_str())) { + fprintf(stderr, + "[module=state_builder, method=buildWorkflowState] " + "delete failed: %s\n", + store_->error().c_str()); + return -1; + } + // Derive step counts from the call graph instead of hardcoding + // (5, 2), which fabricated a 0.4 progress score for every workflow. + // steps_total = outgoing Calls from the workflow entry + // steps_done = those callees that themselves have a caller + // (wired into the graph, not just declared) std::string sql = - "INSERT OR IGNORE INTO workflow_state " + "INSERT INTO workflow_state " "(project_id, name, state, steps_total, steps_done) " - "SELECT DISTINCT ?, e.name, 'Partial', 5, 2 " + "SELECT DISTINCT ?, e.name, " + " CASE WHEN (SELECT COUNT(*) FROM relation rt " + " WHERE rt.project_id = e.project_id AND rt.source_id = e.id " + " AND rt.type = 1) = 0 THEN 'Empty' " + " WHEN (SELECT COUNT(DISTINCT rt.target_id) FROM relation rt " + " WHERE rt.project_id = e.project_id AND rt.source_id = e.id " + " AND rt.type = 1 AND EXISTS (" + " SELECT 1 FROM relation rin " + " WHERE rin.project_id = e.project_id " + " AND rin.target_id = rt.target_id AND rin.type = 1" + " AND rin.source_id != e.id)) " + " >= (SELECT COUNT(*) FROM relation rt " + " WHERE rt.project_id = e.project_id AND rt.source_id = e.id " + " AND rt.type = 1) THEN 'Done' " + " ELSE 'Partial' END, " + " (SELECT COUNT(*) FROM relation rt " + " WHERE rt.project_id = e.project_id AND rt.source_id = e.id " + " AND rt.type = 1), " + " (SELECT COUNT(DISTINCT rt.target_id) FROM relation rt " + " WHERE rt.project_id = e.project_id AND rt.source_id = e.id " + " AND rt.type = 1 AND EXISTS (" + " SELECT 1 FROM relation rin " + " WHERE rin.project_id = e.project_id " + " AND rin.target_id = rt.target_id AND rin.type = 1" + " AND rin.source_id != e.id)) " "FROM entity e " "JOIN relation r ON r.project_id = ? AND r.target_id = e.id " "JOIN entity caller ON r.source_id = caller.id " @@ -292,39 +413,54 @@ int64_t StateBuilder::buildWorkflowState() int64_t StateBuilder::buildArchitectureState() { - // Count architecture violations per layer pair directly from - // architecture_edge, without re-joining entity/relation tables. + // Record cross-module dependency counts. This is a COUNT of call edges + // that cross a module boundary — NOT a violation count, and NOT a layer + // check. // - // The original query did a 4-table JOIN (architecture_edge × entity × - // relation × entity) with non-sargable `file_path LIKE '%layer%'` - // filters, costing ~25s for 110k architecture_edge rows. Even with - // sargable `module_path LIKE 'layer%'` and indexes on - // relation(project_id, target_id), the JOIN cardinality (110k edges × - // N relations per target) made it prohibitively slow. + // ArchitecturePlugin (model/plugins/architecture.cpp) writes one + // architecture_edge row per (caller module, callee module) pair with no + // layer model and no direction test: callee_module / caller_module hold MODULE + // NAMES, not layer names. Counting those rows as `violations` with + // compliance = 0.0 therefore reported every normal cross-module dependency + // as an architecture violation, and pushed the architecture score down in + // proportion to how interconnected the project is. // - // This simplification is SAFE because architecture_edge rows are - // already validated cross-module calls: ArchitecturePlugin (see - // model/plugins/architecture.cpp) creates each row ONLY when a real - // call edge crosses from layer_lower to layer_upper, using the same - // pathStartsWithCI membership test that the LIKE filters re-checked. - // The relation JOIN was therefore redundant validation. + // The count is real information, so it is kept — under its own column. + // `violations` stays 0 and compliance stays 1.0 because nothing here can + // tell a violation from a dependency. `layer` keeps the "a->b" module-pair + // key (the column predates this distinction). // - // Accuracy: the layer pairs, compliance flags (0.0 when violations > - // 0), ORDER BY, and LIMIT are IDENTICAL to the original. The - // violation COUNT differs in magnitude (counts architecture_edge rows - // instead of architecture_edge × relation rows) but preserves - // relative ordering — more cross-module calls per layer pair - // produces a proportionally higher count. - std::string sql = "INSERT OR IGNORE INTO architecture_state " - "(project_id, layer, violations, compliance) " - "SELECT ?, ae.layer_lower || '->' || ae.layer_upper, " - " COUNT(*), " - " CASE WHEN COUNT(*) > 0 THEN 0.0 ELSE 1.0 END " - "FROM architecture_edge ae " - "WHERE ae.project_id = ? " - "GROUP BY ae.layer_lower, ae.layer_upper " - "HAVING COUNT(*) > 0 " - "ORDER BY COUNT(*) DESC LIMIT 10"; + // The original query did a 4-table JOIN (architecture_edge × entity × + // relation × entity) with non-sargable `file_path LIKE '%layer%'` + // filters, costing ~25s for 110k architecture_edge rows. Reading the + // counts straight from architecture_edge avoids that JOIN: the rows are + // already cross-module call edges. + { + // Rebuild idempotently. INSERT OR IGNORE cannot dedupe here — the + // table has no UNIQUE(project_id, layer) — so re-running the builder + // would accumulate a second copy of every row and double every count. + const std::string del = + "DELETE FROM architecture_state WHERE project_id=" + + std::to_string(project_id_); + if (!store_->exec(del.c_str())) { + fprintf(stderr, + "[module=state_builder, method=buildArchitectureState] " + "delete failed: %s\n", + store_->error().c_str()); + return -1; + } + } + std::string sql = + "INSERT INTO architecture_state " + "(project_id, layer, violations, cross_module_edges, " + " compliance) " + "SELECT ?, ae.caller_module || '->' || ae.callee_module, " + " 0, COUNT(*), 1.0 " + "FROM architecture_edge ae " + "WHERE ae.project_id = ? " + "GROUP BY ae.callee_module, ae.caller_module " + "HAVING COUNT(*) > 0 " + "ORDER BY COUNT(*) DESC LIMIT 10"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(store_->handle(), sql.c_str(), -1, &stmt, nullptr) != SQLITE_OK) { diff --git a/engine/src/model/state_builder.h b/engine/src/model/state_builder.h index 110ff4d..d1fc6d3 100644 --- a/engine/src/model/state_builder.h +++ b/engine/src/model/state_builder.h @@ -12,8 +12,7 @@ namespace model // ── Role classifier thresholds (v0.2.2) ─────────────────────────────── // Multi-signal fusion classifier in buildModuleSummaries() uses these // constexpr thresholds so they can be retuned in one place against `bun` -// without touching the SQL string. See docs/dev_plans/role_classifier_plan.md -// for the tuning protocol. Values chosen as v0.2.2 initial estimates. +// without touching the SQL string. Values chosen as v0.2.2 initial estimates. // // role priority order (first hit stops): test -> api -> entry -> core -> utility -> dead -> infra // diff --git a/engine/src/parser/parser.cpp b/engine/src/parser/parser.cpp index 10d8801..4a31672 100644 --- a/engine/src/parser/parser.cpp +++ b/engine/src/parser/parser.cpp @@ -24,11 +24,19 @@ static const TSLanguage *resolveGrammar(const char *name) return tree_sitter_python(); if (strcmp(name, "rust") == 0 || strcmp(name, "rs") == 0) return tree_sitter_rust(); - // Swift grammar temporarily disabled: parser.c incompatible - // with tree-sitter core v0.24.7. Re-enable when upstream - // releases an ABI-compatible version. - // if (strcmp(name, "swift") == 0) - // return tree_sitter_swift(); + // Swift is NOT resolvable here — this is the single authoritative note + // for that state. The grammar is not vendored (it is absent from + // GRAMMAR_SOURCES in CMakeLists.txt) because its parser.c is + // incompatible with tree-sitter core v0.24.7; the IR visitor/translator + // and the Swift builtin table were therefore removed as well rather than + // left unreachable. Consequence, and it is deliberate: a .swift file is + // still detected by extension (filter_policy_detect) and reported as a + // `language_missing` parse failure, which is retried on every run and + // never counts towards the fail-fast skip. The other detected-but- + // ungrammared languages (Kotlin, Ruby, Scala) behave identically. + // Re-enable by restoring the grammar, the visitor/translator pair and the + // builtin table together. + if (strcmp(name, "typescript") == 0 || strcmp(name, "ts") == 0) return tree_sitter_typescript(); if (strcmp(name, "tsx") == 0) diff --git a/engine/src/engine_index_post_parse.cpp b/engine/src/post_parse_phase.cpp similarity index 81% rename from engine/src/engine_index_post_parse.cpp rename to engine/src/post_parse_phase.cpp index 0b9d36d..7bb0758 100644 --- a/engine/src/engine_index_post_parse.cpp +++ b/engine/src/post_parse_phase.cpp @@ -1,5 +1,10 @@ // Shared post-parse sequence for engine_index_project. // +// Internal helper, not an FFI boundary: it takes C++ references and returns +// a dupString()'d JSON payload the caller frees. It deliberately does NOT +// carry an `engine_`/`extern "C"` name, which an FFI audit would otherwise +// have to dismiss one call site at a time. +// // Both the streaming path (BoundedQueue + single writer) and the in-memory // bulk path (store::MemBulkAggregator) produce semantic_records identically. // After the FileResult data is persisted, this single function runs the @@ -21,12 +26,12 @@ using namespace std::chrono; -char *engine_index_post_parse(uint64_t project_id, const std::string &dir, - const std::vector &job_paths, - const FilterPolicy &filter, bool is_reindex, - bool mode_fast, bool mode_deep, - int64_t time_parse_ms, int64_t time_buildgraph_ms, - int total_indexed) +char *postParsePhase(EngineContext *ctx, uint64_t project_id, + const std::string &dir, + const std::vector &job_paths, + const FilterPolicy &filter, bool is_reindex, + bool mode_fast, bool mode_deep, int64_t time_parse_ms, + int64_t time_buildgraph_ms, int total_indexed) { // Total wall-clock for the entire post-parse phase (graph build + // metrics + indexes). Captured here so the breakdown in @@ -49,8 +54,10 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, store::setIndexProgress(p); } - // ── Step 1: buildGraph (SQL-only, graph_nodes + graph_edges + CSR) ── - // Reads semantic_records, creates graph_nodes/graph_edges via SQL JOINs. + // ── Step 1: buildGraph (SQL-only, entity + relation + CSR) ── + // Reads semantic_records, creates the canonical entity/relation rows via + // SQL JOINs (graph_nodes/graph_edges are the LEGACY batch tables and stay + // empty here — see verifier.h on which layer is the source of truth). // NOTE: calls=true builds call edges via all priorities in // buildCallEdgesSQL (store_intern.cpp): P1 (intra-file ref_original_id), // P2 (translator-resolved), P3 (name-based cross-file, language-filtered @@ -59,7 +66,7 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, int64_t time_fts_ms = 0, time_vector_ms = 0; { auto t_bg = steady_clock::now(); - g_store->beginTransaction(); + ctx->store->beginTransaction(); // Incremental re-index: pass changed files so buildGraph uses the // optimized path (only cycles the unique edge index, not the 5 // lookup indexes). First index (is_reindex=false) passes nullptr @@ -93,18 +100,18 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, // propagate a JSON error so the caller (and the user) knows the // index did not fully succeed. (CSR failures are non-fatal inside // buildGraph and still return true.) - if (!g_store->buildGraph(project_id, !defer_csr, - is_reindex ? &changed_files : - nullptr)) { - g_store->rollbackTransaction(); + if (!ctx || !ctx->store->buildGraph( + project_id, !defer_csr, + is_reindex ? &changed_files : nullptr)) { + ctx->store->rollbackTransaction(); return dupString( "{\"ok\":false,\"error\":\"buildGraph failed " "for project " + std::to_string(project_id) + " (resolver stage)\",\"module\":\"engine\"," - "\"method\":\"engine_index_post_parse\"}"); + "\"method\":\"postParsePhase\"}"); } - g_store->commitTransaction(); + ctx->store->commitTransaction(); // Indexing now builds the full call graph (buildGraph above), // so mark every node callgraph_ready. This makes trace_path and // the enhancement-status report reflect that the call graph is @@ -123,12 +130,12 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, // silently misreport readiness. Log on failure so the silent // misreport is at least observable, mirroring the // [module=..., method=...] fprintf pattern used elsewhere. - if (!g_store->exec(up.c_str())) { + if (!ctx || !ctx->store->exec(up.c_str())) { fprintf(stderr, "engine_index_project: callgraph_ready UPDATE " "failed: %s " "[module=engine, method=engine_index_project]\n", - g_store->error().c_str()); + ctx->store->error().c_str()); } } // P0.1: entity/relation dual-write now happens INSIDE buildGraph @@ -153,7 +160,7 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, // count (cyclomatic > 0), so readiness never over-claims. { auto t_metrics = steady_clock::now(); - g_store->resolveStagedMetrics(project_id); + ctx->store->resolveStagedMetrics(project_id); int metrics_rows = 0; { sqlite3_stmt *mstmt = nullptr; @@ -161,7 +168,7 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, "SELECT COUNT(*) FROM entity " "WHERE project_id = ? AND kind IN (0,1) " "AND cyclomatic > 0"; - if (sqlite3_prepare_v2(g_store->handle(), msql, -1, + if (sqlite3_prepare_v2(ctx->store->handle(), msql, -1, &mstmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64( mstmt, 1, @@ -172,14 +179,14 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, sqlite3_finalize(mstmt); } else { fprintf(stderr, - "engine_index_post_parse: metrics count " + "postParsePhase: metrics count " "probe failed: %s [module=engine, " - "method=engine_index_post_parse]\n", - sqlite3_errmsg(g_store->handle())); + "method=postParsePhase]\n", + sqlite3_errmsg(ctx->store->handle())); } } - g_store->setProjectReadiness(project_id, "metrics_ready", - metrics_rows > 0 ? 1 : 0); + ctx->store->setProjectReadiness(project_id, "metrics_ready", + metrics_rows > 0 ? 1 : 0); fprintf(stderr, "engine: resolveStagedMetrics=%lldms " "(metrics_ready=%d) " @@ -199,7 +206,7 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, // rows for this project, so readiness matches canonical data. if (mode_deep) { auto t_v = steady_clock::now(); - g_store->buildVectorsFromGraph(project_id); + ctx->store->buildVectorsFromGraph(project_id); // Probe canonical data: count node_vectors rows for this project. // If the builder wrote nothing (e.g. no function/method entities), // the count stays 0 and the flag stays 0 — exactly what we want, @@ -210,7 +217,7 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, sqlite3_stmt *vstmt = nullptr; const char *vsql = "SELECT COUNT(*) FROM node_vectors WHERE project_id = ?"; - if (sqlite3_prepare_v2(g_store->handle(), vsql, -1, + if (sqlite3_prepare_v2(ctx->store->handle(), vsql, -1, &vstmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64( vstmt, 1, @@ -222,14 +229,14 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, } else { // node_vectors table missing — log and keep flag 0. fprintf(stderr, - "engine_index_post_parse: node_vectors " + "postParsePhase: node_vectors " "count probe failed: %s " - "[module=engine, method=engine_index_post_parse]\n", - sqlite3_errmsg(g_store->handle())); + "[module=engine, method=postParsePhase]\n", + sqlite3_errmsg(ctx->store->handle())); } } - g_store->setProjectReadiness(project_id, "vector_ready", - vec_rows > 0 ? 1 : 0); + ctx->store->setProjectReadiness(project_id, "vector_ready", + vec_rows > 0 ? 1 : 0); time_vector_ms = duration_cast(steady_clock::now() - t_v) .count(); @@ -242,9 +249,9 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, // index recreation on incremental runs (they were never dropped); always // run dedup DELETE + unique edge index creation. { - store::GraphStore::BulkPragmaGuard guard(g_store.get()); + store::GraphStore::BulkPragmaGuard guard(ctx->store.get()); auto t_idx = steady_clock::now(); - g_store->createIndexesAfterBulkLoad(project_id, !is_reindex); + ctx->store->createIndexesAfterBulkLoad(project_id, !is_reindex); fprintf(stderr, "engine: createIndexesAfterBulkLoad=%lldms " "[module=engine, method=engine_index_project]\n", @@ -256,7 +263,7 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, // Set readiness flag — core graph is queryable now. // fts_ready / knowledge_ready are set by the async path // (runModelIndexSync + buildKnowledgeGraphSync). - g_store->setProjectReadiness(project_id, "normal_ready", 1); + ctx->store->setProjectReadiness(project_id, "normal_ready", 1); // P0.2 / P3: Model Engine + State Builder + FTS moved to async path. // Previously ran synchronously here AND inside buildGraph (double @@ -269,9 +276,9 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, // is returned immediately while the knowledge graph materialises // concurrently. Callers can poll isAsyncKnowledgeBuilderRunning() // or check the "knowledge_ready" readiness flag. - // NOTE: This is launched AFTER all g_store reads below, to avoid + // NOTE: This is launched AFTER all engine store reads below, to avoid // concurrent GraphStore access. The builder writes to module_edge - // while the main thread reads graph_nodes/graph_edges counts. + // while the main thread reads the canonical counts below. // ── Result JSON ────────────────────────────────────────────── std::ostringstream result; @@ -293,7 +300,7 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, std::string sql; sql = "SELECT COUNT(*) FROM entity WHERE project_id = " + std::to_string(project_id); - if (sqlite3_prepare_v2(g_store->handle(), sql.c_str(), -1, + if (sqlite3_prepare_v2(ctx->store->handle(), sql.c_str(), -1, &stmt, nullptr) == SQLITE_OK) { if (sqlite3_step(stmt) == SQLITE_ROW) result << ",\"total_nodes\":" @@ -302,17 +309,20 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, } sql = "SELECT COUNT(*) FROM relation WHERE project_id = " + std::to_string(project_id); - if (sqlite3_prepare_v2(g_store->handle(), sql.c_str(), -1, + if (sqlite3_prepare_v2(ctx->store->handle(), sql.c_str(), -1, &stmt, nullptr) == SQLITE_OK) { if (sqlite3_step(stmt) == SQLITE_ROW) result << ",\"total_edges\":" << sqlite3_column_int64(stmt, 0); sqlite3_finalize(stmt); } - // Also report symbols and call_edges counts - sql = "SELECT COUNT(*) FROM graph_nodes WHERE project_id = " + + // Also report the symbol count. It used to count `graph_nodes`, + // which the canonical pipeline leaves empty (only the legacy + // engine_index_batch path writes it), so the field was always 0 + // next to a correct total_edges from `relation`. + sql = "SELECT COUNT(*) FROM entity WHERE project_id = " + std::to_string(project_id); - if (sqlite3_prepare_v2(g_store->handle(), sql.c_str(), -1, + if (sqlite3_prepare_v2(ctx->store->handle(), sql.c_str(), -1, &stmt, nullptr) == SQLITE_OK) { if (sqlite3_step(stmt) == SQLITE_ROW) result << ",\"total_symbols\":" @@ -321,7 +331,7 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, } sql = "SELECT COUNT(*) FROM graph_edges WHERE project_id = " + std::to_string(project_id) + " AND edge_type = 1"; - if (sqlite3_prepare_v2(g_store->handle(), sql.c_str(), -1, + if (sqlite3_prepare_v2(ctx->store->handle(), sql.c_str(), -1, &stmt, nullptr) == SQLITE_OK) { if (sqlite3_step(stmt) == SQLITE_ROW) result << ",\"total_call_edges\":" @@ -349,7 +359,7 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, store::setIndexProgress(p); } - // Launch the async knowledge builder AFTER all g_store reads/writes + // Launch the async knowledge builder AFTER all engine store reads/writes // above are complete, to avoid concurrent GraphStore access. // run_fts=!mode_fast: fast mode skips FTS entirely; normal/deep // modes build FTS in the background thread. @@ -358,14 +368,13 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, // (server/src/scheduler/) skip the ~280ms state-builder work in // per-module workers and run it ONCE on the merged DB at the end. // This keeps the per-module worker fast so the overall wall-clock - // index time stays under 1s on small projects (see - // builtin-scheduler-design.md §4.5). + // index time stays under 1s on small projects. const char *skip_async = getenv("CODESCOPE_SKIP_ASYNC"); if (!skip_async || skip_async[0] == '0') { - launchAsyncKnowledgeBuilder(project_id, !mode_fast); + launchAsyncKnowledgeBuilder(ctx, project_id, !mode_fast); } else { fprintf(stderr, - "[module=engine, method=engine_index_post_parse] " + "[module=engine, method=postParsePhase] " "skipping async model/state/fts build " "(CODESCOPE_SKIP_ASYNC=1) for project %llu\n", (unsigned long long)project_id); @@ -377,7 +386,7 @@ char *engine_index_post_parse(uint64_t project_id, const std::string &dir, auto t_post_parse_end = steady_clock::now(); fprintf(stderr, "engine: post_parse_total=%lldms " - "[module=engine, method=engine_index_post_parse]\n", + "[module=engine, method=postParsePhase]\n", (long long)duration_cast(t_post_parse_end - t_post_parse_start) .count()); diff --git a/engine/src/query/graph_query.cpp b/engine/src/query/graph_query.cpp index a99ea3b..c14b381 100644 --- a/engine/src/query/graph_query.cpp +++ b/engine/src/query/graph_query.cpp @@ -1,7 +1,9 @@ +#include "util/json_writer.h" #include "graph_query.h" #include #include +#include #include #include #include @@ -12,6 +14,17 @@ namespace query { +// ─── Multi-hop query bounds ───────────────────────────────────── +// The hop range comes from a client-supplied DSL string, so it must not be +// able to walk the graph unboundedly. The multi-hop BFS keeps no visited set: +// a node reachable through several paths is re-expanded once per path, so a +// range such as `[Calls*1..1000000]` makes both time and memory grow +// exponentially on a cyclic graph. Truncation is reported in the response +// rather than hidden. +static constexpr int kMultiHopMaxDepth = 8; +static constexpr int kMultiHopMaxRows = 10000; +static constexpr int64_t kMultiHopExpansionBudget = 200000; + // ─── Node/Edge type name → integer mapping ───────────────────── static const std::unordered_map &typeMap() @@ -44,198 +57,27 @@ static void parseNodeSpec(const std::string &spec, std::string &out_type, // ─── SQLite helpers (Cypher escaping + tuple accessors) ────── -// Escape a string for safe inclusion inside a Cypher single-quoted literal. -// Prevents injection / query breakage from symbol names with quotes or -// backslashes. Mirrors the cypherEscape in query_engine.cpp (both are -// static, so TU-local — no ODR clash). -static std::string cypherEscape(const char *s) -{ - if (!s) - return ""; - std::string out; - out.reserve(std::strlen(s) + 8); - for (const char *p = s; *p; ++p) { - if (*p == '\\' || *p == '\'') { - out += '\\'; - } - out += *p; - } - return out; -} - -// Escape a string for safe inclusion inside a JSON double-quoted string. -// Mirrors query::jsonEscape in query_engine.cpp; defined locally so -// graph_query.cpp does not need to pull in the QueryEngine header. -static std::string jsonEscape(const char *s) +/// Build the failure body every graph_query error path returns: +/// `{"total":0,"results":[],"error":""}`. Centralised because the +/// message carries user input (the LIMIT operand, the trailing text after the +/// target node), which the writer escapes instead of it being concatenated +/// into a JSON literal. The `[module=…, method=…]` suffix keeps the trace +/// chain required by plan/rules/code_rules.md. +static std::string graphQueryError(const std::string &message) { - if (!s) - return ""; - std::string out; - out.reserve(std::strlen(s) + 8); - for (const char *p = s; *p; ++p) { - switch (*p) { - case '"': - out += "\\\""; - break; - case '\\': - out += "\\\\"; - break; - case '\n': - out += "\\n"; - break; - case '\r': - out += "\\r"; - break; - case '\t': - out += "\\t"; - break; - default: - out += *p; - break; - } - } - return out; + util::JsonWriter w; + w.beginObject(); + w.key("total").value(0); + w.key("results").beginArray().endArray(); + w.key("error").value(message); + w.endObject(); + return w.str(); } // Map an integer edge_type from the DSL to a Cypher rel-type label. // SQLite stores CALLS (edge_type=1) and RELATES (other edge types) // as relationship labels; CALLS|RELATES matches both in a single // pattern. When edge_type is -1 (unspecified) we match both. -static std::string edgeRelLabel(int edge_type) -{ - // CALLS|RELATES covers all relationship labels currently stored in - // SQLite. The caller may still apply a WHERE r.edge_type = N - // filter to narrow the result set when edge_type is specified. - (void)edge_type; - return "CALLS|RELATES"; -} - -// ─── Build a single-hop Cypher query ─────────────────────────── -// -// Pattern: MATCH (src:GraphNode)-[r:CALLS|RELATES]->(tgt:GraphNode) -// WHERE src.project_id = N AND tgt.project_id = N -// [AND src.name = '...'] [AND tgt.name = '...'] -// [AND src.node_type IN [..] / = N] -// [AND tgt.node_type IN [..] / = N] -// [AND r.edge_type = N] -// RETURN src.graph_node_id, src.name, src.node_type, -// src.file_path, ID(r), r.edge_type, -// tgt.graph_node_id, tgt.name, tgt.node_type, -// tgt.file_path -// LIMIT 10000 -// -// The Cypher is built with project_id spliced inline (a safe integer) -// and name filters spliced via cypherEscape'd single-quoted literals. -// node_type=0 (Function) is treated as IN (0,1) to mirror the legacy -// SQL behaviour where "Function" matched both functions and methods. - -static std::string buildSingleHopCypher(uint64_t project_id, int edge_type, - int src_type_val, int tgt_type_val, - const std::string &src_name, - const std::string &tgt_name) -{ - std::ostringstream c; - c << "MATCH (src:GraphNode)-[r:" << edgeRelLabel(edge_type) - << "]->(tgt:GraphNode) " - << "WHERE src.project_id = " << project_id - << " AND tgt.project_id = " << project_id; - if (edge_type >= 0) - c << " AND r.edge_type = " << edge_type; - if (src_type_val >= 0) { - if (src_type_val == 0) - c << " AND src.node_type IN [0,1]"; - else - c << " AND src.node_type = " << src_type_val; - } - if (tgt_type_val >= 0) { - if (tgt_type_val == 0) - c << " AND tgt.node_type IN [0,1]"; - else - c << " AND tgt.node_type = " << tgt_type_val; - } - if (!src_name.empty()) - c << " AND src.name = '" << cypherEscape(src_name.c_str()) - << "'"; - if (!tgt_name.empty()) - c << " AND tgt.name = '" << cypherEscape(tgt_name.c_str()) - << "'"; - c << " RETURN src.graph_node_id, src.name, src.node_type, " - << "src.file_path, ID(r), r.edge_type, " - << "tgt.graph_node_id, tgt.name, tgt.node_type, tgt.file_path " - << "LIMIT 10000"; - return c.str(); -} - -// ─── Build a multi-hop Cypher query ───────────────────────────── -// -// Pattern: MATCH p = (src:GraphNode)-[:CALLS|RELATES*min..max]->(tgt:GraphNode) -// WHERE src.project_id = N AND tgt.project_id = N -// [AND src.name = '...'] [AND tgt.name = '...'] -// [AND src.node_type IN [..] / = N] -// [AND tgt.node_type IN [..] / = N] -// RETURN src.graph_node_id, src.name, src.node_type, -// src.file_path, -// tgt.graph_node_id, tgt.name, tgt.node_type, -// tgt.file_path, -// length(p), -// [n IN nodes(p) | n.graph_node_id] -// LIMIT 10000 -// -// `length(p)` gives the hop count (1 for a single-edge path). -// `nodes(p)` returns the list of nodes along the path; the list -// comprehension projects each node's graph_node_id, which we join -// into the legacy "1->2->3" chain string in C++. - -static std::string buildMultiHopCypher(uint64_t project_id, int edge_type, - int min_depth, int max_depth, - int src_type_val, int tgt_type_val, - const std::string &src_name, - const std::string &tgt_name) -{ - // Variable-length relationship with optional edge_type filter. - // Cypher syntax: -[:CALLS|RELATES*min..max]-> when edge_type is - // unspecified, otherwise add a WHERE r.edge_type = N filter (the - // edge_type property is preserved on every relationship in the - // path). - std::ostringstream c; - c << "MATCH p = (src:GraphNode)-[:" << edgeRelLabel(edge_type) << "*" - << min_depth << ".." << max_depth << "]->(tgt:GraphNode) " - << "WHERE src.project_id = " << project_id - << " AND tgt.project_id = " << project_id; - if (edge_type >= 0) { - // r in a variable-length pattern refers to the list of - // relationships; filter via ALL(r IN relationships(p) WHERE - // r.edge_type = N) so every hop matches the requested type. - c << " AND ALL(r IN relationships(p) WHERE r.edge_type = " - << edge_type << ")"; - } - if (src_type_val >= 0) { - if (src_type_val == 0) - c << " AND src.node_type IN [0,1]"; - else - c << " AND src.node_type = " << src_type_val; - } - if (tgt_type_val >= 0) { - if (tgt_type_val == 0) - c << " AND tgt.node_type IN [0,1]"; - else - c << " AND tgt.node_type = " << tgt_type_val; - } - if (!src_name.empty()) - c << " AND src.name = '" << cypherEscape(src_name.c_str()) - << "'"; - if (!tgt_name.empty()) - c << " AND tgt.name = '" << cypherEscape(tgt_name.c_str()) - << "'"; - c << " RETURN src.graph_node_id, src.name, src.node_type, " - << "src.file_path, " - << "tgt.graph_node_id, tgt.name, tgt.node_type, tgt.file_path, " - << "length(p), [n IN nodes(p) | n.graph_node_id] LIMIT 10000"; - return c.str(); -} - -// ─── Execute DSL query ───────────────────────────────────────── - std::string executeGraphQuery(uint64_t project_id, const char *dsl_query, store::GraphStore *store) { @@ -325,6 +167,10 @@ std::string executeGraphQuery(uint64_t project_id, const char *dsl_query, min_depth = 1; if (max_depth < min_depth) max_depth = min_depth; + // The DSL is client-supplied: clamp the hop count so a query + // cannot ask for an effectively unbounded walk. + if (max_depth > kMultiHopMaxDepth) + max_depth = kMultiHopMaxDepth; } } else { edge_spec = edge_raw; @@ -359,6 +205,79 @@ std::string executeGraphQuery(uint64_t project_id, const char *dsl_query, parseNodeSpec(src_spec, src_type, src_name); parseNodeSpec(tgt_spec, tgt_type, tgt_name); + // ── Trailing clauses: LIMIT and RETURN ───────── + // Two clauses are accepted after the target node, in either order: + // + // * `LIMIT ` — caps the result set. It is the documented way to bound + // a large query (the README's graph-query benchmarks instruct it) and + // the only one available here, since the MCP transport replaces a tool + // response over its message cap with an error. It used to be ignored + // silently: `LIMIT 10` returned every match. + // * `RETURN ` — accepted for compatibility with the DSL's + // documented surface and currently has no effect, because every match + // is returned as source/edge/target regardless (the response shape the + // corpus and the tests rely on). Its operand runs to the end of the + // query, so write LIMIT before RETURN. + // + // Anything else is an error rather than silently dropped input — the rule + // engine_verify_claim applies to an unknown claim type. + q = q.substr(close_paren + 1); + trim(q); + int user_limit = 0; + while (!q.empty()) { + const size_t sp = q.find_first_of(" \t"); + const std::string tok = + (sp == std::string::npos) ? q : q.substr(0, sp); + std::string upper = tok; + for (auto &c : upper) + c = static_cast( + std::toupper(static_cast(c))); + if (upper == "LIMIT") { + std::string rest = (sp == std::string::npos) ? + std::string() : + q.substr(sp + 1); + trim(rest); + const size_t sp2 = rest.find_first_of(" \t"); + const std::string num = (sp2 == std::string::npos) ? + rest : + rest.substr(0, sp2); + if (num.empty() || + num.find_first_not_of("0123456789") != + std::string::npos) + return graphQueryError( + std::string("LIMIT requires a positive " + "integer, got '") + + num + + "' [module=engine, " + "method=executeGraphQuery]"); + const int want = std::atoi(num.c_str()); + if (want < 1) + return "{\"total\":0,\"results\":[],\"error\":\"LIMIT " + "must be >= 1 [module=engine, " + "method=executeGraphQuery]\"}"; + if (user_limit != 0) + return "{\"total\":0,\"results\":[],\"error\":\"LIMIT " + "given more than once [module=engine, " + "method=executeGraphQuery]\"}"; + user_limit = want; + q = (sp2 == std::string::npos) ? std::string() : + rest.substr(sp2 + 1); + trim(q); + continue; + } + if (upper == "RETURN") { + // Operand runs to the end of the query (see above). + q.clear(); + continue; + } + return graphQueryError( + std::string("unexpected trailing text after the target " + "node: '") + + q + + "' — only 'LIMIT ' and 'RETURN ' are " + "accepted [module=engine, method=executeGraphQuery]"); + } + // Resolve edge type to integer int edge_type = -1; if (!edge_spec.empty()) { @@ -389,21 +308,12 @@ std::string executeGraphQuery(uint64_t project_id, const char *dsl_query, tgt_type_val = it->second; } - // Build Cypher - std::string cypher; - if (multi_hop) { - if (edge_type < 0) { - return "{\"total\":0,\"results\":[],\"error\":\"multi-hop queries " - "require an edge type\"}"; - } - cypher = buildMultiHopCypher(project_id, edge_type, min_depth, - max_depth, src_type_val, - tgt_type_val, src_name, tgt_name); - } else { - cypher = buildSingleHopCypher(project_id, edge_type, - src_type_val, tgt_type_val, - src_name, tgt_name); - } + // The DSL was parsed above and is executed against SQLite below. + // A Cypher string used to be assembled here for a backend that no longer + // exists: `cypher` was written and then never read by any code path, and + // the builders that produced it carried comments about injection + // protection for a string that was never executed. Multi-hop is handled by + // the SQL below (variable-length hop expansion), not by a query string. // Execute via SQLite. Query errors are tagged with // [module=graph_query, method=executeGraphQuery] so callers can @@ -480,10 +390,88 @@ std::string executeGraphQuery(uint64_t project_id, const char *dsl_query, resolveEntities(src_name, src_type_val, src_ids); resolveEntities(tgt_name, tgt_type_val, tgt_ids); + // A node spec that resolved to no entity matches nothing. Omitting + // the `IN (…)` filter for an empty id list (the previous behaviour) + // silently dropped that side of the pattern, so + // MATCH (A:x)-[Calls]->(A:y) with an unknown `y` returned every edge + // out of `x` instead of none. + if (src_ids.empty() || tgt_ids.empty()) { + util::JsonWriter empty; + empty.beginObject(); + empty.key("results").beginArray().endArray(); + empty.key("total").value(0); + // Still worth explaining when a typed name matched no entity: + // the probe below reports same-name kinds the filter missed. + std::string hints; + auto probeOtherKinds = [&](const std::string &name, + int wanted_type, + const char *type_label) { + if (name.empty() || wanted_type < 0) + return; + std::string sql = "SELECT DISTINCT kind FROM entity " + "WHERE project_id=? AND (name=? OR " + "qualified_name=?)"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql.c_str(), -1, &st, + nullptr) != SQLITE_OK) + return; + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + sqlite3_bind_text(st, 2, name.c_str(), -1, + SQLITE_TRANSIENT); + sqlite3_bind_text(st, 3, name.c_str(), -1, + SQLITE_TRANSIENT); + std::string other; + while (sqlite3_step(st) == SQLITE_ROW) { + int k = sqlite3_column_int(st, 0); + if (wanted_type == 0 && (k == 0 || k == 1)) + continue; + if (k == wanted_type) + continue; + if (!other.empty()) + other += ","; + other += std::to_string(k); + } + sqlite3_finalize(st); + if (other.empty()) + return; + if (!hints.empty()) + hints += "; "; + // The hint is plain text; it is escaped once, when the + // writer embeds it in the response below. Escaping the + // name here too would double-escape it. + hints += std::string("no ") + type_label + " named '" + + name + + "' matched; the name exists under entity " + "kind " + + other + + " (kind assignment is translator-specific, " + "e.g. a constructor shares its class's " + "name) — retry without the type label to " + "see them"; + }; + // Only probe the side that actually resolved empty. + if (src_ids.empty()) + probeOtherKinds(src_name, src_type_val, + src_type.c_str()); + if (tgt_ids.empty()) + probeOtherKinds(tgt_name, tgt_type_val, + tgt_type.c_str()); + if (!hints.empty()) + empty.key("hint").value(hints); + empty.endObject(); + return empty.str(); + } + std::ostringstream json; json << "{\"results\":["; bool first_row = true; int row_count = 0; + // Set when a multi-hop bound fires; reported in the response tail. + bool multi_hop_truncated = false; + // Set when the caller's LIMIT cut the result set short. Reported the same + // way, so an unlimited response stays byte-identical to before. + bool user_limit_truncated = false; if (!multi_hop) { // ── Single hop: relation.src→target with filters ── @@ -525,6 +513,12 @@ std::string executeGraphQuery(uint64_t project_id, const char *dsl_query, sqlite3_bind_int64(st, 1, static_cast(project_id)); while (sqlite3_step(st) == SQLITE_ROW) { + // The caller's LIMIT, checked before the row is + // materialised so a large graph stops being scanned. + if (user_limit > 0 && row_count >= user_limit) { + user_limit_truncated = true; + break; + } int64_t eid = sqlite3_column_int64(st, 0); int64_t sid = sqlite3_column_int64(st, 1); std::string sn = @@ -562,16 +556,26 @@ std::string executeGraphQuery(uint64_t project_id, const char *dsl_query, json << ","; first_row = false; ++row_count; - json << "{\"source\":{\"id\":" << sid - << ",\"name\":\"" << jsonEscape(sn.c_str()) - << "\",\"type\":" << sk << ",\"file\":\"" - << jsonEscape(sf.c_str()) << "\"}," - << "\"edge\":{\"id\":" << eid - << ",\"type\":" << edge_type << "}," - << "\"target\":{\"id\":" << tid - << ",\"name\":\"" << jsonEscape(tn.c_str()) - << "\",\"type\":" << tk << ",\"file\":\"" - << jsonEscape(tf.c_str()) << "\"}}"; + util::JsonWriter el; + el.beginObject(); + el.key("source").beginObject(); + el.key("id").value(sid); + el.key("name").value(sn); + el.key("type").value(sk); + el.key("file").value(sf); + el.endObject(); + el.key("edge").beginObject(); + el.key("id").value(eid); + el.key("type").value(edge_type); + el.endObject(); + el.key("target").beginObject(); + el.key("id").value(tid); + el.key("name").value(tn); + el.key("type").value(tk); + el.key("file").value(tf); + el.endObject(); + el.endObject(); + json << el.str(); } sqlite3_finalize(st); } @@ -609,7 +613,9 @@ std::string executeGraphQuery(uint64_t project_id, const char *dsl_query, // BFS level by level. The queue carries the full node path so the // "1->2->3" chain is correct per branch. To keep output bounded we // stop expanding a node once it matches the target set (shortest - // representative paths). + // representative paths). `expansions` bounds the total work and + // `multi_hop_truncated` reports when that bound fired. + int64_t expansions = 0; for (int64_t start : src_set) { // {path, depth} — path includes `start`. std::queue, int>> bfs; @@ -619,6 +625,17 @@ std::string executeGraphQuery(uint64_t project_id, const char *dsl_query, bfs.pop(); if (depth > max_depth) continue; + // Cap the total work before expanding. A node reachable + // through several paths is re-expanded once per path (no + // visited set), so without this the walk is exponential in + // the hop count on a cyclic graph. + if (++expansions > kMultiHopExpansionBudget || + row_count >= kMultiHopMaxRows || + (user_limit > 0 && + row_count >= user_limit)) { + multi_hop_truncated = true; + break; + } int64_t node = path.back(); auto callees = store->getCalleeIds( static_cast(node)); @@ -648,35 +665,42 @@ std::string executeGraphQuery(uint64_t project_id, const char *dsl_query, std::to_string( npath[i]); } - json << "{\"source\":{\"id\":" - << start << ",\"name\":\"" - << jsonEscape(sn.c_str()) - << "\",\"type\":" << sk - << ",\"file\":\"" - << jsonEscape(sf.c_str()) - << "\"}," - << "\"target\":{\"id\":" - << n << ",\"name\":\"" - << jsonEscape(tn.c_str()) - << "\",\"type\":" << tk - << ",\"file\":\"" - << jsonEscape(tf.c_str()) - << "\"}," - << "\"depth\":" << depth - << ",\"chain\":\"" - << jsonEscape( - chain_str.c_str()) - << "\"}"; + util::JsonWriter el; + el.beginObject(); + el.key("source").beginObject(); + el.key("id").value(start); + el.key("name").value(sn); + el.key("type").value(sk); + el.key("file").value(sf); + el.endObject(); + el.key("target").beginObject(); + el.key("id").value(n); + el.key("name").value(tn); + el.key("type").value(tk); + el.key("file").value(tf); + el.endObject(); + el.key("depth").value(depth); + el.key("chain").value( + chain_str); + el.endObject(); + json << el.str(); } if (depth < max_depth && !is_tgt) bfs.push({ std::move(npath), depth + 1 }); } } + if (multi_hop_truncated) + break; } } - json << "],\"total\":" << row_count << "}"; + json << "],\"total\":" << row_count; + // Only present when a bound actually fired, so an untruncated response is + // byte-identical to before. + if (multi_hop_truncated || user_limit_truncated) + json << ",\"truncated\":true"; + json << "}"; return json.str(); } diff --git a/engine/src/query/graph_query.h b/engine/src/query/graph_query.h index 8839fd1..eb115bd 100644 --- a/engine/src/query/graph_query.h +++ b/engine/src/query/graph_query.h @@ -16,9 +16,16 @@ namespace query // Edge types use their string name (Calls, References, Contains, etc.). // Empty type matches any (e.g. ()-[Calls]->(Function)). // Optional :name filters by symbol name (e.g. (Function:add)). +// Optional trailing `LIMIT ` (case-insensitive) caps the result set: +// MATCH (Function)-[Calls]->(Function) LIMIT 100 +// The clause is honoured, not decoration: without it a broad pattern is +// bounded only by the internal scan cap, which for a large project can exceed +// the MCP transport's per-message limit. Any OTHER trailing text is rejected +// with an error instead of being ignored. // // Returns JSON: { "total": N, "results": [ { source: ..., edge: ..., target: -// ... }, ... ] } +// ... }, ... ] } — plus "truncated": true when a bound (LIMIT, the multi-hop +// row cap or the expansion budget) cut the result set short. std::string executeGraphQuery(uint64_t project_id, const char *dsl_query, store::GraphStore *store); diff --git a/engine/src/query/impact_analysis.cpp b/engine/src/query/impact_analysis.cpp index 879f929..44878ff 100644 --- a/engine/src/query/impact_analysis.cpp +++ b/engine/src/query/impact_analysis.cpp @@ -1,9 +1,12 @@ +#include "util/json_writer.h" +#include "util/path_util.h" #include "impact_analysis.h" #include "query_engine.h" #include #include #include +#include #include #include #include @@ -84,12 +87,17 @@ std::string analyzeChangeImpact(uint64_t project_id, store::GraphStore *store, // JSON contract identical regardless of compile configuration. // regardless of compile configuration. auto makeErrorJson = [](const std::string &msg) -> std::string { - std::ostringstream j; - j << "{\"error\":\"" << jsonEscape(msg.c_str()) - << "\",\"modified\":[],\"callers\":[],\"callees\":[]," - << "\"total_impacted\":0,\"max_depth\":" << kImpactMaxDepth - << ",\"approximation\":\"heuristic\"," - << "\"note\":\"" << kImpactNote << "\"}"; + util::JsonWriter j; + j.beginObject(); + j.key("error").value(msg); + j.key("modified").beginArray().endArray(); + j.key("callers").beginArray().endArray(); + j.key("callees").beginArray().endArray(); + j.key("total_impacted").value(0); + j.key("max_depth").value(kImpactMaxDepth); + j.key("approximation").value("heuristic"); + j.key("note").value(kImpactNote); + j.endObject(); return j.str(); }; @@ -107,6 +115,45 @@ std::string analyzeChangeImpact(uint64_t project_id, store::GraphStore *store, fprintf(stderr, "%s\n", err.c_str()); return makeErrorJson(err); } + // The caller names the files as THEY know them — relative to the project + // root, or absolute through a symlink (`/tmp` is `/private/tmp` on macOS) — + // while `entity.file_path` holds the resolved absolute form, because + // createProject() canonicalises the root it registers. The lookup below is + // an exact `file_path IN (...)`, so any other spelling matched nothing and + // the tool answered `modified: []` — "nothing changed" — for a file that + // had changed. Normalise with the SAME rule the store applies to a project + // root (store_core.cpp normalizeRootPath): anchor a relative path at the + // project root, then weakly_canonical, which is idempotent for a path that + // is already in the stored form. + { + const std::string root_path = + store->getProjectRootPath(project_id); + std::vector resolved; + const auto add = [&resolved](const std::string &candidate) { + if (candidate.empty()) + return; + if (std::find(resolved.begin(), resolved.end(), + candidate) == resolved.end()) + resolved.push_back(candidate); + }; + for (const std::string &given : files) { + // The value as given, first: a database driven through the engine + // API keeps the caller's own spelling (measured — a fixture indexed + // as /tmp/... holds /tmp/... rows while the CLI-indexed ones hold + // /private/tmp/...), so the verbatim form has to keep matching. + add(given); + std::filesystem::path p(given); + if (p.is_relative() && !root_path.empty()) + p = std::filesystem::path(root_path) / p; + // Plus the resolved, root-anchored form — the shared rule from + // util/path_util.h, which is what the store builds a project root + // with and therefore what the CLI-indexed rows hold. Accepting + // both is additive: whichever spelling the database holds matches. + if (p.is_absolute()) + add(util::resolvePath(p.string())); + } + files.swap(resolved); + } sqlite3 *db = store->handle(); // Find nodes in modified files (function/method entities). @@ -249,53 +296,49 @@ std::string analyzeChangeImpact(uint64_t project_id, store::GraphStore *store, lookupMeta(need_metadata, name_map, file_map); // ── Build JSON output (mirrors the SQLite branch) ────────── - std::ostringstream json; - json << "{\"error\":null,\"modified\":["; - bool first = true; + util::JsonWriter json; + json.beginObject(); + json.key("error").nullValue(); + json.key("modified").beginArray(); for (const auto &kv : modified_nodes) { - if (!first) - json << ","; - first = false; - json << "{\"id\":" << kv.first << ",\"name\":\"" - << jsonEscape(kv.second.c_str()) << "\"}"; + json.beginObject(); + json.key("id").value(kv.first); + json.key("name").value(kv.second); + json.endObject(); } - json << "],\"callers\":["; - first = true; + json.endArray(); + json.key("callers").beginArray(); for (const auto &e : caller_entries) { - if (!first) - json << ","; - first = false; - json << "{\"id\":" << e.node_id << ",\"name\":\"" - << jsonEscape(name_map[e.node_id].c_str()) - << "\",\"file\":\"" - << jsonEscape(file_map[e.node_id].c_str()) - << "\",\"depth\":" << e.depth << ",\"caller_of\":\"" - << jsonEscape(name_map[e.via_seed].c_str()) << "\"}"; + json.beginObject(); + json.key("id").value(e.node_id); + json.key("name").value(name_map[e.node_id]); + json.key("file").value(file_map[e.node_id]); + json.key("depth").value(e.depth); + json.key("caller_of").value(name_map[e.via_seed]); + json.endObject(); } - json << "],\"callees\":["; - first = true; + json.endArray(); + json.key("callees").beginArray(); for (const auto &e : callee_entries) { - if (!first) - json << ","; - first = false; - json << "{\"id\":" << e.node_id << ",\"name\":\"" - << jsonEscape(name_map[e.node_id].c_str()) - << "\",\"file\":\"" - << jsonEscape(file_map[e.node_id].c_str()) - << "\",\"depth\":" << e.depth << ",\"callee_of\":\"" - << jsonEscape(name_map[e.via_seed].c_str()) << "\"}"; + json.beginObject(); + json.key("id").value(e.node_id); + json.key("name").value(name_map[e.node_id]); + json.key("file").value(file_map[e.node_id]); + json.key("depth").value(e.depth); + json.key("callee_of").value(name_map[e.via_seed]); + json.endObject(); } - json << "],"; + json.endArray(); std::unordered_set all_impacted = modified_ids; for (const auto &e : caller_entries) all_impacted.insert(e.node_id); for (const auto &e : callee_entries) all_impacted.insert(e.node_id); - json << "\"total_impacted\":" << all_impacted.size(); - json << ",\"max_depth\":" << kImpactMaxDepth; - json << ",\"approximation\":\"heuristic\""; - json << ",\"note\":\"" << kImpactNote << "\""; - json << "}"; + json.key("total_impacted").value(all_impacted.size()); + json.key("max_depth").value(kImpactMaxDepth); + json.key("approximation").value("heuristic"); + json.key("note").value(kImpactNote); + json.endObject(); return json.str(); } diff --git a/engine/src/query/query_analysis.cpp b/engine/src/query/query_analysis.cpp index 86d0bcb..5956da2 100644 --- a/engine/src/query/query_analysis.cpp +++ b/engine/src/query/query_analysis.cpp @@ -1,5 +1,7 @@ +#include "util/json_writer.h" #include "query_engine.h" -// community_detection removed — Phase 0 cut +// Community detection lives in query_communities.cpp (restored after the +// Phase-0 cut that removed the original label-propagation implementation). #include "graph_query.h" #include "impact_analysis.h" @@ -18,6 +20,21 @@ namespace query { +namespace +{ +/// Text of a result column, escaped for embedding inside a JSON string +/// literal. Values emitted through this helper are DB-derived (symbol names, +/// file paths) and may contain quotes, backslashes or control characters, +/// which would otherwise produce JSON the MCP client cannot parse. A NULL +/// column becomes the empty string. +std::string columnTextEscaped(sqlite3_stmt *stmt, int col) +{ + const unsigned char *text = sqlite3_column_text(stmt, col); + return util::jsonEscapeString( + text ? reinterpret_cast(text) : ""); +} +} // namespace + std::string QueryEngine::searchCode(uint64_t project_id, const char *query, int limit) { @@ -47,19 +64,6 @@ std::string QueryEngine::detectChanges(uint64_t project_id, return analyzeChangeImpact(project_id, store_, modified_files_json); } -// ─── Community Detection ───────────────────────────────────── - -std::string QueryEngine::getCommunities(uint64_t project_id, int max_members, - int max_communities, - bool include_members) -{ - (void)project_id; - (void)max_members; - (void)max_communities; - (void)include_members; - return "{\"communities\":[],\"total\":0}"; -} - // ─── Hotspot Analysis ─────────────────────────────────────── std::string QueryEngine::getHotspots(uint64_t project_id, int top_n) @@ -121,13 +125,18 @@ std::string QueryEngine::getHotspots(uint64_t project_id, int top_n) j << ","; first = false; ++count; - j << "{\"id\":" << id << ",\"name\":\"" - << jsonEscape(name.c_str()) << "\",\"file\":\"" - << jsonEscape(file.c_str()) << "\",\"type\":" << kind - << ",\"caller_count\":" << caller_count - << ",\"complexity\":" << cyclomatic - << ",\"cognitive\":" << cognitive - << ",\"nesting_depth\":" << nesting << "}"; + util::JsonWriter el; + el.beginObject(); + el.key("id").value(id); + el.key("name").value(name); + el.key("file").value(file); + el.key("type").value(kind); + el.key("caller_count").value(caller_count); + el.key("complexity").value(cyclomatic); + el.key("cognitive").value(cognitive); + el.key("nesting_depth").value(nesting); + el.endObject(); + j << el.str(); } sqlite3_finalize(st); } @@ -179,7 +188,11 @@ std::string QueryEngine::getModuleMap(uint64_t project_id) if (!first_dir) json << ","; first_dir = false; - json << "{\"path\":\"" << dir << "\",\"files\":["; + // This object stays open across the files array below and is + // closed on two different paths, so it keeps its shape; the + // path value goes through the shared escaper. + json << "{\"path\":\"" << util::jsonEscapeString(dir) + << "\",\"files\":["; // Functions in this directory. entity is the canonical fact source // (the legacy graph_nodes table was migrated to entity); metrics @@ -191,7 +204,20 @@ std::string QueryEngine::getModuleMap(uint64_t project_id) "FROM entity e " "WHERE e.project_id = ? AND e.file_path LIKE ? " "AND e.kind IN (0,1) ORDER BY e.file_path"; - sqlite3_prepare_v2(db, func_sql.c_str(), -1, &stmt, nullptr); + // Check prepare: bind/step on a NULL stmt is UB and would + // crash the long-running MCP server through FFI (code_rules + // §3 Error Handling; no silent error handling). + if (sqlite3_prepare_v2(db, func_sql.c_str(), -1, &stmt, + nullptr) != SQLITE_OK) { + fprintf(stderr, + "getModuleMap: prepare functions failed: %s " + "[module=query, method=getModuleMap]\n", + sqlite3_errmsg(db)); + if (stmt) + sqlite3_finalize(stmt); + json << "]}"; + continue; + } sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, (dir + "/%").c_str(), -1, SQLITE_TRANSIENT); @@ -202,19 +228,11 @@ std::string QueryEngine::getModuleMap(uint64_t project_id) json << ","; first_fn = false; json << "{" - << "\"name\":\"" - << (sqlite3_column_text(stmt, 0) ? - reinterpret_cast( - sqlite3_column_text(stmt, 0)) : - "") + << "\"name\":\"" << columnTextEscaped(stmt, 0) << "\"," << "\"type\":" << sqlite3_column_int(stmt, 1) << "," - << "\"file\":\"" - << (sqlite3_column_text(stmt, 2) ? - reinterpret_cast( - sqlite3_column_text(stmt, 2)) : - "") + << "\"file\":\"" << columnTextEscaped(stmt, 2) << "\"," << "\"complexity\":" << sqlite3_column_int(stmt, 3) << "," @@ -283,12 +301,17 @@ std::string QueryEngine::getEntryPoints(uint64_t project_id) j << ","; first = false; ++count; - j << "{\"id\":" << id << ",\"name\":\"" - << jsonEscape(name.c_str()) << "\",\"type\":" << kind - << ",\"file\":\"" << jsonEscape(file.c_str()) - << "\",\"complexity\":" << cyc - << ",\"cognitive\":" << cog << ",\"nesting\":" << nest - << "}"; + util::JsonWriter el; + el.beginObject(); + el.key("id").value(id); + el.key("name").value(name); + el.key("type").value(kind); + el.key("file").value(file); + el.key("complexity").value(cyc); + el.key("cognitive").value(cog); + el.key("nesting").value(nest); + el.endObject(); + j << el.str(); } sqlite3_finalize(st); } @@ -307,6 +330,24 @@ std::string QueryEngine::traceCallChain(uint64_t project_id, !*to_function) { return "{\"error\":\"empty function name\"}"; } + sqlite3 *db_probe = (store_ && store_->handle()) ? store_->handle() : + nullptr; + // Homonym guard: the BFS below is name-keyed, so two entities with + // the same name collapse into one node and the trace silently follows + // whichever edge appeared first. Surface the candidates instead + // (T5 finding #9) — same contract as getCallers. + if (db_probe) { + std::string amb = query::bareNameCandidates( + db_probe, project_id, from_function); + if (!amb.empty()) + return "{\"found\":false,\"chain\":\"\",\"depth\":0," + + amb.substr(1); // merge: keep "ambiguous":… + amb = query::bareNameCandidates(db_probe, project_id, + to_function); + if (!amb.empty()) + return "{\"found\":false,\"chain\":\"\",\"depth\":0," + + amb.substr(1); + } // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── // Load the project's CALLS edges as (src_name → tgt_name) from the @@ -392,10 +433,12 @@ std::string QueryEngine::traceCallChain(uint64_t project_id, std::string chain = path[0]; for (size_t i = 1; i < path.size(); i++) chain += "→" + path[i]; - std::ostringstream json; - json << "{\"found\":true,\"chain\":\"" - << jsonEscape(chain.c_str()) - << "\",\"depth\":" << (path.size() - 1) << "}"; + util::JsonWriter json; + json.beginObject(); + json.key("found").value(true); + json.key("chain").value(chain); + json.key("depth").value(path.size() - 1); + json.endObject(); return json.str(); } return "{\"found\":false,\"chain\":\"\",\"depth\":0}"; @@ -428,57 +471,92 @@ std::string QueryEngine::getProjectOverview(uint64_t project_id) sqlite3_stmt *stmt = nullptr; // v0.2.6: count from the canonical entity/relation tables. // graph_nodes/graph_edges are deprecated and no longer written. - sqlite3_prepare_v2( - db, "SELECT COUNT(*) FROM entity WHERE project_id=?", - -1, &stmt, nullptr); - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) { - json << "\"total_nodes\":" - << sqlite3_column_int(stmt, 0) << ","; + // Every prepare is checked: bind/step on a NULL stmt is UB and + // would crash the long-running MCP server through FFI. + if (sqlite3_prepare_v2( + db, + "SELECT COUNT(*) FROM entity WHERE project_id=?", + -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(stmt, 1, + static_cast(project_id)); + if (sqlite3_step(stmt) == SQLITE_ROW) { + json << "\"total_nodes\":" + << sqlite3_column_int(stmt, 0) << ","; + } + } else { + fprintf(stderr, + "getProjectOverview: prepare entity count " + "failed: %s " + "[module=query, method=getProjectOverview]\n", + sqlite3_errmsg(db)); } sqlite3_finalize(stmt); + stmt = nullptr; - sqlite3_prepare_v2( - db, "SELECT COUNT(*) FROM relation WHERE project_id=?", - -1, &stmt, nullptr); - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) { - json << "\"total_edges\":" - << sqlite3_column_int(stmt, 0) << ","; + if (sqlite3_prepare_v2( + db, + "SELECT COUNT(*) FROM relation WHERE project_id=?", + -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(stmt, 1, + static_cast(project_id)); + if (sqlite3_step(stmt) == SQLITE_ROW) { + json << "\"total_edges\":" + << sqlite3_column_int(stmt, 0) << ","; + } + } else { + fprintf(stderr, + "getProjectOverview: prepare relation count " + "failed: %s " + "[module=query, method=getProjectOverview]\n", + sqlite3_errmsg(db)); } sqlite3_finalize(stmt); + stmt = nullptr; - sqlite3_prepare_v2( - db, "SELECT COUNT(*) FROM files WHERE project_id=?", -1, - &stmt, nullptr); - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) { - json << "\"total_files\":" - << sqlite3_column_int(stmt, 0) << ","; + if (sqlite3_prepare_v2( + db, "SELECT COUNT(*) FROM files WHERE project_id=?", + -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(stmt, 1, + static_cast(project_id)); + if (sqlite3_step(stmt) == SQLITE_ROW) { + json << "\"total_files\":" + << sqlite3_column_int(stmt, 0) << ","; + } + } else { + fprintf(stderr, + "getProjectOverview: prepare files count " + "failed: %s " + "[module=query, method=getProjectOverview]\n", + sqlite3_errmsg(db)); } sqlite3_finalize(stmt); + stmt = nullptr; // Language distribution json << "\"languages\":["; - sqlite3_prepare_v2( - db, - "SELECT language,COUNT(*) FROM files WHERE project_id=? " - "GROUP BY language", - -1, &stmt, nullptr); - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - bool first = true; - while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - json << "{\"lang\":\"" - << (sqlite3_column_text(stmt, 0) ? - reinterpret_cast( - sqlite3_column_text(stmt, 0)) : - "") - << "\"," - << "\"files\":" << sqlite3_column_int(stmt, 1) - << "}"; + if (sqlite3_prepare_v2( + db, + "SELECT language,COUNT(*) FROM files WHERE project_id=? " + "GROUP BY language", + -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(stmt, 1, + static_cast(project_id)); + bool first = true; + while (sqlite3_step(stmt) == SQLITE_ROW) { + if (!first) + json << ","; + first = false; + json << "{\"lang\":\"" + << columnTextEscaped(stmt, 0) << "\"," + << "\"files\":" + << sqlite3_column_int(stmt, 1) << "}"; + } + } else { + fprintf(stderr, + "getProjectOverview: prepare languages " + "failed: %s " + "[module=query, method=getProjectOverview]\n", + sqlite3_errmsg(db)); } json << "]"; sqlite3_finalize(stmt); @@ -507,27 +585,28 @@ static void appendRowsAsJson(sqlite3_stmt *stmt, std::ostringstream &json, if (!first) json << ","; first = false; - json << "{"; + util::JsonWriter el; + el.beginObject(); for (int i = 0; i < col_count; i++) { - if (i > 0) - json << ","; const char *col_name = sqlite3_column_name(stmt, i); - json << "\"" << (col_name ? col_name : "?") << "\":"; + // The writer escapes keys as well as values; the column + // name was concatenated unescaped before. + el.key(col_name ? col_name : "?"); int col_type = sqlite3_column_type(stmt, i); if (col_type == SQLITE_NULL) { - json << "null"; + el.nullValue(); } else if (col_type == SQLITE_INTEGER) { - json << sqlite3_column_int64(stmt, i); + el.value(sqlite3_column_int64(stmt, i)); } else { const char *text = reinterpret_cast( sqlite3_column_text(stmt, i)); - json << "\"" << jsonEscape(text ? text : "") - << "\""; + el.value(text ? text : ""); } } - json << "}"; + el.endObject(); + json << el.str(); } } diff --git a/engine/src/query/query_communities.cpp b/engine/src/query/query_communities.cpp new file mode 100644 index 0000000..1919617 --- /dev/null +++ b/engine/src/query/query_communities.cpp @@ -0,0 +1,392 @@ +// query_communities.cpp — community detection over the CALLS graph. +// +// The Phase-0 cut removed the original label-propagation implementation and +// left `QueryEngine::getCommunities` as a stub returning an empty array while +// the C ABI export stayed in place. This file restores the algorithm on top of +// the canonical `relation` table (type=1 CALLS edges) so the MCP tool can be +// bound to a real implementation. +// +// Design notes (deliberately minimal): +// * The graph is treated as undirected: a CALLS edge relates both +// endpoints, and direction is irrelevant to "which functions form a +// cluster". Self-loops are dropped — a recursive call carries no +// community signal. +// * Only nodes with at least one CALLS edge participate. Isolated entities +// would each become a singleton community and flood the output with noise. +// * Deterministic: nodes are visited in ascending id order and label ties +// prefer the node's current label, then the smallest label id, so the same +// database always yields the same communities (the project's determinism +// contract — see CHANGELOG "Entity ids depended on the row insertion +// order"). +// * Updates are SYNCHRONOUS (Jacobi): every node computes its next label +// from the previous round's labels, and the results are applied together. +// A sequential sweep lets a label spread across an entire connected +// component in one pass, which collapses two densely-connected clusters +// joined by a single bridge edge into one giant community — useless as a +// structural summary. Synchronous updates keep the bridge from merging +// them in the first round, and the "prefer current label" tie-break +// damps the oscillation synchronous LPA is known for. +// * Summary-first output: members are omitted unless the caller asks for +// them, and both the community count and the member count are clamped. An +// earlier revision of this feature returned every member of every +// community — ~342 KB / ~100 K tokens for one query, the opposite of what +// this project exists to do. + +#include "util/json_writer.h" +#include "query_engine.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace query +{ + +namespace +{ + +/// Label-propagation safety bound. Convergence on a call graph takes a +/// handful of rounds; the cap keeps a pathological graph from oscillating +/// forever. +constexpr int kMaxLpaIterations = 20; + +/// Default and maximum output sizes. +constexpr int kDefaultMaxCommunities = 20; +constexpr int kMaxCommunitiesCap = 500; +constexpr int kDefaultMaxMembers = 10; +constexpr int kMaxMembersCap = 200; + +/// Hard ceiling on member rows emitted across ALL communities, so asking for +/// "all members of all communities" cannot produce an unbounded payload. +constexpr size_t kMaxTotalMembersEmitted = 5000; + +/// Entity metadata for one graph node. +struct NodeMeta { + std::string name; + std::string file_path; +}; + +/// The heuristic disclaimer shared by every response shape. The call graph is +/// name-matched (see graph_builder "Known limitations"), so communities are an +/// approximation of the real clustering. +const char *const kCommunityNote = + "Communities inferred by label propagation over CALLS edges, which are " + "resolved by name matching; indirect calls (virtual/pointer) may be " + "missing."; + +} // namespace + +std::string QueryEngine::getCommunities(uint64_t project_id, int max_members, + int max_communities, + bool include_members) +{ + // ── Parameter clamps ──────────────────────────────────────── + // A non-positive value means "use the default" rather than "unlimited": + // an unlimited response is exactly the failure mode described above. + if (max_communities <= 0) + max_communities = kDefaultMaxCommunities; + if (max_communities > kMaxCommunitiesCap) + max_communities = kMaxCommunitiesCap; + if (max_members <= 0) + max_members = kDefaultMaxMembers; + if (max_members > kMaxMembersCap) + max_members = kMaxMembersCap; + + auto emptyResult = [&](const char *error) { + std::ostringstream out; + out << "{\"communities\":[],\"total_communities\":0," + "\"returned_communities\":0,\"inter_community_edges\":0," + "\"truncated\":false,\"approximation\":\"heuristic\"," + "\"note\":\"" + << kCommunityNote << "\""; + if (error != nullptr) + out << ",\"error\":\"" << util::jsonEscapeString(error) + << "\""; + out << "}"; + return out.str(); + }; + + if (store_ == nullptr || store_->handle() == nullptr) + return emptyResult( + "graph not ready [module=query, method=getCommunities]"); + sqlite3 *db = store_->handle(); + + // ── 1. Read the CALLS edges in a deterministic order ──────── + std::vector> edges; + { + const char *sql = "SELECT source_id, target_id FROM relation " + "WHERE project_id=? AND type=1 " + "ORDER BY source_id, target_id"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) != SQLITE_OK) + return emptyResult( + "failed to read call edges " + "[module=query, method=getCommunities]"); + sqlite3_bind_int64(st, 1, static_cast(project_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + uint64_t src = static_cast( + sqlite3_column_int64(st, 0)); + uint64_t tgt = static_cast( + sqlite3_column_int64(st, 1)); + if (src == 0 || tgt == 0 || src == tgt) + continue; // self-loop — no community signal + edges.emplace_back(src, tgt); + } + sqlite3_finalize(st); + } + if (edges.empty()) + return emptyResult(nullptr); + + // ── 2. Node set (ascending id) with an id → index map ─────── + std::vector node_ids; + node_ids.reserve(edges.size() * 2); + for (const auto &e : edges) { + node_ids.push_back(e.first); + node_ids.push_back(e.second); + } + std::sort(node_ids.begin(), node_ids.end()); + node_ids.erase(std::unique(node_ids.begin(), node_ids.end()), + node_ids.end()); + const size_t node_count = node_ids.size(); + std::unordered_map index_of; + index_of.reserve(node_count * 2); + for (size_t i = 0; i < node_count; ++i) + index_of.emplace(node_ids[i], static_cast(i)); + + // ── 3. Undirected, deduplicated adjacency ─────────────────── + std::vector> adj(node_count); + for (const auto &e : edges) { + uint32_t a = index_of[e.first]; + uint32_t b = index_of[e.second]; + adj[a].push_back(b); + adj[b].push_back(a); + } + for (auto &neighbours : adj) { + std::sort(neighbours.begin(), neighbours.end()); + neighbours.erase(std::unique(neighbours.begin(), + neighbours.end()), + neighbours.end()); + } + + // ── 4. Label propagation (synchronous / Jacobi) ───────────── + // Each node starts as its own community. On every round a node adopts + // the most frequent label among its neighbours AS OF THE PREVIOUS ROUND; + // all updates are applied together. Ties prefer the node's current label + // (which damps oscillation), then the smallest label id. + std::vector label(node_count); + std::vector next_label(node_count); + for (size_t i = 0; i < node_count; ++i) + label[i] = static_cast(i); + std::vector label_count(node_count, 0); + std::vector touched; + for (int iter = 0; iter < kMaxLpaIterations; ++iter) { + bool changed = false; + for (size_t i = 0; i < node_count; ++i) { + if (adj[i].empty()) { + next_label[i] = label[i]; + continue; + } + touched.clear(); + for (uint32_t nb : adj[i]) { + uint32_t nb_label = label[nb]; + if (label_count[nb_label] == 0) + touched.push_back(nb_label); + ++label_count[nb_label]; + } + int best_count = 0; + for (uint32_t candidate : touched) + best_count = std::max(best_count, + label_count[candidate]); + uint32_t best = label[i]; + bool current_wins = false; + for (uint32_t candidate : touched) { + if (label_count[candidate] == best_count && + candidate == label[i]) { + current_wins = true; + break; + } + } + if (!current_wins) { + uint32_t smallest = 0; + bool found = false; + for (uint32_t candidate : touched) { + if (label_count[candidate] != + best_count) + continue; + if (!found || candidate < smallest) { + smallest = candidate; + found = true; + } + } + if (found) + best = smallest; + } + for (uint32_t candidate : touched) + label_count[candidate] = 0; + next_label[i] = best; + if (best != label[i]) + changed = true; + } + label.swap(next_label); + if (!changed) + break; + } + + // ── 5. Group nodes per label ──────────────────────────────── + struct CommunityRow { + uint32_t representative = 0; // node index + std::vector members; // node indices, ascending + }; + std::unordered_map> groups; + groups.reserve(node_count); + for (size_t i = 0; i < node_count; ++i) + groups[label[i]].push_back(static_cast(i)); + + std::vector communities; + communities.reserve(groups.size()); + for (auto &entry : groups) { + CommunityRow row; + row.members = std::move(entry.second); + // Members are ascending because i ascends. The representative is + // the highest-degree member (the hub a reader can recognise); + // ties break on the node index for determinism. + uint32_t rep = row.members.front(); + for (uint32_t member : row.members) { + if (adj[member].size() > adj[rep].size() || + (adj[member].size() == adj[rep].size() && + member < rep)) + rep = member; + } + row.representative = rep; + communities.push_back(std::move(row)); + } + // Deterministic ordering: size descending, then representative id + // ascending. An unordered_map iteration order would otherwise leak into + // the response. + std::sort(communities.begin(), communities.end(), + [&](const CommunityRow &a, const CommunityRow &b) { + if (a.members.size() != b.members.size()) + return a.members.size() > b.members.size(); + return node_ids[a.representative] < + node_ids[b.representative]; + }); + + // ── 6. Inter-community edge count ─────────────────────────── + size_t inter_community_edges = 0; + for (const auto &e : edges) { + if (label[index_of[e.first]] != label[index_of[e.second]]) + ++inter_community_edges; + } + + // ── 7. Entity metadata, restricted to participating nodes ─── + // One scan of `entity`; rows outside the node set are skipped so the + // map holds only graph nodes (a project can have far more entities + // than the call graph touches). + std::unordered_map meta; + { + const char *sql = "SELECT id, name, file_path FROM entity " + "WHERE project_id=?"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + uint64_t id = static_cast( + sqlite3_column_int64(st, 0)); + if (index_of.find(id) == index_of.end()) + continue; + NodeMeta node; + const unsigned char *name = + sqlite3_column_text(st, 1); + const unsigned char *path = + sqlite3_column_text(st, 2); + node.name = + (name != nullptr) ? + reinterpret_cast( + name) : + ""; + node.file_path = + (path != nullptr) ? + reinterpret_cast( + path) : + ""; + meta.emplace(id, std::move(node)); + } + sqlite3_finalize(st); + } + } + static const NodeMeta kMissingMeta{ "", "" }; + auto metaOf = [&](uint32_t idx) -> const NodeMeta & { + auto it = meta.find(node_ids[idx]); + return (it != meta.end()) ? it->second : kMissingMeta; + }; + + // ── 8. Emit ───────────────────────────────────────────────── + const size_t total_communities = communities.size(); + const size_t returned_communities = std::min( + total_communities, static_cast(max_communities)); + std::ostringstream json; + json << "{\"communities\":["; + size_t total_members_emitted = 0; + bool members_truncated = false; + for (size_t ci = 0; ci < returned_communities; ++ci) { + const CommunityRow &row = communities[ci]; + if (ci > 0) + json << ","; + // Element stays open across the optional members array below, so + // its shape is kept; the label goes through the shared escaper. + json << "{\"id\":" << node_ids[row.representative] + << ",\"label\":\"" + << util::jsonEscapeString(metaOf(row.representative).name) + << "\",\"member_count\":" << row.members.size(); + if (include_members) { + size_t emit_members = std::min( + row.members.size(), + static_cast(max_members)); + if (emit_members < row.members.size()) + members_truncated = true; + // Global ceiling across all communities. + size_t room = (total_members_emitted < + kMaxTotalMembersEmitted) ? + kMaxTotalMembersEmitted - + total_members_emitted : + 0; + if (emit_members > room) { + emit_members = room; + members_truncated = true; + } + json << ",\"members\":["; + for (size_t mi = 0; mi < emit_members; ++mi) { + uint32_t idx = row.members[mi]; + if (mi > 0) + json << ","; + util::JsonWriter el; + el.beginObject(); + el.key("id").value(node_ids[idx]); + el.key("name").value(metaOf(idx).name); + el.key("file_path").value(metaOf(idx).file_path); + el.endObject(); + json << el.str(); + } + json << "]"; + total_members_emitted += emit_members; + } + json << "}"; + } + const bool truncated = (returned_communities < total_communities) || + members_truncated; + json << "],\"total_communities\":" << total_communities + << ",\"returned_communities\":" << returned_communities + << ",\"inter_community_edges\":" << inter_community_edges + << ",\"truncated\":" << (truncated ? "true" : "false") + << ",\"approximation\":\"heuristic\",\"note\":\"" << kCommunityNote + << "\"}"; + return json.str(); +} + +} // namespace query diff --git a/engine/src/query/query_engine.cpp b/engine/src/query/query_engine.cpp index d55a74a..23bcc09 100644 --- a/engine/src/query/query_engine.cpp +++ b/engine/src/query/query_engine.cpp @@ -1,3 +1,5 @@ +#include "util/json_writer.h" +#include "util/path_util.h" #include "query_engine.h" // community_detection removed — Phase 0 cut #include "graph_query.h" @@ -7,8 +9,6 @@ #include #include #include -#include -#include #include #include #include @@ -18,48 +18,23 @@ namespace query { -// Maximum BFS depth for findShortestPath. Limits traversal to prevent -// unbounded walks over very large graphs; chosen to cover typical call -// chains while keeping query latency bounded. -static constexpr int kShortestPathMaxDepth = 10; - -// Standard note appended to findShortestPath results explaining the -// heuristic nature of the call graph (name-matched, no virtual dispatch). -static const char *const kShortestPathNote = - "Call graph edges are resolved by name matching; indirect calls " - "(virtual/pointer) may be missing."; - -// ─── JSON string escaping ────────────────────────────────────── - -std::string jsonEscape(const char *s) -{ - if (!s) - return ""; - std::string out; - for (const char *p = s; *p; p++) { - switch (*p) { - case '"': - out += "\\\""; - break; - case '\\': - out += "\\\\"; - break; - case '\n': - out += "\\n"; - break; - case '\r': - out += "\\r"; - break; - case '\t': - out += "\\t"; - break; - default: - out += *p; - break; - } - } - return out; -} +// What to do when a bare name matches several entities. +// +// The candidates array alone was not enough: every name-based tool answered +// `{"…":[],"total":0,"ambiguous":true,"candidates":[…]}` for names as ordinary +// as `main` (100+ definitions in one Rust project's examples/), `Agent` or +// `new`, and a caller reading only the empty list concluded "no edges" — the +// next step (`*_by_entity`, which does resolve them) was discoverable only by +// reading the tool catalogue. One sentence, shared by getCallers, getCallees +// and the trace probe so the wording cannot drift. +static const char *const kAmbiguousNameHint = + "Name matches several entities, so no edges were resolved. Pass one of " + "the candidates' graph_node_id to find_callers_by_entity / " + "find_callees_by_entity, or narrow the name with file_filter."; + +// findShortestPath moved to query_engine_traverse.cpp together with +// kShortestPathMaxDepth / kShortestPathNote; the copies that used to live here +// were left behind by that split and only produced unused-variable warnings. QueryEngine::QueryEngine(store::GraphStore *store) : store_(store) @@ -72,57 +47,61 @@ std::string queryToJson(sqlite3 *db, const char *sql, const char *result_key) { sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) != SQLITE_OK) { - return "{\"total\":0,\"results\":[],\"error\":\"" + - std::string(sqlite3_errmsg(db)) + "\"}"; + util::JsonWriter err; + err.beginObject(); + err.key("total").value(0); + err.key("results").beginArray().endArray(); + // The writer escapes the message: sqlite3_errmsg() can carry a + // quote, which previously produced invalid JSON here. + err.key("error").value(sqlite3_errmsg(db)); + err.endObject(); + return err.str(); } - std::ostringstream json; - json << "{\"" << result_key << "\":["; + util::JsonWriter json; + json.beginObject(); + json.key(result_key).beginArray(); int col_count = sqlite3_column_count(stmt); - bool first_row = true; int row_count = 0; while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first_row) - json << ","; - first_row = false; row_count++; - json << "{"; + json.beginObject(); for (int i = 0; i < col_count; i++) { - if (i > 0) - json << ","; const char *col_name = sqlite3_column_name(stmt, i); - // L2 fix: escape the column name so a name containing a quote - // or control char cannot produce invalid JSON. - json << "\"" << jsonEscape(col_name ? col_name : "") - << "\":"; + // The writer escapes keys and values, so a column name + // containing a quote or control char cannot produce + // invalid JSON. + json.key(col_name ? col_name : ""); int col_type = sqlite3_column_type(stmt, i); if (col_type == SQLITE_NULL) { - json << "null"; + json.nullValue(); } else if (col_type == SQLITE_INTEGER) { - json << sqlite3_column_int64(stmt, i); + json.value(sqlite3_column_int64(stmt, i)); } else if (col_type == SQLITE_FLOAT) { double val = sqlite3_column_double(stmt, i); - // Use integer output for whole numbers to avoid "1.000000" + // Use integer output for whole numbers to avoid + // "1.000000" if (val == static_cast(val)) - json << static_cast(val); + json.value(static_cast(val)); else - json << val; + json.value(val); } else { const char *text = reinterpret_cast( sqlite3_column_text(stmt, i)); - json << "\"" << jsonEscape(text ? text : "") - << "\""; + json.value(text ? text : ""); } } - json << "}"; + json.endObject(); } - json << "],\"total\":" << row_count << "}"; + json.endArray(); + json.key("total").value(row_count); + json.endObject(); sqlite3_finalize(stmt); return json.str(); } @@ -150,8 +129,10 @@ std::string QueryEngine::findDefinition(uint64_t project_id, // into the LIKE literal. Splicing let a filter containing a quote or // % break the query or inject SQL; a bound `%filter%` value is safe. bool has_filter = file_filter && strlen(file_filter) > 0; + // Two patterns per filter — the value as given and its resolved form — so a + // database whose rows hold either spelling still matches (util::lookupPath). if (has_filter) - sql += " AND file_path LIKE ?"; + sql += " AND (file_path LIKE ? OR file_path LIKE ?)"; sql += " LIMIT 20"; std::ostringstream json; json << "{\"results\":["; @@ -164,8 +145,10 @@ std::string QueryEngine::findDefinition(uint64_t project_id, sqlite3_bind_text(st, 2, symbol_name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(st, 3, symbol_name, -1, SQLITE_TRANSIENT); if (has_filter) { - std::string like = "%" + std::string(file_filter) + "%"; - sqlite3_bind_text(st, 4, like.c_str(), -1, + const auto patterns = util::lookupPatterns(file_filter); + sqlite3_bind_text(st, 4, patterns[0].c_str(), -1, + SQLITE_TRANSIENT); + sqlite3_bind_text(st, 5, patterns[1].c_str(), -1, SQLITE_TRANSIENT); } while (sqlite3_step(st) == SQLITE_ROW) { @@ -203,16 +186,20 @@ std::string QueryEngine::findDefinition(uint64_t project_id, reinterpret_cast( sqlite3_column_text(st, 9)) : ""; - json << "{\"node_id\":" << node_id << ",\"name\":\"" - << jsonEscape(name.c_str()) - << "\",\"qualified_name\":\"" - << jsonEscape(qn.c_str()) - << "\",\"node_type\":" << ntype - << ",\"file_path\":\"" << jsonEscape(fp.c_str()) - << "\",\"start_row\":" << sr - << ",\"start_col\":" << sc << ",\"end_row\":" << er - << ",\"end_col\":" << ec << ",\"language\":\"" - << jsonEscape(lang.c_str()) << "\"}"; + util::JsonWriter el; + el.beginObject(); + el.key("node_id").value(node_id); + el.key("name").value(name); + el.key("qualified_name").value(qn); + el.key("node_type").value(ntype); + el.key("file_path").value(fp); + el.key("start_row").value(sr); + el.key("start_col").value(sc); + el.key("end_row").value(er); + el.key("end_col").value(ec); + el.key("language").value(lang); + el.endObject(); + json << el.str(); } sqlite3_finalize(st); } @@ -263,8 +250,10 @@ std::string QueryEngine::findReferences(uint64_t project_id, sqlite3_bind_text(st, 3, symbol_name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(st, 4, symbol_name, -1, SQLITE_TRANSIENT); if (has_filter) { - std::string like = "%" + std::string(file_filter) + "%"; - sqlite3_bind_text(st, 5, like.c_str(), -1, + const auto patterns = util::lookupPatterns(file_filter); + sqlite3_bind_text(st, 5, patterns[0].c_str(), -1, + SQLITE_TRANSIENT); + sqlite3_bind_text(st, 6, patterns[1].c_str(), -1, SQLITE_TRANSIENT); } while (sqlite3_step(st) == SQLITE_ROW) { @@ -302,20 +291,41 @@ std::string QueryEngine::findReferences(uint64_t project_id, reinterpret_cast( sqlite3_column_text(st, 9)) : ""; - json << "{\"node_id\":" << node_id << ",\"name\":\"" - << jsonEscape(name.c_str()) - << "\",\"qualified_name\":\"" - << jsonEscape(qn.c_str()) - << "\",\"node_type\":" << ntype - << ",\"file_path\":\"" << jsonEscape(fp.c_str()) - << "\",\"start_row\":" << sr - << ",\"start_col\":" << sc << ",\"end_row\":" << er - << ",\"end_col\":" << ec << ",\"language\":\"" - << jsonEscape(lang.c_str()) << "\"}"; + util::JsonWriter el; + el.beginObject(); + el.key("node_id").value(node_id); + el.key("name").value(name); + el.key("qualified_name").value(qn); + el.key("node_type").value(ntype); + el.key("file_path").value(fp); + el.key("start_row").value(sr); + el.key("start_col").value(sc); + el.key("end_row").value(er); + el.key("end_col").value(ec); + el.key("language").value(lang); + el.endObject(); + json << el.str(); } sqlite3_finalize(st); } - json << "],\"total\":" << count << "}"; + json << "],\"total\":" << count; + if (count == 0) { + // An empty answer must not read as "this symbol is unused". The + // reference table records call sites (type 1 Calls, type 3 + // symbol_reference), and both are emitted for call expressions — a + // symbol that is only USED AS A VALUE has no row at all. Measured + // 2026-10-06 on goagent: get_routes reported the handler `t.handleMCP` + // (transport_server.go:277) while find_references answered 0, because + // registering a handler is not a call. The note names the tools that + // can still locate such a use. + json << ",\"note\":\"No call or symbol-reference edge targets this name. " + "References are recorded per call site, so a symbol that is only " + "used as a value (a route handler, a callback passed by name, a " + "function pointer) has no edge here — an empty result is not proof " + "that it is unused. get_routes, find_symbol and get_type_info can " + "still locate it.\""; + } + json << "}"; return json.str(); } @@ -355,8 +365,12 @@ std::string QueryEngine::getCallers(uint64_t project_id, auto resolveIds = [&](std::vector &ids) { std::string sql = "SELECT id FROM entity WHERE project_id=? " "AND (name=? OR qualified_name=?)"; + // M3 fix: bind the file_filter instead of splicing it into the + // LIKE literal — a filter containing a quote could inject SQL, + // while a bound `%filter%` value is safe. Mirrors the fix + // already applied to findDefinition / findReferences. if (!has_filter.empty()) - sql += " AND file_path LIKE '%" + has_filter + "%'"; + sql += " AND file_path LIKE ?"; sqlite3_stmt *st = nullptr; if (sqlite3_prepare_v2(db, sql.c_str(), -1, &st, nullptr) != SQLITE_OK) @@ -364,6 +378,13 @@ std::string QueryEngine::getCallers(uint64_t project_id, sqlite3_bind_int64(st, 1, static_cast(project_id)); sqlite3_bind_text(st, 2, function_name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(st, 3, function_name, -1, SQLITE_TRANSIENT); + if (!has_filter.empty()) { + const auto patterns = util::lookupPatterns(has_filter); + sqlite3_bind_text(st, 4, patterns[0].c_str(), -1, + SQLITE_TRANSIENT); + sqlite3_bind_text(st, 5, patterns[1].c_str(), -1, + SQLITE_TRANSIENT); + } while (sqlite3_step(st) == SQLITE_ROW) ids.push_back(sqlite3_column_int64(st, 0)); sqlite3_finalize(st); @@ -414,16 +435,20 @@ std::string QueryEngine::getCallers(uint64_t project_id, if (!first_c) cands += ","; first_c = false; - cands += "{\"graph_node_id\":" + std::to_string(id) + - ",\"name\":\"" + jsonEscape(nm.c_str()) + - "\",\"file_path\":\"" + - jsonEscape(fp.c_str()) + - "\",\"start_row\":" + std::to_string(sr) + - ",\"start_col\":" + std::to_string(sc) + "}"; + util::JsonWriter el; + el.beginObject(); + el.key("graph_node_id").value(id); + el.key("name").value(nm); + el.key("file_path").value(fp); + el.key("start_row").value(sr); + el.key("start_col").value(sc); + el.endObject(); + cands += el.str(); } return "{\"callers\":[],\"total\":0,\"ambiguous\":true," "\"candidates\":[" + - cands + "]}"; + cands + "],\"hint\":\"" + + util::jsonEscapeString(kAmbiguousNameHint) + "\"}"; } // Query callers: relation rows where target_id is one of the matched @@ -484,21 +509,21 @@ std::string QueryEngine::getCallers(uint64_t project_id, result += ","; first = false; ++count; - result += - "{\"node_id\":" + std::to_string(node_id) + - ",\"name\":\"" + jsonEscape(name.c_str()) + - "\",\"file_path\":\"" + - jsonEscape(file.c_str()) + "\",\"start_row\":" + - std::to_string(start_row) + - ",\"start_col\":" + std::to_string(start_col) + - ",\"confidence\":" + - std::to_string(confidence) + - ",\"resolver\":\"" + - jsonEscape(resolver.c_str()) + - "\",\"resolution_kind\":\"" + - jsonEscape(rkind.c_str()) + - "\",\"resolve_strategy\":\"" + - jsonEscape(rkind.c_str()) + "\"}"; + util::JsonWriter el; + el.beginObject(); + el.key("node_id").value(node_id); + el.key("name").value(name); + el.key("file_path").value(file); + el.key("start_row").value(start_row); + el.key("start_col").value(start_col); + // raw(): std::to_string keeps the historic "%f" formatting + // for this field. + el.key("confidence").raw(std::to_string(confidence)); + el.key("resolver").value(resolver); + el.key("resolution_kind").value(rkind); + el.key("resolve_strategy").value(rkind); + el.endObject(); + result += el.str(); } sqlite3_finalize(st); } @@ -529,8 +554,12 @@ std::string QueryEngine::getCallees(uint64_t project_id, auto resolveIds = [&](std::vector &ids) { std::string sql = "SELECT id FROM entity WHERE project_id=? " "AND (name=? OR qualified_name=?)"; + // M3 fix: bind the file_filter instead of splicing it into the + // LIKE literal — a filter containing a quote could inject SQL, + // while a bound `%filter%` value is safe. Mirrors the fix + // already applied to findDefinition / findReferences. if (!has_filter.empty()) - sql += " AND file_path LIKE '%" + has_filter + "%'"; + sql += " AND file_path LIKE ?"; sqlite3_stmt *st = nullptr; if (sqlite3_prepare_v2(db, sql.c_str(), -1, &st, nullptr) != SQLITE_OK) @@ -538,6 +567,13 @@ std::string QueryEngine::getCallees(uint64_t project_id, sqlite3_bind_int64(st, 1, static_cast(project_id)); sqlite3_bind_text(st, 2, function_name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(st, 3, function_name, -1, SQLITE_TRANSIENT); + if (!has_filter.empty()) { + const auto patterns = util::lookupPatterns(has_filter); + sqlite3_bind_text(st, 4, patterns[0].c_str(), -1, + SQLITE_TRANSIENT); + sqlite3_bind_text(st, 5, patterns[1].c_str(), -1, + SQLITE_TRANSIENT); + } while (sqlite3_step(st) == SQLITE_ROW) ids.push_back(sqlite3_column_int64(st, 0)); sqlite3_finalize(st); @@ -587,16 +623,20 @@ std::string QueryEngine::getCallees(uint64_t project_id, if (!first_c) cands += ","; first_c = false; - cands += "{\"graph_node_id\":" + std::to_string(id) + - ",\"name\":\"" + jsonEscape(nm.c_str()) + - "\",\"file_path\":\"" + - jsonEscape(fp.c_str()) + - "\",\"start_row\":" + std::to_string(sr) + - ",\"start_col\":" + std::to_string(sc) + "}"; + util::JsonWriter el; + el.beginObject(); + el.key("graph_node_id").value(id); + el.key("name").value(nm); + el.key("file_path").value(fp); + el.key("start_row").value(sr); + el.key("start_col").value(sc); + el.endObject(); + cands += el.str(); } return "{\"callees\":[],\"total\":0,\"ambiguous\":true," "\"candidates\":[" + - cands + "]}"; + cands + "],\"hint\":\"" + + util::jsonEscapeString(kAmbiguousNameHint) + "\"}"; } // Read outgoing Calls edges: relation rows where source_id is the @@ -656,21 +696,21 @@ std::string QueryEngine::getCallees(uint64_t project_id, result += ","; first = false; ++count; - result += - "{\"node_id\":" + std::to_string(node_id) + - ",\"name\":\"" + jsonEscape(name.c_str()) + - "\",\"file_path\":\"" + - jsonEscape(file.c_str()) + "\",\"start_row\":" + - std::to_string(start_row) + - ",\"start_col\":" + std::to_string(start_col) + - ",\"confidence\":" + - std::to_string(confidence) + - ",\"resolver\":\"" + - jsonEscape(resolver.c_str()) + - "\",\"resolution_kind\":\"" + - jsonEscape(rkind.c_str()) + - "\",\"resolve_strategy\":\"" + - jsonEscape(rkind.c_str()) + "\"}"; + util::JsonWriter el; + el.beginObject(); + el.key("node_id").value(node_id); + el.key("name").value(name); + el.key("file_path").value(file); + el.key("start_row").value(start_row); + el.key("start_col").value(start_col); + // raw(): std::to_string keeps the historic "%f" formatting + // for this field. + el.key("confidence").raw(std::to_string(confidence)); + el.key("resolver").value(resolver); + el.key("resolution_kind").value(rkind); + el.key("resolve_strategy").value(rkind); + el.endObject(); + result += el.str(); } sqlite3_finalize(st); } @@ -678,589 +718,6 @@ std::string QueryEngine::getCallees(uint64_t project_id, return result; } -// ── Step 7 (plan §7.2): entity-precise query APIs ──────────────────── -// -// These methods resolve an entity ID to (name, file_path, start_row) in -// SQLite, then build a SQLite Cypher query that filters by all three -// fields. This eliminates the homonym aggregation problem: multiple -// entities named "__init__" in different classes/files are no longer -// merged into a single result set. -// -// The old bare-name APIs (getCallers/getCallees) are retained for -// backward compatibility but now detect ambiguity: when multiple -// entities match the bare name, they return ambiguous=true with a -// candidate list instead of silently aggregating. - -std::string QueryEngine::getCallersByEntity(uint64_t project_id, - uint64_t entity_id) -{ - // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── - // Callers by explicit entity id: read incoming Calls edges - // (relation where target_id = entity_id, type=1) via the - // (project_id, target_id) index. JSON shape mirrors the SQLite - // branch, including the trailing entity_id. - if (!store_ || !store_->handle()) { - return "{\"callers\":[],\"total\":0,\"error\":\"graph not ready " - "[module=query, method=getCallersByEntity]\"}"; - } - sqlite3 *db = store_->handle(); - std::string result = "{\"callers\":["; - bool first = true; - int count = 0; - { - const char *sql = - "SELECT e.id, e.name, e.file_path, e.start_row, " - " e.start_col, r.confidence, r.resolver, " - " r.resolution_kind " - "FROM relation r JOIN entity e ON e.id = r.source_id " - "WHERE r.project_id=? AND r.type=1 " - "AND r.target_id=? " - "GROUP BY e.id ORDER BY e.id LIMIT 1000"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(st, 1, - static_cast(project_id)); - sqlite3_bind_int64(st, 2, - static_cast(entity_id)); - while (sqlite3_step(st) == SQLITE_ROW) { - int64_t node_id = sqlite3_column_int64(st, 0); - std::string name = - reinterpret_cast( - sqlite3_column_text(st, 1)) ? - reinterpret_cast( - sqlite3_column_text( - st, 1)) : - ""; - std::string file = - reinterpret_cast( - sqlite3_column_text(st, 2)) ? - reinterpret_cast( - sqlite3_column_text( - st, 2)) : - ""; - int start_row = sqlite3_column_int(st, 3); - int start_col = sqlite3_column_int(st, 4); - double confidence = - sqlite3_column_double(st, 5); - std::string resolver = - reinterpret_cast( - sqlite3_column_text(st, 6)) ? - reinterpret_cast( - sqlite3_column_text( - st, 6)) : - ""; - std::string rkind = - reinterpret_cast( - sqlite3_column_text(st, 7)) ? - reinterpret_cast( - sqlite3_column_text( - st, 7)) : - ""; - if (!first) - result += ","; - first = false; - ++count; - result += "{\"node_id\":" + - std::to_string(node_id) + - ",\"name\":\"" + - jsonEscape(name.c_str()) + - "\",\"file_path\":\"" + - jsonEscape(file.c_str()) + - "\",\"start_row\":" + - std::to_string(start_row) + - ",\"start_col\":" + - std::to_string(start_col) + - ",\"confidence\":" + - std::to_string(confidence) + - ",\"resolver\":\"" + - jsonEscape(resolver.c_str()) + - "\",\"resolution_kind\":\"" + - jsonEscape(rkind.c_str()) + - "\",\"resolve_strategy\":\"" + - jsonEscape(rkind.c_str()) + "\"}"; - } - sqlite3_finalize(st); - } - } - result += "],\"total\":" + std::to_string(count) + - ",\"entity_id\":" + std::to_string(entity_id) + "}"; - return result; -} - -std::string QueryEngine::getCalleesByEntity(uint64_t project_id, - uint64_t entity_id) -{ - // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── - // Callees by explicit entity id: read outgoing Calls edges - // (relation where source_id = entity_id, type=1) via the - // (project_id, source_id) index. JSON shape mirrors the SQLite - // branch, including the trailing entity_id. - if (!store_ || !store_->handle()) { - return "{\"callees\":[],\"total\":0,\"error\":\"graph not ready " - "[module=query, method=getCalleesByEntity]\"}"; - } - sqlite3 *db = store_->handle(); - std::string result = "{\"callees\":["; - bool first = true; - int count = 0; - { - const char *sql = - "SELECT e.id, e.name, e.file_path, e.start_row, " - " e.start_col, r.confidence, r.resolver, " - " r.resolution_kind " - "FROM relation r JOIN entity e ON e.id = r.target_id " - "WHERE r.project_id=? AND r.type=1 " - "AND r.source_id=? " - "GROUP BY e.id ORDER BY e.id LIMIT 1000"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(st, 1, - static_cast(project_id)); - sqlite3_bind_int64(st, 2, - static_cast(entity_id)); - while (sqlite3_step(st) == SQLITE_ROW) { - int64_t node_id = sqlite3_column_int64(st, 0); - std::string name = - reinterpret_cast( - sqlite3_column_text(st, 1)) ? - reinterpret_cast( - sqlite3_column_text( - st, 1)) : - ""; - std::string file = - reinterpret_cast( - sqlite3_column_text(st, 2)) ? - reinterpret_cast( - sqlite3_column_text( - st, 2)) : - ""; - int start_row = sqlite3_column_int(st, 3); - int start_col = sqlite3_column_int(st, 4); - double confidence = - sqlite3_column_double(st, 5); - std::string resolver = - reinterpret_cast( - sqlite3_column_text(st, 6)) ? - reinterpret_cast( - sqlite3_column_text( - st, 6)) : - ""; - std::string rkind = - reinterpret_cast( - sqlite3_column_text(st, 7)) ? - reinterpret_cast( - sqlite3_column_text( - st, 7)) : - ""; - if (!first) - result += ","; - first = false; - ++count; - result += "{\"node_id\":" + - std::to_string(node_id) + - ",\"name\":\"" + - jsonEscape(name.c_str()) + - "\",\"file_path\":\"" + - jsonEscape(file.c_str()) + - "\",\"start_row\":" + - std::to_string(start_row) + - ",\"start_col\":" + - std::to_string(start_col) + - ",\"confidence\":" + - std::to_string(confidence) + - ",\"resolver\":\"" + - jsonEscape(resolver.c_str()) + - "\",\"resolution_kind\":\"" + - jsonEscape(rkind.c_str()) + - "\",\"resolve_strategy\":\"" + - jsonEscape(rkind.c_str()) + "\"}"; - } - sqlite3_finalize(st); - } - } - result += "],\"total\":" + std::to_string(count) + - ",\"entity_id\":" + std::to_string(entity_id) + "}"; - return result; -} - -std::string QueryEngine::getNeighbors(uint64_t project_id, uint64_t node_id, - int edge_type_filter, int radius) -{ - (void)radius; // reserved for future multi-hop - // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── - // Neighbors by node id: outgoing edges from relation (source_id = - // node_id, type = edge_type_filter) and incoming edges (target_id = - // node_id), each joined to entity metadata, tagged with direction - // "out"/"in" exactly like the SQLite branch. Uses the - // (project_id, source_id) / (project_id, target_id) indexes. - if (!store_ || !store_->handle()) { - return "{\"total\":0,\"neighbors\":[],\"error\":\"graph not " - "ready [module=query, method=getNeighbors]\"}"; - } - sqlite3 *db = store_->handle(); - std::ostringstream json; - json << "{\"neighbors\":["; - bool first = true; - int count = 0; - - auto emitNeighbors = [&](const std::string &dir_clause, - const char *direction) { - std::string sql = - "SELECT e.id, e.name, e.kind, e.file_path, r.type " - "FROM relation r JOIN entity e ON e.id = " - "r.target_id " - "WHERE r.project_id=? AND " + - dir_clause + " "; - if (edge_type_filter > 0) - sql += "AND r.type=" + - std::to_string(edge_type_filter) + " "; - sql += "ORDER BY e.id LIMIT 500"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db, sql.c_str(), -1, &st, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(st, 1, - static_cast(project_id)); - sqlite3_bind_int64(st, 2, - static_cast(node_id)); - while (sqlite3_step(st) == SQLITE_ROW) { - int64_t nid = sqlite3_column_int64(st, 0); - std::string name = - reinterpret_cast( - sqlite3_column_text(st, 1)) ? - reinterpret_cast( - sqlite3_column_text( - st, 1)) : - ""; - int ntype = sqlite3_column_int(st, 2); - std::string file = - reinterpret_cast( - sqlite3_column_text(st, 3)) ? - reinterpret_cast( - sqlite3_column_text( - st, 3)) : - ""; - int etype = sqlite3_column_int(st, 4); - if (!first) - json << ","; - first = false; - ++count; - json << "{\"neighbor_id\":" << nid - << ",\"name\":\"" - << jsonEscape(name.c_str()) - << "\",\"node_type\":" << ntype - << ",\"file_path\":\"" - << jsonEscape(file.c_str()) - << "\",\"edge_type\":" << etype - << ",\"direction\":\"" - << jsonEscape(direction) << "\"}"; - } - sqlite3_finalize(st); - } - }; - - // Outgoing: source_id = node_id → target is the neighbor. - emitNeighbors("r.source_id=?", "out"); - // Incoming: target_id = node_id → source is the neighbor. - { - std::string sql = - "SELECT e.id, e.name, e.kind, e.file_path, r.type " - "FROM relation r JOIN entity e ON e.id = r.source_id " - "WHERE r.project_id=? AND r.target_id=? "; - if (edge_type_filter > 0) - sql += "AND r.type=" + - std::to_string(edge_type_filter) + " "; - sql += "ORDER BY e.id LIMIT 500"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db, sql.c_str(), -1, &st, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(st, 1, - static_cast(project_id)); - sqlite3_bind_int64(st, 2, - static_cast(node_id)); - while (sqlite3_step(st) == SQLITE_ROW) { - int64_t nid = sqlite3_column_int64(st, 0); - std::string name = - reinterpret_cast( - sqlite3_column_text(st, 1)) ? - reinterpret_cast( - sqlite3_column_text( - st, 1)) : - ""; - int ntype = sqlite3_column_int(st, 2); - std::string file = - reinterpret_cast( - sqlite3_column_text(st, 3)) ? - reinterpret_cast( - sqlite3_column_text( - st, 3)) : - ""; - int etype = sqlite3_column_int(st, 4); - if (!first) - json << ","; - first = false; - ++count; - json << "{\"neighbor_id\":" << nid - << ",\"name\":\"" - << jsonEscape(name.c_str()) - << "\",\"node_type\":" << ntype - << ",\"file_path\":\"" - << jsonEscape(file.c_str()) - << "\",\"edge_type\":" << etype - << ",\"direction\":\"in\"}"; - } - sqlite3_finalize(st); - } - } - json << "],\"total\":" << count << "}"; - return json.str(); -} - -std::string QueryEngine::findShortestPath(uint64_t project_id, - uint64_t source_id, - uint64_t target_id) -{ - // Real iterative BFS over the in-memory call graph. - // - // Steps: - // 1. Load all CALLS|RELATES edges for the project from SQLite - // into an adjacency list (unordered_map>). - // 2. BFS from source_id to target_id with a visited set (encoded - // in the depth map) and a parent-pointer map for reconstruction. - // 3. Enforce kShortestPathMaxDepth so traversal stays bounded. - // 4. Reconstruct source→target path via parent pointers. - // - // All errors are reported with [module=query, method=findShortestPath] - // tags; nothing is silently swallowed. - // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── - // Real iterative BFS over the CSR forward adjacency table - // (store_->getCalleeIds, O(E) traversal, no full-table scans). The - // output JSON shape is identical to the SQLite branch: path array - // of {node_id}, found, approximation, note, hops. If source == target - // the path is a single node with 0 hops. - if (!store_ || !store_->handle()) { - return "{\"path\":[],\"found\":false,\"approximation\":" - "\"heuristic\",\"note\":\"" + - std::string(kShortestPathNote) + - "\",\"hops\":0,\"error\":\"graph not ready " - "[module=query, method=findShortestPath]\"}"; - } - std::ostringstream json; - if (source_id == target_id) { - json << "{\"path\":[{\"node_id\":" << source_id << "}]," - << "\"found\":true,\"approximation\":\"heuristic\"," - << "\"note\":\"" << kShortestPathNote << "\",\"hops\":0}"; - return json.str(); - } - // BFS with parent pointers and a visited/depth map; bounded by the - // same kShortestPathMaxDepth as the SQLite branch. - std::unordered_map parent; - std::unordered_map depth_map; - std::queue bfs; - parent[source_id] = source_id; - depth_map[source_id] = 0; - bfs.push(source_id); - bool found = false; - while (!bfs.empty()) { - uint64_t cur = bfs.front(); - bfs.pop(); - int cur_depth = depth_map[cur]; - if (cur_depth >= kShortestPathMaxDepth) - continue; - auto neighbors = store_->getCalleeIds(cur); - for (uint64_t nb : neighbors) { - if (parent.count(nb)) - continue; // already visited - parent[nb] = cur; - depth_map[nb] = cur_depth + 1; - if (nb == target_id) { - found = true; - break; - } - bfs.push(nb); - } - if (found) - break; - } - if (!found) { - // v0.2.5: no-path payload keeps the source node in the path array - // (path:[source]), matching the SQLite emitNotFound contract - // so callers can rely on a stable JSON shape across backends. - json << "{\"path\":[{\"node_id\":" << source_id << "}]," - << "\"found\":false," - << "\"approximation\":\"heuristic\"," - << "\"note\":\"" << kShortestPathNote << "\",\"hops\":0}"; - return json.str(); - } - // Reconstruct target → source via parent pointers, then reverse. - std::vector path; - uint64_t node = target_id; - while (true) { - path.push_back(node); - if (node == source_id) - break; - auto it = parent.find(node); - if (it == parent.end()) { - path.clear(); - found = false; - break; - } - node = it->second; - } - if (!found) { - // See no-path contract above (path:[source]). - json << "{\"path\":[{\"node_id\":" << source_id << "}]," - << "\"found\":false," - << "\"approximation\":\"heuristic\"," - << "\"note\":\"" << kShortestPathNote << "\",\"hops\":0}"; - return json.str(); - } - std::reverse(path.begin(), path.end()); - json << "{\"path\":["; - bool first = true; - for (uint64_t n : path) { - if (!first) - json << ","; - first = false; - json << "{\"node_id\":" << n << "}"; - } - size_t hops = path.size() > 0 ? path.size() - 1 : 0; - json << "],\"found\":true,\"approximation\":\"heuristic\"," - << "\"note\":\"" << kShortestPathNote << "\",\"hops\":" << hops - << "}"; - return json.str(); -} - -std::string QueryEngine::getSubgraph(uint64_t project_id, - uint64_t center_node_id, int radius, - const char *node_type_filter, - const char *edge_type_filter) -{ - (void)radius; // reserved for future multi-hop - // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── - // Subgraph via bidirectional BFS over the CSR adjacency tables - // (getCalleeIds + getCallerIds, O(E) per level). Emits nodes as - // {id, name, node_type, file_path, language} matching the SQLite - // branch. radius is honored (clamped to a sane bound); node/edge type - // filters are applied when given. - if (!store_ || !store_->handle()) { - return "{\"total\":0,\"nodes\":[],\"error\":\"graph not ready " - "[module=query, method=getSubgraph]\"}"; - } - int hops = radius > 0 ? radius : 1; - if (hops > 8) - hops = 8; // bounded traversal (matches SQLite budget) - // Parse node_type_filter (comma-separated kinds) for filtering. - std::unordered_set kind_filter; - if (node_type_filter && *node_type_filter) { - std::string fs(node_type_filter); - std::string token; - std::istringstream iss(fs); - while (std::getline(iss, token, ',')) { - while (!token.empty() && - std::isspace(static_cast( - token.front()))) - token.erase(token.begin()); - if (!token.empty()) { - try { - kind_filter.insert(std::stoi(token)); - } catch (...) { - // skip malformed token - } - } - } - } - - // BFS level by level, collecting visited nodes (undirected: follow - // both callers and callees). - std::unordered_map depth_map; - std::deque frontier{ center_node_id }; - depth_map[center_node_id] = 0; - int cur_depth = 0; - while (!frontier.empty() && cur_depth < hops) { - std::deque next; - for (uint64_t n : frontier) { - int d = depth_map[n]; - for (uint64_t nb : store_->getCalleeIds(n)) { - if (!depth_map.count(nb)) { - depth_map[nb] = d + 1; - next.push_back(nb); - } - } - for (uint64_t nb : store_->getCallerIds(n)) { - if (!depth_map.count(nb)) { - depth_map[nb] = d + 1; - next.push_back(nb); - } - } - } - frontier = std::move(next); - ++cur_depth; - } - - // Emit nodes (center first, then by depth) with entity metadata. - std::ostringstream json; - json << "{\"nodes\":["; - bool first = true; - int count = 0; - auto emitNode = [&](int64_t id, const std::string &name, int kind, - const std::string &file, const std::string &lang) { - if (!first) - json << ","; - first = false; - ++count; - json << "{\"id\":" << id << ",\"name\":\"" - << jsonEscape(name.c_str()) << "\",\"node_type\":" << kind - << ",\"file_path\":\"" << jsonEscape(file.c_str()) - << "\",\"language\":\"" << jsonEscape(lang.c_str()) - << "\"}"; - }; - // Deterministic order: center, then BFS discovery order (depth_map is - // insertion-ordered by BFS, which yields breadth-first order). - std::vector ordered; - for (auto &kv : depth_map) - ordered.push_back(kv.first); - for (uint64_t id : ordered) { - const char *sql = - "SELECT name, kind, file_path, language FROM entity " - "WHERE id=? AND project_id=?"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(store_->handle(), sql, -1, &st, - nullptr) != SQLITE_OK) - continue; - sqlite3_bind_int64(st, 1, static_cast(id)); - sqlite3_bind_int64(st, 2, static_cast(project_id)); - if (sqlite3_step(st) == SQLITE_ROW) { - std::string name = - reinterpret_cast( - sqlite3_column_text(st, 0)) ? - reinterpret_cast( - sqlite3_column_text(st, 0)) : - ""; - int kind = sqlite3_column_int(st, 1); - std::string file = - reinterpret_cast( - sqlite3_column_text(st, 2)) ? - reinterpret_cast( - sqlite3_column_text(st, 2)) : - ""; - std::string lang = - reinterpret_cast( - sqlite3_column_text(st, 3)) ? - reinterpret_cast( - sqlite3_column_text(st, 3)) : - ""; - if (kind_filter.empty() || kind_filter.count(kind)) - emitNode(static_cast(id), name, kind, - file, lang); - } - sqlite3_finalize(st); - } - json << "],\"total\":" << count << "}"; - return json.str(); -} - std::string QueryEngine::locateNode(uint64_t project_id, uint64_t node_id, int context_lines) { @@ -1301,45 +758,43 @@ std::string QueryEngine::locateByName(uint64_t project_id, const char *name) sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, name, -1, SQLITE_TRANSIENT); - std::ostringstream json; - json << "{\"locations\":["; + util::JsonWriter json; + json.beginObject(); + json.key("locations").beginArray(); int col_count = sqlite3_column_count(stmt); - bool first_row = true; int row_count = 0; while (sqlite3_step(stmt) == SQLITE_ROW) { - if (!first_row) - json << ","; - first_row = false; row_count++; - json << "{"; + json.beginObject(); for (int i = 0; i < col_count; i++) { - if (i > 0) - json << ","; const char *col_name = sqlite3_column_name(stmt, i); - // L2 fix: escape the column name so a name containing a quote - // or control char cannot produce invalid JSON. - json << "\"" << jsonEscape(col_name ? col_name : "") - << "\":"; + // The writer escapes keys and values, so a column name + // containing a quote or control char cannot produce + // invalid JSON. + json.key(col_name ? col_name : ""); int col_type = sqlite3_column_type(stmt, i); if (col_type == SQLITE_NULL) { - json << "null"; + json.nullValue(); } else if (col_type == SQLITE_INTEGER) { - json << sqlite3_column_int64(stmt, i); + json.value(sqlite3_column_int64(stmt, i)); } else { + // Non-integer, non-null columns are emitted as + // text, as before. const char *text = reinterpret_cast( sqlite3_column_text(stmt, i)); - json << "\"" << jsonEscape(text ? text : "") - << "\""; + json.value(text ? text : ""); } } - json << "}"; + json.endObject(); } - json << "],\"total\":" << row_count << "}"; + json.endArray(); + json.key("total").value(row_count); + json.endObject(); sqlite3_finalize(stmt); return json.str(); } @@ -1381,22 +836,45 @@ std::string QueryEngine::getGraphStats(uint64_t project_id) } } { - // Distinct file paths across all entities (matches the - // SQLite branch's "count DISTINCT n.file_path"). + // Files the indexer actually wrote, from the `files` table — what the + // field name promises. It used to count `COUNT(DISTINCT file_path) + // FROM entity`, i.e. only the files that produced at least one symbol, + // so a project of 1,579 indexed files reported 672 (goagent) and the + // number looked like "files indexed" while silently excluding + // header-only, re-export-only and comment-only files. The old value is + // still reported, as `files_with_symbols`. + const char *sql = "SELECT COUNT(*) FROM files"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == + SQLITE_OK) { + if (sqlite3_step(st) == SQLITE_ROW) + total_files = sqlite3_column_int64(st, 0); + sqlite3_finalize(st); + } + } + // Files that produced at least one entity: the value the old + // implementation reported as total_files, kept because it answers a + // different (also useful) question — how much of the corpus the graph + // actually covers. + int64_t files_with_symbols = 0; + { const char *sql = "SELECT COUNT(DISTINCT file_path) FROM entity"; sqlite3_stmt *st = nullptr; if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK) { if (sqlite3_step(st) == SQLITE_ROW) - total_files = sqlite3_column_int64(st, 0); + files_with_symbols = + sqlite3_column_int64(st, 0); sqlite3_finalize(st); } } + std::ostringstream json; json << "{\"total_nodes\":" << total_nodes << ",\"total_edges\":" << total_edges - << ",\"total_files\":" << total_files << "}"; + << ",\"total_files\":" << total_files + << ",\"files_with_symbols\":" << files_with_symbols << "}"; return json.str(); } @@ -1420,12 +898,90 @@ std::string QueryEngine::explainSymbol(uint64_t project_id, std::string callees_json = getCallees(project_id, name.c_str()); // 4. Combine into a single response - std::string json = "{"; - json += "\"symbol\":\"" + jsonEscape(name.c_str()) + "\","; - json += "\"definition\":" + def_json + ","; - json += "\"callers\":" + callers_json + ","; - json += "\"callees\":" + callees_json + "}"; - return json; + util::JsonWriter json; + json.beginObject(); + json.key("symbol").value(name); + // The three sub-documents are already serialized by the query + // methods above; raw() embeds them without re-escaping. + json.key("definition").raw(def_json); + json.key("callers").raw(callers_json); + json.key("callees").raw(callees_json); + json.endObject(); + return json.str(); +} + +// ─── Bare-name ambiguity probe ───────────────────────────────── +// +// Shared by the name-based trace/lookup paths (traceCallChain, +// trace_path). Returns "" when the name is unambiguous; otherwise a +// `{"ambiguous":true,"candidates":[…]}` fragment the caller merges into +// its response. Mirrors the getCallers ambiguity contract so a client +// sees one shape for "which entity do you mean?" no matter which tool +// asked (T5 finding #9). + +std::string bareNameCandidates(sqlite3 *db, uint64_t project_id, + const char *name) +{ + if (!db || !name || !*name) + return std::string(); + // Collect every matching entity id first — LIMIT 1 here is exactly + // the silent-first-match behaviour this probe exists to prevent. + std::vector ids; + { + const char *sql = "SELECT id FROM entity WHERE project_id=? " + "AND (name=? OR qualified_name=?) " + "ORDER BY id"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) != SQLITE_OK) + return std::string(); + sqlite3_bind_int64(st, 1, static_cast(project_id)); + sqlite3_bind_text(st, 2, name, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(st, 3, name, -1, SQLITE_TRANSIENT); + while (sqlite3_step(st) == SQLITE_ROW) + ids.push_back(sqlite3_column_int64(st, 0)); + sqlite3_finalize(st); + } + if (ids.size() < 2) + return std::string(); + + std::string cands; + bool first = true; + for (int64_t id : ids) { + std::string nm, fp; + int sr = 0, sc = 0; + const char *q = "SELECT name, file_path, start_row, start_col " + "FROM entity WHERE id=?"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, q, -1, &st, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(st, 1, id); + if (sqlite3_step(st) == SQLITE_ROW) { + const unsigned char *n = + sqlite3_column_text(st, 0); + const unsigned char *f = + sqlite3_column_text(st, 1); + nm = n ? reinterpret_cast(n) : ""; + fp = f ? reinterpret_cast(f) : ""; + sr = sqlite3_column_int(st, 2); + sc = sqlite3_column_int(st, 3); + } + sqlite3_finalize(st); + } + if (!first) + cands += ","; + first = false; + util::JsonWriter el; + el.beginObject(); + el.key("graph_node_id").value(id); + el.key("name").value(nm); + el.key("file_path").value(fp); + el.key("start_row").value(sr); + el.key("start_col").value(sc); + el.endObject(); + cands += el.str(); + } + return "{\"ambiguous\":true,\"candidates\":[" + cands + + "],\"hint\":\"" + util::jsonEscapeString(kAmbiguousNameHint) + + "\"}"; } } // namespace query diff --git a/engine/src/query/query_engine.h b/engine/src/query/query_engine.h index 65ec1f3..ab83bc1 100644 --- a/engine/src/query/query_engine.h +++ b/engine/src/query/query_engine.h @@ -12,12 +12,23 @@ namespace query { // JSON string escaping (shared by query_engine.cpp and query_analysis.cpp) -std::string jsonEscape(const char *s); // Execute a SQL query and return results as JSON array std::string queryToJson(sqlite3 *db, const char *sql, const char *result_key = "results"); +// Bare-name ambiguity probe shared by the name-based trace/lookup paths. +// Returns "" when `name` matches exactly one entity (or none) — the caller +// proceeds normally. When more than one entity shares the name it returns a +// complete JSON fragment in the shape +// {"ambiguous":true,"candidates":[{"graph_node_id":N,"name":"…", +// "file_path":"…","start_row":N,"start_col":N}, …]} +// so the caller can merge it into the response instead of silently tracing +// the first match (T5 finding #9). Query failure returns "" (fail-open, +// matching the getCallers ambiguity helper). +std::string bareNameCandidates(sqlite3 *db, uint64_t project_id, + const char *name); + class QueryEngine { public: explicit QueryEngine(store::GraphStore *store); @@ -124,11 +135,22 @@ class QueryEngine { // ── Community Detection ──────────────────────────────────── - // Run label-propagation community detection on the code graph. - // Returns JSON with communities, their members, and inter-community edges. - // @param max_members Max members per community in output (0 = all). - // @param max_communities Max communities to return (0 = all). - // @param include_members Include member list in output (default false). + // Run deterministic label-propagation community detection over the + // CALLS graph (relation type=1, treated as undirected; self-loops and + // isolated nodes are excluded). Implemented in query_communities.cpp. + // Returns JSON: + // {"communities":[{"id":,"label":, + // "member_count":N[,"members":[...]]}], + // "total_communities":N,"returned_communities":N, + // "inter_community_edges":N,"truncated":bool, + // "approximation":"heuristic","note":"..."} + // Summary-first: "members" is only present when include_members is set. + // @param max_members Members per community when include_members is set. + // Non-positive means "use the default" (10), NOT "unlimited"; + // clamped to 200. + // @param max_communities Communities to return. Non-positive means + // "use the default" (20), NOT "unlimited"; clamped to 500. + // @param include_members Include the member list in the output. std::string getCommunities(uint64_t project_id, int max_members = 10, int max_communities = 20, bool include_members = false); diff --git a/engine/src/query/query_engine_traverse.cpp b/engine/src/query/query_engine_traverse.cpp new file mode 100644 index 0000000..ff1667c --- /dev/null +++ b/engine/src/query/query_engine_traverse.cpp @@ -0,0 +1,639 @@ +// query_engine_traverse.cpp — entity-precise and traversal queries. +// +// Split out of query_engine.cpp (see plan/rules/code_rules.md 1000-line +// rule). These are the Step-7 entity-precise caller/callee APIs plus the +// neighbour / shortest-path / subgraph walks. All of them resolve an +// entity to (name, file_path, start_row) before querying so homonyms +// (many `__init__` across classes) no longer aggregate into one result +// set — which is the whole reason they live together rather than next to +// the bare-name APIs. + +#include "util/json_writer.h" +#include "query_engine.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace query +{ + +// chains while keeping query latency bounded. +static constexpr int kShortestPathMaxDepth = 10; + +// Standard note appended to findShortestPath results explaining the +// heuristic nature of the call graph (name-matched, no virtual dispatch). +static const char *const kShortestPathNote = + "Call graph edges are resolved by name matching; indirect calls " + "(virtual/pointer) may be missing."; + +// ── Step 7 (plan §7.2): entity-precise query APIs ──────────────────── +// +// These methods resolve an entity ID to (name, file_path, start_row) in +// SQLite, then build a SQLite Cypher query that filters by all three +// fields. This eliminates the homonym aggregation problem: multiple +// entities named "__init__" in different classes/files are no longer +// merged into a single result set. +// +// The old bare-name APIs (getCallers/getCallees) are retained for +// backward compatibility but now detect ambiguity: when multiple +// entities match the bare name, they return ambiguous=true with a +// candidate list instead of silently aggregating. + +std::string QueryEngine::getCallersByEntity(uint64_t project_id, + uint64_t entity_id) +{ + // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── + // Callers by explicit entity id: read incoming Calls edges + // (relation where target_id = entity_id, type=1) via the + // (project_id, target_id) index. JSON shape mirrors the SQLite + // branch, including the trailing entity_id. + if (!store_ || !store_->handle()) { + return "{\"callers\":[],\"total\":0,\"error\":\"graph not ready " + "[module=query, method=getCallersByEntity]\"}"; + } + sqlite3 *db = store_->handle(); + std::string result = "{\"callers\":["; + bool first = true; + int count = 0; + { + const char *sql = + "SELECT e.id, e.name, e.file_path, e.start_row, " + " e.start_col, r.confidence, r.resolver, " + " r.resolution_kind " + "FROM relation r JOIN entity e ON e.id = r.source_id " + "WHERE r.project_id=? AND r.type=1 " + "AND r.target_id=? " + "GROUP BY e.id ORDER BY e.id LIMIT 1000"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + sqlite3_bind_int64(st, 2, + static_cast(entity_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + int64_t node_id = sqlite3_column_int64(st, 0); + std::string name = + reinterpret_cast( + sqlite3_column_text(st, 1)) ? + reinterpret_cast( + sqlite3_column_text( + st, 1)) : + ""; + std::string file = + reinterpret_cast( + sqlite3_column_text(st, 2)) ? + reinterpret_cast( + sqlite3_column_text( + st, 2)) : + ""; + int start_row = sqlite3_column_int(st, 3); + int start_col = sqlite3_column_int(st, 4); + double confidence = + sqlite3_column_double(st, 5); + std::string resolver = + reinterpret_cast( + sqlite3_column_text(st, 6)) ? + reinterpret_cast( + sqlite3_column_text( + st, 6)) : + ""; + std::string rkind = + reinterpret_cast( + sqlite3_column_text(st, 7)) ? + reinterpret_cast( + sqlite3_column_text( + st, 7)) : + ""; + if (!first) + result += ","; + first = false; + ++count; + util::JsonWriter el; + el.beginObject(); + el.key("node_id").value(node_id); + el.key("name").value(name); + el.key("file_path").value(file); + el.key("start_row").value(start_row); + el.key("start_col").value(start_col); + // raw(): std::to_string keeps the historic "%f" + // formatting for this field. + el.key("confidence") + .raw(std::to_string(confidence)); + el.key("resolver").value(resolver); + el.key("resolution_kind").value(rkind); + el.key("resolve_strategy").value(rkind); + el.endObject(); + result += el.str(); + } + sqlite3_finalize(st); + } + } + result += "],\"total\":" + std::to_string(count) + + ",\"entity_id\":" + std::to_string(entity_id) + "}"; + return result; +} + +std::string QueryEngine::getCalleesByEntity(uint64_t project_id, + uint64_t entity_id) +{ + // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── + // Callees by explicit entity id: read outgoing Calls edges + // (relation where source_id = entity_id, type=1) via the + // (project_id, source_id) index. JSON shape mirrors the SQLite + // branch, including the trailing entity_id. + if (!store_ || !store_->handle()) { + return "{\"callees\":[],\"total\":0,\"error\":\"graph not ready " + "[module=query, method=getCalleesByEntity]\"}"; + } + sqlite3 *db = store_->handle(); + std::string result = "{\"callees\":["; + bool first = true; + int count = 0; + { + const char *sql = + "SELECT e.id, e.name, e.file_path, e.start_row, " + " e.start_col, r.confidence, r.resolver, " + " r.resolution_kind " + "FROM relation r JOIN entity e ON e.id = r.target_id " + "WHERE r.project_id=? AND r.type=1 " + "AND r.source_id=? " + "GROUP BY e.id ORDER BY e.id LIMIT 1000"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + sqlite3_bind_int64(st, 2, + static_cast(entity_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + int64_t node_id = sqlite3_column_int64(st, 0); + std::string name = + reinterpret_cast( + sqlite3_column_text(st, 1)) ? + reinterpret_cast( + sqlite3_column_text( + st, 1)) : + ""; + std::string file = + reinterpret_cast( + sqlite3_column_text(st, 2)) ? + reinterpret_cast( + sqlite3_column_text( + st, 2)) : + ""; + int start_row = sqlite3_column_int(st, 3); + int start_col = sqlite3_column_int(st, 4); + double confidence = + sqlite3_column_double(st, 5); + std::string resolver = + reinterpret_cast( + sqlite3_column_text(st, 6)) ? + reinterpret_cast( + sqlite3_column_text( + st, 6)) : + ""; + std::string rkind = + reinterpret_cast( + sqlite3_column_text(st, 7)) ? + reinterpret_cast( + sqlite3_column_text( + st, 7)) : + ""; + if (!first) + result += ","; + first = false; + ++count; + util::JsonWriter el; + el.beginObject(); + el.key("node_id").value(node_id); + el.key("name").value(name); + el.key("file_path").value(file); + el.key("start_row").value(start_row); + el.key("start_col").value(start_col); + // raw(): std::to_string keeps the historic "%f" + // formatting for this field. + el.key("confidence") + .raw(std::to_string(confidence)); + el.key("resolver").value(resolver); + el.key("resolution_kind").value(rkind); + el.key("resolve_strategy").value(rkind); + el.endObject(); + result += el.str(); + } + sqlite3_finalize(st); + } + } + result += "],\"total\":" + std::to_string(count) + + ",\"entity_id\":" + std::to_string(entity_id) + "}"; + return result; +} + +std::string QueryEngine::getNeighbors(uint64_t project_id, uint64_t node_id, + int edge_type_filter, int radius) +{ + // Only direct neighbors are returned: multi-hop is not implemented (the + // tool schema says "reserved, currently 1"). The response says so too, so a + // caller that asked for a deeper walk is told what it got instead of + // silently receiving a shallower answer. + (void)radius; + // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── + // Neighbors by node id: outgoing edges from relation (source_id = + // node_id, type = edge_type_filter) and incoming edges (target_id = + // node_id), each joined to entity metadata, tagged with direction + // "out"/"in" exactly like the SQLite branch. Uses the + // (project_id, source_id) / (project_id, target_id) indexes. + if (!store_ || !store_->handle()) { + return "{\"total\":0,\"neighbors\":[],\"error\":\"graph not " + "ready [module=query, method=getNeighbors]\"}"; + } + sqlite3 *db = store_->handle(); + // Report the depth actually walked, so a caller that asked for more than one + // hop can see it got direct neighbors only (see the note above). + std::ostringstream json; + json << "{\"radius_applied\":1"; + if (radius > 1) { + json << ",\"radius_requested\":" << radius + << ",\"note\":\"multi-hop is not implemented; only direct " + "neighbors are returned\""; + } + json << ",\"neighbors\":["; + bool first = true; + int count = 0; + + auto emitNeighbors = [&](const std::string &dir_clause, + const char *direction) { + std::string sql = + "SELECT e.id, e.name, e.kind, e.file_path, r.type " + "FROM relation r JOIN entity e ON e.id = " + "r.target_id " + "WHERE r.project_id=? AND " + + dir_clause + " "; + if (edge_type_filter > 0) + sql += "AND r.type=" + + std::to_string(edge_type_filter) + " "; + sql += "ORDER BY e.id LIMIT 500"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql.c_str(), -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + sqlite3_bind_int64(st, 2, + static_cast(node_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + int64_t nid = sqlite3_column_int64(st, 0); + std::string name = + reinterpret_cast( + sqlite3_column_text(st, 1)) ? + reinterpret_cast( + sqlite3_column_text( + st, 1)) : + ""; + int ntype = sqlite3_column_int(st, 2); + std::string file = + reinterpret_cast( + sqlite3_column_text(st, 3)) ? + reinterpret_cast( + sqlite3_column_text( + st, 3)) : + ""; + int etype = sqlite3_column_int(st, 4); + if (!first) + json << ","; + first = false; + ++count; + util::JsonWriter el; + el.beginObject(); + el.key("neighbor_id").value(nid); + el.key("name").value(name); + el.key("node_type").value(ntype); + el.key("file_path").value(file); + el.key("edge_type").value(etype); + el.key("direction").value(direction); + el.endObject(); + json << el.str(); + } + sqlite3_finalize(st); + } + }; + + // Outgoing: source_id = node_id → target is the neighbor. + emitNeighbors("r.source_id=?", "out"); + // Incoming: target_id = node_id → source is the neighbor. + { + std::string sql = + "SELECT e.id, e.name, e.kind, e.file_path, r.type " + "FROM relation r JOIN entity e ON e.id = r.source_id " + "WHERE r.project_id=? AND r.target_id=? "; + if (edge_type_filter > 0) + sql += "AND r.type=" + + std::to_string(edge_type_filter) + " "; + sql += "ORDER BY e.id LIMIT 500"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db, sql.c_str(), -1, &st, nullptr) == + SQLITE_OK) { + sqlite3_bind_int64(st, 1, + static_cast(project_id)); + sqlite3_bind_int64(st, 2, + static_cast(node_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + int64_t nid = sqlite3_column_int64(st, 0); + std::string name = + reinterpret_cast( + sqlite3_column_text(st, 1)) ? + reinterpret_cast( + sqlite3_column_text( + st, 1)) : + ""; + int ntype = sqlite3_column_int(st, 2); + std::string file = + reinterpret_cast( + sqlite3_column_text(st, 3)) ? + reinterpret_cast( + sqlite3_column_text( + st, 3)) : + ""; + int etype = sqlite3_column_int(st, 4); + if (!first) + json << ","; + first = false; + ++count; + util::JsonWriter el; + el.beginObject(); + el.key("neighbor_id").value(nid); + el.key("name").value(name); + el.key("node_type").value(ntype); + el.key("file_path").value(file); + el.key("edge_type").value(etype); + el.key("direction").value("in"); + el.endObject(); + json << el.str(); + } + sqlite3_finalize(st); + } + } + json << "],\"total\":" << count << "}"; + return json.str(); +} + +std::string QueryEngine::findShortestPath(uint64_t project_id, + uint64_t source_id, + uint64_t target_id) +{ + // Real iterative BFS over the in-memory call graph. + // + // Steps: + // 1. Load all CALLS|RELATES edges for the project from SQLite + // into an adjacency list (unordered_map>). + // 2. BFS from source_id to target_id with a visited set (encoded + // in the depth map) and a parent-pointer map for reconstruction. + // 3. Enforce kShortestPathMaxDepth so traversal stays bounded. + // 4. Reconstruct source→target path via parent pointers. + // + // All errors are reported with [module=query, method=findShortestPath] + // tags; nothing is silently swallowed. + // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── + // Real iterative BFS over the CSR forward adjacency table + // (store_->getCalleeIds, O(E) traversal, no full-table scans). The + // output JSON shape is identical to the SQLite branch: path array + // of {node_id}, found, approximation, note, hops. If source == target + // the path is a single node with 0 hops. + if (!store_ || !store_->handle()) { + return "{\"path\":[],\"found\":false,\"approximation\":" + "\"heuristic\",\"note\":\"" + + std::string(kShortestPathNote) + + "\",\"hops\":0,\"error\":\"graph not ready " + "[module=query, method=findShortestPath]\"}"; + } + std::ostringstream json; + if (source_id == target_id) { + json << "{\"path\":[{\"node_id\":" << source_id << "}]," + << "\"found\":true,\"approximation\":\"heuristic\"," + << "\"note\":\"" << kShortestPathNote << "\",\"hops\":0}"; + return json.str(); + } + // BFS with parent pointers and a visited/depth map; bounded by the + // same kShortestPathMaxDepth as the SQLite branch. + std::unordered_map parent; + std::unordered_map depth_map; + std::queue bfs; + parent[source_id] = source_id; + depth_map[source_id] = 0; + bfs.push(source_id); + bool found = false; + while (!bfs.empty()) { + uint64_t cur = bfs.front(); + bfs.pop(); + int cur_depth = depth_map[cur]; + if (cur_depth >= kShortestPathMaxDepth) + continue; + auto neighbors = store_->getCalleeIds(cur); + for (uint64_t nb : neighbors) { + if (parent.count(nb)) + continue; // already visited + parent[nb] = cur; + depth_map[nb] = cur_depth + 1; + if (nb == target_id) { + found = true; + break; + } + bfs.push(nb); + } + if (found) + break; + } + if (!found) { + // v0.2.5: no-path payload keeps the source node in the path array + // (path:[source]), matching the SQLite emitNotFound contract + // so callers can rely on a stable JSON shape across backends. + json << "{\"path\":[{\"node_id\":" << source_id << "}]," + << "\"found\":false," + << "\"approximation\":\"heuristic\"," + << "\"note\":\"" << kShortestPathNote << "\",\"hops\":0}"; + return json.str(); + } + // Reconstruct target → source via parent pointers, then reverse. + std::vector path; + uint64_t node = target_id; + while (true) { + path.push_back(node); + if (node == source_id) + break; + auto it = parent.find(node); + if (it == parent.end()) { + path.clear(); + found = false; + break; + } + node = it->second; + } + if (!found) { + // See no-path contract above (path:[source]). + json << "{\"path\":[{\"node_id\":" << source_id << "}]," + << "\"found\":false," + << "\"approximation\":\"heuristic\"," + << "\"note\":\"" << kShortestPathNote << "\",\"hops\":0}"; + return json.str(); + } + std::reverse(path.begin(), path.end()); + json << "{\"path\":["; + bool first = true; + for (uint64_t n : path) { + if (!first) + json << ","; + first = false; + json << "{\"node_id\":" << n << "}"; + } + size_t hops = path.size() > 0 ? path.size() - 1 : 0; + json << "],\"found\":true,\"approximation\":\"heuristic\"," + << "\"note\":\"" << kShortestPathNote << "\",\"hops\":" << hops + << "}"; + return json.str(); +} + +std::string QueryEngine::getSubgraph(uint64_t project_id, + uint64_t center_node_id, int radius, + const char *node_type_filter, + const char *edge_type_filter) +{ + // NOTE: no `(void)radius;` here — this function honors it (clamped just + // below into `hops`). The stale cast that used to sit here read as "radius + // is ignored", contradicting the code three lines down. + // ── v0.2.5: SQLite graph-query backend (Windows / SQLite-only) ── + // Subgraph via bidirectional BFS over the CSR adjacency tables + // (getCalleeIds + getCallerIds, O(E) per level). Emits nodes as + // {id, name, node_type, file_path, language} matching the SQLite + // branch. radius is honored (clamped to a sane bound); node/edge type + // filters are applied when given. + if (!store_ || !store_->handle()) { + return "{\"total\":0,\"nodes\":[],\"error\":\"graph not ready " + "[module=query, method=getSubgraph]\"}"; + } + int hops = radius > 0 ? radius : 1; + if (hops > 8) + hops = 8; // bounded traversal (matches SQLite budget) + // Parse node_type_filter (comma-separated kinds) for filtering. + std::unordered_set kind_filter; + if (node_type_filter && *node_type_filter) { + std::string fs(node_type_filter); + std::string token; + std::istringstream iss(fs); + while (std::getline(iss, token, ',')) { + while (!token.empty() && + std::isspace(static_cast( + token.front()))) + token.erase(token.begin()); + if (!token.empty()) { + try { + kind_filter.insert(std::stoi(token)); + } catch (...) { + // skip malformed token + } + } + } + } + + // BFS level by level, collecting visited nodes (undirected: follow + // both callers and callees). + std::unordered_map depth_map; + std::deque frontier{ center_node_id }; + depth_map[center_node_id] = 0; + int cur_depth = 0; + while (!frontier.empty() && cur_depth < hops) { + std::deque next; + for (uint64_t n : frontier) { + int d = depth_map[n]; + for (uint64_t nb : store_->getCalleeIds(n)) { + if (!depth_map.count(nb)) { + depth_map[nb] = d + 1; + next.push_back(nb); + } + } + for (uint64_t nb : store_->getCallerIds(n)) { + if (!depth_map.count(nb)) { + depth_map[nb] = d + 1; + next.push_back(nb); + } + } + } + frontier = std::move(next); + ++cur_depth; + } + + // Emit nodes (center first, then by depth) with entity metadata. + std::ostringstream json; + json << "{\"nodes\":["; + bool first = true; + int count = 0; + auto emitNode = [&](int64_t id, const std::string &name, int kind, + const std::string &file, const std::string &lang) { + if (!first) + json << ","; + first = false; + ++count; + util::JsonWriter el; + el.beginObject(); + el.key("id").value(id); + el.key("name").value(name); + el.key("node_type").value(kind); + el.key("file_path").value(file); + el.key("language").value(lang); + el.endObject(); + json << el.str(); + }; + // Deterministic order: center, then BFS discovery order (depth_map is + // insertion-ordered by BFS, which yields breadth-first order). + std::vector ordered; + for (auto &kv : depth_map) + ordered.push_back(kv.first); + for (uint64_t id : ordered) { + const char *sql = + "SELECT name, kind, file_path, language FROM entity " + "WHERE id=? AND project_id=?"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(store_->handle(), sql, -1, &st, + nullptr) != SQLITE_OK) + continue; + sqlite3_bind_int64(st, 1, static_cast(id)); + sqlite3_bind_int64(st, 2, static_cast(project_id)); + if (sqlite3_step(st) == SQLITE_ROW) { + std::string name = + reinterpret_cast( + sqlite3_column_text(st, 0)) ? + reinterpret_cast( + sqlite3_column_text(st, 0)) : + ""; + int kind = sqlite3_column_int(st, 1); + std::string file = + reinterpret_cast( + sqlite3_column_text(st, 2)) ? + reinterpret_cast( + sqlite3_column_text(st, 2)) : + ""; + std::string lang = + reinterpret_cast( + sqlite3_column_text(st, 3)) ? + reinterpret_cast( + sqlite3_column_text(st, 3)) : + ""; + if (kind_filter.empty() || kind_filter.count(kind)) + emitNode(static_cast(id), name, kind, + file, lang); + } + sqlite3_finalize(st); + } + json << "],\"total\":" << count << "}"; + return json.str(); +} + +} // namespace query diff --git a/engine/src/resolver/factors.cpp b/engine/src/resolver/factors.cpp index 5d7ea7c..79030ee 100644 --- a/engine/src/resolver/factors.cpp +++ b/engine/src/resolver/factors.cpp @@ -1,143 +1,47 @@ #include "factors.h" #include #include +#include +#include #include #include namespace resolver { -namespace +const std::string &disabledFactor() { -/// Return true if any target_path in `targets` matches the LIKE pattern -/// `"%%"` (semantically identical to the original SQL -/// `SELECT COUNT(*) ... WHERE target_path LIKE '%module_name%' > 0`). -bool anyImportMatches(const std::vector &targets, - const std::string &module_name) -{ - std::string pattern = "%" + module_name + "%"; - for (const auto &target : targets) { - if (sqliteLikeMatch(pattern, target)) - return true; - } - return false; -} - -/// Extract the module-name token used for import matching. Mirrors the -/// original factors.cpp logic: strip the file name (last '/'), then take -/// the last directory component. For "a/b/c.go" this yields "b"; for a -/// bare "c.go" it yields "c.go"; for "/c.go" it yields "" (which, like -/// the original SQL `LIKE '%%'`, matches any imported target_path). -std::string moduleTokenFromPath(const std::string &file_path) -{ - std::string module_path = file_path; - size_t slash = module_path.rfind('/'); - if (slash != std::string::npos) - module_path = module_path.substr(0, slash); - size_t prev_slash = module_path.rfind('/'); - if (prev_slash != std::string::npos) - return module_path.substr(prev_slash + 1); - return module_path; -} -} // namespace - -double factorImportMatch( - const std::unordered_map> - &import_index, - const std::string &caller_file, const std::string &candidate_file, - const std::string &candidate_name) -{ - (void)candidate_name; // unused — kept for caller compatibility - - // Same-module calls don't need an import statement — score 1.0. - // (Identical to the original early return: pure string comparison, - // no index lookup, so it is also the fast path for same-directory - // candidates.) - { - size_t c_slash = caller_file.rfind('/'); - size_t t_slash = candidate_file.rfind('/'); - if (c_slash != std::string::npos && - t_slash != std::string::npos) { - std::string caller_dir = caller_file.substr(0, c_slash); - std::string cand_dir = - candidate_file.substr(0, t_slash); - if (caller_dir == cand_dir) - return 1.0; + static const std::string value = []() -> std::string { + const char *raw = getenv("CODESCOPE_RESOLVER_DISABLE_FACTOR"); + if (raw == nullptr || *raw == '\0') + return std::string(); + const std::string name(raw); + // Validated here, once, rather than in the scoring loop: a typo must + // not look like "this factor changes nothing". + static const char *const kFactors[] = { + "ModuleMatch", "ImportMatch", + "ImportModuleMatch", "NamespaceMatch", + "SignatureMatch", "DistanceMatch", + "ConstructorMatch", "ReceiverMatch", + "CommonNamePenalty", "CallKindMatch", + "DefinitionMatch", + }; + for (const char *factor : kFactors) { + if (name == factor) + return name; } - } - - // Forward check: does the caller's file import the candidate's - // module? Replaces `SELECT COUNT(*) FROM import WHERE file_path=? - // AND target_path LIKE '%candidate_module%'` with an in-memory - // lookup over the pre-loaded import_index. - std::string candidate_module = moduleTokenFromPath(candidate_file); - double result = 0.0; - auto fwd_it = import_index.find(caller_file); - if (fwd_it != import_index.end() && - anyImportMatches(fwd_it->second, candidate_module)) - result = 1.0; - - // Reverse check: does the candidate's file import the caller's - // module? Only runs when the forward check found nothing, matching - // the original `result == 0.0` guard. - if (result == 0.0) { - std::string caller_module = moduleTokenFromPath(caller_file); - auto rev_it = import_index.find(candidate_file); - if (rev_it != import_index.end() && - anyImportMatches(rev_it->second, caller_module)) - result = 1.0; - } - - return result; -} - -double factorNamespaceMatch(const std::string &caller_file, - const std::string &candidate_file) -{ - // Extract directory paths and compare. - size_t c_slash = caller_file.rfind('/'); - size_t t_slash = candidate_file.rfind('/'); - if (c_slash == std::string::npos || t_slash == std::string::npos) - return 0.0; - - std::string caller_dir = caller_file.substr(0, c_slash); - std::string cand_dir = candidate_file.substr(0, t_slash); - - if (caller_dir == cand_dir) - return kScoreExactMatch; // Same package - - // Check if they share a common parent directory - // (sibling packages within the same module) - size_t c_parent = caller_dir.rfind('/'); - size_t t_parent = cand_dir.rfind('/'); - if (c_parent == std::string::npos || t_parent == std::string::npos) - return 0.0; - - std::string caller_parent = caller_dir.substr(0, c_parent); - std::string cand_parent = cand_dir.substr(0, t_parent); - if (caller_parent == cand_parent && caller_parent.length() > 0) - return kScoreSiblingModule; // Sibling packages - - return 0.0; -} - -double factorDistanceMatch(const std::string &caller_file, - const std::string &candidate_file) -{ - if (caller_file == candidate_file) - return kScoreExactMatch; // Same file - - size_t c_slash = caller_file.rfind('/'); - size_t t_slash = candidate_file.rfind('/'); - if (c_slash == std::string::npos || t_slash == std::string::npos) - return 0.0; - - std::string caller_dir = caller_file.substr(0, c_slash); - std::string cand_dir = candidate_file.substr(0, t_slash); - if (caller_dir == cand_dir) - return kScoreSameDirectory; // Same directory - - return 0.0; + fprintf(stderr, + "[module=resolver, method=disabledFactor] " + "CODESCOPE_RESOLVER_DISABLE_FACTOR=\"%s\" is not a factor " + "name, so no factor is disabled. Valid names: ModuleMatch, " + "ImportMatch, ImportModuleMatch, NamespaceMatch, " + "SignatureMatch, DistanceMatch, ConstructorMatch, " + "ReceiverMatch, CommonNamePenalty, CallKindMatch, " + "DefinitionMatch\n", + raw); + return std::string(); + }(); + return value; } double factorSignatureMatch(int caller_arity, int candidate_arity) @@ -164,124 +68,37 @@ double factorConstructorMatch(const std::string &ref_name, return 0.0; } -double factorReceiverMatch(const std::string &ref_name, - const std::string &caller_file, - const std::string &candidate_name, - const std::string &candidate_file) -{ - // Receiver match: for method calls like a.method(), - // check if the candidate is a method of the caller's receiver type. - // This is a simplified check: if caller and candidate share - // the same package prefix, boost the score. - size_t c_slash = caller_file.rfind('/'); - size_t t_slash = candidate_file.rfind('/'); - if (c_slash == std::string::npos || t_slash == std::string::npos) - return 0.0; - - std::string caller_pkg = caller_file.substr(0, c_slash); - std::string cand_pkg = candidate_file.substr(0, t_slash); - - // Same package: likely receiver match - if (caller_pkg == cand_pkg) - return kScoreExactMatch; - - // Different package: check if candidate is in a sub-package - if (cand_pkg.find(caller_pkg) == 0) - return kScorePartialMatch; - - return 0.0; -} - -// Step 5 (plan §5.3): receiver type evidence factor. -// -// Replaces the directory-heuristic factorReceiverMatch with actual -// type-based matching. The key insight: when a reference carries a -// known receiver_type (e.g. "Box" from `let b: Box = ...; b.draw()`), -// the candidate's qualified_name should contain that type as a prefix -// (e.g. "Box::draw", "Box.draw", "Box::draw"). This is strong -// structural evidence — far more reliable than "same directory". -// -// When receiver_type is empty (dynamic/unknown receiver), we return -// 0.5 (neutral) rather than 0.0. This is critical: returning 0.0 would -// penalize ALL candidates equally (no differentiation), while 0.5 -// ensures the receiver factor does not distort the ranking when we -// lack type evidence. The decision then falls to other factors -// (Import, Namespace, Signature) as before. -double factorReceiverTypeMatch(const std::string &receiver_type, - const std::string &candidate_qname, - const std::string &candidate_name, - const std::string &candidate_file) -{ - // No receiver type evidence → neutral, do not fabricate evidence. - if (receiver_type.empty()) - return 0.5; - - // Strong match: qualified_name contains the receiver type as a - // prefix. Covers "Box::draw", "Box.draw", "MyClass::method", etc. - if (!candidate_qname.empty()) { - // Check "Type::method" and "Type.method" patterns. - std::string prefix1 = receiver_type + "::"; - std::string prefix2 = receiver_type + "."; - if (candidate_qname.find(prefix1) == 0 || - candidate_qname.find(prefix2) == 0) - return kScoreExactMatch; - // Also check if receiver_type appears as a component anywhere - // in the qualified_name (e.g. "module::Box::draw"). - if (candidate_qname.find(prefix1) != std::string::npos || - candidate_qname.find(prefix2) != std::string::npos) - return kScorePartialMatch; - } - - // Weak fallback: if the candidate's file path contains the receiver - // type name (e.g. file "box.go" containing methods of Box), give a - // partial score. This is less reliable than qualified_name but - // better than nothing for languages that don't populate - // qualified_name (e.g. Go, where methods are defined as - // `func (b Box) draw()` and qualified_name may be empty). - size_t slash = candidate_file.rfind('/'); - std::string fname = (slash != std::string::npos) ? - candidate_file.substr(slash + 1) : - candidate_file; - // Convert to lowercase for case-insensitive comparison (Go file - // names are typically lowercase: "box.go", "renderer.ts"). - std::string fname_lower = fname; - std::string rtype_lower = receiver_type; - for (auto &ch : fname_lower) - ch = static_cast( - std::tolower(static_cast(ch))); - for (auto &ch : rtype_lower) - ch = static_cast( - std::tolower(static_cast(ch))); - if (!rtype_lower.empty() && - fname_lower.find(rtype_lower) != std::string::npos) - return kScorePartialMatch; - - return 0.0; -} - // v0.2.5 (perf): pre-parsed receiver matching. Build the ref-level context // (prefix1/prefix2/rtype_lower) once per reference so the resolver hot loop // does not reallocate them for every candidate. +// +// This builder sits between two scorers that were deleted as dead code +// (factorReceiverMatch, factorReceiverTypeMatch) — it is NOT dead itself: +// pipeline_apply.cpp calls it once per reference. Kept here, adjacent to the +// scorer that consumes the context. ReceiverMatchContext buildReceiverMatchContext(const std::string &receiver_type) { + // Canonical spelling (see canonicalTypeName): a reference recorded as + // `*Manager` must produce the same prefixes as one recorded as `Manager`, + // because the candidate side compares against `Manager::Get` / + // `Manager.Get`. Without this the strong match was unreachable for every + // pointer-typed receiver and the factor silently fell through to its weak + // file-name fallback. + const std::string canonical = canonicalTypeName(receiver_type); ReceiverMatchContext ctx; - if (receiver_type.empty()) { + if (canonical.empty()) { ctx.empty = true; return ctx; } - ctx.prefix1 = receiver_type + "::"; - ctx.prefix2 = receiver_type + "."; - ctx.rtype_lower = receiver_type; + ctx.prefix1 = canonical + "::"; + ctx.prefix2 = canonical + "."; + ctx.rtype_lower = canonical; for (auto &ch : ctx.rtype_lower) ch = static_cast( std::tolower(static_cast(ch))); return ctx; } -// Scoring is byte-identical to factorReceiverTypeMatch: same prefixes, same -// strong/partial/file-fallback rules. Only the string construction differs -// (precomputed ref-level strings; per-candidate file basename is still -// derived here because it is candidate-specific). double factorReceiverTypeMatchPrecomp(const ReceiverMatchContext &ctx, const std::string &candidate_qname, const std::string &candidate_file) @@ -300,7 +117,7 @@ double factorReceiverTypeMatchPrecomp(const ReceiverMatchContext &ctx, } // Weak fallback: candidate file basename contains the lowercased - // receiver type (mirrors factorReceiverTypeMatch lines 292-308). + // receiver type. size_t slash = candidate_file.rfind('/'); std::string fname = (slash != std::string::npos) ? candidate_file.substr(slash + 1) : @@ -309,9 +126,15 @@ double factorReceiverTypeMatchPrecomp(const ReceiverMatchContext &ctx, for (auto &ch : fname_lower) ch = static_cast( std::tolower(static_cast(ch))); + // A filename coincidence is weaker evidence than a qualified_name that + // really contains the receiver type, so it gets its own value instead of + // kScorePartialMatch: returning 0.5 here made `Widgets.cpp` — a file whose + // name happens to mention the type — score exactly the same as a candidate + // whose qualified_name contains it, which is the difference between a + // coincidence and receiver evidence. if (!ctx.rtype_lower.empty() && fname_lower.find(ctx.rtype_lower) != std::string::npos) - return kScorePartialMatch; + return kScoreWeakFilenameMatch; return 0.0; } diff --git a/engine/src/resolver/factors.h b/engine/src/resolver/factors.h index 22ddb49..9bf7be6 100644 --- a/engine/src/resolver/factors.h +++ b/engine/src/resolver/factors.h @@ -4,11 +4,187 @@ #include #include #include +#include #include +#include namespace resolver { +/// Infer the source language from a file path's extension. Returns "" when +/// the extension is unrecognized. +/// +/// NOTE: every C/C++ extension — including `.c` — maps to "cpp". The +/// visitors, by contrast, label a `.c` translation unit "c", so callers +/// that compare this value against an entity's language MUST use +/// languagesCompatible() rather than string equality. +inline std::string languageFromPath(const std::string &file_path) +{ + size_t dot = file_path.rfind('.'); + if (dot == std::string::npos) + return ""; + std::string ext = file_path.substr(dot); + // Normalize to lowercase for case-insensitive comparison. + std::string lower; + lower.reserve(ext.size()); + for (char ch : ext) + lower.push_back(static_cast( + std::tolower(static_cast(ch)))); + if (lower == ".cpp" || lower == ".cc" || lower == ".cxx" || + lower == ".c" || lower == ".h" || lower == ".hpp" || + lower == ".hh" || lower == ".hxx") + return "cpp"; + if (lower == ".rs") + return "rust"; + if (lower == ".py") + return "python"; + if (lower == ".go") + return "go"; + if (lower == ".ts" || lower == ".tsx") + return "typescript"; + if (lower == ".js" || lower == ".jsx") + return "javascript"; + if (lower == ".java") + return "java"; + return ""; +} + +/// True for the languages where a call may supply FEWER arguments than the +/// declaration lists, because the declaration has a default value or the call +/// is variadic in a way the producer cannot see. +/// +/// This matters for the arity factor: it penalises a call whose argument count +/// differs from the candidate's parameter count, but in C++ the default lives in +/// the HEADER and is omitted in the DEFINITION, and it is the definition the +/// entity and its `arity` come from. Measured on this repository: every lost +/// call edge between the previous commit and the arity work was of this shape — +/// `policyFor -> loadIgnoreFile` (definition declares `(root, append)`, the +/// header defaults `append`, the call passes one argument), +/// `indexProjectImpl -> loadGitignore`, `insertFileResultBatch` — and each was +/// penalised by factorSignatureMatch for being "different" when it was valid. +/// Go, Rust and C have no default arguments, so there a mismatch really is +/// evidence against the candidate (`New(a)` against a three-parameter `New`), +/// which is the precision the factor was added for and which this keeps. +inline bool allowsDefaultArguments(const std::string &file_path) +{ + const std::string lang = languageFromPath(file_path); + return lang == "cpp" || lang == "python" || lang == "javascript" || + lang == "typescript" || lang == "java"; +} + +/// True when two language labels may describe the same code. +/// +/// The labels come from two different vocabularies: languageFromPath() +/// reports every C/C++ extension (including `.c`) as "cpp", while the +/// visitors label a `.c` translation unit "c" and a `.h` header may belong +/// to either. Comparing the raw strings would therefore reject every +/// legitimate C call site, so the whole C family is treated as one +/// language. +/// +/// The JS/TS family needs the same treatment for the same reason, and this +/// was a measured false-negative class rather than a hypothetical one: +/// languageFromPath() reports a `.tsx` path as "typescript" while the tsx +/// visitor labels the entities it defines "tsx", so the two labels never +/// matched. On two real TypeScript projects the resolver produced ZERO call +/// edges — AIScope (220 tsx + 3 typescript entities, 1164 references of +/// which 37 name locally-defined symbols) and PolyScope (27 tsx + 14 +/// typescript, 80 references, 9 naming local symbols). A `.ts` module +/// calling a `.tsx` component, or a `.js` file calling into either, is the +/// ordinary shape of a React/TS codebase; the language label is one gate +/// among several (name, arity, directory, import evidence all still apply), +/// so treating the family as one language cannot manufacture edges on its +/// own. +/// +/// Everything else must match exactly; an empty label means "unknown" and is +/// always allowed through (never over-filter). +inline bool languagesCompatible(const std::string &a, const std::string &b) +{ + if (a.empty() || b.empty() || a == b) + return true; + static const std::unordered_set kCFamily = { + "c", "cpp", "c++", "cxx", "objc", "objective-c", + }; + static const std::unordered_set kJsFamily = { + "javascript", + "typescript", + "tsx", + }; + if (kCFamily.count(a) > 0 && kCFamily.count(b) > 0) + return true; + return kJsFamily.count(a) > 0 && kJsFamily.count(b) > 0; +} + +/// True when `candidate_file` is the module named by a RELATIVE import +/// specifier written in `caller_dir`. +/// +/// This is the evidence TypeScript/JavaScript projects never had. Their calls +/// are bare names (`helper()`), the language has no receiver to match on, and +/// the only import signal the visitors produced was a *plain set of imported +/// names* — which says "imported" but not "imported from here". So every +/// cross-directory call had no factor that could separate the candidates and +/// the resolver abstained (measured: PolyScope 0 edges from 80 references). +/// The `import` table does store the module specifier +/// (`./helper` → alias `helper`), so the missing piece was this comparison. +/// +/// `./helper` names helper.ts / helper.tsx / helper.js, so the match is +/// extension-agnostic, and it accepts a directory match (`./utils` → +/// `utils/index.ts`). A bare package specifier ("react", "@/lib/utils") names +/// no file inside the project and returns false: the caller must not invent +/// path evidence from an alias it cannot resolve. +inline bool relativeImportMatchesFile(const std::string &caller_dir, + const std::string &spec, + const std::string &candidate_file) +{ + if (spec.empty() || spec[0] != '.') + return false; + // Join the specifier onto the caller's directory and collapse the "." and + // ".." segments, so "../lib/x" and "./y" compare against real paths. + const std::string joined = caller_dir.empty() ? spec : + caller_dir + "/" + spec; + std::vector parts; + size_t i = 0; + while (i <= joined.size()) { + const size_t slash = joined.find('/', i); + const std::string seg = joined.substr( + i, slash == std::string::npos ? std::string::npos : + slash - i); + if (seg == "..") { + if (!parts.empty()) + parts.pop_back(); + } else if (!seg.empty() && seg != ".") { + parts.push_back(seg); + } + if (slash == std::string::npos) + break; + i = slash + 1; + } + std::string base; + for (const auto &p : parts) { + if (!base.empty()) + base += '/'; + base += p; + } + if (base.empty()) + return false; + // The collapse above drops the empty first segment of an absolute path; + // put the root back or no absolute candidate can ever match. + if (joined[0] == '/') + base.insert(base.begin(), '/'); + auto strip_ext = [](const std::string &p) -> std::string { + const size_t dot = p.rfind('.'); + const size_t slash = p.rfind('/'); + if (dot != std::string::npos && + (slash == std::string::npos || dot > slash)) + return p.substr(0, dot); + return p; + }; + if (candidate_file == base || strip_ext(candidate_file) == base) + return true; + // `./utils` may name a directory: any file inside it can be that module's + // entry point (index.ts and friends). + return candidate_file.rfind(base + "/", 0) == 0; +} + /// Fold an ASCII byte to lowercase. SQLite's default LIKE folds only /// ASCII upper-case letters; all other bytes (including non-ASCII) are /// returned unchanged so they compare byte-for-byte, matching SQLite. @@ -61,8 +237,31 @@ inline bool sqliteLikeMatch(const std::string &pattern, const std::string &text) } // ── Named constants for factor weights ────────────────────────────── +/// The one factor whose contribution this process drops, from +/// CODESCOPE_RESOLVER_DISABLE_FACTOR; empty when unset. +/// +/// This is the instrument for tuning the weights below, not a product switch. +/// `relation.reason` already records the factor with the largest contribution +/// (`decided_by=`), but that field answers "which weight is biggest" rather than +/// "which factor discriminated": ImportMatch carries 0.80 against 0.15 or less +/// for everything else, so on this repository it is the deciding factor for all +/// 1201 scored edges and the field carries no information about the rest. The +/// question that does have an answer is counterfactual — how many edges move +/// when this factor is taken away — and that is what this switch measures: index +/// the same tree twice, once with the factor disabled, and diff the edge sets. +/// +/// A misspelt name would silently disable nothing and look like "this factor +/// does not matter", so an unknown value is reported on stderr, once, with the +/// list of valid names. +const std::string &disabledFactor(); + constexpr double kWeightModuleMatch = 0.15; constexpr double kWeightImportMatch = 0.80; // Dominant for cross-module +// The caller imports this callee from a RELATIVE module and the candidate is +// that module: exact path evidence, so it outweighs the general import signal. +// Only applied when such a specifier was actually recorded — see +// relativeImportMatchesFile. +constexpr double kWeightImportModuleMatch = 0.90; constexpr double kWeightNamespaceMatch = 0.10; constexpr double kWeightSignatureMatch = 0.10; constexpr double kWeightDistanceMatch = 0.05; @@ -87,6 +286,11 @@ constexpr double kScorePartialMatch = 0.5; constexpr double kScorePenalty = -0.5; constexpr double kScoreSiblingModule = 0.5; constexpr double kScoreSameDirectory = 0.3; +// A candidate whose FILE NAME happens to contain the receiver type. Weaker +// than a qualified_name that really contains it (kScorePartialMatch), and it +// must not be confused with that hit: the two used to be the same value, so a +// filename coincidence was indistinguishable from receiver evidence. +constexpr double kScoreWeakFilenameMatch = 0.2; // ── Threshold ─────────────────────────────────────────────────────── constexpr double kResolutionThreshold = 0.40; @@ -113,28 +317,40 @@ struct FactorResult { std::string detail; // Human-readable explanation }; -/// A candidate entity with per-factor scores. -struct ScoredCandidate { - uint64_t entity_id; - std::string name; - std::string file_path; - std::string module_path; - std::string qualified_name; - int arity; - double total_score; // Weighted average of all factor scores - std::vector factors; -}; - -/// Compute the total score as weighted average of factors. -inline double computeTotalScore(const std::vector &factors) +/// Map a scoring factor name — the ones applyConstraints accumulates, see +/// pipeline_apply.cpp — to the `resolution_kind` recorded on the edge. +/// +/// The kind answers "which evidence decided this match", and the label is what +/// per-kind accuracy audits group by, so it must name the decisive evidence +/// rather than whichever reference field happened to be non-empty. An empty +/// result means the factor is unknown to this mapping, and the caller falls +/// back to its own labelling. +inline std::string resolutionKindFromFactor(const std::string &factor) { - double sum_weight = 0.0; - double sum_scored = 0.0; - for (auto &f : factors) { - sum_weight += f.weight; - sum_scored += f.weight * f.score; - } - return (sum_weight > 0.0) ? (sum_scored / sum_weight) : 0.0; + if (factor == "ReceiverMatch") + return "receiver_type"; + if (factor == "ImportMatch") + return "imported"; + // A distinct label from the general import signal: this one means the + // candidate IS the module the caller imported the name from, which is a + // stronger and auditable statement than "the caller has imports". + if (factor == "ImportModuleMatch") + return "import_module"; + if (factor == "SignatureMatch") + return "signature"; + if (factor == "DefinitionMatch") + return "definition"; + if (factor == "ConstructorMatch") + return "constructor"; + if (factor == "CallKindMatch") + return "call_kind"; + if (factor == "DistanceMatch") + return "distance"; + if (factor == "NamespaceMatch") + return "namespace"; + if (factor == "ModuleMatch") + return "module"; + return ""; } /// Check if the caller's file imports the candidate's module (forward) @@ -214,14 +430,96 @@ double factorReceiverTypeMatch(const std::string &receiver_type, const std::string &candidate_name, const std::string &candidate_file); +// ── Type-name spelling ───────────────────────────────────────────── +// +// A type name reaches the resolver in whatever form its use site was written +// in, while every table it is looked up in — interface_impl_index_, +// global_struct_fields_, global_var_types_ — is keyed by the bare name the +// declaration recorded. `*Manager`, `&ares_runtime.Manager` and `Manager` are +// one type, so comparisons and lookups go through canonicalTypeName(). + +/// Sigils a use site can put in front of a type name. +constexpr char kPointerSigil = '*'; +constexpr char kReferenceSigil = '&'; + +/// The bare type name: last `.`/`::`-separated segment, with pointer/reference +/// sigils, whitespace and a trailing generic argument list removed. +/// +/// `*PluginBus`, `&pkg.PluginBus`, `PluginBus*`, `PluginBus` → `PluginBus`. +/// +/// A template INSTANTIATION yields "" too, brackets and all, even though the +/// name inside them looks like a type: which name owns the method depends on the +/// template. `std::vector` holds its own methods, `std::unique_ptr` +/// forwards to `Store`, and `Handle` is whatever the library says — a rule +/// that keeps the last segment cannot tell them apart, and stripping the +/// argument list (an earlier version of this function did exactly that) actively +/// hurts: `std::unique_ptr` became `unique_ptr`, which matches no +/// declaration, so a call the old code resolved by unique name through +/// `policy->loadIgnoreFile(…)` now lost its edge while other receiver-carrying +/// calls gained false ones. Measured on this repository by rebuilding the commit +/// before the change and diffing call edges: -85 / +62 with that rule, against +/// -0 / +0 for the same tree when instantiations are simply unknown. Composite +/// spellings never matched a table key anyway (no declaration name contains a +/// bracket), so nothing legitimate is lost by abstaining. +/// +/// Anything that is NOT a plain (optionally qualified) name yields "": Go's +/// channel, slice, map and function types are recorded verbatim +/// (`<-chan os.Signal`, `[]<-chan *Event`, `map[string]int`, +/// `func(ctx context.Context, …) (…)`), and mining those for a "last segment" +/// produces a plausible-looking but unrelated symbol (`os.Signal` → `Signal`), +/// which is exactly the kind of coincidence the field-chain walk would then +/// resolve a receiver through. Measured on goagent: 1802 of the recorded type +/// spellings are plain names and ~730 are composite, none of which can be a +/// table key (no declaration name contains a bracket, sigil or separator). +/// +/// Why this exists: the field-chain walk (pipeline.cpp, Step 8.1c) looked the +/// RAW spelling of each segment up in global_struct_fields_, so a parameter +/// recorded as `*HolderA` never reached the entry keyed `HolderA` — the chain +/// was reported unresolvable and the walk moved on to the next same-named +/// variable's type, which is how a `h.registry.Get(...)` call site ended up +/// resolving through an unrelated type (measured on goagent: six `dispatch` +/// call sites, and a fixture where the pointer spelling produced no edge at +/// all while the value spelling produced two). The receiver-match factor had +/// the same hole from the other side: `prefix1 = receiver_type + "::"` built +/// from `*Manager` can never occur in a candidate's `Manager::Get`. +/// +/// @param type_name Type spelling as recorded; may be empty. +/// @return The bare name, or "" when the input is empty, all sigils, or not a +/// plain type name (in which case callers must treat it as unknown). +inline std::string canonicalTypeName(const std::string &type_name) +{ + const auto is_sigil_or_space = [](char ch) { + return ch == kPointerSigil || ch == kReferenceSigil || + std::isspace(static_cast(ch)) != 0; + }; + std::string text = type_name; + size_t begin = 0; + while (begin < text.size() && is_sigil_or_space(text[begin])) + ++begin; + text.erase(0, begin); + while (!text.empty() && is_sigil_or_space(text.back())) + text.pop_back(); + // Everything left must be a plain name: letters, digits, '_' and the two + // qualifier separators. Composite spellings are rejected as unknown rather + // than mined for a coincidental symbol. + for (char ch : text) + if (ch != '.' && ch != ':' && + !std::isalnum(static_cast(ch)) && ch != '_') + return {}; + const size_t sep = text.find_last_of(".:"); + if (sep != std::string::npos) + text.erase(0, sep + 1); + return text; +} + // ── v0.2.5 (perf): receiver-type match with pre-parsed ref-level strings ── // -// factorReceiverTypeMatch re-derives `receiver_type + "::"`, +// factorReceiverTypeMatchPrecomp re-derives `receiver_type + "::"`, // `receiver_type + "."` and the lowercased receiver_type on EVERY candidate. // All three depend only on the REF's receiver_type (fixed across candidates), // so in the resolver hot loop we build them once per ref and hand them to a -// pre-parsed variant that skips those allocations. Scoring is IDENTICAL to -// factorReceiverTypeMatch — do not change it independently. +// pre-parsed variant that skips those allocations. The receiver type is +// canonicalised here, once per ref, so both spellings of one type score alike. struct ReceiverMatchContext { std::string prefix1; // receiver_type + "::" std::string prefix2; // receiver_type + "." diff --git a/engine/src/resolver/pipeline.cpp b/engine/src/resolver/pipeline.cpp index 0aaa274..de04fea 100644 --- a/engine/src/resolver/pipeline.cpp +++ b/engine/src/resolver/pipeline.cpp @@ -60,41 +60,10 @@ bool shouldSkipFuzzy(const std::string &name) // Confidence thresholds were removed when resolved_reference table was // deprecated. Score is now stored directly in relation.confidence. - -// Infer the source language from a file path's extension. Used by the -// ScopeConstraint to prefer same-language candidates (a Rust symbol is -// unlikely to be the target of a C++ call site, etc.). Returns "" when -// the extension is unrecognized. -std::string languageFromPath(const std::string &file_path) -{ - size_t dot = file_path.rfind('.'); - if (dot == std::string::npos) - return ""; - std::string ext = file_path.substr(dot); - // Normalize to lowercase for case-insensitive comparison. - std::string lower; - lower.reserve(ext.size()); - for (char ch : ext) - lower.push_back(static_cast( - std::tolower(static_cast(ch)))); - if (lower == ".cpp" || lower == ".cc" || lower == ".cxx" || - lower == ".c" || lower == ".h" || lower == ".hpp" || - lower == ".hh" || lower == ".hxx") - return "cpp"; - if (lower == ".rs") - return "rust"; - if (lower == ".py") - return "python"; - if (lower == ".go") - return "go"; - if (lower == ".ts" || lower == ".tsx") - return "typescript"; - if (lower == ".js" || lower == ".jsx") - return "javascript"; - if (lower == ".java") - return "java"; - return ""; -} +// +// languageFromPath() and languagesCompatible() live in factors.h so every +// pipeline translation unit shares one implementation (this file previously +// carried a private copy that pipeline_load.cpp duplicated). } // namespace ResolverPipeline::ResolverPipeline(store::GraphStore *store, @@ -105,7 +74,7 @@ ResolverPipeline::ResolverPipeline(store::GraphStore *store, { // Import matching no longer uses prepared SQL statements: run() // pre-loads all import rows into import_index_ (file_path -> - // target_path list) and factorImportMatch matches against it + // target_path list) and ImportMatch matches against it // in-memory with SQLite-exact LIKE semantics. This removes the // 313k per-candidate full table scans that dominated run(). } @@ -304,77 +273,13 @@ int64_t ResolverPipeline::run() bool fuzzy_budget_exhausted = false; // ── Batch: read all references into memory first ──────────────── - // Instead of sqlite3_step per row in the hot loop, read all 108k refs - // into a vector at once. This avoids 108k individual sqlite3_step - // calls and lets the hot loop run entirely in memory (~10MB for 108k refs). - struct RefRow { - uint64_t ref_id; - std::string name; - uint64_t caller_id; - std::string caller_file; - int call_kind; - int arity; // caller arity from reference row (column r.arity) - int start_row; // Step 6: call site row for provenance - int start_col; // Step 6: call site col for provenance - std::string resolve_strategy; - // Step 3 (plan §3.1): structured call facts. Populated by - // per-language Visitors; used by the exact-first candidate - // generation in Step 5. Empty = unknown. - std::string qualified_target; // full call text, e.g. "b.Get" - std::string receiver_text; // syntactic receiver, e.g. "b" - std::string receiver_type; // inferred receiver type, e.g. "Box" - std::string import_alias; // import alias used, e.g. "fmt" - std::string call_site_file; // file path of the call site - }; + // loadReferences (pipeline_load.cpp) reads every project reference + // into `refs` in one pass so the hot loop runs with no SQLite + // round-trips. RefRow itself lives in pipeline.h alongside + // ResolvedEdge (same 1000-line-rule split). std::vector refs; - refs.reserve(65536); // pre-allocate for 108k typical - - while (sqlite3_step(ref_st) == SQLITE_ROW) { - RefRow r; - r.ref_id = - static_cast(sqlite3_column_int64(ref_st, 0)); - const char *name_c = reinterpret_cast( - sqlite3_column_text(ref_st, 1)); - r.caller_id = - static_cast(sqlite3_column_int64(ref_st, 2)); - // Column 3 is r.arity — the call site's arity. Previously this - // column was selected but never read, so the caller arity was - // always 0 in applyConstraints, breaking overload resolution. - r.arity = sqlite3_column_int(ref_st, 3); - // Step 6: read call site position for provenance (columns 4-5). - r.start_row = sqlite3_column_int(ref_st, 4); - r.start_col = sqlite3_column_int(ref_st, 5); - const char *fp_c = reinterpret_cast( - sqlite3_column_text(ref_st, 8)); - r.call_kind = sqlite3_column_int(ref_st, 6); - const char *rs_c = reinterpret_cast( - sqlite3_column_text(ref_st, 7)); - // Step 3: read structured call facts (columns 9-13). - const char *qt_c = reinterpret_cast( - sqlite3_column_text(ref_st, 9)); - const char *rtx_c = reinterpret_cast( - sqlite3_column_text(ref_st, 10)); - const char *rty_c = reinterpret_cast( - sqlite3_column_text(ref_st, 11)); - const char *ia_c = reinterpret_cast( - sqlite3_column_text(ref_st, 12)); - const char *csf_c = reinterpret_cast( - sqlite3_column_text(ref_st, 13)); - if (!name_c || !*name_c || !fp_c) - continue; - r.name = name_c; - r.caller_file = fp_c; - r.resolve_strategy = rs_c ? rs_c : ""; - r.qualified_target = qt_c ? qt_c : ""; - r.receiver_text = rtx_c ? rtx_c : ""; - r.receiver_type = rty_c ? rty_c : ""; - r.import_alias = ia_c ? ia_c : ""; - r.call_site_file = csf_c ? csf_c : fp_c; - refs.push_back(std::move(r)); - } - sqlite3_finalize(ref_st); + loadReferences(ref_st, refs, total_refs); ref_st = nullptr; - total_refs = static_cast(refs.size()); // Free the entity_index right after the hot loop — it's no longer needed. // Store results in a vector for batch insert. @@ -416,6 +321,11 @@ int64_t ResolverPipeline::run() // expansion only read, so they use the shared reference — // results are bit-identical. const std::vector *cands = nullptr; + // Whether the candidate set came from the fuzzy fallback rather than the + // exact-name index. The single-candidate fast path below labels its edge + // `exact_local`; for a fuzzy candidate that label was simply untrue — + // the name did not match exactly, it was only similar. + bool used_fuzzy = false; auto it = entity_index.find(ref.name); if (it != entity_index.end()) { cands = &it->second; // borrow — index stays intact @@ -482,6 +392,7 @@ int64_t ResolverPipeline::run() // point cands at it so subsequent reads (size/front/ // dispatch) see them. cands = &candidates; + used_fuzzy = true; } total_candidates_seen += static_cast(cands->size()); @@ -491,9 +402,16 @@ int64_t ResolverPipeline::run() continue; } + // Language of the call site, derived once per reference. Both the + // single-candidate fast path immediately below and the main + // hard-filter loop further down consume it, so the two paths + // apply exactly the same language rule. + const std::string caller_lang = + languageFromPath(ref.caller_file); + // ── Single-candidate fast path (semantically safe) ─────── // When exactly one candidate exists AND it shares the caller's - // directory, factorImportMatch early-returns 1.0 (ImportMatch, + // directory, the ImportMatch factor returns 1.0 (ImportMatch, // weight 0.80) and the other same-module factors are also high, // so the weighted total_score is always >= ~0.65 — well above // kResolutionThreshold (0.40) for every call_kind. Therefore the @@ -502,12 +420,33 @@ int64_t ResolverPipeline::run() // edge directly produces an IDENTICAL result while skipping the // full applyConstraints factor allocation/sort. // + // Notably this is a deliberate recall-over-precision trade-off for + // the FUZZY fallback too: a lone same-directory candidate whose name + // only matches by prefix is accepted here (Case D of + // test_resolution_kind: "DeltaThi" -> "DeltaThing"), which is why the + // edge's resolution_kind says fuzzy_local instead of claiming + // exactness. The name gate in that factor (pipeline_apply.cpp) lives + // on the scored path only and deliberately does not apply here. + // // Cross-module single candidates are NOT short-circuited: their // threshold outcome depends on the import match, so the exact // score must be computed to preserve identical edges. if (cands->size() == 1) { const Candidate &c = cands->front(); if (c.entity_id != ref.caller_id) { + // Hard language rule, identical to the main loop + // below: a .cpp call site can never resolve to a .py + // entity. The fast path previously omitted this check, + // so a lone same-directory candidate in another + // language produced a cross-language CALLS edge with + // confidence 0.85 that the full path rejects. The check + // must use the same compatibility rule as the main loop + // (C/C++ share a family) or valid C calls are rejected. + if (!languagesCompatible(caller_lang, + c.language)) { + skipped_lang_mismatch++; + continue; + } size_t c_slash = ref.caller_file.rfind('/'); size_t t_slash = c.file_path.rfind('/'); bool same_dir = @@ -516,22 +455,41 @@ int64_t ResolverPipeline::run() ref.caller_file.substr(0, c_slash) == c.file_path.substr(0, t_slash)); - if (same_dir && + // A fuzzy candidate is only acceptable for a BARE call: the + // fallback exists for visitors that record a call by its + // bare name (factors.h, ImportMatch); a receiver-carrying + // reference is a member access, and a same-directory FREE + // function is not that member (measured: every such edge was + // false — `Instant::now()` → `now_ms`, `math.Pow` → + // `powFunc`; all 50 here, 13 in goagent). + const bool fuzzy_ok = !used_fuzzy || + ref.call_kind == 0; + if (same_dir && fuzzy_ok && factorVisibilityCheck(c.language, c.name, ref.caller_file, c.file_path) >= 0.5) { resolved_count++; // Step 6: provenance for single-candidate // fast path. High confidence — only one - // candidate in the same directory. + // candidate in the same directory. The kind + // says how the NAME was matched, so a fuzzy + // candidate is not labelled "exact_local": + // the fast path also runs for the fuzzy + // fallback's single same-directory candidate + // (cands may point at that vector), and the + // audits group by this column. resolved_edges.push_back( { ref.caller_id, c.entity_id, kRelationTypeCall, ref.resolve_strategy, 0.85, // confidence "pipeline", // resolver - "exact_local", // resolution_kind - "single same-module candidate", + used_fuzzy ? + "fuzzy_local" : + "exact_local", // resolution_kind + used_fuzzy ? + "single same-module fuzzy candidate" : + "single same-module candidate", ref.call_site_file, ref.start_row, ref.start_col }); @@ -586,8 +544,15 @@ int64_t ResolverPipeline::run() if (fv != global_var_types_.end()) { for (const auto &cand_type : fv->second) { + // Canonical spelling: a variable + // is recorded as `*HolderA` or + // `pkg.Holder`, while the field + // table is keyed by the declared + // bare name (see + // canonicalTypeName). std::string cur_type = - cand_type; + canonicalTypeName( + cand_type); bool chain_ok = true; size_t pos = first_dot; while (chain_ok && @@ -622,35 +587,18 @@ int64_t ResolverPipeline::run() false; break; } - cur_type = fld->second; + cur_type = canonicalTypeName( + fld->second); pos = next; } if (chain_ok && !cur_type.empty()) { - // Normalize the resolved type so it - // can hit interface_impl_index_: - // strip a leading pointer marker - // (`*PluginBus` → `PluginBus`) and - // drop a package qualifier - // (`ares_runtime.PluginBus` → - // `PluginBus`), matching how the - // visitor records interface names. - std::string norm = + // Every segment was + // canonicalised above, so the + // result can hit + // interface_impl_index_. + resolved_receiver = cur_type; - if (!norm.empty() && - norm[0] == '*') - norm.erase(0, - 1); - size_t last_dot = - norm.rfind('.'); - if (last_dot != - std::string::npos) - norm = norm.substr( - last_dot + - 1); - if (!norm.empty()) - resolved_receiver = - norm; break; } } @@ -700,6 +648,21 @@ int64_t ResolverPipeline::run() impl_len + 1 && qn[impl_len + 1] == ':')) continue; + // Hard language filter, identical to the + // main loop and the fast path. This + // expansion selects targets by + // qualified_name prefix only, so without + // the filter a same-named entity in + // another language became a dispatch + // target: a .cpp call site could gain a + // CALLS edge to a Python implementation + // of the interface. Empty language + // (unknown) is allowed through, as in the + // main loop. + if (!languagesCompatible( + caller_lang, + c.language)) + continue; // Visibility check. if (factorVisibilityCheck( c.language, c.name, @@ -781,7 +744,11 @@ int64_t ResolverPipeline::run() double best_score = -1.0; uint64_t second_id = 0; double second_score = -1.0; - std::string caller_lang = languageFromPath(ref.caller_file); + // The winner itself, not just its id: Step 6 labels the edge with the + // evidence that decided this match (its deciding_factor). + const Candidate *best_cand = nullptr; + // caller_lang is hoisted above the single-candidate fast path + // (see the top of this loop body) so both paths share one rule. for (auto &c : candidates) { if (c.entity_id == ref.caller_id) continue; @@ -793,9 +760,10 @@ int64_t ResolverPipeline::run() // Step 5: hard filter — language match. A call site in // a .go file cannot resolve to a .py entity; skip the // candidate entirely. Empty language (unknown) is allowed - // through to avoid over-filtering edge cases. - if (!caller_lang.empty() && !c.language.empty() && - caller_lang != c.language) { + // through to avoid over-filtering edge cases. C and C++ + // share a family because the path-based classifier reports + // ".c" as "cpp" while the C visitor labels the unit "c". + if (!languagesCompatible(caller_lang, c.language)) { skipped_lang_mismatch++; continue; } @@ -804,6 +772,7 @@ int64_t ResolverPipeline::run() second_score = best_score; best_id = c.entity_id; best_score = c.total_score; + best_cand = &c; } else if (c.total_score > second_score) { second_id = c.entity_id; second_score = c.total_score; @@ -867,37 +836,80 @@ int64_t ResolverPipeline::run() // Fuzzy name similarity is inherently weaker than exact-name // matching, so require a higher confidence before writing a // CALLS edge from a fuzzy candidate. - bool from_fuzzy = (exact_hits == 0); // approximated; see note - (void)from_fuzzy; // not used for now — threshold is uniform + // The old `(exact_hits == 0)` approximation is gone: exact_hits is a + // running total over ALL references, so it said nothing about this one. + // The per-ref `used_fuzzy` flag set above is exact, and the fast path + // labels its edges from it. + // + // kFuzzyResolutionThreshold is still not applied, and the reason is + // worth stating: the edge confidence is the weighted factor score, in + // which the fuzzy name similarity is NOT a factor — the name is only + // the lookup key. Gating a fuzzy hit on that score would therefore + // filter by evidence strength (namespace/import/distance), not by match + // kind. Applying the threshold needs a name-similarity factor first; + // that is registered in the review doc instead of guessed at here. // Note: the fuzzy threshold kFuzzyResolutionThreshold is // reserved for when we can precisely track which candidates // came from fuzzy vs exact. For now, the evidence gate above // (fuzzy only fires with structured evidence) plus the // ambiguity gate provide sufficient FP protection. - // Step 6 (plan §6.2): determine resolution_kind from evidence. - // Priority: receiver_type > qualified_target > import_alias > - // name_arity. The kind records which evidence path produced - // the edge, enabling per-kind accuracy tracking and FP audits. + // Step 6 (plan §6.2): resolution_kind records the evidence that + // ACTUALLY decided the match. + // + // It used to be read off "which reference field is non-empty" + // (receiver_type > qualified_target > import_alias > name), which is a + // different question: a call carrying a receiver_type field but + // resolved in fact by name+arity — the receiver evidence contributing + // nothing to the winning score — was still labelled `receiver_type`, + // and every per-kind accuracy audit grouped by these labels inherited + // that skew. + // + // The label now comes from the winning candidate's largest positive + // factor contribution (Candidate::deciding_factor, recorded by + // applyConstraints), with one exception checked explicitly because it is + // not a scoring factor: an exact qualified-name hit, which is decisive + // on its own evidence. std::string res_kind; std::string reason; - if (!ref.receiver_type.empty()) { - res_kind = "receiver_type"; - reason = "receiver_type=" + ref.receiver_type + - " score=" + std::to_string(best_score); - } else if (!ref.qualified_target.empty()) { + if (!ref.qualified_target.empty() && best_cand != nullptr && + best_cand->qualified_name == ref.qualified_target) { res_kind = "qualified"; reason = "qualified_target=" + ref.qualified_target + - " score=" + std::to_string(best_score); - } else if (!ref.import_alias.empty()) { - res_kind = "imported"; - reason = "import_alias=" + ref.import_alias + - " score=" + std::to_string(best_score); - } else { - res_kind = "name_arity"; - reason = "name=" + ref.name + + " (exact) score=" + std::to_string(best_score); + } else if (best_cand != nullptr && + !best_cand->deciding_factor.empty()) { + res_kind = resolutionKindFromFactor( + best_cand->deciding_factor); + if (res_kind.empty()) + res_kind = "name_arity"; + reason = "decided_by=" + best_cand->deciding_factor + + " name=" + ref.name + " arity=" + std::to_string(ref.arity) + " score=" + std::to_string(best_score); + } else { + // No factor was recorded for the winner (every factor scored + // zero, or the candidate came from a path that does not run + // applyConstraints). Fall back to the field-presence label and say + // so, rather than inventing a kind. + if (!ref.receiver_type.empty()) { + res_kind = "receiver_type"; + reason = + "receiver_type=" + ref.receiver_type + + " score=" + std::to_string(best_score) + + " (field present; no factor recorded)"; + } else if (!ref.import_alias.empty()) { + res_kind = "imported"; + reason = + "import_alias=" + ref.import_alias + + " score=" + std::to_string(best_score) + + " (field present; no factor recorded)"; + } else { + res_kind = "name_arity"; + reason = "name=" + ref.name + + " arity=" + std::to_string(ref.arity) + + " score=" + std::to_string(best_score); + } } resolved_count++; @@ -926,7 +938,17 @@ int64_t ResolverPipeline::run() // Staging temp-table insert in one transaction, then bulk-copy into // relation + graph_edges. Finalizes ins_st and reports elapsed ms. int64_t sql_batch_ms = 0; - flushResolvedEdges(resolved_edges, ins_st, sql_batch_ms); + // A failed flush invalidates the build: return -1 so buildGraph + // rolls back to its savepoint instead of committing a graph that is + // missing the resolved CALLS edges. + if (!flushResolvedEdges(resolved_edges, ins_st, sql_batch_ms)) { + fprintf(stderr, + "[module=resolver, method=run] flush of resolved " + "edges failed — reporting failure to buildGraph\n"); + store_->exec("DROP TABLE IF EXISTS _resolved_edges"); + mark("sql_batch"); + return -1; + } mark("sql_batch"); // ── Cleanup staging table ── diff --git a/engine/src/resolver/pipeline.h b/engine/src/resolver/pipeline.h index c4afb30..c4473f7 100644 --- a/engine/src/resolver/pipeline.h +++ b/engine/src/resolver/pipeline.h @@ -76,6 +76,15 @@ class ResolverPipeline { // performed in-memory with SQLite-exact LIKE semantics, so the // resolved edges are IDENTICAL to the previous SQL implementation. std::unordered_map> import_index_; + // file_path -> (imported local name -> module specifier), from the `import` + // table. `./helper` written in app.tsx lands here as + // {"app.tsx": {"helper": "./helper"}} — the lookup that lets the resolver + // tell which module a bare `helper()` call came from. The specifier is used + // only when it is RELATIVE (see relativeImportMatchesFile); bare package + // names are recorded but never matched against a file. + std::unordered_map> + import_alias_index_; // Step 8 (plan §8.1): interface/trait implementation index. // Maps interface/trait name → list of implementing type names. @@ -124,6 +133,14 @@ class ResolverPipeline { int kind = 0; int score = 0; double total_score = 0.0; + // The factor that contributed most to `total_score` — i.e. the evidence + // that actually decided this candidate's ranking. Recorded by + // applyConstraints while the weighted sum is accumulated (the hot path + // deliberately does not materialise a FactorResult vector), so run() can + // label the edge with the evidence that decided it instead of with + // whichever reference field happens to be non-empty. Empty when nothing + // scored positively. + std::string deciding_factor; // v0.6 (perf): precomputed path components derived once when the // entity_index is loaded. applyConstraints recomputed dir/parent/ // module via rfind+substr for every candidate on every ref; since a @@ -166,14 +183,40 @@ class ResolverPipeline { int call_site_col; }; + /// One unresolved reference row, read fully into memory before the + /// hot loop (no SQLite round-trips inside the loop). Moved to class + /// scope so loadReferences can live in pipeline_load.cpp under the + /// 1000-line rule. + struct RefRow { + uint64_t ref_id; + std::string name; + uint64_t caller_id; + std::string caller_file; + int call_kind; + int arity; // caller arity from reference row (column r.arity) + int start_row; // Step 6: call site row for provenance + int start_col; // Step 6: call site col for provenance + std::string resolve_strategy; + // Step 3 (plan §3.1): structured call facts. Populated by + // per-language Visitors; used by the exact-first candidate + // generation in Step 5. Empty = unknown. + std::string qualified_target; // full call text, e.g. "b.Get" + std::string receiver_text; // syntactic receiver, e.g. "b" + std::string receiver_type; // inferred receiver type, e.g. "Box" + std::string import_alias; // import alias used, e.g. "fmt" + std::string call_site_file; // file path of the call site + }; + /// Flush the staged resolved edges into _resolved_edges (staging temp - /// table) in one transaction, then bulk-copy into relation and + /// table) inside a nested savepoint, then bulk-copy into relation and /// graph_edges. Finalizes ins_st. Extracted from run() so this TU stays /// under the 1000-line rule. /// @param resolved_edges Staged edges accumulated by the resolve loop. /// @param ins_st Prepared staging INSERT (finalized here). /// @param sql_batch_ms [out] milliseconds spent in the SQL flush. - void flushResolvedEdges(std::vector &resolved_edges, + /// @return true when every batch write succeeded; false lets run() + /// return -1 so buildGraph rolls the savepoint back. + bool flushResolvedEdges(std::vector &resolved_edges, sqlite3_stmt *ins_st, int64_t &sql_batch_ms); /// Apply constraints to rank candidates. @@ -216,6 +259,17 @@ class ResolverPipeline { /// global_var_types_) and semantic_records; no caller state. void loadDispatchIndex(); + /// Read every project reference into `refs` in one pass (no SQLite + /// round-trips in the hot loop). Extracted from run() + /// (pipeline_load.cpp) so this TU stays under the 1000-line rule. + /// @param ref_st Prepared SELECT over the reference table + /// (finalized here on success; left to the caller + /// when the statement is exhausted normally). + /// @param refs [out] filled with one RefRow per valid row. + /// @param total_refs [out] number of rows kept. + void loadReferences(sqlite3_stmt *ref_st, std::vector &refs, + int64_t &total_refs); + /// Check if `callee_name` is imported in the file at `caller_file`. /// Returns the import target path if found, empty string otherwise. std::string checkImport(const std::string &caller_file, diff --git a/engine/src/resolver/pipeline_apply.cpp b/engine/src/resolver/pipeline_apply.cpp index 8d28903..ab02518 100644 --- a/engine/src/resolver/pipeline_apply.cpp +++ b/engine/src/resolver/pipeline_apply.cpp @@ -15,15 +15,16 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, // // v0.2.5 (perf fix): the original code built a full // std::vector (name/detail heap strings) per candidate and - // fed it to computeTotalScore(). On large projects that is ~166k + // fed it to the weighted-average total. On large projects that is ~166k // candidate evaluations × ~20 string allocations each ≈ 3.3M heap // allocations — measured as the 90µs-per-candidate cost that made the // resolver 93.8% of index time (goagent: 14.9s of 15.9s). Here we // accumulate the weighted sum directly (pure double math, no strings) // and capture only the ReceiverMatch score that the ambiguity gate needs // (see receiver_bypass in run()). Every factor's weight/score pair and - // the final weighted-average formula are identical to the previous - // computeTotalScore path, so resolved edges are byte-for-byte unchanged. + // the final weighted-average formula are identical to the array-based + // scorer that was used before this optimisation, so resolved edges are + // byte-for-byte unchanged. // // v0.2.5 (perf fix #2): the path-based factors (ImportMatch, // NamespaceMatch, DistanceMatch) each re-derived caller_file's directory @@ -32,9 +33,11 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, // caller_parent / caller_module) and parse each candidate's path ONCE // inside the loop, then compute the three path factors from those // pre-parsed values. This removes ~N×3 redundant substring allocations - // per candidate. The scoring rules are kept IDENTICAL to - // factorImportMatch/factorNamespaceMatch/factorDistanceMatch in - // factors.cpp — do not change them independently. + // per candidate. THE RULES BELOW ARE the ImportMatch / NamespaceMatch / + // DistanceMatch definitions — the factor* functions in factors.cpp that an + // earlier comment told readers to keep in sync were dead code and have + // been deleted (batch 14). There is no second copy left to diverge from, + // so any change here changes resolution and must be re-measured. size_t caller_slash = caller_file.rfind('/'); std::string caller_dir = (caller_slash != std::string::npos) ? caller_file.substr(0, caller_slash) : @@ -45,8 +48,9 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, caller_dir.substr(0, caller_parent_slash) : ""; // moduleTokenFromPath: the last path token (file's directory name). - // Mirrors the anonymous helper in factors.cpp used by - // factorImportMatch. When there is no slash, the whole dir is returned. + // Same rule as moduleTokenFromPath() in factors.cpp (that helper served + // the deleted factorImportMatch; this inline copy is the live one). + // When there is no slash, the whole dir is returned. std::string caller_module = caller_dir; { size_t ms = caller_dir.rfind('/'); @@ -61,6 +65,26 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, } return false; }; + // ── Ref-level import-module lookup ── + // When the caller imports this callee from a RELATIVE module + // (`import { helper } from "./helper"`), that specifier is the one piece of + // evidence that can separate cross-directory candidates — the language has + // no receiver to match on and `import_alias` only says "imported", not + // "imported from here". Resolved once per reference, outside the candidate + // loop; bare package names are ignored because they name no file inside + // the project. Empty for every language that does not record imports this + // way, which keeps the factor below inert for them. + std::string import_spec; + { + auto fit = import_alias_index_.find(caller_file); + if (fit != import_alias_index_.end()) { + auto sit = fit->second.find(callee_name); + if (sit != fit->second.end() && !sit->second.empty() && + sit->second[0] == '.') + import_spec = sit->second; + } + } + // ── Ref-level factor precompute (perf fix #3) ── // factorCommonNamePenalty depends ONLY on the ref's callee_name, which // is fixed across all candidates of this ref, yet it was called once per @@ -102,15 +126,36 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, caller_fwd_joined += '\x00'; } } + // Read once per call rather than per candidate; the value is a cached + // static, and it is empty unless the tuning switch was set. + const std::string &disabled = disabledFactor(); for (auto &c : candidates) { double sum_weight = 0.0; double sum_scored = 0.0; // Accumulate one (weight, score) pair into the weighted sums. - // This mirrors computeTotalScore's loop without materializing a + // This is the weighted-average loop, kept free of the per-candidate // vector per candidate. - auto acc = [&](double weight, double score) { + // The factor with the largest positive contribution (weight * score) + // to this candidate's weighted average: the evidence that actually + // decided its ranking. A penalty has a non-positive contribution, so it + // is never chosen — a penalty cannot "decide" a match. + double best_contribution = 0.0; + const char *deciding = ""; + // CODESCOPE_RESOLVER_DISABLE_FACTOR: the counterfactual for weight + // tuning. Dropping a factor here also drops its weight from + // sum_weight, so the average is what it would have been had the factor + // never existed — and `deciding` can never be the disabled one. + auto acc = [&](double weight, double score, + const char *factor) { + if (disabled == factor) + return; sum_weight += weight; sum_scored += weight * score; + const double contribution = weight * score; + if (contribution > best_contribution) { + best_contribution = contribution; + deciding = factor; + } }; // ── Candidate path components (v0.6) ── @@ -123,7 +168,7 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, // ── NamespaceMatch score (reused by ModuleMatch and the // CommonNamePenalty same-module gate) ── - // Mirrors factorNamespaceMatch(caller_file, c.file_path): + // NamespaceMatch (same dir → 1.0; same parent dir → 0.5; else 0.0): // same dir → 1.0; same parent dir → 0.5; else 0.0. double ns_score = 0.0; if (!cand_dir.empty() && !caller_dir.empty()) { @@ -135,7 +180,7 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, } // Factor 1: ModuleMatch - acc(kWeightModuleMatch, ns_score); + acc(kWeightModuleMatch, ns_score, "ModuleMatch"); // Factor 2: ImportMatch — dominant weight for cross-module calls. // Mirrors factorImportMatch(import_index_, caller_file, @@ -144,33 +189,74 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, // 2. forward: caller imports candidate_module → 1.0 // 3. reverse: candidate imports caller_module → 1.0 // 4. else 0.0 + // + // Name gate: every clause above is path/module evidence, and none of + // them looks at the CALLEE NAME — so with this factor's 0.80 weight + // the fuzzy fallback cleared kFuzzyResolutionThreshold (0.55) on + // location alone. Measured in this repository: `count()` on a + // std::chrono::duration inside a timing helper is recorded as a + // reference named `count` (the parser had already marked it + // `external`), yet it resolved to the unrelated function `countLines` + // just because both files sit in engine/src — reason + // "decided_by=ImportMatch name=count arity=0 score=0.723684". That + // target collected 10 such callers (async_knowledge.cpp, + // engine_lifecycle.cpp, engine_queries.cpp, engine_verify_ffi.cpp, + // post_parse_phase.cpp, engine_index_metrics.cpp) while its only real + // caller is ingestReadmeDocument. The factor claims "the caller + // imported THIS callee", which a differently named candidate cannot + // be, so it only fires when the candidate carries the reference's + // name. Exact-name candidates keep it bit-for-bit, so every + // non-fuzzy path is unchanged; alias imports are unaffected too, + // because they are keyed by name in import_alias_index_ and scored by + // ImportModuleMatch (0.90). { + const bool name_agrees = (c.name == callee_name); double import_score = 0.0; - if (!caller_dir.empty() && caller_dir == cand_dir) { - import_score = 1.0; - } else { - // Forward: caller imports candidate module. - // Uses the ref-level fused string (perf fix #5): - // one find() over the contiguous buffer replaces - // the per-path scan of anyImportMatches. - if (!caller_fwd_joined.empty() && - caller_fwd_joined.find(cand_module) != - std::string::npos) + if (name_agrees) { + if (!caller_dir.empty() && + caller_dir == cand_dir) { import_score = 1.0; - else { - auto rev_it = - import_index_.find(c.file_path); - if (rev_it != import_index_.end() && - anyImportMatches(rev_it->second, - caller_module)) + } else { + // Forward: caller imports candidate module. + // Uses the ref-level fused string (perf fix #5): + // one find() over the contiguous buffer replaces + // the per-path scan of anyImportMatches. + if (!caller_fwd_joined.empty() && + caller_fwd_joined.find( + cand_module) != + std::string::npos) import_score = 1.0; + else { + auto rev_it = + import_index_.find( + c.file_path); + if (rev_it != import_index_ + .end() && + anyImportMatches( + rev_it->second, + caller_module)) + import_score = 1.0; + } } } - acc(kWeightImportMatch, import_score); + acc(kWeightImportMatch, import_score, "ImportMatch"); } + // Factor 2b: ImportModuleMatch — the caller imports this callee from a + // RELATIVE module and this candidate IS that module. Accumulated only + // when such a specifier exists, so every other reference keeps its + // previous weighted average exactly (the guarded-acc pattern the + // ImportMatch block above already uses). + if (!import_spec.empty()) + acc(kWeightImportModuleMatch, + relativeImportMatchesFile(caller_dir, import_spec, + c.file_path) ? + 1.0 : + 0.0, + "ImportModuleMatch"); + // Factor 3: NamespaceMatch - acc(kWeightNamespaceMatch, ns_score); + acc(kWeightNamespaceMatch, ns_score, "NamespaceMatch"); // Factor 4: SignatureMatch — compares the call site's arity // (from the reference row) against each candidate's arity. @@ -180,10 +266,22 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, // arity (returning +0.5) — the exact opposite of correct // overload resolution. Thread the real reference arity through // so exact-arity overloads score highest. - acc(kWeightSignatureMatch, - factorSignatureMatch(caller_arity, c.arity)); + { + double signature = + factorSignatureMatch(caller_arity, c.arity); + // Only supplying MORE arguments than the candidate declares is + // impossible. Fewer can be legitimate wherever defaults exist, and + // the default is usually declared somewhere this candidate's own + // arity cannot see (C++ declares it in the header, omits it in the + // definition) — see allowsDefaultArguments. + if (signature == kScorePenalty && + caller_arity < c.arity && + allowsDefaultArguments(c.file_path)) + signature = kScorePartialMatch; + acc(kWeightSignatureMatch, signature, "SignatureMatch"); + } - // Factor 5: DistanceMatch — mirrors factorDistanceMatch: + // Factor 5: DistanceMatch: // same file → 1.0; same directory → 0.3; else 0.0. { double dist_score = 0.0; @@ -191,15 +289,16 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, dist_score = kScoreExactMatch; else if (!caller_dir.empty() && caller_dir == cand_dir) dist_score = kScoreSameDirectory; - acc(kWeightDistanceMatch, dist_score); + acc(kWeightDistanceMatch, dist_score, "DistanceMatch"); } // Factor 6: ConstructorMatch acc(kWeightConstructorMatch, - factorConstructorMatch(callee_name, c.name, c.kind)); + factorConstructorMatch(callee_name, c.name, c.kind), + "ConstructorMatch"); // Factor 7: ReceiverMatch (Step 5: now type-based, not directory) - // Replaced factorReceiverMatch (directory heuristic) with + // This replaced the old directory-heuristic receiver factor with // factorReceiverTypeMatch (actual receiver_type evidence). // When receiver_type is known (e.g. "Box"), candidates whose // qualified_name contains "Box::" or "Box." score 1.0. When @@ -222,7 +321,7 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, c.file_path); } c.receiver_score = recv; - acc(kWeightReceiverMatch, recv); + acc(kWeightReceiverMatch, recv, "ReceiverMatch"); } // Factor 8: CommonNamePenalty — reduce score for very common names @@ -234,7 +333,8 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, bool same_module = (ns_score > 0.0); double penalty = same_module ? 0.0 : common_name_penalty; - acc(kWeightCommonNamePenalty, -penalty); + acc(kWeightCommonNamePenalty, -penalty, + "CommonNamePenalty"); } // Factor 9: CallKindMatch — adjust scoring based on call kind. @@ -252,7 +352,7 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, else if (call_kind == kCallKindMethod) kscore = -0.1; // slight penalty: methods usually same-module - acc(kWeightCallKindMatch, kscore); + acc(kWeightCallKindMatch, kscore, "CallKindMatch"); } // Factor 10: DefinitionMatch — for C/C++, prefer symbols defined @@ -262,7 +362,8 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, // identically (Finding #8). Non-C/C++ languages return 1.0 // (neutral), so their ranking is unaffected. acc(kWeightDefinitionMatch, - factorDefinitionMatch(c.language, c.file_path)); + factorDefinitionMatch(c.language, c.file_path), + "DefinitionMatch"); // VisibilityCheck was moved to a hard filter in run() to // ensure language visibility rules (e.g. Go unexported names) @@ -270,7 +371,8 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, // weighted factor can be overcome by other factors, but a // hard language rule must be absolute. - // Weighted average, identical to computeTotalScore's formula. + // Weighted average (the scorer's only formula). + c.deciding_factor = deciding; c.total_score = (sum_weight > 0.0) ? (sum_scored / sum_weight) : 0.0; // c.factors is intentionally NOT populated here (perf); the hot @@ -278,9 +380,17 @@ void ResolverPipeline::applyConstraints(std::vector &candidates, // rebuild it lazily for the resolved candidate only. } + // Deterministic ordering: primary key is the score (descending); ties + // are broken by entity_id (ascending) so the winning candidate is + // reproducible regardless of the input row order. A comparator that + // only compared total_score would leave equal-scoring homonyms to + // std::sort's unspecified ordering, making the resolved edge depend + // on the index write order. std::sort(candidates.begin(), candidates.end(), [](const Candidate &a, const Candidate &b) { - return a.total_score > b.total_score; + if (a.total_score != b.total_score) + return a.total_score > b.total_score; + return a.entity_id < b.entity_id; }); } diff --git a/engine/src/resolver/pipeline_flush.cpp b/engine/src/resolver/pipeline_flush.cpp index 6713d5d..511e684 100644 --- a/engine/src/resolver/pipeline_flush.cpp +++ b/engine/src/resolver/pipeline_flush.cpp @@ -6,17 +6,33 @@ namespace resolver { -void ResolverPipeline::flushResolvedEdges( +bool ResolverPipeline::flushResolvedEdges( std::vector &resolved_edges, sqlite3_stmt *ins_st, int64_t &sql_batch_ms) { using Clock = std::chrono::steady_clock; + bool ok = true; // ── Batch insert all resolved edges ──────────────────────────── // Single INSERT with multiple rows is faster than per-row INSERTs. - // Use a single transaction wrapping the batch for minimal WAL overhead. + // A nested SAVEPOINT wraps the batch — NOT BEGIN/COMMIT. This code + // runs inside buildGraph's SAVEPOINT and, usually, the caller's + // BEGIN: a plain BEGIN fails there ("cannot start a transaction + // within a transaction") and the matching COMMIT then commits the + // caller's transaction and destroys buildGraph's savepoint, so a + // later failure could no longer roll the build back — the + // all-or-nothing guarantee was silently lost, and the final + // RELEASE SAVEPOINT errored with "no such savepoint". if (!resolved_edges.empty()) { - store_->exec("BEGIN"); + bool savepoint_ok = + store_->exec("SAVEPOINT flush_resolved_edges"); + if (!savepoint_ok) { + fprintf(stderr, + "[module=resolver, method=flushResolvedEdges] " + "SAVEPOINT failed: %s\n", + store_->error().c_str()); + ok = false; + } for (auto &e : resolved_edges) { sqlite3_bind_int64(ins_st, 1, static_cast(e.caller_id)); @@ -40,15 +56,28 @@ void ResolverPipeline::flushResolvedEdges( sqlite3_bind_int(ins_st, 11, e.call_site_row); sqlite3_bind_int(ins_st, 12, e.call_site_col); int st_rc = sqlite3_step(ins_st); - if (st_rc != SQLITE_DONE && st_rc != SQLITE_CONSTRAINT) + if (st_rc != SQLITE_DONE && + st_rc != SQLITE_CONSTRAINT) { fprintf(stderr, "[module=resolver, method=run] " "staging insert failed (rc=%d): %s\n", st_rc, sqlite3_errmsg(store_->handle())); + ok = false; + } sqlite3_reset(ins_st); } - store_->exec("COMMIT"); + // Release the savepoint only if it was actually created; + // releasing a missing savepoint would report a second, + // misleading error. + if (savepoint_ok && + !store_->exec("RELEASE SAVEPOINT flush_resolved_edges")) { + fprintf(stderr, + "[module=resolver, method=flushResolvedEdges] " + "RELEASE SAVEPOINT failed: %s\n", + store_->error().c_str()); + ok = false; + } } resolved_edges.clear(); resolved_edges.shrink_to_fit(); @@ -73,6 +102,7 @@ void ResolverPipeline::flushResolvedEdges( "[module=resolver, method=run] " "batch relation insert failed: %s\n", store_->error().c_str()); + ok = false; } if (!store_->exec("INSERT OR IGNORE INTO graph_edges " @@ -85,11 +115,13 @@ void ResolverPipeline::flushResolvedEdges( "[module=resolver, method=run] " "batch graph_edges insert failed: %s\n", store_->error().c_str()); + ok = false; } sql_batch_ms = std::chrono::duration_cast( Clock::now() - t_sql) .count(); + return ok; } } // namespace resolver diff --git a/engine/src/resolver/pipeline_load.cpp b/engine/src/resolver/pipeline_load.cpp index 60e4a97..6d90dfe 100644 --- a/engine/src/resolver/pipeline_load.cpp +++ b/engine/src/resolver/pipeline_load.cpp @@ -8,43 +8,6 @@ namespace resolver { -namespace -{ -// Infer the source language from a file path's extension. Mirrors the -// helper in pipeline.cpp (kept as a per-TU copy because it lives in an -// anonymous namespace there; ODR-safe since anonymous namespaces isolate -// each translation unit). -std::string languageFromPath(const std::string &file_path) -{ - size_t dot = file_path.rfind('.'); - if (dot == std::string::npos) - return ""; - std::string ext = file_path.substr(dot); - std::string lower; - lower.reserve(ext.size()); - for (char ch : ext) - lower.push_back(static_cast( - std::tolower(static_cast(ch)))); - if (lower == ".cpp" || lower == ".cc" || lower == ".cxx" || - lower == ".c" || lower == ".h" || lower == ".hpp" || - lower == ".hh" || lower == ".hxx") - return "cpp"; - if (lower == ".rs") - return "rust"; - if (lower == ".py") - return "python"; - if (lower == ".go") - return "go"; - if (lower == ".ts" || lower == ".tsx") - return "typescript"; - if (lower == ".js" || lower == ".jsx") - return "javascript"; - if (lower == ".java") - return "java"; - return ""; -} -} // namespace - int ResolverPipeline::loadEntityIndex( std::unordered_map> &entity_index, std::unordered_map &entity_by_id, @@ -60,7 +23,6 @@ int ResolverPipeline::loadEntityIndex( // Without this column in the SELECT, c.arity defaulted to 0 // and every candidate scored kScorePartialMatch (0.5), // letting std::sort pick the winner by unstable order. - // See CODE_REVIEW_FINDINGS_2026-07-19.md C2. // Include kind (appended as column 5) so // factorConstructorMatch can prefer Class/Struct targets; // previously kind was hardcoded 0 in the call, so the @@ -68,9 +30,16 @@ int ResolverPipeline::loadEntityIndex( // Step 5: include qualified_name (column 6) so // factorReceiverTypeMatch can match "Box::draw" against // receiver_type="Box" instead of using directory heuristics. + // ORDER BY id makes the per-name candidate order deterministic + // (rowid order). Without it SQLite may return rows in any order + // after a parallel index merge, so equal-scoring homonyms could + // be ranked differently between runs. entity_index[name] keeps + // that order, and applyConstraints() uses entity_id as the final + // tie-break, so the winner is reproducible. "SELECT id, name, file_path, language, arity, kind, qualified_name " "FROM entity " - "WHERE project_id=? AND name != ''"; + "WHERE project_id=? AND name != '' " + "ORDER BY id"; sqlite3_stmt *idx_st = nullptr; if (sqlite3_prepare_v2(store_->handle(), idx_sql.c_str(), -1, &idx_st, nullptr) != SQLITE_OK) { @@ -202,9 +171,117 @@ int ResolverPipeline::loadEntityIndex( } sqlite3_finalize(imp_st); } + + // ── Step 0c: imported binding -> module specifier, from the IR records ── + // This is the index that lets a bare `widgetHelper()` call be traced to the + // module it came from, which is the only evidence that can separate + // cross-directory candidates in JS/TS (no receiver to match on, and + // `import_alias` records that a name IS imported, not where from). The + // `import` table cannot answer it: its alias column is the module path's + // last segment (`../lib/Widget` → `Widget`), not the bound name. Only the + // visitors that know the binding emit these records today (JS/TS, via + // SemanticEmitter::emitImportBinding), so for every other language this map + // stays empty and the factor reading it never fires. + import_alias_index_.clear(); + { + // RecordKind::ImportBinding as stored in semantic_records.kind. The + // store layer writes raw ordinals — store_graph.cpp persists this one + // as kKindImportBinding with the same value. + static constexpr int kKindImportBinding = 21; + const std::string binding_sql = + "SELECT file_path, name, type_name FROM semantic_records " + "WHERE project_id=? AND kind=? AND name != '' AND " + "type_name != ''"; + sqlite3_stmt *bind_st = nullptr; + if (sqlite3_prepare_v2(store_->handle(), binding_sql.c_str(), + -1, &bind_st, nullptr) != SQLITE_OK) { + fprintf(stderr, + "[module=resolver, method=run] " + "prepare import-binding index failed: %s\n", + sqlite3_errmsg(store_->handle())); + return -1; + } + sqlite3_bind_int64(bind_st, 1, + static_cast(project_id_)); + sqlite3_bind_int(bind_st, 2, kKindImportBinding); + while (sqlite3_step(bind_st) == SQLITE_ROW) { + const char *fp = reinterpret_cast( + sqlite3_column_text(bind_st, 0)); + const char *nm = reinterpret_cast( + sqlite3_column_text(bind_st, 1)); + const char *tn = reinterpret_cast( + sqlite3_column_text(bind_st, 2)); + if (!fp || !nm || !tn || !*fp || !*nm || !*tn) + continue; + import_alias_index_[fp][nm] = tn; + } + sqlite3_finalize(bind_st); + } return 0; } +void ResolverPipeline::loadReferences(sqlite3_stmt *ref_st, + std::vector &refs, + int64_t &total_refs) +{ + // Read all project references into memory first. Instead of + // sqlite3_step per row in the hot loop, read every ref into a vector + // at once. This avoids per-row sqlite3_step calls and lets the hot + // loop run entirely in memory (~10MB for 108k refs). Extracted from + // run() so pipeline.cpp stays under the 1000-line rule. + refs.clear(); + refs.reserve(65536); // pre-allocate for 108k typical + + while (sqlite3_step(ref_st) == SQLITE_ROW) { + RefRow r; + r.ref_id = + static_cast(sqlite3_column_int64(ref_st, 0)); + const char *name_c = reinterpret_cast( + sqlite3_column_text(ref_st, 1)); + r.caller_id = + static_cast(sqlite3_column_int64(ref_st, 2)); + // Column 3 is r.arity — the call site's arity. Previously this + // column was selected but never read, so the caller arity was + // always 0 in applyConstraints, breaking overload resolution. + r.arity = sqlite3_column_int(ref_st, 3); + // Step 6: read call site position for provenance (columns 4-5). + r.start_row = sqlite3_column_int(ref_st, 4); + r.start_col = sqlite3_column_int(ref_st, 5); + const char *fp_c = reinterpret_cast( + sqlite3_column_text(ref_st, 8)); + r.call_kind = sqlite3_column_int(ref_st, 6); + const char *rs_c = reinterpret_cast( + sqlite3_column_text(ref_st, 7)); + // Step 3: read structured call facts (columns 9-13). + const char *qt_c = reinterpret_cast( + sqlite3_column_text(ref_st, 9)); + const char *rtx_c = reinterpret_cast( + sqlite3_column_text(ref_st, 10)); + const char *rty_c = reinterpret_cast( + sqlite3_column_text(ref_st, 11)); + const char *ia_c = reinterpret_cast( + sqlite3_column_text(ref_st, 12)); + const char *csf_c = reinterpret_cast( + sqlite3_column_text(ref_st, 13)); + if (!name_c || !*name_c || !fp_c) + continue; + r.name = name_c; + r.caller_file = fp_c; + r.resolve_strategy = rs_c ? rs_c : ""; + r.qualified_target = qt_c ? qt_c : ""; + r.receiver_text = rtx_c ? rtx_c : ""; + r.receiver_type = rty_c ? rty_c : ""; + r.import_alias = ia_c ? ia_c : ""; + r.call_site_file = csf_c ? csf_c : fp_c; + refs.push_back(std::move(r)); + } + // Ownership: ref_st is passed by value, so this finalize is the + // transfer point — the caller's pointer is stale afterwards and is + // nulled there (pipeline.cpp). Do not dereference ref_st after this. + sqlite3_finalize(ref_st); + total_refs = static_cast(refs.size()); +} + void ResolverPipeline::loadDispatchIndex() { // ── Step 8 (plan §8.1): Pre-load interface/trait implementations ── @@ -391,6 +468,14 @@ void ResolverPipeline::loadDispatchIndex() // another file) can be resolved before dispatch expansion. global_struct_fields_.clear(); { + // ORDER BY key, not insertion order: the assignment below is + // last-write-wins, so when one (struct, field) is recorded twice with + // different types the winner was whatever row the scan reached last — + // i.e. the rowid order, which is the MERGE order of the module + // databases and therefore changes with the worker partitioning + // (semantic_records.rowid is re-assigned at merge time). Ordering by + // content makes the table a function of the records, not of how they + // were packed into modules. std::string field_sql = "SELECT p.name, t.name, t.type_name " "FROM semantic_records t " @@ -398,7 +483,8 @@ void ResolverPipeline::loadDispatchIndex() "AND p.project_id = t.project_id " "AND p.file_path = t.file_path " "WHERE t.project_id=? AND t.kind=17 AND p.kind=2 " - "AND t.name != '' AND t.type_name != ''"; + "AND t.name != '' AND t.type_name != '' " + "ORDER BY p.name, t.name, t.type_name"; sqlite3_stmt *fst = nullptr; if (sqlite3_prepare_v2(store_->handle(), field_sql.c_str(), -1, &fst, nullptr) == SQLITE_OK) { @@ -430,6 +516,18 @@ void ResolverPipeline::loadDispatchIndex() // type ("r" -> "Runner") when resolving "r.pluginBus.AfterStep". global_var_types_.clear(); { + // ORDER BY key, not insertion order: the walk in run() takes the FIRST + // candidate type whose field chain resolves ("the first type that + // resolves the whole chain wins"), so the vector's order decided the + // answer. Without this the order was the rowid order — the MERGE order + // of the module databases, which changes with the worker partitioning + // (semantic_records.rowid is re-assigned at merge time) — and the same + // project resolved a field-chain receiver to an interface or to + // nothing depending on how its directories were spread over workers: + // measured on goagent, `-w 4` produced 684 dispatch edges and `-w 6` + // produced 696, from byte-identical entities, references and + // semantic_records. Ordering by content makes the resolution a + // function of the records. std::string vtype_sql = "SELECT t.name, t.type_name FROM semantic_records t " "JOIN semantic_records p ON t.parent_id = p.original_id " @@ -437,7 +535,8 @@ void ResolverPipeline::loadDispatchIndex() "AND p.file_path = t.file_path " "WHERE t.project_id=? AND t.kind=17 " "AND p.kind IN (0,1) " - "AND t.name != '' AND t.type_name != ''"; + "AND t.name != '' AND t.type_name != '' " + "ORDER BY t.name, t.type_name"; sqlite3_stmt *vst = nullptr; if (sqlite3_prepare_v2(store_->handle(), vtype_sql.c_str(), -1, &vst, nullptr) == SQLITE_OK) { diff --git a/engine/src/resolver/project_index.cpp b/engine/src/resolver/project_index.cpp deleted file mode 100644 index c42864b..0000000 --- a/engine/src/resolver/project_index.cpp +++ /dev/null @@ -1,30 +0,0 @@ -#include "project_index.h" - -namespace resolver -{ - -void ProjectSymbolIndex::addEntry(const IndexEntry &entry) -{ - std::lock_guard lock(mutex_); - name_index_[entry.name].push_back(entry); - entries_.emplace_back(entry.name, entry); -} - -const std::vector * -ProjectSymbolIndex::lookup(const std::string &name) const -{ - std::lock_guard lock(mutex_); - auto it = name_index_.find(name); - if (it != name_index_.end()) - return &it->second; - return nullptr; -} - -void ProjectSymbolIndex::clear() -{ - std::lock_guard lock(mutex_); - name_index_.clear(); - entries_.clear(); -} - -} // namespace resolver diff --git a/engine/src/resolver/project_index.h b/engine/src/resolver/project_index.h deleted file mode 100644 index 63b2a6e..0000000 --- a/engine/src/resolver/project_index.h +++ /dev/null @@ -1,71 +0,0 @@ -#ifndef PROJECT_INDEX_H -#define PROJECT_INDEX_H - -#include -#include -#include -#include - -#include "../ir/ir.h" - -namespace resolver -{ - -// ─── Symbol Index Entry ───────────────────────────────────────── -// -// A single entry in the project-wide symbol index. -// Populated by pre-scanning all source files before translation. - -struct IndexEntry { - std::string name; - std::string file_path; - std::string module_path; // e.g. "src/util" from "src/util/helper.c" - ir::NodeKind kind; - ir::SourceLocation loc; - bool is_static = false; // static functions limited to file scope -}; - -// ─── Project Symbol Index ────────────────────────────────────── -// -// In-memory, project-wide symbol lookup table. -// Built by a pre-scan pass before main translation begins. -// Not backed by SQL — purely an unordered_map for O(1) lookup. - -class ProjectSymbolIndex { - public: - // Add a single symbol entry to the index (thread-safe). - void addEntry(const IndexEntry &entry); - - // Look up all entries with the given name (thread-safe). - // Returns empty vector if no entries found. - const std::vector *lookup(const std::string &name) const; - - // Clear all entries (for re-indexing). - void clear(); - - // Return total number of indexed symbols. - size_t size() const - { - std::lock_guard lock(mutex_); - return entries_.size(); - } - - // Check if the index has been populated. - bool empty() const - { - std::lock_guard lock(mutex_); - return name_index_.empty(); - } - - private: - // Primary index: name → entries - mutable std::mutex mutex_; - std::unordered_map> name_index_; - - // Flat list of all entries (for iteration / stats) - std::vector> entries_; -}; - -} // namespace resolver - -#endif // PROJECT_INDEX_H diff --git a/engine/src/resolver/project_resolver.cpp b/engine/src/resolver/project_resolver.cpp deleted file mode 100644 index 3de2382..0000000 --- a/engine/src/resolver/project_resolver.cpp +++ /dev/null @@ -1,160 +0,0 @@ -#include "project_resolver.h" - -#include -#include - -#include "factors.h" - -namespace resolver -{ - -ResolutionResult ProjectResolver::resolve(const std::string &name, - const std::string &file_path, - const ir::Node *context) -{ - ResolutionResult result; - result.name = name; - result.resolver_name = "project_resolver"; - - if (!index_ || index_->empty()) { - result.status = ResolutionStatus::NotFound; - return result; - } - - const auto *candidates = index_->lookup(name); - if (!candidates || candidates->empty()) { - result.status = ResolutionStatus::NotFound; - return result; - } - - // Score each candidate - struct Scored { - const IndexEntry *entry; - int score; - }; - std::vector scored; - scored.reserve(candidates->size()); - - for (const auto &entry : *candidates) { - // Skip static declarations that are not in the caller's file - if (entry.is_static && entry.file_path != file_path) - continue; - - int score = rankCandidate(entry, file_path); - scored.push_back({ &entry, score }); - } - - if (scored.empty()) { - result.status = ResolutionStatus::NotFound; - return result; - } - - // Sort by score descending - std::sort(scored.begin(), scored.end(), - [](const Scored &a, const Scored &b) { - return a.score > b.score; - }); - - // Build ResolvedSymbol results - for (const auto &s : scored) { - ResolvedSymbol rs; - rs.name = s.entry->name; - rs.file_path = s.entry->file_path; - rs.kind = s.entry->kind; - rs.loc = s.entry->loc; - rs.rank_score = s.score; - result.candidates.push_back(rs); - } - - // Determine status: - // If best score > 0 and strictly higher than second best → Resolved - // If best score == second best → Ambiguous - // Otherwise → Resolved (single candidate) - if (scored.size() == 1) { - result.status = ResolutionStatus::Resolved; - } else if (scored[0].score > scored[1].score) { - result.status = ResolutionStatus::Resolved; - } else { - result.status = ResolutionStatus::Ambiguous; - result.error_message = - "Multiple candidates with equal rank for '" + name + - "'"; - } - - return result; -} - -int ProjectResolver::rankCandidate(const IndexEntry &candidate, - const std::string &caller_file) const -{ - int score = 0; - - // Same directory: strong signal (likely in the same module) - std::string caller_dir = extractModule(caller_file); - std::string candidate_dir = extractModule(candidate.file_path); - if (!caller_dir.empty() && caller_dir == candidate_dir) - score += 3; - - // Same file: already handled by local resolver, but if we got here - // it means local didn't find it. Still score it. - if (caller_file == candidate.file_path) - score += 1; - - // Not static: more likely to be visible across files - if (!candidate.is_static) - score += 1; - - // C/C++ definition priority (Finding #8): prefer symbols defined in - // a source file (.c/.cpp/.cc/...) over header-only prototypes - // (.h/.hpp/...). Applied only to C/C++ extensions, so other - // languages are unaffected. - if (isCppSourceFile(candidate.file_path)) - score += 2; - else if (isCppHeaderFile(candidate.file_path)) - score -= 1; - - // Shorter path distance = more likely related - // (heuristic: files in nearby directories are more likely related) - try { - std::filesystem::path cp(caller_file); - std::filesystem::path dp(candidate.file_path); - auto ci = cp.begin(), di = dp.begin(); - while (ci != cp.end() && di != dp.end() && *ci == *di) { - ++ci; - ++di; - } - // Count remaining components as distance - int remaining = 0; - for (; ci != cp.end(); ++ci) - remaining++; - for (; di != dp.end(); ++di) - remaining++; - if (remaining <= 1) - score += 2; - else if (remaining <= 3) - score += 1; - } catch (const std::exception &e) { - fprintf(stderr, - "RESOLVER: rankCandidate filesystem error: %s [module=resolver, method=rankCandidate]\n", - e.what()); - } - - return score; -} - -std::string ProjectResolver::extractModule(const std::string &file_path) -{ - try { - std::filesystem::path p(file_path); - if (p.has_parent_path()) { - return p.parent_path().lexically_normal().string(); - } - } catch (const std::exception &e) { - fprintf(stderr, - "RESOLVER: extractModule filesystem error: %s [module=resolver, method=extractModule]\n", - e.what()); - } - return ""; -} - -} // namespace resolver diff --git a/engine/src/resolver/project_resolver.h b/engine/src/resolver/project_resolver.h deleted file mode 100644 index abcab7d..0000000 --- a/engine/src/resolver/project_resolver.h +++ /dev/null @@ -1,54 +0,0 @@ -#ifndef PROJECT_RESOLVER_H -#define PROJECT_RESOLVER_H - -#include "resolver.h" -#include "project_index.h" - -namespace resolver -{ - -// ─── Project Resolver ─────────────────────────────────────────── -// -// Uses ProjectSymbolIndex to resolve symbols that were not found -// by the local scope resolver. When multiple candidates exist, -// applies ranking heuristics: -// -// 1. Same directory (+3) -// 2. Same module/package (+2) -// 3. Included in the same file (+2) -// 4. Not static (+1) -// 5. File path lexical distance (+0..1) -// -// The highest-ranked unique candidate is returned as Resolved. -// If the top two candidates have the same score → Ambiguous. - -class ProjectResolver : public Resolver { - public: - explicit ProjectResolver(const ProjectSymbolIndex *index) - : index_(index) - { - } - - ResolutionResult resolve(const std::string &name, - const std::string &file_path, - const ir::Node *context) override; - - const char *name() const override - { - return "project_resolver"; - } - - private: - const ProjectSymbolIndex *index_; - - // Compute a rank score for a candidate relative to the call site. - int rankCandidate(const IndexEntry &candidate, - const std::string &caller_file) const; - - // Extract module path from a file path (directory portion). - static std::string extractModule(const std::string &file_path); -}; - -} // namespace resolver - -#endif // PROJECT_RESOLVER_H diff --git a/engine/src/resolver/resolve_cache.cpp b/engine/src/resolver/resolve_cache.cpp deleted file mode 100644 index 543c6db..0000000 --- a/engine/src/resolver/resolve_cache.cpp +++ /dev/null @@ -1,85 +0,0 @@ -#include "resolve_cache.h" - -namespace resolver -{ - -CachingResolver::CachingResolver(std::unique_ptr inner) - : inner_(std::move(inner)) -{ -} - -void CachingResolver::setFile(const std::string &file_path) -{ - // Per-file caches are keyed by file_path already — no explicit clear needed. - // resolve_cache_ entries are naturally scoped by (file, name) key. - (void)file_path; -} - -void CachingResolver::addImport(const std::string &file_path, - const std::string &imported_module) -{ - reach_cache_[file_path].insert(imported_module); -} - -void CachingResolver::addImportMap(const std::string &prefix, - const std::string &module_scope) -{ - import_map_cache_[prefix] = module_scope; -} - -ResolutionResult CachingResolver::resolve(const std::string &name, - const std::string &file_path, - const ir::Node *context) -{ - // Check resolve_cache first: (file, name) → last resolved result - CacheKey key{ file_path, name }; - { - auto it = resolve_cache_.find(key); - if (it != resolve_cache_.end()) { - stats_.resolve_cache_hit++; - return it->second; - } - stats_.resolve_cache_miss++; - } - - // Check reach_cache: is name reachable from this file's imports? - { - auto file_it = reach_cache_.find(file_path); - if (file_it != reach_cache_.end()) { - // Quick prefix match: name like "fmt.Println" → check if "fmt" is imported - auto dot = name.find('.'); - if (dot != std::string::npos) { - std::string prefix = name.substr(0, dot); - if (file_it->second.find(prefix) != - file_it->second.end()) { - stats_.reach_cache_hit++; - // Known import — proceed to inner resolve - } - } - } - } - - // Check import_map_cache: map import prefix to module scope - { - auto dot = name.find('.'); - if (dot != std::string::npos) { - std::string prefix = name.substr(0, dot); - auto it = import_map_cache_.find(prefix); - if (it != import_map_cache_.end()) { - stats_.import_map_hit++; - // prefix known, proceed to inner resolve with context - } - } - } - - // Fall through to inner resolver - ResolutionResult result = inner_->resolve(name, file_path, context); - - // Cache the result for this (file, name) pair - resolve_cache_[key] = result; - stats_.resolve_cache_stored++; - - return result; -} - -} // namespace resolver diff --git a/engine/src/resolver/resolve_cache.h b/engine/src/resolver/resolve_cache.h deleted file mode 100644 index 23f67e3..0000000 --- a/engine/src/resolver/resolve_cache.h +++ /dev/null @@ -1,100 +0,0 @@ -#ifndef RESOLVE_CACHE_H -#define RESOLVE_CACHE_H - -#include "resolver.h" -#include -#include -#include - -namespace resolver -{ - -/** - * Resolve cache trio: reachability, import map, and per-file-name cache. - * - * Wraps an existing Resolver and adds three caches: - * 1. reach_cache: per-file imported module set (which files are reachable) - * 2. import_map_cache: import prefix → module/symbol scope - * 3. resolve_cache: (caller_file, callee_name) → last result, avoids repeat work - * - * All caches are file-scoped and cleared when the file changes. - */ -class CachingResolver : public Resolver { - public: - explicit CachingResolver(std::unique_ptr inner); - - ResolutionResult resolve(const std::string &name, - const std::string &file_path, - const ir::Node *context) override; - - const char *name() const override - { - return "caching_resolver"; - } - - /// Switch to a new file — clears per-file caches. - void setFile(const std::string &file_path); - - /// Record that `file_path` imports from `imported_module`. - void addImport(const std::string &file_path, - const std::string &imported_module); - - /// Record that `prefix` maps to `module_scope`. - void addImportMap(const std::string &prefix, - const std::string &module_scope); - - /// Cache stats for observability. - struct Stats { - uint64_t reach_cache_hit = 0; - uint64_t import_map_hit = 0; - uint64_t resolve_cache_hit = 0; - uint64_t resolve_cache_miss = 0; - uint64_t resolve_cache_stored = 0; - }; - const Stats &stats() const - { - return stats_; - } - void resetStats() - { - stats_ = Stats{}; - } - - private: - std::unique_ptr inner_; - - // Key: caller_file_path - // Value: set of imported module/file paths - std::unordered_map> - reach_cache_; - - // Key: import prefix (e.g. "os", "fmt") - // Value: resolved module scope - std::unordered_map import_map_cache_; - - // Key: (caller_file, callee_name) - // Value: cached resolution result - struct CacheKey { - std::string file; - std::string name; - bool operator==(const CacheKey &o) const - { - return file == o.file && name == o.name; - } - }; - struct CacheKeyHash { - size_t operator()(const CacheKey &k) const - { - return std::hash{}(k.file) ^ - (std::hash{}(k.name) << 1); - } - }; - std::unordered_map - resolve_cache_; - - Stats stats_; -}; - -} // namespace resolver - -#endif // RESOLVE_CACHE_H diff --git a/engine/src/resolver/resolver.cpp b/engine/src/resolver/resolver.cpp deleted file mode 100644 index 1e3878b..0000000 --- a/engine/src/resolver/resolver.cpp +++ /dev/null @@ -1,32 +0,0 @@ -#include "resolver.h" - -namespace resolver -{ - -void ResolverChain::addResolver(std::unique_ptr resolver) -{ - if (resolver) - resolvers_.push_back(std::move(resolver)); -} - -ResolutionResult ResolverChain::resolve(const std::string &name, - const std::string &file_path, - const ir::Node *context) -{ - for (auto &r : resolvers_) { - if (!r) - continue; - ResolutionResult result = r->resolve(name, file_path, context); - if (result.status != ResolutionStatus::NotFound) { - return result; - } - } - - ResolutionResult not_found; - not_found.name = name; - not_found.status = ResolutionStatus::NotFound; - not_found.resolver_name = "resolver_chain"; - return not_found; -} - -} // namespace resolver diff --git a/engine/src/resolver/resolver.h b/engine/src/resolver/resolver.h deleted file mode 100644 index a89daca..0000000 --- a/engine/src/resolver/resolver.h +++ /dev/null @@ -1,111 +0,0 @@ -#ifndef RESOLVER_H -#define RESOLVER_H - -#include -#include -#include - -#include "../ir/ir.h" - -namespace resolver -{ - -// ─── Resolution Result ────────────────────────────────────────── -// -// Represents the outcome of attempting to resolve a symbol name. -// A resolved result carries enough information to create an IR -// SemanticEdge with Relation::CallTarget or Relation::SymbolRef -// pointing to the canonical definition, even when that definition -// lives in a different file (cross-file resolution). - -enum class ResolutionStatus { - Resolved, // A matching symbol was found (not necessarily unique) - Ambiguous, // Multiple equally-ranked candidates (caller must decide) - NotFound, // No matching symbol in any resolver -}; - -struct ResolvedSymbol { - uint64_t ir_node_id = 0; // ID of the IR node in the DB - uint64_t graph_node_id = 0; // ID of the graph node in the DB - std::string name; - std::string file_path; // File where the target is defined - std::string qualified_name; // Full qualified name if available - ir::NodeKind kind; - ir::SourceLocation loc; - int rank_score = 0; // Higher = better match (for disambiguation) -}; - -struct ResolutionResult { - ResolutionStatus status = ResolutionStatus::NotFound; - std::string name; // Original name being resolved - std::vector candidates; // Ranked candidates (best first) - std::string resolver_name; // Which resolver produced this result - std::string error_message; // Human-readable diagnostic - - bool isResolved() const - { - return status == ResolutionStatus::Resolved && - !candidates.empty(); - } - - const ResolvedSymbol *best() const - { - return candidates.empty() ? nullptr : &candidates[0]; - } -}; - -// ─── Resolver Interface ───────────────────────────────────────── -// -// All resolvers (Local, Project, LSP, etc.) implement this interface. -// The chain calls each resolver in order and stops at the first -// non-NotFound result. - -class Resolver { - public: - virtual ~Resolver() = default; - - // Resolve a symbol name within the given context. - // @param name The symbol name to resolve (e.g. "add") - // @param file_path The file where the call site resides - // @param context The containing IR node (typically a FunctionDecl), - // or nullptr if not available - // @return ResolutionResult with status + candidates - virtual ResolutionResult resolve(const std::string &name, - const std::string &file_path, - const ir::Node *context) = 0; - - virtual const char *name() const = 0; -}; - -// ─── Resolver Chain ───────────────────────────────────────────── -// -// Chains multiple resolvers together: Local → Project → LSP → ... -// Each resolver is tried in order. If a resolver returns Resolved, -// the chain stops and returns that result. If NotFound, it falls -// through to the next resolver. - -class ResolverChain : public Resolver { - public: - void addResolver(std::unique_ptr resolver); - - ResolutionResult resolve(const std::string &name, - const std::string &file_path, - const ir::Node *context) override; - - const char *name() const override - { - return "resolver_chain"; - } - - size_t size() const - { - return resolvers_.size(); - } - - private: - std::vector> resolvers_; -}; - -} // namespace resolver - -#endif // RESOLVER_H diff --git a/engine/src/store/store.cpp b/engine/src/store/store.cpp index 4d62709..793428c 100644 --- a/engine/src/store/store.cpp +++ b/engine/src/store/store.cpp @@ -1,4 +1,5 @@ #include "store.h" +#include "util/json_writer.h" #include "store_internal.h" #include "platform_win.h" @@ -44,7 +45,7 @@ bool GraphStore::rollbackTransaction() // // Each thread owns its own prepared statements. A single sqlite3_stmt* is // therefore never shared across threads, fixing the M-2 cross-thread race on -// a shared cached statement under concurrent g_store access. The connection is +// a shared cached statement under concurrent engine store access. The connection is // already serialized via SQLITE_CONFIG_SERIALIZED (see open()), so concurrent // use of DIFFERENT per-thread statements on the same connection is safe. // Bounded to kStmtCacheMax entries per thread; finalized by clearStmtCache() @@ -112,40 +113,4 @@ void GraphStore::clearStmtCache() // ── Shared JSON helper ───────────────────────────────────────── // Declared in store_internal.h; used by all store_*.cpp split files. -std::string jsonEscape(const std::string &s) -{ - std::string out; - out.reserve(s.size() + 4); - for (char c : s) { - switch (c) { - case '"': - out += "\\\""; - break; - case '\\': - out += "\\\\"; - break; - case '\n': - out += "\\n"; - break; - case '\r': - out += "\\r"; - break; - case '\t': - out += "\\t"; - break; - default: - if (static_cast(c) < 0x20) { - char buf[8]; - snprintf(buf, sizeof(buf), "\\u%04x", - static_cast(c)); - out += buf; - } else { - out += c; - } - break; - } - } - return out; -} - } // namespace store diff --git a/engine/src/store/store.h b/engine/src/store/store.h index b6f86ad..54cf544 100644 --- a/engine/src/store/store.h +++ b/engine/src/store/store.h @@ -2,6 +2,7 @@ #define STORE_H #include +#include #include #include #include @@ -45,8 +46,9 @@ namespace store * cognitive, lines, param_count, call_count, branch_count, loop_count) * project_id is filled by the writer, not stored in the row. * - * name/line/col are used to JOIN with the symbols table at resolve time - * (symbol IDs are not known until after buildGraph + populateSymbolsFromGraph). + * A staged row is matched back to its entity row by (project_id, file_path, + * start_row, start_col) in resolveStagedMetrics; `name` travels with it for + * diagnostics and is not part of that match. */ struct MetricRow { std::string name; // symbol name, for symbol JOIN at resolve time @@ -105,6 +107,21 @@ class GraphStore { uint64_t createProject(const char *root_path, const char *name); uint64_t getProjectId(const char *root_path); + + /** + * Read a project's registered root path. + * + * Used where a caller must know the tree it is dealing with without + * receiving it as a parameter — the enhance pass ingests the project's + * root README, and that pass is reached from several index paths that do + * not all carry the directory. + * + * @param project_id Project identifier. + * @return The stored `projects.root_path`, or an empty string when the + * project does not exist. A failed query is logged through + * error(). + */ + std::string getProjectRootPath(uint64_t project_id); uint64_t getLatestProjectId(); /** @@ -127,22 +144,11 @@ class GraphStore { uint64_t insertGraphNode(uint64_t project_id, const graph::GraphNode &node); - /** Batch insert graph nodes — prepare once, bind/step/reset in a loop. - * Up to ~10x faster than calling insertGraphNode per node. */ - void insertGraphNodes(uint64_t project_id, - const std::vector &nodes); - - bool deleteGraphNodesByFile(uint64_t project_id, const char *file_path); - // ── Graph Edges ──────────────────────────────────────────── uint64_t insertGraphEdge(uint64_t project_id, const graph::GraphEdge &edge); - /** Batch insert graph edges — prepare once, bind/step/reset in a loop. */ - void insertGraphEdges(uint64_t project_id, - const std::vector &edges); - bool deleteGraphEdgesByFile(uint64_t project_id, const char *file_path); /** @@ -156,17 +162,6 @@ class GraphStore { */ bool deleteGraphDataByFile(uint64_t project_id, const char *file_path); - /** - * Delete ALL data for a file across every table: semantic_records, - * relation, graph_edges, graph_nodes, and entity. Used by index_file - * to ensure a clean slate before re-inserting parsed data. - * - * @param project_id Project identifier. - * @param file_path Source file path. - * @return true on success. - */ - bool deleteFileData(uint64_t project_id, const char *file_path); - // ── Entity/Relation (Phase 1.1) ───────────────────────── uint64_t insertEntity(uint64_t project_id, @@ -189,20 +184,6 @@ class GraphStore { const std::string &file_path, const std::vector &records); - /** - * Batch insert semantic records for MULTIPLE files in one call. - * Prepares the SQL statement ONCE and reuses it for all records - * across all files — significantly reducing per-file prepare/finalize - * overhead compared to calling insertSemanticRecords per file. - * - * @param project_id Project identifier. - * @param file_records Vector of (file_path, records) pairs. - */ - void insertSemanticRecordsBatch( - uint64_t project_id, - const std::vector>> - &file_records); - /** * Streaming pipeline: batch-insert a complete FileResult bundle. * @@ -223,11 +204,13 @@ class GraphStore { bool is_reindex = true); /** - * Resolve pre-computed metrics from the staging temp table into - * the metrics and symbol_status tables. + * Resolve pre-computed metrics from the staging temp table onto the + * matching entity rows. * - * Must be called AFTER buildGraph + populateSymbolsFromGraph, because - * metrics reference symbol IDs which are created during populate. + * Must be called after the entities exist: each staged row is matched to + * its entity by (project_id, file_path, start_row, start_col), so a row + * staged before its entity exists is not applied. Only function/method + * entities (kind 0/1) carry code metrics. * * Does a single SQL JOIN to resolve all staged metrics in O(n) time. * @@ -237,14 +220,45 @@ class GraphStore { bool resolveStagedMetrics(uint64_t project_id); /** - * Populate the symbols table from graph_nodes for a project. - * Called after buildGraph to create symbol entries with node_id - * back-references for cross-file edge copy. + * Snapshot the project's current metric columns before a rebuild. * - * @param project_id Project identifier. - * @return Number of symbols inserted. + * buildGraph deletes the entity rows of every file it rebuilds and + * re-inserts them from semantic_records, whose column list carries a + * declaration's identity but not its derived metrics. Without this + * snapshot those values (cyclomatic, cognitive, nesting_depth, lines, ...) + * are lost on every rebuild — including the post-index pass the parallel + * scheduler runs on the merged database, which is why every project + * indexed with `index-parallel` reported 0 metrics while its per-module + * worker databases held the real numbers. + * + * Values go into the temp table `_prev_metrics`, keyed exactly the way + * resolveStagedMetrics keys staged rows: (project_id, file_path, + * start_row, start_col). Must be called after the rebuild set is known and + * BEFORE those rows are deleted. + * + * @param project_id Project identifier. + * @return true on success. false is logged; the caller can still rebuild, + * it just cannot restore the previous metric values. + */ + bool snapshotMetricsForRebuild(uint64_t project_id); + + /** + * Re-apply metric columns after a rebuild re-inserted the entity rows. + * + * Two steps in order: restore the pre-rebuild values from `_prev_metrics`, + * then overlay freshly staged values from `_staged_metrics` (a fresh + * producer result is newer than a snapshot). The staged rows are consumed + * afterwards, so a path that never calls resolveStagedMetrics — the + * file-list path (engine_index_files) — cannot leave them behind. + * + * Must be called after snapshotMetricsForRebuild() and after the entity + * rows exist again. + * + * @param project_id Project identifier. + * @return true on success; false is logged and means the caller should + * report that metrics may be incomplete. */ - int64_t populateSymbolsFromGraph(uint64_t project_id); + bool applyMetricsAfterRebuild(uint64_t project_id); /** * Build the knowledge graph from previously stored semantic records. @@ -258,11 +272,21 @@ class GraphStore { * only symbol/containment graph (~2x faster). * @param changed_files If non-null, only rebuild graph for these files * (incremental mode). When null, rebuilds all. + * @param include_test_files Promote declarations from test/bench/spec + * files too (default false). The automatic index + * path drops them on purpose — "AI only needs + * production code" — but a caller that explicitly + * bypassed the discovery skip rules + * (`force_index_files`) asked for exactly those + * files, and leaving the filter on made the tool + * report `ok:true, files_indexed:115` while the + * graph gained nothing. * @return true on success. */ bool buildGraph( uint64_t project_id, bool build_calls = true, - const std::unordered_set *changed_files = nullptr); + const std::unordered_set *changed_files = nullptr, + bool include_test_files = false); // ── CSR Adjacency (BLOB-packed call edges) ────────────────── @@ -289,11 +313,6 @@ class GraphStore { // ── FTS index helpers ─────────────────────────────────────── - void insertIntoFTS(uint64_t node_id, uint64_t project_id, - const char *name, const char *qualified_name, - const char *file_path, const char *content, - int node_kind = -1); - /** * Bulk-build FTS index from graph_nodes for a project. * Scans all graph_nodes with a non-empty name and inserts into @@ -304,10 +323,6 @@ class GraphStore { // ── Vector search (semantic) ───────────────────────────────── - bool storeVector(uint64_t node_id, uint64_t project_id, - const void *vec_data, size_t vec_bytes); - std::string searchSemantic(uint64_t project_id, const void *query_vec, - size_t vec_bytes, int limit); // deleteFTSByFile removed — FTS is built inline during buildGraph /** @@ -319,9 +334,6 @@ class GraphStore { // ── Complexity ─────────────────────────────────────────────── - bool setComplexity(uint64_t project_id, uint64_t graph_node_id, - uint64_t cyclomatic, uint64_t cognitive, - uint64_t nesting_depth, uint64_t decision_points); std::string getComplexityJson(uint64_t project_id, uint64_t graph_node_id); @@ -335,25 +347,6 @@ class GraphStore { const char *name, const char *path, const char *language); - // ── New Schema (Phase A): Symbols ───────────────────────── - - /** - * Insert a symbol (fast scan result). - * Automatically creates a corresponding row in symbol_status. - * Returns the symbol id. - */ - uint64_t insertSymbol(uint64_t project_id, uint64_t module_id, - const char *kind, const char *name, - const char *signature, const char *visibility, - const char *language, const char *file_path, - int line, int column, int span_start, - int span_end); - - // ── New Schema (Phase A): Entry Points ──────────────────── - - bool insertEntryPoint(uint64_t symbol_id, uint64_t project_id, - const char *kind); - // ── New Schema (Phase A): Queries ───────────────────────── /** Get module tree as JSON. */ @@ -362,39 +355,13 @@ class GraphStore { /** Find symbols by name as JSON. */ std::string findSymbolJson(uint64_t project_id, const char *name); - // ── Phase B: Enhancement — Call Edges ───────────────────── - - uint64_t insertCallEdge(uint64_t project_id, uint64_t caller_symbol_id, - uint64_t callee_symbol_id, - const char *provenance, int line, int col); - - // ── Phase B: Enhancement — Embeddings ───────────────────── - - bool insertEmbedding(uint64_t symbol_id, const float *vector_data, - int dim); - // ── Symbol Status (separate from symbols, keeps main table lean) ── - /** Mark a symbol's callgraph and metrics as ready (embedding is separate - * because it may fail when vec0 is unavailable). */ - bool markCallgraphAndMetricsReady(uint64_t symbol_id); - /** Mark a symbol's embedding as ready. Only call after insertEmbedding - * succeeded, so that failed embeddings can self-heal on rerun. */ - bool markEmbeddingReady(uint64_t symbol_id); - bool setSymbolStub(uint64_t symbol_id, bool is_stub); - - /** Get files where symbols have a status flag = 0 (not ready). */ - std::vector getUnreadyFiles(uint64_t project_id, - const char *ready_field); - /** Get ratio of symbols with a status flag = 1 (0.0 - 1.0). */ - double getReadyRatio(uint64_t project_id); double getReadyRatio(uint64_t project_id, const char *ready_field); // ── Incremental Indexing ────────────────────────────────── - bool isFileUnchanged(uint64_t project_id, const char *file_path, - int64_t mtime, int64_t size); /** Load all (file_path, mtime, size) tuples for a project into an * in-memory set for O(1) membership checks during file discovery. * Replaces N per-file isFileUnchanged queries with a single SELECT. @@ -403,45 +370,9 @@ class GraphStore { * @return unordered_set of "path|mtime|size" strings. */ std::unordered_set loadFileScanStateBatch(uint64_t project_id); - void updateFileScanState(uint64_t project_id, const char *file_path, - int64_t mtime, int64_t size); void cleanupStaleFiles(uint64_t project_id, const std::vector &existing_files); - // ── Path Tracing (BFS on call_edges) ────────────────────── - - /** - * Trace the shortest call path between two functions using BFS on call_edges. - * Returns JSON: {"path": [{"name":"...","file":"...","line":N}, ...]} - * Requires callgraph_ready (enhancement must be run first). - */ - std::string tracePathJson(uint64_t project_id, const char *from_name, - const char *to_name); - - /** - * Explore a function's callers/callees recursively as a JSON tree. - * Returns hierarchical JSON: {"name":"...","file":"...","line":N,"callers":[...],"callees":[...]} - * Each level nests up to `depth` levels (0 = just the function metadata). - * @param function_name Starting function. - * @param depth How many levels to recurse (max 5). - * @param direction "callers", "callees", or "both". - */ - std::string exploreFunctionJson(uint64_t project_id, - const char *function_name, int depth, - const char *direction); - - // ── Index Tasks (Tokio background task tracking) ────────── - - /** Create a new task record. Returns task id. */ - uint64_t createTask(uint64_t project_id, const char *task_type); - - /** Update task status and progress. */ - bool updateTask(uint64_t task_id, const char *status, int progress, - const char *error); - - /** Get latest task for a project as JSON. */ - std::string getTaskStatusJson(uint64_t project_id); - // ── Phase C: Unified Queries (adaptive) ─────────────────── /** @@ -531,34 +462,7 @@ class GraphStore { int saved_cache_ = -2000; // ~2 MB default }; - /** - * Find callers from the new call_edges table (requires callgraph_ready). - * Returns JSON array of caller symbols. - * - * @param file_filter Optional absolute file path. When non-empty, - * restricts callee node matching to the given file, - * disambiguating homonyms (same name across files/classes, - * e.g. __init__, run, main). Empty = aggregate all files - * (legacy behavior, may produce noise on common names). - */ - std::string findCallersJson(uint64_t project_id, - const char *symbol_name, - const char *file_filter = nullptr); - - /** - * Find callees from the new call_edges table (requires callgraph_ready). - * Returns JSON array of callee symbols. - * - * @param file_filter Optional absolute file path. When non-empty, - * restricts caller node matching to the given file, - * disambiguating homonyms. See findCallersJson doc above. - */ - std::string findCalleesJson(uint64_t project_id, - const char *symbol_name, - const char *file_filter = nullptr); - /** Get entry points from the new entry_points table. */ - std::string getEntryPointsJson(uint64_t project_id); // ── On-demand call graph queries (from semantic_records, not pre-built) ── @@ -626,37 +530,6 @@ class GraphStore { // ── Type Registry ───────────────────────────────────────────── - /** - * Batch-insert type_info rows (type definitions). - * Each entry contains: name, qualified_name, kind, file_path, language, location. - * Uses a single multi-VALUES INSERT for efficiency. - * - * @param project_id Project identifier. - * @param rows Vector of (name, qualified_name, kind, file_path, language, - * start_row, start_col, end_row, end_col) tuples. - * @return true on success. - */ - bool insertTypeInfoBatch( - uint64_t project_id, - const std::vector< - std::tuple> &rows); - - /** - * Batch-insert type_ref rows (type references). - * Each entry contains: entity_id, type_name, kind, file_path, location. - * Uses a single multi-VALUES INSERT for efficiency. - * - * @param project_id Project identifier. - * @param rows Vector of (entity_id, type_name, kind, file_path, - * start_row, start_col) tuples. - * @return true on success. - */ - bool insertTypeRefBatch( - uint64_t project_id, - const std::vector> &rows); - sqlite3 *handle() const { return db_; @@ -679,9 +552,6 @@ class GraphStore { /** Get a readiness flag for a project, or 0 if not set. */ int getProjectReadiness(uint64_t project_id, const char *field); - /** Get all readiness flags as JSON: "fast_ready":1,"fts_ready":0,... */ - std::string getProjectReadinessJson(uint64_t project_id); - // ── Shared Artifact ───────────────────────────────────────── /** @@ -708,12 +578,24 @@ class GraphStore { const std::string &name, int64_t scope_id, int arity, int start_row, int start_col, int call_kind = 0); - int64_t insertScope(uint64_t project_id, int64_t parent_id, int kind, - const std::string &name, int start_row, - int end_row); - int64_t insertImport(uint64_t project_id, int64_t source_scope_id, - const std::string &target_path, - const std::string &alias, int is_pub); + + /** + * Delete one document row so a re-ingest replaces it instead of + * accumulating duplicates. + * + * The drift tools concatenate every README row of a project before + * extracting language claims, so a duplicate would silently double a + * claim's weight. Ingesting the root README on every index run (see + * engine_index_discover::ingestProjectRootReadme) therefore deletes the + * previous row first. + * + * @param project_id Project the document belongs to. + * @param type Document type (0 == README / knowledge document). + * @param file_path Exact path the document was stored under. + * @return true on success, including when no row matched. + */ + bool deleteDocument(uint64_t project_id, int type, + const std::string &file_path); bool insertDocument(uint64_t project_id, int type, const std::string &file_path, @@ -724,8 +606,8 @@ class GraphStore { bool insertWorkflowStep(int64_t workflow_id, int step_order, int64_t entity_id, const std::string &label); bool insertArchitectureEdge(uint64_t project_id, - const std::string &layer_upper, - const std::string &layer_lower, + const std::string &caller_module, + const std::string &callee_module, int64_t entity_id); bool insertCapability(uint64_t project_id, const std::string &name, @@ -766,11 +648,6 @@ class GraphStore { const std::string &description, double confidence); - /** Delete all knowledge/evidence rows for a project. - * Deletes in FK order: evidence_fact -> evidence -> finding -> - * claim -> contract -> capability. Returns true on success. */ - bool clearProjectKnowledge(uint64_t project_id); - /** List (id, name) pairs of capabilities for a project. */ std::vector> listCapabilities(uint64_t project_id); @@ -810,11 +687,6 @@ class GraphStore { std::string error_; std::string db_path_; - // Cached prepared statements (initialized in open(), finalized in close()) - sqlite3_stmt *stmt_fts_map_ = nullptr; // INSERT INTO fts_node_map - sqlite3_stmt *stmt_fts_ = nullptr; // INSERT INTO code_fts - sqlite3_stmt *stmt_vector_ = nullptr; // INSERT INTO node_vectors - // Dynamic statement cache: one prepared statement PER THREAD (see // getCachedStmt in store.cpp). A single sqlite3_stmt* is never shared // across threads, so there is no cross-thread race on a shared cached @@ -838,6 +710,22 @@ class GraphStore { private: bool createSchema(); + + /** Patch columns added in later versions onto a database that already + * exists. SQLite has no "ADD COLUMN IF NOT EXISTS", so createSchema() + * creates the main DDL and delegates every per-column migration here + * (own TU: the migrations outgrew the 1000-line rule). + * @return false on a hard failure; individual failures are logged. */ + bool runSchemaMigrations(); + + /** Migration group for the route / type_info / type_ref tables (own TU: + * the migration pass outgrew the 1000-line rule — see + * store_schema_migrations_types.cpp). + * @param record Runs one statement and records its failure together with + * the failing SQL; see runSchemaMigrations. + * @return false when any statement failed (each failure has already been + * reported through `record`). */ + bool migrateTypeTables(const std::function &record); }; // ── Index Progress (global, for client polling) ───────────── diff --git a/engine/src/store/store_batch.cpp b/engine/src/store/store_batch.cpp index 065bb3a..33689da 100644 --- a/engine/src/store/store_batch.cpp +++ b/engine/src/store/store_batch.cpp @@ -100,169 +100,26 @@ void GraphStore::insertSemanticRecords(uint64_t project_id, SQLITE_STATIC); int rc = sqlite3_step(stmt); - if (rc != SQLITE_DONE) + if (rc != SQLITE_DONE) { + // Surface the failure on the store instead of only logging + // it: a silent step error left a partially-written file with + // error() empty (code_rules §5). This entry point is + // test-only now (production uses insertFileResultBatch), but + // the same contract applies. + error_ = + std::string( + "insertSemanticRecords: step failed: ") + + sqlite3_errmsg(db_); fprintf(stderr, - "insertSemanticRecords: step error %d: %s\n", + "insertSemanticRecords: step error %d: %s " + "[module=store, method=insertSemanticRecords]\n", rc, sqlite3_errmsg(db_)); + break; + } sqlite3_reset(stmt); } sqlite3_finalize(stmt); } - -void GraphStore::insertSemanticRecordsBatch( - uint64_t project_id, - const std::vector>> - &file_records) -{ - // Count total records to pre-compute size - size_t total = 0; - for (auto &fr : file_records) - total += fr.second.size(); - if (total == 0) - return; - - constexpr size_t kBatchSize = 500; - // 23 columns in semantic_records: original_id, project_id, kind, - // name, qualified_name, parent_id, ref_original_id, arity, - // is_static, type_name, call_kind, resolve_strategy, visibility, - // start_row, start_col, end_row, end_col, file_path, language, - // qualified_target, receiver_text, receiver_type, import_alias. - // (Step 3 added the last 4 call-fact columns.) Must match the - // column count AND the placeholder count below. - constexpr int kColsPerRow = 23; - - // Step 1: Flatten records into a contiguous vector for efficient batching. - // Each element stores (file_path, record_index) to reference the original. - struct FlatRecord { - const ir::Record *rec; - const std::string *file_path; - }; - std::vector flat; - flat.reserve(total); - for (auto &fr : file_records) - for (auto &r : fr.second) - flat.push_back({ &r, &fr.first }); - - // Step 2: Process in batches using multi-VALUES INSERT. - // Build SQL: INSERT INTO t VALUES (?,?,...), (?,?,...), ... - // This reduces prepare/bind/step/reset overhead by ~13x per batch. - size_t offset = 0; - while (offset < flat.size()) { - size_t batch = flat.size() - offset; - if (batch > kBatchSize) - batch = kBatchSize; - - // Build multi-VALUES SQL - std::string sql = "INSERT INTO semantic_records " - "(original_id, project_id, kind, name, " - "qualified_name, parent_id, ref_original_id, " - "arity, is_static, type_name, call_kind, " - "resolve_strategy, visibility, " - "start_row, start_col, end_row, end_col, " - "file_path, language, " - "qualified_target, receiver_text, " - "receiver_type, import_alias) VALUES "; - for (size_t i = 0; i < batch; i++) { - if (i > 0) - sql += ","; - sql += "(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"; - } - - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql.c_str(), -1, &stmt, nullptr) != - SQLITE_OK) { - error_ = "insertSemanticRecordsBatch: prepare failed"; - return; - } - - // Build intra-file declaration maps for ref_original_id - std::unordered_map decl_by_name; - for (size_t i = 0; i < batch; i++) { - auto &fr = flat[offset + i]; - auto k = static_cast(fr.rec->kind); - if ((k == 0 || k == 1) && !fr.rec->name.empty()) - decl_by_name[fr.rec->name] = fr.rec->id; - } - - // Bind all rows in batch - for (size_t i = 0; i < batch; i++) { - auto &fr = flat[offset + i]; - auto &r = *fr.rec; - int base = static_cast(i * kColsPerRow); - - uint64_t ref_id = 0; - if (static_cast(r.kind) == 9 && !r.name.empty()) { - auto it = decl_by_name.find(r.name); - if (it != decl_by_name.end()) - ref_id = it->second; - } - - sqlite3_bind_int64(stmt, base + 1, - static_cast(r.id)); - sqlite3_bind_int64(stmt, base + 2, - static_cast(project_id)); - sqlite3_bind_int(stmt, base + 3, - static_cast(r.kind)); - sqlite3_bind_text(stmt, base + 4, r.name.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, base + 5, - r.qualified_name.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_int64(stmt, base + 6, - static_cast(r.parent_id)); - sqlite3_bind_int64(stmt, base + 7, - static_cast(ref_id)); - sqlite3_bind_int(stmt, base + 8, r.arity); - sqlite3_bind_int(stmt, base + 9, r.is_static ? 1 : 0); - sqlite3_bind_text(stmt, base + 10, r.type_name.c_str(), - -1, SQLITE_STATIC); - sqlite3_bind_int(stmt, base + 11, - static_cast(r.call_kind)); - sqlite3_bind_text(stmt, base + 12, - r.resolve_strategy.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_int(stmt, base + 13, r.visibility); - sqlite3_bind_int(stmt, base + 14, - static_cast(r.loc.start_row)); - sqlite3_bind_int(stmt, base + 15, - static_cast(r.loc.start_col)); - sqlite3_bind_int(stmt, base + 16, - static_cast(r.loc.end_row)); - sqlite3_bind_int(stmt, base + 17, - static_cast(r.loc.end_col)); - sqlite3_bind_text(stmt, base + 18, - fr.file_path->c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, base + 19, r.language.c_str(), - -1, SQLITE_STATIC); - // Step 3: bind structured call facts. - sqlite3_bind_text(stmt, base + 20, - r.qualified_target.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, base + 21, - r.receiver_text.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, base + 22, - r.receiver_type.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, base + 23, - r.import_alias.c_str(), -1, - SQLITE_STATIC); - } - - int rc = sqlite3_step(stmt); - if (rc != SQLITE_DONE) - fprintf(stderr, - "insertSemanticRecordsBatch: step error %d: %s " - "(batch %zu-%zu)\n", - rc, sqlite3_errmsg(db_), offset, - offset + batch); - sqlite3_finalize(stmt); - - offset += batch; - } -} - // ─── Streaming Pipeline ───────────────────────────────────────── bool GraphStore::insertFileResultBatch(uint64_t project_id, @@ -272,8 +129,6 @@ bool GraphStore::insertFileResultBatch(uint64_t project_id, if (batch.empty()) return true; - // _staged_metrics temp table removed — metrics no longer stored. - // ── Prepare statements ───────────────────────────────────── const char *sr_sql = "INSERT INTO semantic_records " @@ -366,7 +221,22 @@ bool GraphStore::insertFileResultBatch(uint64_t project_id, SQLITE_TRANSIENT); sqlite3_bind_text(file_st, 3, fr.language.c_str(), -1, SQLITE_TRANSIENT); - sqlite3_step(file_st); + // The files row is the per-file record: a silent step + // failure would leave the file untracked while its + // semantic_records are still inserted below. + if (sqlite3_step(file_st) != SQLITE_DONE) { + error_ = + "[module=store, method=" + "insertFileResultBatch] INSERT INTO files " + "failed: " + + std::string(sqlite3_errmsg(db_)); + sqlite3_reset(file_st); + sqlite3_finalize(file_st); + sqlite3_finalize(fss_st); + sqlite3_finalize(del_sr_st); + sqlite3_finalize(sr_st); + return false; + } sqlite3_reset(file_st); } @@ -383,7 +253,20 @@ bool GraphStore::insertFileResultBatch(uint64_t project_id, sqlite3_bind_text(fss_st, 5, ch.c_str(), -1, SQLITE_TRANSIENT); } - sqlite3_step(fss_st); + // file_scan_state drives incremental re-index freshness; a + // silent failure would make the next run re-parse the file + // (or, worse, believe a changed file is unchanged). + if (sqlite3_step(fss_st) != SQLITE_DONE) { + error_ = "[module=store, method=insertFileResultBatch] " + "INSERT INTO file_scan_state failed: " + + std::string(sqlite3_errmsg(db_)); + sqlite3_reset(fss_st); + sqlite3_finalize(fss_st); + sqlite3_finalize(file_st); + sqlite3_finalize(del_sr_st); + sqlite3_finalize(sr_st); + return false; + } sqlite3_reset(fss_st); // Delete old semantic_records for this file to prevent @@ -395,7 +278,22 @@ bool GraphStore::insertFileResultBatch(uint64_t project_id, static_cast(project_id)); sqlite3_bind_text(del_sr_st, 2, fr.file_path.c_str(), -1, SQLITE_TRANSIENT); - sqlite3_step(del_sr_st); + // A failed DELETE leaves the previous run's rows in + // place; the plain INSERT below would then append + // duplicates that buildGraph turns into duplicate + // entities. Fail instead of accumulating. + if (sqlite3_step(del_sr_st) != SQLITE_DONE) { + error_ = "[module=store, method=" + "insertFileResultBatch] DELETE FROM " + "semantic_records failed: " + + std::string(sqlite3_errmsg(db_)); + sqlite3_reset(del_sr_st); + sqlite3_finalize(del_sr_st); + sqlite3_finalize(fss_st); + sqlite3_finalize(file_st); + sqlite3_finalize(sr_st); + return false; + } sqlite3_reset(del_sr_st); } @@ -415,6 +313,12 @@ bool GraphStore::insertFileResultBatch(uint64_t project_id, // Step 2: Batch-insert semantic_records via multi-VALUES // Simple flat iteration: collect (record_ptr, file_path) for all records + // + // The DELETE above already removed each file's previous rows, so a failed + // INSERT here would commit a file that has no records at all — buildGraph + // would then see the file but none of its symbols. Track the failure and + // report it instead of letting the caller commit a half-written batch. + bool records_write_ok = true; if (!batch_records.empty()) { struct RecRef { const ir::Record *rec; @@ -471,7 +375,17 @@ bool GraphStore::insertFileResultBatch(uint64_t project_id, sqlite3_stmt *batch_st = nullptr; if (sqlite3_prepare_v2(db_, sql.c_str(), -1, &batch_st, - nullptr) == SQLITE_OK) { + nullptr) != SQLITE_OK) { + // Prepare failure after DELETE: without this + // branch the function returned true with old + // rows already gone and new rows never written. + error_ = "[module=store, method=" + "insertFileResultBatch] " + "semantic_records batch prepare " + "failed: " + + std::string(sqlite3_errmsg(db_)); + records_write_ok = false; + } else { for (size_t i = 0; i < batch_sz; i++) { auto &r = *all_recs[off + i].rec; int base = static_cast(i * 23); @@ -564,11 +478,17 @@ bool GraphStore::insertFileResultBatch(uint64_t project_id, SQLITE_STATIC); } int rc = sqlite3_step(batch_st); - if (rc != SQLITE_DONE) - fprintf(stderr, - "insertFileResultBatch: records " - "multi-VALUES step %d: %s\n", - rc, sqlite3_errmsg(db_)); + if (rc != SQLITE_DONE) { + error_ = "[module=store, method=" + "insertFileResultBatch] " + "semantic_records batch step " + "failed: " + + std::string( + sqlite3_errmsg(db_)); + sqlite3_finalize(batch_st); + records_write_ok = false; + break; + } sqlite3_finalize(batch_st); } } @@ -655,7 +575,9 @@ bool GraphStore::insertFileResultBatch(uint64_t project_id, // Graph data is stored in SQLite: semantic_records → buildGraph → // graph_nodes/edges is the source of truth for all graph queries. - return true; + // Fail-closed: a failed records batch must not be committed by the caller + // (see records_write_ok above). + return records_write_ok; } // Resolve pre-computed metrics from the _staged_metrics staging table onto @@ -668,8 +590,7 @@ bool GraphStore::insertFileResultBatch(uint64_t project_id, // // Metrics are real measurements (cyclomatic/cognitive/nesting) produced in the // parse worker — see engine_index_metrics.cpp. This restores the metrics -// capability that Step 10 of ACCURACY_IMPROVEMENT_DEVELOPMENT_PLAN.md had -// sunset; the plan's completion criterion (no placeholder 0, real data) is met +// capability that had been sunset; the criterion (no placeholder 0, real data) is met // because we write the actual computed values and mark the project's // metrics_ready flag from the canonical entity coverage. bool GraphStore::resolveStagedMetrics(uint64_t project_id) @@ -734,7 +655,12 @@ bool GraphStore::resolveStagedMetrics(uint64_t project_id) "DELETE FROM _staged_metrics WHERE project_id = ?"; if (sqlite3_prepare_v2(db_, del_sql, -1, &del, nullptr) == SQLITE_OK) { sqlite3_bind_int64(del, 1, static_cast(project_id)); - sqlite3_step(del); + if (sqlite3_step(del) != SQLITE_DONE) { + fprintf(stderr, + "resolveStagedMetrics: cleanup step failed: %s " + "[module=store, method=resolveStagedMetrics]\n", + sqlite3_errmsg(db_)); + } sqlite3_finalize(del); } else { fprintf(stderr, diff --git a/engine/src/store/store_core.cpp b/engine/src/store/store_core.cpp index 3c57793..9406471 100644 --- a/engine/src/store/store_core.cpp +++ b/engine/src/store/store_core.cpp @@ -1,4 +1,6 @@ #include "store.h" +#include "util/json_writer.h" +#include "util/path_util.h" #include "platform_win.h" #include "posix_compat.h" @@ -13,6 +15,7 @@ #include #include #include +#include #include #ifndef _WIN32 #include @@ -27,6 +30,11 @@ namespace store // Performance PRAGMA values static constexpr int kCacheSizePages = -64000; +// Retried sqlite3_open failures (SQLITE_BUSY / SQLITE_LOCKED under a +// concurrent writer). 5 x 50ms covers the transient window without +// stalling a genuinely unavailable file. +static constexpr int kOpenRetryAttempts = 5; +static constexpr int kOpenRetryDelayMs = 50; // 64 MB cache static constexpr int kPageSize = @@ -141,7 +149,7 @@ bool GraphStore::open(const char *db_path) { // Enable SQLite serialized threading mode so the same handle can be // used from multiple threads safely (worker pool in engine_index.cpp - // + MCP server main loop both access g_store concurrently). + // + MCP server main loop both access the engine store concurrently). // This MUST be called before any sqlite3_open() call. // If already initialized (SQLITE_MISUSE), the default is likely // serialized already (most builds have SQLITE_THREADSAFE=1). @@ -157,11 +165,31 @@ bool GraphStore::open(const char *db_path) config_rc); } - int rc = sqlite3_open(db_path, &db_); - if (rc != SQLITE_OK) { + // Bounded retry: sqlite3_open fails transiently with SQLITE_BUSY / + // SQLITE_LOCKED when another connection holds the write lock (e.g. two + // CLI invocations over the same DB). busy_timeout cannot help — it only + // applies to statements after the handle is open. + int rc = SQLITE_BUSY; + for (int attempt = 0; attempt < kOpenRetryAttempts; ++attempt) { + rc = sqlite3_open(db_path, &db_); + if (rc == SQLITE_OK) + break; error_ = sqlite3_errmsg(db_); - return false; + // sqlite3_open returns a handle even on failure; close it before + // the next attempt or the retries leak handles. + sqlite3_close(db_); + db_ = nullptr; + if (rc != SQLITE_BUSY && rc != SQLITE_LOCKED) + return false; + std::this_thread::sleep_for( + std::chrono::milliseconds(kOpenRetryDelayMs)); } + if (rc != SQLITE_OK) + return false; + // A retried attempt leaves its failure text in error_; open() has now + // succeeded, so error() must reflect THIS call (pass-8 contract — + // same rationale as buildFTSFromGraph's entry clear). + error_.clear(); db_path_ = db_path; // page_size MUST be set before any tables are created. Larger pages @@ -241,39 +269,12 @@ bool GraphStore::open(const char *db_path) if (!createSchema()) return false; - // Pre-cache prepared statements for hot insert paths - sqlite3_prepare_v2( - db_, - "INSERT OR REPLACE INTO fts_node_map (node_id, project_id, file_id) " - "VALUES (?, ?, 0)", - -1, &stmt_fts_map_, nullptr); - sqlite3_prepare_v2( - db_, - "INSERT OR REPLACE INTO code_fts (rowid, name, qualified_name, " - "file_path, content, project_id, node_id, node_kind) " - "VALUES (?, ?, ?, ?, ?, ?, ?, ?)", - -1, &stmt_fts_, nullptr); - sqlite3_prepare_v2( - db_, - "INSERT OR REPLACE INTO node_vectors (node_id, project_id, vector) " - "VALUES (?, ?, ?)", - -1, &stmt_vector_, nullptr); - return true; } void GraphStore::close() { if (db_) { - // Finalize cached prepared statements - if (stmt_fts_map_) - sqlite3_finalize(stmt_fts_map_); - if (stmt_fts_) - sqlite3_finalize(stmt_fts_); - if (stmt_vector_) - sqlite3_finalize(stmt_vector_); - stmt_fts_map_ = stmt_fts_ = stmt_vector_ = nullptr; - // Finalize all dynamically cached statements clearStmtCache(); @@ -338,37 +339,7 @@ IndexProgress getIndexProgress() // Handles ", \, and control characters per RFC 8259. static std::string jsonEscapeForProgress(const std::string &s) { - std::string out; - out.reserve(s.size() + 8); - for (char c : s) { - switch (c) { - case '"': - out += "\\\""; - break; - case '\\': - out += "\\\\"; - break; - case '\n': - out += "\\n"; - break; - case '\r': - out += "\\r"; - break; - case '\t': - out += "\\t"; - break; - default: - if (static_cast(c) < 0x20) { - char buf[8]; - snprintf(buf, sizeof(buf), "\\u%04x", - static_cast(c)); - out += buf; - } else { - out += c; - } - } - } - return out; + return util::jsonEscapeString(s); } std::string getIndexProgressJson(uint64_t project_id) @@ -396,15 +367,31 @@ void GraphStore::setProjectReadiness(uint64_t project_id, const char *field, if (!field || allowed_fields.find(field) == allowed_fields.end()) return; + // exec() sets error_ on failure but never clears it on success; clear + // here so error() reflects only THIS call (callers chain several + // readiness writes and check error() between them). + error_.clear(); + // Ensure the readiness row exists - exec(std::string( - "INSERT OR IGNORE INTO project_readiness (project_id) VALUES (" + - std::to_string(project_id) + ")") - .c_str()); - exec(std::string("UPDATE project_readiness SET " + std::string(field) + - "=" + std::to_string(value) + - " WHERE project_id=" + std::to_string(project_id)) - .c_str()); + if (!exec(std::string( + "INSERT OR IGNORE INTO project_readiness (project_id) VALUES (" + + std::to_string(project_id) + ")") + .c_str())) { + error_ = "[module=store, method=setProjectReadiness] insert: " + + error_; + return; + } + if (!exec(std::string("UPDATE project_readiness SET " + + std::string(field) + "=" + std::to_string(value) + + " WHERE project_id=" + std::to_string(project_id)) + .c_str())) { + error_ = "[module=store, method=setProjectReadiness] update: " + + error_; + fprintf(stderr, + "setProjectReadiness: update %s failed: %s " + "[module=store, method=setProjectReadiness]\n", + field, error_.c_str()); + } } int GraphStore::getProjectReadiness(uint64_t project_id, const char *field) @@ -430,33 +417,6 @@ int GraphStore::getProjectReadiness(uint64_t project_id, const char *field) sqlite3_finalize(stmt); return val; } - -std::string GraphStore::getProjectReadinessJson(uint64_t project_id) -{ - sqlite3_stmt *stmt = nullptr; - const char *sql = "SELECT fast_ready, normal_ready, deep_ready, " - "fts_ready, vector_ready " - "FROM project_readiness WHERE project_id=?"; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) - return ""; - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - std::string json; - if (sqlite3_step(stmt) == SQLITE_ROW) { - json = "\"fast_ready\":" + - std::to_string(sqlite3_column_int(stmt, 0)) + - ",\"normal_ready\":" + - std::to_string(sqlite3_column_int(stmt, 1)) + - ",\"deep_ready\":" + - std::to_string(sqlite3_column_int(stmt, 2)) + - ",\"fts_ready\":" + - std::to_string(sqlite3_column_int(stmt, 3)) + - ",\"vector_ready\":" + - std::to_string(sqlite3_column_int(stmt, 4)); - } - sqlite3_finalize(stmt); - return json; -} - // ─── Shared Artifact ───────────────────────────────────────────── // Uses fork() + execvp() for zstd to avoid command injection via system(). // Validates paths reject single-quote chars to prevent SQL injection in VACUUM/ATTACH. @@ -678,21 +638,15 @@ static std::string normalizeRootPath(const char *root_path) { if (!root_path || !*root_path) return {}; - namespace fs = std::filesystem; - std::error_code ec; - // weakly_canonical resolves symlinks AND makes the path absolute, - // even if the path doesn't fully exist (it canonicalizes the - // existing prefix). This handles ".", "./foo", "foo/../bar", - // and absolute paths uniformly. - fs::path can = fs::weakly_canonical(fs::path(root_path), ec); - if (ec) - return root_path; - // Use native string form so macOS / Linux DB rows match exactly. -#ifdef _WIN32 - return can.string(); -#else - return can.native(); -#endif + // util::resolvePath is the single implementation of "absolute, + // symlink-resolved" (util/path_util.h): weakly_canonical resolves the + // prefix that exists and normalises the rest, so ".", "./foo", + // "foo/../bar" and absolute paths are handled uniformly, and it falls back + // to the raw input when resolution fails (non-existent path, permission + // denied) so create-by-name flows still work. Every stored file_path is + // built from this form, which is why readers of those rows must resolve + // their own paths the same way (util::lookupPath). + return util::resolvePath(root_path); } uint64_t GraphStore::createProject(const char *root_path, const char *name) @@ -735,6 +689,29 @@ uint64_t GraphStore::createProject(const char *root_path, const char *name) return getProjectId(effective_path); } +std::string GraphStore::getProjectRootPath(uint64_t project_id) +{ + sqlite3_stmt *stmt = nullptr; + const char *sql = "SELECT root_path FROM projects WHERE id = ?"; + if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) { + fprintf(stderr, + "getProjectRootPath prepare failed: %s " + "[module=store, method=getProjectRootPath]\n", + sqlite3_errmsg(db_)); + return std::string(); + } + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + std::string root; + if (sqlite3_step(stmt) == SQLITE_ROW) { + const char *value = reinterpret_cast( + sqlite3_column_text(stmt, 0)); + if (value) + root = value; + } + sqlite3_finalize(stmt); + return root; +} + uint64_t GraphStore::getProjectId(const char *root_path) { std::string norm = normalizeRootPath(root_path); diff --git a/engine/src/store/store_graph.cpp b/engine/src/store/store_graph.cpp index 3060cf8..d442373 100644 --- a/engine/src/store/store_graph.cpp +++ b/engine/src/store/store_graph.cpp @@ -36,62 +36,17 @@ constexpr int kKindTypeAssign = 18; #include "../graph/graph_builder.h" #include "../ir/semantic_unit.h" +// PathRec + flushImportBatch live in their own TU: this file's 1000-line +// rule (see plan/rules/code_rules.md) is easier to keep when the import +// batch writer is not inlined here. +#include "store_graph_imports.h" -// ── Batch import helper ───────────────────────────────────────── namespace store { -// ── Batch import helper ───────────────────────────────────────── -struct PathRec { - std::string path; - std::string alias; - std::string file; -}; - -static void flushImportBatch(sqlite3 *db, uint64_t project_id, - const std::vector &batch) -{ - if (batch.empty()) - return; - std::string sql = "INSERT OR IGNORE INTO import " - "(project_id, source_scope_id, target_path, alias, " - " file_path, is_pub) VALUES "; - for (size_t i = 0; i < batch.size(); i++) { - if (i > 0) - sql += ","; - sql += "(?,0,?,?,?,0)"; - } - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) != - SQLITE_OK) { - fprintf(stderr, - "[module=store, method=flushImportBatch] " - "prepare failed: %s\n", - sqlite3_errmsg(db)); - return; - } - for (size_t i = 0; i < batch.size(); i++) { - int base = static_cast(i * 4); - sqlite3_bind_int64(stmt, base + 1, - static_cast(project_id)); - sqlite3_bind_text(stmt, base + 2, batch[i].path.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, base + 3, batch[i].alias.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, base + 4, batch[i].file.c_str(), -1, - SQLITE_STATIC); - } - int rc = sqlite3_step(stmt); - if (rc != SQLITE_DONE && rc != SQLITE_CONSTRAINT) - fprintf(stderr, - "[module=store, method=flushImportBatch] " - "step failed (rc=%d): %s\n", - rc, sqlite3_errmsg(db)); - sqlite3_finalize(stmt); -} - bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, - const std::unordered_set *changed_files) + const std::unordered_set *changed_files, + bool include_test_files) { using Clock = std::chrono::steady_clock; @@ -105,11 +60,32 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, // a subset of files are re-indexed — lookup indexes stay valid. const bool full_rebuild = (changed_files == nullptr); - // Step 1: determine which files to rebuild + // Test/bench/spec filter, applied to every graph row derived below + // (entity, reference, import). The automatic index path keeps it: those + // declarations are deliberately kept out of the graph ("AI only needs + // production code"), and the discovery walk already skips the test/ + // directories it recognises. `include_test_files` is the decision of a + // caller that explicitly bypassed that walk — `force_index_files` + // promises "index these paths regardless of the default skip rules", and + // without this switch that promise held for the parse stage only: + // measured on this repository, forcing `engine/tests` added 13174 + // semantic_records rows and **zero** entities, while the tool answered + // {"ok":true,"files_indexed":115}. + const std::string test_file_filter = + include_test_files ? + std::string() : + std::string( + " AND sr.file_path NOT LIKE '%\\_test.%' ESCAPE '\\'" + " AND sr.file_path NOT LIKE '%/tests/%'" + " AND sr.file_path NOT LIKE '%\\_spec.%' ESCAPE '\\'" + " AND sr.file_path NOT LIKE '%/benches/%'" + " AND sr.file_path NOT LIKE '%\\_\\_test\\_\\_%' ESCAPE '\\'"); + + // Step 1: determine which files to rebuild. project_id is bound, not + // concatenated (code_rules §5: one parameterized form for all queries). auto t0 = Clock::now(); std::string file_list_sql = - "SELECT DISTINCT file_path FROM semantic_records WHERE project_id=" + - std::to_string(project_id); + "SELECT DISTINCT file_path FROM semantic_records WHERE project_id=?"; sqlite3_stmt *fl_stmt = nullptr; if (sqlite3_prepare_v2(db_, file_list_sql.c_str(), -1, &fl_stmt, nullptr) != SQLITE_OK) { @@ -117,9 +93,15 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, "buildGraph: prepare file_list failed: %s " "[module=store, method=buildGraph]\n", sqlite3_errmsg(db_)); - // Fall through: empty rebuild_files is a no-op, not a fatal error. - // The transaction is still committed below. + // Fail closed: an empty rebuild_files used to RELEASE the + // savepoint and return true, so the caller committed an index + // with no graph and reported ok:true (code_rules: no silent + // error handling). + if (fl_stmt) + sqlite3_finalize(fl_stmt); + return false; } + sqlite3_bind_int64(fl_stmt, 1, static_cast(project_id)); std::vector rebuild_files; if (fl_stmt) { @@ -150,6 +132,15 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, return true; } + // Metric columns are derived data that this rebuild would otherwise drop: + // the entity INSERT below re-creates rows from semantic_records, whose + // column list carries no metrics. Carry the current values across the + // rebuild (store_metrics.cpp documents the two field failures this fixes: + // the parallel path's post-index rebuild and the file-list path). + // The return value is advisory — a failure is logged and only means the + // values cannot be restored; the graph itself still rebuilds correctly. + snapshotMetricsForRebuild(project_id); + // Delete existing graph data for files being rebuilt. // deleteGraphDataByFile cleans entity, relation (graph_nodes/graph_edges are deprecated) // AND entity — the old code only deleted edges+nodes, leaving entity @@ -161,18 +152,54 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, std::string pid = std::to_string(project_id); + // Critical graph writes are checked. A failed INSERT..SELECT (schema + // drift, SQLITE_BUSY, disk full) used to go unnoticed: buildGraph then + // RELEASEd the savepoint and returned true, so the caller committed a + // partially-populated graph and reported success. `graph_write_ok` + // accumulates failures and forces a ROLLBACK TO SAVEPOINT below. + bool graph_write_ok = true; + auto exec_write = [&](const std::string &sql, const char *step) { + if (exec(sql.c_str())) + return; + graph_write_ok = false; + fprintf(stderr, + "[module=store, method=buildGraph] %s failed: %s\n", + step, error().c_str()); + }; + // ── 2a: Create file filter temp table ── exec("DROP TABLE IF EXISTS _rf"); exec("CREATE TEMP TABLE _rf (file_path TEXT PRIMARY KEY)"); { sqlite3_stmt *ins = nullptr; - sqlite3_prepare_v2( - db_, "INSERT OR IGNORE INTO _rf (file_path) VALUES (?)", - -1, &ins, nullptr); + // Check prepare: bind/step on a NULL stmt is UB (and a failed + // _rf fill makes every `file_path IN (SELECT ... FROM _rf)` + // insert a silent no-op while graph_write_ok stays true). + if (sqlite3_prepare_v2( + db_, + "INSERT OR IGNORE INTO _rf (file_path) VALUES (?)", + -1, &ins, nullptr) != SQLITE_OK) { + fprintf(stderr, + "buildGraph: prepare _rf insert failed: %s " + "[module=store, method=buildGraph]\n", + sqlite3_errmsg(db_)); + exec("ROLLBACK TO SAVEPOINT buildGraph"); + exec("RELEASE SAVEPOINT buildGraph"); + return false; + } for (auto &fp : rebuild_files) { sqlite3_bind_text(ins, 1, fp.c_str(), -1, SQLITE_TRANSIENT); - sqlite3_step(ins); + if (sqlite3_step(ins) != SQLITE_DONE) { + fprintf(stderr, + "buildGraph: _rf insert step failed: " + "%s [module=store, method=buildGraph]\n", + sqlite3_errmsg(db_)); + sqlite3_finalize(ins); + exec("ROLLBACK TO SAVEPOINT buildGraph"); + exec("RELEASE SAVEPOINT buildGraph"); + return false; + } sqlite3_reset(ins); } sqlite3_finalize(ins); @@ -180,13 +207,21 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, auto t_rf = Clock::now(); // ── 2b: Create _r2n mapping table (unsorted for speed) ── - // Note: ROW_NUMBER() OVER () avoids ORDER BY sort cost. - // Node IDs are sequential but not sorted by file_path — sorting is - // not required for correctness since JOINs use indexes, not sequential scans. + // Node ids are assigned by ROW_NUMBER() ordered on each record's + // semantic identity (see the _r2n CREATE below), so the same input + // always yields the same ids. JOINs use indexes, not sequential scans, + // so the ordering itself is irrelevant to query correctness — it only + // makes the assignment reproducible. static constexpr int kR2nKinds[] = { kKindFunction, kKindMethod, kKindClass, kKindInterface, kKindEnum, kKindTypeAlias, kKindTypeDecl, kKindTypeRef, kKindTypeAssign, + // RecordKind::ImportBinding is deliberately NOT here: this list + // decides which kinds are promoted to ENTITIES (the _r2n mapping and + // the entity INSERT below). An import binding is a fact, not a symbol + // — listing it here turned every imported name into a node (measured: + // AIScope 223 → 839 entities). The Resolver reads it from + // semantic_records, which stores every record kind unfiltered. }; static constexpr int kNumR2nKinds = sizeof(kR2nKinds) / sizeof(kR2nKinds[0]); @@ -200,27 +235,33 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, } kind_list += ")"; - // C1 (data-loss fix): in incremental rebuilds (changed_files != null), - // `deleteGraphDataByFile` above only removed entity rows for the files - // being rebuilt, so unchanged files still hold entity.id values starting - // at 1. ROW_NUMBER() OVER () restarts at 1 every call, which collides - // with those retained ids and makes the downstream INSERT OR IGNORE INTO - // entity silently skip the rebuilt file's entities — losing data on every - // incremental run (confirmed: 1477 -> 1476 entities after editing one - // file). Shift new node_ids above the current max entity.id in the - // incremental case; a full rebuild deletes ALL entity rows first, so - // MAX(entity.id) is NULL and the offset is 0. All downstream edges - // (relation, type_ref, import) reference r2n.node_id, so they stay - // consistent with the shifted id automatically. + // C1 (data-loss fix): entity ids are a GLOBAL primary key + // (`entity.id INTEGER PRIMARY KEY`, not per project), while + // deleteGraphDataByFile above removes only the rows of the files being + // rebuilt, so rows with ids starting at 1 are always still there. A bare + // ROW_NUMBER() OVER () restarts at 1 every call, collides with those + // retained ids, and the downstream INSERT OR IGNORE INTO entity then + // silently skips the new rows — losing data on every incremental run + // (confirmed: 1477 -> 1476 entities after editing one file). + // + // The offset must therefore clear the WHOLE table and must be computed on + // every call, not only for incremental rebuilds. The old code applied it + // only when `changed_files != nullptr` and scoped MAX(id) to the project, + // on the assumption that "a full rebuild deletes ALL entity rows first" — + // true for the project being indexed, false for the table. Indexing a + // SECOND project into a database that already holds one therefore started + // at id 1, collided with the first project's ids, and lost every entity + // row of the new project: the project had semantic_records but zero + // entities, so every tool answered "not found" for a project that had just + // indexed successfully (silently — INSERT OR IGNORE reports nothing). + // All downstream edges (relation, type_ref, import) reference + // r2n.node_id, so they stay consistent with the shifted id automatically. long long id_offset = 0; - if (changed_files != nullptr) { + { sqlite3_stmt *mx = nullptr; if (sqlite3_prepare_v2(db_, - "SELECT COALESCE(MAX(id),0) FROM entity " - "WHERE project_id=?", + "SELECT COALESCE(MAX(id),0) FROM entity", -1, &mx, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(mx, 1, - static_cast(project_id)); if (sqlite3_step(mx) == SQLITE_ROW) id_offset = sqlite3_column_int64(mx, 0); sqlite3_finalize(mx); @@ -228,10 +269,19 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, } exec("DROP TABLE IF EXISTS _r2n"); + // Deterministic id assignment. ROW_NUMBER() must be ordered by the + // record's semantic identity, NOT by semantic_records' scan order: + // that order is the insertion order, which varies with parse-worker + // scheduling, so a bare `OVER ()` produced different entity ids for + // identical input (the perf report measured entity.id drift on parallel + // indexes). Everything downstream (relation, type_ref, import, CSR, + // fixture comparisons) then shifted with it. The key is unique for real + // records — one declaration per (file, kind, position). std::string r2n_sql = "CREATE TEMP TABLE _r2n AS " "SELECT sr.rowid as rid, sr.original_id, sr.file_path, sr.name," - " CAST(ROW_NUMBER() OVER () AS INTEGER) + " + + " CAST(ROW_NUMBER() OVER (ORDER BY sr.file_path, sr.kind, " + " sr.start_row, sr.start_col, sr.original_id) AS INTEGER) + " + std::to_string(id_offset) + " as node_id " "FROM semantic_records sr " @@ -270,30 +320,33 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, // before scope creation. The late insert after dropQueryIndexes is // now redundant but kept as a safety net (INSERT OR IGNORE). // Includes sr.arity so the Resolver Pipeline can disambiguate - // same-name overloads via factorSignatureMatch. See - // CODE_REVIEW_FINDINGS_2026-07-19.md C2. - exec(std::string( - "INSERT OR IGNORE INTO entity " - "(id, project_id, kind, name, qualified_name, " - " file_path, language, start_row, start_col, " - " end_row, end_col, module_path, visibility, arity) " - "SELECT r2n.node_id, sr.project_id, " - " CASE sr.kind WHEN 0 THEN 0 WHEN 1 THEN 1 WHEN 2 THEN 2 " - " WHEN 3 THEN 4 WHEN 4 THEN 3 WHEN 5 THEN 3 ELSE 7 END, " - " sr.name, COALESCE(NULLIF(sr.qualified_name, ''), sr.name), " - " sr.file_path, sr.language, " - " sr.start_row, sr.start_col, sr.end_row, sr.end_col, " - " rtrim(sr.file_path, replace(sr.file_path, '/', 'x')), " - " sr.visibility, " - " sr.arity " - "FROM semantic_records sr " - "JOIN _r2n r2n ON sr.rowid = r2n.rid " - "WHERE sr.file_path NOT LIKE '%\\_test.%' ESCAPE '\\'" - " AND sr.file_path NOT LIKE '%/tests/%'" - " AND sr.file_path NOT LIKE '%\\_spec.%' ESCAPE '\\'" - " AND sr.file_path NOT LIKE '%/benches/%'" - " AND sr.file_path NOT LIKE '%\\_\\_test\\_\\_%' ESCAPE '\\'") - .c_str()); + // same-name overloads via factorSignatureMatch. + exec_write( + std::string( + "INSERT OR IGNORE INTO entity " + "(id, project_id, kind, name, qualified_name, " + " file_path, language, start_row, start_col, " + " end_row, end_col, module_path, visibility, arity) " + "SELECT r2n.node_id, sr.project_id, " + " CASE sr.kind WHEN 0 THEN 0 WHEN 1 THEN 1 WHEN 2 THEN 2 " + " WHEN 3 THEN 4 WHEN 4 THEN 3 WHEN 5 THEN 3 ELSE 7 END, " + " sr.name, COALESCE(NULLIF(sr.qualified_name, ''), sr.name), " + " sr.file_path, sr.language, " + " sr.start_row, sr.start_col, sr.end_row, sr.end_col, " + " rtrim(sr.file_path, replace(sr.file_path, '/', 'x')), " + " sr.visibility, " + " sr.arity " + "FROM semantic_records sr " + "JOIN _r2n r2n ON sr.rowid = r2n.rid " + "WHERE sr.project_id=" + + pid + test_file_filter), + "INSERT INTO entity"); + + // The rows above were just re-created without metric columns: re-apply the + // snapshot taken before the rebuild and any freshly staged values. Order + // matters — staging is the newer producer result. Failure is logged, not + // fatal: metrics are derived data and must not fail an index run. + applyMetricsAfterRebuild(project_id); auto t_nodes = Clock::now(); // ── 2d: Containment edges (edge_type=3) ── @@ -314,22 +367,23 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, // Phase 1.3: Type edges — create USES_TYPE edges from TypeRef records { // Populate route table from Route records (kind=19 in semantic_records) - exec(std::string( - "INSERT OR IGNORE INTO route " - "(project_id, method, path, handler_name, " - " file_path, start_row, start_col) " - "SELECT sr.project_id, " - " SUBSTR(sr.name, 1, INSTR(sr.name, ' ') - 1), " - " SUBSTR(sr.name, INSTR(sr.name, ' ') + 1), " - " sr.qualified_name, " - " sr.file_path, sr.start_row, sr.start_col " - "FROM semantic_records sr " - "WHERE sr.project_id=" + - pid + - " AND sr.kind = 19" // Route - " AND sr.name != '' AND sr.name LIKE '% %'" - " AND sr.file_path IN (SELECT file_path FROM _rf)") - .c_str()); + exec_write( + std::string( + "INSERT OR IGNORE INTO route " + "(project_id, method, path, handler_name, " + " file_path, start_row, start_col) " + "SELECT sr.project_id, " + " SUBSTR(sr.name, 1, INSTR(sr.name, ' ') - 1), " + " SUBSTR(sr.name, INSTR(sr.name, ' ') + 1), " + " sr.qualified_name, " + " sr.file_path, sr.start_row, sr.start_col " + "FROM semantic_records sr " + "WHERE sr.project_id=" + + pid + + " AND sr.kind = 19" // Route + " AND sr.name != '' AND sr.name LIKE '% %'" + " AND sr.file_path IN (SELECT file_path FROM _rf)"), + "INSERT INTO route"); } auto t_route = Clock::now(); @@ -377,55 +431,59 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, std::to_string(kKindInterface) + "," + std::to_string(kKindEnum) + "," + std::to_string(kKindTypeAlias) + ")"; - exec(std::string( - "INSERT OR IGNORE INTO type_info " - "(project_id, name, qualified_name, kind, " - " file_path, language, start_row, start_col, " - " end_row, end_col) " - "SELECT sr.project_id, sr.name, " - " COALESCE(NULLIF(sr.qualified_name, ''), sr.name), " - " CASE sr.kind" - " WHEN " + - std::to_string(kKindClass) + - " THEN 0 " - " WHEN " + - std::to_string(kKindEnum) + - " THEN 1 " - " WHEN " + - std::to_string(kKindInterface) + - " THEN 3 " - " WHEN " + - std::to_string(kKindTypeAlias) + - " THEN 4 " - " ELSE 0 END, " - " sr.file_path, sr.language, " - " sr.start_row, sr.start_col, sr.end_row, sr.end_col " - "FROM semantic_records sr " - "WHERE sr.project_id=" + - pid + " AND sr.kind IN " + type_kind_list + - " AND sr.name != ''" - " AND sr.file_path IN (SELECT file_path FROM _rf)") - .c_str()); + exec_write( + std::string( + "INSERT OR IGNORE INTO type_info " + "(project_id, name, qualified_name, kind, " + " file_path, language, start_row, start_col, " + " end_row, end_col) " + "SELECT sr.project_id, sr.name, " + " COALESCE(NULLIF(sr.qualified_name, ''), sr.name), " + " CASE sr.kind" + " WHEN " + + std::to_string(kKindClass) + + " THEN 0 " + " WHEN " + + std::to_string(kKindEnum) + + " THEN 1 " + " WHEN " + + std::to_string(kKindInterface) + + " THEN 3 " + " WHEN " + + std::to_string(kKindTypeAlias) + + " THEN 4 " + " ELSE 0 END, " + " sr.file_path, sr.language, " + " sr.start_row, sr.start_col, " + " sr.end_row, sr.end_col " + "FROM semantic_records sr " + "WHERE sr.project_id=" + + pid + " AND sr.kind IN " + type_kind_list + + " AND sr.name != ''" + " AND sr.file_path IN " + "(SELECT file_path FROM _rf)"), + "INSERT INTO type_info"); } auto t_type_info = Clock::now(); // Populate type_ref table from TypeRef records. { - exec(std::string( - "INSERT OR IGNORE INTO type_ref " - "(project_id, entity_id, type_name, kind, " - " file_path, start_row, start_col) " - "SELECT sr.project_id, r2n.node_id, sr.type_name, " - " CASE WHEN sr.name LIKE '%.return' THEN 2 ELSE 0 END, " - " sr.file_path, sr.start_row, sr.start_col " - "FROM semantic_records sr " - "JOIN _r2n r2n ON sr.rowid = r2n.rid " - "WHERE sr.project_id=" + - pid + - " AND sr.kind = " + std::to_string(kKindTypeRef) + - " AND sr.name != '' AND sr.type_name != ''" - " AND sr.file_path IN (SELECT file_path FROM _rf)") - .c_str()); + exec_write( + std::string( + "INSERT OR IGNORE INTO type_ref " + "(project_id, entity_id, type_name, kind, " + " file_path, start_row, start_col) " + "SELECT sr.project_id, r2n.node_id, sr.type_name, " + " CASE WHEN sr.name LIKE '%.return' THEN 2 ELSE 0 END, " + " sr.file_path, sr.start_row, sr.start_col " + "FROM semantic_records sr " + "JOIN _r2n r2n ON sr.rowid = r2n.rid " + "WHERE sr.project_id=" + + pid + " AND sr.kind = " + + std::to_string(kKindTypeRef) + + " AND sr.name != '' AND sr.type_name != ''" + " AND sr.file_path IN (SELECT file_path FROM _rf)"), + "INSERT INTO type_ref"); } auto t_type_ref = Clock::now(); @@ -478,21 +536,24 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, " AND sr.file_path = r2n.file_path " "WHERE sr.project_id=" + std::to_string(project_id) + - " AND sr.kind = 9 AND sr.name != '' AND sr.file_path NOT LIKE '%\\_test.%' ESCAPE '\\' AND sr.file_path NOT LIKE '%/tests/%' AND sr.file_path NOT LIKE '%\\_spec.%' ESCAPE '\\' AND sr.file_path NOT LIKE '%/benches/%' AND sr.file_path NOT LIKE '%\\_\\_test\\_\\_%' ESCAPE '\\'" - " AND sr.file_path IN (SELECT file_path FROM _rf)"; - exec(ref_sql.c_str()); + " AND sr.kind = 9 AND sr.name != ''" + " AND sr.file_path IN (SELECT file_path FROM _rf)" + + test_file_filter; + exec_write(ref_sql, "INSERT INTO reference"); } auto t_reference = Clock::now(); // Phase 1.2: populate import table from semantic_records Import // Parse raw import text to extract individual import paths. { - const char *fetch_sql = - "SELECT sr.name, sr.project_id, sr.file_path FROM semantic_records sr " - "WHERE sr.project_id=? AND sr.kind=11 AND sr.name != '' AND sr.file_path NOT LIKE '%\\_test.%' ESCAPE '\\' AND sr.file_path NOT LIKE '%/tests/%' AND sr.file_path NOT LIKE '%\\_spec.%' ESCAPE '\\' AND sr.file_path NOT LIKE '%/benches/%'" - " AND sr.file_path IN (SELECT file_path FROM _rf)"; + const std::string fetch_sql = + std::string( + "SELECT sr.name, sr.project_id, sr.file_path FROM semantic_records sr " + "WHERE sr.project_id=? AND sr.kind=11 AND sr.name != ''" + " AND sr.file_path IN (SELECT file_path FROM _rf)") + + test_file_filter; sqlite3_stmt *fetch_st = nullptr; - if (sqlite3_prepare_v2(db_, fetch_sql, -1, &fetch_st, + if (sqlite3_prepare_v2(db_, fetch_sql.c_str(), -1, &fetch_st, nullptr) == SQLITE_OK) { sqlite3_bind_int64(fetch_st, 1, static_cast(project_id)); @@ -571,14 +632,16 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, pr.file = fp_c ? fp_c : ""; batch.push_back(pr); if (batch.size() >= kMaxBatch) { - flushImportBatch( - db_, project_id, batch); + if (!flushImportBatch( + db_, project_id, + batch)) + graph_write_ok = false; batch.clear(); } } - if (!batch.empty()) - flushImportBatch(db_, project_id, - batch); + if (!batch.empty() && + !flushImportBatch(db_, project_id, batch)) + graph_write_ok = false; } sqlite3_finalize(fetch_st); } @@ -588,6 +651,19 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, // Phase 1.2: populate scope table from entity module_path. // Module scopes: one per unique directory (module_path is the // denormalized directory portion of file_path, populated at INSERT). + // + // Idempotency: `scope` has no UNIQUE key on (project_id, kind, name) + // (kind=2 function scopes legitimately repeat a name across modules), + // so `INSERT OR IGNORE` has no conflict target and would re-append every + // kind=1 module row on each buildGraph. buildGraph runs more than once + // per project (the parallel indexer merges per-module worker DBs, then + // the post-index enhance runs buildGraph again on the merged DB), so + // without a guard each module scope is duplicated — inflating + // project_overview.total_modules and doubling module_summary rows (and + // thus dead_code counts, since buildModuleSummaries GROUP BYs s.id). + // The NOT EXISTS clause makes the insert a true upsert-by-name for + // module scopes and is correct for both full and incremental passes + // (an incremental pass only adds modules that do not yet have a scope). { std::string scope_sql = "INSERT OR IGNORE INTO scope " @@ -599,8 +675,12 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, "FROM entity WHERE project_id=" + std::to_string(project_id) + " AND module_path != ''" - " AND entity.file_path IN (SELECT file_path FROM _rf)"; - exec(scope_sql.c_str()); + " AND entity.file_path IN (SELECT file_path FROM _rf)" + " AND NOT EXISTS (SELECT 1 FROM scope s2 WHERE" + " s2.project_id=" + + std::to_string(project_id) + + " AND s2.kind=1 AND s2.name=entity.module_path)"; + exec_write(scope_sql, "INSERT INTO scope (module)"); } // Function scopes: each entity within its module scope. { @@ -618,7 +698,46 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, "WHERE e.project_id=" + std::to_string(project_id) + " AND e.file_path IN (SELECT file_path FROM _rf)"; - exec(func_sql.c_str()); + exec_write(func_sql, "INSERT INTO scope (function)"); + } + // Phase 1.3: drop module scopes whose module no longer exists. + // + // Nothing else removes kind=1 rows: a module scope is shared by every file + // in its directory, so the per-file cleanup in deleteGraphDataByFile only + // deletes function scopes. When the last file of a module disappears — the + // directory was deleted, or a later index excluded it — its module scope + // stayed behind. project_overview counts kind=1 rows as total_modules and + // buildModuleSummaries GROUPs BY them, so a stale row inflated both (and + // the dead_code counts derived from those groups). + // + // `entity` is the right reference set: the INSERT above creates a kind=1 + // row exclusively from `entity.module_path`, so every legitimate module + // name is an entity.module_path of this project. The entity rows of files + // that vanished are already gone by now (deleteGraphDataByFile above, or + // GraphStore::cleanupStaleFiles before this call) and the surviving files' + // rows were inserted earlier in this function, so no live module can look + // orphaned here. NOT IN (…) keeps it to a single index scan of + // idx_entity_module instead of a correlated probe per scope row. + { + std::string orphan_modules = + "DELETE FROM scope WHERE project_id=" + + std::to_string(project_id) + + " AND kind=1 AND name NOT IN (" + "SELECT DISTINCT module_path FROM entity" + " WHERE project_id=" + + std::to_string(project_id) + " AND module_path != '')"; + exec_write(orphan_modules, "DELETE FROM scope (orphan module)"); + // Their function scopes go with them: a kind=2 row whose parent is gone + // can never be joined to a module again, and it would keep inflating + // the table on every later pass. + std::string orphan_functions = + "DELETE FROM scope WHERE project_id=" + + std::to_string(project_id) + + " AND kind=2 AND parent_id NOT IN (" + "SELECT id FROM scope WHERE project_id=" + + std::to_string(project_id) + " AND kind=1)"; + exec_write(orphan_functions, + "DELETE FROM scope (orphan function)"); } // Update import.source_scope_id to point to the file's module scope. // v0.6 (perf): the old form nested a second correlated subquery to look @@ -640,13 +759,22 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, // NOTE: an UPDATE ... FROM rewrite was attempted but SQLite rejects // referencing the target table (import) inside the FROM join clause // ("no such column: import.id"), so the correlated form is kept. - // exec() result is now checked so a failed update can never be - // silently swallowed again (previous code ignored the return value, - // which left every import.source_scope_id at its 0 default). + // The result is routed through exec_write so a failed update flips + // graph_write_ok and rolls the savepoint back (all-or-nothing): + // previously it only logged, so a failure left every + // import.source_scope_id at its 0 default while buildGraph still + // returned true. + // + // import.source_scope_id is NOT NULL. The scalar subquery yields NULL + // when an import has no matching kind=1 module scope, so COALESCE must + // wrap the SUBQUERY (not sit inside it): the old inner COALESCE only + // guarded a matched row's s.id and was never evaluated when the + // subquery returned zero rows, so the whole UPDATE aborted with + // "NOT NULL constraint failed" the moment any import had no scope. { std::string imp_scope_sql = "UPDATE import SET source_scope_id = " - "(SELECT COALESCE(s.id, 0) FROM scope s " + "COALESCE((SELECT s.id FROM scope s " " JOIN entity e ON e.project_id = s.project_id" " AND s.kind = 1" " AND s.name = e.module_path " @@ -654,15 +782,10 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, " AND sr.project_id = import.project_id" " AND sr.file_path = e.file_path" " WHERE e.project_id = import.project_id" - " LIMIT 1) " + " LIMIT 1), 0) " "WHERE project_id=" + std::to_string(project_id); - if (!exec(imp_scope_sql.c_str())) { - fprintf(stderr, - "[module=store, method=buildGraph] " - "UPDATE import.source_scope_id failed: %s\n", - error().c_str()); - } + exec_write(imp_scope_sql, "UPDATE import.source_scope_id"); } auto t_scope = Clock::now(); @@ -779,315 +902,31 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, (long long)ms(t_resolver, t_csr), (long long)ms(t_csr, t_cleanup), (long long)ms(t0, t_cleanup)); - exec("RELEASE SAVEPOINT buildGraph"); - return true; -} - -// ── CSR Adjacency (BLOB-packed call edges) ───────────────────── - -bool GraphStore::buildCSR(uint64_t project_id) -{ - // Wrap the full rebuild (DELETE + forward inserts + reverse inserts) - // in a SAVEPOINT so a mid-build failure rolls back atomically: - // without this, a crash between the DELETE and the final INSERT leaves - // a half-populated CSR (some edges silently missing from queries). - // SAVEPOINT (not BEGIN) is required because buildGraph always runs - // with an active transaction (SAVEPOINT buildGraph / caller BEGIN), - // and SQLite forbids BEGIN inside an active transaction. - if (!exec("SAVEPOINT buildCSR")) { + // Fail closed: any unchecked critical write failure above must not be + // committed. Rolling back to the savepoint restores the pre-buildGraph + // state (all or nothing) and the false return value tells the caller + // the index did not produce a complete graph. + if (!graph_write_ok) { fprintf(stderr, - "[module=store, method=buildCSR] SAVEPOINT buildCSR " - "failed: %s\n", - sqlite3_errmsg(db_)); - return false; - } - - // Clear previous entries for this project - exec(std::string("DELETE FROM adjacency WHERE project_id=" + - std::to_string(project_id)) - .c_str()); - - // Read all call edges from relation table, ordered by source_id for streaming group-by. - // ORDER BY ensures same caller rows are contiguous so we only flush - // to the BLOB when the source changes. - std::string sql = "SELECT source_id, target_id " - "FROM relation " - "WHERE type=1 AND project_id=" + - std::to_string(project_id) + " ORDER BY source_id"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db_, sql.c_str(), -1, &st, nullptr) != - SQLITE_OK) { - exec("ROLLBACK TO SAVEPOINT buildCSR"); - exec("RELEASE SAVEPOINT buildCSR"); - return false; - } - - // v0.6 (perf): rows for this project were just DELETEd above (line 767), - // so no PK conflicts can exist — plain INSERT is equivalent to INSERT OR - // REPLACE here but skips the replace path's conflict bookkeeping. - const char *ins_sql = - "INSERT INTO adjacency (src_id, project_id, tgt_blob) " - "VALUES (?, ?, ?)"; - sqlite3_stmt *ins = nullptr; - if (sqlite3_prepare_v2(db_, ins_sql, -1, &ins, nullptr) != SQLITE_OK) { - sqlite3_finalize(st); - exec("ROLLBACK TO SAVEPOINT buildCSR"); - exec("RELEASE SAVEPOINT buildCSR"); - return false; - } - - int64_t pid_i = static_cast(project_id); - int64_t current_src = -1; - std::vector buf; - buf.reserve(1024); - int64_t count = 0; - - while (sqlite3_step(st) == SQLITE_ROW) { - int64_t src = sqlite3_column_int64(st, 0); - int64_t tgt = sqlite3_column_int64(st, 1); - if (src == tgt) - continue; // skip self-loops - - if (src != current_src) { - // Flush previous group - if (current_src >= 0 && !buf.empty()) { - sqlite3_bind_int64(ins, 1, current_src); - sqlite3_bind_int64(ins, 2, pid_i); - sqlite3_bind_blob( - ins, 3, buf.data(), - static_cast(buf.size() * - sizeof(uint64_t)), - SQLITE_STATIC); - if (sqlite3_step(ins) == SQLITE_DONE) - count++; - else - fprintf(stderr, - "buildCSR: forward flush" - " failed: %s\n", - sqlite3_errmsg(db_)); - sqlite3_reset(ins); - } - current_src = src; - buf.clear(); - } - buf.push_back(static_cast(tgt)); - } - // Flush last group - if (current_src >= 0 && !buf.empty()) { - sqlite3_bind_int64(ins, 1, current_src); - sqlite3_bind_int64(ins, 2, pid_i); - sqlite3_bind_blob( - ins, 3, buf.data(), - static_cast(buf.size() * sizeof(uint64_t)), - SQLITE_STATIC); - if (sqlite3_step(ins) == SQLITE_DONE) - count++; - else - fprintf(stderr, - "buildCSR: final forward flush failed: %s\n", - sqlite3_errmsg(db_)); - sqlite3_reset(ins); - } - - sqlite3_finalize(ins); - sqlite3_finalize(st); - fprintf(stderr, "buildCSR: %lld forward groups from relation(type=1)\n", - (long long)count); - - // ── Build reverse adjacency (adjacency_rev) ── - // Mirror of forward adjacency: group by target_id (callee) instead - // of source_id (caller). Enables O(1) getCallerIds() lookups. - exec(std::string("DELETE FROM adjacency_rev WHERE project_id=" + - std::to_string(project_id)) - .c_str()); - - std::string rev_sql = "SELECT target_id, source_id " - "FROM relation " - "WHERE type=1 AND project_id=" + - std::to_string(project_id) + - " ORDER BY target_id"; - sqlite3_stmt *rev_st = nullptr; - if (sqlite3_prepare_v2(db_, rev_sql.c_str(), -1, &rev_st, nullptr) != - SQLITE_OK) { - exec("ROLLBACK TO SAVEPOINT buildCSR"); - exec("RELEASE SAVEPOINT buildCSR"); - return false; - } - - // v0.6 (perf): rows for this project were just DELETEd above, so no PK - // conflicts can exist — plain INSERT equals INSERT OR REPLACE here. - const char *rev_ins_sql = - "INSERT INTO adjacency_rev (tgt_id, project_id, " - "src_blob) VALUES (?, ?, ?)"; - sqlite3_stmt *rev_ins = nullptr; - if (sqlite3_prepare_v2(db_, rev_ins_sql, -1, &rev_ins, nullptr) != - SQLITE_OK) { - sqlite3_finalize(rev_st); - exec("ROLLBACK TO SAVEPOINT buildCSR"); - exec("RELEASE SAVEPOINT buildCSR"); + "buildGraph: graph writes failed for project %s — " + "rolling back savepoint " + "[module=store, method=buildGraph]\n", + pid.c_str()); + exec("ROLLBACK TO SAVEPOINT buildGraph"); + exec("RELEASE SAVEPOINT buildGraph"); return false; } - - int64_t current_tgt = -1; - std::vector rev_buf; - rev_buf.reserve(1024); - int64_t rev_count = 0; - - while (sqlite3_step(rev_st) == SQLITE_ROW) { - int64_t tgt = sqlite3_column_int64(rev_st, 0); - int64_t src = sqlite3_column_int64(rev_st, 1); - if (src == tgt) - continue; - - if (tgt != current_tgt) { - if (current_tgt >= 0 && !rev_buf.empty()) { - sqlite3_bind_int64(rev_ins, 1, current_tgt); - sqlite3_bind_int64(rev_ins, 2, pid_i); - sqlite3_bind_blob( - rev_ins, 3, rev_buf.data(), - static_cast(rev_buf.size() * - sizeof(uint64_t)), - SQLITE_STATIC); - if (sqlite3_step(rev_ins) == SQLITE_DONE) - rev_count++; - else - fprintf(stderr, - "buildCSR: rev flush" - " failed: %s\n", - sqlite3_errmsg(db_)); - sqlite3_reset(rev_ins); - } - current_tgt = tgt; - rev_buf.clear(); - } - rev_buf.push_back(static_cast(src)); - } - if (current_tgt >= 0 && !rev_buf.empty()) { - sqlite3_bind_int64(rev_ins, 1, current_tgt); - sqlite3_bind_int64(rev_ins, 2, pid_i); - sqlite3_bind_blob( - rev_ins, 3, rev_buf.data(), - static_cast(rev_buf.size() * sizeof(uint64_t)), - SQLITE_STATIC); - if (sqlite3_step(rev_ins) == SQLITE_DONE) - rev_count++; - else - fprintf(stderr, - "buildCSR: final rev flush failed: %s\n", - sqlite3_errmsg(db_)); - sqlite3_reset(rev_ins); - } - - sqlite3_finalize(rev_ins); - sqlite3_finalize(rev_st); - fprintf(stderr, "buildCSR: %lld reverse groups from relation(type=1)\n", - (long long)rev_count); - // Release the atomic rebuild savepoint; a failure here leaves the - // savepoint open, so roll back explicitly rather than leaking a - // pending savepoint into the caller's transaction. - if (!exec("RELEASE SAVEPOINT buildCSR")) { + if (!exec("RELEASE SAVEPOINT buildGraph")) { fprintf(stderr, - "[module=store, method=buildCSR] RELEASE SAVEPOINT " - "buildCSR failed: %s\n", - sqlite3_errmsg(db_)); - exec("ROLLBACK TO SAVEPOINT buildCSR"); - exec("RELEASE SAVEPOINT buildCSR"); + "buildGraph: RELEASE SAVEPOINT failed: %s " + "[module=store, method=buildGraph]\n", + error().c_str()); + exec("ROLLBACK TO SAVEPOINT buildGraph"); return false; } return true; } -/// Decode a packed uint64_t BLOB into a vector of node IDs. -/// -/// The CSR adjacency tables store neighbor IDs as a packed array of -/// uint64_t. The BLOB length must be an exact multiple of sizeof(uint64_t); -/// a non-multiple indicates corruption or an externally-written row. The -/// trailing partial element is dropped and a diagnostic is emitted so the -/// caller is never silently handed a truncated neighbor list. -/// -/// @param blob Pointer to the BLOB bytes (may be null when length is 0). -/// @param bytes Length of the BLOB in bytes. -/// @return The decoded neighbor IDs. -static std::vector decodeAdjacencyBlob(const void *blob, int bytes) -{ - constexpr int kUint64Bytes = static_cast(sizeof(uint64_t)); - if (bytes % kUint64Bytes != 0) { - fprintf(stderr, - "[module=store, method=decodeAdjacencyBlob] BLOB " - "length %d is not a multiple of %d — trailing %d " - "byte(s) dropped\n", - bytes, kUint64Bytes, bytes % kUint64Bytes); - } - const int n = bytes / kUint64Bytes; - std::vector ids; - ids.reserve(static_cast(n)); - if (n > 0) { - const auto *arr = static_cast(blob); - for (int i = 0; i < n; i++) - ids.push_back(static_cast(arr[i])); - } - return ids; -} - -std::vector GraphStore::getCalleeIds(uint64_t node_id) -{ - std::vector ids; - const char *sql = "SELECT tgt_blob FROM adjacency WHERE src_id=?"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &st, nullptr) != SQLITE_OK) - return ids; - sqlite3_bind_int64(st, 1, static_cast(node_id)); - if (sqlite3_step(st) == SQLITE_ROW) { - ids = decodeAdjacencyBlob(sqlite3_column_blob(st, 0), - sqlite3_column_bytes(st, 0)); - } - sqlite3_finalize(st); - return ids; -} - -std::vector GraphStore::getCallerIds(uint64_t node_id) -{ - // O(1) reverse adjacency lookup via adjacency_rev table. - // Falls back to O(n) full-scan if adjacency_rev is not populated - // (e.g., buildCSR was called before the reverse adjacency feature). - std::vector ids; - const char *sql = "SELECT src_blob FROM adjacency_rev WHERE tgt_id=?"; - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &st, nullptr) != SQLITE_OK) - return ids; - sqlite3_bind_int64(st, 1, static_cast(node_id)); - if (sqlite3_step(st) == SQLITE_ROW) { - ids = decodeAdjacencyBlob(sqlite3_column_blob(st, 0), - sqlite3_column_bytes(st, 0)); - sqlite3_finalize(st); - return ids; - } - sqlite3_finalize(st); - - // Fallback: O(n) full-scan of forward adjacency (legacy path) - const char *fallback_sql = - "SELECT src_id, tgt_blob FROM adjacency WHERE project_id IN " - "(SELECT project_id FROM entity WHERE id=?)"; - if (sqlite3_prepare_v2(db_, fallback_sql, -1, &st, nullptr) != - SQLITE_OK) - return ids; - sqlite3_bind_int64(st, 1, static_cast(node_id)); - while (sqlite3_step(st) == SQLITE_ROW) { - int64_t src = sqlite3_column_int64(st, 0); - auto src_ids = decodeAdjacencyBlob(sqlite3_column_blob(st, 1), - sqlite3_column_bytes(st, 1)); - uint64_t target = node_id; - for (uint64_t id : src_ids) { - if (id == target) { - ids.push_back(static_cast(src)); - break; - } - } - } - sqlite3_finalize(st); - return ids; -} - // ── BulkPragmaGuard: RAII bulk-load PRAGMA tuning ─────────────── // Matches codebase-memory-mcp cbm_store_begin_bulk/end_bulk. Saves // synchronous + cache_size, sets OFF + 64 MB, restores on destruction. diff --git a/engine/src/store/store_graph_csr.cpp b/engine/src/store/store_graph_csr.cpp new file mode 100644 index 0000000..a54e4c9 --- /dev/null +++ b/engine/src/store/store_graph_csr.cpp @@ -0,0 +1,354 @@ +// store_graph_csr.cpp — BLOB-packed call-edge adjacency (CSR). +// +// Split out of store_graph.cpp (see plan/rules/code_rules.md 1000-line +// rule). buildCSR packs relation(type=1) edges into per-node BLOBs in the +// adjacency / adjacency_rev tables so caller/callee lookups are O(1) +// instead of an O(n) scan of the relation table. +// +// The rebuild runs in a SAVEPOINT, so a mid-build failure rolls back +// atomically — a crash can never leave a half-populated CSR, which would +// silently drop query edges. + +#include "store.h" + +#include +#include +#include +#include +#include + +namespace store +{ + +// ── CSR Adjacency (BLOB-packed call edges) ───────────────────── + +bool GraphStore::buildCSR(uint64_t project_id) +{ + // Wrap the full rebuild (DELETE + forward inserts + reverse inserts) + // in a SAVEPOINT so a mid-build failure rolls back atomically: + // without this, a crash between the DELETE and the final INSERT leaves + // a half-populated CSR (some edges silently missing from queries). + // SAVEPOINT (not BEGIN) is required because buildGraph always runs + // with an active transaction (SAVEPOINT buildGraph / caller BEGIN), + // and SQLite forbids BEGIN inside an active transaction. + if (!exec("SAVEPOINT buildCSR")) { + fprintf(stderr, + "[module=store, method=buildCSR] SAVEPOINT buildCSR " + "failed: %s\n", + sqlite3_errmsg(db_)); + return false; + } + + // Clear previous entries for this project + exec(std::string("DELETE FROM adjacency WHERE project_id=" + + std::to_string(project_id)) + .c_str()); + + // Read all call edges from relation table, ordered by source_id for streaming group-by. + // ORDER BY ensures same caller rows are contiguous so we only flush + // to the BLOB when the source changes. + std::string sql = "SELECT source_id, target_id " + "FROM relation " + "WHERE type=1 AND project_id=" + + std::to_string(project_id) + " ORDER BY source_id"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db_, sql.c_str(), -1, &st, nullptr) != + SQLITE_OK) { + exec("ROLLBACK TO SAVEPOINT buildCSR"); + exec("RELEASE SAVEPOINT buildCSR"); + return false; + } + + // v0.6 (perf): rows for this project were just DELETEd above (line 767), + // so no PK conflicts can exist — plain INSERT is equivalent to INSERT OR + // REPLACE here but skips the replace path's conflict bookkeeping. + const char *ins_sql = + "INSERT INTO adjacency (src_id, project_id, tgt_blob) " + "VALUES (?, ?, ?)"; + sqlite3_stmt *ins = nullptr; + if (sqlite3_prepare_v2(db_, ins_sql, -1, &ins, nullptr) != SQLITE_OK) { + sqlite3_finalize(st); + exec("ROLLBACK TO SAVEPOINT buildCSR"); + exec("RELEASE SAVEPOINT buildCSR"); + return false; + } + + int64_t pid_i = static_cast(project_id); + int64_t current_src = -1; + std::vector buf; + buf.reserve(1024); + int64_t count = 0; + // Groups the INSERT rejected. A partial CSR must not be published as a + // successful build: see the check before the final RELEASE. + int failed_groups = 0; + + while (sqlite3_step(st) == SQLITE_ROW) { + int64_t src = sqlite3_column_int64(st, 0); + int64_t tgt = sqlite3_column_int64(st, 1); + if (src == tgt) + continue; // skip self-loops + + if (src != current_src) { + // Flush previous group + if (current_src >= 0 && !buf.empty()) { + sqlite3_bind_int64(ins, 1, current_src); + sqlite3_bind_int64(ins, 2, pid_i); + sqlite3_bind_blob( + ins, 3, buf.data(), + static_cast(buf.size() * + sizeof(uint64_t)), + SQLITE_STATIC); + if (sqlite3_step(ins) == SQLITE_DONE) + count++; + else { + ++failed_groups; + fprintf(stderr, + "buildCSR: forward flush" + " failed: %s\n", + sqlite3_errmsg(db_)); + } + sqlite3_reset(ins); + } + current_src = src; + buf.clear(); + } + buf.push_back(static_cast(tgt)); + } + // Flush last group + if (current_src >= 0 && !buf.empty()) { + sqlite3_bind_int64(ins, 1, current_src); + sqlite3_bind_int64(ins, 2, pid_i); + sqlite3_bind_blob( + ins, 3, buf.data(), + static_cast(buf.size() * sizeof(uint64_t)), + SQLITE_STATIC); + if (sqlite3_step(ins) == SQLITE_DONE) + count++; + else { + ++failed_groups; + fprintf(stderr, + "buildCSR: final forward flush failed: %s\n", + sqlite3_errmsg(db_)); + } + sqlite3_reset(ins); + } + + sqlite3_finalize(ins); + sqlite3_finalize(st); + fprintf(stderr, "buildCSR: %lld forward groups from relation(type=1)\n", + (long long)count); + + // ── Build reverse adjacency (adjacency_rev) ── + // Mirror of forward adjacency: group by target_id (callee) instead + // of source_id (caller). Enables O(1) getCallerIds() lookups. + exec(std::string("DELETE FROM adjacency_rev WHERE project_id=" + + std::to_string(project_id)) + .c_str()); + + std::string rev_sql = "SELECT target_id, source_id " + "FROM relation " + "WHERE type=1 AND project_id=" + + std::to_string(project_id) + + " ORDER BY target_id"; + sqlite3_stmt *rev_st = nullptr; + if (sqlite3_prepare_v2(db_, rev_sql.c_str(), -1, &rev_st, nullptr) != + SQLITE_OK) { + exec("ROLLBACK TO SAVEPOINT buildCSR"); + exec("RELEASE SAVEPOINT buildCSR"); + return false; + } + + // v0.6 (perf): rows for this project were just DELETEd above, so no PK + // conflicts can exist — plain INSERT equals INSERT OR REPLACE here. + const char *rev_ins_sql = + "INSERT INTO adjacency_rev (tgt_id, project_id, " + "src_blob) VALUES (?, ?, ?)"; + sqlite3_stmt *rev_ins = nullptr; + if (sqlite3_prepare_v2(db_, rev_ins_sql, -1, &rev_ins, nullptr) != + SQLITE_OK) { + sqlite3_finalize(rev_st); + exec("ROLLBACK TO SAVEPOINT buildCSR"); + exec("RELEASE SAVEPOINT buildCSR"); + return false; + } + + int64_t current_tgt = -1; + std::vector rev_buf; + rev_buf.reserve(1024); + int64_t rev_count = 0; + + while (sqlite3_step(rev_st) == SQLITE_ROW) { + int64_t tgt = sqlite3_column_int64(rev_st, 0); + int64_t src = sqlite3_column_int64(rev_st, 1); + if (src == tgt) + continue; + + if (tgt != current_tgt) { + if (current_tgt >= 0 && !rev_buf.empty()) { + sqlite3_bind_int64(rev_ins, 1, current_tgt); + sqlite3_bind_int64(rev_ins, 2, pid_i); + sqlite3_bind_blob( + rev_ins, 3, rev_buf.data(), + static_cast(rev_buf.size() * + sizeof(uint64_t)), + SQLITE_STATIC); + if (sqlite3_step(rev_ins) == SQLITE_DONE) + rev_count++; + else { + ++failed_groups; + fprintf(stderr, + "buildCSR: rev flush" + " failed: %s\n", + sqlite3_errmsg(db_)); + } + sqlite3_reset(rev_ins); + } + current_tgt = tgt; + rev_buf.clear(); + } + rev_buf.push_back(static_cast(src)); + } + if (current_tgt >= 0 && !rev_buf.empty()) { + sqlite3_bind_int64(rev_ins, 1, current_tgt); + sqlite3_bind_int64(rev_ins, 2, pid_i); + sqlite3_bind_blob( + rev_ins, 3, rev_buf.data(), + static_cast(rev_buf.size() * sizeof(uint64_t)), + SQLITE_STATIC); + if (sqlite3_step(rev_ins) == SQLITE_DONE) + rev_count++; + else { + ++failed_groups; + fprintf(stderr, + "buildCSR: final rev flush failed: %s\n", + sqlite3_errmsg(db_)); + } + sqlite3_reset(rev_ins); + } + + sqlite3_finalize(rev_ins); + sqlite3_finalize(rev_st); + fprintf(stderr, "buildCSR: %lld reverse groups from relation(type=1)\n", + (long long)rev_count); + // Release the atomic rebuild savepoint; a failure here leaves the + // savepoint open, so roll back explicitly rather than leaking a + // pending savepoint into the caller's transaction. + // Every group that failed to write leaves the CSR incomplete. The previous + // version logged those failures, released the savepoint anyway and returned + // true, so a partial adjacency table was published as a successful build — + // and queries then silently saw a graph with edges missing. Rolling back + // leaves the PREVIOUS CSR in place, which is the honest outcome. + if (failed_groups > 0) { + fprintf(stderr, + "buildCSR: %d adjacency group(s) failed to write; the CSR for " + "project %llu would be incomplete — rolling back " + "[module=store, method=buildCSR]\n", + failed_groups, (unsigned long long)project_id); + exec("ROLLBACK TO SAVEPOINT buildCSR"); + exec("RELEASE SAVEPOINT buildCSR"); + return false; + } + + if (!exec("RELEASE SAVEPOINT buildCSR")) { + fprintf(stderr, + "[module=store, method=buildCSR] RELEASE SAVEPOINT " + "buildCSR failed: %s\n", + sqlite3_errmsg(db_)); + exec("ROLLBACK TO SAVEPOINT buildCSR"); + exec("RELEASE SAVEPOINT buildCSR"); + return false; + } + return true; +} + +/// Decode a packed uint64_t BLOB into a vector of node IDs. +/// +/// The CSR adjacency tables store neighbor IDs as a packed array of +/// uint64_t. The BLOB length must be an exact multiple of sizeof(uint64_t); +/// a non-multiple indicates corruption or an externally-written row. The +/// trailing partial element is dropped and a diagnostic is emitted so the +/// caller is never silently handed a truncated neighbor list. +/// +/// @param blob Pointer to the BLOB bytes (may be null when length is 0). +/// @param bytes Length of the BLOB in bytes. +/// @return The decoded neighbor IDs. +static std::vector decodeAdjacencyBlob(const void *blob, int bytes) +{ + constexpr int kUint64Bytes = static_cast(sizeof(uint64_t)); + if (bytes % kUint64Bytes != 0) { + fprintf(stderr, + "[module=store, method=decodeAdjacencyBlob] BLOB " + "length %d is not a multiple of %d — trailing %d " + "byte(s) dropped\n", + bytes, kUint64Bytes, bytes % kUint64Bytes); + } + const int n = bytes / kUint64Bytes; + std::vector ids; + ids.reserve(static_cast(n)); + if (n > 0) { + const auto *arr = static_cast(blob); + for (int i = 0; i < n; i++) + ids.push_back(static_cast(arr[i])); + } + return ids; +} + +std::vector GraphStore::getCalleeIds(uint64_t node_id) +{ + std::vector ids; + const char *sql = "SELECT tgt_blob FROM adjacency WHERE src_id=?"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db_, sql, -1, &st, nullptr) != SQLITE_OK) + return ids; + sqlite3_bind_int64(st, 1, static_cast(node_id)); + if (sqlite3_step(st) == SQLITE_ROW) { + ids = decodeAdjacencyBlob(sqlite3_column_blob(st, 0), + sqlite3_column_bytes(st, 0)); + } + sqlite3_finalize(st); + return ids; +} + +std::vector GraphStore::getCallerIds(uint64_t node_id) +{ + // O(1) reverse adjacency lookup via adjacency_rev table. + // Falls back to O(n) full-scan if adjacency_rev is not populated + // (e.g., buildCSR was called before the reverse adjacency feature). + std::vector ids; + const char *sql = "SELECT src_blob FROM adjacency_rev WHERE tgt_id=?"; + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(db_, sql, -1, &st, nullptr) != SQLITE_OK) + return ids; + sqlite3_bind_int64(st, 1, static_cast(node_id)); + if (sqlite3_step(st) == SQLITE_ROW) { + ids = decodeAdjacencyBlob(sqlite3_column_blob(st, 0), + sqlite3_column_bytes(st, 0)); + sqlite3_finalize(st); + return ids; + } + sqlite3_finalize(st); + + // Fallback: O(n) full-scan of forward adjacency (legacy path) + const char *fallback_sql = + "SELECT src_id, tgt_blob FROM adjacency WHERE project_id IN " + "(SELECT project_id FROM entity WHERE id=?)"; + if (sqlite3_prepare_v2(db_, fallback_sql, -1, &st, nullptr) != + SQLITE_OK) + return ids; + sqlite3_bind_int64(st, 1, static_cast(node_id)); + while (sqlite3_step(st) == SQLITE_ROW) { + int64_t src = sqlite3_column_int64(st, 0); + auto src_ids = decodeAdjacencyBlob(sqlite3_column_blob(st, 1), + sqlite3_column_bytes(st, 1)); + uint64_t target = node_id; + for (uint64_t id : src_ids) { + if (id == target) { + ids.push_back(static_cast(src)); + break; + } + } + } + sqlite3_finalize(st); + return ids; +} +} // namespace store diff --git a/engine/src/store/store_graph_imports.cpp b/engine/src/store/store_graph_imports.cpp new file mode 100644 index 0000000..6f57da3 --- /dev/null +++ b/engine/src/store/store_graph_imports.cpp @@ -0,0 +1,57 @@ +// store_graph_imports.cpp — batch import writer used by buildGraph. +// See store_graph_imports.h for why this lives in its own TU. + +#include "store_graph_imports.h" + +#include + +namespace store +{ + +bool flushImportBatch(sqlite3 *db, uint64_t project_id, + const std::vector &batch) +{ + if (batch.empty()) + return true; + std::string sql = "INSERT OR IGNORE INTO import " + "(project_id, source_scope_id, target_path, alias, " + " file_path, is_pub) VALUES "; + for (size_t i = 0; i < batch.size(); i++) { + if (i > 0) + sql += ","; + sql += "(?,0,?,?,?,0)"; + } + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) != + SQLITE_OK) { + fprintf(stderr, + "[module=store, method=flushImportBatch] " + "prepare failed: %s\n", + sqlite3_errmsg(db)); + return false; + } + for (size_t i = 0; i < batch.size(); i++) { + int base = static_cast(i * 4); + sqlite3_bind_int64(stmt, base + 1, + static_cast(project_id)); + sqlite3_bind_text(stmt, base + 2, batch[i].path.c_str(), -1, + SQLITE_STATIC); + sqlite3_bind_text(stmt, base + 3, batch[i].alias.c_str(), -1, + SQLITE_STATIC); + sqlite3_bind_text(stmt, base + 4, batch[i].file.c_str(), -1, + SQLITE_STATIC); + } + int rc = sqlite3_step(stmt); + if (rc != SQLITE_DONE && rc != SQLITE_CONSTRAINT) { + fprintf(stderr, + "[module=store, method=flushImportBatch] " + "step failed (rc=%d): %s\n", + rc, sqlite3_errmsg(db)); + sqlite3_finalize(stmt); + return false; + } + sqlite3_finalize(stmt); + return true; +} + +} // namespace store diff --git a/engine/src/store/store_graph_imports.h b/engine/src/store/store_graph_imports.h new file mode 100644 index 0000000..e2bc02a --- /dev/null +++ b/engine/src/store/store_graph_imports.h @@ -0,0 +1,33 @@ +// store_graph_imports.h — batch import writer used by buildGraph. +// +// Split out of store_graph.cpp (the Rust/C++ rule is 1000 lines per file). +// The import scan collects (path, alias, file) triples and flushes them in +// batches so that one multi-row INSERT replaces one per row. +#ifndef CODESCOPE_STORE_GRAPH_IMPORTS_H +#define CODESCOPE_STORE_GRAPH_IMPORTS_H + +#include +#include +#include +#include + +namespace store +{ + +/// One collected import statement. +struct PathRec { + std::string path; + std::string alias; + std::string file; +}; + +/// Flush one batch of import rows. +/// @return false when the batch could not be written, so the caller can +/// fail the surrounding buildGraph instead of silently dropping +/// imports (a dropped import means a lost edge). +bool flushImportBatch(sqlite3 *db, uint64_t project_id, + const std::vector &batch); + +} // namespace store + +#endif // CODESCOPE_STORE_GRAPH_IMPORTS_H diff --git a/engine/src/store/store_insert.cpp b/engine/src/store/store_insert.cpp index edac379..61f1f82 100644 --- a/engine/src/store/store_insert.cpp +++ b/engine/src/store/store_insert.cpp @@ -62,11 +62,14 @@ uint64_t GraphStore::insertGraphNode(uint64_t project_id, // The previous code bound slot 17 to a hardcoded 0 and slot 18 to // the real value — slot 18 is past the 17-placeholder SQL and // sqlite3_bind_int returned SQLITE_RANGE, silently dropping the - // value so is_entry_point was ALWAYS 0. See - // CODE_REVIEW_FINDINGS_2026-07-19.md C1. + // value so is_entry_point was ALWAYS 0. sqlite3_bind_int(stmt, 17, node.is_entry_point ? 1 : 0); int rc = sqlite3_step(stmt); + // Reset before returning so the cached stmt releases its SQLITE_STATIC + // bindings (which point into `node`) in this frame, not at the next + // cache reuse — same discipline as store_knowledge.cpp. + sqlite3_reset(stmt); if (rc != SQLITE_DONE) fprintf(stderr, "insertGraphNode: step failed (rc=%d): %s " @@ -75,74 +78,6 @@ uint64_t GraphStore::insertGraphNode(uint64_t project_id, insertEntity(project_id, node); return node.id; } - -void GraphStore::insertGraphNodes(uint64_t project_id, - const std::vector &nodes) -{ - if (nodes.empty()) - return; - - const char *sql = - "INSERT INTO graph_nodes (id, project_id, ir_node_id, node_type, " - "name, qualified_name, module_path, package_name, class_name, " - "start_row, start_col, end_row, end_col, " - "file_path, language, signature, is_entry_point) " - "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, " - "?, ?, ?, ?, ?, ?, ?, ?)"; - - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) { - error_ = "insertGraphNodes: prepare failed"; - return; - } - - for (auto &node : nodes) { - sqlite3_bind_int64(stmt, 1, static_cast(node.id)); - sqlite3_bind_int64(stmt, 2, static_cast(project_id)); - sqlite3_bind_int64(stmt, 3, - static_cast(node.ir_node_id)); - sqlite3_bind_int(stmt, 4, static_cast(node.type)); - sqlite3_bind_text(stmt, 5, node.name.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, 6, node.qualified_name.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, 7, node.module_path.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, 8, node.package_name.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, 9, node.class_name.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_int(stmt, 10, static_cast(node.start_row)); - sqlite3_bind_int(stmt, 11, static_cast(node.start_col)); - sqlite3_bind_int(stmt, 12, static_cast(node.end_row)); - sqlite3_bind_int(stmt, 13, static_cast(node.end_col)); - sqlite3_bind_text(stmt, 14, node.file_path.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, 15, node.language.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, 16, node.signature.c_str(), -1, - SQLITE_STATIC); - // Bind is_entry_point to slot 17 (matches the 17th ? placeholder). - // Previous code bound slot 17 to hardcoded 0 and slot 18 to the - // real value — slot 18 is past the 17-placeholder SQL, SQLITE_RANGE - // silently dropped the value so is_entry_point was ALWAYS 0. - // See CODE_REVIEW_FINDINGS_2026-07-19.md C1. - sqlite3_bind_int(stmt, 17, node.is_entry_point ? 1 : 0); - - int rc = sqlite3_step(stmt); - if (rc != SQLITE_DONE) - fprintf(stderr, - "insertGraphNodes: step failed (rc=%d) " - "for node '%s': %s " - "[module=store, method=insertGraphNodes]\n", - rc, node.name.c_str(), sqlite3_errmsg(db_)); - sqlite3_reset(stmt); - insertEntity(project_id, node); - } - - sqlite3_finalize(stmt); -} - // ── Entity (Phase 1.1) ────────────────────────────────────── uint64_t GraphStore::insertEntity(uint64_t project_id, @@ -182,6 +117,7 @@ uint64_t GraphStore::insertEntity(uint64_t project_id, constexpr int kEntityArityUnknown = 0; sqlite3_bind_int(stmt, 12, kEntityArityUnknown); int rc = sqlite3_step(stmt); + sqlite3_reset(stmt); if (rc != SQLITE_DONE && rc != SQLITE_CONSTRAINT) fprintf(stderr, "insertEntity step failed (rc=%d): %s " @@ -204,33 +140,13 @@ void GraphStore::insertRelation(uint64_t project_id, uint64_t source_id, sqlite3_bind_int64(stmt, 3, static_cast(target_id)); sqlite3_bind_int(stmt, 4, type); int rc = sqlite3_step(stmt); + sqlite3_reset(stmt); if (rc != SQLITE_DONE && rc != SQLITE_CONSTRAINT) fprintf(stderr, "insertRelation step failed (rc=%d): %s " "[module=store, method=insertRelation]\n", rc, sqlite3_errmsg(db_)); } - -bool GraphStore::deleteGraphNodesByFile(uint64_t project_id, - const char *file_path) -{ - // Delete edges first - deleteGraphEdgesByFile(project_id, file_path); - - // Use prepared statement to prevent SQL injection via file_path - sqlite3_stmt *stmt = nullptr; - const char *sql = - "DELETE FROM graph_nodes WHERE project_id = ? AND file_path = ?"; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_text(stmt, 2, file_path, -1, SQLITE_STATIC); - int rc = sqlite3_step(stmt); - sqlite3_finalize(stmt); - return (rc == SQLITE_DONE); - } - return false; -} - // ─── Graph Edges ─────────────────────────────────────────────── uint64_t GraphStore::insertGraphEdge(uint64_t project_id, @@ -256,6 +172,9 @@ uint64_t GraphStore::insertGraphEdge(uint64_t project_id, sqlite3_bind_text(stmt, 8, edge.label.c_str(), -1, SQLITE_STATIC); int rc = sqlite3_step(stmt); + // Reset before the SQLITE_CONSTRAINT branch so the cached stmt releases + // its SQLITE_STATIC bindings (pointing into `edge`) in this frame. + sqlite3_reset(stmt); if (rc == SQLITE_CONSTRAINT) { // INSERT OR IGNORE dropped the row (unique violation on // (project_id, source_node_id, target_node_id, edge_type, @@ -300,53 +219,6 @@ uint64_t GraphStore::insertGraphEdge(uint64_t project_id, rc, sqlite3_errmsg(db_)); return static_cast(sqlite3_last_insert_rowid(db_)); } - -void GraphStore::insertGraphEdges(uint64_t project_id, - const std::vector &edges) -{ - if (edges.empty()) - return; - - const char *sql = - "INSERT OR IGNORE INTO graph_edges (project_id, source_node_id, " - "target_node_id, edge_type, graph_type, " - "call_site_file, call_site_line, label) " - "VALUES (?, ?, ?, ?, ?, ?, ?, ?)"; - - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) { - error_ = "insertGraphEdges: prepare failed"; - return; - } - - for (auto &edge : edges) { - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_int64(stmt, 2, - static_cast(edge.source_id)); - sqlite3_bind_int64(stmt, 3, - static_cast(edge.target_id)); - sqlite3_bind_int(stmt, 4, static_cast(edge.type)); - sqlite3_bind_text(stmt, 5, edge.graph_type.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(stmt, 6, edge.call_site_file.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_int(stmt, 7, edge.call_site_line); - sqlite3_bind_text(stmt, 8, edge.label.c_str(), -1, - SQLITE_STATIC); - - int rc = sqlite3_step(stmt); - if (rc != SQLITE_DONE) { - error_ = "insertGraphEdges: step error (" + - std::to_string(rc) + ")"; - } - sqlite3_reset(stmt); - insertRelation(project_id, edge.source_id, edge.target_id, - static_cast(edge.type)); - } - - sqlite3_finalize(stmt); -} - bool GraphStore::deleteGraphEdgesByFile(uint64_t project_id, const char *file_path) { @@ -503,8 +375,13 @@ bool GraphStore::deleteGraphDataByFile(uint64_t project_id, ")", "scope(function)"); - // 4+5. graph_nodes/graph_edges are deprecated — no longer written. - // The old deleteGraphNodesByFile call is removed. + // 4+5. graph_nodes/graph_edges are legacy: the index paths no longer + // populate graph_nodes (the only writer left is the + // engine_index_batch path), while the resolver still mirrors every + // resolved edge into graph_edges alongside the canonical `relation` + // (resolver/pipeline_flush.cpp). No query reads either — see + // engine_queries_context.cpp — so the mirror is a compatibility + // artefact, and the old deleteGraphNodesByFile call is removed. // 6. Delete entity rows for this file. deleteByFile( @@ -529,52 +406,4 @@ bool GraphStore::deleteGraphDataByFile(uint64_t project_id, return ok; } - -bool GraphStore::deleteFileData(uint64_t project_id, const char *file_path) -{ - if (!file_path || !*file_path) - return false; - - // Track success so callers can detect partial cleanup failures. - // Previously this function unconditionally returned true after merely - // fprintf-ing on prepare/step failures, which masked SQLite errors - // (SQLITE_BUSY, schema drift) and left stale rows — downstream - // buildGraph uses semantic_records to decide the rebuild file set; - // stale rows would cause buildGraph to attempt rebuilding a file whose - // graph was already cleared, leading to inconsistency. - bool ok = deleteGraphDataByFile(project_id, file_path); - - // Delete semantic_records for this file - { - sqlite3_stmt *stmt = nullptr; - const char *sql = - "DELETE FROM semantic_records WHERE project_id = ? AND file_path = ?"; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(stmt, 1, - static_cast(project_id)); - sqlite3_bind_text(stmt, 2, file_path, -1, - SQLITE_STATIC); - int rc = sqlite3_step(stmt); - sqlite3_finalize(stmt); - if (rc != SQLITE_DONE) { - fprintf(stderr, - "deleteFileData: semantic_records delete " - "failed (rc=%d): %s " - "[module=store, method=deleteFileData]\n", - rc, sqlite3_errmsg(db_)); - ok = false; - } - } else { - fprintf(stderr, - "deleteFileData: semantic_records prepare failed: %s " - "[module=store, method=deleteFileData]\n", - sqlite3_errmsg(db_)); - ok = false; - } - } - - return ok; -} - } // namespace store diff --git a/engine/src/store/store_internal.h b/engine/src/store/store_internal.h index 90b1751..d77141d 100644 --- a/engine/src/store/store_internal.h +++ b/engine/src/store/store_internal.h @@ -7,7 +7,6 @@ namespace store // Escape a string for safe embedding in JSON. // Shared across store_*.cpp split files. -std::string jsonEscape(const std::string &s); // Build a safe FTS5 phrase query string for the trigram tokenizer. // Wraps the raw query in double quotes so it is treated as a literal diff --git a/engine/src/store/store_knowledge.cpp b/engine/src/store/store_knowledge.cpp index a8a896f..167e2cb 100644 --- a/engine/src/store/store_knowledge.cpp +++ b/engine/src/store/store_knowledge.cpp @@ -30,8 +30,16 @@ bool GraphStore::insertCapability(uint64_t project_id, const std::string &name, const std::string &source_kind, const std::string &source_ref) { - const char *sql = "INSERT INTO capability (project_id, name, summary, " - "source_kind, source_ref) VALUES (?, ?, ?, ?, ?)"; + // Idempotent: the capability pass runs on every index, and an unguarded + // INSERT appended another copy of every capability each run — inflating + // capability_state and the drift counts derived from it. The identity is + // (project_id, name, source_kind, source_ref); a migration removes what + // earlier runs left behind (store_schema_migrations.cpp). + const char *sql = + "INSERT INTO capability (project_id, name, summary, source_kind, " + "source_ref) SELECT ?, ?, ?, ?, ? WHERE NOT EXISTS " + "(SELECT 1 FROM capability WHERE project_id = ? AND name = ? " + "AND source_kind = ? AND source_ref = ?)"; sqlite3_stmt *stmt = getCachedStmt(sql); if (!stmt) { return false; @@ -41,8 +49,13 @@ bool GraphStore::insertCapability(uint64_t project_id, const std::string &name, sqlite3_bind_text(stmt, 3, summary.c_str(), -1, SQLITE_STATIC); sqlite3_bind_text(stmt, 4, source_kind.c_str(), -1, SQLITE_STATIC); sqlite3_bind_text(stmt, 5, source_ref.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_int64(stmt, 6, static_cast(project_id)); + sqlite3_bind_text(stmt, 7, name.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_text(stmt, 8, source_kind.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_text(stmt, 9, source_ref.c_str(), -1, SQLITE_STATIC); int rc = sqlite3_step(stmt); + sqlite3_reset(stmt); if (rc != SQLITE_DONE) { error_ = std::string("insertCapability: step failed: ") + sqlite3_errmsg(db_); @@ -62,11 +75,22 @@ bool GraphStore::insertContract(uint64_t project_id, const std::string &name, const std::string &claim_text, const std::string &source_file, int source_line) { + // Idempotent: ContractPlugin re-runs on every runModelIndexSync. + // A bare INSERT appended a duplicate row per rebuild (the same defect + // insertCapability had). WHERE NOT EXISTS keeps one row per + // (project_id, name, origin, source_file, source_line) identity. const char *sql = "INSERT INTO contract (project_id, name, origin, claim_text, " - "source_file, source_line) VALUES (?, ?, ?, ?, ?, ?)"; + "source_file, source_line) " + "SELECT ?, ?, ?, ?, ?, ? " + "WHERE NOT EXISTS (" + " SELECT 1 FROM contract " + " WHERE project_id=? AND name=? AND origin=? " + " AND source_file=? AND source_line=?)"; sqlite3_stmt *stmt = getCachedStmt(sql); if (!stmt) { + error_ = "[module=store, method=insertContract] " + "prepare failed"; return false; } sqlite3_bind_int64(stmt, 1, static_cast(project_id)); @@ -75,10 +99,17 @@ bool GraphStore::insertContract(uint64_t project_id, const std::string &name, sqlite3_bind_text(stmt, 4, claim_text.c_str(), -1, SQLITE_STATIC); sqlite3_bind_text(stmt, 5, source_file.c_str(), -1, SQLITE_STATIC); sqlite3_bind_int(stmt, 6, source_line); + sqlite3_bind_int64(stmt, 7, static_cast(project_id)); + sqlite3_bind_text(stmt, 8, name.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_text(stmt, 9, origin.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_text(stmt, 10, source_file.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_int(stmt, 11, source_line); int rc = sqlite3_step(stmt); + sqlite3_reset(stmt); if (rc != SQLITE_DONE) { - error_ = std::string("insertContract: step failed: ") + + error_ = std::string("[module=store, method=insertContract] " + "step failed: ") + sqlite3_errmsg(db_); fprintf(stderr, "insertContract: step failed (rc=%d): %s " @@ -113,6 +144,10 @@ int64_t GraphStore::insertClaim(uint64_t project_id, const verify::Claim &claim) sqlite3_bind_text(stmt, 8, claim.source_ref.c_str(), -1, SQLITE_STATIC); int rc = sqlite3_step(stmt); + // Reset before returning so SQLITE_STATIC bindings (which point into + // the caller's std::string) are released here, not at the next cache + // reuse — the strings die with this frame. + sqlite3_reset(stmt); if (rc != SQLITE_DONE) { error_ = std::string("insertClaim: step failed: ") + sqlite3_errmsg(db_); @@ -147,6 +182,7 @@ int64_t GraphStore::insertEvidence(int64_t claim_id, verify::Verdict verdict, sqlite3_bind_text(stmt, 5, detail.c_str(), -1, SQLITE_STATIC); int rc = sqlite3_step(stmt); + sqlite3_reset(stmt); if (rc != SQLITE_DONE) { error_ = std::string("insertEvidence: step failed: ") + sqlite3_errmsg(db_); @@ -177,6 +213,7 @@ bool GraphStore::insertEvidenceFact(int64_t evidence_id, int fact_kind, sqlite3_bind_text(stmt, 4, detail.c_str(), -1, SQLITE_STATIC); int rc = sqlite3_step(stmt); + sqlite3_reset(stmt); if (rc != SQLITE_DONE) { error_ = std::string("insertEvidenceFact: step failed: ") + sqlite3_errmsg(db_); @@ -216,6 +253,7 @@ int64_t GraphStore::insertFinding(uint64_t project_id, const std::string &rule, sqlite3_bind_double(stmt, 6, confidence); int rc = sqlite3_step(stmt); + sqlite3_reset(stmt); if (rc != SQLITE_DONE) { error_ = std::string("insertFinding: step failed: ") + sqlite3_errmsg(db_); @@ -227,60 +265,6 @@ int64_t GraphStore::insertFinding(uint64_t project_id, const std::string &rule, } return static_cast(sqlite3_last_insert_rowid(db_)); } - -// ── clearProjectKnowledge ─────────────────────────────────────────── -// -// Deletes in reverse FK dependency order so no FK violation can occur: -// evidence_fact -> evidence -> finding -> claim -> contract -> capability -// evidence_fact and evidence have no project_id column, so they are scoped -// via a subquery on claim.project_id. finding/contract/capability carry -// project_id directly and use a bound parameter. - -bool GraphStore::clearProjectKnowledge(uint64_t project_id) -{ - // Each DELETE uses a cached, parameterized statement so repeated calls - // (e.g. KnowledgeBuilder::build() on re-index) reuse the same plan. - static const char *const kDeleteSql[] = { - // 1. evidence_fact (via evidence -> claim.project_id) - "DELETE FROM evidence_fact WHERE evidence_id IN " - "(SELECT id FROM evidence WHERE claim_id IN " - " (SELECT id FROM claim WHERE project_id=?))", - // 2. evidence (via claim.project_id) - ("DELETE FROM evidence WHERE claim_id IN " - " (SELECT id FROM claim WHERE project_id=?)"), - // 3. finding (has project_id) - "DELETE FROM finding WHERE project_id=?", - // 4. claim - "DELETE FROM claim WHERE project_id=?", - // 5. contract - "DELETE FROM contract WHERE project_id=?", - // 6. capability - "DELETE FROM capability WHERE project_id=?", - }; - - bool ok = true; - for (const char *sql : kDeleteSql) { - sqlite3_stmt *stmt = getCachedStmt(sql); - if (!stmt) { - ok = false; - continue; - } - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - int rc = sqlite3_step(stmt); - if (rc != SQLITE_DONE) { - error_ = std::string("clearProjectKnowledge: ") + - sqlite3_errmsg(db_); - fprintf(stderr, - "clearProjectKnowledge: delete failed (rc=%d): " - "%s [module=store, " - "method=clearProjectKnowledge]\n", - rc, sqlite3_errmsg(db_)); - ok = false; - } - } - return ok; -} - // ── listCapabilities / listContracts ─────────────────────────────── std::vector> @@ -299,6 +283,7 @@ GraphStore::listCapabilities(uint64_t project_id) sqlite3_column_text(stmt, 1)); out.emplace_back(id, name ? name : ""); } + sqlite3_reset(stmt); return out; } @@ -318,6 +303,7 @@ GraphStore::listContracts(uint64_t project_id) sqlite3_column_text(stmt, 1)); out.emplace_back(id, name ? name : ""); } + sqlite3_reset(stmt); return out; } @@ -342,6 +328,7 @@ bool GraphStore::insertDocument(uint64_t project_id, int type, sqlite3_bind_int(stmt, 5, start_line); sqlite3_bind_int(stmt, 6, end_line); int rc = sqlite3_step(stmt); + sqlite3_reset(stmt); if (rc != SQLITE_DONE) { error_ = std::string("insertDocument: step failed: ") + sqlite3_errmsg(db_); @@ -350,22 +337,73 @@ bool GraphStore::insertDocument(uint64_t project_id, int type, return true; } +bool GraphStore::deleteDocument(uint64_t project_id, int type, + const std::string &file_path) +{ + // Same cached-statement pattern as insertDocument above: prepared once per + // connection, reset after use. Deleting zero rows is success — the point + // is that a re-ingest cannot leave the previous copy behind. + const char *sql = "DELETE FROM document " + "WHERE project_id = ? AND type = ? AND file_path = ?"; + sqlite3_stmt *stmt = getCachedStmt(sql); + if (!stmt) + return false; + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_int(stmt, 2, type); + sqlite3_bind_text(stmt, 3, file_path.c_str(), -1, SQLITE_STATIC); + const int rc = sqlite3_step(stmt); + sqlite3_reset(stmt); + if (rc != SQLITE_DONE) { + error_ = std::string("deleteDocument: step failed: ") + + sqlite3_errmsg(db_); + return false; + } + return true; +} + // ── workflow ────────────────────────────────────────────────────── int64_t GraphStore::insertWorkflow(uint64_t project_id, const std::string &name) { + // Idempotent across model rebuilds (same identity as insertContract / + // insertCapability): return the existing row instead of appending a + // duplicate on every runModelIndexSync. const char *sql = - "INSERT INTO workflow (project_id, name) VALUES (?,?)"; + "INSERT INTO workflow (project_id, name) " + "SELECT ?,? WHERE NOT EXISTS (" + " SELECT 1 FROM workflow WHERE project_id=? AND name=?)"; sqlite3_stmt *stmt = getCachedStmt(sql); - if (!stmt) + if (!stmt) { + error_ = "[module=store, method=insertWorkflow] " + "prepare failed"; return -1; + } sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, name.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_int64(stmt, 3, static_cast(project_id)); + sqlite3_bind_text(stmt, 4, name.c_str(), -1, SQLITE_STATIC); if (sqlite3_step(stmt) != SQLITE_DONE) { - error_ = "insertWorkflow: step failed"; + sqlite3_reset(stmt); + error_ = "[module=store, method=insertWorkflow] step failed"; return -1; } - return sqlite3_last_insert_rowid(db_); + sqlite3_reset(stmt); + // Return the (possibly pre-existing) row id so workflow_step rows + // attach to a single workflow instead of a fresh duplicate. + sqlite3_stmt *sel = getCachedStmt( + "SELECT id FROM workflow WHERE project_id=? AND name=?"); + if (!sel) { + error_ = "[module=store, method=insertWorkflow] " + "select failed"; + return -1; + } + sqlite3_bind_int64(sel, 1, static_cast(project_id)); + sqlite3_bind_text(sel, 2, name.c_str(), -1, SQLITE_STATIC); + int64_t id = -1; + if (sqlite3_step(sel) == SQLITE_ROW) + id = sqlite3_column_int64(sel, 0); + sqlite3_reset(sel); + return id; } bool GraphStore::insertWorkflowStep(int64_t workflow_id, int step_order, @@ -382,8 +420,10 @@ bool GraphStore::insertWorkflowStep(int64_t workflow_id, int step_order, sqlite3_bind_int64(stmt, 3, entity_id); sqlite3_bind_text(stmt, 4, label.c_str(), -1, SQLITE_STATIC); int rc = sqlite3_step(stmt); + sqlite3_reset(stmt); if (rc != SQLITE_DONE) { - error_ = "insertWorkflowStep: step failed"; + error_ = "[module=store, method=insertWorkflowStep] " + "step failed"; return false; } return true; @@ -392,23 +432,26 @@ bool GraphStore::insertWorkflowStep(int64_t workflow_id, int step_order, // ── architecture_edge ───────────────────────────────────────────── bool GraphStore::insertArchitectureEdge(uint64_t project_id, - const std::string &layer_upper, - const std::string &layer_lower, + const std::string &caller_module, + const std::string &callee_module, int64_t entity_id) { - const char *sql = "INSERT INTO architecture_edge " - "(project_id, layer_upper, layer_lower, entity_id) " - "VALUES (?,?,?,?)"; + const char *sql = + "INSERT INTO architecture_edge " + "(project_id, caller_module, callee_module, entity_id) " + "VALUES (?,?,?,?)"; sqlite3_stmt *stmt = getCachedStmt(sql); if (!stmt) return false; sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_text(stmt, 2, layer_upper.c_str(), -1, SQLITE_STATIC); - sqlite3_bind_text(stmt, 3, layer_lower.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_text(stmt, 2, caller_module.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_text(stmt, 3, callee_module.c_str(), -1, SQLITE_STATIC); sqlite3_bind_int64(stmt, 4, entity_id); int rc = sqlite3_step(stmt); + sqlite3_reset(stmt); if (rc != SQLITE_DONE) { - error_ = "insertArchitectureEdge: step failed"; + error_ = "[module=store, method=insertArchitectureEdge] " + "step failed"; return false; } return true; @@ -437,66 +480,16 @@ int64_t GraphStore::insertReference(uint64_t project_id, uint64_t caller_id, sqlite3_bind_int(stmt, 7, start_col); sqlite3_bind_int(stmt, 8, call_kind); if (sqlite3_step(stmt) != SQLITE_DONE) { - error_ = "insertReference: step failed"; - return -1; - } - return sqlite3_last_insert_rowid(db_); -} - -// ── scope ──────────────────────────────────────────────────────── - -int64_t GraphStore::insertScope(uint64_t project_id, int64_t parent_id, - int kind, const std::string &name, - int start_row, int end_row) -{ - const char *sql = - "INSERT INTO scope " - "(project_id, parent_id, kind, name, start_row, end_row) " - "VALUES (?,?,?,?,?,?)"; - sqlite3_stmt *stmt = getCachedStmt(sql); - if (!stmt) - return -1; - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_int64(stmt, 2, parent_id); - sqlite3_bind_int(stmt, 3, kind); - sqlite3_bind_text(stmt, 4, name.c_str(), -1, SQLITE_STATIC); - sqlite3_bind_int(stmt, 5, start_row); - sqlite3_bind_int(stmt, 6, end_row); - if (sqlite3_step(stmt) != SQLITE_DONE) { - error_ = "insertScope: step failed"; + error_ = "[module=store, method=insertReference] " + "step failed"; + sqlite3_reset(stmt); return -1; } + sqlite3_reset(stmt); return sqlite3_last_insert_rowid(db_); } - -// ── import ─────────────────────────────────────────────────────── - -int64_t GraphStore::insertImport(uint64_t project_id, int64_t source_scope_id, - const std::string &target_path, - const std::string &alias, int is_pub) -{ - const char *sql = - "INSERT INTO import " - "(project_id, source_scope_id, target_path, alias, is_pub) " - "VALUES (?,?,?,?,?)"; - sqlite3_stmt *stmt = getCachedStmt(sql); - if (!stmt) - return -1; - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_int64(stmt, 2, source_scope_id); - sqlite3_bind_text(stmt, 3, target_path.c_str(), -1, SQLITE_STATIC); - sqlite3_bind_text(stmt, 4, alias.c_str(), -1, SQLITE_STATIC); - sqlite3_bind_int(stmt, 5, is_pub); - if (sqlite3_step(stmt) != SQLITE_DONE) { - error_ = "insertImport: step failed"; - return -1; - } - return sqlite3_last_insert_rowid(db_); -} - -// ── resolved_reference ────────────────────────────────────────── - -// insertResolvedReference has been removed. -// The resolved_reference table was replaced by relation.confidence + reason. +// `insertResolvedReference` and the `resolved_reference` table are gone: the +// resolver writes relation.confidence + relation.reason instead. Kept as a note +// so it does not get reintroduced. } // namespace store diff --git a/engine/src/store/store_membulk.cpp b/engine/src/store/store_membulk.cpp index 56b4f40..11517b5 100644 --- a/engine/src/store/store_membulk.cpp +++ b/engine/src/store/store_membulk.cpp @@ -44,9 +44,11 @@ MemBulkAggregator::~MemBulkAggregator() = default; bool GraphStore::dropSemanticRecordIndexes() { - // Drop the 9 lookup indexes on semantic_records before a bulk INSERT. - // Each live index costs one B-tree update per inserted row; with 9 - // indexes and ~16k rows that is ~144k random B-tree writes. Dropping + // Drop the 11 lookup indexes on semantic_records before a bulk INSERT. + // Each live index costs one B-tree update per inserted row; with 11 + // indexes and ~16k rows that is ~176k random B-tree writes. The list + // below must stay in step with createSemanticRecordIndexes: an index that + // is dropped but never recreated is lost until the next schema pass. Dropping // them and rebuilding in bulk after the insert (one sorted scan per // index) is 5–10x faster for large modules. // @@ -65,6 +67,7 @@ bool GraphStore::dropSemanticRecordIndexes() "DROP INDEX IF EXISTS idx_sr_kind_name", "DROP INDEX IF EXISTS idx_sr_kind_fp", "DROP INDEX IF EXISTS idx_sr_proj_file_oid", + "DROP INDEX IF EXISTS idx_sr_oid", }; bool ok = true; for (auto *sql : drop_sqls) { @@ -81,7 +84,7 @@ bool GraphStore::dropSemanticRecordIndexes() bool GraphStore::createSemanticRecordIndexes() { - // Recreate the 9 semantic_records indexes after a bulk INSERT. + // Recreate the 11 semantic_records indexes after a bulk INSERT. // SQLite builds each index in a single sorted scan of the table // (O(n log n) per index, sequential I/O), which is far cheaper than // maintaining the index row-by-row during INSERT (O(n log n) per @@ -100,6 +103,7 @@ bool GraphStore::createSemanticRecordIndexes() "CREATE INDEX IF NOT EXISTS idx_sr_kind_name ON semantic_records(project_id, kind, name, language)", "CREATE INDEX IF NOT EXISTS idx_sr_kind_fp ON semantic_records(project_id, kind, file_path)", "CREATE INDEX IF NOT EXISTS idx_sr_proj_file_oid ON semantic_records(project_id, file_path, original_id)", + "CREATE INDEX IF NOT EXISTS idx_sr_oid ON semantic_records(project_id, original_id)", }; bool ok = true; for (auto *sql : create_sqls) { @@ -139,8 +143,8 @@ bool MemBulkAggregator::flush(GraphStore &store, uint64_t project_id, total); } - // On a fresh DB (worker mode), drop the 9 semantic_records indexes - // before the bulk INSERT so SQLite doesn't maintain 9 B-trees per + // On a fresh DB (worker mode), drop the 11 semantic_records indexes + // before the bulk INSERT so SQLite doesn't maintain 11 B-trees per // row. They are rebuilt in bulk after commit (one sorted scan per // index) which is 5–10x faster for large modules. if (!is_reindex && !store.dropSemanticRecordIndexes()) { @@ -174,9 +178,20 @@ bool MemBulkAggregator::flush(GraphStore &store, uint64_t project_id, } } - store.commitTransaction(); + // A failed COMMIT leaves the connection inside an open transaction, so every + // later BEGIN fails — and flush() reported success regardless, claiming a + // bulk load that never landed. Roll back so the connection is usable again + // and report the failure to the caller. + if (!store.commitTransaction()) { + fprintf(stderr, + "store_membulk: commitTransaction failed: %s " + "[module=store_membulk, method=flush]\n", + store.error().c_str()); + store.rollbackTransaction(); + return false; + } - // Rebuild the 9 semantic_records indexes in bulk now that all rows + // Rebuild the 11 semantic_records indexes in bulk now that all rows // are inserted. Must happen BEFORE buildGraph (called by the caller // after flush returns) — buildGraph JOINs semantic_records on // (project_id, file_path) via idx_sr_file / idx_sr_kind_fp. diff --git a/engine/src/store/store_metrics.cpp b/engine/src/store/store_metrics.cpp new file mode 100644 index 0000000..012e8c2 --- /dev/null +++ b/engine/src/store/store_metrics.cpp @@ -0,0 +1,257 @@ +// store_metrics.cpp — metric columns must survive a graph rebuild. +// +// Why this file exists +// -------------------- +// buildGraph() re-creates every entity row it rebuilds: it deletes the rows of +// the files in the rebuild set (deleteGraphDataByFile) and re-inserts them from +// `semantic_records` with an explicit column list (store_graph.cpp §2c). That +// list carries a declaration's semantic identity — id, name, file, position, +// arity — but NOT the derived metrics, because metrics are produced separately +// and staged in `_staged_metrics` for resolveStagedMetrics() to apply after the +// entities exist. +// +// Two measured consequences (2026-10-06, nine languages, real projects): +// +// 1. Parallel path (`index-parallel`, the recommended one): each module +// worker resolves its own metrics — a goagent worker database held 5543 +// entities with cyclomatic > 0 — but the post-index pass re-runs +// buildGraph on the MERGED database, so every metric column came back as +// 0. One and the same entity (`handleMCP`) was `cyclomatic=4, cognitive=5, +// nesting_depth=1, lines=15` in the worker and `0,0,0,0` after the merge. +// `analysis_progress.metrics`, `metrics_ready`, and every +// complexity/cognitive/nesting/lines value in tool output were 0 for +// every project checked. +// 2. File-list path (`force-index`, `force_index_files`): metrics are staged +// but resolveStagedMetrics() is never called on that path — a goagent +// database held 15160 unresolved `_staged_metrics` rows and published 0 +// metrics as well. +// +// The fix lives here rather than in buildGraph() so that function — already +// close to the 1000-line limit of code_rules §1 — gains two calls instead of +// another SQL block. Both halves use the (project_id, file_path, start_row, +// start_col) key that resolveStagedMetrics() and the staging writer already +// agree on, and only function/method entities (kind 0/1) carry code metrics. + +#include "store.h" +#include "store_internal.h" + +#include +#include +#include +#include +#include + +namespace +{ + +/// Metric columns, declared once so the snapshot and the re-apply can never +/// drift apart (code_rules §5: no duplicated literals). `entity` and +/// `_staged_metrics` both carry exactly these columns. +constexpr const char *kMetricColumns[] = { + "cyclomatic", "nesting_depth", "cognitive", + "param_count", "call_count", "branch_count", + "loop_count", "lines", "is_stub", +}; + +constexpr size_t kMetricColumnCount = + sizeof(kMetricColumns) / sizeof(kMetricColumns[0]); + +/// Build the assignment list of an `UPDATE ... FROM` statement: +/// `a = alias.a, b = alias.b, ...`. +/// +/// @param alias Table alias the values are read from. +/// @return The assignment list, without a trailing comma. +std::string metricAssignments(const char *alias) +{ + std::string sql; + sql.reserve(kMetricColumnCount * 24); + for (size_t i = 0; i < kMetricColumnCount; ++i) { + if (i > 0) + sql += ", "; + sql += kMetricColumns[i]; + sql += " = "; + sql += alias; + sql += "."; + sql += kMetricColumns[i]; + } + return sql; +} + +/// Comma-separated column list, used by the snapshot's `CREATE ... AS SELECT`. +std::string metricColumnList() +{ + std::string sql; + sql.reserve(kMetricColumnCount * 16); + for (size_t i = 0; i < kMetricColumnCount; ++i) { + if (i > 0) + sql += ", "; + sql += kMetricColumns[i]; + } + return sql; +} + +/// Run a statement that takes the project id as its single bound parameter. +/// +/// Metrics are derived data: a failure must be reported (code_rules §1) but +/// must not fail the index run that triggered it, which is why the caller +/// decides what to do with a false return. +/// +/// @param db Open store connection. +/// @param sql Statement with exactly one `?` placeholder. +/// @param project_id Value bound to that placeholder. +/// @param caller `[module=store, method=...]` tag for the log line. +/// @return true when the statement completed (SQLITE_DONE). +bool runForProject(sqlite3 *db, const std::string &sql, uint64_t project_id, + const char *caller) +{ + if (!db) + return false; + + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) != + SQLITE_OK) { + fprintf(stderr, + "%s: prepare failed: %s [module=store, method=%s]\n", + caller, sqlite3_errmsg(db), caller); + return false; + } + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + const int rc = sqlite3_step(stmt); + sqlite3_finalize(stmt); + if (rc != SQLITE_DONE) { + fprintf(stderr, + "%s: step failed (rc=%d): %s [module=store, method=%s]\n", + caller, rc, sqlite3_errmsg(db), caller); + return false; + } + return true; +} + +} // namespace + +namespace store +{ + +bool GraphStore::snapshotMetricsForRebuild(uint64_t project_id) +{ + if (!db_) + return false; + + // Temp table keyed like _staged_metrics so the re-apply below is a plain + // JOIN. DROP+CREATE keeps the column set in step with kMetricColumns and + // makes a second buildGraph on the same connection start from a clean + // slate instead of accumulating rows from the previous run. + if (!exec("DROP TABLE IF EXISTS _prev_metrics")) + return false; + + std::string create = + "CREATE TEMP TABLE _prev_metrics AS SELECT project_id, " + " file_path, start_row, start_col, " + + metricColumnList() + + " FROM entity WHERE project_id = ? AND kind IN (0,1)"; + if (!runForProject(db_, create, project_id, + "snapshotMetricsForRebuild")) + return false; + + int64_t rows = -1; + sqlite3_stmt *count_stmt = nullptr; + if (sqlite3_prepare_v2(db_, "SELECT COUNT(*) FROM _prev_metrics", -1, + &count_stmt, nullptr) == SQLITE_OK) { + if (sqlite3_step(count_stmt) == SQLITE_ROW) + rows = sqlite3_column_int64(count_stmt, 0); + sqlite3_finalize(count_stmt); + } + fprintf(stderr, + "engine: snapshotMetricsForRebuild=%lld row(s) carried across the " + "rebuild [module=store, method=snapshotMetricsForRebuild]\n", + (long long)rows); + return true; +} + +bool GraphStore::applyMetricsAfterRebuild(uint64_t project_id) +{ + if (!db_) + return false; + + // Step 1: values that existed before this rebuild. This is what makes an + // already-resolved project (the merged database of a parallel index, an + // incremental rebuild) keep its metrics. + std::string restore_prev = "UPDATE entity SET " + + metricAssignments("p") + + " FROM _prev_metrics p" + " WHERE entity.project_id = p.project_id" + " AND entity.file_path = p.file_path" + " AND entity.start_row = p.start_row" + " AND entity.start_col = p.start_col" + " AND entity.project_id = ?" + " AND entity.kind IN (0,1)"; + if (!runForProject(db_, restore_prev, project_id, + "applyMetricsAfterRebuild")) + return false; + + // Step 2: freshly staged values win — a producer result from this run is + // newer than the snapshot. + std::string apply_staged = "UPDATE entity SET " + + metricAssignments("m") + + " FROM _staged_metrics m" + " WHERE entity.project_id = m.project_id" + " AND entity.file_path = m.file_path" + " AND entity.start_row = m.start_row" + " AND entity.start_col = m.start_col" + " AND entity.project_id = ?" + " AND entity.kind IN (0,1)"; + if (!runForProject(db_, apply_staged, project_id, + "applyMetricsAfterRebuild")) + return false; + + // Consume the staged rows: they are now on the entity rows, and a path + // that never calls resolveStagedMetrics() (engine_index_files) would + // otherwise accumulate them forever. resolveStagedMetrics() stays in + // place as the belt-and-braces producer step and becomes a no-op here. + const std::string consume = + "DELETE FROM _staged_metrics WHERE project_id = ?"; + if (!runForProject(db_, consume, project_id, + "applyMetricsAfterRebuild")) + return false; + + // Keep the readiness flag consistent with the data it describes. + // postParsePhase sets it too, but only on the project-index path: the + // parallel scheduler's post-index pass rebuilds the MERGED database, whose + // readiness row the merge does not carry across, so `metrics_ready` stayed + // 0 while `analysis_progress.metrics` (a live probe) reported the resolved + // rows — a contradiction the user sees as `ready_features.metrics: false` + // next to real complexity numbers. Probing here means every path that + // rebuilds the graph publishes the same answer. + const std::string metric_probe = + "SELECT COUNT(*) FROM entity WHERE project_id = ?" + " AND kind IN (0,1) AND cyclomatic > 0"; + sqlite3_stmt *probe_stmt = nullptr; + int64_t metric_rows = -1; + if (sqlite3_prepare_v2(db_, metric_probe.c_str(), -1, &probe_stmt, + nullptr) == SQLITE_OK) { + sqlite3_bind_int64(probe_stmt, 1, + static_cast(project_id)); + if (sqlite3_step(probe_stmt) == SQLITE_ROW) + metric_rows = sqlite3_column_int64(probe_stmt, 0); + sqlite3_finalize(probe_stmt); + } else { + fprintf(stderr, + "applyMetricsAfterRebuild: metrics count probe failed: %s " + "[module=store, method=applyMetricsAfterRebuild]\n", + sqlite3_errmsg(db_)); + // Falls through: the values themselves are applied and correct; only + // the flag cannot be refreshed, and the failure is already reported. + } + if (metric_rows >= 0) + setProjectReadiness(project_id, "metrics_ready", + metric_rows > 0 ? 1 : 0); + + fprintf(stderr, + "engine: applyMetricsAfterRebuild done (previous values restored, " + "staged rows applied and consumed, metrics_ready=%lld) " + "[module=store, method=applyMetricsAfterRebuild]\n", + (long long)(metric_rows < 0 ? -1 : metric_rows)); + return true; +} + +} // namespace store diff --git a/engine/src/store/store_parse_failure.cpp b/engine/src/store/store_parse_failure.cpp index e056e77..82ebdb4 100644 --- a/engine/src/store/store_parse_failure.cpp +++ b/engine/src/store/store_parse_failure.cpp @@ -1,29 +1,34 @@ // store_parse_failure.cpp — Persistent parse-failure tracking. // -// Implements the fail-fast design from DYNAMIC_SCHED_REDESIGN.md §7.3 -// and CODE_REVIEW_DYNAMIC_SCHED_2026-07-19.md (Part B, Phase 0). +// Implements the fail-fast design from DYNAMIC_SCHED_REDESIGN.md §7.3. // Files that fail to parse N times (CODESCOPE_FAIL_RETRY_MAX, default // 1) are skipped on subsequent index runs. Reset via CLI reset-failures. // -// Uses g_store (defined in engine.cpp, declared in engine_internal.h) -// via the public handle() accessor. Prepared statements are wrapped in -// StmtPtr (unique_ptr with custom deleter) so sqlite3_finalize always -// runs, even on early return or exception. +// Reaches the engine store through the EngineContext it is HANDED — TD-1 +// knife 3 made the ABI handle-based and deleted the process-global accessor, so +// every entry here takes `EngineContext *ctx` and goes through that context's +// public handle() accessor. Prepared statements are wrapped in StmtPtr +// (unique_ptr with custom deleter) so sqlite3_finalize always runs, even on +// early return or exception. +#include "util/json_writer.h" #include "store_parse_failure.h" #include "store.h" #include "store_internal.h" +// engine_context.h forward-declares its member types and defines both special +// members out-of-line, so including it to take an `EngineContext *` does NOT +// pull engine_internal.h's heavy transitive includes (parser.h, ir.h, ...) into +// this store TU. Previously this file re-declared the `g_store` global by hand +// to avoid that; TD-1 passes the context in as a parameter instead. +#include "engine_context.h" + #include #include #include #include #include -// Re-declared here to avoid pulling engine_internal.h's heavy -// transitive includes (parser.h, ir.h, etc.) into this store TU. -extern std::unique_ptr g_store; - namespace store { @@ -66,19 +71,25 @@ const char *failReasonToString(FailReason r) return "unknown"; } -bool isKnownParseFailure(uint64_t project_id, const std::string &file_path, - int retry_max) +bool isKnownParseFailure(EngineContext *ctx, uint64_t project_id, + const std::string &file_path, int retry_max) { - sqlite3 *db = g_store ? g_store->handle() : nullptr; + sqlite3 *db = ctx->store ? ctx->store->handle() : nullptr; if (!db) { - fprintf(stderr, "store: g_store not initialised " + fprintf(stderr, "store: engine store not initialised " "[module=store, method=isKnownParseFailure]\n"); return false; } + // `language_missing` is deliberately excluded: the file is not broken, + // the engine just has no grammar for it (a disabled grammar, or one that + // failed to load). Counting it as a known failure would SKIP the file + // forever — including after the grammar is enabled — because the skip set + // is keyed by path only and never expires. sqlite3_stmt *raw = nullptr; if (sqlite3_prepare_v2(db, "SELECT fail_count FROM parse_failures " - "WHERE project_id=? AND file_path=?", + "WHERE project_id=? AND file_path=? " + "AND fail_reason != 'language_missing'", -1, &raw, nullptr) != SQLITE_OK) { logErr(db, "prepare", "isKnownParseFailure"); return false; @@ -102,7 +113,7 @@ bool isKnownParseFailure(uint64_t project_id, const std::string &file_path, // writer transaction rolls back (M1 in CODE_REVIEW_SCHED_CHANGES). The // connection is cached and re-opened if the underlying DB path changes // (e.g. several index runs within one process). -static sqlite3 *auxFailureDb() +static sqlite3 *auxFailureDb(EngineContext *ctx) { // GUARD: parse-worker threads call recordParseFailure concurrently // (engine_index_project.cpp spawns N workers). The static `db` / @@ -116,9 +127,9 @@ static sqlite3 *auxFailureDb() static std::string cached_path; static sqlite3 *db = nullptr; - if (!g_store) + if (!ctx || !ctx->store) return nullptr; - const std::string path = g_store->dbPath(); + const std::string path = ctx->store->dbPath(); if (db && cached_path != path) { sqlite3_close(db); db = nullptr; @@ -149,10 +160,11 @@ static sqlite3 *auxFailureDb() return db; } -void recordParseFailure(uint64_t project_id, const std::string &file_path, - const std::string &lang, const std::string &reason) +void recordParseFailure(EngineContext *ctx, uint64_t project_id, + const std::string &file_path, const std::string &lang, + const std::string &reason) { - sqlite3 *db = auxFailureDb(); + sqlite3 *db = auxFailureDb(ctx); if (!db) { fprintf(stderr, "store: aux failure db unavailable " "[module=store, method=recordParseFailure]\n"); @@ -203,7 +215,7 @@ void bufferParseFailure(uint64_t project_id, const std::string &file_path, g_failure_buf.push_back({ project_id, file_path, lang, reason }); } -int flushParseFailures() +int flushParseFailures(EngineContext *ctx) { // Swap out the buffer under the lock so we can flush without // holding the lock during SQLite writes. @@ -215,7 +227,7 @@ int flushParseFailures() if (batch.empty()) return 0; - sqlite3 *db = auxFailureDb(); + sqlite3 *db = auxFailureDb(ctx); if (!db) { fprintf(stderr, "store: aux failure db unavailable " "[module=store, method=flushParseFailures]\n"); @@ -238,8 +250,16 @@ int flushParseFailures() } StmtPtr stmt(raw); - // Single transaction for all buffered rows. - sqlite3_exec(db, "BEGIN", nullptr, nullptr, nullptr); + // Single transaction for all buffered rows. BEGIN, each step and COMMIT are + // all checked: the previous version ignored the BEGIN/COMMIT results and + // merely logged a failed row while still committing, so a partially written + // batch was indistinguishable from a complete one (and a failed COMMIT left + // the connection inside an open transaction, failing every later BEGIN). + if (sqlite3_exec(db, "BEGIN", nullptr, nullptr, nullptr) != SQLITE_OK) { + logErr(db, "BEGIN", "flushParseFailures"); + return -1; + } + size_t failed_rows = 0; for (auto &r : batch) { sqlite3_bind_int64(stmt.get(), 1, static_cast(r.project_id)); @@ -249,20 +269,35 @@ int flushParseFailures() SQLITE_TRANSIENT); sqlite3_bind_text(stmt.get(), 4, r.reason.c_str(), -1, SQLITE_TRANSIENT); - if (sqlite3_step(stmt.get()) != SQLITE_DONE) + if (sqlite3_step(stmt.get()) != SQLITE_DONE) { + ++failed_rows; logErr(db, "step", "flushParseFailures"); + } sqlite3_reset(stmt.get()); } - sqlite3_exec(db, "COMMIT", nullptr, nullptr, nullptr); + if (failed_rows > 0) { + sqlite3_exec(db, "ROLLBACK", nullptr, nullptr, nullptr); + fprintf(stderr, + "store: %zu of %zu parse-failure rows failed to write; " + "batch rolled back [module=store, method=flushParseFailures]\n", + failed_rows, batch.size()); + return -1; + } + if (sqlite3_exec(db, "COMMIT", nullptr, nullptr, nullptr) != + SQLITE_OK) { + logErr(db, "COMMIT", "flushParseFailures"); + sqlite3_exec(db, "ROLLBACK", nullptr, nullptr, nullptr); + return -1; + } return static_cast(batch.size()); } -int resetParseFailures(uint64_t project_id) +int resetParseFailures(EngineContext *ctx, uint64_t project_id) { - sqlite3 *db = g_store ? g_store->handle() : nullptr; + sqlite3 *db = ctx->store ? ctx->store->handle() : nullptr; if (!db) { - fprintf(stderr, "store: g_store not initialised " + fprintf(stderr, "store: engine store not initialised " "[module=store, method=resetParseFailures]\n"); return -1; } @@ -282,12 +317,13 @@ int resetParseFailures(uint64_t project_id) return sqlite3_changes(db); } -std::string getParseFailuresJson(uint64_t project_id, int limit) +std::string getParseFailuresJson(EngineContext *ctx, uint64_t project_id, + int limit) { - sqlite3 *db = g_store ? g_store->handle() : nullptr; + sqlite3 *db = ctx->store ? ctx->store->handle() : nullptr; if (!db) { fprintf(stderr, - "store: g_store not initialised " + "store: engine store not initialised " "[module=store, method=getParseFailuresJson]\n"); return "[]"; } @@ -322,31 +358,40 @@ std::string getParseFailuresJson(uint64_t project_id, int limit) int fc = sqlite3_column_int(stmt.get(), 3); int64_t first_seen = sqlite3_column_int64(stmt.get(), 4); int64_t last_seen = sqlite3_column_int64(stmt.get(), 5); - json += "{\"file_path\":\"" + jsonEscape(fp ? fp : ""); - json += "\",\"language\":\"" + jsonEscape(lg ? lg : ""); - json += "\",\"fail_reason\":\"" + jsonEscape(rs ? rs : ""); - json += "\",\"fail_count\":" + std::to_string(fc); - json += ",\"first_seen\":" + std::to_string(first_seen); - json += ",\"last_seen\":" + std::to_string(last_seen) + "}"; + util::JsonWriter el; + el.beginObject(); + el.key("file_path").value(fp ? fp : ""); + el.key("language").value(lg ? lg : ""); + el.key("fail_reason").value(rs ? rs : ""); + el.key("fail_count").value(fc); + el.key("first_seen").value(first_seen); + el.key("last_seen").value(last_seen); + el.endObject(); + json += el.str(); } json += "]"; return json; } -bool loadKnownParseFailures(uint64_t project_id, int retry_max, - std::vector &out_paths) +bool loadKnownParseFailures(EngineContext *ctx, uint64_t project_id, + int retry_max, std::vector &out_paths) { - sqlite3 *db = g_store ? g_store->handle() : nullptr; + sqlite3 *db = ctx->store ? ctx->store->handle() : nullptr; if (!db) { fprintf(stderr, - "store: g_store not initialised " + "store: engine store not initialised " "[module=store, method=loadKnownParseFailures]\n"); return false; } + // Same `language_missing` exclusion as isKnownParseFailure: a file whose + // only failure was an unavailable grammar must be re-attempted on every + // run, so enabling the grammar (or fixing GRAMMARS_DIR) is enough to pick + // it up — no manual `codescope reset-failures` needed. sqlite3_stmt *raw = nullptr; if (sqlite3_prepare_v2(db, "SELECT file_path FROM parse_failures " - "WHERE project_id=? AND fail_count >= ?", + "WHERE project_id=? AND fail_count >= ? " + "AND fail_reason != 'language_missing'", -1, &raw, nullptr) != SQLITE_OK) { logErr(db, "prepare", "loadKnownParseFailures"); return false; diff --git a/engine/src/store/store_parse_failure.h b/engine/src/store/store_parse_failure.h index dad4822..b1332a0 100644 --- a/engine/src/store/store_parse_failure.h +++ b/engine/src/store/store_parse_failure.h @@ -6,6 +6,10 @@ #include #include +// The auxiliary parse-failure connection is opened from the engine instance's +// store (TD-1 knife 3: the instance is passed in, not looked up globally). +#include "engine_context.h" + namespace store { @@ -30,10 +34,18 @@ const char *failReasonToString(FailReason r); /// fail_count >= retry_max). Returns false otherwise (including when /// the file is not in the table, or fail_count < retry_max). /// -/// `retry_max` is read from `CODESCOPE_FAIL_RETRY_MAX` env var -/// (default 3) by the caller and passed in to avoid repeated env reads. -bool isKnownParseFailure(uint64_t project_id, const std::string &file_path, - int retry_max); +/// Rows whose reason is `language_missing` never count: the file is intact and +/// the engine simply has no grammar for it, so it must be re-attempted every +/// run (a disabled grammar or a failed .so load is fixable without touching +/// the file). +/// +/// `retry_max` is passed in by the caller to avoid repeated env reads. The +/// index paths read `CODESCOPE_FAIL_RETRY_MAX`, defaulting to +/// `engine_index_sched.h`'s `kDefaultFailRetryMax` (currently 1). The forced +/// path (`engine_index_files.cpp` with bypass_fail_fast) loads no set at all — +/// a file handed to `force_index_files` is always re-attempted. +bool isKnownParseFailure(EngineContext *ctx, uint64_t project_id, + const std::string &file_path, int retry_max); /// Record a parse failure for a file. If the row already exists, /// increments fail_count and updates last_seen. Otherwise inserts a @@ -46,8 +58,11 @@ bool isKnownParseFailure(uint64_t project_id, const std::string &file_path, /// `lang` may be empty if the language was never identified. /// `reason` is one of FailReason above (exception variants should /// append the what() text after a colon, e.g. "exception: bad_alloc"). -void recordParseFailure(uint64_t project_id, const std::string &file_path, - const std::string &lang, const std::string &reason); +/// @param ctx Engine instance whose store supplies the database path for +/// the auxiliary connection. May be null (no-op). +void recordParseFailure(EngineContext *ctx, uint64_t project_id, + const std::string &file_path, const std::string &lang, + const std::string &reason); /// Thread-safe in-memory buffer for parse failures. /// Parse workers call this instead of recordParseFailure to avoid @@ -62,26 +77,29 @@ void bufferParseFailure(uint64_t project_id, const std::string &file_path, /// or -1 on error. The buffer is cleared after successful flush. /// Idempotent: consecutive calls with no new bufferParseFailure calls /// write zero rows. -int flushParseFailures(); +/// @param ctx Engine instance whose store supplies the database path for +/// the auxiliary connection. +int flushParseFailures(EngineContext *ctx); /// Reset (delete) all parse_failures rows for the given project_id. /// Called by the `codescope reset-failures` CLI. Returns the number /// of rows deleted, or -1 on error. -int resetParseFailures(uint64_t project_id); +int resetParseFailures(EngineContext *ctx, uint64_t project_id); /// Return a JSON array of all parse_failures rows for a project, /// limited to `limit` rows (default 100). Each row is an object: /// {"file_path":"...","language":"...","fail_reason":"...", /// "fail_count":N,"first_seen":N,"last_seen":N} /// Returns "[]" on error or empty table. -std::string getParseFailuresJson(uint64_t project_id, int limit); +std::string getParseFailuresJson(EngineContext *ctx, uint64_t project_id, + int limit); /// Load all file_path values for the given project_id where /// fail_count >= retry_max into the provided vector. Used at index /// startup to pre-populate a skip set without per-file DB queries. /// Returns true on success, false on DB error (caller logs). -bool loadKnownParseFailures(uint64_t project_id, int retry_max, - std::vector &out_paths); +bool loadKnownParseFailures(EngineContext *ctx, uint64_t project_id, + int retry_max, std::vector &out_paths); } // namespace store diff --git a/engine/src/store/store_project.cpp b/engine/src/store/store_project.cpp index 1ab42d3..198c0e6 100644 --- a/engine/src/store/store_project.cpp +++ b/engine/src/store/store_project.cpp @@ -1,3 +1,4 @@ +#include "util/json_writer.h" #include "store.h" #include "store_internal.h" #include "platform_win.h" @@ -69,102 +70,6 @@ uint64_t GraphStore::insertModule(uint64_t project_id, uint64_t parent_id, sqlite3_finalize(stmt); return id; } - -// ── New Schema (Phase A): Symbols ───────────────────────────── - -uint64_t GraphStore::insertSymbol(uint64_t project_id, uint64_t module_id, - const char *kind, const char *name, - const char *signature, const char *visibility, - const char *language, const char *file_path, - int line, int column, int span_start, - int span_end) -{ - const char *sql = - "INSERT OR IGNORE INTO graph_nodes " - "(project_id, node_type, name, signature, file_path, language, " - " start_row, start_col) " - "VALUES (?, ?, ?, ?, ?, ?, ?, ?)"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) { - error_ = "insertSymbol: prepare failed"; - return 0; - } - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - // NOTE: module_id is intentionally NOT bound here — graph_nodes has no - // module_id column (see the INSERT column list above: - // project_id, node_type, name, signature, file_path, language, - // start_row, start_col). The removed code bound ?2 to module_id and then - // OVERWROTE it with node_type, so ?2 always ended up as node_type (the - // correct value) but module_id was silently dropped. ?2 belongs to - // node_type only. (void)module_id; below documents the unused parameter. - // Map string kind to node_type integer - int node_type = 7; - if (kind) { - std::string ks = kind; - if (ks == "function") - node_type = 0; - else if (ks == "method") - node_type = 1; - else if (ks == "class") - node_type = 2; - else if (ks == "struct") - node_type = 3; - else if (ks == "interface") - node_type = 4; - else if (ks == "enum") - node_type = 5; - else if (ks == "type_alias") - node_type = 6; - } - (void)module_id; - (void)visibility; - (void)span_start; - (void)span_end; - sqlite3_bind_int(stmt, 2, node_type); - sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); - sqlite3_bind_text(stmt, 4, signature, -1, SQLITE_TRANSIENT); - sqlite3_bind_text(stmt, 5, file_path, -1, SQLITE_TRANSIENT); - sqlite3_bind_text(stmt, 6, language, -1, SQLITE_TRANSIENT); - sqlite3_bind_int(stmt, 7, line); - sqlite3_bind_int(stmt, 8, column); - - if (sqlite3_step(stmt) != SQLITE_DONE) { - error_ = "insertSymbol: step failed"; - sqlite3_finalize(stmt); - return 0; - } - uint64_t id = static_cast(sqlite3_last_insert_rowid(db_)); - sqlite3_finalize(stmt); - - return id; -} - -// ── New Schema (Phase A): Entry Points ──────────────────────── - -bool GraphStore::insertEntryPoint(uint64_t symbol_id, uint64_t project_id, - const char *kind) -{ - const char *sql = - "INSERT OR REPLACE INTO entry_points (symbol_id, project_id, kind) " - "VALUES (?, ?, ?)"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) { - error_ = "insertEntryPoint: prepare failed"; - return false; - } - sqlite3_bind_int64(stmt, 1, static_cast(symbol_id)); - sqlite3_bind_int64(stmt, 2, static_cast(project_id)); - sqlite3_bind_text(stmt, 3, kind, -1, SQLITE_TRANSIENT); - - if (sqlite3_step(stmt) != SQLITE_DONE) { - error_ = "insertEntryPoint: step failed"; - sqlite3_finalize(stmt); - return false; - } - sqlite3_finalize(stmt); - return true; -} - // ── New Schema (Phase A): Queries ───────────────────────────── std::string GraphStore::getModuleTreeJson(uint64_t project_id) @@ -239,13 +144,18 @@ std::string GraphStore::getModuleTreeJson(uint64_t project_id) if (!first) json << ","; first = false; + // The element stays open across the optional children array + // below (and the recursion writes nested elements into the + // same stream), so its shape is kept; the string values go + // through the shared escaper. json << "{\"id\":" << it->id << ",\"parent_id\":" << it->parent_id << ",\"depth\":" << depth << ",\"name\":\"" - << jsonEscape(it->name) << "\"" - << ",\"path\":\"" << jsonEscape(it->path) << "\"" - << ",\"language\":\"" << jsonEscape(it->language) - << "\"" + << util::jsonEscapeString(it->name) << "\"" + << ",\"path\":\"" + << util::jsonEscapeString(it->path) << "\"" + << ",\"language\":\"" + << util::jsonEscapeString(it->language) << "\"" << ",\"file_count\":" << it->file_count; auto ci = children_of.find(id); if (ci != children_of.end() && !ci->second.empty()) { @@ -311,21 +221,30 @@ std::string GraphStore::findSymbolJson(uint64_t project_id, const char *name) int line = sqlite3_column_int(stmt, 6); int col = sqlite3_column_int(stmt, 7); - json << "{" - << "\"id\":" << id << "," - << "\"kind\":\"" << (kind ? kind : "") << "\"," - << "\"name\":\"" << jsonEscape(sym_name ? sym_name : "") - << "\"," - << "\"signature\":\"" << jsonEscape(sig ? sig : "") - << "\"," - << "\"language\":\"" << (lang ? lang : "") << "\"," - << "\"file_path\":\"" << jsonEscape(fp ? fp : "") << "\"," - << "\"line\":" << line << "," - << "\"column\":" << col << "}"; + util::JsonWriter el; + el.beginObject(); + el.key("id").value(id); + // kind and language went through the writer now too: they are + // free text read from the database and were previously emitted + // unescaped. + el.key("kind").value(kind ? kind : ""); + el.key("name").value(sym_name ? sym_name : ""); + el.key("signature").value(sig ? sig : ""); + el.key("language").value(lang ? lang : ""); + el.key("file_path").value(fp ? fp : ""); + el.key("line").value(line); + el.key("column").value(col); + el.endObject(); + json << el.str(); } + sqlite3_reset(stmt); json << "]}"; - // If symbols table was empty, fall back to graph_nodes (new pipeline) + // If the entity table returned nothing, fall back to graph_nodes. That + // table is written only by the LEGACY engine_index_batch path, so this + // fallback serves DBs built by it; the canonical pipeline (and the + // primary query above) reads `entity`. Kept deliberately: a batch-built + // DB must still resolve symbols. if (first) { std::ostringstream gn_json; gn_json << "{\"results\":["; @@ -367,18 +286,20 @@ std::string GraphStore::findSymbolJson(uint64_t project_id, const char *name) const char *type_name = (nt >= 0 && nt < 7) ? type_names[nt] : "symbol"; - gn_json << "{" - << "\"id\":" << id << "," - << "\"kind\":\"" << type_name << "\"," - << "\"name\":\"" - << jsonEscape(n ? n : "") << "\"," - << "\"file_path\":\"" - << jsonEscape(fp ? fp : "") << "\"," - << "\"line\":" << sr << "," - << "\"column\":" << sc << "," - << "\"language\":\"" - << (lang ? lang : "") << "\"" - << "}"; + util::JsonWriter el; + el.beginObject(); + el.key("id").value(id); + // kind and language are free text and were emitted + // unescaped before; the writer escapes them now. + el.key("kind").value(type_name ? type_name : + ""); + el.key("name").value(n ? n : ""); + el.key("file_path").value(fp ? fp : ""); + el.key("line").value(sr); + el.key("column").value(sc); + el.key("language").value(lang ? lang : ""); + el.endObject(); + gn_json << el.str(); } sqlite3_finalize(gn_stmt); } @@ -388,168 +309,69 @@ std::string GraphStore::findSymbolJson(uint64_t project_id, const char *name) return json.str(); } +// ── Phase C: Unified Queries ────────────────────────────────── -// ── Phase B: Enhancement — Embeddings ───────────────────────── - -bool GraphStore::insertEmbedding(uint64_t symbol_id, const float *vector_data, - int dim) +namespace { - // Guard against negative dim (would wrap to a huge size_t in the - // memcpy below and cause a heap buffer overflow) and null input. - if (dim <= 0 || !vector_data) - return false; - - // The vec0 table expects a blob of float32 values - // Pad/truncate to 384 (schema definition) - constexpr int TARGET_DIM = 384; - std::vector vec(TARGET_DIM, 0.0f); - int copy_dim = dim < TARGET_DIM ? dim : TARGET_DIM; - memcpy(vec.data(), vector_data, - static_cast(copy_dim) * sizeof(float)); - - // Write to node_vectors FIRST — this is the table that searchSemantic - // actually reads from. This always works because node_vectors is a - // regular SQLite table, not a vec0 virtual table. - // On platforms where vec0.dll isn't available (e.g. Windows without - // the extension), the embeddings INSERT below will fail, but the - // node_vectors path still succeeds, providing graceful degradation. - // v0.2.5: resolve the project id from the canonical `entity` table - // (entity.id) — the deprecated `graph_nodes` table is empty in the - // canonical schema, so the old query could never resolve a project. - { - uint64_t proj_id = 0; - sqlite3_stmt *pid_st = nullptr; - if (sqlite3_prepare_v2( - db_, "SELECT project_id FROM entity WHERE id=?", -1, - &pid_st, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(pid_st, 1, - static_cast(symbol_id)); - if (sqlite3_step(pid_st) == SQLITE_ROW) - proj_id = static_cast( - sqlite3_column_int64(pid_st, 0)); - sqlite3_finalize(pid_st); - } - if (proj_id > 0 && stmt_vector_) { - sqlite3_reset(stmt_vector_); - sqlite3_bind_int64(stmt_vector_, 1, - static_cast(symbol_id)); - sqlite3_bind_int64(stmt_vector_, 2, - static_cast(proj_id)); - sqlite3_bind_blob( - stmt_vector_, 3, vec.data(), - TARGET_DIM * static_cast(sizeof(float)), - SQLITE_TRANSIENT); - sqlite3_step(stmt_vector_); - } - } - - // node_vectors was already written above (the only vector storage). - // The vec0 embeddings table was removed — searchSemantic reads - // node_vectors directly. - return true; -} - -// ── Phase B: Enhancement — Ready Flags ──────────────────────── +// ── Canonical readiness probes ────────────────────────────────── +// The coverage ratios below are the same questions engine_queries_status.cpp +// answers for engine_get_enhancement_status, and they deliberately use the +// same SQL so both APIs report identical numbers. They read the canonical +// tables the pipeline actually writes (`entity`, `relation`, `node_vectors`). // -// v0.2.5: the metrics and embedding producers are RESTORED (see -// resolveStagedMetrics / buildVectorsFromGraph). These two setters remain as -// defensive seams that operate on the DEPRECATED `graph_nodes` table (empty -// in the canonical schema — `entity`/`relation`/`node_vectors` are the source -// of truth). They deliberately do NOT flip canonical readiness: true -// metrics_ready / vector_ready are derived from the canonical entity -// cyclomatic count and node_vectors row count in -// engine_get_enhancement_status / engine_get_capabilities / -// engine_index_post_parse, so readiness always matches real data and the -// A18/A19 "fake ready" bugs cannot recur regardless of these seams. - -bool GraphStore::markCallgraphAndMetricsReady(uint64_t symbol_id) -{ - // Compatibility seam: only touches the deprecated graph_nodes row. - // Canonical callgraph readiness is computed from relation.type=1 - // coverage; canonical metrics_ready from entity cyclomatic — never - // set here. - const char *sql = - "UPDATE graph_nodes SET callgraph_ready=1 WHERE id = ?"; - sqlite3_stmt *stmt = getCachedStmt(sql); - if (!stmt) { - return false; - } - sqlite3_bind_int64(stmt, 1, static_cast(symbol_id)); - if (sqlite3_step(stmt) != SQLITE_DONE) { - error_ = "markCallgraphAndMetricsReady: step failed"; - return false; - } - return true; -} - -bool GraphStore::markEmbeddingReady(uint64_t symbol_id) -{ - // v0.2.5: embedding producer (buildVectorsFromGraph) is restored and - // populates node_vectors. This seam deliberately does NOT flip a flag: - // canonical embedding readiness is derived from the node_vectors row - // count, so it tracks real data and the A19 "fake ready" bug cannot - // recur. Retained so a caller cannot silently over-claim readiness. - (void)symbol_id; - return true; -} - -bool GraphStore::setSymbolStub(uint64_t symbol_id, bool is_stub) +// Step 10.1: they used to read `graph_nodes._ready`, a table the +// canonical pipeline never populates (only the legacy engine_index_batch path +// does), so every ratio was 0.0 — and consumers drew the wrong conclusion from +// it: `project_overview` reported a working call graph as +// `"callgraph_available": false`, `ready_features.call_graph: false` and an +// empty `sample_call_edges`, while find_callers/find_callees answered with +// hundreds of callers. verifier.h states the rule this violated: the canonical +// fact layer is the production source of truth, never graph_nodes. + +/// Population the per-entity ratios are computed over: function and method +/// entities. +constexpr const char *kReadyEligibleSql = + "SELECT COUNT(*) FROM entity WHERE project_id=? AND kind IN (0,1)"; + +/// Function/method entities participating in the call graph (either end of a +/// Calls relation). +constexpr const char *kReadyCallgraphSql = + "SELECT COUNT(*) FROM (SELECT DISTINCT node FROM (" + "SELECT source_id AS node FROM relation WHERE project_id=? AND type=1 " + "UNION " + "SELECT target_id AS node FROM relation WHERE project_id=? AND type=1))"; + +/// Function/method entities whose metrics were resolved (cyclomatic > 0). +constexpr const char *kReadyMetricsSql = + "SELECT COUNT(*) FROM entity WHERE project_id=? AND kind IN (0,1) " + "AND cyclomatic > 0"; + +/// Entities carrying a semantic vector: the canonical embedding coverage. +constexpr const char *kReadyEmbeddingSql = + "SELECT COUNT(*) FROM node_vectors WHERE project_id=?"; + +/// Run a count probe. Returns -1 when the statement cannot be run, so the +/// caller can report the failure instead of silently returning 0.0. +/// \param db Open store handle. +/// \param sql Probe with one placeholder per entry of `binds`. +/// \param binds Values bound to the placeholders, in order. +int64_t readyCount(sqlite3 *db, const char *sql, + const std::vector &binds) { - const char *sql = "UPDATE graph_nodes SET is_stub = ? WHERE id = ?"; + if (!db || !sql) + return -1; sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) { - error_ = "setSymbolStub: prepare failed"; - return false; - } - sqlite3_bind_int(stmt, 1, is_stub ? 1 : 0); - sqlite3_bind_int64(stmt, 2, static_cast(symbol_id)); - if (sqlite3_step(stmt) != SQLITE_DONE) { - error_ = "setSymbolStub: step failed"; - sqlite3_finalize(stmt); - return false; - } + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) != SQLITE_OK) + return -1; + for (size_t i = 0; i < binds.size(); ++i) + sqlite3_bind_int64(stmt, static_cast(i + 1), binds[i]); + int64_t n = -1; + if (sqlite3_step(stmt) == SQLITE_ROW) + n = sqlite3_column_int64(stmt, 0); sqlite3_finalize(stmt); - return true; + return n; } - -std::vector GraphStore::getUnreadyFiles(uint64_t project_id, - const char *ready_field) -{ - // Whitelist allowed field names to prevent SQL injection - static const std::unordered_set allowed_fields = { - "fast_ready", "normal_ready", "deep_ready", - "fts_ready", "vector_ready", "callgraph_ready", - "metrics_ready", "embedding_ready" - }; - if (!ready_field || - allowed_fields.find(ready_field) == allowed_fields.end()) { - return {}; - } - - std::string sql = "SELECT DISTINCT gn.file_path FROM graph_nodes gn " - "WHERE gn.project_id = ? AND gn." + - std::string(ready_field) + - " = 0 " - "ORDER BY gn.file_path"; - sqlite3_stmt *stmt = nullptr; - std::vector files; - if (sqlite3_prepare_v2(db_, sql.c_str(), -1, &stmt, nullptr) != - SQLITE_OK) { - error_ = "getUnreadyFiles: prepare failed"; - return files; - } - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - while (sqlite3_step(stmt) == SQLITE_ROW) { - const char *fp = reinterpret_cast( - sqlite3_column_text(stmt, 0)); - if (fp) - files.emplace_back(fp); - } - sqlite3_finalize(stmt); - return files; -} - -// ── Phase C: Unified Queries ────────────────────────────────── +} // namespace double GraphStore::getReadyRatio(uint64_t project_id, const char *ready_field) { @@ -563,45 +385,50 @@ double GraphStore::getReadyRatio(uint64_t project_id, const char *ready_field) allowed_fields.find(ready_field) == allowed_fields.end()) { return 0.0; } + if (!db_) + return 0.0; - std::string sql = "SELECT CASE WHEN COUNT(*) > 0 THEN " - "CAST(SUM(gn." + - std::string(ready_field) + - ") AS REAL) / COUNT(*) " - "ELSE 0.0 END FROM graph_nodes gn " - "WHERE gn.project_id = ? AND gn.node_type IN (0,1)"; - sqlite3_stmt *stmt = nullptr; - double ratio = 0.0; - if (sqlite3_prepare_v2(db_, sql.c_str(), -1, &stmt, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) { - ratio = sqlite3_column_double(stmt, 0); + const int64_t pid = static_cast(project_id); + const std::string field(ready_field); + + // Per-entity coverage: ratio of the eligible population that carries the + // producer's output. + const char *probe = nullptr; + std::vector binds; + if (field == "callgraph_ready") { + probe = kReadyCallgraphSql; + binds = { pid, pid }; + } else if (field == "metrics_ready") { + probe = kReadyMetricsSql; + binds = { pid }; + } else if (field == "embedding_ready" || field == "vector_ready") { + probe = kReadyEmbeddingSql; + binds = { pid }; + } + if (probe) { + const int64_t eligible = + readyCount(db_, kReadyEligibleSql, { pid }); + const int64_t ready = readyCount(db_, probe, binds); + if (eligible < 0 || ready < 0) { + fprintf(stderr, + "store: readiness probe failed for %s " + "[module=store, method=getReadyRatio]\n", + ready_field); + return 0.0; } - sqlite3_finalize(stmt); + if (eligible == 0) + return 0.0; + return static_cast(ready) / + static_cast(eligible); } - return ratio; -} - -// ── Index Tasks (Tokio background task tracking) ──────────── -bool GraphStore::isFileUnchanged(uint64_t project_id, const char *file_path, - int64_t mtime, int64_t size) -{ - const char *sql = - "SELECT 1 FROM file_scan_state WHERE project_id=? AND file_path=? AND file_mtime=? AND file_size=?"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) - return false; - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_text(stmt, 2, file_path, -1, SQLITE_TRANSIENT); - sqlite3_bind_int64(stmt, 3, mtime); - sqlite3_bind_int64(stmt, 4, size); - bool unchanged = (sqlite3_step(stmt) == SQLITE_ROW); - sqlite3_finalize(stmt); - return unchanged; + // The remaining fields are pipeline-stage FLAGS, not per-entity coverage: + // project_readiness records whether the producer ran, which is what its + // schema comment says it is for (see engine_get_enhancement_status for the + // canonical count). No per-entity probe exists for them, so the ratio is + // 1.0 or 0.0. + return getProjectReadiness(project_id, ready_field) > 0 ? 1.0 : 0.0; } - std::unordered_set GraphStore::loadFileScanStateBatch(uint64_t project_id) { @@ -645,88 +472,120 @@ GraphStore::loadFileScanStateBatch(uint64_t project_id) sqlite3_finalize(stmt); return result; } - -void GraphStore::updateFileScanState(uint64_t project_id, const char *file_path, - int64_t mtime, int64_t size) -{ - const char *sql = - "INSERT OR REPLACE INTO file_scan_state (project_id, file_path, file_mtime, file_size) VALUES (?,?,?,?)"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) - return; - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_text(stmt, 2, file_path, -1, SQLITE_TRANSIENT); - sqlite3_bind_int64(stmt, 3, mtime); - sqlite3_bind_int64(stmt, 4, size); - sqlite3_step(stmt); - sqlite3_finalize(stmt); -} - void GraphStore::cleanupStaleFiles(uint64_t project_id, const std::vector &active_files) { - // Wrap in a transaction for atomicity + throughput (1 commit vs N). - exec("BEGIN IMMEDIATE"); - - // Create temp table if not exists (idempotent). - sqlite3_exec( - db_, - "CREATE TEMP TABLE IF NOT EXISTS _active_files (path TEXT PRIMARY KEY)", - nullptr, nullptr, nullptr); + // Nested SAVEPOINT instead of BEGIN IMMEDIATE/COMMIT: this runs inside + // the index transaction, where a plain BEGIN fails silently and the + // matching COMMIT would commit — and end — the caller's transaction. + if (!exec("SAVEPOINT cleanup_stale_files")) { + fprintf(stderr, + "store: cleanupStaleFiles SAVEPOINT failed: %s " + "[module=store, method=cleanupStaleFiles]\n", + error_.c_str()); + return; + } - // Clear previous active files in one shot. - sqlite3_exec(db_, "DELETE FROM _active_files", nullptr, nullptr, - nullptr); + bool ok = true; - // Reuse a single prepared INSERT for all files (1 prepare vs N prepares). - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2( - db_, - "INSERT OR IGNORE INTO _active_files (path) VALUES (?)", -1, - &stmt, nullptr) != SQLITE_OK) { + // Create the temp table (idempotent) and clear it in one shot. + if (!exec("CREATE TEMP TABLE IF NOT EXISTS _active_files " + "(path TEXT PRIMARY KEY)") || + !exec("DELETE FROM _active_files")) { fprintf(stderr, - "store: cleanupStaleFiles prepare insert failed: %s " + "store: cleanupStaleFiles temp table setup failed: %s " "[module=store, method=cleanupStaleFiles]\n", - sqlite3_errmsg(db_)); - exec("ROLLBACK"); - return; + error_.c_str()); + ok = false; } - for (const auto &f : active_files) { - // SQLITE_STATIC: avoid SQLite internal memcpy (caller owns the - // string for the duration of the step call). - sqlite3_bind_text(stmt, 1, f.c_str(), - static_cast(f.size()), SQLITE_STATIC); - sqlite3_step(stmt); - sqlite3_reset(stmt); + + if (ok) { + // Reuse a single prepared INSERT for all files (1 prepare vs N). + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2( + db_, + "INSERT OR IGNORE INTO _active_files (path) " + "VALUES (?)", + -1, &stmt, nullptr) != SQLITE_OK) { + fprintf(stderr, + "store: cleanupStaleFiles prepare insert " + "failed: %s " + "[module=store, method=cleanupStaleFiles]\n", + sqlite3_errmsg(db_)); + ok = false; + } else { + for (const auto &f : active_files) { + // SQLITE_STATIC: avoid SQLite internal memcpy + // (caller owns the string for the duration of + // the step call). + sqlite3_bind_text(stmt, 1, f.c_str(), + static_cast(f.size()), + SQLITE_STATIC); + if (sqlite3_step(stmt) != SQLITE_DONE) { + fprintf(stderr, + "store: cleanupStaleFiles " + "insert failed: %s " + "[module=store, method=" + "cleanupStaleFiles]\n", + sqlite3_errmsg(db_)); + ok = false; + sqlite3_reset(stmt); + break; + } + sqlite3_reset(stmt); + } + sqlite3_finalize(stmt); + } } - sqlite3_finalize(stmt); // Delete stale file_scan_state entries in one shot. - { + if (ok) { sqlite3_stmt *del = nullptr; if (sqlite3_prepare_v2(db_, "DELETE FROM file_scan_state " "WHERE project_id=? AND file_path NOT IN " "(SELECT path FROM _active_files)", - -1, &del, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(del, 1, - static_cast(project_id)); - sqlite3_step(del); - sqlite3_finalize(del); - } else { + -1, &del, nullptr) != SQLITE_OK) { fprintf(stderr, - "store: cleanupStaleFiles prepare delete failed: %s " + "store: cleanupStaleFiles prepare delete " + "failed: %s " "[module=store, method=cleanupStaleFiles]\n", sqlite3_errmsg(db_)); + ok = false; + } else { + sqlite3_bind_int64(del, 1, + static_cast(project_id)); + if (sqlite3_step(del) != SQLITE_DONE) { + fprintf(stderr, + "store: cleanupStaleFiles delete " + "failed: %s " + "[module=store, method=" + "cleanupStaleFiles]\n", + sqlite3_errmsg(db_)); + ok = false; + } + sqlite3_finalize(del); } } + if (!ok) { + exec("ROLLBACK TO SAVEPOINT cleanup_stale_files"); + exec("RELEASE SAVEPOINT cleanup_stale_files"); + return; + } + // Drop temp table (kept for now to match existing behavior; could // be retained across calls with a TRUNCATE pattern for further savings). sqlite3_exec(db_, "DROP TABLE IF EXISTS _active_files", nullptr, nullptr, nullptr); - exec("COMMIT"); + if (!exec("RELEASE SAVEPOINT cleanup_stale_files")) { + fprintf(stderr, + "store: cleanupStaleFiles RELEASE SAVEPOINT failed: %s " + "[module=store, method=cleanupStaleFiles]\n", + error_.c_str()); + exec("ROLLBACK TO SAVEPOINT cleanup_stale_files"); + } } // ─── Interactive Function Exploration ────────────────────────── diff --git a/engine/src/store/store_query.cpp b/engine/src/store/store_query.cpp index 099f187..f1eb601 100644 --- a/engine/src/store/store_query.cpp +++ b/engine/src/store/store_query.cpp @@ -1,3 +1,4 @@ +#include "util/json_writer.h" #include "store.h" #include "store_internal.h" #include "platform_win.h" @@ -5,9 +6,7 @@ #include #include #include -#include #include -#include #include #include #include @@ -22,108 +21,6 @@ namespace store { - -uint64_t GraphStore::createTask(uint64_t project_id, const char *task_type) -{ - const char *sql = - "INSERT INTO index_tasks (project_id, task_type) VALUES (?, ?)"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) { - error_ = "createTask: prepare failed"; - return 0; - } - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_text(stmt, 2, task_type, -1, SQLITE_TRANSIENT); - if (sqlite3_step(stmt) != SQLITE_DONE) { - error_ = "createTask: step failed"; - sqlite3_finalize(stmt); - return 0; - } - uint64_t id = static_cast(sqlite3_last_insert_rowid(db_)); - sqlite3_finalize(stmt); - return id; -} - -bool GraphStore::updateTask(uint64_t task_id, const char *status, int progress, - const char *error) -{ - // Guard against null status — strcmp below would dereference it - if (!status) - return false; - std::string sql = - "UPDATE index_tasks SET status = ?, progress = ?, error = ?"; - if (strcmp(status, "running") == 0) - sql += ", started_at = datetime('now')"; - else if (strcmp(status, "completed") == 0 || - strcmp(status, "failed") == 0) - sql += ", completed_at = datetime('now')"; - sql += " WHERE id = ?"; - - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql.c_str(), -1, &stmt, nullptr) != - SQLITE_OK) { - error_ = "updateTask: prepare failed"; - return false; - } - sqlite3_bind_text(stmt, 1, status, -1, SQLITE_TRANSIENT); - sqlite3_bind_int(stmt, 2, progress); - sqlite3_bind_text(stmt, 3, error ? error : "", -1, SQLITE_TRANSIENT); - sqlite3_bind_int64(stmt, 4, static_cast(task_id)); - if (sqlite3_step(stmt) != SQLITE_DONE) { - error_ = "updateTask: step failed"; - sqlite3_finalize(stmt); - return false; - } - sqlite3_finalize(stmt); - return true; -} - -std::string GraphStore::getTaskStatusJson(uint64_t project_id) -{ - const char *sql = "SELECT id, task_type, status, progress, error, " - "COALESCE(started_at,''), COALESCE(completed_at,'') " - "FROM index_tasks WHERE project_id = ? " - "ORDER BY id DESC LIMIT 1"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) { - return "{\"error\":\"getTaskStatusJson: prepare failed\"}"; - } - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) { - int64_t id = sqlite3_column_int64(stmt, 0); - const char *tt = reinterpret_cast( - sqlite3_column_text(stmt, 1)); - const char *st = reinterpret_cast( - sqlite3_column_text(stmt, 2)); - int pr = sqlite3_column_int(stmt, 3); - const char *er = reinterpret_cast( - sqlite3_column_text(stmt, 4)); - const char *sa = reinterpret_cast( - sqlite3_column_text(stmt, 5)); - const char *ca = reinterpret_cast( - sqlite3_column_text(stmt, 6)); - std::ostringstream json; - // Mirror sibling functions (getEntryPointsJson, tracePathJson, - // exploreFunctionJson) and escape all free-form text fields so a - // '"' or '\' in task_type/status/error/timestamps cannot break the - // JSON. [module=store, method=getTaskStatusJson] - json << "{" - << "\"id\":" << id << "," - << "\"task_type\":\"" << jsonEscape(tt ? tt : "") << "\"," - << "\"status\":\"" << jsonEscape(st ? st : "") << "\"," - << "\"progress\":" << pr << "," - << "\"error\":\"" << jsonEscape(er ? er : "") << "\"," - << "\"started_at\":\"" << jsonEscape(sa ? sa : "") << "\"," - << "\"completed_at\":\"" << jsonEscape(ca ? ca : "") - << "\"" - << "}"; - sqlite3_finalize(stmt); - return json.str(); - } - sqlite3_finalize(stmt); - return "{\"status\":\"none\"}"; -} - std::string GraphStore::searchUnifiedJson(uint64_t project_id, const char *query, int limit) { @@ -160,7 +57,13 @@ std::string GraphStore::searchUnifiedJson(uint64_t project_id, sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(db_, sql.c_str(), -1, &stmt, nullptr) == SQLITE_OK) { - std::string fts_query = query; + // fts5Phrase() wraps the raw query in double quotes so FTS5 + // treats it as a literal phrase instead of syntax: an unescaped + // `"` or `(` made the MATCH fail, and the failure was silent — + // callers saw "no results" for a query that was never run. The + // trailing `*` must sit OUTSIDE the quotes to stay a prefix + // search. + std::string fts_query = fts5Phrase(query); if (!fts_query.empty() && fts_query.back() != '*') fts_query += "*"; sqlite3_bind_text(stmt, 1, fts_query.c_str(), -1, @@ -192,6 +95,17 @@ std::string GraphStore::searchUnifiedJson(uint64_t project_id, "ms [module=store, " "method=searchUnifiedJson]\"}"; } + if (rc != SQLITE_DONE) { + // Any other step error used to fall through to the + // fallbacks with no trace, so a failed FTS query looked + // like a query that simply matched nothing. + error_ = sqlite3_errmsg(db_); + fprintf(stderr, + "searchUnifiedJson: fts step failed (rc=%d): " + "%s [module=store, " + "method=searchUnifiedJson]\n", + rc, error_.c_str()); + } } else { error_ = sqlite3_errmsg(db_); fprintf(stderr, @@ -256,6 +170,17 @@ std::string GraphStore::searchUnifiedJson(uint64_t project_id, "ms [module=store, " "method=searchUnifiedJson]\"}"; } + if (rc != SQLITE_DONE) { + // Any other step error used to fall through to the + // fallbacks with no trace, so a failed FTS query looked + // like a query that simply matched nothing. + error_ = sqlite3_errmsg(db_); + fprintf(stderr, + "searchUnifiedJson: fts step failed (rc=%d): " + "%s [module=store, " + "method=searchUnifiedJson]\n", + rc, error_.c_str()); + } } else { error_ = sqlite3_errmsg(db_); fprintf(stderr, @@ -332,676 +257,18 @@ std::string GraphStore::searchUnifiedJson(uint64_t project_id, for (size_t i = 0; i < results.size(); i++) { if (i > 0) json << ","; - json << "{" - << "\"node_id\":" << results[i].node_id << "," - << "\"name\":\"" << jsonEscape(results[i].name) << "\"," - << "\"qualified_name\":\"" - << jsonEscape(results[i].qualified_name) << "\"," - << "\"file_path\":\"" << jsonEscape(results[i].file_path) - << "\"" - << "}"; - } - json << "]}"; - return json.str(); -} - -std::string GraphStore::findCallersJson(uint64_t project_id, - const char *symbol_name, - const char *file_filter) -{ - // Collect ALL node IDs matching the name (a symbol may be declared - // many times: function/method/class/interface share a name). Edges - // point at specific declarations, so looking up only one id misses - // callers of the other declarations. No node_type restriction — - // all node_types qualify (function/method/class/interface/etc). - // - // file_filter: when non-NULL/non-empty, restricts the matched - // callee nodes to the given file. This disambiguates homonyms - // — symbols that share a name across files/classes (e.g. - // __init__, run, main). Without this filter, findCallersJson - // on a common name returns callers aggregated across ALL files - // in the project, producing heavy noise on large codebases. - std::vector gn_ids; - { - std::string id_sql = "SELECT id FROM graph_nodes " - "WHERE project_id = ? AND name = ?"; - if (file_filter && *file_filter) - id_sql += " AND file_path = ?"; - sqlite3_stmt *id_stmt = nullptr; - if (sqlite3_prepare_v2(db_, id_sql.c_str(), -1, &id_stmt, - nullptr) == SQLITE_OK) { - int b = 1; - sqlite3_bind_int64(id_stmt, b++, - static_cast(project_id)); - sqlite3_bind_text(id_stmt, b++, symbol_name, -1, - SQLITE_TRANSIENT); - if (file_filter && *file_filter) - sqlite3_bind_text(id_stmt, b++, file_filter, -1, - SQLITE_TRANSIENT); - while (sqlite3_step(id_stmt) == SQLITE_ROW) - gn_ids.push_back(static_cast( - sqlite3_column_int64(id_stmt, 0))); - sqlite3_finalize(id_stmt); - } - } - if (gn_ids.empty()) - return "{\"callers\":[]}"; - - std::ostringstream json; - json << "{\"callers\":["; - // edge_type 1=call, 3=symbol_reference (caller→callee). Both are - // call-like; restricting to edge_type=1 alone dropped 83% of edges. - // Build a parameterized IN (...) clause for all matching node ids. - std::string id_list; - for (size_t i = 0; i < gn_ids.size(); ++i) { - if (i > 0) - id_list += ","; - id_list += std::to_string(gn_ids[i]); - } - // Read resolve_strategy so the frontend can filter out third-party - // and unresolved callees. Populated by the Resolver Pipeline for - // edge_type=1 (call) edges; edge_type=3 (symbol_reference) edges are - // back-filled by buildTypeRefEdges(). - std::string ge_sql_str = - std::string("SELECT DISTINCT gn.id, gn.name, gn.node_type, " - "gn.file_path, gn.start_row, " - "ge.resolve_strategy " - "FROM graph_edges ge " - "JOIN graph_nodes gn ON gn.id = ge.source_node_id " - "WHERE ge.project_id = ? AND ge.edge_type IN (1,3) " - "AND ge.target_node_id IN (") + - id_list + ") ORDER BY gn.name LIMIT 50"; - sqlite3_stmt *ge_stmt = nullptr; - static const char *type_names[] = { - "function", "method", "class", "interface", "enum", "typealias", - }; - if (sqlite3_prepare_v2(db_, ge_sql_str.c_str(), -1, &ge_stmt, - nullptr) == SQLITE_OK) { - sqlite3_bind_int64(ge_stmt, 1, - static_cast(project_id)); - bool first = true; - while (sqlite3_step(ge_stmt) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - uint64_t gid = static_cast( - sqlite3_column_int64(ge_stmt, 0)); - int nt = sqlite3_column_int(ge_stmt, 2); - const char *gn = reinterpret_cast( - sqlite3_column_text(ge_stmt, 1)); - const char *fp = reinterpret_cast( - sqlite3_column_text(ge_stmt, 3)); - int ln = sqlite3_column_int(ge_stmt, 4); - const char *rs = reinterpret_cast( - sqlite3_column_text(ge_stmt, 5)); - const char *tn = (nt >= 0 && nt < 6) ? type_names[nt] : - "symbol"; - // Escape free-form text fields (name/file_path/resolve_strategy) - // so a '"' or '\' cannot break the JSON. kind (tn) is a fixed - // enum label and needs no escaping. - // [module=store, method=findCallersJson/findCalleesJson] - json << "{\"id\":" << gid << ",\"name\":\"" - << jsonEscape(gn ? gn : "") << "\"" - << ",\"kind\":\"" << tn << "\"" - << ",\"file_path\":\"" << jsonEscape(fp ? fp : "") - << "\"" - << ",\"line\":" << ln << ",\"resolve_strategy\":\"" - << jsonEscape(rs ? rs : "") << "\"}"; - } - sqlite3_finalize(ge_stmt); - } - json << "]}"; - return json.str(); -} - -std::string GraphStore::findCalleesJson(uint64_t project_id, - const char *symbol_name, - const char *file_filter) -{ - // Collect ALL node IDs matching the name (a symbol may be declared - // many times: function/method/class/interface share a name). Edges - // point at specific declarations, so looking up only one id misses - // callees of the other declarations. No node_type restriction — - // all node_types qualify (function/method/class/interface/etc). - // - // file_filter: when non-NULL/non-empty, restricts the matched - // caller nodes to the given file. See findCallersJson doc. - std::vector gn_ids; - { - std::string id_sql = "SELECT id FROM graph_nodes " - "WHERE project_id = ? AND name = ?"; - if (file_filter && *file_filter) - id_sql += " AND file_path = ?"; - sqlite3_stmt *id_stmt = nullptr; - if (sqlite3_prepare_v2(db_, id_sql.c_str(), -1, &id_stmt, - nullptr) == SQLITE_OK) { - int b = 1; - sqlite3_bind_int64(id_stmt, b++, - static_cast(project_id)); - sqlite3_bind_text(id_stmt, b++, symbol_name, -1, - SQLITE_TRANSIENT); - if (file_filter && *file_filter) - sqlite3_bind_text(id_stmt, b++, file_filter, -1, - SQLITE_TRANSIENT); - while (sqlite3_step(id_stmt) == SQLITE_ROW) - gn_ids.push_back(static_cast( - sqlite3_column_int64(id_stmt, 0))); - sqlite3_finalize(id_stmt); - } - } - if (gn_ids.empty()) - return "{\"callees\":[]}"; - - std::ostringstream json; - json << "{\"callees\":["; - // edge_type 1=call, 3=symbol_reference (caller→callee). Both are - // call-like; restricting to edge_type=1 alone dropped 83% of edges. - // Build a parameterized IN (...) clause for all matching node ids. - std::string id_list; - for (size_t i = 0; i < gn_ids.size(); ++i) { - if (i > 0) - id_list += ","; - id_list += std::to_string(gn_ids[i]); - } - std::string ge_sql_str = - std::string("SELECT DISTINCT gn.id, gn.name, gn.node_type, " - "gn.file_path, gn.start_row, " - "ge.resolve_strategy " - "FROM graph_edges ge " - "JOIN graph_nodes gn ON gn.id = ge.target_node_id " - "WHERE ge.project_id = ? AND ge.edge_type IN (1,3) " - "AND ge.source_node_id IN (") + - id_list + ") ORDER BY gn.name LIMIT 50"; - sqlite3_stmt *ge_stmt = nullptr; - static const char *type_names[] = { - "function", "method", "class", "interface", "enum", "typealias", - }; - if (sqlite3_prepare_v2(db_, ge_sql_str.c_str(), -1, &ge_stmt, - nullptr) == SQLITE_OK) { - sqlite3_bind_int64(ge_stmt, 1, - static_cast(project_id)); - bool first = true; - while (sqlite3_step(ge_stmt) == SQLITE_ROW) { - if (!first) - json << ","; - first = false; - uint64_t gid = static_cast( - sqlite3_column_int64(ge_stmt, 0)); - int nt = sqlite3_column_int(ge_stmt, 2); - const char *gn = reinterpret_cast( - sqlite3_column_text(ge_stmt, 1)); - const char *fp = reinterpret_cast( - sqlite3_column_text(ge_stmt, 3)); - int ln = sqlite3_column_int(ge_stmt, 4); - const char *rs = reinterpret_cast( - sqlite3_column_text(ge_stmt, 5)); - const char *tn = (nt >= 0 && nt < 6) ? type_names[nt] : - "symbol"; - // Escape free-form text fields (name/file_path/resolve_strategy) - // so a '"' or '\' cannot break the JSON. kind (tn) is a fixed - // enum label and needs no escaping. - // [module=store, method=findCallersJson/findCalleesJson] - json << "{\"id\":" << gid << ",\"name\":\"" - << jsonEscape(gn ? gn : "") << "\"" - << ",\"kind\":\"" << tn << "\"" - << ",\"file_path\":\"" << jsonEscape(fp ? fp : "") - << "\"" - << ",\"line\":" << ln << ",\"resolve_strategy\":\"" - << jsonEscape(rs ? rs : "") << "\"}"; - } - sqlite3_finalize(ge_stmt); - } - json << "]}"; - return json.str(); -} - -std::string GraphStore::getEntryPointsJson(uint64_t project_id) -{ - const char *sql = - "SELECT gn.id, gn.name, gn.node_type, gn.file_path, gn.start_row, ep.kind as ep_kind " - "FROM entry_points ep " - "JOIN graph_nodes gn ON gn.id = ep.symbol_id " - "WHERE ep.project_id = ? " - "ORDER BY ep.kind"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != SQLITE_OK) { - return "{\"error\":\"getEntryPointsJson: prepare " - "failed\",\"entry_points\":[]}"; - } - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - - std::ostringstream json; - // Collect entries grouped by kind - struct Ep { - int64_t id; - std::string name, kind, file_path; - int line; - std::string ep_kind; - }; - std::vector entries; - while (sqlite3_step(stmt) == SQLITE_ROW) { - Ep e; - e.id = sqlite3_column_int64(stmt, 0); - { - const char *s = reinterpret_cast( - sqlite3_column_text(stmt, 1)); - e.name = s ? s : ""; - } - { - const char *s = reinterpret_cast( - sqlite3_column_text(stmt, 2)); - e.kind = s ? s : ""; - } - { - const char *s = reinterpret_cast( - sqlite3_column_text(stmt, 3)); - e.file_path = s ? s : ""; - } - e.line = sqlite3_column_int(stmt, 4); - { - const char *s = reinterpret_cast( - sqlite3_column_text(stmt, 5)); - e.ep_kind = s ? s : ""; - } - entries.push_back(std::move(e)); - } - sqlite3_finalize(stmt); - - if (entries.empty()) - return "{\"entry_points\":[]}"; - - // Group by ep_kind - json << "{\"entry_points\":{"; - bool first_kind = true; - std::string current_kind; - std::sort(entries.begin(), entries.end(), [](const Ep &a, const Ep &b) { - return a.ep_kind < b.ep_kind; - }); - for (size_t i = 0; i < entries.size(); i++) { - if (entries[i].ep_kind != current_kind) { - if (!first_kind) - json << "]},"; - first_kind = false; - current_kind = entries[i].ep_kind; - json << "\"" << current_kind << "\":["; - } else { - json << ","; - } - json << "{\"id\":" << entries[i].id << ",\"name\":\"" - << jsonEscape(entries[i].name) << "\"" - << ",\"kind\":\"" << entries[i].kind << "\"" - << ",\"file\":\"" << jsonEscape(entries[i].file_path) - << "\"" - << ",\"line\":" << entries[i].line << "}"; - } - json << "]}}"; - return json.str(); -} - -// ── Path Tracing (BFS on call_edges) ────────────────────────── - -std::string GraphStore::tracePathJson(uint64_t project_id, - const char *from_name, - const char *to_name) -{ - // 1. Find symbol IDs - auto syms = [&](const char *name) -> uint64_t { - const char *sql = - "SELECT id FROM graph_nodes WHERE project_id = ? AND name = ? " - "ORDER BY node_type LIMIT 1"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != - SQLITE_OK) - return 0; - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_text(stmt, 2, name, -1, SQLITE_TRANSIENT); - uint64_t id = 0; - if (sqlite3_step(stmt) == SQLITE_ROW) - id = static_cast( - sqlite3_column_int64(stmt, 0)); - sqlite3_finalize(stmt); - return id; - }; - - uint64_t from_id = syms(from_name); - uint64_t to_id = syms(to_name); - if (!from_id || !to_id) - return "{\"path\":[],\"error\":\"symbol not found\"}"; - if (from_id == to_id) - return "{\"path\":[{\"name\":\"" + std::string(from_name) + - "\"}],\"trivial\":true}"; - - // 2. BFS through call_edges: parent map = callee → caller - std::unordered_map parent; - std::queue q; - std::unordered_set visited; - - q.push(from_id); - visited.insert(from_id); - bool found = false; - - while (!q.empty() && !found) { - uint64_t cur = q.front(); - q.pop(); - - const char *sql = - "SELECT target_node_id FROM graph_edges " - "WHERE project_id = ? AND source_node_id = ? AND edge_type IN (1,3)"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != - SQLITE_OK) - continue; - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_int64(stmt, 2, static_cast(cur)); - - while (sqlite3_step(stmt) == SQLITE_ROW) { - uint64_t callee = static_cast( - sqlite3_column_int64(stmt, 0)); - if (visited.count(callee)) - continue; - visited.insert(callee); - parent[callee] = cur; - if (callee == to_id) { - found = true; - break; - } - q.push(callee); - } - sqlite3_finalize(stmt); - } - - if (!found) - return "{\"path\":[],\"error\":\"no path found\"}"; - - // 3. Reconstruct path: to_id → ... → from_id - std::vector path_ids; - for (uint64_t id = to_id; id != from_id; id = parent[id]) - path_ids.push_back(id); - path_ids.push_back(from_id); - std::reverse(path_ids.begin(), path_ids.end()); - - // 4. Build JSON with name, file, line - std::ostringstream json; - json << "{\"path\":["; - bool first = true; - for (auto id : path_ids) { - if (!first) - json << ","; - first = false; - - const char *sql = - "SELECT name, file_path, start_row FROM graph_nodes WHERE id = ?"; - sqlite3_stmt *stmt = nullptr; - std::string name, file; - int line = 0; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(stmt, 1, static_cast(id)); - if (sqlite3_step(stmt) == SQLITE_ROW) { - const char *n = reinterpret_cast( - sqlite3_column_text(stmt, 0)); - const char *f = reinterpret_cast( - sqlite3_column_text(stmt, 1)); - if (n) - name = n; - if (f) - file = f; - line = sqlite3_column_int(stmt, 2); - } - sqlite3_finalize(stmt); - } - json << "{\"name\":\"" << jsonEscape(name) << "\"," - << "\"file\":\"" << jsonEscape(file) << "\"," - << "\"line\":" << line << "}"; + util::JsonWriter el; + el.beginObject(); + el.key("node_id").value(results[i].node_id); + el.key("name").value(results[i].name); + el.key("qualified_name").value(results[i].qualified_name); + el.key("file_path").value(results[i].file_path); + el.endObject(); + json << el.str(); } json << "]}"; return json.str(); } - -// ── Incremental Indexing ───────────────────────────────────── - -std::string GraphStore::exploreFunctionJson(uint64_t project_id, - const char *function_name, - int depth, const char *direction) -{ - // Limit depth to prevent runaway recursion - if (depth > 5) - depth = 5; - if (depth < 0) - depth = 0; - // Guard against null string parameters — strcmp / std::string - // construction below would dereference them - if (!direction) - direction = ""; - if (!function_name) - function_name = ""; - - bool show_callers = (strcmp(direction, "callers") == 0 || - strcmp(direction, "both") == 0); - bool show_callees = (strcmp(direction, "callees") == 0 || - strcmp(direction, "both") == 0); - - // 1. Find the function in graph_nodes (new pipeline) or symbols (legacy) - auto findFuncId = [&](const char *name) -> uint64_t { - // Try graph_nodes first (new pipeline) - { - const char *sql = - "SELECT id FROM graph_nodes WHERE project_id = ? AND name = ? AND node_type IN (0,1,6) LIMIT 1"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64( - stmt, 1, - static_cast(project_id)); - sqlite3_bind_text(stmt, 2, name, -1, - SQLITE_TRANSIENT); - uint64_t id = 0; - if (sqlite3_step(stmt) == SQLITE_ROW) - id = static_cast( - sqlite3_column_int64(stmt, 0)); - sqlite3_finalize(stmt); - if (id) - return id; - } - } - // Fallback: try symbols table (legacy/scanner pipeline) - { - const char *sql = - "SELECT id FROM symbols WHERE project_id = ? AND name = ? LIMIT 1"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != - SQLITE_OK) - return 0; - sqlite3_bind_int64(stmt, 1, - static_cast(project_id)); - sqlite3_bind_text(stmt, 2, name, -1, SQLITE_TRANSIENT); - uint64_t id = 0; - if (sqlite3_step(stmt) == SQLITE_ROW) - id = static_cast( - sqlite3_column_int64(stmt, 0)); - sqlite3_finalize(stmt); - return id; - } - }; - - // 2. Recursive JSON builder - std::function buildNode = - [&](std::ostringstream &json, uint64_t id, int remaining) { - // Get function metadata — try graph_nodes first - const char *gn_sql = - "SELECT name, file_path, start_row FROM graph_nodes WHERE id = ? AND project_id = ?"; - sqlite3_stmt *stmt = nullptr; - std::string name = "?"; - std::string file_path = ""; - int line = 0; - bool found = false; - if (sqlite3_prepare_v2(db_, gn_sql, -1, &stmt, - nullptr) == SQLITE_OK) { - sqlite3_bind_int64(stmt, 1, - static_cast(id)); - sqlite3_bind_int64( - stmt, 2, - static_cast(project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) { - const char *n = - reinterpret_cast( - sqlite3_column_text( - stmt, 0)); - if (n) - name = n; - const char *f = - reinterpret_cast( - sqlite3_column_text( - stmt, 1)); - if (f) - file_path = f; - line = sqlite3_column_int(stmt, 2); - found = true; - } - sqlite3_finalize(stmt); - } - // Fallback: symbols table - if (!found) { - const char *s_sql = - "SELECT name, file_path, line FROM symbols WHERE id = ? AND project_id = ?"; - if (sqlite3_prepare_v2(db_, s_sql, -1, &stmt, - nullptr) == SQLITE_OK) { - sqlite3_bind_int64( - stmt, 1, - static_cast(id)); - sqlite3_bind_int64(stmt, 2, - static_cast( - project_id)); - if (sqlite3_step(stmt) == SQLITE_ROW) { - const char *n = reinterpret_cast< - const char *>( - sqlite3_column_text( - stmt, 0)); - if (n) - name = n; - const char *f = reinterpret_cast< - const char *>( - sqlite3_column_text( - stmt, 1)); - if (f) - file_path = f; - line = sqlite3_column_int(stmt, - 2); - } - sqlite3_finalize(stmt); - } - } - - json << "{\"name\":\"" << jsonEscape(name) - << "\",\"file\":\"" << jsonEscape(file_path) - << "\",\"line\":" << line; - - if (remaining <= 0) { - json << "}"; - return; - } - - bool has_fields = - true; // name, file, line already written - - // Callers: graph_edges where target_node_id = id (call + symbol_reference) - if (show_callers) { - if (has_fields) - json << ","; - has_fields = true; - json << "\"callers\":["; - const char *csql = - "SELECT source_node_id FROM graph_edges " - "WHERE project_id = ? AND target_node_id = ? AND edge_type IN (1,3) " - "AND source_node_id != ? LIMIT 20"; - sqlite3_stmt *cstmt = nullptr; - bool first = true; - if (sqlite3_prepare_v2(db_, csql, -1, &cstmt, - nullptr) == SQLITE_OK) { - sqlite3_bind_int64(cstmt, 1, - static_cast( - project_id)); - sqlite3_bind_int64( - cstmt, 2, - static_cast(id)); - sqlite3_bind_int64( - cstmt, 3, - static_cast(id)); - while (sqlite3_step(cstmt) == - SQLITE_ROW) { - uint64_t caller_id = static_cast< - uint64_t>( - sqlite3_column_int64( - cstmt, 0)); - if (!first) - json << ","; - first = false; - buildNode(json, caller_id, - remaining - 1); - } - sqlite3_finalize(cstmt); - } - json << "]"; - } - - // Callees: graph_edges where source_node_id = id (call + symbol_reference) - if (show_callees) { - if (has_fields) - json << ","; - has_fields = true; - json << "\"callees\":["; - const char *csql = - "SELECT target_node_id FROM graph_edges " - "WHERE project_id = ? AND source_node_id = ? AND edge_type IN (1,3) " - "AND target_node_id != ? LIMIT 20"; - sqlite3_stmt *cstmt = nullptr; - bool first = true; - if (sqlite3_prepare_v2(db_, csql, -1, &cstmt, - nullptr) == SQLITE_OK) { - sqlite3_bind_int64(cstmt, 1, - static_cast( - project_id)); - sqlite3_bind_int64( - cstmt, 2, - static_cast(id)); - sqlite3_bind_int64( - cstmt, 3, - static_cast(id)); - while (sqlite3_step(cstmt) == - SQLITE_ROW) { - uint64_t callee_id = static_cast< - uint64_t>( - sqlite3_column_int64( - cstmt, 0)); - if (!first) - json << ","; - first = false; - buildNode(json, callee_id, - remaining - 1); - } - sqlite3_finalize(cstmt); - } - json << "]"; - } - - json << "}"; - }; - - // 3. Find starting function and build tree - uint64_t func_id = findFuncId(function_name); - if (!func_id) { - std::ostringstream err; - err << "{\"error\":\"function '" << jsonEscape(function_name) - << "' not found\",\"name\":\"" << jsonEscape(function_name) - << "\",\"callers\":[],\"callees\":[]}"; - return err.str(); - } - - std::ostringstream result; - buildNode(result, func_id, depth); - return result.str(); -} - // ─── Semantic Records (DB-first pipeline) ─────────────────── } // namespace store diff --git a/engine/src/store/store_schema.cpp b/engine/src/store/store_schema.cpp index 89a7c96..0d7a0a7 100644 --- a/engine/src/store/store_schema.cpp +++ b/engine/src/store/store_schema.cpp @@ -124,7 +124,7 @@ bool GraphStore::createSchema() module_path TEXT NOT NULL DEFAULT '', -- v0.5+: mirrors semantic_records.arity so the Resolver Pipeline -- can disambiguate same-name overloads (init()/init(int)) without - -- a JOIN per candidate. See CODE_REVIEW_FINDINGS_2026-07-19.md C2. + -- a JOIN per candidate. arity INTEGER NOT NULL DEFAULT 0, -- v0.2.5: per-function code metrics. Computed once in the parse -- worker (computeMetricsFromCST/computeMetricsFromUnit), staged in @@ -682,6 +682,14 @@ bool GraphStore::createSchema() FOREIGN KEY (module_id) REFERENCES scope(id) ); + -- One summary row per module. Without this key, the model + -- builder's INSERT OR REPLACE never conflicted (the primary + -- key is an AUTOINCREMENT id), so every enhance pass appended + -- a fresh set of rows and SUM(dead_entities) multiplied by the + -- number of passes — the T5 "dead_code.entities > total" bug. + CREATE UNIQUE INDEX IF NOT EXISTS idx_module_summary_unique + ON module_summary(project_id, module_id); + CREATE TABLE IF NOT EXISTS capability_state ( id INTEGER PRIMARY KEY AUTOINCREMENT, project_id INTEGER NOT NULL, @@ -708,6 +716,10 @@ CREATE TABLE IF NOT EXISTS architecture_state ( project_id INTEGER NOT NULL, layer TEXT NOT NULL, violations INTEGER NOT NULL DEFAULT 0, + -- Real cross-module call count for this module pair. Normal dependencies + -- are NOT violations: there is no layer model, so `violations` stays 0 and + -- the dependency count lives here (see buildArchitectureState). + cross_module_edges INTEGER NOT NULL DEFAULT 0, compliance REAL NOT NULL DEFAULT 1.0, evidence TEXT NOT NULL DEFAULT '[]', FOREIGN KEY (project_id) REFERENCES projects(id) @@ -716,8 +728,13 @@ CREATE TABLE IF NOT EXISTS architecture_state ( CREATE TABLE IF NOT EXISTS architecture_edge ( id INTEGER PRIMARY KEY AUTOINCREMENT, project_id INTEGER NOT NULL, - layer_upper TEXT NOT NULL, -- e.g. "Controller" - layer_lower TEXT NOT NULL, -- e.g. "Service" + -- Module PATHS, not layers: the caller's module and the + -- callee's. They were called layer_upper/layer_lower with + -- "-- e.g. \"Controller\"" comments until v0.7, which made every + -- reader treat them as a layer model (the root cause of the false + -- layer-violation reporting, 2026-09-18 review #3). + caller_module TEXT NOT NULL, -- module path of the caller + callee_module TEXT NOT NULL, -- module path of the callee entity_id INTEGER NOT NULL, -- FK to entity.id FOREIGN KEY (project_id) REFERENCES projects(id), FOREIGN KEY (entity_id) REFERENCES entity(id) @@ -813,828 +830,13 @@ CREATE TABLE IF NOT EXISTS architecture_edge ( // Execute main schema bool ok = exec(schema); - // ── Schema migrations for pre-existing databases ─────────────── - // CREATE TABLE IF NOT EXISTS skips when the table already exists, so columns - // added in later versions must be patched in here. SQLite has no - // "ADD COLUMN IF NOT EXISTS", so we probe PRAGMA table_info first. - - // Migration: add knowledge_ready column to project_readiness (v0.5+) - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, - "PRAGMA table_info(project_readiness)", - -1, &probe, nullptr) == SQLITE_OK) { - bool has_knowledge_ready = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col && - std::string(col) == "knowledge_ready") - has_knowledge_ready = true; - } - sqlite3_finalize(probe); - if (!has_knowledge_ready) { - exec("ALTER TABLE project_readiness " - "ADD COLUMN knowledge_ready INTEGER DEFAULT 0"); - } - } - } - - // Migration: add module_state column to entity table (v0.5+) - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, "PRAGMA table_info(entity)", -1, - &probe, nullptr) == SQLITE_OK) { - bool has_module_state = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col && std::string(col) == "module_state") - has_module_state = true; - } - sqlite3_finalize(probe); - if (!has_module_state) { - exec("ALTER TABLE entity " - "ADD COLUMN module_state " - "INTEGER NOT NULL DEFAULT 0"); - } - } - } - - // Migration: add module_path column to entity table (v0.9+) - // Denormalizes the directory portion of file_path so that scope - // JOINs and module_edge grouping can use an indexed equality predicate - // instead of the non-sargable rtrim(file_path, replace(...)) expression. - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, "PRAGMA table_info(entity)", -1, - &probe, nullptr) == SQLITE_OK) { - bool has_module_path = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col && std::string(col) == "module_path") - has_module_path = true; - } - sqlite3_finalize(probe); - if (!has_module_path) { - // Wrap ALTER + backfill UPDATE + CREATE INDEX in - // a single transaction. Without this, a crash after - // ALTER leaves the column existing with '' values; - // the next startup sees the column and skips the - // migration, so the backfill never reruns and - // scope/state_builder JOINs break silently. - if (!exec("BEGIN IMMEDIATE")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "BEGIN module_path migration " - "failed: %s\n", - error_.c_str()); - return false; - } - if (!exec("ALTER TABLE entity " - "ADD COLUMN module_path " - "TEXT NOT NULL DEFAULT ''")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "ALTER TABLE entity ADD module_path " - "failed: %s\n", - error_.c_str()); - exec("ROLLBACK"); - return false; - } - // Backfill module_path for pre-existing entity rows - // using the same rtrim(replace(...)) expression - // used at INSERT time in store_graph.cpp. Without - // this, migrated databases would have module_path='' - // for all existing entities, breaking scope creation, - // state_builder JOINs, and module_edge grouping. - if (!exec("UPDATE entity SET module_path = " - "rtrim(file_path, " - "replace(file_path, '/', 'x')) " - "WHERE module_path = ''")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "UPDATE entity backfill module_path " - "failed: %s\n", - error_.c_str()); - exec("ROLLBACK"); - return false; - } - if (!exec("CREATE INDEX IF NOT EXISTS " - "idx_entity_module " - "ON entity(project_id, module_path)")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "CREATE INDEX idx_entity_module " - "failed: %s\n", - error_.c_str()); - exec("ROLLBACK"); - return false; - } - if (!exec("COMMIT")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "COMMIT module_path migration " - "failed: %s\n", - error_.c_str()); - exec("ROLLBACK"); - return false; - } - } - } - } - - // Migration: add arity + is_static columns to semantic_records (v0.5+) - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, - "PRAGMA table_info(semantic_records)", - -1, &probe, nullptr) == SQLITE_OK) { - bool has_arity = false; - bool has_is_static = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col) { - if (std::string(col) == "arity") - has_arity = true; - if (std::string(col) == "is_static") - has_is_static = true; - } - } - sqlite3_finalize(probe); - if (!has_arity) { - exec("ALTER TABLE semantic_records " - "ADD COLUMN arity INTEGER DEFAULT 0"); - } - if (!has_is_static) { - exec("ALTER TABLE semantic_records " - "ADD COLUMN is_static INTEGER DEFAULT 0"); - } - } - } - - // Migration (v0.2.5): add code-metrics columns to entity for databases - // created before the metrics restore. Each column defaults to 0 so - // existing rows stay valid; resolveStagedMetrics fills them on the next - // index/enhance run. Mirrors the entity DDL in createSchema(). - { - struct EntityMetricCol { - const char *name; - const char *dflt; - }; - static const EntityMetricCol kCols[] = { - { "cyclomatic", "0" }, { "nesting_depth", "0" }, - { "cognitive", "0" }, { "param_count", "0" }, - { "call_count", "0" }, { "branch_count", "0" }, - { "loop_count", "0" }, { "lines", "0" }, - { "is_stub", "0" }, - }; - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, "PRAGMA table_info(entity)", -1, - &probe, nullptr) == SQLITE_OK) { - std::vector existing; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col) - existing.emplace_back(col); - } - sqlite3_finalize(probe); - for (const auto &c : kCols) { - if (std::find(existing.begin(), existing.end(), - c.name) != existing.end()) - continue; - exec(("ALTER TABLE entity ADD COLUMN " + - std::string(c.name) + - " INTEGER NOT NULL DEFAULT " + c.dflt) - .c_str()); - } - } else { - fprintf(stderr, - "createSchema: entity metrics migration probe " - "failed: %s [module=store, method=createSchema]\n", - sqlite3_errmsg(db_)); - } - } - - // Migration (v0.2.5): add metrics_ready to project_readiness for - // databases created before the metrics restore. Mirrors the DDL column - // in createSchema(). - { - sqlite3_stmt *rprobe = nullptr; - bool has_metrics_ready = false; - if (sqlite3_prepare_v2(db_, - "PRAGMA table_info(project_readiness)", - -1, &rprobe, nullptr) == SQLITE_OK) { - while (sqlite3_step(rprobe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(rprobe, 1)); - if (col && std::string(col) == "metrics_ready") - has_metrics_ready = true; - } - sqlite3_finalize(rprobe); - } - if (!has_metrics_ready) { - exec("ALTER TABLE project_readiness " - "ADD COLUMN metrics_ready INTEGER DEFAULT 0"); - } - } - - // Migration: add type_info + type_ref tables (v0.6+) - { - // Add route table if missing - sqlite3_stmt *rprobe = nullptr; - if (sqlite3_prepare_v2(db_, - "SELECT name FROM sqlite_master " - "WHERE type='table' AND name='route'", - -1, &rprobe, nullptr) == SQLITE_OK) { - if (sqlite3_step(rprobe) != SQLITE_ROW) { - sqlite3_finalize(rprobe); - exec("CREATE TABLE IF NOT EXISTS route (" - " id INTEGER PRIMARY KEY AUTOINCREMENT," - " project_id INTEGER NOT NULL," - " method TEXT NOT NULL," - " path TEXT NOT NULL," - " handler_name TEXT DEFAULT ''," - " file_path TEXT NOT NULL," - " start_row INTEGER DEFAULT 0," - " start_col INTEGER DEFAULT 0," - " FOREIGN KEY (project_id) REFERENCES projects(id)" - ")"); - exec("CREATE INDEX IF NOT EXISTS idx_route_path " - "ON route(project_id, method, path)"); - } else { - sqlite3_finalize(rprobe); - } - } - - // Add type_name column to semantic_records if missing - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, - "PRAGMA table_info(semantic_records)", - -1, &probe, nullptr) == SQLITE_OK) { - bool has_type_name = false; - bool has_call_kind = false; - bool has_resolve_strategy = false; - bool has_qualified_target = false; - bool has_receiver_text = false; - bool has_receiver_type = false; - bool has_import_alias = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col) { - const std::string c(col); - if (c == "type_name") - has_type_name = true; - if (c == "call_kind") - has_call_kind = true; - if (c == "resolve_strategy") - has_resolve_strategy = true; - if (c == "qualified_target") - has_qualified_target = true; - if (c == "receiver_text") - has_receiver_text = true; - if (c == "receiver_type") - has_receiver_type = true; - if (c == "import_alias") - has_import_alias = true; - } - } - sqlite3_finalize(probe); - if (!has_type_name) { - exec("ALTER TABLE semantic_records " - "ADD COLUMN type_name TEXT DEFAULT ''"); - } - if (!has_call_kind) { - exec("ALTER TABLE semantic_records " - "ADD COLUMN call_kind INTEGER DEFAULT 0"); - } - if (!has_resolve_strategy) { - exec("ALTER TABLE semantic_records " - "ADD COLUMN resolve_strategy " - "TEXT DEFAULT ''"); - } - // Step 3 (plan §3.1): structured call-fact columns. - if (!has_qualified_target) { - exec("ALTER TABLE semantic_records " - "ADD COLUMN qualified_target " - "TEXT DEFAULT ''"); - } - if (!has_receiver_text) { - exec("ALTER TABLE semantic_records " - "ADD COLUMN receiver_text " - "TEXT DEFAULT ''"); - } - if (!has_receiver_type) { - exec("ALTER TABLE semantic_records " - "ADD COLUMN receiver_type " - "TEXT DEFAULT ''"); - } - if (!has_import_alias) { - exec("ALTER TABLE semantic_records " - "ADD COLUMN import_alias " - "TEXT DEFAULT ''"); - } - } - - // Step 3 (plan §3.1): migrate the `reference` table with the - // same structured call-fact columns plus call_site_file. SQLite - // has no ADD COLUMN IF NOT EXISTS, so probe table_info first. - { - sqlite3_stmt *ref_probe = nullptr; - if (sqlite3_prepare_v2( - db_, "PRAGMA table_info(reference)", -1, - &ref_probe, nullptr) == SQLITE_OK) { - bool has_qualified_target = false; - bool has_receiver_text = false; - bool has_receiver_type = false; - bool has_import_alias = false; - bool has_call_site_file = false; - while (sqlite3_step(ref_probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text( - ref_probe, 1)); - if (col) { - const std::string c(col); - if (c == "qualified_target") - has_qualified_target = - true; - if (c == "receiver_text") - has_receiver_text = - true; - if (c == "receiver_type") - has_receiver_type = - true; - if (c == "import_alias") - has_import_alias = true; - if (c == "call_site_file") - has_call_site_file = - true; - } - } - sqlite3_finalize(ref_probe); - if (!has_qualified_target) - exec("ALTER TABLE reference ADD COLUMN " - "qualified_target TEXT DEFAULT ''"); - if (!has_receiver_text) - exec("ALTER TABLE reference ADD COLUMN " - "receiver_text TEXT DEFAULT ''"); - if (!has_receiver_type) - exec("ALTER TABLE reference ADD COLUMN " - "receiver_type TEXT DEFAULT ''"); - if (!has_import_alias) - exec("ALTER TABLE reference ADD COLUMN " - "import_alias TEXT DEFAULT ''"); - if (!has_call_site_file) - exec("ALTER TABLE reference ADD COLUMN " - "call_site_file TEXT DEFAULT ''"); - } - } - - // Step 6 (plan §6.1): migrate the `relation` table with - // provenance columns. SQLite has no ADD COLUMN IF NOT EXISTS, - // so probe table_info first. Each new column is nullable with - // a default so pre-existing rows and non-call relations are - // not affected. - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2( - db_, "PRAGMA table_info(relation)", -1, - &probe, nullptr) == SQLITE_OK) { - bool has_confidence = false; - bool has_resolver = false; - bool has_res_kind = false; - bool has_reason = false; - bool has_csf = false; - bool has_csr = false; - bool has_csc = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text( - probe, 1)); - if (!col) - continue; - std::string c = col; - if (c == "confidence") - has_confidence = true; - else if (c == "resolver") - has_resolver = true; - else if (c == "resolution_kind") - has_res_kind = true; - else if (c == "reason") - has_reason = true; - else if (c == "call_site_file") - has_csf = true; - else if (c == "call_site_row") - has_csr = true; - else if (c == "call_site_col") - has_csc = true; - } - sqlite3_finalize(probe); - if (!has_confidence) - exec("ALTER TABLE relation ADD COLUMN " - "confidence REAL DEFAULT 0.0"); - if (!has_resolver) - exec("ALTER TABLE relation ADD COLUMN " - "resolver TEXT DEFAULT ''"); - if (!has_res_kind) - exec("ALTER TABLE relation ADD COLUMN " - "resolution_kind TEXT DEFAULT ''"); - if (!has_reason) - exec("ALTER TABLE relation ADD COLUMN " - "reason TEXT DEFAULT ''"); - if (!has_csf) - exec("ALTER TABLE relation ADD COLUMN " - "call_site_file TEXT DEFAULT ''"); - if (!has_csr) - exec("ALTER TABLE relation ADD COLUMN " - "call_site_row INTEGER DEFAULT 0"); - if (!has_csc) - exec("ALTER TABLE relation ADD COLUMN " - "call_site_col INTEGER DEFAULT 0"); - } - } - - // Create type_info table if missing - sqlite3_stmt *probe2 = nullptr; - if (sqlite3_prepare_v2(db_, - "SELECT name FROM sqlite_master " - "WHERE type='table' AND name='type_info'", - -1, &probe2, nullptr) == SQLITE_OK) { - if (sqlite3_step(probe2) != SQLITE_ROW) { - sqlite3_finalize(probe2); - exec("CREATE TABLE IF NOT EXISTS type_info (" - " id INTEGER PRIMARY KEY AUTOINCREMENT," - " project_id INTEGER NOT NULL," - " name TEXT NOT NULL," - " qualified_name TEXT DEFAULT ''," - " kind INTEGER NOT NULL," - " file_path TEXT NOT NULL," - " language TEXT DEFAULT ''," - " start_row INTEGER DEFAULT 0," - " start_col INTEGER DEFAULT 0," - " end_row INTEGER DEFAULT 0," - " end_col INTEGER DEFAULT 0," - " FOREIGN KEY (project_id) REFERENCES projects(id)" - ")"); - exec("CREATE INDEX IF NOT EXISTS idx_ti_name " - "ON type_info(project_id, name)"); - exec("CREATE INDEX IF NOT EXISTS idx_ti_qn " - "ON type_info(project_id, qualified_name)"); - } else { - sqlite3_finalize(probe); - } - } - } - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, - "SELECT name FROM sqlite_master " - "WHERE type='table' AND name='type_ref'", - -1, &probe, nullptr) == SQLITE_OK) { - if (sqlite3_step(probe) != SQLITE_ROW) { - sqlite3_finalize(probe); - exec("CREATE TABLE IF NOT EXISTS type_ref (" - " id INTEGER PRIMARY KEY AUTOINCREMENT," - " project_id INTEGER NOT NULL," - " entity_id INTEGER NOT NULL," - " type_name TEXT NOT NULL," - " kind INTEGER NOT NULL," - " file_path TEXT NOT NULL," - " start_row INTEGER DEFAULT 0," - " start_col INTEGER DEFAULT 0," - " FOREIGN KEY (project_id) REFERENCES projects(id)," - " FOREIGN KEY (entity_id) REFERENCES entity(id)" - ")"); - exec("CREATE INDEX IF NOT EXISTS idx_tr_type " - "ON type_ref(project_id, type_name)"); - exec("CREATE INDEX IF NOT EXISTS idx_tr_entity " - "ON type_ref(project_id, entity_id)"); - } else { - sqlite3_finalize(probe); - } - } - } - - // Note: vec0 embeddings table is created in engine_init() after - // sqlite-vec extension is loaded via dlopen. Not needed here. - - // Migration: add role column to module_summary table (v0.7+) - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, "PRAGMA table_info(module_summary)", - -1, &probe, nullptr) == SQLITE_OK) { - bool has_role = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col && std::string(col) == "role") - has_role = true; - } - sqlite3_finalize(probe); - if (!has_role) { - exec("ALTER TABLE module_summary " - "ADD COLUMN role TEXT DEFAULT ''"); - } - } - } - - // Migration: add visibility column to entity table (v0.2.2) - // 0 = private (default), 1 = pub/public/export, 2 = protected (Java/C# reserved) - // Populated by Visitors per language: Rust pub→1, Go exported-uppercase→1, - // Python __leading→0 else→1, C/C++ header-declared→1 static-anon→0, - // Java public→1 private→0 protected→2, JS/TS export→1 else→0, Swift public/open→1. - // role classifier (state_builder.cpp buildModuleSummaries) fuses pub_count - // from this column with call-graph counts — see docs/dev_plans/role_classifier_plan.md. - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, "PRAGMA table_info(entity)", -1, - &probe, nullptr) == SQLITE_OK) { - bool has_visibility = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col && std::string(col) == "visibility") - has_visibility = true; - } - sqlite3_finalize(probe); - if (!has_visibility) { - exec("ALTER TABLE entity " - "ADD COLUMN visibility INTEGER NOT NULL DEFAULT 0"); - } - } - } - - // Migration: add arity column to entity table (v0.5+, C2) - // The Resolver Pipeline (resolver/pipeline.cpp) SELECTs entity.arity to - // score same-name overload candidates via factorSignatureMatch. Without - // this column the SELECT fails with "no such column: arity", breaking - // the entire resolve run. Backfill from semantic_records (which already - // has arity populated by Visitors) so pre-existing entity rows get the - // correct arity without a re-index. Matches by (project_id, file_path, - // name, start_row) — the same identity used by buildGraph's _r2n JOIN. - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, "PRAGMA table_info(entity)", -1, - &probe, nullptr) == SQLITE_OK) { - bool has_arity = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col && std::string(col) == "arity") - has_arity = true; - } - sqlite3_finalize(probe); - if (!has_arity) { - if (!exec("ALTER TABLE entity " - "ADD COLUMN arity INTEGER NOT NULL DEFAULT 0")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "ALTER TABLE entity ADD arity " - "failed: %s\n", - error_.c_str()); - return false; - } - // Backfill arity from semantic_records. Each - // declaration record (kind 0/1) carries the - // visitor-computed argument count. Call records - // (kind 9) are skipped — entity rows are - // declarations only. - if (!exec("UPDATE entity SET arity = COALESCE(" - " (SELECT sr.arity FROM semantic_records sr" - " WHERE sr.project_id = entity.project_id" - " AND sr.file_path = entity.file_path" - " AND sr.name = entity.name" - " AND sr.start_row = entity.start_row" - " AND sr.kind IN (0, 1)), 0)")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "UPDATE entity backfill arity " - "failed: %s\n", - error_.c_str()); - return false; - } - } - } - } - - // Migration: add call_kind column to reference table (v0.7+) - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, "PRAGMA table_info(reference)", -1, - &probe, nullptr) == SQLITE_OK) { - bool has_ck = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col && std::string(col) == "call_kind") - has_ck = true; - } - sqlite3_finalize(probe); - if (!has_ck) { - exec("ALTER TABLE reference " - "ADD COLUMN call_kind INTEGER DEFAULT 0"); - } - // Check for resolve_strategy column (v0.9+) - bool has_ref_rs = false; - // Reuse the same probe - re-prepare - probe = nullptr; - if (sqlite3_prepare_v2( - db_, "PRAGMA table_info(reference)", -1, - &probe, nullptr) == SQLITE_OK) { - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text( - probe, 1)); - if (col && std::string(col) == - "resolve_strategy") - has_ref_rs = true; - } - sqlite3_finalize(probe); - probe = nullptr; - } - if (!has_ref_rs) { - exec("ALTER TABLE reference " - "ADD COLUMN resolve_strategy " - "TEXT DEFAULT ''"); - } - } - } - - // Migration: add parent_id column to graph_nodes (v0.8+) - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, "PRAGMA table_info(graph_nodes)", - -1, &probe, nullptr) == SQLITE_OK) { - bool has_pid = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col && std::string(col) == "parent_id") - has_pid = true; - } - sqlite3_finalize(probe); - if (!has_pid) { - exec("ALTER TABLE graph_nodes " - "ADD COLUMN parent_id INTEGER DEFAULT 0"); - exec("CREATE INDEX IF NOT EXISTS idx_gn_parent " - "ON graph_nodes(project_id, parent_id)"); - } - } - } - - // Migration: add resolve_strategy column to graph_edges (v0.9+) - // Propagated from semantic_records → reference → _resolved_edges - // by the Resolver Pipeline. Stores the resolution strategy for - // each call edge: "p1_intra" (intra-file resolved), - // "external" (known builtin/third-party), "unresolved" (unknown). - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2(db_, "PRAGMA table_info(graph_edges)", - -1, &probe, nullptr) == SQLITE_OK) { - bool has_rs = false; - while (sqlite3_step(probe) == SQLITE_ROW) { - const char *col = - reinterpret_cast( - sqlite3_column_text(probe, 1)); - if (col && - std::string(col) == "resolve_strategy") - has_rs = true; - } - sqlite3_finalize(probe); - if (!has_rs) { - exec("ALTER TABLE graph_edges " - "ADD COLUMN resolve_strategy " - "TEXT DEFAULT ''"); - } - } - } - - // Migration: add semantic_fact table (v0.3 Phase 1). - // The table is in the main schema string, but pre-existing - // databases created before v0.3 need it added here. Probing - // sqlite_master (not PRAGMA table_info) because the table may not - // exist at all on legacy databases. - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2( - db_, - "SELECT name FROM sqlite_master " - "WHERE type='table' AND name='semantic_fact'", - -1, &probe, nullptr) == SQLITE_OK) { - if (sqlite3_step(probe) != SQLITE_ROW) { - sqlite3_finalize(probe); - if (!exec("CREATE TABLE IF NOT EXISTS semantic_fact (" - " id INTEGER PRIMARY KEY," - " project_id INTEGER NOT NULL," - " function_id INTEGER NOT NULL," - " category TEXT NOT NULL," - " primitive TEXT NOT NULL," - " kind TEXT NOT NULL," - " symbol TEXT NOT NULL DEFAULT ''," - " confidence REAL NOT NULL DEFAULT 1.0," - " detail_json TEXT," - " created_at TEXT DEFAULT (datetime('now'))," - " FOREIGN KEY (project_id) REFERENCES projects(id)," - " FOREIGN KEY (function_id) REFERENCES graph_nodes(id)" - ")")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "CREATE TABLE semantic_fact failed: %s\n", - error_.c_str()); - return false; - } - if (!exec("CREATE INDEX IF NOT EXISTS idx_sf_category " - "ON semantic_fact(project_id, category)")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "CREATE INDEX idx_sf_category failed: %s\n", - error_.c_str()); - return false; - } - if (!exec("CREATE INDEX IF NOT EXISTS idx_sf_primitive " - "ON semantic_fact(project_id, primitive)")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "CREATE INDEX idx_sf_primitive failed: %s\n", - error_.c_str()); - return false; - } - if (!exec("CREATE INDEX IF NOT EXISTS idx_sf_category_primitive " - "ON semantic_fact(project_id, category, primitive)")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "CREATE INDEX idx_sf_category_primitive failed: %s\n", - error_.c_str()); - return false; - } - if (!exec("CREATE INDEX IF NOT EXISTS idx_sf_function " - "ON semantic_fact(project_id, function_id)")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "CREATE INDEX idx_sf_function failed: %s\n", - error_.c_str()); - return false; - } - } else { - sqlite3_finalize(probe); - } - } - } - - // Migration: add project_state table (v0.3 Phase 4 prep). - // Schema is added now to avoid a future migration; the table is - // unused until Phase 4. Same sqlite_master probe pattern as above. - { - sqlite3_stmt *probe = nullptr; - if (sqlite3_prepare_v2( - db_, - "SELECT name FROM sqlite_master " - "WHERE type='table' AND name='project_state'", - -1, &probe, nullptr) == SQLITE_OK) { - if (sqlite3_step(probe) != SQLITE_ROW) { - sqlite3_finalize(probe); - if (!exec("CREATE TABLE IF NOT EXISTS project_state (" - " id INTEGER PRIMARY KEY," - " project_id INTEGER NOT NULL UNIQUE," - " confidence REAL NOT NULL DEFAULT 0.0," - " snapshot_json TEXT NOT NULL," - " created_at TEXT DEFAULT (datetime('now'))," - " updated_at TEXT DEFAULT (datetime('now'))," - " FOREIGN KEY (project_id) REFERENCES projects(id)" - ")")) { - fprintf(stderr, - "[module=store, method=createSchema] " - "CREATE TABLE project_state failed: %s\n", - error_.c_str()); - return false; - } - } else { - sqlite3_finalize(probe); - } - } - } + // The per-column migrations that follow live in + // store_schema_migrations.cpp (1000-line rule). They run regardless of + // `ok`, exactly as before this split, and a hard migration failure + // still fails the whole schema build. + bool migrations_ok = runSchemaMigrations(); (void)ok; - - return ok; + return ok && migrations_ok; } bool GraphStore::createIndexesAfterBulkLoad(uint64_t project_id, diff --git a/engine/src/store/store_schema_migrations.cpp b/engine/src/store/store_schema_migrations.cpp new file mode 100644 index 0000000..9858977 --- /dev/null +++ b/engine/src/store/store_schema_migrations.cpp @@ -0,0 +1,882 @@ +// store_schema_migrations.cpp — per-column schema migrations. +// +// Split out of store_schema.cpp (see plan/rules/code_rules.md 1000-line +// rule). createSchema() executes the main DDL and then calls this to patch +// databases that already exist: SQLite has no "ADD COLUMN IF NOT EXISTS", +// so every column added in a later version is probed with PRAGMA +// table_info and ALTER TABLE'd in here. CREATE TABLE IF NOT EXISTS does +// nothing for a table that already exists, so these migrations are the +// only reason an old database can be re-indexed without a full rebuild. +// +// A hard failure returns false so createSchema() fails loudly instead of +// leaving a half-migrated schema that later queries misinterpret. + +#include "store.h" + +#include +#include +#include +#include + +namespace store +{ + +bool GraphStore::runSchemaMigrations() +{ + // ── Schema migrations for pre-existing databases ─────────────── + // CREATE TABLE IF NOT EXISTS skips when the table already exists, so columns + // added in later versions must be patched in here. SQLite has no + // "ADD COLUMN IF NOT EXISTS", so we probe PRAGMA table_info first. + + // ── Migration failure collection ─────────────────────────────── + // Every ALTER TABLE below runs through migrationExec so a failure is + // recorded and reported at the end of the pass. Thirty of these call sites + // used to ignore the result while the function still returned true, leaving + // a half-migrated schema whose queries later failed with "no such column" — + // far from the cause (2026-09-18 review #19). Wrapping the + // calls in one place covers every existing site and any future one, and + // catches any cause of a failed ALTER, not only a missing column. + bool migration_ok = true; + auto migrationExec = [&](const char *sql) -> bool { + if (exec(sql)) + return true; + migration_ok = false; + fprintf(stderr, + "[module=store, method=runSchemaMigrations] migration " + "statement failed: %s | sql=%.100s\n", + error().c_str(), sql ? sql : "(null)"); + return false; + }; + // Probe helper: a PRAGMA prepare failure must fail the pass, not + // silently skip the migration (a skipped column later dies with + // "no such column", far from the cause). + auto migrationProbe = [&](sqlite3_stmt **probe, const char *sql, + const char *what) -> bool { + if (sqlite3_prepare_v2(db_, sql, -1, probe, nullptr) == + SQLITE_OK) + return true; + migration_ok = false; + fprintf(stderr, + "[module=store, method=runSchemaMigrations] probe " + "%s failed: %s | sql=%.100s\n", + what ? what : "(unknown)", sqlite3_errmsg(db_), + sql ? sql : "(null)"); + return false; + }; + + // Migration: add knowledge_ready column to project_readiness (v0.5+) + { + sqlite3_stmt *probe = nullptr; + if (sqlite3_prepare_v2(db_, + "PRAGMA table_info(project_readiness)", + -1, &probe, nullptr) != SQLITE_OK) { + // A failed probe must fail the schema pass: skipping + // the migration leaves a half-migrated schema and + // later queries die with "no such column" (code_rules: + // no silent error handling). + fprintf(stderr, + "runSchemaMigrations: probe project_readiness " + "failed: %s " + "[module=store, method=runSchemaMigrations]\n", + sqlite3_errmsg(db_)); + if (probe) + sqlite3_finalize(probe); + return false; + } + bool has_knowledge_ready = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col && std::string(col) == "knowledge_ready") + has_knowledge_ready = true; + } + sqlite3_finalize(probe); + if (!has_knowledge_ready) { + if (!migrationExec( + "ALTER TABLE project_readiness " + "ADD COLUMN knowledge_ready INTEGER DEFAULT 0")) + return false; + } + } + + // Migration: rename architecture_edge's layer_upper/layer_lower to + // caller_module/callee_module (v0.7+). + // + // The columns never held layers — architecture.cpp writes module paths into + // them (caller's and callee's respectively) — and the old names plus the + // old `-- e.g. "Controller"` comments are why the model layer treated a + // normal cross-module dependency as a layer violation. The behaviour was + // corrected separately; this removes the cause. + // + // Probe for the OLD name: a database created by the current schema already + // has the new columns and must not be touched. RENAME COLUMN needs SQLite + // >= 3.25; a failure is reported through migrationExec rather than leaving + // a half-renamed schema (see #19). + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe(&probe, + "PRAGMA table_info(architecture_edge)", + "architecture_edge")) + return false; + if (probe) { + bool has_old_name = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col && std::string(col) == "layer_upper") + has_old_name = true; + } + sqlite3_finalize(probe); + if (has_old_name) { + migrationExec( + "ALTER TABLE architecture_edge " + "RENAME COLUMN layer_upper TO caller_module"); + migrationExec( + "ALTER TABLE architecture_edge " + "RENAME COLUMN layer_lower TO callee_module"); + } + } + } + + // Migration: add module_state column to entity table (v0.5+) + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe(&probe, "PRAGMA table_info(entity)", + "schema-probe")) + return false; + if (probe) { + bool has_module_state = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col && std::string(col) == "module_state") + has_module_state = true; + } + sqlite3_finalize(probe); + if (!has_module_state) { + migrationExec("ALTER TABLE entity " + "ADD COLUMN module_state " + "INTEGER NOT NULL DEFAULT 0"); + } + } + } + + // Migration: add module_path column to entity table (v0.9+) + // Denormalizes the directory portion of file_path so that scope + // JOINs and module_edge grouping can use an indexed equality predicate + // instead of the non-sargable rtrim(file_path, replace(...)) expression. + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe(&probe, "PRAGMA table_info(entity)", + "schema-probe")) + return false; + if (probe) { + bool has_module_path = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col && std::string(col) == "module_path") + has_module_path = true; + } + sqlite3_finalize(probe); + if (!has_module_path) { + // Wrap ALTER + backfill UPDATE + CREATE INDEX in + // a single transaction. Without this, a crash after + // ALTER leaves the column existing with '' values; + // the next startup sees the column and skips the + // migration, so the backfill never reruns and + // scope/state_builder JOINs break silently. + if (!exec("BEGIN IMMEDIATE")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "BEGIN module_path migration " + "failed: %s\n", + error_.c_str()); + return false; + } + if (!migrationExec( + "ALTER TABLE entity " + "ADD COLUMN module_path " + "TEXT NOT NULL DEFAULT ''")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "ALTER TABLE entity ADD module_path " + "failed: %s\n", + error_.c_str()); + exec("ROLLBACK"); + return false; + } + // Backfill module_path for pre-existing entity rows + // using the same rtrim(replace(...)) expression + // used at INSERT time in store_graph.cpp. Without + // this, migrated databases would have module_path='' + // for all existing entities, breaking scope creation, + // state_builder JOINs, and module_edge grouping. + if (!exec("UPDATE entity SET module_path = " + "rtrim(file_path, " + "replace(file_path, '/', 'x')) " + "WHERE module_path = ''")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "UPDATE entity backfill module_path " + "failed: %s\n", + error_.c_str()); + exec("ROLLBACK"); + return false; + } + if (!exec("CREATE INDEX IF NOT EXISTS " + "idx_entity_module " + "ON entity(project_id, module_path)")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "CREATE INDEX idx_entity_module " + "failed: %s\n", + error_.c_str()); + exec("ROLLBACK"); + return false; + } + if (!exec("COMMIT")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "COMMIT module_path migration " + "failed: %s\n", + error_.c_str()); + exec("ROLLBACK"); + return false; + } + } + } + } + + // Migration: add arity + is_static columns to semantic_records (v0.5+) + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe(&probe, + "PRAGMA table_info(semantic_records)", + "schema-probe")) + return false; + if (probe) { + bool has_arity = false; + bool has_is_static = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col) { + if (std::string(col) == "arity") + has_arity = true; + if (std::string(col) == "is_static") + has_is_static = true; + } + } + sqlite3_finalize(probe); + if (!has_arity) { + migrationExec( + "ALTER TABLE semantic_records " + "ADD COLUMN arity INTEGER DEFAULT 0"); + } + if (!has_is_static) { + migrationExec( + "ALTER TABLE semantic_records " + "ADD COLUMN is_static INTEGER DEFAULT 0"); + } + } + } + + // Migration (v0.2.5): add code-metrics columns to entity for databases + // created before the metrics restore. Each column defaults to 0 so + // existing rows stay valid; resolveStagedMetrics fills them on the next + // index/enhance run. Mirrors the entity DDL in createSchema(). + { + struct EntityMetricCol { + const char *name; + const char *dflt; + }; + static const EntityMetricCol kCols[] = { + { "cyclomatic", "0" }, { "nesting_depth", "0" }, + { "cognitive", "0" }, { "param_count", "0" }, + { "call_count", "0" }, { "branch_count", "0" }, + { "loop_count", "0" }, { "lines", "0" }, + { "is_stub", "0" }, + }; + sqlite3_stmt *probe = nullptr; + if (!migrationProbe(&probe, "PRAGMA table_info(entity)", + "schema-probe")) + return false; + if (probe) { + std::vector existing; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col) + existing.emplace_back(col); + } + sqlite3_finalize(probe); + for (const auto &c : kCols) { + if (std::find(existing.begin(), existing.end(), + c.name) != existing.end()) + continue; + migrationExec( + ("ALTER TABLE entity ADD COLUMN " + + std::string(c.name) + + " INTEGER NOT NULL DEFAULT " + c.dflt) + .c_str()); + } + } else { + fprintf(stderr, + "createSchema: entity metrics migration probe " + "failed: %s [module=store, method=createSchema]\n", + sqlite3_errmsg(db_)); + } + } + + // Migration (v0.2.5): add metrics_ready to project_readiness for + // databases created before the metrics restore. Mirrors the DDL column + // in createSchema(). + { + sqlite3_stmt *rprobe = nullptr; + bool has_metrics_ready = false; + if (!migrationProbe(&rprobe, + "PRAGMA table_info(project_readiness)", + "schema-probe")) + return false; + if (rprobe) { + while (sqlite3_step(rprobe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(rprobe, 1)); + if (col && std::string(col) == "metrics_ready") + has_metrics_ready = true; + } + sqlite3_finalize(rprobe); + } + if (!has_metrics_ready) { + migrationExec( + "ALTER TABLE project_readiness " + "ADD COLUMN metrics_ready INTEGER DEFAULT 0"); + } + } + + // Route / type_info / type_ref live in their own TU (1000-line rule): see + // store_schema_migrations_types.cpp. Failures inside it are recorded by + // migrationExec; the return value is surfaced here so no path is silent. + if (!migrateTypeTables(migrationExec)) + migration_ok = false; + + // Note: vec0 embeddings table is created in engine_init() after + // sqlite-vec extension is loaded via dlopen. Not needed here. + + // Migration: add role column to module_summary table (v0.7+) + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe(&probe, "PRAGMA table_info(module_summary)", + "schema-probe")) + return false; + if (probe) { + bool has_role = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col && std::string(col) == "role") + has_role = true; + } + sqlite3_finalize(probe); + if (!has_role) { + migrationExec( + "ALTER TABLE module_summary " + "ADD COLUMN role TEXT DEFAULT ''"); + } + } + } + + // Migration: one module_summary row per (project_id, module_id). + // The model builder writes with INSERT OR REPLACE, but before the + // UNIQUE index existed the only conflict target was the + // AUTOINCREMENT id — so every enhance pass appended another full + // set of rows and SUM(dead_entities) grew by one module's worth per + // pass (T5 finding #4: dead_code.entities > dead_code.total). + // Existing DBs must be deduplicated before the index can be + // created; keep the newest row (highest id) per module. + { + migrationExec("DELETE FROM module_summary WHERE id NOT IN (" + " SELECT MAX(id) FROM module_summary" + " GROUP BY project_id, module_id)"); + migrationExec("CREATE UNIQUE INDEX IF NOT EXISTS " + "idx_module_summary_unique " + "ON module_summary(project_id, module_id)"); + } + + // Migration: add visibility column to entity table (v0.2.2) + // 0 = private (default), 1 = pub/public/export, 2 = protected (Java/C# reserved) + // Populated by Visitors per language: Rust pub→1, Go exported-uppercase→1, + // Python __leading→0 else→1, C/C++ header-declared→1 static-anon→0, + // Java public→1 private→0 protected→2, JS/TS export→1 else→0, Swift public/open→1. + // role classifier (state_builder.cpp buildModuleSummaries) fuses pub_count + // from this column with call-graph counts. + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe(&probe, "PRAGMA table_info(entity)", + "schema-probe")) + return false; + if (probe) { + bool has_visibility = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col && std::string(col) == "visibility") + has_visibility = true; + } + sqlite3_finalize(probe); + if (!has_visibility) { + migrationExec( + "ALTER TABLE entity " + "ADD COLUMN visibility INTEGER NOT NULL DEFAULT 0"); + } + } + } + + // Migration: add arity column to entity table (v0.5+, C2) + // The Resolver Pipeline (resolver/pipeline.cpp) SELECTs entity.arity to + // score same-name overload candidates via factorSignatureMatch. Without + // this column the SELECT fails with "no such column: arity", breaking + // the entire resolve run. Backfill from semantic_records (which already + // has arity populated by Visitors) so pre-existing entity rows get the + // correct arity without a re-index. Matches by (project_id, file_path, + // name, start_row) — the same identity used by buildGraph's _r2n JOIN. + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe(&probe, "PRAGMA table_info(entity)", + "schema-probe")) + return false; + if (probe) { + bool has_arity = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col && std::string(col) == "arity") + has_arity = true; + } + sqlite3_finalize(probe); + if (!has_arity) { + if (!migrationExec( + "ALTER TABLE entity " + "ADD COLUMN arity INTEGER NOT NULL DEFAULT 0")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "ALTER TABLE entity ADD arity " + "failed: %s\n", + error_.c_str()); + return false; + } + // Backfill arity from semantic_records. Each + // declaration record (kind 0/1) carries the + // visitor-computed argument count. Call records + // (kind 9) are skipped — entity rows are + // declarations only. + if (!exec("UPDATE entity SET arity = COALESCE(" + " (SELECT sr.arity FROM semantic_records sr" + " WHERE sr.project_id = entity.project_id" + " AND sr.file_path = entity.file_path" + " AND sr.name = entity.name" + " AND sr.start_row = entity.start_row" + " AND sr.kind IN (0, 1)), 0)")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "UPDATE entity backfill arity " + "failed: %s\n", + error_.c_str()); + return false; + } + } + } + } + + // Migration: add call_kind column to reference table (v0.7+) + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe(&probe, "PRAGMA table_info(reference)", + "schema-probe")) + return false; + if (probe) { + bool has_ck = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col && std::string(col) == "call_kind") + has_ck = true; + } + sqlite3_finalize(probe); + if (!has_ck) { + migrationExec( + "ALTER TABLE reference " + "ADD COLUMN call_kind INTEGER DEFAULT 0"); + } + // Check for resolve_strategy column (v0.9+) + bool has_ref_rs = false; + // Reuse the same probe - re-prepare + probe = nullptr; + if (!migrationProbe(&probe, + "PRAGMA table_info(reference)", + "schema-probe")) + return false; + if (probe) { + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text( + probe, 1)); + if (col && std::string(col) == + "resolve_strategy") + has_ref_rs = true; + } + sqlite3_finalize(probe); + probe = nullptr; + } + if (!has_ref_rs) { + migrationExec("ALTER TABLE reference " + "ADD COLUMN resolve_strategy " + "TEXT DEFAULT ''"); + } + } + } + + // Migration: add parent_id column to graph_nodes (v0.8+) + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe(&probe, "PRAGMA table_info(graph_nodes)", + "schema-probe")) + return false; + if (probe) { + bool has_pid = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col && std::string(col) == "parent_id") + has_pid = true; + } + sqlite3_finalize(probe); + if (!has_pid) { + migrationExec( + "ALTER TABLE graph_nodes " + "ADD COLUMN parent_id INTEGER DEFAULT 0"); + exec("CREATE INDEX IF NOT EXISTS idx_gn_parent " + "ON graph_nodes(project_id, parent_id)"); + } + } + } + + // Migration: add resolve_strategy column to graph_edges (v0.9+) + // Propagated from semantic_records → reference → _resolved_edges + // by the Resolver Pipeline. Stores the resolution strategy for + // each call edge: "p1_intra" (intra-file resolved), + // "external" (known builtin/third-party), "unresolved" (unknown). + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe(&probe, "PRAGMA table_info(graph_edges)", + "schema-probe")) + return false; + if (probe) { + bool has_rs = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col && + std::string(col) == "resolve_strategy") + has_rs = true; + } + sqlite3_finalize(probe); + if (!has_rs) { + migrationExec("ALTER TABLE graph_edges " + "ADD COLUMN resolve_strategy " + "TEXT DEFAULT ''"); + } + } + } + + // Migration: add semantic_fact table (v0.3 Phase 1). + // The table is in the main schema string, but pre-existing + // databases created before v0.3 need it added here. Probing + // sqlite_master (not PRAGMA table_info) because the table may not + // exist at all on legacy databases. + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe( + &probe, + "SELECT name FROM sqlite_master " + "WHERE type='table' AND name='semantic_fact'", + "schema-probe")) + return false; + if (probe) { + if (sqlite3_step(probe) != SQLITE_ROW) { + sqlite3_finalize(probe); + if (!exec("CREATE TABLE IF NOT EXISTS semantic_fact (" + " id INTEGER PRIMARY KEY," + " project_id INTEGER NOT NULL," + " function_id INTEGER NOT NULL," + " category TEXT NOT NULL," + " primitive TEXT NOT NULL," + " kind TEXT NOT NULL," + " symbol TEXT NOT NULL DEFAULT ''," + " confidence REAL NOT NULL DEFAULT 1.0," + " detail_json TEXT," + " created_at TEXT DEFAULT (datetime('now'))," + " FOREIGN KEY (project_id) REFERENCES projects(id)," + " FOREIGN KEY (function_id) REFERENCES graph_nodes(id)" + ")")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "CREATE TABLE semantic_fact failed: %s\n", + error_.c_str()); + return false; + } + if (!exec("CREATE INDEX IF NOT EXISTS idx_sf_category " + "ON semantic_fact(project_id, category)")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "CREATE INDEX idx_sf_category failed: %s\n", + error_.c_str()); + return false; + } + if (!exec("CREATE INDEX IF NOT EXISTS idx_sf_primitive " + "ON semantic_fact(project_id, primitive)")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "CREATE INDEX idx_sf_primitive failed: %s\n", + error_.c_str()); + return false; + } + if (!exec("CREATE INDEX IF NOT EXISTS idx_sf_category_primitive " + "ON semantic_fact(project_id, category, primitive)")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "CREATE INDEX idx_sf_category_primitive failed: %s\n", + error_.c_str()); + return false; + } + if (!exec("CREATE INDEX IF NOT EXISTS idx_sf_function " + "ON semantic_fact(project_id, function_id)")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "CREATE INDEX idx_sf_function failed: %s\n", + error_.c_str()); + return false; + } + } else { + sqlite3_finalize(probe); + } + } + } + + // Migration: add project_state table (v0.3 Phase 4 prep). + // Schema is added now to avoid a future migration; the table is + // unused until Phase 4. Same sqlite_master probe pattern as above. + { + sqlite3_stmt *probe = nullptr; + if (!migrationProbe( + &probe, + "SELECT name FROM sqlite_master " + "WHERE type='table' AND name='project_state'", + "schema-probe")) + return false; + if (probe) { + if (sqlite3_step(probe) != SQLITE_ROW) { + sqlite3_finalize(probe); + if (!exec("CREATE TABLE IF NOT EXISTS project_state (" + " id INTEGER PRIMARY KEY," + " project_id INTEGER NOT NULL UNIQUE," + " confidence REAL NOT NULL DEFAULT 0.0," + " snapshot_json TEXT NOT NULL," + " created_at TEXT DEFAULT (datetime('now'))," + " updated_at TEXT DEFAULT (datetime('now'))," + " FOREIGN KEY (project_id) REFERENCES projects(id)" + ")")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "CREATE TABLE project_state failed: %s\n", + error_.c_str()); + return false; + } + } else { + sqlite3_finalize(probe); + } + } + } + + // Migration: collapse duplicate capability rows. The capability pass runs on + // every index and the table has no uniqueness rule, so each run used to + // append another copy of every capability; the insert is guarded now + // (store_knowledge.cpp::insertCapability) and this removes the copies that + // earlier runs left behind, so capability_state and the drift counts derived + // from it stop double-counting. + migrationExec( + "DELETE FROM capability WHERE id NOT IN " + "(SELECT MIN(id) FROM capability GROUP BY project_id, name, " + " source_kind, source_ref)"); + + // Migration: add cross_module_edges to architecture_state. + // + // architecture_state.violations used to hold the COUNT of cross-module + // call pairs per module pair, with compliance forced to 0.0 — so every + // normal dependency was reported as an architecture violation and + // dragged the architecture score down. The count is real information; + // the label was wrong. It now lives in its own column, `violations` + // stays 0 until a real layer violation can be detected, and + // compliance stays 1.0. Hard failures fail the schema build loudly + // (see the header) instead of leaving a half-migrated table that + // buildArchitectureState then fails to INSERT into. + { + sqlite3_stmt *arch_probe = nullptr; + bool has_cross_module_edges = false; + if (!migrationProbe(&arch_probe, + "PRAGMA table_info(architecture_state)", + "schema-probe")) + return false; + if (arch_probe) { + while (sqlite3_step(arch_probe) == SQLITE_ROW) { + const char *col = reinterpret_cast( + sqlite3_column_text(arch_probe, 1)); + if (col && + std::string(col) == "cross_module_edges") + has_cross_module_edges = true; + } + sqlite3_finalize(arch_probe); + arch_probe = nullptr; + } else { + fprintf(stderr, + "[module=store, method=createSchema] " + "architecture_state column probe " + "failed: %s\n", + error_.c_str()); + return false; + } + if (!has_cross_module_edges) { + if (!migrationExec("ALTER TABLE architecture_state " + "ADD COLUMN cross_module_edges " + "INTEGER NOT NULL DEFAULT 0")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "ALTER TABLE architecture_state ADD " + " cross_module_edges failed: %s\n", + error_.c_str()); + return false; + } + } + // Data correction. Both statements run on every open and + // are idempotent, so a database that acquires bogus rows + // later (a pre-fix binary after a downgrade) is repaired + // on the next open instead of never. + // + // 1) Collapse duplicate (project_id, layer) rows. The old + // buildArchitectureState used INSERT OR IGNORE with no + // unique key to ignore on, so every rebuild appended a + // second copy of every row — and a re-index after the + // edge counts changed appended rows with different + // counts too. The NEWEST row is what the last rebuild + // wrote; summing the copies would inflate + // cross_module_edges by the number of rebuilds. + if (!migrationExec("DELETE FROM architecture_state " + "WHERE rowid NOT IN " + "(SELECT MAX(rowid) FROM architecture_state " + " GROUP BY project_id, layer)")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "architecture_state de-duplication " + "failed: %s\n", + error_.c_str()); + return false; + } + // 2) Move the pre-fix `violations` value into the column + // that names it honestly rather than discarding it, + // and clear the bogus violation flag. Unconditional: + // gating it on the ALTER above would miss a table that + // already has the column but pre-fix rows. Afterwards + // violations is 0 and the WHERE matches nothing. + if (!migrationExec("UPDATE architecture_state " + "SET cross_module_edges = violations, " + " violations = 0, " + " compliance = 1.0 " + "WHERE violations > 0")) { + fprintf(stderr, + "[module=store, method=createSchema] " + "architecture_state violations move " + "failed: %s\n", + error_.c_str()); + return false; + } + } + + // Migration: collapse duplicate scopes (kind=1 module + kind=2 function). + // A pre-fix buildGraph used `INSERT OR IGNORE INTO scope` with no UNIQUE + // key to ignore on, so each buildGraph re-appended every module row — the + // parallel indexer merges per-module worker DBs and then the post-index + // enhance runs buildGraph again, so every module ended up with ≥2 kind=1 + // rows. buildModuleSummaries GROUP BYs scope.id, so the duplicates + // doubled module_summary rows (and dead_code counts) and inflated + // project_overview.total_modules. The insert is guarded now + // (store_graph.cpp NOT EXISTS clause); this repairs databases the + // pre-fix binary left behind. Idempotent: after the first pass no + // duplicate remains and all three statements match nothing. + // + // 1) Repoint kind=2 function scopes whose parent points at a duplicate + // module scope to the surviving (lowest-id) module scope with the + // same (project_id, name), so no child is left dangling. + migrationExec("UPDATE scope AS child SET parent_id = (" + " SELECT MIN(keep.id) FROM scope keep, scope dup" + " WHERE dup.id = child.parent_id AND keep.kind = 1" + " AND keep.project_id = dup.project_id" + " AND keep.name = dup.name)" + " WHERE child.kind = 2 AND child.parent_id IN (" + " SELECT id FROM scope WHERE kind = 1 AND id NOT IN (" + " SELECT MIN(id) FROM scope WHERE kind = 1" + " GROUP BY project_id, name))"); + // 2) Delete the duplicate module scopes, keeping the lowest id per + // (project_id, name). + migrationExec("DELETE FROM scope WHERE kind = 1 AND id NOT IN (" + " SELECT MIN(id) FROM scope WHERE kind = 1" + " GROUP BY project_id, name)"); + // 3) Delete duplicate FUNCTION scopes (kind=2), keeping the lowest id per + // fully-identical row. + // + // The identity of a kind=2 row is the tuple buildGraph inserts — + // (project_id, parent_id, name, start_row, end_row) — NOT + // (project_id, parent_id, name) alone: several entities in one module + // legitimately share a name (one `init` per file) and are distinguished + // only by their source range. Grouping without the range would delete + // those distinct rows. + // + // Measured on the 1.1 GB self-index: 117,718 kind=2 rows collapse to + // 29,799 on this identity — 29,674 tuples carried duplicates and 87,919 + // rows were surplus, the residue of pre-fix accumulate-and-append runs. + // Runs AFTER step 1 on purpose: reparenting is what brings rows that + // share a logical (module, name, range) onto one parent_id, making them + // recognisable as duplicates. Nothing references a kind=2 scope id — + // import.source_scope_id targets the kind=1 module scope only (see + // buildGraph's UPDATE) and reference.scope_id is never written — so + // deleting these rows leaves no dangling reference. Idempotent. + migrationExec( + "DELETE FROM scope WHERE kind = 2 AND id NOT IN (" + " SELECT MIN(id) FROM scope WHERE kind = 2" + " GROUP BY project_id, parent_id, name, start_row, end_row)"); + + if (!migration_ok) { + fprintf(stderr, + "[module=store, method=runSchemaMigrations] one or more " + "migrations failed; refusing to report a migrated schema " + "(later queries would fail with \"no such column\")\n"); + return false; + } + + return true; +} + +} // namespace store diff --git a/engine/src/store/store_schema_migrations_types.cpp b/engine/src/store/store_schema_migrations_types.cpp new file mode 100644 index 0000000..c72e86f --- /dev/null +++ b/engine/src/store/store_schema_migrations_types.cpp @@ -0,0 +1,344 @@ +// store_schema_migrations_types.cpp — route / type_info / type_ref migration. +// +// Split out of store_schema_migrations.cpp (plan/rules/code_rules.md: no source +// file may exceed 1000 lines). It runs between the other migration groups in +// GraphStore::runSchemaMigrations, which hands its failure-recording callback in +// as `record`. +// +// Every statement runs through `run`, which delegates to `record` so a failure +// is recorded together with the failing SQL (see GraphStore::runSchemaMigrations). +// Eight DDL calls in this block called exec() directly and ignored the result, so +// a database could end up without the type_ref indexes while the migration pass +// still reported success. + +#include "store.h" + +#include + +#include +#include + +namespace store +{ + +bool GraphStore::migrateTypeTables( + const std::function &record) +{ + // `run` routes the statements that used to call exec() directly through + // `record`, so their failures are reported like every other one, and keeps + // the local flag so the caller learns whether the whole group completed. + bool ok = true; + auto run = [&](const char *sql) -> bool { + const bool done = record(sql); + ok = done && ok; + return done; + }; + // Probe helper: a failed prepare must fail this migration group, not + // silently skip the block. A skipped ALTER leaves a half-migrated schema + // whose queries later die with "no such column", far from the cause + // (code_rules §5: no silent error handling). Mirrors migrationProbe in + // store_schema_migrations.cpp. + auto probeStmt = [&](sqlite3_stmt **stmt, const char *sql, + const char *what) -> bool { + if (sqlite3_prepare_v2(db_, sql, -1, stmt, nullptr) == + SQLITE_OK) + return true; + ok = false; + fprintf(stderr, + "[module=store, method=migrateTypeTables] probe %s " + "failed: %s | sql=%.100s\n", + what ? what : "(unknown)", sqlite3_errmsg(db_), + sql ? sql : "(null)"); + return false; + }; + + // Migration: add type_info + type_ref tables (v0.6+) + { + // Add route table if missing + sqlite3_stmt *rprobe = nullptr; + if (probeStmt(&rprobe, + "SELECT name FROM sqlite_master " + "WHERE type='table' AND name='route'", + "route")) { + if (sqlite3_step(rprobe) != SQLITE_ROW) { + sqlite3_finalize(rprobe); + run("CREATE TABLE IF NOT EXISTS route (" + " id INTEGER PRIMARY KEY AUTOINCREMENT," + " project_id INTEGER NOT NULL," + " method TEXT NOT NULL," + " path TEXT NOT NULL," + " handler_name TEXT DEFAULT ''," + " file_path TEXT NOT NULL," + " start_row INTEGER DEFAULT 0," + " start_col INTEGER DEFAULT 0," + " FOREIGN KEY (project_id) REFERENCES projects(id)" + ")"); + run("CREATE INDEX IF NOT EXISTS idx_route_path " + "ON route(project_id, method, path)"); + } else { + sqlite3_finalize(rprobe); + } + } + + // Add type_name column to semantic_records if missing + sqlite3_stmt *probe = nullptr; + if (probeStmt(&probe, "PRAGMA table_info(semantic_records)", + "semantic_records")) { + bool has_type_name = false; + bool has_call_kind = false; + bool has_resolve_strategy = false; + bool has_qualified_target = false; + bool has_receiver_text = false; + bool has_receiver_type = false; + bool has_import_alias = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text(probe, 1)); + if (col) { + const std::string c(col); + if (c == "type_name") + has_type_name = true; + if (c == "call_kind") + has_call_kind = true; + if (c == "resolve_strategy") + has_resolve_strategy = true; + if (c == "qualified_target") + has_qualified_target = true; + if (c == "receiver_text") + has_receiver_text = true; + if (c == "receiver_type") + has_receiver_type = true; + if (c == "import_alias") + has_import_alias = true; + } + } + sqlite3_finalize(probe); + // Null out the handle so a later accidental reuse is a + // no-op (sqlite3_finalize(nullptr) is safe) instead of a + // use-after-free. See the type_info block below. + probe = nullptr; + if (!has_type_name) { + record("ALTER TABLE semantic_records " + "ADD COLUMN type_name TEXT DEFAULT ''"); + } + if (!has_call_kind) { + record("ALTER TABLE semantic_records " + "ADD COLUMN call_kind INTEGER DEFAULT 0"); + } + if (!has_resolve_strategy) { + record("ALTER TABLE semantic_records " + "ADD COLUMN resolve_strategy " + "TEXT DEFAULT ''"); + } + // Step 3 (plan §3.1): structured call-fact columns. + if (!has_qualified_target) { + record("ALTER TABLE semantic_records " + "ADD COLUMN qualified_target " + "TEXT DEFAULT ''"); + } + if (!has_receiver_text) { + record("ALTER TABLE semantic_records " + "ADD COLUMN receiver_text " + "TEXT DEFAULT ''"); + } + if (!has_receiver_type) { + record("ALTER TABLE semantic_records " + "ADD COLUMN receiver_type " + "TEXT DEFAULT ''"); + } + if (!has_import_alias) { + record("ALTER TABLE semantic_records " + "ADD COLUMN import_alias " + "TEXT DEFAULT ''"); + } + } + + // Step 3 (plan §3.1): migrate the `reference` table with the + // same structured call-fact columns plus call_site_file. SQLite + // has no ADD COLUMN IF NOT EXISTS, so probe table_info first. + { + sqlite3_stmt *ref_probe = nullptr; + if (probeStmt(&ref_probe, + "PRAGMA table_info(reference)", + "reference")) { + bool has_qualified_target = false; + bool has_receiver_text = false; + bool has_receiver_type = false; + bool has_import_alias = false; + bool has_call_site_file = false; + while (sqlite3_step(ref_probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text( + ref_probe, 1)); + if (col) { + const std::string c(col); + if (c == "qualified_target") + has_qualified_target = + true; + if (c == "receiver_text") + has_receiver_text = + true; + if (c == "receiver_type") + has_receiver_type = + true; + if (c == "import_alias") + has_import_alias = true; + if (c == "call_site_file") + has_call_site_file = + true; + } + } + sqlite3_finalize(ref_probe); + if (!has_qualified_target) + record("ALTER TABLE reference ADD COLUMN " + "qualified_target TEXT DEFAULT ''"); + if (!has_receiver_text) + record("ALTER TABLE reference ADD COLUMN " + "receiver_text TEXT DEFAULT ''"); + if (!has_receiver_type) + record("ALTER TABLE reference ADD COLUMN " + "receiver_type TEXT DEFAULT ''"); + if (!has_import_alias) + record("ALTER TABLE reference ADD COLUMN " + "import_alias TEXT DEFAULT ''"); + if (!has_call_site_file) + record("ALTER TABLE reference ADD COLUMN " + "call_site_file TEXT DEFAULT ''"); + } + } + + // Step 6 (plan §6.1): migrate the `relation` table with + // provenance columns. SQLite has no ADD COLUMN IF NOT EXISTS, + // so probe table_info first. Each new column is nullable with + // a default so pre-existing rows and non-call relations are + // not affected. + { + sqlite3_stmt *probe = nullptr; + if (probeStmt(&probe, "PRAGMA table_info(relation)", + "relation")) { + bool has_confidence = false; + bool has_resolver = false; + bool has_res_kind = false; + bool has_reason = false; + bool has_csf = false; + bool has_csr = false; + bool has_csc = false; + while (sqlite3_step(probe) == SQLITE_ROW) { + const char *col = + reinterpret_cast( + sqlite3_column_text( + probe, 1)); + if (!col) + continue; + std::string c = col; + if (c == "confidence") + has_confidence = true; + else if (c == "resolver") + has_resolver = true; + else if (c == "resolution_kind") + has_res_kind = true; + else if (c == "reason") + has_reason = true; + else if (c == "call_site_file") + has_csf = true; + else if (c == "call_site_row") + has_csr = true; + else if (c == "call_site_col") + has_csc = true; + } + sqlite3_finalize(probe); + if (!has_confidence) + record("ALTER TABLE relation ADD COLUMN " + "confidence REAL DEFAULT 0.0"); + if (!has_resolver) + record("ALTER TABLE relation ADD COLUMN " + "resolver TEXT DEFAULT ''"); + if (!has_res_kind) + record("ALTER TABLE relation ADD COLUMN " + "resolution_kind TEXT DEFAULT ''"); + if (!has_reason) + record("ALTER TABLE relation ADD COLUMN " + "reason TEXT DEFAULT ''"); + if (!has_csf) + record("ALTER TABLE relation ADD COLUMN " + "call_site_file TEXT DEFAULT ''"); + if (!has_csr) + record("ALTER TABLE relation ADD COLUMN " + "call_site_row INTEGER DEFAULT 0"); + if (!has_csc) + record("ALTER TABLE relation ADD COLUMN " + "call_site_col INTEGER DEFAULT 0"); + } + } + + // Create type_info table if missing + sqlite3_stmt *probe2 = nullptr; + if (probeStmt(&probe2, + "SELECT name FROM sqlite_master " + "WHERE type='table' AND name='type_info'", + "type_info")) { + if (sqlite3_step(probe2) != SQLITE_ROW) { + sqlite3_finalize(probe2); + run("CREATE TABLE IF NOT EXISTS type_info (" + " id INTEGER PRIMARY KEY AUTOINCREMENT," + " project_id INTEGER NOT NULL," + " name TEXT NOT NULL," + " qualified_name TEXT DEFAULT ''," + " kind INTEGER NOT NULL," + " file_path TEXT NOT NULL," + " language TEXT DEFAULT ''," + " start_row INTEGER DEFAULT 0," + " start_col INTEGER DEFAULT 0," + " end_row INTEGER DEFAULT 0," + " end_col INTEGER DEFAULT 0," + " FOREIGN KEY (project_id) REFERENCES projects(id)" + ")"); + run("CREATE INDEX IF NOT EXISTS idx_ti_name " + "ON type_info(project_id, name)"); + run("CREATE INDEX IF NOT EXISTS idx_ti_qn " + "ON type_info(project_id, qualified_name)"); + } else { + // Finalize probe2, NOT the outer `probe` from the + // semantic_records migration above: that statement + // was already finalized and would be a + // use-after-free here (and probe2 would leak). + sqlite3_finalize(probe2); + } + } + } + { + sqlite3_stmt *probe = nullptr; + if (probeStmt(&probe, + "SELECT name FROM sqlite_master " + "WHERE type='table' AND name='type_ref'", + "type_ref")) { + if (sqlite3_step(probe) != SQLITE_ROW) { + sqlite3_finalize(probe); + run("CREATE TABLE IF NOT EXISTS type_ref (" + " id INTEGER PRIMARY KEY AUTOINCREMENT," + " project_id INTEGER NOT NULL," + " entity_id INTEGER NOT NULL," + " type_name TEXT NOT NULL," + " kind INTEGER NOT NULL," + " file_path TEXT NOT NULL," + " start_row INTEGER DEFAULT 0," + " start_col INTEGER DEFAULT 0," + " FOREIGN KEY (project_id) REFERENCES projects(id)," + " FOREIGN KEY (entity_id) REFERENCES entity(id)" + ")"); + run("CREATE INDEX IF NOT EXISTS idx_tr_type " + "ON type_ref(project_id, type_name)"); + run("CREATE INDEX IF NOT EXISTS idx_tr_entity " + "ON type_ref(project_id, entity_id)"); + } else { + sqlite3_finalize(probe); + } + } + } + + return ok; +} + +} // namespace store diff --git a/engine/src/store/store_search.cpp b/engine/src/store/store_search.cpp index ab9192f..bce5689 100644 --- a/engine/src/store/store_search.cpp +++ b/engine/src/store/store_search.cpp @@ -1,3 +1,4 @@ +#include "util/json_writer.h" #include "store.h" #include "store_internal.h" #include "platform_win.h" @@ -19,103 +20,12 @@ #include #include +#include "store_search_words.h" #include "../graph/graph_builder.h" #include "../ir/semantic_unit.h" namespace store { - -// ─── FTS5 Full-Text Search ───────────────────────────────────── - -// Split a camelCase / snake_case / kebab-case identifier into its -// constituent words so an FTS5 query can match both styles: -// "findByLastName" -> find by last name -// "find_by_last_name" -> find by last name -// The unicode61 tokenizer treats underscore and case boundaries as part -// of a single token, so a bare `"findByLastName"` MATCH never hits -// snake_case code and vice versa. Splitting at lower->upper boundaries -// and at '_'/'-' yields the shared words. -static std::vector splitIdentifierWords(const std::string &word) -{ - std::vector out; - std::string cur; - auto flush = [&]() { - if (!cur.empty()) { - out.push_back(cur); - cur.clear(); - } - }; - for (size_t i = 0; i < word.size(); ++i) { - char c = word[i]; - if (c == '_' || c == '-' || c == '.' || c == '/' || c == ':' || - c == '(' || c == ')') { - flush(); - continue; - } - if (std::isupper(static_cast(c)) && - !cur.empty()) { - // Lower -> upper boundary (camelCase: "lastName"). - // Keep an acronym run together ("JSONParser" stays one - // split unless the next char is lower). - char prev = cur.back(); - if (!std::isupper(static_cast(prev)) || - (i + 1 < word.size() && - std::islower(static_cast( - word[i + 1])))) { - flush(); - } - } - cur += static_cast( - std::tolower(static_cast(c))); - } - flush(); - return out; -} - -void GraphStore::insertIntoFTS(uint64_t node_id, uint64_t project_id, - const char *name, const char *qualified_name, - const char *file_path, const char *content, - int node_kind) -{ - // Skip empty entries - if ((!name || !*name) && (!qualified_name || !*qualified_name) && - (!file_path || !*file_path) && (!content || !*content)) { - return; - } - if (node_kind < 0) - node_kind = 0; - - // Update mapping table (reuses cached prepared statement) - if (stmt_fts_map_) { - sqlite3_reset(stmt_fts_map_); - sqlite3_bind_int64(stmt_fts_map_, 1, - static_cast(node_id)); - sqlite3_bind_int64(stmt_fts_map_, 2, - static_cast(project_id)); - sqlite3_step(stmt_fts_map_); - } - - // Insert into FTS5 (reuses cached prepared statement) - if (stmt_fts_) { - sqlite3_reset(stmt_fts_); - sqlite3_bind_int64(stmt_fts_, 1, static_cast(node_id)); - sqlite3_bind_text(stmt_fts_, 2, name ? name : "", -1, - SQLITE_TRANSIENT); - sqlite3_bind_text(stmt_fts_, 3, - qualified_name ? qualified_name : "", -1, - SQLITE_TRANSIENT); - sqlite3_bind_text(stmt_fts_, 4, file_path ? file_path : "", -1, - SQLITE_TRANSIENT); - sqlite3_bind_text(stmt_fts_, 5, content ? content : "", -1, - SQLITE_TRANSIENT); - sqlite3_bind_int64(stmt_fts_, 6, - static_cast(project_id)); - sqlite3_bind_int64(stmt_fts_, 7, static_cast(node_id)); - sqlite3_bind_int(stmt_fts_, 8, node_kind); - sqlite3_step(stmt_fts_); - } -} - // deleteFTSByFile removed — FTS is indexed inline during buildGraph. // Single-file index paths no longer write FTS entries per-node. @@ -124,40 +34,88 @@ void GraphStore::buildFTSFromGraph(uint64_t project_id) // Bulk-build FTS from entity: single SQL INSERT-SELECT // No per-node prepare/finalize overhead. // graph_nodes is deprecated; entity is the canonical source. - exec(std::string( - "INSERT OR IGNORE INTO code_fts (rowid, name, qualified_name, " - " file_path, content, project_id, node_id, node_kind) " - "SELECT e.id, e.name, e.qualified_name, e.file_path, '', " + - std::to_string(project_id) + - ", e.id, e.kind " - "FROM entity e " - "WHERE e.project_id=" + - std::to_string(project_id) + " AND e.name != ''") - .c_str()); + // exec() sets error_ on failure but never clears it on success, so a + // stale error_ from an earlier operation would make the callers' + // `error().empty()` check report an FTS failure even when every INSERT + // below succeeded. Clear it up front so error() reflects THIS call. + error_.clear(); + // Rebuild is delete-first: buildFTSFromGraph repopulates the WHOLE + // project's FTS from the current entity table, but the FTS tables are + // never otherwise cleaned (deleteGraphDataByFile does not touch them). + // The inserts below key each row on rowid = entity.id, so without a + // delete they collide with stale rows via INSERT OR IGNORE: after any + // edit that re-uses an entity id, the old symbol stays in code_fts (a + // ghost that searchUnifiedJson returns because its FTS branch does not + // JOIN entity) AND the new symbol is silently dropped (its rowid is + // already taken) — so `search` returns deleted names and misses renamed + // ones. Clearing this project's FTS rows first makes the rebuild + // idempotent and collision-free. [module=store, method=buildFTSFromGraph] + { + const std::string pid = std::to_string(project_id); + const std::string dels[] = { + "DELETE FROM code_fts WHERE project_id=" + pid, + "DELETE FROM name_trgm WHERE project_id=" + pid, + "DELETE FROM fts_node_map WHERE project_id=" + pid, + }; + for (const auto &del : dels) { + if (!exec(del.c_str())) { + error_ = + "[module=store, method=buildFTSFromGraph] " + "FTS delete-first failed: " + + error_; + return; + } + } + } + // exec() sets error_ on failure; callers must not set fts_ready + // or report ok when error_ is non-empty after this call. + if (!exec(std::string( + "INSERT OR IGNORE INTO code_fts (rowid, name, qualified_name, " + " file_path, content, project_id, node_id, node_kind) " + "SELECT e.id, e.name, e.qualified_name, e.file_path, '', " + + std::to_string(project_id) + + ", e.id, e.kind " + "FROM entity e " + "WHERE e.project_id=" + + std::to_string(project_id) + " AND e.name != ''") + .c_str())) { + error_ = "[module=store, method=buildFTSFromGraph] code_fts: " + + error_; + return; + } // Build fts_node_map mapping - exec(std::string( - "INSERT OR IGNORE INTO fts_node_map (node_id, project_id, file_id) " - "SELECT e.id, e.project_id, COALESCE(f.id, 0) " - "FROM entity e " - "LEFT JOIN files f ON f.path = e.file_path AND f.project_id=e.project_id " - "WHERE e.project_id=" + - std::to_string(project_id)) - .c_str()); + if (!exec(std::string( + "INSERT OR IGNORE INTO fts_node_map (node_id, project_id, file_id) " + "SELECT e.id, e.project_id, COALESCE(f.id, 0) " + "FROM entity e " + "LEFT JOIN files f ON f.path = e.file_path AND f.project_id=e.project_id " + "WHERE e.project_id=" + + std::to_string(project_id)) + .c_str())) { + error_ = + "[module=store, method=buildFTSFromGraph] fts_node_map: " + + error_; + return; + } // Bulk-build the trigram FTS5 index (name_trgm) in parallel with // code_fts. Same source (entity), same WHERE filter. Uses // INSERT OR IGNORE so re-runs after partial indexing are idempotent. // The trigram index powers O(log n) substring search via MATCH, // replacing the O(n) LIKE '%query%' scan in searchGraphFallback. - exec(std::string( - "INSERT OR IGNORE INTO name_trgm " - "(rowid, name, qualified_name, project_id, node_id, node_type) " - "SELECT e.id, e.name, e.qualified_name, " + - std::to_string(project_id) + - ", e.id, e.kind " - "FROM entity e " - "WHERE e.project_id=" + - std::to_string(project_id) + " AND e.name != ''") - .c_str()); + if (!exec(std::string( + "INSERT OR IGNORE INTO name_trgm " + "(rowid, name, qualified_name, project_id, node_id, node_type) " + "SELECT e.id, e.name, e.qualified_name, " + + std::to_string(project_id) + + ", e.id, e.kind " + "FROM entity e " + "WHERE e.project_id=" + + std::to_string(project_id) + " AND e.name != ''") + .c_str())) { + error_ = + "[module=store, method=buildFTSFromGraph] name_trgm: " + + error_; + } } bool GraphStore::isTrigramAvailable() @@ -182,448 +140,6 @@ bool GraphStore::isTrigramAvailable() return available; } -namespace -{ -// Fixed dimension of the n-gram hash vector. 192 floats = 768 bytes per -// row, small enough for a BLOB column and for a full in-memory scan of a -// large module. Larger dimensions hurt cosine separation at this feature -// scale; smaller ones increase collision noise. -constexpr int kVecDim = 192; -// Cosine-similarity floor for semantic search results (see the accuracy-first -// gate in searchSemanticJson). Strong n-gram matches score > 0.6; unrelated -// names cluster below 0.23, so 0.3 cleanly separates signal from noise. -constexpr float kSemanticScoreFloor = 0.3f; - -// Double-hash the n-gram into two buckets and accumulate signed weights, so -// the resulting vector is a standard hashing-vectorizer (like -// sklearn HashingVectorizer). No external model is involved — this is the -// n-gram hash scheme the schema comment for node_vectors always intended. -// `seed` decorrelates the two hash passes. -static inline uint64_t hashMix(uint64_t h) -{ - h ^= h >> 30; - h *= 0xbf58476d1ce4e5b9ULL; - h ^= h >> 27; - h *= 0x94d049bb133111ebULL; - h ^= h >> 31; - return h; -} -} // namespace - -// Build n-gram hash vectors for every function/method entity of the project -// and store them in node_vectors. This restores the semantic-search producer -// that Step 10 sunset (the previous body was a no-op leaving node_vectors -// empty). Readiness is derived from the actual node_vectors row count, so the -// A19 "fake ready" regression cannot recur: if this loop writes rows, -// embedding_ready reflects it; if it writes nothing, readiness stays 0. -// -// The vector is built from the entity's qualified_name + name n-grams. This -// gives lexical-similarity search (find "user_dao" given "user_repository") -// which is the practical "semantic" signal available without an embedding -// model. It is NOT a meaning vector; the tool description and capabilities -// JSON say so explicitly. -void GraphStore::buildVectorsFromGraph(uint64_t project_id) -{ - if (!db_) - return; - - // Collect (id, qualified_name, name) for function/method entities. - // entity.id is the canonical primary key (it preserves the legacy graph - // node identity after the graph_nodes→entity migration). - struct Ent { - int64_t id; - std::string text; - }; - std::vector ents; - { - const char *sql = - "SELECT id, COALESCE(NULLIF(qualified_name, ''), name), " - " name FROM entity " - "WHERE project_id = ? AND kind IN (0,1)"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != - SQLITE_OK) { - fprintf(stderr, - "buildVectorsFromGraph: prepare collect failed: %s " - "[module=store, method=buildVectorsFromGraph]\n", - sqlite3_errmsg(db_)); - return; - } - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - while (sqlite3_step(stmt) == SQLITE_ROW) { - Ent e; - e.id = sqlite3_column_int64(stmt, 0); - const char *qn = reinterpret_cast( - sqlite3_column_text(stmt, 1)); - const char *nm = reinterpret_cast( - sqlite3_column_text(stmt, 2)); - std::string text = qn ? qn : ""; - if (nm && *nm) { - if (!text.empty()) - text.push_back(' '); - text += nm; - } - e.text = std::move(text); - ents.push_back(std::move(e)); - } - sqlite3_finalize(stmt); - } - if (ents.empty()) - return; - - // ── TF-IDF identifier weighting (v0.2.5) ────────────────────── - // Tokenize every entity's qualified_name + name via camel/snake/kebab - // splitting and count per-entity token frequencies, so we can weight - // each token's vector contribution by inverse document frequency (idf = - // log(1 + N/(1+df))). Rare, discriminative tokens (e.g. "ledger" in - // getUserByLedgerId) then contribute far more to the vector than common - // ones ("get"), which sharply improves semantic-search precision: a - // query matching a rare token ranks the correct entity far above - // incidental trigram-overlap noise. The same split is applied on the - // query side, so no project statistics are needed at query time. - struct TokMap { - std::vector toks; - std::vector idfs; - }; - std::vector ent_toks(ents.size()); - { - std::unordered_map df; - df.reserve(ents.size() * 4); - for (const auto &e : ents) { - auto toks = splitIdentifierWords(e.text); - // Dedupe within this entity (df counts entities, not - // occurrences). - std::sort(toks.begin(), toks.end()); - toks.erase(std::unique(toks.begin(), toks.end()), - toks.end()); - for (auto &t : toks) { - if (t.empty()) - continue; - ++df[t]; - } - ent_toks[&e - ents.data()].toks = std::move(toks); - } - const size_t N = ents.size(); - for (size_t ei = 0; ei < ents.size(); ++ei) { - auto &tm = ent_toks[ei]; - tm.idfs.reserve(tm.toks.size()); - for (const auto &t : tm.toks) { - size_t d = 0; - auto it = df.find(t); - if (it != df.end()) - d = it->second; - float idf = static_cast(std::log( - 1.0 + - static_cast(N) / - (1.0 + static_cast(d)))); - // Cap the weight so one dominant token cannot - // overwhelm the trigram signal entirely. - if (idf > 3.0f) - idf = 3.0f; - tm.idfs.push_back(idf); - } - } - } - - // Clear stale vectors for this project, then write fresh ones. - const char *clear_sql = "DELETE FROM node_vectors WHERE project_id = ?"; - sqlite3_stmt *del = nullptr; - if (sqlite3_prepare_v2(db_, clear_sql, -1, &del, nullptr) == - SQLITE_OK) { - sqlite3_bind_int64(del, 1, static_cast(project_id)); - sqlite3_step(del); - sqlite3_finalize(del); - } else { - fprintf(stderr, - "buildVectorsFromGraph: prepare clear failed: %s " - "[module=store, method=buildVectorsFromGraph]\n", - sqlite3_errmsg(db_)); - return; - } - - const char *ins_sql = - "INSERT OR REPLACE INTO node_vectors (node_id, project_id, vector) " - "VALUES (?,?,?)"; - sqlite3_stmt *ins = nullptr; - if (sqlite3_prepare_v2(db_, ins_sql, -1, &ins, nullptr) != SQLITE_OK) { - fprintf(stderr, - "buildVectorsFromGraph: prepare insert failed: %s " - "[module=store, method=buildVectorsFromGraph]\n", - sqlite3_errmsg(db_)); - return; - } - - // v0.2.5 (perf fix): wrap the whole batch of INSERTs in a single - // transaction. In autocommit mode every row INSERT issues its own - // fsync/commit, which made vector build take tens of seconds on large - // projects (thousands of function entities). One BEGIN/COMMIT collapses - // all writes into a single commit — order-of-magnitude faster, and the - // table is our own scratch (vector_ready is derived from the row count, - // so partial/rolled-back writes still yield correct readiness). - exec("BEGIN IMMEDIATE TRANSACTION"); - - std::vector vec(kVecDim, 0.0f); - for (const auto &e : ents) { - std::fill(vec.begin(), vec.end(), 0.0f); - const std::string &t = e.text; - // Character trigrams (lowercased) — captures identifier substrings - // and cross-casing boundaries ("userDao" → "use","ser","erD",...). - // Kept unweighted as a lexical-similarity fallback so a query - // that only partially overlaps an identifier (or crosses a casing - // boundary) still gets a signal. - for (size_t i = 0; i + 3 <= t.size(); ++i) { - std::string gram = t.substr(i, 3); - for (char &ch : gram) - ch = static_cast(std::tolower( - static_cast(ch))); - uint64_t h = hashMix(std::hash{}(gram) ^ - static_cast(project_id)); - int b1 = static_cast(h % kVecDim); - int b2 = static_cast(hashMix(h) % kVecDim); - vec[b1] += (h & 1) ? 1.0f : -1.0f; - vec[b2] += (h & 2) ? 1.0f : -1.0f; - } - // TF-IDF weighted camel/snake token contribution (v0.2.5). Each - // split token (get/user/by/id ...) is hashed and accumulated with - // magnitude proportional to its idf — rare discriminative tokens - // dominate, so semantically distinctive names rank correctly. - { - const TokMap &tm = ent_toks[&e - ents.data()]; - for (size_t ti = 0; ti < tm.toks.size(); ++ti) { - const std::string &tok = tm.toks[ti]; - if (tok.empty()) - continue; - uint64_t h = hashMix( - std::hash{}(tok) ^ - static_cast(project_id)); - int b1 = static_cast(h % kVecDim); - int b2 = static_cast(hashMix(h) % kVecDim); - const float w = tm.idfs[ti]; - vec[b1] += (h & 1) ? w : -w; - vec[b2] += (h & 2) ? w : -w; - } - } - // L2-normalize. - double norm = 0.0; - for (float v : vec) - norm += static_cast(v) * v; - if (norm > 0.0) { - const float inv = - static_cast(1.0 / std::sqrt(norm)); - for (float &v : vec) - v *= inv; - } - // Serialize as raw float32 little-endian. - std::vector blob(kVecDim * sizeof(float)); - for (int d = 0; d < kVecDim; ++d) { - uint32_t bits; - memcpy(&bits, &vec[d], sizeof(bits)); - for (int b = 0; b < 4; ++b) - blob[d * 4 + b] = static_cast( - (bits >> (8 * b)) & 0xFF); - } - sqlite3_bind_int64(ins, 1, e.id); - sqlite3_bind_int64(ins, 2, static_cast(project_id)); - sqlite3_bind_blob(ins, 3, blob.data(), - static_cast(blob.size()), - SQLITE_TRANSIENT); - if (sqlite3_step(ins) != SQLITE_DONE) { - fprintf(stderr, - "buildVectorsFromGraph: insert step failed: %s " - "[module=store, method=buildVectorsFromGraph]\n", - sqlite3_errmsg(db_)); - } - sqlite3_reset(ins); - } - sqlite3_finalize(ins); - // Commit the batch (see the BEGIN above). exec() logs on failure. - exec("COMMIT"); -} - -std::string GraphStore::searchSemanticJson(uint64_t project_id, - const char *query, int limit) -{ - static constexpr const char *kMethod = "searchSemanticJson"; - if (!db_ || !query || !*query) - return "{\"method\":\"semantic\",\"results\":[]}"; - if (limit <= 0 || limit > 100) - limit = 20; - - // Vectorize the query to mirror buildVectorsFromGraph: lowercased - // character trigrams + camel/snake-split identifier tokens, double-hash - // accumulated, L2-normalized. Cosine similarity over the stored - // normalized vectors is then a dot product. - // - // v0.2.5: the query is also split into identifier tokens (matching the - // TF-IDF-weighted token contribution the builder wrote). Query tokens are - // hashed at equal magnitude — the builder already baked each token's idf - // weight into the stored entity vectors, so an equal-weight query token - // automatically scores higher against the entity that shares that token - // at high weight (i.e. the rare, discriminative one). - std::vector qvec(kVecDim, 0.0f); - { - const std::string t = query; - for (size_t i = 0; i + 3 <= t.size(); ++i) { - std::string gram = t.substr(i, 3); - for (char &ch : gram) - ch = static_cast(std::tolower( - static_cast(ch))); - uint64_t h = hashMix(std::hash{}(gram) ^ - static_cast(project_id)); - int b1 = static_cast(h % kVecDim); - int b2 = static_cast(hashMix(h) % kVecDim); - qvec[b1] += (h & 1) ? 1.0f : -1.0f; - qvec[b2] += (h & 2) ? 1.0f : -1.0f; - } - // Identifier-token contributions (equal weight; idf lives in the - // stored entity vectors). - for (const std::string &tok : splitIdentifierWords(t)) { - if (tok.empty()) - continue; - uint64_t h = hashMix(std::hash{}(tok) ^ - static_cast(project_id)); - int b1 = static_cast(h % kVecDim); - int b2 = static_cast(hashMix(h) % kVecDim); - qvec[b1] += (h & 1) ? 1.0f : -1.0f; - qvec[b2] += (h & 2) ? 1.0f : -1.0f; - } - double norm = 0.0; - for (float v : qvec) - norm += static_cast(v) * v; - if (norm > 0.0) { - const float inv = - static_cast(1.0 / std::sqrt(norm)); - for (float &v : qvec) - v *= inv; - } - } - - // Early-exit when no vectors exist for this project: nothing to match, - // report empty with a reason so callers can fall back to FTS. - { - sqlite3_stmt *chk = nullptr; - const char *csql = - "SELECT COUNT(*) FROM node_vectors WHERE project_id = ?"; - if (sqlite3_prepare_v2(db_, csql, -1, &chk, nullptr) != - SQLITE_OK) { - fprintf(stderr, - "[module=store, method=%s] count prepare failed: " - "%s\n", - kMethod, sqlite3_errmsg(db_)); - return "{\"method\":\"semantic\",\"results\":[]}"; - } - sqlite3_bind_int64(chk, 1, static_cast(project_id)); - bool has_rows = false; - if (sqlite3_step(chk) == SQLITE_ROW && - sqlite3_column_int64(chk, 0) > 0) - has_rows = true; - sqlite3_finalize(chk); - if (!has_rows) - return "{\"method\":\"semantic\",\"results\":[]," - "\"reason\":\"embedding_not_built\"}"; - } - - // Full scan of node_vectors joined to entity, computing cosine. - struct Hit { - int64_t node_id; - std::string name; - std::string qualified_name; - std::string file_path; - float score; - }; - std::vector hits; - { - const char *sql = - "SELECT v.node_id, v.vector, " - " COALESCE(NULLIF(e.qualified_name,''),e.name), " - " e.name, e.file_path " - "FROM node_vectors v JOIN entity e ON e.id = v.node_id " - "WHERE v.project_id = ?"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != - SQLITE_OK) { - fprintf(stderr, - "[module=store, method=%s] scan prepare failed: %s\n", - kMethod, sqlite3_errmsg(db_)); - return "{\"method\":\"semantic\",\"results\":[]," - "\"error\":\"scan_failed\"}"; - } - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - while (sqlite3_step(stmt) == SQLITE_ROW) { - int64_t nid = sqlite3_column_int64(stmt, 0); - const void *blob = sqlite3_column_blob(stmt, 1); - int nbytes = sqlite3_column_bytes(stmt, 1); - const char *qn = reinterpret_cast( - sqlite3_column_text(stmt, 2)); - const char *nm = reinterpret_cast( - sqlite3_column_text(stmt, 3)); - const char *fp = reinterpret_cast( - sqlite3_column_text(stmt, 4)); - if (!blob || - nbytes < static_cast(kVecDim * sizeof(float))) - continue; - // Deserialize float32 little-endian and dot with qvec. - float dot = 0.0f; - for (int d = 0; d < kVecDim; ++d) { - uint32_t bits = 0; - const uint8_t *b = - static_cast(blob) + - d * 4; - bits |= static_cast(b[0]); - bits |= static_cast(b[1]) << 8; - bits |= static_cast(b[2]) << 16; - bits |= static_cast(b[3]) << 24; - float f; - memcpy(&f, &bits, sizeof(f)); - dot += f * qvec[d]; - } - // Accuracy-first gate (0LLM design): only strong matches are - // reported. Empirically the true positive for an n-gram hash - // vector (a name sharing the query's trigrams) scores > 0.6, - // while unrelated names cluster in the 0.02–0.23 band. A - // 0.3 floor therefore keeps every relevant hit while - // rejecting the noise, so semantic search never pollutes - // results with weak/incidental matches (the accuracy - // fixtures depend on exact FTS/trigram and must stay clean). - if (dot <= kSemanticScoreFloor) - continue; - Hit hit; - hit.node_id = nid; - hit.name = nm ? nm : ""; - hit.qualified_name = qn ? qn : ""; - hit.file_path = fp ? fp : ""; - hit.score = dot; - hits.push_back(std::move(hit)); - } - sqlite3_finalize(stmt); - } - - // Rank by descending similarity, cap at limit. - std::partial_sort(hits.begin(), - hits.begin() + std::min(static_cast(limit), - hits.size()), - hits.end(), [](const Hit &a, const Hit &b) { - return a.score > b.score; - }); - if (hits.size() > static_cast(limit)) - hits.resize(static_cast(limit)); - - std::ostringstream json; - json << "{\"method\":\"semantic\",\"total\":" << hits.size() - << ",\"results\":["; - for (size_t i = 0; i < hits.size(); ++i) { - if (i > 0) - json << ","; - json << "{\"node_id\":" << hits[i].node_id << ",\"name\":\"" - << jsonEscape(hits[i].name) << "\",\"qualified_name\":\"" - << jsonEscape(hits[i].qualified_name) - << "\",\"file_path\":\"" << jsonEscape(hits[i].file_path) - << "\",\"score\":" << hits[i].score << "}"; - } - json << "]}"; - return json.str(); -} - std::string GraphStore::searchCode(uint64_t project_id, const char *query, int limit) { @@ -836,19 +352,20 @@ std::string GraphStore::searchCode(uint64_t project_id, const char *query, for (size_t i = 0; i < results.size(); i++) { if (i > 0) json << ","; - json << "{" - << "\"node_id\":" << results[i].node_id << "," - << "\"name\":\"" << jsonEscape(results[i].name) << "\"," - << "\"node_type\":" << results[i].node_type << "," - << "\"file_path\":\"" << jsonEscape(results[i].file_path) - << "\"," - << "\"start_row\":" << results[i].start_row << "," - << "\"start_col\":" << results[i].start_col << "," - << "\"end_row\":" << results[i].end_row << "," - << "\"end_col\":" << results[i].end_col << "," - << "\"language\":\"" << jsonEscape(results[i].language) - << "\"," - << "\"score\":" << results[i].score << "}"; + util::JsonWriter el; + el.beginObject(); + el.key("node_id").value(results[i].node_id); + el.key("name").value(results[i].name); + el.key("node_type").value(results[i].node_type); + el.key("file_path").value(results[i].file_path); + el.key("start_row").value(results[i].start_row); + el.key("start_col").value(results[i].start_col); + el.key("end_row").value(results[i].end_row); + el.key("end_col").value(results[i].end_col); + el.key("language").value(results[i].language); + el.key("score").value(results[i].score); + el.endObject(); + json << el.str(); } json << "],\"total\":" << results.size() << "}"; return json.str(); @@ -916,17 +433,16 @@ std::string GraphStore::searchGraphFallback(uint64_t project_id, const char *fp_raw = reinterpret_cast( sqlite3_column_text(stmt, 2)); - json << "{" - << "\"node_id\":" - << sqlite3_column_int64(stmt, 0) << "," - << "\"name\":\"" - << jsonEscape(name_raw ? name_raw : "") - << "\"," - << "\"file_path\":\"" - << jsonEscape(fp_raw ? fp_raw : "") - << "\"," - << "\"type\":" - << sqlite3_column_int(stmt, 3) << "}"; + util::JsonWriter el; + el.beginObject(); + el.key("node_id").value( + sqlite3_column_int64(stmt, 0)); + el.key("name").value(name_raw ? name_raw : ""); + el.key("file_path").value(fp_raw ? fp_raw : ""); + el.key("type").value( + sqlite3_column_int(stmt, 3)); + el.endObject(); + json << el.str(); } sqlite3_finalize(stmt); if (rc == SQLITE_INTERRUPT) { @@ -976,15 +492,14 @@ std::string GraphStore::searchGraphFallback(uint64_t project_id, sqlite3_column_text(stmt, 1)); const char *fp_raw = reinterpret_cast( sqlite3_column_text(stmt, 2)); - json << "{" - << "\"node_id\":" << sqlite3_column_int64(stmt, 0) - << "," - << "\"name\":\"" - << jsonEscape(name_raw ? name_raw : "") << "\"," - << "\"file_path\":\"" - << jsonEscape(fp_raw ? fp_raw : "") << "\"," - << "\"type\":" << sqlite3_column_int(stmt, 3) - << "}"; + util::JsonWriter el; + el.beginObject(); + el.key("node_id").value(sqlite3_column_int64(stmt, 0)); + el.key("name").value(name_raw ? name_raw : ""); + el.key("file_path").value(fp_raw ? fp_raw : ""); + el.key("type").value(sqlite3_column_int(stmt, 3)); + el.endObject(); + json << el.str(); } sqlite3_finalize(stmt); } else { @@ -1005,34 +520,6 @@ std::string GraphStore::searchGraphFallback(uint64_t project_id, json << "}"; return json.str(); } - -// ─── Complexity ─────────────────────────────────────────────── -// -// v0.2.5: metrics are restored. The canonical write path is the parse worker -// (engine_index_metrics.cpp) → `_staged_metrics` (insertFileResultBatch) → -// `resolveStagedMetrics()`, which resolves the staged values onto the -// canonical `entity` columns. The read API (getComplexityJson) returns the -// real measurements from `entity`. `setComplexity` below is a retained -// compatibility seam with no callers; it is intentionally inert so a stray -// caller cannot bypass the canonical staged-metrics pipeline and write -// metrics that were never computed. - -bool GraphStore::setComplexity(uint64_t project_id, uint64_t graph_node_id, - uint64_t cyclomatic, uint64_t cognitive, - uint64_t nesting_depth, uint64_t decision_points) -{ - // Inert compatibility seam: canonical metrics flow through - // _staged_metrics → resolveStagedMetrics. Returns false so a future - // caller can detect the write did not go through the canonical path. - (void)project_id; - (void)graph_node_id; - (void)cyclomatic; - (void)cognitive; - (void)nesting_depth; - (void)decision_points; - return false; -} - // Return the per-function code metrics for a single graph node, sourced from // the canonical entity row (the Knowledge Graph single source of truth). // Metrics are populated during indexing (staged in _staged_metrics, resolved @@ -1084,18 +571,22 @@ std::string GraphStore::getComplexityJson(uint64_t project_id, int calls = sqlite3_column_int(stmt, 8); int lines = sqlite3_column_int(stmt, 9); int is_stub = sqlite3_column_int(stmt, 10); - std::ostringstream j; - j << "{\"name\":\"" << jsonEscape(name ? name : "") - << "\",\"file_path\":\"" << jsonEscape(file ? file : "") - << "\",\"cyclomatic\":" << cyclomatic - << ",\"cognitive\":" << cognitive - << ",\"nesting_depth\":" << nesting - << ",\"branch_count\":" << branches - << ",\"loop_count\":" << loops - << ",\"param_count\":" << params - << ",\"call_count\":" << calls << ",\"lines\":" << lines - << ",\"is_stub\":" << (is_stub ? "true" : "false") - << ",\"complexity\":" << cyclomatic << ",\"available\":true}"; + util::JsonWriter j; + j.beginObject(); + j.key("name").value(name ? name : ""); + j.key("file_path").value(file ? file : ""); + j.key("cyclomatic").value(cyclomatic); + j.key("cognitive").value(cognitive); + j.key("nesting_depth").value(nesting); + j.key("branch_count").value(branches); + j.key("loop_count").value(loops); + j.key("param_count").value(params); + j.key("call_count").value(calls); + j.key("lines").value(lines); + j.key("is_stub").value(is_stub != 0); + j.key("complexity").value(cyclomatic); + j.key("available").value(true); + j.endObject(); sqlite3_finalize(stmt); return j.str(); } @@ -1115,40 +606,6 @@ std::string GraphStore::getComplexityJson(uint64_t project_id, // ─── Vector Search (removed) ────────────────────────────────── -bool // storeVector removed — Phase 0 cut -GraphStore::storeVector(uint64_t node_id, uint64_t project_id, - const void *vec_data, size_t vec_bytes) -{ - if (!stmt_vector_) { - error_ = "storeVector: statement not prepared"; - return false; - } - sqlite3_reset(stmt_vector_); - sqlite3_bind_int64(stmt_vector_, 1, static_cast(node_id)); - sqlite3_bind_int64(stmt_vector_, 2, static_cast(project_id)); - // sqlite3_bind_blob takes an int length; reject vectors that would - // overflow it rather than silently truncating the size_t value. - if (vec_bytes > static_cast(INT_MAX)) { - error_ = "storeVector: vector too large for blob binding"; - return false; - } - sqlite3_bind_blob(stmt_vector_, 3, vec_data, - static_cast(vec_bytes), SQLITE_TRANSIENT); - int rc = sqlite3_step(stmt_vector_); - return rc == SQLITE_DONE; -} - -std::string // searchSemantic removed — Phase 0 cut -GraphStore::searchSemantic(uint64_t project_id, const void *query_vec, - size_t vec_bytes, int limit) -{ - (void)project_id; - (void)query_vec; - (void)vec_bytes; - (void)limit; - return "{\"total\":0,\"results\":[]}"; -} - // ── New Schema (Phase A): Modules ───────────────────────────── } // namespace store diff --git a/engine/src/store/store_search_vector.cpp b/engine/src/store/store_search_vector.cpp new file mode 100644 index 0000000..fef86ec --- /dev/null +++ b/engine/src/store/store_search_vector.cpp @@ -0,0 +1,528 @@ +// store_search_vector.cpp — lexical-similarity ("semantic") search. +// +// Split out of store_search.cpp (see plan/rules/code_rules.md 1000-line +// rule). This is the n-gram hashing producer/consumer pair: it is NOT a +// meaning vector, it is the lexical-similarity signal the search pipeline +// falls back on when FTS5 comes up short. The tool description and the +// capabilities JSON both say so. +// +// Vectors are built from the entity's qualified_name + name n-grams, so +// "user_dao" is findable given "user_repository" with no model loaded. + +#include "util/json_writer.h" +#include "store.h" +#include "store_internal.h" +#include "store_search_words.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace store +{ + +namespace +{ +// Fixed dimension of the n-gram hash vector. 192 floats = 768 bytes per +// row, small enough for a BLOB column and for a full in-memory scan of a +// large module. Larger dimensions hurt cosine separation at this feature +// scale; smaller ones increase collision noise. +constexpr int kVecDim = 192; +// Cosine-similarity floor for semantic search results (see the accuracy-first +// gate in searchSemanticJson). Strong n-gram matches score > 0.6; unrelated +// names cluster below 0.23, so 0.3 cleanly separates signal from noise. +constexpr float kSemanticScoreFloor = 0.3f; + +// Double-hash the n-gram into two buckets and accumulate signed weights, so +// the resulting vector is a standard hashing-vectorizer (like +// sklearn HashingVectorizer). No external model is involved — this is the +// n-gram hash scheme the schema comment for node_vectors always intended. +// `seed` decorrelates the two hash passes. +static inline uint64_t hashMix(uint64_t h) +{ + h ^= h >> 30; + h *= 0xbf58476d1ce4e5b9ULL; + h ^= h >> 27; + h *= 0x94d049bb133111ebULL; + h ^= h >> 31; + return h; +} +} // namespace + +// Build n-gram hash vectors for every function/method entity of the project +// and store them in node_vectors. This restores the semantic-search producer +// that Step 10 sunset (the previous body was a no-op leaving node_vectors +// empty). Readiness is derived from the actual node_vectors row count, so the +// A19 "fake ready" regression cannot recur: if this loop writes rows, +// embedding_ready reflects it; if it writes nothing, readiness stays 0. +// +// The vector is built from the entity's qualified_name + name n-grams. This +// gives lexical-similarity search (find "user_dao" given "user_repository") +// which is the practical "semantic" signal available without an embedding +// model. It is NOT a meaning vector; the tool description and capabilities +// JSON say so explicitly. +void GraphStore::buildVectorsFromGraph(uint64_t project_id) +{ + if (!db_) + return; + + // Collect (id, qualified_name, name) for function/method entities. + // entity.id is the canonical primary key (it preserves the legacy graph + // node identity after the graph_nodes→entity migration). + struct Ent { + int64_t id; + std::string text; + }; + std::vector ents; + { + const char *sql = + "SELECT id, COALESCE(NULLIF(qualified_name, ''), name), " + " name FROM entity " + "WHERE project_id = ? AND kind IN (0,1)"; + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != + SQLITE_OK) { + fprintf(stderr, + "buildVectorsFromGraph: prepare collect failed: %s " + "[module=store, method=buildVectorsFromGraph]\n", + sqlite3_errmsg(db_)); + return; + } + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + while (sqlite3_step(stmt) == SQLITE_ROW) { + Ent e; + e.id = sqlite3_column_int64(stmt, 0); + const char *qn = reinterpret_cast( + sqlite3_column_text(stmt, 1)); + const char *nm = reinterpret_cast( + sqlite3_column_text(stmt, 2)); + std::string text = qn ? qn : ""; + if (nm && *nm) { + if (!text.empty()) + text.push_back(' '); + text += nm; + } + e.text = std::move(text); + ents.push_back(std::move(e)); + } + sqlite3_finalize(stmt); + } + if (ents.empty()) + return; + + // ── TF-IDF identifier weighting (v0.2.5) ────────────────────── + // Tokenize every entity's qualified_name + name via camel/snake/kebab + // splitting and count per-entity token frequencies, so we can weight + // each token's vector contribution by inverse document frequency (idf = + // log(1 + N/(1+df))). Rare, discriminative tokens (e.g. "ledger" in + // getUserByLedgerId) then contribute far more to the vector than common + // ones ("get"), which sharply improves semantic-search precision: a + // query matching a rare token ranks the correct entity far above + // incidental trigram-overlap noise. The same split is applied on the + // query side, so no project statistics are needed at query time. + struct TokMap { + std::vector toks; + std::vector idfs; + }; + std::vector ent_toks(ents.size()); + { + std::unordered_map df; + df.reserve(ents.size() * 4); + for (const auto &e : ents) { + auto toks = splitIdentifierWords(e.text); + // Dedupe within this entity (df counts entities, not + // occurrences). + std::sort(toks.begin(), toks.end()); + toks.erase(std::unique(toks.begin(), toks.end()), + toks.end()); + for (auto &t : toks) { + if (t.empty()) + continue; + ++df[t]; + } + ent_toks[&e - ents.data()].toks = std::move(toks); + } + const size_t N = ents.size(); + for (size_t ei = 0; ei < ents.size(); ++ei) { + auto &tm = ent_toks[ei]; + tm.idfs.reserve(tm.toks.size()); + for (const auto &t : tm.toks) { + size_t d = 0; + auto it = df.find(t); + if (it != df.end()) + d = it->second; + float idf = static_cast(std::log( + 1.0 + + static_cast(N) / + (1.0 + static_cast(d)))); + // Cap the weight so one dominant token cannot + // overwhelm the trigram signal entirely. + if (idf > 3.0f) + idf = 3.0f; + tm.idfs.push_back(idf); + } + } + } + + // Clear stale vectors for this project, then write fresh ones. + const char *clear_sql = "DELETE FROM node_vectors WHERE project_id = ?"; + sqlite3_stmt *del = nullptr; + if (sqlite3_prepare_v2(db_, clear_sql, -1, &del, nullptr) != + SQLITE_OK) { + fprintf(stderr, + "buildVectorsFromGraph: prepare clear failed: %s " + "[module=store, method=buildVectorsFromGraph]\n", + sqlite3_errmsg(db_)); + return; + } + sqlite3_bind_int64(del, 1, static_cast(project_id)); + if (sqlite3_step(del) != SQLITE_DONE) { + fprintf(stderr, + "buildVectorsFromGraph: clear step failed: %s " + "[module=store, method=buildVectorsFromGraph]\n", + sqlite3_errmsg(db_)); + sqlite3_finalize(del); + return; + } + sqlite3_finalize(del); + + const char *ins_sql = + "INSERT OR REPLACE INTO node_vectors (node_id, project_id, vector) " + "VALUES (?,?,?)"; + sqlite3_stmt *ins = nullptr; + if (sqlite3_prepare_v2(db_, ins_sql, -1, &ins, nullptr) != SQLITE_OK) { + fprintf(stderr, + "buildVectorsFromGraph: prepare insert failed: %s " + "[module=store, method=buildVectorsFromGraph]\n", + sqlite3_errmsg(db_)); + return; + } + + // v0.2.5 (perf fix): wrap the whole batch of INSERTs in a single + // transaction. In autocommit mode every row INSERT issues its own + // fsync/commit, which made vector build take tens of seconds on large + // projects (thousands of function entities). Collapsing all writes into + // one commit is an order-of-magnitude faster, and the table is our own + // scratch (vector_ready is derived from the row count, so + // partial/rolled-back writes still yield correct readiness). + // + // SAVEPOINT rather than BEGIN: this may run inside the index + // transaction, where a plain BEGIN fails and the matching COMMIT would + // commit the caller's transaction. + if (!exec("SAVEPOINT build_vectors")) { + fprintf(stderr, + "buildVectorsFromGraph: SAVEPOINT failed: %s " + "[module=store, method=buildVectorsFromGraph]\n", + error_.c_str()); + return; + } + + // RAII: any early return below — or an allocation that throws inside + // the loop — rolls the savepoint back, so a failed vector build can + // never leave the connection sitting inside a transaction (which would + // make every later BEGIN fail with "cannot start a transaction within + // a transaction"). + struct SavepointGuard { + GraphStore *store = nullptr; + bool released = false; + ~SavepointGuard() + { + if (!released) { + store->exec( + "ROLLBACK TO SAVEPOINT build_vectors"); + store->exec("RELEASE SAVEPOINT build_vectors"); + } + } + } guard{ this, false }; + + bool ok = true; + std::vector vec(kVecDim, 0.0f); + for (const auto &e : ents) { + std::fill(vec.begin(), vec.end(), 0.0f); + const std::string &t = e.text; + // Character trigrams (lowercased) — captures identifier substrings + // and cross-casing boundaries ("userDao" → "use","ser","erD",...). + // Kept unweighted as a lexical-similarity fallback so a query + // that only partially overlaps an identifier (or crosses a casing + // boundary) still gets a signal. + for (size_t i = 0; i + 3 <= t.size(); ++i) { + std::string gram = t.substr(i, 3); + for (char &ch : gram) + ch = static_cast(std::tolower( + static_cast(ch))); + uint64_t h = hashMix(std::hash{}(gram) ^ + static_cast(project_id)); + int b1 = static_cast(h % kVecDim); + int b2 = static_cast(hashMix(h) % kVecDim); + vec[b1] += (h & 1) ? 1.0f : -1.0f; + vec[b2] += (h & 2) ? 1.0f : -1.0f; + } + // TF-IDF weighted camel/snake token contribution (v0.2.5). Each + // split token (get/user/by/id ...) is hashed and accumulated with + // magnitude proportional to its idf — rare discriminative tokens + // dominate, so semantically distinctive names rank correctly. + { + const TokMap &tm = ent_toks[&e - ents.data()]; + for (size_t ti = 0; ti < tm.toks.size(); ++ti) { + const std::string &tok = tm.toks[ti]; + if (tok.empty()) + continue; + uint64_t h = hashMix( + std::hash{}(tok) ^ + static_cast(project_id)); + int b1 = static_cast(h % kVecDim); + int b2 = static_cast(hashMix(h) % kVecDim); + const float w = tm.idfs[ti]; + vec[b1] += (h & 1) ? w : -w; + vec[b2] += (h & 2) ? w : -w; + } + } + // L2-normalize. + double norm = 0.0; + for (float v : vec) + norm += static_cast(v) * v; + if (norm > 0.0) { + const float inv = + static_cast(1.0 / std::sqrt(norm)); + for (float &v : vec) + v *= inv; + } + // Serialize as raw float32 little-endian. + std::vector blob(kVecDim * sizeof(float)); + for (int d = 0; d < kVecDim; ++d) { + uint32_t bits; + memcpy(&bits, &vec[d], sizeof(bits)); + for (int b = 0; b < 4; ++b) + blob[d * 4 + b] = static_cast( + (bits >> (8 * b)) & 0xFF); + } + sqlite3_bind_int64(ins, 1, e.id); + sqlite3_bind_int64(ins, 2, static_cast(project_id)); + sqlite3_bind_blob(ins, 3, blob.data(), + static_cast(blob.size()), + SQLITE_TRANSIENT); + if (sqlite3_step(ins) != SQLITE_DONE) { + fprintf(stderr, + "buildVectorsFromGraph: insert step failed: %s " + "[module=store, method=buildVectorsFromGraph]\n", + sqlite3_errmsg(db_)); + ok = false; + } + sqlite3_reset(ins); + } + sqlite3_finalize(ins); + + // Abort (letting the guard roll back) rather than release a savepoint + // holding a half-built vector table: semantic search would otherwise + // mix new vectors with the stale ones the DELETE was meant to remove. + if (!ok) { + fprintf(stderr, + "buildVectorsFromGraph: batch failed, rolling back " + "savepoint " + "[module=store, method=buildVectorsFromGraph]\n"); + return; + } + if (!exec("RELEASE SAVEPOINT build_vectors")) { + fprintf(stderr, + "buildVectorsFromGraph: RELEASE SAVEPOINT failed: %s " + "[module=store, method=buildVectorsFromGraph]\n", + error_.c_str()); + return; + } + guard.released = true; +} + +std::string GraphStore::searchSemanticJson(uint64_t project_id, + const char *query, int limit) +{ + static constexpr const char *kMethod = "searchSemanticJson"; + if (!db_ || !query || !*query) + return "{\"method\":\"semantic\",\"results\":[]}"; + if (limit <= 0 || limit > 100) + limit = 20; + + // Vectorize the query to mirror buildVectorsFromGraph: lowercased + // character trigrams + camel/snake-split identifier tokens, double-hash + // accumulated, L2-normalized. Cosine similarity over the stored + // normalized vectors is then a dot product. + // + // v0.2.5: the query is also split into identifier tokens (matching the + // TF-IDF-weighted token contribution the builder wrote). Query tokens are + // hashed at equal magnitude — the builder already baked each token's idf + // weight into the stored entity vectors, so an equal-weight query token + // automatically scores higher against the entity that shares that token + // at high weight (i.e. the rare, discriminative one). + std::vector qvec(kVecDim, 0.0f); + { + const std::string t = query; + for (size_t i = 0; i + 3 <= t.size(); ++i) { + std::string gram = t.substr(i, 3); + for (char &ch : gram) + ch = static_cast(std::tolower( + static_cast(ch))); + uint64_t h = hashMix(std::hash{}(gram) ^ + static_cast(project_id)); + int b1 = static_cast(h % kVecDim); + int b2 = static_cast(hashMix(h) % kVecDim); + qvec[b1] += (h & 1) ? 1.0f : -1.0f; + qvec[b2] += (h & 2) ? 1.0f : -1.0f; + } + // Identifier-token contributions (equal weight; idf lives in the + // stored entity vectors). + for (const std::string &tok : splitIdentifierWords(t)) { + if (tok.empty()) + continue; + uint64_t h = hashMix(std::hash{}(tok) ^ + static_cast(project_id)); + int b1 = static_cast(h % kVecDim); + int b2 = static_cast(hashMix(h) % kVecDim); + qvec[b1] += (h & 1) ? 1.0f : -1.0f; + qvec[b2] += (h & 2) ? 1.0f : -1.0f; + } + double norm = 0.0; + for (float v : qvec) + norm += static_cast(v) * v; + if (norm > 0.0) { + const float inv = + static_cast(1.0 / std::sqrt(norm)); + for (float &v : qvec) + v *= inv; + } + } + + // Early-exit when no vectors exist for this project: nothing to match, + // report empty with a reason so callers can fall back to FTS. + { + sqlite3_stmt *chk = nullptr; + const char *csql = + "SELECT COUNT(*) FROM node_vectors WHERE project_id = ?"; + if (sqlite3_prepare_v2(db_, csql, -1, &chk, nullptr) != + SQLITE_OK) { + fprintf(stderr, + "[module=store, method=%s] count prepare failed: " + "%s\n", + kMethod, sqlite3_errmsg(db_)); + return "{\"method\":\"semantic\",\"results\":[]}"; + } + sqlite3_bind_int64(chk, 1, static_cast(project_id)); + bool has_rows = false; + if (sqlite3_step(chk) == SQLITE_ROW && + sqlite3_column_int64(chk, 0) > 0) + has_rows = true; + sqlite3_finalize(chk); + if (!has_rows) + return "{\"method\":\"semantic\",\"results\":[]," + "\"reason\":\"embedding_not_built\"}"; + } + + // Full scan of node_vectors joined to entity, computing cosine. + struct Hit { + int64_t node_id; + std::string name; + std::string qualified_name; + std::string file_path; + float score; + }; + std::vector hits; + { + const char *sql = + "SELECT v.node_id, v.vector, " + " COALESCE(NULLIF(e.qualified_name,''),e.name), " + " e.name, e.file_path " + "FROM node_vectors v JOIN entity e ON e.id = v.node_id " + "WHERE v.project_id = ?"; + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db_, sql, -1, &stmt, nullptr) != + SQLITE_OK) { + fprintf(stderr, + "[module=store, method=%s] scan prepare failed: %s\n", + kMethod, sqlite3_errmsg(db_)); + return "{\"method\":\"semantic\",\"results\":[]," + "\"error\":\"scan_failed\"}"; + } + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + while (sqlite3_step(stmt) == SQLITE_ROW) { + int64_t nid = sqlite3_column_int64(stmt, 0); + const void *blob = sqlite3_column_blob(stmt, 1); + int nbytes = sqlite3_column_bytes(stmt, 1); + const char *qn = reinterpret_cast( + sqlite3_column_text(stmt, 2)); + const char *nm = reinterpret_cast( + sqlite3_column_text(stmt, 3)); + const char *fp = reinterpret_cast( + sqlite3_column_text(stmt, 4)); + if (!blob || + nbytes < static_cast(kVecDim * sizeof(float))) + continue; + // Deserialize float32 little-endian and dot with qvec. + float dot = 0.0f; + for (int d = 0; d < kVecDim; ++d) { + uint32_t bits = 0; + const uint8_t *b = + static_cast(blob) + + d * 4; + bits |= static_cast(b[0]); + bits |= static_cast(b[1]) << 8; + bits |= static_cast(b[2]) << 16; + bits |= static_cast(b[3]) << 24; + float f; + memcpy(&f, &bits, sizeof(f)); + dot += f * qvec[d]; + } + // Accuracy-first gate (0LLM design): only strong matches are + // reported. Empirically the true positive for an n-gram hash + // vector (a name sharing the query's trigrams) scores > 0.6, + // while unrelated names cluster in the 0.02–0.23 band. A + // 0.3 floor therefore keeps every relevant hit while + // rejecting the noise, so semantic search never pollutes + // results with weak/incidental matches (the accuracy + // fixtures depend on exact FTS/trigram and must stay clean). + if (dot <= kSemanticScoreFloor) + continue; + Hit hit; + hit.node_id = nid; + hit.name = nm ? nm : ""; + hit.qualified_name = qn ? qn : ""; + hit.file_path = fp ? fp : ""; + hit.score = dot; + hits.push_back(std::move(hit)); + } + sqlite3_finalize(stmt); + } + + // Rank by descending similarity, cap at limit. + std::partial_sort(hits.begin(), + hits.begin() + std::min(static_cast(limit), + hits.size()), + hits.end(), [](const Hit &a, const Hit &b) { + return a.score > b.score; + }); + if (hits.size() > static_cast(limit)) + hits.resize(static_cast(limit)); + + std::ostringstream json; + json << "{\"method\":\"semantic\",\"total\":" << hits.size() + << ",\"results\":["; + for (size_t i = 0; i < hits.size(); ++i) { + if (i > 0) + json << ","; + util::JsonWriter el; + el.beginObject(); + el.key("node_id").value(hits[i].node_id); + el.key("name").value(hits[i].name); + el.key("qualified_name").value(hits[i].qualified_name); + el.key("file_path").value(hits[i].file_path); + el.key("score").value(hits[i].score); + el.endObject(); + json << el.str(); + } + json << "]}"; + return json.str(); +} +} // namespace store diff --git a/engine/src/store/store_search_words.h b/engine/src/store/store_search_words.h new file mode 100644 index 0000000..2d796d3 --- /dev/null +++ b/engine/src/store/store_search_words.h @@ -0,0 +1,61 @@ +// store_search_words.h — identifier word splitter shared by the FTS5 +// indexer (store_search.cpp) and the n-gram vectorizer +// (store_search_vector.cpp), which were split into separate TUs. +#ifndef CODESCOPE_STORE_SEARCH_WORDS_H +#define CODESCOPE_STORE_SEARCH_WORDS_H + +#include +#include +#include + +namespace store +{ + +/// Split a camelCase / snake_case / kebab-case identifier into its +/// constituent words so an FTS5 query can match both styles: +/// "findByLastName" -> find by last name +/// "find_by_last_name" -> find by last name +/// The unicode61 tokenizer treats underscore and case boundaries as part +/// of a single token, so a bare `"findByLastName"` MATCH never hits +/// snake_case code and vice versa. Splitting at lower->upper boundaries +/// and at '_'/'-' yields the shared words. +inline std::vector splitIdentifierWords(const std::string &word) +{ + std::vector out; + std::string cur; + auto flush = [&]() { + if (!cur.empty()) { + out.push_back(cur); + cur.clear(); + } + }; + for (size_t i = 0; i < word.size(); ++i) { + char c = word[i]; + if (c == '_' || c == '-' || c == '.' || c == '/' || c == ':' || + c == '(' || c == ')') { + flush(); + continue; + } + if (std::isupper(static_cast(c)) && + !cur.empty()) { + // Lower -> upper boundary (camelCase: "lastName"). + // Keep an acronym run together ("JSONParser" stays one + // split unless the next char is lower). + char prev = cur.back(); + if (!std::isupper(static_cast(prev)) || + (i + 1 < word.size() && + std::islower(static_cast( + word[i + 1])))) { + flush(); + } + } + cur += static_cast( + std::tolower(static_cast(c))); + } + flush(); + return out; +} + +} // namespace store + +#endif // CODESCOPE_STORE_SEARCH_WORDS_H diff --git a/engine/src/store/store_semantic_fact.cpp b/engine/src/store/store_semantic_fact.cpp index e92b5ef..c041541 100644 --- a/engine/src/store/store_semantic_fact.cpp +++ b/engine/src/store/store_semantic_fact.cpp @@ -125,6 +125,10 @@ bool GraphStore::clearSemanticFacts(uint64_t project_id) } sqlite3_bind_int64(stmt, 1, static_cast(project_id)); int rc = sqlite3_step(stmt); + // Reset before returning so the cached statement is reusable and its + // bindings (SQLITE_STATIC, pointing at caller-owned storage) are + // released here rather than at the next cache hit. + sqlite3_reset(stmt); if (rc != SQLITE_DONE) { error_ = std::string("clearSemanticFacts: step failed: ") + sqlite3_errmsg(db_); diff --git a/engine/src/store/store_type.cpp b/engine/src/store/store_type.cpp deleted file mode 100644 index 0a8fdc7..0000000 --- a/engine/src/store/store_type.cpp +++ /dev/null @@ -1,174 +0,0 @@ -// store_type.cpp — Type registry batch insert implementation. -// -// Provides batch insert methods for type_info and type_ref tables, -// following the same multi-VALUES pattern as store_batch.cpp. -// Extracted into its own file to keep each translation unit under -// the 1000-line limit imposed by plan/rules/code_rules.md. -// -// Reference: codebase-memory-mcp (MIT, https://github.com/DeusData/codebase-memory-mcp) -// internal/cbm/extract_type_refs.c — type reference extraction pattern -// internal/cbm/extract_type_assigns.c — type assignment tracking pattern - -#include "store.h" - -#include -#include -#include -#include -#include -#include - -namespace store -{ - -// ─── Constants ──────────────────────────────────────────────────── - -/** Max rows per multi-VALUES batch (keeps SQL string under ~64 KB). */ -static constexpr int kTypeBatchSize = 500; - -// ─── Type Info Batch Insert ─────────────────────────────────────── - -bool GraphStore::insertTypeInfoBatch( - uint64_t project_id, - const std::vector> &rows) -{ - if (rows.empty()) - return true; - - beginTransaction(); - - for (size_t off = 0; off < rows.size(); off += kTypeBatchSize) { - size_t batch = rows.size() - off; - if (batch > static_cast(kTypeBatchSize)) - batch = static_cast(kTypeBatchSize); - - std::string sql = - "INSERT INTO type_info " - "(project_id, name, qualified_name, kind, " - " file_path, language, " - " start_row, start_col, end_row, end_col) VALUES "; - - for (size_t i = 0; i < batch; i++) { - if (i > 0) - sql += ","; - char buf[32]; - snprintf(buf, sizeof(buf), "%llu", - (unsigned long long)project_id); - sql += "(" + std::string(buf) + ",?,?,?,?,?,?,?,?)"; - } - - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql.c_str(), -1, &stmt, nullptr) != - SQLITE_OK) { - fprintf(stderr, - "[module=store, method=insertTypeInfoBatch] " - "prepare failed: %s\n", - sqlite3_errmsg(db_)); - commitTransaction(); - return false; - } - - int idx = 1; - for (size_t i = 0; i < batch; i++) { - const auto &row = rows[off + i]; - sqlite3_bind_text(stmt, idx++, std::get<0>(row).c_str(), - -1, SQLITE_TRANSIENT); - sqlite3_bind_text(stmt, idx++, std::get<1>(row).c_str(), - -1, SQLITE_TRANSIENT); - sqlite3_bind_int(stmt, idx++, std::get<2>(row)); - sqlite3_bind_text(stmt, idx++, std::get<3>(row).c_str(), - -1, SQLITE_TRANSIENT); - sqlite3_bind_text(stmt, idx++, std::get<4>(row).c_str(), - -1, SQLITE_TRANSIENT); - sqlite3_bind_int(stmt, idx++, std::get<5>(row)); - sqlite3_bind_int(stmt, idx++, std::get<6>(row)); - sqlite3_bind_int(stmt, idx++, std::get<7>(row)); - sqlite3_bind_int(stmt, idx++, std::get<8>(row)); - } - - int rc = sqlite3_step(stmt); - if (rc != SQLITE_DONE) { - fprintf(stderr, - "[module=store, method=insertTypeInfoBatch] " - "step failed (rc=%d): %s\n", - rc, sqlite3_errmsg(db_)); - } - sqlite3_finalize(stmt); - } - - commitTransaction(); - return true; -} - -// ─── Type Ref Batch Insert ─────────────────────────────────────── - -bool GraphStore::insertTypeRefBatch( - uint64_t project_id, - const std::vector> &rows) -{ - if (rows.empty()) - return true; - - beginTransaction(); - - for (size_t off = 0; off < rows.size(); off += kTypeBatchSize) { - size_t batch = rows.size() - off; - if (batch > static_cast(kTypeBatchSize)) - batch = static_cast(kTypeBatchSize); - - std::string sql = "INSERT INTO type_ref " - "(project_id, entity_id, type_name, kind, " - " file_path, start_row, start_col) VALUES "; - - for (size_t i = 0; i < batch; i++) { - if (i > 0) - sql += ","; - char buf[32]; - snprintf(buf, sizeof(buf), "%llu", - (unsigned long long)project_id); - sql += "(" + std::string(buf) + ",?,?,?,?,?,?)"; - } - - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db_, sql.c_str(), -1, &stmt, nullptr) != - SQLITE_OK) { - fprintf(stderr, - "[module=store, method=insertTypeRefBatch] " - "prepare failed: %s\n", - sqlite3_errmsg(db_)); - commitTransaction(); - return false; - } - - int idx = 1; - for (size_t i = 0; i < batch; i++) { - const auto &row = rows[off + i]; - sqlite3_bind_int64( - stmt, idx++, - static_cast(std::get<0>(row))); - sqlite3_bind_text(stmt, idx++, std::get<1>(row).c_str(), - -1, SQLITE_TRANSIENT); - sqlite3_bind_int(stmt, idx++, std::get<2>(row)); - sqlite3_bind_text(stmt, idx++, std::get<3>(row).c_str(), - -1, SQLITE_TRANSIENT); - sqlite3_bind_int(stmt, idx++, std::get<4>(row)); - sqlite3_bind_int(stmt, idx++, std::get<5>(row)); - } - - int rc = sqlite3_step(stmt); - if (rc != SQLITE_DONE) { - fprintf(stderr, - "[module=store, method=insertTypeRefBatch] " - "step failed (rc=%d): %s\n", - rc, sqlite3_errmsg(db_)); - } - sqlite3_finalize(stmt); - } - - commitTransaction(); - return true; -} - -} // namespace store \ No newline at end of file diff --git a/engine/src/util/json_writer.cpp b/engine/src/util/json_writer.cpp new file mode 100644 index 0000000..84c59ee --- /dev/null +++ b/engine/src/util/json_writer.cpp @@ -0,0 +1,314 @@ +#include "util/json_writer.h" + +#include +#include +#include +#include +#include + +namespace util +{ + +std::string jsonEscapeString(const std::string &s) +{ + std::string out; + out.reserve(s.size() + 8); + for (char c : s) { + switch (c) { + case '"': + out += "\\\""; + break; + case '\\': + out += "\\\\"; + break; + case '\n': + out += "\\n"; + break; + case '\r': + out += "\\r"; + break; + case '\t': + out += "\\t"; + break; + default: + if (static_cast(c) < 0x20) { + // Remaining C0 control bytes: \u00XX. The buffer holds + // "\uXXXX" plus NUL and is always NUL-terminated. + char buf[8]; + snprintf(buf, sizeof(buf), "\\u%04x", + static_cast(c)); + out += buf; + } else { + out += c; + } + break; + } + } + return out; +} + +namespace +{ + +/// Shared body of the two envelope helpers: build +/// `{"":,"error":"[module=…, method=…] "}`, or without +/// the flag for the plain error envelope. +/// @param with_ok_flag Emit `"ok":false` before the error member. +/// @param module Module name for the trace chain. +/// @param method Function name for the trace chain. +/// @param message Raw message, escaped by the writer. +/// @return The complete JSON object. +std::string envelope(bool with_ok_flag, const char *module, const char *method, + const std::string &message) +{ + std::string prefix = "[module="; + prefix += module ? module : "unknown"; + prefix += ", method="; + prefix += method ? method : "unknown"; + prefix += "] "; + prefix += message; + + JsonWriter w; + w.beginObject(); + if (with_ok_flag) + w.key("ok").value(false); + w.key("error").value(prefix); + w.endObject(); + return w.str(); +} + +} // namespace + +std::string errorEnvelope(const char *module, const char *method, + const std::string &message) +{ + return envelope(false, module, method, message); +} + +std::string okFalseEnvelope(const char *module, const char *method, + const std::string &message) +{ + return envelope(true, module, method, message); +} + +JsonWriter::JsonWriter() +{ + out_.reserve(256); +} + +void JsonWriter::reset() +{ + out_.clear(); + error_.clear(); + stack_.clear(); + awaiting_value_ = false; + started_ = false; +} + +void JsonWriter::setError(const char *msg) +{ + // Keep the FIRST error: it is the root cause; later misuse is a + // consequence of the writer having stopped emitting. + if (error_.empty()) + error_ = msg; +} + +void JsonWriter::beforeValue() +{ + // Called before emitting any value (scalar, container open, raw). + if (stack_.empty()) { + if (started_) + setError("multiple top-level JSON values"); + return; + } + Frame &f = stack_.back(); + if (f.is_object) { + // Inside an object a value is only legal right after a key. + if (!awaiting_value_) + setError("object value without a key"); + return; + } + // Inside an array: separate from the previous element. + if (f.has_items) + out_ += ','; +} + +void JsonWriter::markValueDone() +{ + if (stack_.empty()) { + started_ = true; + return; + } + Frame &f = stack_.back(); + if (f.is_object) + awaiting_value_ = false; + f.has_items = true; +} + +void JsonWriter::appendEscapedString(const std::string &v) +{ + out_ += '"'; + out_ += jsonEscapeString(v); + out_ += '"'; +} + +void JsonWriter::beginContainer(char open, bool is_object) +{ + if (!error_.empty()) + return; + beforeValue(); + if (!error_.empty()) + return; + out_ += open; + // The container itself is a value in the enclosing scope; mark it done + // BEFORE pushing the new frame so the parent's has_items/started_ flags + // are updated against the parent, not the child. + markValueDone(); + stack_.push_back(Frame{ is_object, false }); + awaiting_value_ = false; +} + +void JsonWriter::endContainer(char close, bool is_object) +{ + if (!error_.empty()) + return; + if (stack_.empty() || stack_.back().is_object != is_object) { + setError(is_object ? "endObject without beginObject" : + "endArray without beginArray"); + return; + } + if (is_object && awaiting_value_) { + setError("object key without a value"); + return; + } + stack_.pop_back(); + out_ += close; + awaiting_value_ = false; +} + +JsonWriter &JsonWriter::beginObject() +{ + beginContainer('{', true); + return *this; +} + +JsonWriter &JsonWriter::endObject() +{ + endContainer('}', true); + return *this; +} + +JsonWriter &JsonWriter::beginArray() +{ + beginContainer('[', false); + return *this; +} + +JsonWriter &JsonWriter::endArray() +{ + endContainer(']', false); + return *this; +} + +JsonWriter &JsonWriter::key(const std::string &k) +{ + if (!error_.empty()) + return *this; + if (stack_.empty() || !stack_.back().is_object) { + setError("key outside an object"); + return *this; + } + if (awaiting_value_) { + setError("object key without a value for the previous key"); + return *this; + } + Frame &f = stack_.back(); + if (f.has_items) + out_ += ','; + appendEscapedString(k); + out_ += ':'; + awaiting_value_ = true; + return *this; +} + +JsonWriter &JsonWriter::key(const char *k) +{ + return key(std::string(k ? k : "")); +} + +JsonWriter &JsonWriter::value(const std::string &v) +{ + if (!error_.empty()) + return *this; + beforeValue(); + if (!error_.empty()) + return *this; + appendEscapedString(v); + markValueDone(); + return *this; +} + +JsonWriter &JsonWriter::value(const char *v) +{ + return value(std::string(v ? v : "")); +} + +JsonWriter &JsonWriter::value(bool v) +{ + if (!error_.empty()) + return *this; + beforeValue(); + if (!error_.empty()) + return *this; + out_ += v ? "true" : "false"; + markValueDone(); + return *this; +} + +JsonWriter &JsonWriter::value(double v) +{ + // JSON cannot represent NaN/Inf; degrade to null rather than emit a + // token no parser accepts. + if (!std::isfinite(v)) + return nullValue(); + if (!error_.empty()) + return *this; + beforeValue(); + if (!error_.empty()) + return *this; + // Match std::ostringstream's default double formatting in a fixed + // locale: defaultfloat, precision 6. Keeping the exact format preserves + // byte-compatibility with the string-concatenation code this replaces. + std::ostringstream oss; + oss.imbue(std::locale::classic()); + oss << v; + out_ += oss.str(); + markValueDone(); + return *this; +} + +JsonWriter &JsonWriter::nullValue() +{ + if (!error_.empty()) + return *this; + beforeValue(); + if (!error_.empty()) + return *this; + out_ += "null"; + markValueDone(); + return *this; +} + +JsonWriter &JsonWriter::raw(const std::string &json) +{ + if (!error_.empty()) + return *this; + beforeValue(); + if (!error_.empty()) + return *this; + out_ += json; + markValueDone(); + return *this; +} + +} // namespace util diff --git a/engine/src/util/json_writer.h b/engine/src/util/json_writer.h new file mode 100644 index 0000000..a118cbe --- /dev/null +++ b/engine/src/util/json_writer.h @@ -0,0 +1,189 @@ +#ifndef CODESCOPE_UTIL_JSON_WRITER_H +#define CODESCOPE_UTIL_JSON_WRITER_H + +#include +#include +#include + +namespace util +{ + +/// Escape a UTF-8 string for embedding inside a JSON string literal. +/// +/// Handles the two mandatory escapes (`"` and `\`), the shorthand escapes +/// (`\n`, `\r`, `\t`), and `\uXXXX` for every other C0 control byte +/// (0x00-0x1f). Bytes >= 0x20 pass through unchanged, so valid UTF-8 stays +/// valid. This is the single source of truth for JSON escaping: the legacy +/// `jsonEscape()` wrappers are gone, so callers use either this function or +/// JsonWriter (which escapes through it). +/// +/// @param s Raw bytes to escape. +/// @return The escaped contents, WITHOUT the surrounding quotes. +std::string jsonEscapeString(const std::string &s); + +/// Build the error envelope every FFI export returns on failure: +/// `{"error":"[module=, method=] "}`. +/// +/// Centralised because the shape was hand-concatenated in ~150 places, and each +/// one had to remember `jsonEscape` around the message: an exception message +/// containing a quote, a backslash or a control byte produced invalid JSON that +/// the Rust server then fed straight into serde_json. The message is escaped +/// here, unconditionally, and the `[module=…, method=…]` prefix is what makes +/// the error traceable to a module and a method (plan/rules/code_rules.md: +/// errors must never be silent and must carry a trace chain). +/// +/// @param module Module name for the trace chain, e.g. "ffi". +/// @param method Function name for the trace chain, e.g. "engine_get_routes". +/// @param message Raw message; escaped, so `e.what()` can be passed directly. +/// @return A complete JSON object, ready for `dupString()`. +std::string errorEnvelope(const char *module, const char *method, + const std::string &message); + +/// Build the error envelope for the entry points that report an explicit `ok` +/// flag: `{"ok":false,"error":"[module=, method=] "}`. +/// Same escaping and trace-chain guarantees as errorEnvelope(); the two shapes +/// coexist because callers distinguish them (see engine.h), so this helper +/// keeps each one byte-identical to what it replaces. +/// +/// @param module Module name for the trace chain, e.g. "ffi". +/// @param method Function name for the trace chain. +/// @param message Raw message; escaped before it is embedded. +/// @return A complete JSON object, ready for `dupString()`. +std::string okFalseEnvelope(const char *module, const char *method, + const std::string &message); + +/// Incremental, always-escaping JSON serializer. +/// +/// Replaces the hand-rolled +/// `std::string j = "{\"k\":\"" + v + "\"}";` pattern (plan/rules/ +/// code_rules.md: no manual JSON, no silently omitted escaping) with a +/// builder that inserts separators and escapes every string value and key. +/// +/// Typical use: +/// @code +/// util::JsonWriter w; +/// w.beginObject(); +/// w.key("name").value(symbol_name); +/// w.key("total").value(rows.size()); +/// w.key("items").beginArray(); +/// for (const auto &r : rows) +/// w.value(r.name); +/// w.endArray(); +/// w.endObject(); +/// return dupString(w.str()); +/// @endcode +/// +/// Structural misuse (a value with no key inside an object, a key without a +/// value, mismatched endObject/endArray, a second top-level value) records +/// an error() and stops emitting rather than producing malformed JSON. A +/// caller that wants a fail-closed envelope can check ok(). +/// +/// Not thread-safe; use one instance per logical output. +class JsonWriter { + public: + JsonWriter(); + + /// Open a `{`. Begins an object whose members are added with key(). + JsonWriter &beginObject(); + /// Close the current object. Errors if the top container is not one. + JsonWriter &endObject(); + /// Open a `[`. Begins an array whose items are added with value(). + JsonWriter &beginArray(); + /// Close the current array. Errors if the top container is not one. + JsonWriter &endArray(); + + /// Emit an object member key. Must be inside an object and must be + /// followed by exactly one value. + JsonWriter &key(const std::string &k); + JsonWriter &key(const char *k); + + /// Emit a string value (escaped). + JsonWriter &value(const std::string &v); + JsonWriter &value(const char *v); + + /// Emit a boolean value. + JsonWriter &value(bool v); + + /// Emit any integral value. Excludes bool (handled above) so an int is + /// never serialized as `true`/`false`. Covers int, int64_t, uint64_t, + /// size_t, ... without ambiguous overload resolution. + template && + !std::is_same_v, bool>, + int>::type = 0> + JsonWriter &value(T v) + { + if (!error_.empty()) + return *this; + beforeValue(); + if (!error_.empty()) + return *this; + out_ += std::to_string(v); + markValueDone(); + return *this; + } + + /// Emit a number for a finite double, or `null` for NaN/Inf (JSON has no + /// representation for them). Formatting matches `std::ostringstream`'s + /// default (defaultfloat, precision 6) in the classic locale, so output + /// is byte-compatible with the code this utility replaces. + JsonWriter &value(double v); + + /// Emit the JSON literal `null`. + JsonWriter &nullValue(); + + /// Append already-serialized JSON verbatim. The caller guarantees a + /// single well-formed JSON value; it is NOT escaped or validated. Use + /// for a nested fragment produced elsewhere. + JsonWriter &raw(const std::string &json); + + /// The JSON text built so far. + const std::string &str() const + { + return out_; + } + /// Move the JSON text out (leaves the writer empty). + std::string take() + { + return std::move(out_); + } + /// True when no structural error has been recorded. + bool ok() const + { + return error_.empty(); + } + /// The first structural error message, or "" when ok(). + const std::string &error() const + { + return error_; + } + /// Clear all state for reuse. + void reset(); + + private: + void beforeValue(); + void markValueDone(); + void beginContainer(char open, bool is_object); + void endContainer(char close, bool is_object); + void appendEscapedString(const std::string &v); + void setError(const char *msg); + + struct Frame { + bool is_object; + bool has_items; + }; + + std::string out_; + std::string error_; + std::vector stack_; + /// True after an object key is emitted and before its value arrives. + bool awaiting_value_ = false; + /// True once a top-level value has been emitted (so a second one is an + /// error). + bool started_ = false; +}; + +} // namespace util + +#endif // CODESCOPE_UTIL_JSON_WRITER_H diff --git a/engine/src/util/path_util.h b/engine/src/util/path_util.h new file mode 100644 index 0000000..0ceb09e --- /dev/null +++ b/engine/src/util/path_util.h @@ -0,0 +1,79 @@ +#ifndef CODESCOPE_UTIL_PATH_UTIL_H +#define CODESCOPE_UTIL_PATH_UTIL_H + +#include +#include +#include + +namespace util +{ + +/// Absolute, symlink-resolved form of a path — the shape the store keeps. +/// +/// `weakly_canonical` rather than `canonical`: it resolves the prefix that +/// exists and normalises the rest lexically, so ".", "./foo", "foo/../bar" and +/// absolute paths are all handled uniformly, including paths that do not exist +/// yet (an index target named on the command line, a project root that is +/// about to be created). Falls back to the input when resolution fails, so +/// callers that only need a best effort keep working. +/// +/// This is the ONE implementation of the rule. It is what `createProject` uses +/// to register a project root (store_core.cpp), and therefore what every stored +/// `file_path` is built from; callers that build a path to look up must use it +/// too, or an exact / prefix match against those rows misses. +inline std::string resolvePath(const std::string &path) +{ + namespace fs = std::filesystem; + std::error_code ec; + const fs::path resolved = fs::weakly_canonical(fs::path(path), ec); + if (ec) + return path; + // Native form on POSIX so the string is byte-identical to what was stored. +#ifdef _WIN32 + return resolved.string(); +#else + return resolved.native(); +#endif +} + +/// The form of a caller-supplied path to match against stored `file_path` +/// values. +/// +/// Only an ABSOLUTE input is resolved. These arguments are documented as "the +/// absolute file path" but the SQL wraps them in `file_path LIKE '%'||?||'%'`, +/// so they double as a substring filter: "src/main.go" or "impact_analysis" +/// must stay exactly as given, because resolving either against the current +/// directory would turn a working filter into a wrong one. An absolute path, +/// however, is meant to name the file, and the stored row is usually resolved — +/// a caller reaching the project through a symlink matched nothing and read as +/// "no such symbol" (reproduced: indexing /tmp/pn/real and filtering on +/// /tmp/pn/link/src/main.go returned `{"results":[],"total":0}` for a symbol +/// that exists). +inline std::string lookupPath(const std::string &path) +{ + if (!std::filesystem::path(path).is_absolute()) + return path; + return resolvePath(path); +} + +/// The LIKE patterns a caller-supplied path filter must be matched with: the +/// value exactly as given, and its resolved form. +/// +/// Two patterns rather than one, because the stored rows are not guaranteed to +/// be resolved either: `createProject` resolves the root it registers, but an +/// index driven through the engine API keeps whatever prefix the caller passed +/// (measured — a fixture indexed as `/tmp/...` holds `/tmp/...` file_path rows, +/// while the CLI-indexed ones hold `/private/tmp/...`). Resolving the filter +/// alone therefore trades a miss on one database for a miss on the other; +/// accepting both is additive and matches whichever form that database holds. +/// When the two forms are identical, so are the patterns, and the second test +/// is a harmless repeat of the first. +inline std::vector lookupPatterns(const std::string &path) +{ + const std::string resolved = lookupPath(path); + return { "%" + path + "%", "%" + resolved + "%" }; +} + +} // namespace util + +#endif // CODESCOPE_UTIL_PATH_UTIL_H diff --git a/engine/src/verify/capability_drift.cpp b/engine/src/verify/capability_drift.cpp index c768ada..4544484 100644 --- a/engine/src/verify/capability_drift.cpp +++ b/engine/src/verify/capability_drift.cpp @@ -1,4 +1,6 @@ #include "capability_drift.h" +#include "capability_verifier.h" +#include "registry.h" #include #include @@ -11,60 +13,30 @@ int64_t countImplementingEntities(store::GraphStore &store, uint64_t project_id, { if (cap_name.empty()) return 0; - - sqlite3 *db = store.handle(); - if (!db) { + if (!store.handle()) { fprintf(stderr, "[module=verify, method=countImplementingEntities] " "db handle is null\n"); - return 0; - } - - // An implementing entity is one whose name matches the capability - // name and that has at least one incoming call edge (a caller in the - // relation table with type=1). Without callers the entity is dead - // code and cannot be said to "implement" an active capability. - // - // Match direction: bidirectional prefix LIKE. The capability name is - // README-derived PascalCase (e.g. "IncrementalIndexing") while the - // entity name is a short code symbol (e.g. "incremental_index" or - // "IncrementalIndex"). A single direction LIKE name||'%' misses one - // side: exact `e.name=?` (the previous code) missed almost every - // capability because README-derived names rarely equal code symbols. - // We accept a match when either name starts with the other, so both - // "IncrementalIndex" → "IncrementalIndexing" and the reverse work. - const char *sql = "SELECT COUNT(*) FROM entity e " - "WHERE e.project_id=? " - "AND (LOWER(e.name) LIKE LOWER(?) || '%' " - " OR LOWER(?) LIKE LOWER(e.name) || '%') " - "AND EXISTS (SELECT 1 FROM relation r " - " WHERE r.project_id=? AND r.type=1 " - " AND r.target_id=e.id)"; - sqlite3_stmt *stmt = nullptr; - if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) != SQLITE_OK) { - fprintf(stderr, - "[module=verify, method=countImplementingEntities] " - "prepare failed: %s\n", - sqlite3_errmsg(db)); - return 0; + return -1; } - sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_text(stmt, 2, cap_name.c_str(), -1, SQLITE_STATIC); - sqlite3_bind_text(stmt, 3, cap_name.c_str(), -1, SQLITE_STATIC); - sqlite3_bind_int64(stmt, 4, static_cast(project_id)); - int64_t count = 0; - int rc = sqlite3_step(stmt); - if (rc == SQLITE_ROW) { - count = sqlite3_column_int64(stmt, 0); - } else if (rc != SQLITE_DONE) { - fprintf(stderr, - "[module=verify, method=countImplementingEntities] " - "step failed with rc=%d: %s\n", - rc, sqlite3_errmsg(db)); - } - sqlite3_finalize(stmt); - return count; + // The rule (name match + evidence) lives in + // CapabilityVerifier::implementingEntitiesFor; this counts its result so + // the drift report and verify_claim cannot disagree (see the note at the + // end of this function). + // One rule, one implementation: this delegates to the verifier's + // implementingEntitiesFor (see capability_verifier.h). Before, this file + // carried its own copy of the name match and the caller requirement, so + // detect_capability_drift and verify_claim could answer differently for + // the same declared capability — and did, on this repository, once the + // C-ABI spelling (`engine_verify_claim`) and exported-but-uncalled + // symbols were accounted for in only one of the two copies. + bool ok = true; + const std::vector ids = + implementingEntitiesFor(&store, project_id, cap_name, ok); + if (!ok) + return -1; + return static_cast(ids.size()); } std::vector detectCapabilityDrift(store::GraphStore &store, @@ -78,6 +50,24 @@ std::vector detectCapabilityDrift(store::GraphStore &store, return drifts; } + // Evidence gate (same one the four registry verifiers use). With an + // empty entity/relation table countImplementingEntities() returns 0 for + // every capability, so each declared capability would be reported as a + // severity-2 drift — "declared in README but not implemented" — purely + // because nothing has been indexed yet. That is a hard conclusion drawn + // from missing evidence, which is exactly what this project must not do. + int64_t gate_entities = 0; + int64_t gate_relations = 0; + if (!evidence_backend_ready(&store, project_id, &gate_entities, + &gate_relations)) { + fprintf(stderr, + "[module=verify, method=detectCapabilityDrift] evidence " + "backend not ready (entity=%lld, relation=%lld): no " + "drift conclusions reported\n", + (long long)gate_entities, (long long)gate_relations); + return drifts; + } + // Read all declared capabilities for this project. Each row in the // capability table represents a feature the project claims to provide // (typically extracted from the README by the CapabilityPlugin). @@ -101,6 +91,17 @@ std::vector detectCapabilityDrift(store::GraphStore &store, int64_t impl_count = countImplementingEntities(store, project_id, name); + if (impl_count < 0) { + // Query failure, not "no implementors": reporting drift + // here would draw a hard conclusion from a failed query + // (same rule as the verifiers' Unknown-on-error). + fprintf(stderr, + "[module=verify, method=detectCapabilityDrift] " + "countImplementingEntities failed for '%s' — " + "skipping drift conclusion\n", + name.c_str()); + continue; + } if (impl_count == 0) { DriftItem item; item.type = "CapabilityDrift"; @@ -109,7 +110,7 @@ std::vector detectCapabilityDrift(store::GraphStore &store, item.detail = "Capability '" + name + "' declared in README but no implementing " - "entity with callers found in codebase"; + "implementing entity (called or exported) found in codebase"; drifts.push_back(item); } } diff --git a/engine/src/verify/capability_verifier.cpp b/engine/src/verify/capability_verifier.cpp index 5e62b01..97a72c3 100644 --- a/engine/src/verify/capability_verifier.cpp +++ b/engine/src/verify/capability_verifier.cpp @@ -20,6 +20,9 @@ static constexpr double kConfNoStore = 0.0; // IN (1,3) mixed the legacy graph_edges numbering (1=call_graph, // 3=symbol_reference) and let non-call references pollute the caller set. static constexpr int kRelationTypeCalls = 1; +/// Minimum length of the shorter side of a capability/entity PREFIX match +/// (see entitiesWithCallers). Exact equality always matches. +static constexpr int kMinCapabilityPrefixLen = 4; namespace verify { @@ -38,9 +41,12 @@ CapabilityVerifier::CapabilityVerifier(store::GraphStore *store, // (relation type=1 incoming). A claim is Supported only when all three // links are present. If the capability is not declared, the claim is // Contradicted with high confidence (the knowledge layer explicitly -// denies it). If declared but no implementing entity has callers, the -// claim is Contradicted with lower confidence (the code may exist under -// a different name). +// denies it). If declared but nothing implements it, the claim is +// Contradicted with lower confidence (the code may exist under a +// different name). An entity implements it when its name matches (see +// implementingEntitiesFor for the spellings tried) AND it is either called +// or exported — requiring callers alone marked every C-ABI export +// unimplemented, because its callers live in the other language. bool CapabilityVerifier::accepts(const Claim &claim) const { @@ -49,8 +55,9 @@ bool CapabilityVerifier::accepts(const Claim &claim) const // Step 1: check whether the capability is declared in the knowledge layer. // Uses LOWER() on both sides so the LIKE match is case-insensitive. The -// subject is bound as the LIKE pattern, so callers can include % wildcards -// for fuzzy matching; a plain subject matches exactly (case-insensitive). +// subject is bound as the LIKE prefix pattern (escaped wildcards + trailing +// '%'), so `_` in a subject is literal and not a single-character wildcard +// (same treatment as ContractVerifier). // // Match direction: LOWER(name) LIKE LOWER(subject) || '%' // — capability name must start with (or equal) the subject. The subject @@ -63,12 +70,16 @@ bool CapabilityVerifier::accepts(const Claim &claim) const // (subject LIKE name||'%'), which required the README-derived subject to // *start with* the short stored name — almost never true, so every // capability_exists claim was Contradicted even on perfect matches. -static bool capabilityDeclared(store::GraphStore *store, uint64_t project_id, - const std::string &subject) +// +// @param subject Claim subject; LIKE wildcards are escaped before bind. +// @return 1 = declared, 0 = not declared, -1 = query failed. Callers must +// NOT treat -1 as "not declared" (that maps an error to Contradicted). +static int capabilityDeclared(store::GraphStore *store, uint64_t project_id, + const std::string &subject) { const char *sql = "SELECT 1 FROM capability " - "WHERE project_id=? AND LOWER(name) LIKE LOWER(?) || '%' " + "WHERE project_id=? AND LOWER(name) LIKE LOWER(?) ESCAPE '\\' " "LIMIT 1"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(store->handle(), sql, -1, &stmt, nullptr) != @@ -77,12 +88,22 @@ static bool capabilityDeclared(store::GraphStore *store, uint64_t project_id, "CapabilityVerifier: prepare failed: %s " "[module=verify, method=capabilityDeclared]\n", sqlite3_errmsg(store->handle())); - return false; + return -1; + } + // Escaped LIKE prefix pattern: wildcards in subject are literal, + // trailing '%' is the intentional prefix-match wildcard. + std::string pattern; + pattern.reserve(subject.size() + 1); + for (char c : subject) { + if (c == '\\' || c == '%' || c == '_') + pattern.push_back('\\'); + pattern.push_back(c); } + pattern.push_back('%'); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); - sqlite3_bind_text(stmt, 2, subject.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_text(stmt, 2, pattern.c_str(), -1, SQLITE_TRANSIENT); - bool found = (sqlite3_step(stmt) == SQLITE_ROW); + int found = (sqlite3_step(stmt) == SQLITE_ROW) ? 1 : 0; sqlite3_finalize(stmt); return found; } @@ -90,7 +111,7 @@ static bool capabilityDeclared(store::GraphStore *store, uint64_t project_id, // Step 2: collect entity ids that (a) match the subject name and (b) have // at least one incoming Calls relation (relation.type=1). Returns the ids // in the order produced by SQLite. Empty result means "no implementing -// entity with callers" -> the claim cannot be Supported. +// implementing (called or exported) entity" -> the claim cannot be Supported. // // Step 9.5: migrated from graph_nodes/graph_edges to canonical // entity/relation. Per plan/rules/relation_contract.md only relation.type=1 @@ -106,33 +127,95 @@ static bool capabilityDeclared(store::GraphStore *store, uint64_t project_id, // (BUG 2026-07-17) flipped the direction but kept a single-sided test, // so it still failed whenever the subject was longer than the entity name. // We now accept a match when either side starts with the other. -static std::vector entitiesWithCallers(store::GraphStore *store, - uint64_t project_id, - const std::string &subject) +// +// @param subject Claim subject; LIKE wildcards are escaped before bind. +// @param out_ok Set to false on prepare failure (caller must map that to +// Unknown, not Contradicted). True on success even when the +// result is empty ("no implementing entity"). +// @return Entity ids in SQLite row order; empty when no match. +std::vector implementingEntitiesFor(store::GraphStore *store, + uint64_t project_id, + const std::string &subject, + bool &out_ok) { std::vector ids; + out_ok = true; + // Prefix matches need a length floor on BOTH sides. Without it the rule + // degenerates: for subject "GetNeighbors" the reverse direction + // `LOWER(?) LIKE LOWER(e.name) || '%'` is satisfied by any 1-3 + // character symbol (`get`, or a single-letter variable), so unrelated + // entities were reported as implementing the capability. + // kMinCapabilityPrefixLen (4) clears the extremely common code symbols + // of length <= 3 (get/set/add/map/run/put/len/i/x …). Exact equality + // still matches at any length, so a genuine short symbol such as `Run` + // is not lost — only the accidental prefix is. const char *sql = "SELECT e.id FROM entity e " "WHERE e.project_id=? " - "AND (LOWER(e.name) LIKE LOWER(?) || '%' " - " OR LOWER(?) LIKE LOWER(e.name) || '%') " - "AND EXISTS (SELECT 1 FROM relation r " + "AND (LOWER(e.name) = LOWER(?) " + " OR (LENGTH(?) >= ? AND LENGTH(e.name) >= ? " + " AND (LOWER(e.name) LIKE LOWER(?) || '%' " + // Spelled differently, same symbol: the README names the capability + // ("VerifyClaim…") while the exported symbol carries a language + // prefix ("engine_verify_claim"). Compare the de-underscored names + // and the entity name without its first '_'-delimited segment, each + // behind its own length floor. The reverse direction escapes '\\', + // '%' and '_' in the entity name so they stay literal. + " OR LOWER(?) LIKE LOWER(REPLACE(REPLACE(REPLACE(" + " e.name, '\\', '\\\\'), '%', '\\%'), '_', '\\_'))" + " || '%' ESCAPE '\\' " + " OR (LENGTH(REPLACE(e.name, '_', '')) >= ? " + " AND (LOWER(REPLACE(e.name, '_', '')) LIKE " + " LOWER(REPLACE(?, '_', '')) || '%' " + " OR LOWER(REPLACE(?, '_', '')) LIKE " + " LOWER(REPLACE(e.name, '_', '')) || '%'))" + " OR (INSTR(e.name, '_') > 0 AND LENGTH(REPLACE(SUBSTR(" + " e.name, INSTR(e.name, '_') + 1), '_', '')) >= ? " + " AND (LOWER(REPLACE(SUBSTR(e.name, " + " INSTR(e.name, '_') + 1), '_', '')) LIKE " + " LOWER(REPLACE(?, '_', '')) || '%' " + " OR LOWER(REPLACE(?, '_', '')) LIKE LOWER(" + " REPLACE(SUBSTR(e.name, INSTR(e.name, '_') " + " + 1), '_', '')) || '%'))))) " + // Evidence: something calls it, or it is an exported/public entry + // point. Callers alone reported every C-ABI capability of this + // repository as unimplemented — an extern "C" export is called from + // the other language, which records no edge in this graph. + "AND (EXISTS (SELECT 1 FROM relation r " " WHERE r.project_id=? AND r.target_id=e.id " - " AND r.type=?)"; + " AND r.type=?) " + " OR e.visibility=1)"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(store->handle(), sql, -1, &stmt, nullptr) != SQLITE_OK) { fprintf(stderr, "CapabilityVerifier: prepare entities failed: %s " - "[module=verify, method=entitiesWithCallers]\n", + "[module=verify, method=implementingEntitiesFor]\n", sqlite3_errmsg(store->handle())); + out_ok = false; return ids; } sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, subject.c_str(), -1, SQLITE_STATIC); sqlite3_bind_text(stmt, 3, subject.c_str(), -1, SQLITE_STATIC); - sqlite3_bind_int64(stmt, 4, static_cast(project_id)); - sqlite3_bind_int(stmt, 5, kRelationTypeCalls); + sqlite3_bind_int(stmt, 4, kMinCapabilityPrefixLen); + sqlite3_bind_int(stmt, 5, kMinCapabilityPrefixLen); + // Forward direction: name LIKE subject||'%' — the raw subject; the + // literal '%' lives in the SQL. Reverse direction: subject LIKE + // name||'%' — the SQL appends the wildcard on the entity-name side. + sqlite3_bind_text(stmt, 6, subject.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_text(stmt, 7, subject.c_str(), -1, SQLITE_STATIC); + // De-underscored arm: floor, then both directions. + sqlite3_bind_int(stmt, 8, kMinCapabilityPrefixLen); + sqlite3_bind_text(stmt, 9, subject.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_text(stmt, 10, subject.c_str(), -1, SQLITE_STATIC); + // Language-prefix-stripped arm: floor, then both directions. + sqlite3_bind_int(stmt, 11, kMinCapabilityPrefixLen); + sqlite3_bind_text(stmt, 12, subject.c_str(), -1, SQLITE_STATIC); + sqlite3_bind_text(stmt, 13, subject.c_str(), -1, SQLITE_STATIC); + // Evidence subquery. + sqlite3_bind_int64(stmt, 14, static_cast(project_id)); + sqlite3_bind_int(stmt, 15, kRelationTypeCalls); while (sqlite3_step(stmt) == SQLITE_ROW) { ids.push_back(sqlite3_column_int64(stmt, 0)); @@ -173,7 +256,18 @@ EvidenceRecord CapabilityVerifier::verify(const Claim &claim) } // Step 1: the capability must be declared in the knowledge layer. - if (!capabilityDeclared(store_, project_id_, claim.subject)) { + // A query failure is Unknown, not Contradicted (code_rules: no hard + // conclusion from a failed query). + int declared = capabilityDeclared(store_, project_id_, claim.subject); + if (declared < 0) { + rec.verdict = Verdict::Unknown; + rec.confidence = kConfBackendNotReady; + rec.detail = + "CapabilityVerifier: capability lookup failed " + "[module=verify, method=CapabilityVerifier::verify]"; + return rec; + } + if (declared == 0) { rec.verdict = Verdict::Contradicted; rec.confidence = kConfCapabilityNotDeclared; rec.detail = "Capability '" + claim.subject + @@ -182,14 +276,23 @@ EvidenceRecord CapabilityVerifier::verify(const Claim &claim) } // Step 2: at least one implementing entity must have callers. - std::vector ids = - entitiesWithCallers(store_, project_id_, claim.subject); + bool query_ok = true; + std::vector ids = implementingEntitiesFor( + store_, project_id_, claim.subject, query_ok); + if (!query_ok) { + rec.verdict = Verdict::Unknown; + rec.confidence = kConfBackendNotReady; + rec.detail = + "CapabilityVerifier: entity lookup failed " + "[module=verify, method=CapabilityVerifier::verify]"; + return rec; + } if (ids.empty()) { rec.verdict = Verdict::Contradicted; rec.confidence = kConfCapabilityNoCallers; rec.detail = "Capability '" + claim.subject + - "' declared but no implementing entity with callers"; + "' declared but no implementing entity (nothing calls it and it is not exported)"; return rec; } diff --git a/engine/src/verify/capability_verifier.h b/engine/src/verify/capability_verifier.h index 2f0554e..e1d7714 100644 --- a/engine/src/verify/capability_verifier.h +++ b/engine/src/verify/capability_verifier.h @@ -11,13 +11,37 @@ namespace verify { +/** + * Entities that can be the implementation of `subject`, a documented + * capability name. The one place this rule lives: CapabilityVerifier uses it, + * and capability_drift.cpp delegates to it so a capability cannot be + * "implemented" for one caller and "missing" for the other. + * + * Name match: exact, or a bidirectional prefix with the kMinCapabilityPrefixLen + * floor, over the raw names and over two normalised spellings (de-underscored, + * and without the entity's first '_'-delimited segment) that bridge the + * README spelling and a language-prefixed export such as `engine_verify_claim`. + * + * Evidence: the entity has a caller (relation type=1 incoming) or is exported + * (entity.visibility=1). + * + * @param out_ok False only when the query itself failed; callers must map that + * to Unknown, never to Contradicted (no silent error handling). + * @return Entity ids in SQLite row order; empty when nothing matches. + */ +std::vector implementingEntitiesFor(store::GraphStore *store, + uint64_t project_id, + const std::string &subject, + bool &out_ok); + /** * CapabilityVerifier checks that claimed capabilities actually exist in the * codebase. In the Claim-driven flow it accepts CapabilityExists claims and * returns an EvidenceRecord with supporting or contradicting evidence. * * Evidence chain (Step 9.5 migrated to canonical facts): - * Capability row -> entity row -> Callers (relation type=1 incoming) + * Capability row -> entity row -> implementingEntitiesFor (callers or + * exported/public — see above) * * Legacy path: * The no-argument verify() returning std::vector is preserved so diff --git a/engine/src/verify/claim_parser.cpp b/engine/src/verify/claim_parser.cpp index aa8d7b4..29cf414 100644 --- a/engine/src/verify/claim_parser.cpp +++ b/engine/src/verify/claim_parser.cpp @@ -43,6 +43,50 @@ std::string truncateAtBoundary(const std::string &raw) return s; } +// Strip a single trailing generic noun ("class", "function", …) from a +// captured noun phrase. "has a LeaderAgent class" captures "LeaderAgent +// class"; the noun carries no identity and must not become part of the +// subject. Only a whole trailing word is removed — "error class mapping" +// keeps its tail because "mapping" is not in the list. Returns the input +// unchanged when the last word is not one of the generic nouns. +std::string stripTrailingNoun(const std::string &raw) +{ + static const char *kGenericNouns[] = { + "class", "function", "method", "module", "struct", + "interface", "type", "component", "service", "endpoint", + "route", "handler", "capability", "feature", "table", + "column", "field", "variable", "constant", "file", + "package", "library", "dependency", "test", "tests", + "support", "handler", "controller", "manager", + }; + // Split off the last word. + size_t end = raw.size(); + while (end > 0 && (raw[end - 1] == ' ' || raw[end - 1] == '\t')) + --end; + if (end == 0) + return raw; + size_t start = end; + while (start > 0 && raw[start - 1] != ' ' && raw[start - 1] != '\t') + --start; + std::string last = raw.substr(start, end - start); + // Inline lowercase compare (toLower is defined further down). + std::string last_lower; + last_lower.reserve(last.size()); + for (char c : last) + last_lower.push_back(static_cast( + std::tolower(static_cast(c)))); + for (const char *noun : kGenericNouns) { + if (last_lower == noun) { + std::string head = raw.substr(0, start); + while (!head.empty() && + (head.back() == ' ' || head.back() == '\t')) + head.pop_back(); + return head; + } + } + return raw; +} + // Convert an ASCII string to lowercase (locale-independent) for // whitelist comparison. std::string toLower(const std::string &s) @@ -121,6 +165,8 @@ bool isLayerName(const std::string &token) // boundary-truncation helper is applied before stamping the subject): // 1. supports? -> CapabilityExists(supported_by) // 2. implements? -> CapabilityExists(implemented_by) +// 2b. has/contains -> CapabilityExists(has) +// 2c. should -> FunctionImplements(should) // 3. thread-safe -> ContractHolds(ThreadSafe) // 4. memory-safe -> ContractHolds(MemorySafe) // 5. zero-copy -> ContractHolds(ZeroCopy) @@ -197,6 +243,113 @@ std::vector ClaimParser::parse(const std::string &text, e.what()); } + // ── Pattern 2b: "has / contains " ─────────────────── + // Existence claims ("The project has a LeaderAgent class"). The + // article is optional and the trailing generic noun is stripped so + // the subject is the identity token, not the prose. Negation + // ("has no X", "doesn't have X") must NOT become a positive claim. + try { + static const std::regex kHas( + "\\b(?:has|contains?)[[:space:]]+(?:an?[[:space:]]+|the[[:space:]]+)?" + "([A-Za-z][A-Za-z0-9_\\- ]{1,50})", + std::regex::icase); + std::sregex_iterator it(text.begin(), text.end(), kHas); + std::sregex_iterator end; + for (; it != end; ++it) { + std::string before = text.substr( + 0, static_cast((*it).position())); + std::string raw = (*it)[1].str(); + // "has no X" / "has not X" / "has n't" / "never has X" + // are denials — skip them rather than flip polarity. + static const std::regex kNegation( + "(\\bnot|isn't|is not|never|n't)\\s*$", + std::regex::icase); + std::smatch neg; + if (std::regex_search(before, neg, kNegation)) + continue; + std::string lower_raw = raw; + for (char &c : lower_raw) + c = static_cast(std::tolower( + static_cast(c))); + if (lower_raw.rfind("no ", 0) == 0 || + lower_raw.rfind("not ", 0) == 0 || + lower_raw.rfind("never ", 0) == 0) + continue; + std::string subject = toPascalCase( + stripTrailingNoun(truncateAtBoundary(raw))); + if (subject.empty()) + continue; + Claim c; + c.type = ClaimType::CapabilityExists; + c.subject = subject; + c.predicate = "has"; + c.object = ""; + c.scope = "repository"; + c.source_kind = source_kind; + c.source_ref = source_ref; + claims.push_back(std::move(c)); + } + } catch (const std::regex_error &e) { + fprintf(stderr, + "ClaimParser: has regex error: %s " + "[module=verify, method=parse]\n", + e.what()); + } + + // ── Pattern 2c: " should " ──────────────── + // Behavioral claims ("c_print should handle null input"). The + // subject is a function/method name and is kept VERBATIM — + // PascalCase-mangling would break the entity lookup in + // FunctionImplementsVerifier. The verb phrase goes to `object`. + try { + static const std::regex kShould( + "\\b([A-Za-z_][A-Za-z0-9_]{0,60})[[:space:]]+should[[:space:]]+" + "([A-Za-z][A-Za-z0-9_\\- ,]{0,80})", + std::regex::icase); + std::sregex_iterator it(text.begin(), text.end(), kShould); + std::sregex_iterator end; + for (; it != end; ++it) { + std::string before = text.substr( + 0, static_cast((*it).position())); + std::string subject = (*it)[1].str(); + std::string behavior = + truncateAtBoundary((*it)[2].str()); + // "should never X" / "should not X" are obligations + // about absence — not positive FunctionImplements + // claims. Reject the match instead of flipping it. + static const std::regex kNegation( + "(\\bnot|isn't|is not|never|n't)\\s*$", + std::regex::icase); + std::smatch neg; + if (std::regex_search(before, neg, kNegation)) + continue; + std::string lower_beh = behavior; + for (char &c : lower_beh) + c = static_cast(std::tolower( + static_cast(c))); + if (lower_beh.rfind("not ", 0) == 0 || + lower_beh.rfind("never ", 0) == 0 || + lower_beh.rfind("n't", 0) == 0) + continue; + if (subject.empty() || behavior.empty()) + continue; + Claim c; + c.type = ClaimType::FunctionImplements; + c.subject = subject; + c.predicate = "should"; + c.object = behavior; + c.scope = "repository"; + c.source_kind = source_kind; + c.source_ref = source_ref; + claims.push_back(std::move(c)); + } + } catch (const std::regex_error &e) { + fprintf(stderr, + "ClaimParser: should regex error: %s " + "[module=verify, method=parse]\n", + e.what()); + } + // ── Patterns 3-6: keyword contracts ────────────────────────── // Each keyword contract ("thread-safe", "memory-safe", "zero-copy", // "lock-free") emits a ContractHolds claim with a canonical subject @@ -215,7 +368,25 @@ std::vector ClaimParser::parse(const std::string &text, for (const auto &kc : kKeywords) { try { std::regex re(kc.pattern, std::regex::icase); - if (std::regex_search(text, re)) { + std::smatch m; + if (std::regex_search(text, m, re)) { + // Negation guard: "not thread-safe" / "isn't + // thread-safe" / "never thread-safe" must NOT + // become a positive ContractHolds claim — the + // polarity of the source sentence is inverted + // otherwise and ContractVerifier can answer + // Supported on the README's denial. + // Anchored to the word end so "cannot" (which + // ends in "not") is not mistaken for a denial. + static const std::regex kNegation( + "(\\bnot|isn't|is not|never|n't)\\s*$", + std::regex::icase); + std::smatch neg; + std::string before = text.substr( + 0, static_cast(m.position())); + if (std::regex_search(before, neg, kNegation)) { + continue; + } Claim c; c.type = ClaimType::ContractHolds; c.subject = kc.subject; diff --git a/engine/src/verify/contract_verifier.cpp b/engine/src/verify/contract_verifier.cpp index f22c73f..b3c460b 100644 --- a/engine/src/verify/contract_verifier.cpp +++ b/engine/src/verify/contract_verifier.cpp @@ -15,7 +15,9 @@ static constexpr double kConfZeroCopySupported = 0.6; static constexpr double kConfZeroCopyNotFound = 0.3; static constexpr double kConfUnrecognisedContract = 0.3; static constexpr double kConfThreadSafeSupported = 0.7; -static constexpr double kConfThreadSafeContradicted = 0.6; +// Absence of a matching synchronisation name is NOT evidence that the code is +// unsafe, so the negative outcome is Unknown (see verifyThreadSafe). +static constexpr double kConfThreadSafeUnknown = 0.4; static constexpr double kConfMemorySafeSupported = 0.6; static constexpr double kConfMemorySafeNotFound = 0.4; static constexpr double kConfBackendNotReady = 0.2; @@ -102,13 +104,19 @@ entitiesMatchingAny(store::GraphStore *store, uint64_t project_id, if (patterns.empty()) return ids; - // Build "LOWER(name) LIKE LOWER(?) OR LOWER(name) LIKE LOWER(?) ..." - // dynamically. The number of ? placeholders equals patterns.size(). + // Build "LOWER(name) LIKE LOWER(?) ESCAPE '\' OR ..." dynamically. The + // number of ? placeholders equals patterns.size(). + // + // `%` stays a wildcard (callers intend it), but `_` is escaped to a + // literal underscore: a code identifier contains real underscores, while + // LIKE would otherwise treat `_` as "any single character" and silently + // over-match — `%_lock` matched `Block` ("B" + "lock"), which is how a + // class named `Block` was read as synchronisation evidence. std::string sql = "SELECT id FROM entity WHERE project_id=? AND ("; for (size_t i = 0; i < patterns.size(); ++i) { if (i > 0) sql += " OR "; - sql += "LOWER(name) LIKE LOWER(?)"; + sql += "LOWER(name) LIKE LOWER(?) ESCAPE '\\'"; } sql += ")"; @@ -123,9 +131,19 @@ entitiesMatchingAny(store::GraphStore *store, uint64_t project_id, } sqlite3_bind_int64(stmt, 1, static_cast(project_id)); for (size_t i = 0; i < patterns.size(); ++i) { + // Escape the escape character itself first, then `_`, so a pattern + // reads as "names containing this text" with `%` as the only + // wildcard. `%` is deliberately left alone. + std::string escaped; + escaped.reserve(patterns[i].size()); + for (char c : patterns[i]) { + if (c == '\\' || c == '_') + escaped.push_back('\\'); + escaped.push_back(c); + } // +2 because parameter 1 is project_id; patterns start at 2. sqlite3_bind_text(stmt, static_cast(i + 2), - patterns[i].c_str(), -1, SQLITE_STATIC); + escaped.c_str(), -1, SQLITE_TRANSIENT); } while (sqlite3_step(stmt) == SQLITE_ROW) { @@ -200,9 +218,13 @@ EvidenceRecord ContractVerifier::verify(const Claim &claim) } if (subject == "zerocopy") { // ZeroCopy: search for view/span/slice entities as evidence of - // non-owning reference types. Falls into verifyGeneric. + // non-owning reference types. The `view` patterns are anchored to + // the start of the name or to a separator: an unanchored `%view%` + // also matches Review / Preview / Overview, which reported a + // project as zero-copy on the strength of a class name. std::vector ids = entitiesMatchingAny( - store_, project_id_, { "%view%", "%span%", "%slice%" }); + store_, project_id_, + { "%span%", "%slice%", "view%", "%.view", "%_view" }); if (!ids.empty()) { return makeRecord(Verdict::Supported, kConfZeroCopySupported, @@ -225,21 +247,42 @@ EvidenceRecord ContractVerifier::verify(const Claim &claim) // ── Contract-specific helpers ──────────────────────────────────────── // ThreadSafe: the codebase should reference synchronisation primitives -// (mutex, lock, atomic). Their presence is supporting evidence; their -// absence strongly contradicts a thread-safety claim. +// (mutex, atomic, lock guards). Their presence is supporting evidence. +// +// The patterns are deliberately narrow. An unanchored `%lock%` also matches +// Block, BlockStore, Clock, Deadlock and Unlock, so a project that merely +// contains a `Block` class used to be reported as thread-safe. +// +// Absence of a match is NOT evidence of absence: code can be thread-safe +// through means these names do not cover (Rust `std::sync`, Java +// `synchronized`, an OS lock created in C, a message-passing design). The +// negative outcome is therefore Unknown, matching verifyMemorySafe — asserting +// "thread safe: contradicted" from a failed name pattern would be a claim +// about the code that the evidence does not support. +static const std::vector kSyncEvidencePatterns = { + "%mutex%", "%atomic%", "%spinlock%", + "%rwlock%", "%semaphore%", "%condition_variable%", + "%.lock", "%.unlock", "%.rlock", + "%.wlock", "%_lock", "%_unlock", + "%lock_guard%", "%lockguard%", "%unique_lock%", + "%shared_lock%", "%uniquelock%", "%sharedlock%", +}; + EvidenceRecord ContractVerifier::verifyThreadSafe(const Claim &claim) { - std::vector ids = entitiesMatchingAny( - store_, project_id_, { "%mutex%", "%lock%", "%atomic%" }); + std::vector ids = + entitiesMatchingAny(store_, project_id_, kSyncEvidencePatterns); if (!ids.empty()) { return makeRecord( Verdict::Supported, kConfThreadSafeSupported, - "ThreadSafe: found mutex/lock/atomic entities", ids); + "ThreadSafe: found mutex/atomic/lock entities", ids); } - return makeRecord(Verdict::Contradicted, kConfThreadSafeContradicted, - "No mutex/lock/atomic found despite ThreadSafe " - "claim", - {}); + return makeRecord( + Verdict::Unknown, kConfThreadSafeUnknown, + "ThreadSafe: no synchronisation primitive matched " + "(mutex/atomic/lock guard); a name pattern not matching is not " + "evidence that the code is unsafe", + {}); } // MemorySafe: the codebase should use memory-management primitives diff --git a/engine/src/verify/dead_code_inspector.cpp b/engine/src/verify/dead_code_inspector.cpp index 0eede90..14f6870 100644 --- a/engine/src/verify/dead_code_inspector.cpp +++ b/engine/src/verify/dead_code_inspector.cpp @@ -1,5 +1,6 @@ #include "dead_code_inspector.h" #include "claim.h" +#include "registry.h" #include "../store/store.h" #include @@ -22,32 +23,91 @@ DeadCodeInspector::DeadCodeInspector(store::GraphStore *store, std::vector DeadCodeInspector::findOrphanModules() { std::vector out; + + // Evidence gate. The orphan test below is `NOT EXISTS (SELECT 1 FROM + // import …)`, which is vacuously TRUE when the import table is empty: a + // project that was never indexed, or a language whose visitor records no + // imports, would have every module with >=10 entities reported as an + // orphan module — a hard conclusion drawn from missing evidence. Unlike + // the shared evidence_backend_ready() gate this also has to check + // `import`, because entity rows can exist while imports do not. + auto count_for_project = [&](const char *sql) -> int64_t { + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(store_->handle(), sql, -1, &st, + nullptr) != SQLITE_OK) { + fprintf(stderr, + "[module=verify, method=findOrphanModules] " + "prepare failed: %s\n", + sqlite3_errmsg(store_->handle())); + return -1; + } + sqlite3_bind_int64(st, 1, static_cast(project_id_)); + int64_t n = (sqlite3_step(st) == SQLITE_ROW) ? + sqlite3_column_int64(st, 0) : + -1; + sqlite3_finalize(st); + return n; + }; + const int64_t entity_rows = count_for_project( + "SELECT COUNT(*) FROM entity WHERE project_id=?"); + const int64_t import_rows = count_for_project( + "SELECT COUNT(*) FROM import WHERE project_id=?"); + if (entity_rows <= 0 || import_rows <= 0) { + fprintf(stderr, + "[module=verify, method=findOrphanModules] evidence " + "backend not ready (entity=%lld, import=%lld): " + "orphan-module conclusions suppressed\n", + (long long)entity_rows, (long long)import_rows); + return out; + } + // Find modules that are never imported from outside the module itself. // Uses the import table, filtered by file_path to exclude self-imports. // A module is orphaned if no external file imports it. // This matches the manual audit methodology (grep for import paths, // excluding self-references). - std::string sql = "SELECT s.name, COUNT(e.id) as entities, " - " MIN(e.file_path) as sample_file " - "FROM scope s " - "JOIN entity e ON e.project_id = s.project_id " - " AND e.file_path LIKE s.name || '%' " - "WHERE s.kind = 1 AND s.project_id = ? " - " AND NOT EXISTS (" - " SELECT 1 FROM import i " - " WHERE i.project_id = ? " - " AND i.target_path LIKE '%' || " - " substr(s.name, length(s.name) - " - " instr(reverse(s.name), '/') + 2) || '%'" - " AND i.file_path NOT LIKE s.name || '%'" - " ) " - "GROUP BY s.name " - "HAVING entities >= 10 " - "ORDER BY entities DESC LIMIT 500"; + std::string sql = + "SELECT s.name, COUNT(e.id) as entities, " + " MIN(e.file_path) as sample_file " + "FROM scope s " + "JOIN entity e ON e.project_id = s.project_id " + " AND e.file_path LIKE s.name || '%' " + "WHERE s.kind = 1 AND s.project_id = ? " + " AND NOT EXISTS (" + " SELECT 1 FROM import i " + " WHERE i.project_id = ? " + // An import counts if its target path contains the + // module's last path component ("src/foo/bar" -> "bar"). + // + // SQLite has no reverse(), which the previous expression + // called: the statement failed to prepare on EVERY call and + // the inspector silently reported "no orphan modules" for + // every project. The basename is extracted with rtrim + // instead — rtrim(X,'/') drops trailing slashes, and + // rtrim(X, replace(X,'/','')) strips trailing non-slash + // characters, so its length is the index of the last '/'. + " AND i.target_path LIKE '%' || " + " substr(rtrim(s.name, '/'), " + " length(rtrim(rtrim(s.name, '/'), " + " replace(rtrim(s.name, '/'), " + " '/', ''))) + 1) || '%'" + " AND i.file_path NOT LIKE s.name || '%'" + " ) " + "GROUP BY s.name " + "HAVING entities >= 10 " + "ORDER BY entities DESC LIMIT 500"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(store_->handle(), sql.c_str(), -1, &stmt, - nullptr) != SQLITE_OK) + nullptr) != SQLITE_OK) { + // No silent error handling: a failed prepare must not look like + // "the inspection ran and found nothing". The SQL prefix + // identifies which of this class's queries failed. + fprintf(stderr, + "[module=verify, method=dead_code_inspector] prepare " + "failed: %s | sql=%.120s\n", + sqlite3_errmsg(store_->handle()), sql.c_str()); return out; + } sqlite3_bind_int64(stmt, 1, static_cast(project_id_)); sqlite3_bind_int64(stmt, 2, static_cast(project_id_)); @@ -60,11 +120,29 @@ std::vector DeadCodeInspector::findOrphanModules() Finding f; f.type = "DeadModule"; + // The old wording said "entities with zero callers", which this query + // never measures — it counts the module's entities and tests whether + // anything outside imports the module. Say what was checked. f.description = std::string("Module '") + (mod ? mod : "") + "' has " + std::to_string(entities) + - " entities with zero callers — orphan module. " - "Sample: " + + " entities and is imported by no other module " + "(checked via the import table). Sample: " + (sample ? sample : ""); + // A scope holding most of the project is a top-level directory, not a + // dependency: nothing outside it imports it *because* it contains + // everything. Measured on real projects — a Rust crate's `src/` + // (6747 of 6850 entities), a Python project root (847 of 847), and + // this repo's `engine/src/` (1766 of 2265) — the finding is still + // emitted (the query contract is pinned by + // test_verifier_evidence_gates Case 6), but the reader is told why + // the absence of an importer is expected there. Deleting the finding + // instead was tried and breaks that contract. + if (static_cast(entities) * 2 > entity_rows) { + f.description += + " (this scope holds most of the project's " + "entities — it is a top-level directory, so no " + "external importer is expected)"; + } f.confidence = 0.95; out.push_back(f); } @@ -75,6 +153,40 @@ std::vector DeadCodeInspector::findOrphanModules() std::vector DeadCodeInspector::findOrphanFunctions() { std::vector out; + + // Evidence gate (same class as findOrphanModules): when the relation + // table is empty the orphan test `NOT EXISTS (SELECT 1 FROM relation …)` + // is vacuously TRUE for every non-public, non-entry-point function, so + // a pre-index or call-edge-less project would emit mass DeadFunction + // findings at 0.90 confidence — hard conclusions drawn from missing + // evidence. Suppress the check when relation is empty. + { + sqlite3_stmt *st = nullptr; + if (sqlite3_prepare_v2(store_->handle(), + "SELECT COUNT(*) FROM relation WHERE " + "project_id=?", + -1, &st, nullptr) != SQLITE_OK) { + fprintf(stderr, + "[module=verify, method=findOrphanFunctions] " + "prepare relation count failed: %s\n", + sqlite3_errmsg(store_->handle())); + return out; + } + sqlite3_bind_int64(st, 1, static_cast(project_id_)); + int64_t relation_rows = (sqlite3_step(st) == SQLITE_ROW) ? + sqlite3_column_int64(st, 0) : + -1; + sqlite3_finalize(st); + if (relation_rows <= 0) { + fprintf(stderr, + "[module=verify, method=findOrphanFunctions] " + "evidence backend not ready (relation=%lld): " + "dead-function conclusions suppressed\n", + (long long)relation_rows); + return out; + } + } + // Find functions/types with 0 incoming edges and 0 outgoing edges. // Exclusions (otherwise main(), FFI exports, and callback entry // points get misclassified as dead — they have no in/out relation @@ -112,8 +224,16 @@ std::vector DeadCodeInspector::findOrphanFunctions() "LIMIT 500"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(store_->handle(), sql.c_str(), -1, &stmt, - nullptr) != SQLITE_OK) + nullptr) != SQLITE_OK) { + // No silent error handling: a failed prepare must not look like + // "the inspection ran and found nothing". The SQL prefix + // identifies which of this class's queries failed. + fprintf(stderr, + "[module=verify, method=dead_code_inspector] prepare " + "failed: %s | sql=%.120s\n", + sqlite3_errmsg(store_->handle()), sql.c_str()); return out; + } sqlite3_bind_int64(stmt, 1, static_cast(project_id_)); sqlite3_bind_int64(stmt, 2, static_cast(project_id_)); @@ -161,12 +281,30 @@ std::vector DeadCodeInspector::findArchitectureDrift() " AND tgt_mod.kind = 1 " "WHERE r.project_id = ? AND r.type = 1 " " AND src_mod.id != tgt_mod.id " + // A file under a nested scope prefix-matches BOTH the child and + // its ancestor, so an intra-module call is emitted twice and + // surfaces as "Module 'src/' calls 'src/ir/'" — a parent/child + // pair is not a boundary crossing (T5 finding #6). substr/= + // rather than LIKE so '_' in a path is a literal, not a + // single-char wildcard. + " AND NOT (substr(src_mod.name, 1, length(tgt_mod.name)) = " + "tgt_mod.name " + " OR substr(tgt_mod.name, 1, length(src_mod.name)) = " + "src_mod.name) " "GROUP BY src_mod.name, tgt_mod.name " "ORDER BY edges DESC LIMIT 15"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(store_->handle(), sql.c_str(), -1, &stmt, - nullptr) != SQLITE_OK) + nullptr) != SQLITE_OK) { + // No silent error handling: a failed prepare must not look like + // "the inspection ran and found nothing". The SQL prefix + // identifies which of this class's queries failed. + fprintf(stderr, + "[module=verify, method=dead_code_inspector] prepare " + "failed: %s | sql=%.120s\n", + sqlite3_errmsg(store_->handle()), sql.c_str()); return out; + } sqlite3_bind_int64(stmt, 1, static_cast(project_id_)); sqlite3_bind_int64(stmt, 2, static_cast(project_id_)); sqlite3_bind_int64(stmt, 3, static_cast(project_id_)); @@ -192,46 +330,76 @@ std::vector DeadCodeInspector::findArchitectureDrift() } sqlite3_finalize(stmt); - // Layer violation check: detect lower-layer modules calling upper-layer - // modules. Uses the architecture_edge table for known layer relationships. - std::string layer_sql = - "SELECT ae.layer_lower, ae.layer_upper, COUNT(*) as violations " + // Module coupling: the module pairs with the most cross-module calls. + // + // This used to be reported as a layer violation ("lower layer should not + // depend on upper layer"). There is no layer model: architecture_edge rows + // are written per (caller module, callee module) pair with no direction + // test, and callee_module / caller_module hold module NAMES. The old finding + // therefore asserted a violation on the strength of two module names. It + // is now reported for what it is — coupling, ordered by call count. + std::string coupling_sql = + "SELECT ae.caller_module, ae.callee_module, COUNT(*) as calls " "FROM architecture_edge ae " "JOIN entity e ON ae.entity_id = e.id " "JOIN relation r ON r.project_id = ? AND r.target_id = e.id " "JOIN entity caller ON r.source_id = caller.id " "WHERE ae.project_id = ? " - " AND caller.file_path LIKE '%' || ae.layer_lower || '%'" - " AND e.file_path LIKE '%' || ae.layer_upper || '%'" - " GROUP BY ae.layer_lower, ae.layer_upper" - " HAVING violations > 0" - " ORDER BY violations DESC LIMIT 10"; - sqlite3_stmt *layer_st = nullptr; - if (sqlite3_prepare_v2(store_->handle(), layer_sql.c_str(), -1, - &layer_st, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(layer_st, 1, + // A module calling itself is not a boundary crossing. The + // architecture_edge writer already skips same-scope pairs, but + // the read side must not trust that: without this predicate a + // self-pair row would be reported as "Module 'X' calls 'X' … + // across a module boundary" (T5 finding #6). + " AND ae.callee_module != ae.caller_module" + // Same for parent/child nesting: a file under `src/ir/` also + // prefix-matches `src/`, so intra-module calls surface as + // 'src/' ↔ 'src/ir/' pairs. substr/= keeps '_' literal. + " AND NOT (substr(ae.callee_module, 1, length(ae.caller_module)) " + "= ae.caller_module " + " OR substr(ae.caller_module, 1, length(ae.callee_module)) " + "= ae.callee_module)" + // The call edge runs caller -> callee (relation.target_id is the + // callee, so `caller` is the source side). Matching the caller + // against callee_module here read every pair backwards and made + // the rule report only reversed edges — once the parent/child + // filter above removed those, this rule went silent on real data. + " AND caller.file_path LIKE '%' || ae.caller_module || '%'" + " AND e.file_path LIKE '%' || ae.callee_module || '%'" + " GROUP BY ae.caller_module, ae.callee_module" + " HAVING calls > 0" + " ORDER BY calls DESC LIMIT 10"; + sqlite3_stmt *coupling_st = nullptr; + if (sqlite3_prepare_v2(store_->handle(), coupling_sql.c_str(), -1, + &coupling_st, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(coupling_st, 1, static_cast(project_id_)); - sqlite3_bind_int64(layer_st, 2, + sqlite3_bind_int64(coupling_st, 2, static_cast(project_id_)); - while (sqlite3_step(layer_st) == SQLITE_ROW) { - const char *lower = reinterpret_cast( - sqlite3_column_text(layer_st, 0)); - const char *upper = reinterpret_cast( - sqlite3_column_text(layer_st, 1)); - int violations = sqlite3_column_int(layer_st, 2); + while (sqlite3_step(coupling_st) == SQLITE_ROW) { + const char *caller_mod = reinterpret_cast( + sqlite3_column_text(coupling_st, 0)); + const char *callee_mod = reinterpret_cast( + sqlite3_column_text(coupling_st, 1)); + int calls = sqlite3_column_int(coupling_st, 2); Finding f; - f.type = "LayerViolation"; - f.description = - std::string("Layer violation: '") + - (lower ? lower : "") + "' calls '" + - (upper ? upper : "") + "' " + - std::to_string(violations) + - " times — lower layer should not depend on upper layer."; + f.type = "ModuleCoupling"; + f.description = std::string("Module coupling: '") + + (caller_mod ? caller_mod : "") + + "' calls '" + + (callee_mod ? callee_mod : "") + "' " + + std::to_string(calls) + + " times across a module boundary — a " + "dependency, not a layer violation."; f.confidence = 0.90; out.push_back(f); } - sqlite3_finalize(layer_st); + sqlite3_finalize(coupling_st); + } else { + fprintf(stderr, + "[module=verify, method=dead_code_inspector] " + "coupling prepare failed: %s\n", + sqlite3_errmsg(store_->handle())); } return out; @@ -261,8 +429,16 @@ std::vector DeadCodeInspector::findConnectedComponents() "WHERE project_id = ? AND type = 1"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(store_->handle(), sql.c_str(), -1, &stmt, - nullptr) != SQLITE_OK) + nullptr) != SQLITE_OK) { + // No silent error handling: a failed prepare must not look like + // "the inspection ran and found nothing". The SQL prefix + // identifies which of this class's queries failed. + fprintf(stderr, + "[module=verify, method=dead_code_inspector] prepare " + "failed: %s | sql=%.120s\n", + sqlite3_errmsg(store_->handle()), sql.c_str()); return out; + } sqlite3_bind_int64(stmt, 1, static_cast(project_id_)); while (sqlite3_step(stmt) == SQLITE_ROW) { uint64_t src = diff --git a/engine/src/verify/documentation_drift.cpp b/engine/src/verify/documentation_drift.cpp index 2ff346d..ab1a260 100644 --- a/engine/src/verify/documentation_drift.cpp +++ b/engine/src/verify/documentation_drift.cpp @@ -77,25 +77,260 @@ bool isWordBoundaryAfter(const std::string &text, size_t end_pos) return !std::isalnum(static_cast(next)); } -// Count how many times "go" appears as a standalone word (case-insensitive). -// This avoids matching "Google", "Going", etc. -size_t countStandaloneWord(const std::string &text, const std::string &word) +} // namespace + +// ─── Claim context ────────────────────────────────────────────── +// +// A language mention is a CLAIM ABOUT THIS PROJECT only when its context says +// so. Counting every occurrence anywhere in the README made this detector drift +// itself: a "Supported Languages" capability matrix — or a benchmark table +// listing other projects — was read as "this project is written in Python, Go, +// TypeScript, …", and each of those was then reported as documented but absent +// from the code. A drift detector that invents drift is worse than none, +// because it teaches the reader to ignore its output. +// +// Two contexts are rejected, each judged from the mention's own line, the +// header of the markdown table it sits in, and the nearest heading above it: +// +// 1. CAPABILITY — the mention is about what the tool can handle +// ("Supports C++, Python…", "| Language | Parser |"). The prose cues are +// deliberately narrow: "The cpp parser is fast" is a statement about this +// project, so a bare "parser" in prose must not veto it — but the same +// word as a COLUMN HEADER names the capability axis and does. +// 2. THIRD PARTY — the mention is a value in a table whose header has a +// `Project` column ("| Project | Language | Index Time |"), so the +// language there describes the listed projects, not this one. +// +// Everything else still counts, so a genuine claim ("Written in Rust") with no +// Rust in the tree is still reported as drift. + +struct ReadmeLine { + size_t start = 0; + size_t end = 0; + bool table = false; // markdown table row (leading '|') + int heading = -1; // nearest heading at or above this line + int table_header = -1; // header row of the table this line belongs to +}; + +/// Split the README into lines and annotate each with the context a language +/// mention on it has to be judged against. +std::vector annotateReadmeLines(const std::string &text) +{ + std::vector lines; + size_t pos = 0; + while (true) { + size_t nl = text.find('\n', pos); + if (nl == std::string::npos) + nl = text.size(); + ReadmeLine l; + l.start = pos; + l.end = nl; + size_t first = pos; + while (first < nl && + std::isspace(static_cast(text[first]))) + first++; + l.table = first < nl && text[first] == '|'; + lines.push_back(l); + if (nl >= text.size()) + break; + pos = nl + 1; + } + + int last_heading = -1; + for (size_t i = 0; i < lines.size(); i++) { + size_t first = lines[i].start; + while (first < lines[i].end && + std::isspace(static_cast(text[first]))) + first++; + if (first < lines[i].end && text[first] == '#') + last_heading = static_cast(i); + lines[i].heading = last_heading; + if (lines[i].table) { + // The header is the topmost row of the contiguous table block. + size_t top = i; + while (top > 0 && lines[top - 1].table) + --top; + lines[i].table_header = static_cast(top); + } + } + return lines; +} + +/// Line index containing `pos`, or -1. READMEs are small, so the linear scan +/// costs nothing next to the string search that produced `pos`. +int lineOfMention(const std::vector &lines, size_t pos) +{ + for (size_t i = 0; i < lines.size(); i++) + if (pos >= lines[i].start && pos <= lines[i].end) + return static_cast(i); + return -1; +} + +/// True when `haystack` contains any of `cues` (case-insensitive). +bool containsAnyCue(const std::string &haystack, const char *const *cues, + size_t cue_count) +{ + for (size_t i = 0; i < cue_count; i++) + if (findCaseInsensitive(haystack, cues[i], 0) != + std::string::npos) + return true; + return false; +} + +// Prose cues: the sentence is about what the tool does to code. "supports" +// covers "supports X" / "supported languages"; the CJK entries cover the same +// phrasing in Chinese READMEs. +const char *const kCapabilityProseCues[] = { + "supports", "supported", "supporting", "support", "can parse", "parses", + "parsed", "handles", "支持", "解析", "兼容", +}; + +// Header cues: the capability axis as a column name or section title. +const char *const kCapabilityHeaderCues[] = { + "support", "parse", "parser", "handle", "recogni", + "verified", "支持", "解析", "兼容", +}; + +// A table with a `Project` column lists OTHER projects, so its language values +// describe those, not this repository. +const char *const kThirdPartyHeaderCues[] = { "project" }; + +// Handling cues: the surrounding paragraph documents what the indexer DOES to +// files of that language — skips, excludes, rejects, or special-cases them. +// Scanned over the whole paragraph, not the single line, because such notes +// wrap: this repository's own "→ For Java: the Layer-1 names (test, docs, +// samples, …) are also top-only" bullet only says "skipped" on the line above. +// +// The verb forms are listed rather than the bare stem "parse" so that a +// sentence about this project's own "cpp parser" is still a claim — "parse" is +// a substring of "parser", "parses"/"parsing" are not. +const char *const kHandlingCues[] = { + "skip", "exclude", "filter", "reject", "top-only", "handled", + "handles", "namespace", "convention", "parses", "parsed", "parsing", + "can parse", "支持", "解析", "兼容", +}; + +/// The contiguous non-blank block of lines containing `idx` — a markdown +/// paragraph or bullet group, which is how per-language handling notes appear. +std::string paragraphAround(const std::string &text, + const std::vector &lines, size_t idx) +{ + auto blank = [&](size_t i) { + for (size_t p = lines[i].start; p < lines[i].end; p++) + if (!std::isspace(static_cast(text[p]))) + return false; + return true; + }; + size_t lo = idx; + size_t hi = idx; + while (lo > 0 && !blank(lo - 1)) + lo--; + while (hi + 1 < lines.size() && !blank(hi + 1)) + hi++; + std::string out; + for (size_t i = lo; i <= hi; i++) { + out += text.substr(lines[i].start, + lines[i].end - lines[i].start); + out += '\n'; + } + return out; +} + +/// True when the mention at `pos` is a capability statement or a fact about +/// another project — i.e. not a claim about this one. +bool isNonProjectContext(const std::string &text, + const std::vector &lines, size_t pos) +{ + const int idx = lineOfMention(lines, pos); + if (idx < 0) + return false; + const ReadmeLine &l = lines[static_cast(idx)]; + + const std::string line = text.substr(l.start, l.end - l.start); + if (containsAnyCue(line, kCapabilityProseCues, + sizeof(kCapabilityProseCues) / + sizeof(kCapabilityProseCues[0]))) + return true; + if (containsAnyCue(paragraphAround(text, lines, + static_cast(idx)), + kHandlingCues, + sizeof(kHandlingCues) / sizeof(kHandlingCues[0]))) + return true; + + const size_t header_cues = sizeof(kCapabilityHeaderCues) / + sizeof(kCapabilityHeaderCues[0]); + const size_t third_party_cues = sizeof(kThirdPartyHeaderCues) / + sizeof(kThirdPartyHeaderCues[0]); + + if (l.heading >= 0) { + const ReadmeLine &h = lines[static_cast(l.heading)]; + if (containsAnyCue(text.substr(h.start, h.end - h.start), + kCapabilityHeaderCues, header_cues)) + return true; + } + if (l.table_header >= 0) { + const ReadmeLine &t = + lines[static_cast(l.table_header)]; + const std::string header = + text.substr(t.start, t.end - t.start); + if (containsAnyCue(header, kCapabilityHeaderCues, + header_cues) || + containsAnyCue(header, kThirdPartyHeaderCues, + third_party_cues)) + return true; + } + return false; +} + +/// How a mention is matched in the README text. +enum class MatchMode { + Substring, // case-insensitive substring ("c++", "python", ...) + Word, // case-insensitive standalone word ("Go", "Java") + WordCaseSensitive, // standalone case-sensitive word (bare "C") +}; + +/// Count the mentions of `needle` that are claims about this project. +size_t countClaimMentions(const std::string &text, + const std::vector &lines, + const std::string &needle, MatchMode mode) { size_t count = 0; + if (mode == MatchMode::Substring) { + size_t pos = 0; + while ((pos = findCaseInsensitive(text, needle, pos)) != + std::string::npos) { + if (!isNonProjectContext(text, lines, pos)) + count++; + pos += needle.size(); + } + return count; + } + const bool case_sensitive = mode == MatchMode::WordCaseSensitive; size_t pos = 0; - while ((pos = findCaseInsensitive(text, word, pos)) != - std::string::npos) { - size_t abs_end = pos + word.size(); - if (isWordBoundary(text, pos) && - isWordBoundaryAfter(text, abs_end)) + while (pos < text.size()) { + size_t found = case_sensitive ? + text.find(needle, pos) : + findCaseInsensitive(text, needle, pos); + if (found == std::string::npos) + break; + pos = found; + const size_t abs_end = pos + needle.size(); + // "C++" / "C#" are their own spellings, not a bare-"C" mention. + // isWordBoundaryAfter() only tests word characters and '+' is not one, + // so without this the same "C++" occurrence was counted twice — once + // by the "c++" pattern and once by the bare-"C" rule. + const bool spelling_suffix = + abs_end < text.size() && + (text[abs_end] == '+' || text[abs_end] == '#'); + if (!spelling_suffix && isWordBoundary(text, pos) && + isWordBoundaryAfter(text, abs_end) && + !isNonProjectContext(text, lines, pos)) count++; pos = abs_end; } return count; } -} // namespace - std::vector extractLanguageClaims(const std::string &readme_text) { std::vector result; @@ -143,15 +378,12 @@ std::vector extractLanguageClaims(const std::string &readme_text) } } + const std::vector lines = annotateReadmeLines(masked); + for (const auto &pat : kLanguagePatterns) { - std::string pattern(pat.pattern); - size_t count = 0; - size_t pos = 0; - while ((pos = findCaseInsensitive(masked, pattern, pos)) != - std::string::npos) { - count++; - pos += pattern.size(); - } + const std::string pattern(pat.pattern); + const size_t count = countClaimMentions(masked, lines, pattern, + MatchMode::Substring); if (count == 0) continue; @@ -177,7 +409,8 @@ std::vector extractLanguageClaims(const std::string &readme_text) // Special handling for "Go" as a standalone word — not part of // kLanguagePatterns because "go" is too common as a substring. { - size_t go_count = countStandaloneWord(masked, "go"); + const size_t go_count = countClaimMentions(masked, lines, "go", + MatchMode::Word); if (go_count > 0) { // Check if "go" is already claimed via "golang" bool already = false; @@ -201,7 +434,8 @@ std::vector extractLanguageClaims(const std::string &readme_text) // Special handling for "Java" — use word-boundary matching to avoid // false positives on "JavaScript" which contains "Java" as a substring. { - size_t java_count = countStandaloneWord(masked, "java"); + const size_t java_count = countClaimMentions( + masked, lines, "java", MatchMode::Word); if (java_count > 0) { bool already = false; for (auto &c : result) { @@ -229,16 +463,8 @@ std::vector extractLanguageClaims(const std::string &readme_text) // consistent with the existing "c language" → "cpp" rule and with // countEntitiesByLanguage's c/cpp equivalence. { - size_t c_count = 0; - size_t pos = 0; - const std::string needle = "C"; - while ((pos = masked.find(needle, pos)) != std::string::npos) { - size_t abs_end = pos + needle.size(); - if (isWordBoundary(masked, pos) && - isWordBoundaryAfter(masked, abs_end)) - c_count++; - pos = abs_end; - } + const size_t c_count = countClaimMentions( + masked, lines, "C", MatchMode::WordCaseSensitive); if (c_count > 0) { bool already = false; for (auto &c : result) { @@ -360,6 +586,43 @@ std::vector detectDocumentationDrift(store::GraphStore &store, if (claims.empty()) return drifts; + // Evidence gate: countEntitiesByLanguage() returns 0 when nothing has been + // indexed, which would mark every language named in the README as missing. + // + // Readiness here is "the entity table has rows", NOT the shared + // evidence_backend_ready() (which also demands a relation row): this check + // reads only `entity`, and a project whose languages are all single-file + // has no resolvable call edges, so requiring relations would suppress the + // check for exactly the projects it still applies to. + int64_t entity_rows = 0; + { + const char *count_sql = + "SELECT COUNT(*) FROM entity WHERE project_id=?"; + sqlite3_stmt *count_stmt = nullptr; + if (sqlite3_prepare_v2(db, count_sql, -1, &count_stmt, + nullptr) == SQLITE_OK) { + sqlite3_bind_int64(count_stmt, 1, + static_cast(project_id)); + if (sqlite3_step(count_stmt) == SQLITE_ROW) + entity_rows = + sqlite3_column_int64(count_stmt, 0); + sqlite3_finalize(count_stmt); + } else { + fprintf(stderr, + "[module=verify, method=detectDocumentationDrift] " + "entity count prepare failed: %s\n", + sqlite3_errmsg(db)); + } + } + if (entity_rows <= 0) { + fprintf(stderr, + "[module=verify, method=detectDocumentationDrift] " + "evidence backend not ready (entity=%lld): no drift " + "conclusions reported\n", + (long long)entity_rows); + return drifts; + } + // Step 3: Cross-reference each claimed language with the entity table. for (const auto &claim : claims) { int64_t entity_count = countEntitiesByLanguage( diff --git a/engine/src/verify/ffi_internal.h b/engine/src/verify/ffi_internal.h index b9fdf63..6acb901 100644 --- a/engine/src/verify/ffi_internal.h +++ b/engine/src/verify/ffi_internal.h @@ -53,8 +53,9 @@ struct VerifyResult { // // THREAD SAFETY: single-threaded only (relies on the GraphStore // single-writer invariant documented in store.h). The caller must hold -// the global g_store singleton. -VerifyResult verify_one_claim(uint64_t project_id, const verify::Claim &claim); +// the engine store singleton. +VerifyResult verify_one_claim(EngineContext *ctx, uint64_t project_id, + const verify::Claim &claim); // BatchResult bundles the output of verify_claim_batch so callers can // wrap it in their own JSON envelope (summary, reality, review, etc.) @@ -68,6 +69,21 @@ struct BatchResult { int contradicted = 0; int unknown = 0; std::string results_json; // "[,...]" + + /// How much of the input was actually judgeable, so a caller can tell + /// "nothing contradicted us" apart from "nothing was checkable": + /// * `"no_claims"` — the text carried no recognisable claim pattern + /// (ClaimParser is deliberately conservative), so there is nothing to + /// score; + /// * `"no_verdicts"` — claims were parsed but every verdict was Unknown; + /// * `"verified"` — at least one claim was decided. + /// + /// The wrapped tools used to disagree on this case: engine_verify_summary + /// answered `trust_score: 1.0` for text with no claims ("nothing to + /// dispute") while engine_verify_review answered `0.0` for the same + /// vacuous input — one number reads as "fully trusted", the other as + /// "worthless", and neither means anything. Both now carry this status. + const char *status = "no_claims"; }; // Parse claims from `text`, verify each via verify_one_claim, and return @@ -81,8 +97,8 @@ struct BatchResult { // @param source_ref Short reference string for the evidence table. // // THREAD SAFETY: single-threaded only (same invariant as verify_one_claim). -BatchResult verify_claim_batch(uint64_t project_id, const std::string &text, - const char *source_kind, +BatchResult verify_claim_batch(EngineContext *ctx, uint64_t project_id, + const std::string &text, const char *source_kind, const std::string &source_ref); } // namespace verify_ffi diff --git a/engine/src/verify/function_implements_verifier.cpp b/engine/src/verify/function_implements_verifier.cpp index e3eabc2..f7ef5d4 100644 --- a/engine/src/verify/function_implements_verifier.cpp +++ b/engine/src/verify/function_implements_verifier.cpp @@ -18,8 +18,15 @@ static constexpr double kConfFunctionIsolated = 0.55; // Function exists AND is wired into the call graph — but only presence // + edges are confirmed, not that the function semantically implements // the claimed behavior (the claim's object field is not validated). -// Downgraded from Supported to PartiallyVerified with low confidence. -static constexpr double kConfFunctionPartiallyVerified = 0.55; +// +// Reported as Supported with reduced confidence, and the detail string says the +// check was structural only. The comment here used to say the verdict was +// "downgraded to PartiallyVerified" — a value the claim Verdict enum +// (claim.h: Supported / Contradicted / Unknown) does not have, so the code +// never did what the comment described. The confidence split is what separates +// "structurally plausible" (this value) from "object-linked" +// (kConfFunctionObjectLinked below). +static constexpr double kConfFunctionStructuralOnly = 0.55; static constexpr double kConfNoStore = 0.0; static constexpr double kConfBackendNotReady = 0.2; @@ -53,12 +60,19 @@ bool FunctionImplementsVerifier::accepts(const Claim &claim) const // case-insensitive exact name match (LOWER(name) = LOWER(?)). We do NOT // use LIKE wildcards here because a function-implements claim names a // specific symbol; prefix matching would over-match (e.g. "run" would -// match "runtime"). Returns ids in SQLite row order; empty when no match. +// match "runtime"). +// +// @param out_ok Set to false on prepare failure (caller must map that to +// Unknown, not Contradicted). True on success even when the +// result is empty ("function not found"). +// @return Matching entity ids in SQLite row order; empty when no match. static std::vector findFunctionEntities(store::GraphStore *store, uint64_t project_id, - const std::string &subject) + const std::string &subject, + bool &out_ok) { std::vector ids; + out_ok = true; if (!store || !store->handle() || subject.empty()) return ids; @@ -73,6 +87,7 @@ static std::vector findFunctionEntities(store::GraphStore *store, "failed: %s " "[module=verify, method=findFunctionEntities]\n", sqlite3_errmsg(store->handle())); + out_ok = false; return ids; } sqlite3_bind_int64(stmt, 1, static_cast(project_id)); @@ -278,8 +293,20 @@ EvidenceRecord FunctionImplementsVerifier::verify(const Claim &claim) } // Step 1: the subject function must exist as an entity. - std::vector fn_ids = - findFunctionEntities(store_, project_id_, claim.subject); + // A query failure is Unknown, not Contradicted (code_rules: no hard + // conclusion from a failed query). + bool query_ok = true; + std::vector fn_ids = findFunctionEntities( + store_, project_id_, claim.subject, query_ok); + if (!query_ok) { + rec.verdict = Verdict::Unknown; + rec.confidence = kConfBackendNotReady; + rec.detail = + "FunctionImplementsVerifier: function lookup failed " + "[module=verify, " + "method=FunctionImplementsVerifier::verify]"; + return rec; + } if (fn_ids.empty()) { rec.verdict = Verdict::Contradicted; rec.confidence = kConfFunctionNotFound; @@ -313,7 +340,7 @@ EvidenceRecord FunctionImplementsVerifier::verify(const Claim &claim) // object-linked call edges and thus stays at structural confidence, // so callers can tell "structurally plausible" from "object-linked". rec.verdict = Verdict::Supported; - rec.confidence = kConfFunctionPartiallyVerified; + rec.confidence = kConfFunctionStructuralOnly; rec.detail = "Function '" + claim.subject + "' exists and participates in the call graph " "(structural check only); semantic implementation of '" + diff --git a/engine/src/verify/intent.h b/engine/src/verify/intent.h deleted file mode 100644 index f93d2d9..0000000 --- a/engine/src/verify/intent.h +++ /dev/null @@ -1,66 +0,0 @@ -#ifndef CODESCOPE_VERIFY_INTENT_H -#define CODESCOPE_VERIFY_INTENT_H - -#include -#include - -namespace verify::planner -{ - -// ─── Intent + Evidence Requirements (VP1) ───────────────────────── -// -// Intent is the structured representation of a natural-language -// verification question. The IntentParser turns free-form claim text -// into an Intent; the Planner uses the Intent's requirements to -// decide which evidence rules to execute; the VerdictBuilder combines -// the executed evidence back against the Intent to produce a Verdict. -// -// These types live in the `verify::planner` sub-namespace to avoid a -// name collision with the existing `verify::Verdict` enum in claim.h -// (which has only 3 values and a stable 0/1/2 DB mapping). The -// planner's Verdict adds PartiallyVerified for the case where some -// requirements are satisfied but not enough to Support. - -/// One evidence requirement inside an Intent. `id` is a human-readable -/// label (e.g. "MemoryOwnership", "PatternMatch"). `rule_names` lists -/// the evidence::Rule names that, if matched, contribute to satisfying -/// this requirement. `weight` is the relative weight of this -/// requirement in the overall verdict aggregation (0.0..1.0). -struct EvidenceRequirement { - std::string id; - std::vector rule_names; - double weight = 1.0; -}; - -/// Intent is the structured representation of a natural-language -/// verification question. `type` categorizes the question -/// ("safety_question", "pattern_question", "capability_question", -/// or "unknown" when no keyword matched). `subject` is the topic of -/// the question (e.g. "CString"). `raw_claim` is the original text. -/// `requirements` lists the EvidenceRequirement values that, if -/// satisfied, support the claim. -struct Intent { - std::string type; - std::string subject; - std::string raw_claim; - std::vector requirements; -}; - -/// Verdict is the outcome of evidence-based verification. Extends the -/// 3-value verify::Verdict in claim.h with PartiallyVerified for the -/// case where some requirements are satisfied but the weighted ratio -/// falls between 0.2 and 0.8. -enum class Verdict { - Supported, - Contradicted, - PartiallyVerified, - Unknown, -}; - -/// Human-readable name for a Verdict (e.g. "Supported"). -/// Used by the FFI layer to serialize the verdict into JSON. -std::string verdictToString(Verdict v); - -} // namespace verify::planner - -#endif // CODESCOPE_VERIFY_INTENT_H diff --git a/engine/src/verify/intent_parser.cpp b/engine/src/verify/intent_parser.cpp deleted file mode 100644 index 2e85408..0000000 --- a/engine/src/verify/intent_parser.cpp +++ /dev/null @@ -1,279 +0,0 @@ -// intent_parser.cpp — IntentParser implementation (VP1). -// -// Turns a free-form natural-language claim into a structured Intent -// via case-insensitive keyword matching. Each keyword maps to a fixed -// Intent type with a predefined set of EvidenceRequirement values. -// -// The matching is intentionally simple (substring search on a -// lowercased copy of the input). This keeps the parser deterministic -// and easy to extend: adding a new keyword is one entry in the -// dispatch table below. An LLM-based parser can be plugged in later -// by implementing the same IntentParser interface. -// -// Unmatched claims produce an Intent with type="unknown" and empty -// requirements, which the VerdictBuilder turns into Verdict::Unknown. -// The unmatched text is logged to stderr with the -// [module=verify, method=IntentParser::parse] trace chain so callers -// can diagnose why a claim produced no verdict. - -#include "intent_parser.h" - -#include -#include -#include -#include - -namespace verify::planner -{ - -// ─── Local helpers ─────────────────────────────────────────────── - -namespace -{ - -// Convert an ASCII string to lowercase (locale-independent). Used for -// case-insensitive substring matching against the keyword table. -std::string toLower(const std::string &s) -{ - std::string out; - out.reserve(s.size()); - for (char c : s) { - out.push_back(static_cast( - std::tolower(static_cast(c)))); - } - return out; -} - -// Case-insensitive substring search. Returns true if `haystack` -// contains `needle` (both compared lowercased). -bool containsCI(const std::string &haystack, const std::string &needle) -{ - if (needle.empty()) - return true; - if (haystack.size() < needle.size()) - return false; - std::string h = toLower(haystack); - std::string n = toLower(needle); - return h.find(n) != std::string::npos; -} - -// Named weights for the CString safety question. Centralized here so -// the Intent construction below reads as data, not magic numbers. -constexpr double kWeightMemoryOwnership = 0.5; -constexpr double kWeightFfiBoundary = 0.3; -constexpr double kWeightLifetime = 0.2; - -constexpr double kWeightPatternMatch = 1.0; - -constexpr double kWeightCapabilityExistence = 0.4; -constexpr double kWeightImplementation = 0.4; -constexpr double kWeightWorkflowCompleteness = 0.2; - -constexpr double kWeightThreadSafety = 1.0; - -// Build the 3-requirement Intent for the "safely handle CString" -// safety question. MemoryOwnership (cstring_leak + malloc_no_free) -// carries the highest weight; FFIBoundary (extern_call + cgo_callback) -// and Lifetime (cstring_alloc_vs_free) are secondary signals. -Intent buildCStringSafetyIntent(const std::string &raw_claim) -{ - Intent intent; - intent.type = "safety_question"; - intent.subject = "CString"; - intent.raw_claim = raw_claim; - - EvidenceRequirement mem; - mem.id = "MemoryOwnership"; - mem.rule_names = { "cstring_leak", "malloc_no_free" }; - mem.weight = kWeightMemoryOwnership; - intent.requirements.push_back(std::move(mem)); - - EvidenceRequirement ffi; - ffi.id = "FFIBoundary"; - ffi.rule_names = { "extern_call", "cgo_callback" }; - ffi.weight = kWeightFfiBoundary; - intent.requirements.push_back(std::move(ffi)); - - EvidenceRequirement life; - life.id = "Lifetime"; - life.rule_names = { "cstring_alloc_vs_free" }; - life.weight = kWeightLifetime; - intent.requirements.push_back(std::move(life)); - - return intent; -} - -// Build the single-requirement Intent for a bare-except pattern -// question. The bare_except_collect rule covers all bare except -// clauses that swallow exceptions. -Intent buildBareExceptIntent(const std::string &raw_claim) -{ - Intent intent; - intent.type = "pattern_question"; - intent.subject = "bare_except"; - intent.raw_claim = raw_claim; - - EvidenceRequirement req; - req.id = "PatternMatch"; - req.rule_names = { "bare_except_collect" }; - req.weight = kWeightPatternMatch; - intent.requirements.push_back(std::move(req)); - - return intent; -} - -// Build the 3-requirement Intent for a JWT capability question. -// CapabilityExistence checks for the declared capability, -// Implementation checks for JWT entities in code, and -// WorkflowCompleteness checks for the end-to-end login workflow. -Intent buildJwtCapabilityIntent(const std::string &raw_claim) -{ - Intent intent; - intent.type = "capability_question"; - intent.subject = "JWT"; - intent.raw_claim = raw_claim; - - EvidenceRequirement cap; - cap.id = "CapabilityExistence"; - cap.rule_names = { "capability_declared" }; - cap.weight = kWeightCapabilityExistence; - intent.requirements.push_back(std::move(cap)); - - EvidenceRequirement impl; - impl.id = "Implementation"; - impl.rule_names = { "jwt_entities" }; - impl.weight = kWeightImplementation; - intent.requirements.push_back(std::move(impl)); - - EvidenceRequirement wf; - wf.id = "WorkflowCompleteness"; - wf.rule_names = { "workflow_complete" }; - wf.weight = kWeightWorkflowCompleteness; - intent.requirements.push_back(std::move(wf)); - - return intent; -} - -// Build the single-requirement Intent for a CString leak pattern -// question. The cstring_leak rule detects C.CString allocations -// without matching C.free in the same function. -Intent buildCStringLeakIntent(const std::string &raw_claim) -{ - Intent intent; - intent.type = "pattern_question"; - intent.subject = "CString_leak"; - intent.raw_claim = raw_claim; - - EvidenceRequirement req; - req.id = "PatternMatch"; - req.rule_names = { "cstring_leak" }; - req.weight = kWeightPatternMatch; - intent.requirements.push_back(std::move(req)); - - return intent; -} - -// Build the single-requirement Intent for a thread-safety question. -// The mutex_without_defer_unlock rule detects mutex locks without a -// matching defer Unlock. -Intent buildThreadSafetyIntent(const std::string &raw_claim) -{ - Intent intent; - intent.type = "safety_question"; - intent.subject = "thread_safety"; - intent.raw_claim = raw_claim; - - EvidenceRequirement req; - req.id = "ThreadSafety"; - req.rule_names = { "mutex_without_defer_unlock" }; - req.weight = kWeightThreadSafety; - intent.requirements.push_back(std::move(req)); - - return intent; -} - -// Build the default Unknown Intent for unmatched claims. The empty -// requirements vector causes the VerdictBuilder to return -// Verdict::Unknown. -Intent buildUnknownIntent(const std::string &raw_claim) -{ - Intent intent; - intent.type = "unknown"; - intent.subject = ""; - intent.raw_claim = raw_claim; - return intent; -} - -} // namespace - -// ─── IntentParser public API ───────────────────────────────────── - -Intent IntentParser::parse(const std::string &claim_text) const -{ - if (claim_text.empty()) { - return buildUnknownIntent(""); - } - - // Rule 1: "safely handle CString" / "safely handles CString" - // Covers both "safely handle" and "safely handles" forms. - if (containsCI(claim_text, "safely handle") && - containsCI(claim_text, "cstring")) { - return buildCStringSafetyIntent(claim_text); - } - if (containsCI(claim_text, "safely handles") && - containsCI(claim_text, "cstring")) { - return buildCStringSafetyIntent(claim_text); - } - - // Rule 2: "bare except" - if (containsCI(claim_text, "bare except")) { - return buildBareExceptIntent(claim_text); - } - - // Rule 3: "supports JWT" / "login module supports" - if (containsCI(claim_text, "supports jwt") || - containsCI(claim_text, "login module supports")) { - return buildJwtCapabilityIntent(claim_text); - } - - // Rule 4: "CString leak" / "has a leak" - if ((containsCI(claim_text, "cstring") && - containsCI(claim_text, "leak")) || - containsCI(claim_text, "has a leak")) { - return buildCStringLeakIntent(claim_text); - } - - // Rule 5: "mutex without defer" / "thread safe" / "thread-safe" - if (containsCI(claim_text, "mutex without defer") || - containsCI(claim_text, "thread safe") || - containsCI(claim_text, "thread-safe")) { - return buildThreadSafetyIntent(claim_text); - } - - // Default: unmatched claim. Log to stderr so callers can - // diagnose why a claim produced no verdict. - std::fprintf(stderr, - "[module=verify, method=IntentParser::parse] " - "unmatched claim: %s\n", - claim_text.c_str()); - return buildUnknownIntent(claim_text); -} - -// ─── verdictToString ───────────────────────────────────────────── - -std::string verdictToString(Verdict v) -{ - switch (v) { - case Verdict::Supported: - return "Supported"; - case Verdict::Contradicted: - return "Contradicted"; - case Verdict::PartiallyVerified: - return "PartiallyVerified"; - case Verdict::Unknown: - return "Unknown"; - } - return "Unknown"; -} - -} // namespace verify::planner diff --git a/engine/src/verify/intent_parser.h b/engine/src/verify/intent_parser.h deleted file mode 100644 index cd620c2..0000000 --- a/engine/src/verify/intent_parser.h +++ /dev/null @@ -1,63 +0,0 @@ -#ifndef CODESCOPE_VERIFY_INTENT_PARSER_H -#define CODESCOPE_VERIFY_INTENT_PARSER_H - -#include - -#include "intent.h" - -namespace verify::planner -{ - -// ─── IntentParser (VP1) ────────────────────────────────────────── -// -// IntentParser turns a free-form natural-language claim into a -// structured Intent. The parser is keyword-based and case-insensitive: -// it matches high-confidence phrases like "safely handle CString" or -// "bare except" to a fixed set of Intent types, each with a predefined -// set of EvidenceRequirement values. -// -// The parser is conservative: text that does not match any keyword -// produces an Intent with type="unknown" and empty requirements, which -// the VerdictBuilder will turn into a Verdict::Unknown. This is always -// a legal outcome — better to return Unknown than a wrong verdict. -// -// Matching rules (evaluated in order, first match wins): -// 1. "safely handle CString" / "safely handles CString" -// → type="safety_question", requirements: -// - MemoryOwnership(cstring_leak, malloc_no_free, w=0.5) -// - FFIBoundary(extern_call, cgo_callback, w=0.3) -// - Lifetime(cstring_alloc_vs_free, w=0.2) -// 2. "bare except" -// → type="pattern_question", requirements: -// - PatternMatch(bare_except_collect, w=1.0) -// 3. "supports JWT" / "login module supports" -// → type="capability_question", requirements: -// - CapabilityExistence(capability_declared, w=0.4) -// - Implementation(jwt_entities, w=0.4) -// - WorkflowCompleteness(workflow_complete, w=0.2) -// 4. "CString leak" / "has a leak" -// → type="pattern_question", requirements: -// - PatternMatch(cstring_leak, w=1.0) -// 5. "mutex without defer" / "thread safe" / "thread-safe" -// → type="safety_question", requirements: -// - ThreadSafety(mutex_without_defer_unlock, w=1.0) -// 6. Default: type="unknown", requirements=[] (Unknown verdict) -// -// Unmatched claims are logged to stderr with the -// [module=verify, method=IntentParser::parse] trace chain so callers -// can diagnose why a claim produced no verdict. - -class IntentParser { - public: - /// Parse `claim_text` into an Intent. Matching is case-insensitive - /// and keyword-based. Unmatched text produces an Intent with - /// type="unknown" and empty requirements. - /// @param claim_text The natural-language claim to parse. - /// @return An Intent describing the claim's type, subject, and - /// evidence requirements. - Intent parse(const std::string &claim_text) const; -}; - -} // namespace verify::planner - -#endif // CODESCOPE_VERIFY_INTENT_PARSER_H diff --git a/engine/src/verify/planner.cpp b/engine/src/verify/planner.cpp deleted file mode 100644 index 2b31bfb..0000000 --- a/engine/src/verify/planner.cpp +++ /dev/null @@ -1,97 +0,0 @@ -// planner.cpp — Planner implementation (VP2). -// -// Planner turns an Intent into a Plan (list of PlanSteps), then -// executes the Plan by calling EvidenceBuilder::buildByRule for each -// step's rule_name. The resulting Evidence values are aggregated into -// a single vector and returned to the caller (the FFI layer or the -// VerdictBuilder). -// -// The execute() method loads rule files from the directory pointed to -// by CODESCOPE_RULES_DIR (falling back to "engine/src/evidence/rules" -// relative to CWD) on each call. This makes the Planner self-contained -// for tests and matches the FFI layer's behaviour. The store_ pointer -// is borrowed from the caller and must outlive the Planner. - -#include "planner.h" - -#include -#include -#include - -#include "../store/store.h" - -namespace verify::planner -{ - -// ─── Local helpers ─────────────────────────────────────────────── - -namespace -{ - -// Default rules directory relative to CWD. Mirrors the FFI layer's -// fallback so the Planner works identically when called from tests -// or from the FFI. -constexpr const char *kDefaultRulesDir = "engine/src/evidence/rules"; - -// Resolve the rules directory from CODESCOPE_RULES_DIR (if set and -// non-empty), falling back to kDefaultRulesDir. Returns a std::string -// so the caller can pass it to EvidenceBuilder::loadRules without -// lifetime concerns. -std::string resolveRulesDir() -{ - const char *env_dir = std::getenv("CODESCOPE_RULES_DIR"); - if (env_dir && *env_dir) { - return env_dir; - } - return kDefaultRulesDir; -} - -} // namespace - -// ─── Planner public API ────────────────────────────────────────── - -Plan Planner::plan(const Intent &intent) const -{ - Plan out; - for (const auto &req : intent.requirements) { - for (const auto &rule_name : req.rule_names) { - if (rule_name.empty()) { - continue; - } - PlanStep step; - step.rule_name = rule_name; - // category is left empty; EvidenceBuilder fills it - // in when the rule runs. - step.category = ""; - out.steps.push_back(std::move(step)); - } - } - return out; -} - -std::vector Planner::execute(const Plan &plan, - uint64_t project_id) const -{ - std::vector result; - if (!store_) { - std::fprintf(stderr, "[module=verify::planner, method=execute] " - "store is null\n"); - return result; - } - // Build a fresh EvidenceBuilder on each call so the Planner is - // stateless across invocations. loadRules is required before - // buildByRule can produce any output. - evidence::EvidenceBuilder builder(store_); - std::string rules_dir = resolveRulesDir(); - builder.loadRules(rules_dir); - - for (const auto &step : plan.steps) { - auto evs = builder.buildByRule(project_id, step.rule_name); - for (auto &ev : evs) { - result.push_back(std::move(ev)); - } - } - return result; -} - -} // namespace verify::planner diff --git a/engine/src/verify/planner.h b/engine/src/verify/planner.h deleted file mode 100644 index f075d4c..0000000 --- a/engine/src/verify/planner.h +++ /dev/null @@ -1,80 +0,0 @@ -#ifndef CODESCOPE_VERIFY_PLANNER_H -#define CODESCOPE_VERIFY_PLANNER_H - -#include -#include -#include - -#include "intent.h" -#include "../evidence/evidence_builder.h" - -namespace store -{ -class GraphStore; -} // namespace store - -namespace verify::planner -{ - -// ─── Planner (VP2) ─────────────────────────────────────────────── -// -// Planner turns an Intent into an executable Plan, then runs the Plan -// against the project's semantic_fact rows via the EvidenceBuilder. -// -// A Plan is a flat list of PlanSteps, one per rule_name mentioned in -// the Intent's requirements. Duplicate rule_names are preserved (the -// same rule may be needed by multiple requirements). The Planner does -// NOT deduplicate because the cost of running the same rule twice is -// small and the resulting Evidence vector is aggregated linearly. -// -// The store pointer is borrowed; the caller owns it and must keep it -// alive for the lifetime of the Planner. - -/// One step in a Plan: execute the named rule and collect its Evidence. -struct PlanStep { - std::string rule_name; - std::string category; -}; - -/// A Plan is a list of PlanSteps. The Planner produces a Plan from an -/// Intent; the execute() method runs the Plan against the project. -struct Plan { - std::vector steps; -}; - -class Planner { - public: - /// Construct with the store handle used for read-only SELECTs - /// against semantic_fact. The pointer is borrowed; the caller - /// owns it and must keep it alive for the lifetime of the - /// Planner. - explicit Planner(store::GraphStore *store) - : store_(store) - { - } - - /// Build a Plan from an Intent. Collects all rule_names from - /// all EvidenceRequirements into PlanSteps. The `category` - /// field of each PlanStep is left empty (the EvidenceBuilder - /// fills in the category when the rule runs). - /// @param intent The Intent to plan for. - /// @return A Plan with one PlanStep per rule_name. - Plan plan(const Intent &intent) const; - - /// Execute a Plan against the project's semantic_fact rows. - /// For each PlanStep, calls EvidenceBuilder::buildByRule to - /// produce 0+ Evidence values, then aggregates them into a - /// single vector. The caller owns the returned vector. - /// @param plan The Plan to execute. - /// @param project_id The project to query. - /// @return Aggregated vector of Evidence values from all steps. - std::vector execute(const Plan &plan, - uint64_t project_id) const; - - private: - store::GraphStore *store_; -}; - -} // namespace verify::planner - -#endif // CODESCOPE_VERIFY_PLANNER_H diff --git a/engine/src/verify/registry.cpp b/engine/src/verify/registry.cpp index 46e81c6..ad2e60f 100644 --- a/engine/src/verify/registry.cpp +++ b/engine/src/verify/registry.cpp @@ -50,10 +50,10 @@ void VerifierRegistry::register_default_verifiers(store::GraphStore *store, // // This replaces the old `static bool initialized` flag in // engine_verify_ffi.cpp. The flag was the root cause of the lifecycle bug -// (A15): engine_shutdown() cleared the registry but the flag stayed true, +// (A15): engine_destroy() cleared the registry but the flag stayed true, // so the next ensureVerifiersRegistered() was a no-op and the registry // stayed empty. Checking the actual registry state makes the function -// symmetric with engine_shutdown()'s clear() — repeatable any number of +// symmetric with engine_destroy()'s clear() — repeatable any number of // times. void VerifierRegistry::ensureDefaultVerifiers(store::GraphStore *store, uint64_t project_id) diff --git a/engine/src/verify/registry.h b/engine/src/verify/registry.h index 08d373d..1f5683f 100644 --- a/engine/src/verify/registry.h +++ b/engine/src/verify/registry.h @@ -35,9 +35,9 @@ namespace verify * Lifecycle contract (Step 9.1): * - `ensureDefaultVerifiers(store, pid)` is IDEMPOTENT: it checks the * actual registry state and only re-registers when empty. It does NOT - * rely on any process-level static flag, so `engine_shutdown()` -> - * `engine_init()` -> `ensureDefaultVerifiers()` always restores a - * healthy registry. `engine_shutdown()` calls `clear()` so the next + * rely on any process-level static flag, so `engine_destroy()` -> + * `engine_create()` -> `ensureDefaultVerifiers()` always restores a + * healthy registry. `engine_destroy()` calls `clear()` so the next * `ensureDefaultVerifiers()` re-populates from scratch. * - The sentinel verifiers registered here use nullptr/0 for store/pid * because their `accepts()` only inspects `claim.type`. The actual @@ -67,7 +67,7 @@ class VerifierRegistry { /// Idempotent registration of the default sentinel verifiers. /// If the registry already has verifiers registered, this is a no-op. /// This replaces the old `static bool initialized` flag and fixes the - /// lifecycle bug where `engine_shutdown()` cleared the registry but + /// lifecycle bug where `engine_destroy()` cleared the registry but /// the flag stayed true, leaving the registry empty after re-init. /// The sentinels use nullptr/0 because accepts() only reads claim.type. /// @param store GraphStore handle (unused by sentinels; kept for diff --git a/engine/src/verify/verdict_builder.cpp b/engine/src/verify/verdict_builder.cpp deleted file mode 100644 index c4ccac0..0000000 --- a/engine/src/verify/verdict_builder.cpp +++ /dev/null @@ -1,321 +0,0 @@ -// verdict_builder.cpp — VerdictBuilder implementation (VP3). -// -// Combines an Intent with the Evidence values produced by the Planner -// to produce a VerdictResult. The matching is heuristic (see header -// for details); the aggregation is a weighted sum of satisfied -// requirement weights. The raw_json field is pre-serialized so the -// FFI layer can return it directly without re-serialization. - -#include "verdict_builder.h" - -#include -#include -#include -#include -#include -#include - -namespace verify::planner -{ - -// ─── Local helpers ─────────────────────────────────────────────── - -namespace -{ - -// Verdict thresholds. Centralized here so they can be tuned in one -// place. The spec mandates 0.8 for Supported and 0.2 for Contradicted. -constexpr double kSupportedThreshold = 0.8; -constexpr double kContradictedThreshold = 0.2; - -// Minimum length for an underscore-separated rule_name part to be -// considered meaningful for substring matching. Parts shorter than -// this (e.g. "vs", "no") are skipped to reduce false positives. -constexpr size_t kMinPartLength = 3; - -// Convert an ASCII string to lowercase (locale-independent). -std::string toLower(const std::string &s) -{ - std::string out; - out.reserve(s.size()); - for (char c : s) { - out.push_back(static_cast( - std::tolower(static_cast(c)))); - } - return out; -} - -// Case-insensitive substring search. Returns true if `haystack` -// contains `needle` (both compared lowercased). -bool containsCI(const std::string &haystack, const std::string &needle) -{ - if (needle.empty()) - return true; - if (haystack.size() < needle.size()) - return false; - std::string h = toLower(haystack); - std::string n = toLower(needle); - return h.find(n) != std::string::npos; -} - -// Split a rule_name on underscores into lowercase parts. Used for -// heuristic matching against evidence text fields. -std::vector splitRuleNameParts(const std::string &rule_name) -{ - std::vector parts; - std::string current; - for (char c : rule_name) { - if (c == '_') { - if (!current.empty()) { - parts.push_back(toLower(current)); - current.clear(); - } - } else { - current += c; - } - } - if (!current.empty()) { - parts.push_back(toLower(current)); - } - return parts; -} - -// Check if a rule_name matches an Evidence value. The match is -// heuristic: the rule_name itself (case-insensitive) or any of its -// underscore-separated parts (length > kMinPartLength) must appear as -// a substring in the Evidence's category, title, or any item's -// category / primitive / kind / symbol fields. -bool evidenceMatchesRule(const evidence::Evidence &ev, - const std::string &rule_name) -{ - // Direct rule_name substring match (case-insensitive) on - // category and title. - if (containsCI(ev.category, rule_name) || - containsCI(ev.title, rule_name)) { - return true; - } - // Split rule_name on underscores and check each significant - // part against category, title, and item fields. - auto parts = splitRuleNameParts(rule_name); - for (const auto &part : parts) { - if (part.size() <= kMinPartLength) { - continue; - } - if (containsCI(ev.category, part) || - containsCI(ev.title, part)) { - return true; - } - for (const auto &item : ev.items) { - if (containsCI(item.category, part) || - containsCI(item.primitive, part) || - containsCI(item.kind, part) || - containsCI(item.symbol, part)) { - return true; - } - } - } - return false; -} - -// JSON-escape a string for inclusion in a JSON string literal. -// Mirrors the jsonEscape helper in engine_internal.h but kept local -// to avoid pulling that header's full set of includes. -std::string escapeJson(const std::string &s) -{ - std::string out; - out.reserve(s.size() + 8); - for (char c : s) { - switch (c) { - case '"': - out += "\\\""; - break; - case '\\': - out += "\\\\"; - break; - case '\n': - out += "\\n"; - break; - case '\r': - out += "\\r"; - break; - case '\t': - out += "\\t"; - break; - default: - if (static_cast(c) < 0x20) { - char buf[8]; - std::snprintf(buf, sizeof(buf), "\\u%04x", - static_cast(c)); - out += buf; - } else { - out += c; - } - } - } - return out; -} - -// Per-requirement state used during aggregation and JSON serialization. -struct ReqState { - std::string id; - double weight = 0.0; - bool satisfied = false; - double confidence = 0.0; -}; - -// Serialize the VerdictResult to a JSON string. The structure is: -// {"verdict":"...","confidence":N, -// "requirements":[{"id":"...","weight":N,"satisfied":bool, -// "confidence":N},...], -// "evidence":[{"category":"...","title":"...", -// "confidence":N,"item_count":N},...]} -std::string serializeResult(const VerdictResult &result, - const std::vector &req_states, - const std::vector &evidences) -{ - std::ostringstream ss; - ss << "{\"verdict\":\"" << escapeJson(verdictToString(result.verdict)) - << "\"" - << ",\"confidence\":" << result.confidence << ",\"requirements\":["; - for (size_t i = 0; i < req_states.size(); ++i) { - if (i) - ss << ","; - const auto &r = req_states[i]; - ss << "{\"id\":\"" << escapeJson(r.id) << "\"" - << ",\"weight\":" << r.weight - << ",\"satisfied\":" << (r.satisfied ? "true" : "false") - << ",\"confidence\":" << r.confidence << "}"; - } - ss << "],\"evidence\":["; - bool first = true; - for (const auto &ev : evidences) { - if (ev.items.empty()) { - continue; - } - if (!first) - ss << ","; - first = false; - ss << "{\"category\":\"" << escapeJson(ev.category) << "\"" - << ",\"title\":\"" << escapeJson(ev.title) << "\"" - << ",\"confidence\":" << ev.confidence - << ",\"item_count\":" << ev.items.size() << "}"; - } - ss << "]}"; - return ss.str(); -} - -} // namespace - -// ─── VerdictBuilder public API ─────────────────────────────────── - -VerdictResult -VerdictBuilder::build(const Intent &intent, - const std::vector &evidences) const -{ - VerdictResult result; - - // If the Intent has no requirements (type="unknown"), the - // verdict is Unknown regardless of evidence. - if (intent.requirements.empty()) { - result.verdict = Verdict::Unknown; - result.confidence = 0.0; - // Still emit any evidence we received so the caller can - // inspect what was found. - for (const auto &ev : evidences) { - if (ev.items.empty()) { - continue; - } - std::string summary = ev.title + " (" + - std::to_string(ev.items.size()) + - " item(s))"; - result.evidence_summary.push_back(std::move(summary)); - } - std::vector empty_states; - result.raw_json = - serializeResult(result, empty_states, evidences); - return result; - } - - double total_weight = 0.0; - double weighted_sum = 0.0; - int satisfied_count = 0; - - std::vector req_states; - req_states.reserve(intent.requirements.size()); - - for (const auto &req : intent.requirements) { - ReqState st; - st.id = req.id; - st.weight = req.weight; - total_weight += req.weight; - - // Find the best matching evidence (highest confidence - // among matches with items.size() > 0). - double best_conf = 0.0; - bool matched = false; - for (const auto &ev : evidences) { - if (ev.items.empty()) { - continue; - } - bool rule_matches = false; - for (const auto &rule_name : req.rule_names) { - if (evidenceMatchesRule(ev, rule_name)) { - rule_matches = true; - break; - } - } - if (!rule_matches) { - continue; - } - matched = true; - if (ev.confidence > best_conf) { - best_conf = ev.confidence; - } - } - - if (matched) { - st.satisfied = true; - st.confidence = best_conf; - weighted_sum += req.weight * best_conf; - ++satisfied_count; - } - req_states.push_back(std::move(st)); - } - - // Verdict rules. - if (satisfied_count == 0) { - result.verdict = Verdict::Unknown; - result.confidence = 0.0; - } else if (total_weight <= 0.0) { - // Defensive: zero total weight with non-empty requirements - // is a degenerate Intent. Treat as Unknown. - result.verdict = Verdict::Unknown; - result.confidence = 0.0; - } else { - double ratio = weighted_sum / total_weight; - result.confidence = ratio; - if (ratio >= kSupportedThreshold) { - result.verdict = Verdict::Supported; - } else if (ratio <= kContradictedThreshold) { - result.verdict = Verdict::Contradicted; - } else { - result.verdict = Verdict::PartiallyVerified; - } - } - - // Evidence summary: one string per Evidence (title + item count). - for (const auto &ev : evidences) { - if (ev.items.empty()) { - continue; - } - std::string summary = ev.title + " (" + - std::to_string(ev.items.size()) + - " item(s))"; - result.evidence_summary.push_back(std::move(summary)); - } - - result.raw_json = serializeResult(result, req_states, evidences); - return result; -} - -} // namespace verify::planner diff --git a/engine/src/verify/verdict_builder.h b/engine/src/verify/verdict_builder.h deleted file mode 100644 index fbec738..0000000 --- a/engine/src/verify/verdict_builder.h +++ /dev/null @@ -1,83 +0,0 @@ -#ifndef CODESCOPE_VERIFY_VERDICT_BUILDER_H -#define CODESCOPE_VERIFY_VERDICT_BUILDER_H - -#include -#include - -#include "intent.h" -#include "../evidence/evidence_builder.h" - -namespace verify::planner -{ - -// ─── VerdictBuilder (VP3) ──────────────────────────────────────── -// -// VerdictBuilder combines an Intent with the Evidence values produced -// by the Planner to produce a VerdictResult. The VerdictResult -// includes the final Verdict, an overall confidence, a list of -// human-readable evidence summaries, and a raw JSON blob for the FFI -// layer to return to the MCP server. -// -// Matching heuristic: -// For each EvidenceRequirement, the builder checks whether any of -// its rule_names matches any of the supplied Evidence values. A -// match is detected when the rule_name (or any underscore-separated -// part of it with length > 2) appears as a case-insensitive -// substring in the Evidence's category, title, or any item's -// category / primitive / kind / symbol fields. This heuristic -// handles the common case where rule names like "cstring_leak" map -// to evidence titles like "1 function(s) leak C.CString ...". -// -// Aggregation: -// - A requirement is "satisfied" if at least one matching Evidence -// has items.size() > 0. The contribution to the weighted sum is -// weight * (matching evidence's confidence, default 1.0). -// - total_weight = sum of all requirement weights. -// - weighted_sum = sum of (weight * confidence) for satisfied -// requirements. -// - ratio = weighted_sum / total_weight. -// -// Verdict rules: -// - If NO requirements are satisfied → Unknown (regardless of ratio) -// - If ratio >= 0.8 → Supported -// - If ratio <= 0.2 → Contradicted -// - Otherwise → PartiallyVerified - -/// The result of evidence-based verification for one Intent. -struct VerdictResult { - /// The final verdict (Supported / Contradicted / - /// PartiallyVerified / Unknown). - Verdict verdict = Verdict::Unknown; - /// Overall confidence in the verdict: the weighted ratio of - /// satisfied requirement weights to total weights (0.0..1.0). - double confidence = 0.0; - /// Human-readable summaries of each contributing Evidence - /// (title + item count). Empty when no Evidence was matched. - std::vector evidence_summary; - /// Pre-serialized JSON blob with the structure: - /// {"verdict":"...","confidence":N, - /// "requirements":[{"id":"...","weight":N,"satisfied":bool, - /// "confidence":N},...], - /// "evidence":[{"category":"...","title":"...", - /// "confidence":N,"item_count":N},...]} - std::string raw_json; -}; - -class VerdictBuilder { - public: - /// Combine an Intent with the Evidence values produced by the - /// Planner to produce a VerdictResult. The Intent's - /// requirements drive the matching; the Evidence values are - /// matched against each requirement's rule_names. - /// @param intent The Intent whose requirements drive matching. - /// @param evidences The Evidence values produced by the Planner. - /// @return A VerdictResult with the final verdict, confidence, - /// summaries, and raw JSON blob. - VerdictResult - build(const Intent &intent, - const std::vector &evidences) const; -}; - -} // namespace verify::planner - -#endif // CODESCOPE_VERIFY_VERDICT_BUILDER_H diff --git a/engine/tests/accuracy/fixtures/cpp/a.cpp b/engine/tests/accuracy/fixtures/cpp/a.cpp index da881b9..a3715fa 100644 --- a/engine/tests/accuracy/fixtures/cpp/a.cpp +++ b/engine/tests/accuracy/fixtures/cpp/a.cpp @@ -11,25 +11,36 @@ #include struct Point { - int x; - int y; - int adder(Point &a) { return a.x; } - static int create() { return 0; } - int helper() { return this->adder(*this); } + int x; + int y; + int adder(Point &a) + { + return a.x; + } + static int create() + { + return 0; + } + int helper() + { + return this->adder(*this); + } }; // alpha calls bravo (cross-file, bare name). -int alpha(int x) { - return bravo(x); +int alpha(int x) +{ + return bravo(x); } // mainFunc calls alpha (intra-file), std::sort (stdlib), // p.adder() (method call on instance), Point::create() (static method), // and p.helper() (which internally calls this->adder()). -int mainFunc() { - int r = alpha(1); - std::vector v = {3, 1, 2}; - std::sort(v.begin(), v.end()); - Point p{1, 2}; - return p.adder(p) + Point::create() + p.helper(); +int mainFunc() +{ + int r = alpha(1); + std::vector v = { 3, 1, 2 }; + std::sort(v.begin(), v.end()); + Point p{ 1, 2 }; + return p.adder(p) + Point::create() + p.helper(); } diff --git a/engine/tests/accuracy/fixtures/cpp/b.cpp b/engine/tests/accuracy/fixtures/cpp/b.cpp index 331740f..1790ff6 100644 --- a/engine/tests/accuracy/fixtures/cpp/b.cpp +++ b/engine/tests/accuracy/fixtures/cpp/b.cpp @@ -1,10 +1,12 @@ // Second C++ fixture file: cross-file callee + homonym. -int bravo(int x) { - return x; +int bravo(int x) +{ + return x; } -int helper() { - // Same-name function in b.cpp; a.cpp does NOT call it. - return 42; +int helper() +{ + // Same-name function in b.cpp; a.cpp does NOT call it. + return 42; } diff --git a/engine/tests/accuracy/fixtures/java/A.java b/engine/tests/accuracy/fixtures/java/A.java index fa47416..e0ebdf9 100644 --- a/engine/tests/accuracy/fixtures/java/A.java +++ b/engine/tests/accuracy/fixtures/java/A.java @@ -6,8 +6,9 @@ public static int alpha(int x) { return B.bravo(x); } - // mainFunc calls alpha (intra-file), Math.max (stdlib), and - // new B() (constructor). + // mainFunc calls alpha (intra-file), Math.max (stdlib), + // new B() (cross-file constructor), and b.run() (cross-file + // instance method with receiver_type=B). public static void mainFunc() { int r = alpha(1); int m = Math.max(1, 2); diff --git a/engine/tests/accuracy/fixtures/java/ground_truth.json b/engine/tests/accuracy/fixtures/java/ground_truth.json index 4826483..50fd6e7 100644 --- a/engine/tests/accuracy/fixtures/java/ground_truth.json +++ b/engine/tests/accuracy/fixtures/java/ground_truth.json @@ -3,16 +3,14 @@ "expected_calls": [ {"caller": "mainFunc", "caller_file": "A.java", "callee": "alpha", "callee_file": "A.java"}, {"caller": "mainFunc", "caller_file": "A.java", "callee": "B", "callee_file": "B.java", "note": "constructor call new B()"}, - {"caller": "alpha", "caller_file": "A.java", "callee": "bravo", "callee_file": "B.java"} + {"caller": "mainFunc", "caller_file": "A.java", "callee": "run", "callee_file": "B.java", "note": "cross-file instance method b.run() with receiver_type=B"}, + {"caller": "alpha", "caller_file": "A.java", "callee": "bravo", "callee_file": "B.java", "note": "cross-file static call B.bravo() from a method body"} ], "forbidden_calls": [ {"caller": "mainFunc", "callee": "max"}, {"caller": "mainFunc", "callee": "println"}, {"caller": "mainFunc", "callee": "out"} ], - "allowed_unresolved": [ - {"caller": "mainFunc", "callee": "run", "note": "instance method via b.run(); receiver type not yet propagated"}, - {"caller": "mainFunc", "callee": "bravo", "note": "static call B.bravo; qualified target not yet propagated"} - ], + "allowed_unresolved": [], "external_calls": ["Math.max", "System.out.println"] } diff --git a/engine/tests/accuracy/fixtures/ts/ground_truth.json b/engine/tests/accuracy/fixtures/ts/ground_truth.json index 67d39fc..c15c6e2 100644 --- a/engine/tests/accuracy/fixtures/ts/ground_truth.json +++ b/engine/tests/accuracy/fixtures/ts/ground_truth.json @@ -4,14 +4,13 @@ {"caller": "mainFunc", "caller_file": "a.ts", "callee": "alpha", "callee_file": "a.ts"}, {"caller": "mainFunc", "caller_file": "a.ts", "callee": "Renderer", "callee_file": "a.ts", "note": "constructor call new Renderer()"}, {"caller": "mainFunc", "caller_file": "a.ts", "callee": "Logger", "callee_file": "a.ts", "note": "constructor call new Logger()"}, + {"caller": "mainFunc", "caller_file": "a.ts", "callee": "render", "callee_file": "a.ts", "note": "obj.render() with receiver_type=Renderer"}, + {"caller": "mainFunc", "caller_file": "a.ts", "callee": "log", "callee_file": "a.ts", "note": "logger.log() with receiver_type=Logger"}, {"caller": "alpha", "caller_file": "a.ts", "callee": "bravo", "callee_file": "b.ts"} ], "forbidden_calls": [ {"caller": "mainFunc", "callee": "max"} ], - "allowed_unresolved": [ - {"caller": "mainFunc", "callee": "render", "note": "instance method call obj.render(); receiver_type=Renderer now extracted but cross-method resolution still in progress"}, - {"caller": "mainFunc", "callee": "log", "note": "instance method call logger.log(); receiver_type=Logger now extracted; bare-name edge ignored as allowed_unresolved"} - ], + "allowed_unresolved": [], "external_calls": ["Math.max"] } diff --git a/engine/tests/grammar_loader.h b/engine/tests/grammar_loader.h new file mode 100644 index 0000000..b05c246 --- /dev/null +++ b/engine/tests/grammar_loader.h @@ -0,0 +1,55 @@ +#ifndef CODESCOPE_TESTS_GRAMMAR_LOADER_H +#define CODESCOPE_TESTS_GRAMMAR_LOADER_H + +// The grammar a test parses with, taken from the library the test already links. +// +// engine/src/codescope_grammars.h states the rule outright — "These are compiled +// into the binary — no dlopen, no .so loading" — and every test binary links +// astgraph_engine, so the language is one call away. Five tests instead dlopen()ed +// `/tree-sitter-.so` over a candidate list of `$GRAMMARS_DIR`, +// `../grammars`, `grammars`, which does not contain `engine/grammars` at all +// (three of them ended the list with a hard-coded personal path behind a +// full-width "～"). No target in this repository produces those files — +// engine/grammars/*.so are untracked leftovers on a developer machine — so the +// tests could only pass where the artefacts happened to sit next to a matching +// working directory. On CI, which runs `engine/build-tests/test_*` from the +// repository root with GRAMMARS_DIR unset, they failed on `grammar loaded`: +// test_js_visitor, test_ts_visitor, test_tsx_visitor and, through the same +// loader, test_translator_depth_guard and test_ir_edge_coverage. Asking the +// library removes the working directory, the environment and the stale +// artefact from the answer. + +#include "../src/codescope_grammars.h" + +#include +#include + +/// The statically linked grammar called `name` ("c", "cpp", "go", "java", +/// "javascript", "python", "rust", "typescript", "tsx"), or nullptr when the +/// name is not one of the vendored grammars. A nullptr is a test bug — the +/// table below is the whole set — so callers keep a CHECK on the result. +inline const TSLanguage *testGrammar(const char *name) +{ + struct Entry { + const char *name; + const TSLanguage *(*language)(); + }; + static const Entry kGrammars[] = { + { "c", tree_sitter_c }, + { "cpp", tree_sitter_cpp }, + { "go", tree_sitter_go }, + { "java", tree_sitter_java }, + { "javascript", tree_sitter_javascript }, + { "python", tree_sitter_python }, + { "rust", tree_sitter_rust }, + { "typescript", tree_sitter_typescript }, + { "tsx", tree_sitter_tsx }, + }; + for (const Entry &entry : kGrammars) { + if (std::strcmp(entry.name, name) == 0) + return entry.language(); + } + return nullptr; +} + +#endif // CODESCOPE_TESTS_GRAMMAR_LOADER_H diff --git a/engine/tests/test_accuracy_baseline.cpp b/engine/tests/test_accuracy_baseline.cpp index 21fbd71..e785d49 100644 --- a/engine/tests/test_accuracy_baseline.cpp +++ b/engine/tests/test_accuracy_baseline.cpp @@ -29,6 +29,7 @@ #include #include #include +#include "test_engine_handle.h" static void check(bool cond, const char *msg) { @@ -38,24 +39,6 @@ static void check(bool cond, const char *msg) } } -// Run a single SELECT that returns one integer column, accumulating the -// rows into `out`. Returns true on success. -static bool collectIntColumn(sqlite3 *db, const std::string &sql, - std::vector &out) -{ - sqlite3_stmt *st = nullptr; - if (sqlite3_prepare_v2(db, sql.c_str(), -1, &st, nullptr) != - SQLITE_OK) { - sqlite3_finalize(st); - return false; - } - while (sqlite3_step(st) == SQLITE_ROW) { - out.push_back(sqlite3_column_int64(st, 0)); - } - sqlite3_finalize(st); - return true; -} - // Run a SELECT that returns a single integer row. Returns -1 on miss. static int64_t scalarInt(sqlite3 *db, const std::string &sql) { @@ -110,13 +93,14 @@ int main() char db_path[] = "/tmp/test_accuracy_baseline.db"; unlink(db_path); - check(engine_init(db_path) == 0, "engine_init"); + g_engine = engine_create(db_path); + check(g_engine != nullptr, "engine_init"); uint64_t pid = - engine_create_project(proj_dir, "accuracy-baseline"); + engine_create_project(g_engine, proj_dir, "accuracy-baseline"); check(pid > 0, "create_project"); - char *idx = engine_index_project(pid, proj_dir, nullptr); + char *idx = engine_index_project(g_engine, pid, proj_dir, nullptr); check(idx != nullptr, "index_project"); check(strstr(idx, "\"ok\":true") != nullptr, "index_project ok"); engine_free_string(idx); @@ -128,43 +112,37 @@ int main() sqlite3 *db = nullptr; check(sqlite3_open(db_path, &db) == SQLITE_OK, "sqlite3_open"); - int64_t entity_count = scalarInt( - db, "SELECT COUNT(*) FROM entity WHERE project_id=" + - std::to_string(pid)); + int64_t entity_count = + scalarInt(db, "SELECT COUNT(*) FROM entity WHERE project_id=" + + std::to_string(pid)); int64_t relation_total = scalarInt( db, "SELECT COUNT(*) FROM relation WHERE project_id=" + std::to_string(pid)); int64_t relation_calls = scalarInt( - db, - "SELECT COUNT(*) FROM relation WHERE project_id=" + - std::to_string(pid) + " AND type=1"); + db, "SELECT COUNT(*) FROM relation WHERE project_id=" + + std::to_string(pid) + " AND type=1"); int64_t relation_refs = scalarInt( - db, - "SELECT COUNT(*) FROM relation WHERE project_id=" + - std::to_string(pid) + " AND type=0"); + db, "SELECT COUNT(*) FROM relation WHERE project_id=" + + std::to_string(pid) + " AND type=0"); int64_t relation_defines = scalarInt( - db, - "SELECT COUNT(*) FROM relation WHERE project_id=" + - std::to_string(pid) + " AND type=2"); + db, "SELECT COUNT(*) FROM relation WHERE project_id=" + + std::to_string(pid) + " AND type=2"); int64_t relation_contains = scalarInt( - db, - "SELECT COUNT(*) FROM relation WHERE project_id=" + - std::to_string(pid) + " AND type=3"); + db, "SELECT COUNT(*) FROM relation WHERE project_id=" + + std::to_string(pid) + " AND type=3"); int64_t relation_imports = scalarInt( - db, - "SELECT COUNT(*) FROM relation WHERE project_id=" + - std::to_string(pid) + " AND type>=4"); + db, "SELECT COUNT(*) FROM relation WHERE project_id=" + + std::to_string(pid) + " AND type>=4"); // Duplicate typed relation count — the contract requires this to be // 0 once Step 1 lands the unique index. - int64_t duplicate_typed = scalarInt( - db, - "SELECT COUNT(*) FROM relation r1 WHERE EXISTS (" - " SELECT 1 FROM relation r2 WHERE " - " r2.project_id=r1.project_id AND " - " r2.source_id=r1.source_id AND " - " r2.target_id=r1.target_id AND " - " r2.type=r1.type AND r2.id +#include "test_check.h" #include #include #include @@ -39,12 +39,12 @@ static void insertEntity(store::GraphStore &store, uint64_t project_id, "start_col, end_row, end_col) " "VALUES (?,?,0,?,'',?,'cpp',0,0,0,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, file_path, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -56,11 +56,11 @@ static void insertCall(store::GraphStore &store, uint64_t project_id, const char *sql = "INSERT INTO relation (project_id, source_id, " "target_id, type) VALUES (?,?,?,1)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, source_id); sqlite3_bind_int64(stmt, 3, target_id); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -69,59 +69,60 @@ int main() unlink(kDbPath); store::GraphStore store; - assert(store.open(kDbPath)); + CHECK(store.open(kDbPath)); uint64_t project_id = store.createProject("/test", "test_arch_drift"); - assert(project_id > 0); + CHECK(project_id > 0); // ── Test 1: classifyEntityLayer — Controller by name ──────────── { - assert(classifyEntityLayer("UserController", "/src/main.cpp") == - kLayerController); - assert(classifyEntityLayer("usercontroller", "/src/main.cpp") == - kLayerController); // case-insensitive + CHECK(classifyEntityLayer("UserController", "/src/main.cpp") == + kLayerController); + CHECK(classifyEntityLayer("usercontroller", "/src/main.cpp") == + kLayerController); // case-insensitive printf(" [PASS] classifyEntityLayer: Controller by name suffix\n"); } // ── Test 2: classifyEntityLayer — Controller by path ──────────── { - assert(classifyEntityLayer("handler", "/src/controllers/user.go") == - kLayerController); - assert(classifyEntityLayer("handler", "/api/v1/routes.ts") == - kLayerController); + CHECK(classifyEntityLayer("handler", + "/src/controllers/user.go") == + kLayerController); + CHECK(classifyEntityLayer("handler", "/api/v1/routes.ts") == + kLayerController); printf(" [PASS] classifyEntityLayer: Controller by file path\n"); } // ── Test 3: classifyEntityLayer — Service ─────────────────────── { - assert(classifyEntityLayer("UserService", "/src/main.cpp") == - kLayerService); - assert(classifyEntityLayer("auth", "/services/auth.py") == - kLayerService); + CHECK(classifyEntityLayer("UserService", "/src/main.cpp") == + kLayerService); + CHECK(classifyEntityLayer("auth", "/services/auth.py") == + kLayerService); printf(" [PASS] classifyEntityLayer: Service by name/path\n"); } // ── Test 4: classifyEntityLayer — Repository variants ─────────── { - assert(classifyEntityLayer("UserRepository", "/src/main.cpp") == - kLayerRepository); - assert(classifyEntityLayer("UserRepo", "/src/main.cpp") == - kLayerRepository); - assert(classifyEntityLayer("UserStore", "/src/main.cpp") == - kLayerRepository); - assert(classifyEntityLayer("UserDAO", "/src/main.cpp") == - kLayerRepository); - assert(classifyEntityLayer("user", "/repository/user.go") == - kLayerRepository); - assert(classifyEntityLayer("user", "/data/user.go") == - kLayerRepository); + CHECK(classifyEntityLayer("UserRepository", "/src/main.cpp") == + kLayerRepository); + CHECK(classifyEntityLayer("UserRepo", "/src/main.cpp") == + kLayerRepository); + CHECK(classifyEntityLayer("UserStore", "/src/main.cpp") == + kLayerRepository); + CHECK(classifyEntityLayer("UserDAO", "/src/main.cpp") == + kLayerRepository); + CHECK(classifyEntityLayer("user", "/repository/user.go") == + kLayerRepository); + CHECK(classifyEntityLayer("user", "/data/user.go") == + kLayerRepository); printf(" [PASS] classifyEntityLayer: Repository variants\n"); } // ── Test 5: classifyEntityLayer — unclassified ────────────────── { - assert(classifyEntityLayer("helper", "/src/utils.cpp").empty()); - assert(classifyEntityLayer("main", "/src/main.go").empty()); + CHECK(classifyEntityLayer("helper", "/src/utils.cpp").empty()); + CHECK(classifyEntityLayer("main", "/src/main.go").empty()); printf(" [PASS] classifyEntityLayer: unclassified → empty\n"); } @@ -140,7 +141,7 @@ int main() insertCall(store, project_id, 10, 11); // Controller → Service auto drifts = detectArchitectureDrift(store, project_id); - assert(drifts.empty()); + CHECK(drifts.empty()); printf(" [PASS] detectArchitectureDrift: Controller→Service OK\n"); } @@ -157,13 +158,14 @@ int main() "/repository/user.go"); insertEntity(store, project_id, 21, "AdminController", "/controllers/admin.go"); - insertCall(store, project_id, 20, 21); // Repository → Controller + insertCall(store, project_id, 20, + 21); // Repository → Controller auto drifts = detectArchitectureDrift(store, project_id); - assert(drifts.size() == 1); - assert(drifts[0].type == "ArchitectureDrift"); - assert(drifts[0].severity == kDriftSeverityArch); - assert(drifts[0].subject == "Repository->Controller"); + CHECK(drifts.size() == 1); + CHECK(drifts[0].type == "ArchitectureDrift"); + CHECK(drifts[0].severity == kDriftSeverityArch); + CHECK(drifts[0].subject == "Repository->Controller"); printf(" [PASS] detectArchitectureDrift: Repository→Controller = reverse call\n"); } @@ -180,12 +182,13 @@ int main() "/controllers/user.go"); insertEntity(store, project_id, 31, "OrderController", "/controllers/order.go"); - insertCall(store, project_id, 30, 31); // Controller → Controller + insertCall(store, project_id, 30, + 31); // Controller → Controller auto drifts = detectArchitectureDrift(store, project_id); - assert(drifts.size() == 1); - assert(drifts[0].type == "ArchitectureDrift"); - assert(drifts[0].subject == "Controller->Controller"); + CHECK(drifts.size() == 1); + CHECK(drifts[0].type == "ArchitectureDrift"); + CHECK(drifts[0].subject == "Controller->Controller"); printf(" [PASS] detectArchitectureDrift: Controller→Controller = same-layer bypass\n"); } @@ -204,7 +207,7 @@ int main() insertCall(store, project_id, 40, 41); // unclassified → Service auto drifts = detectArchitectureDrift(store, project_id); - assert(drifts.empty()); + CHECK(drifts.empty()); printf(" [PASS] detectArchitectureDrift: unclassified entities skipped\n"); } @@ -222,7 +225,7 @@ int main() insertCall(store, project_id, 50, 50); // self-edge auto drifts = detectArchitectureDrift(store, project_id); - assert(drifts.empty()); + CHECK(drifts.empty()); printf(" [PASS] detectArchitectureDrift: self-edges excluded\n"); } @@ -242,7 +245,7 @@ int main() insertCall(store, project_id, 60, 61); // Service → Repository auto drifts = detectArchitectureDrift(store, project_id); - assert(drifts.empty()); + CHECK(drifts.empty()); printf(" [PASS] detectArchitectureDrift: Service→Repository OK\n"); } @@ -250,5 +253,5 @@ int main() unlink(kDbPath); printf("=== test_architecture_drift PASSED ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_bench_enhance.cpp b/engine/tests/test_bench_enhance.cpp index c7e1294..f5d3f5d 100644 --- a/engine/tests/test_bench_enhance.cpp +++ b/engine/tests/test_bench_enhance.cpp @@ -5,77 +5,100 @@ #include #include #include +#include "test_engine_handle.h" using Clock = std::chrono::steady_clock; -static double elapsed_s(Clock::time_point start) { - return std::chrono::duration(Clock::now() - start).count(); +static double elapsed_s(Clock::time_point start) +{ + return std::chrono::duration(Clock::now() - start).count(); } -int main(int argc, char** argv) { - const char* project_dir = argc > 1 ? argv[1] : "."; - const char* lang_filter = argc > 2 ? argv[2] : "typescript,tsx,javascript,go,c,cpp,rust,python,java"; +int main(int argc, char **argv) +{ + const char *project_dir = argc > 1 ? argv[1] : "."; + const char *lang_filter = + argc > 2 ? + argv[2] : + "typescript,tsx,javascript,go,c,cpp,rust,python,java"; - unlink("/tmp/bench_enhance.db"); - unlink("/tmp/bench_enhance.db-wal"); - unlink("/tmp/bench_enhance.db-shm"); + unlink("/tmp/bench_enhance.db"); + unlink("/tmp/bench_enhance.db-wal"); + unlink("/tmp/bench_enhance.db-shm"); - setenv("CODESCOPE_VERBOSE", "1", 1); + setenv("CODESCOPE_VERBOSE", "1", 1); - int rc = engine_init("/tmp/bench_enhance.db"); - if (rc != 0) { fprintf(stderr, "FAIL: engine_init\n"); return 1; } - fprintf(stderr, "engine_init OK\n"); + g_engine = engine_create("/tmp/bench_enhance.db"); + const int rc = g_engine != nullptr ? 0 : -1; + if (rc != 0) { + fprintf(stderr, "FAIL: engine_init\n"); + return 1; + } + fprintf(stderr, "engine_init OK\n"); - uint64_t pid = engine_create_project(project_dir, "bench-enhance"); - if (pid == 0) { fprintf(stderr, "FAIL: create_project\n"); return 1; } - fprintf(stderr, "create_project OK (pid=%llu)\n", (unsigned long long)pid); + uint64_t pid = + engine_create_project(g_engine, project_dir, "bench-enhance"); + if (pid == 0) { + fprintf(stderr, "FAIL: create_project\n"); + return 1; + } + fprintf(stderr, "create_project OK (pid=%llu)\n", + (unsigned long long)pid); - fprintf(stderr, "\n=== INDEX ===\n"); - auto t0 = Clock::now(); - char* result = engine_index_project(pid, project_dir, lang_filter); - double index_s = elapsed_s(t0); - fprintf(stderr, "index: %.2f s\n", index_s); - fprintf(stderr, "result: %s\n", result); + fprintf(stderr, "\n=== INDEX ===\n"); + auto t0 = Clock::now(); + char *result = + engine_index_project(g_engine, pid, project_dir, lang_filter); + double index_s = elapsed_s(t0); + fprintf(stderr, "index: %.2f s\n", index_s); + fprintf(stderr, "result: %s\n", result); - bool ok = strstr(result, "\"ok\":true") != nullptr; - engine_free_string(result); - if (!ok) { fprintf(stderr, "FAIL: index\n"); engine_shutdown(); return 1; } + bool ok = strstr(result, "\"ok\":true") != nullptr; + engine_free_string(result); + if (!ok) { + fprintf(stderr, "FAIL: index\n"); + engine_destroy(g_engine); + g_engine = nullptr; + return 1; + } - fprintf(stderr, "\n=== ENHANCE (default mode — skip_vectors) ===\n"); - t0 = Clock::now(); - char* enhance_result = engine_enhance_project(pid); - double enhance_s = elapsed_s(t0); - fprintf(stderr, "enhance: %.2f s\n", enhance_s); - fprintf(stderr, "enhance result: %s\n", enhance_result); - engine_free_string(enhance_result); + fprintf(stderr, "\n=== ENHANCE (default mode — skip_vectors) ===\n"); + t0 = Clock::now(); + char *enhance_result = engine_enhance_project(g_engine, pid); + double enhance_s = elapsed_s(t0); + fprintf(stderr, "enhance: %.2f s\n", enhance_s); + fprintf(stderr, "enhance result: %s\n", enhance_result); + engine_free_string(enhance_result); - fprintf(stderr, "\n=== ENHANCE AGAIN (idempotent — should be 0) ===\n"); - t0 = Clock::now(); - enhance_result = engine_enhance_project(pid); - double enhance2_s = elapsed_s(t0); - fprintf(stderr, "enhance2: %.2f s\n", enhance2_s); - fprintf(stderr, "enhance2 result: %s\n", enhance_result); - engine_free_string(enhance_result); + fprintf(stderr, "\n=== ENHANCE AGAIN (idempotent — should be 0) ===\n"); + t0 = Clock::now(); + enhance_result = engine_enhance_project(g_engine, pid); + double enhance2_s = elapsed_s(t0); + fprintf(stderr, "enhance2: %.2f s\n", enhance2_s); + fprintf(stderr, "enhance2 result: %s\n", enhance_result); + engine_free_string(enhance_result); - fprintf(stderr, "\n=== ENHANCE DEEP (with vectors) ===\n"); - setenv("CODESCOPE_ENHANCE_MODE", "deep", 1); - t0 = Clock::now(); - enhance_result = engine_enhance_project(pid); - double enhance_deep_s = elapsed_s(t0); - fprintf(stderr, "enhance deep: %.2f s\n", enhance_deep_s); - fprintf(stderr, "enhance deep result: %s\n", enhance_result); - engine_free_string(enhance_result); + fprintf(stderr, "\n=== ENHANCE DEEP (with vectors) ===\n"); + setenv("CODESCOPE_ENHANCE_MODE", "deep", 1); + t0 = Clock::now(); + enhance_result = engine_enhance_project(g_engine, pid); + double enhance_deep_s = elapsed_s(t0); + fprintf(stderr, "enhance deep: %.2f s\n", enhance_deep_s); + fprintf(stderr, "enhance deep result: %s\n", enhance_result); + engine_free_string(enhance_result); - char* status = engine_get_enhancement_status(pid); - fprintf(stderr, "\nstatus: %s\n", status); - engine_free_string(status); + char *status = engine_get_enhancement_status(g_engine, pid); + fprintf(stderr, "\nstatus: %s\n", status); + engine_free_string(status); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; - fprintf(stderr, "\n=== SUMMARY ===\n"); - fprintf(stderr, "index: %.2f s\n", index_s); - fprintf(stderr, "enhance: %.2f s\n", enhance_s); - fprintf(stderr, "enhance deep: %.2f s\n", enhance_deep_s); - fprintf(stderr, "total: %.2f s\n", index_s + enhance_s + enhance_deep_s); - return 0; + fprintf(stderr, "\n=== SUMMARY ===\n"); + fprintf(stderr, "index: %.2f s\n", index_s); + fprintf(stderr, "enhance: %.2f s\n", enhance_s); + fprintf(stderr, "enhance deep: %.2f s\n", enhance_deep_s); + fprintf(stderr, "total: %.2f s\n", + index_s + enhance_s + enhance_deep_s); + return 0; } diff --git a/engine/tests/test_builtin_method_calls.cpp b/engine/tests/test_builtin_method_calls.cpp new file mode 100644 index 0000000..bde2be1 --- /dev/null +++ b/engine/tests/test_builtin_method_calls.cpp @@ -0,0 +1,307 @@ +// test_builtin_method_calls.cpp — regression test for the builtin filter. +// +// Defect guarded: the Go / Java / Python / JS visitors applied their +// builtin-name filter to the LAST SEGMENT of a qualified call, i.e. to the +// method name. tree-sitter exposes `obj.map(x)` and a bare `map(x)` with +// the same method name, so every user-defined method whose name collided +// with a builtin was dropped with NO call record at all — a silent +// false negative (the Resolver can only emit edges for records that exist). +// +// The collisions were not exotic: the Java list contains `map`, `filter`, +// `forEach`, `collect`, `reduce`, `indexOf`, `replace`, `format`, +// `compareTo`, `startsWith`, `equals`, `toString`, `hashCode`, `clone`; +// Go adds `copy`, `close`, `delete`, `new`, `len`, `min`, `max`, `clear`. +// +// Rule now enforced: the builtin filter applies ONLY to unqualified calls. +// A call with a receiver / namespace cannot resolve to a pre-declared +// identifier, so it keeps its record and its receiver evidence. +// +// Second rule, added with the same fixtures: a name THIS FILE defines is user +// code whatever the builtin list says. `def format()`, `function Map()`, +// `fn write()`, `void free()` and a Java method `format()` are declared in the +// fixture itself, yet a BARE call to them was still dropped — the exemption +// (JsVisitor::defined_names_, see js_visitor_defined_names.cpp) is what keeps +// them. Each language below therefore has one locally-defined collision that +// must be EMITTED and one builtin that is NOT defined in the file and must +// stay FILTERED, so the exemption can never be mistaken for "the filter is off". +// +// Boundary cases covered: +// * Go `b.copy(...)` emitted; bare `len(x)` still filtered +// * Java `b.map(...)`, `s.indexOf(...)` emitted +// * Python `b.format(...)` emitted; bare `len(x)` still filtered +// * JS `c.map(...)` emitted; bare `parseInt(...)` still filtered +// * C `o->free(...)` still filtered — DELIBERATE (see note below). +// C's list is stdlib names (malloc/free/memcpy/printf) and +// vtable-style structs name callback fields exactly that way, +// while the resolver has no receiver-type evidence for a field +// call. This assertion pins that decision so it cannot drift. +// +// Exit code: 0 on success, 1 on any failed assertion. + +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include +#include "test_engine_handle.h" + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "FAIL: %s\n", msg); + exit(1); + } +} + +/// Number of CallExpr (kind=9) rows named `name` in files ending with +/// `file_suffix`. A built-in that is filtered produces no row at all. +static int countCalls(sqlite3 *db, uint64_t project_id, const char *name, + const char *file_suffix) +{ + const char *sql = + "SELECT COUNT(*) FROM semantic_records " + "WHERE project_id=? AND kind=9 AND name=? AND file_path LIKE ?"; + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK, + "prepare countCalls"); + sqlite3_bind_int64(st, 1, static_cast(project_id)); + sqlite3_bind_text(st, 2, name, -1, SQLITE_TRANSIENT); + std::string pattern = std::string("%") + file_suffix; + sqlite3_bind_text(st, 3, pattern.c_str(), -1, SQLITE_TRANSIENT); + int n = 0; + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + return n; +} + +static void writeFile(const std::string &path, const char *content) +{ + std::filesystem::create_directories( + std::filesystem::path(path).parent_path()); + FILE *f = fopen(path.c_str(), "w"); + check(f != nullptr, "fopen fixture"); + fputs(content, f); + fclose(f); +} + +/// One assertion: a colliding method call must keep its record. +static void expectEmitted(sqlite3 *db, uint64_t pid, const char *name, + const char *file_suffix, const char *why) +{ + if (countCalls(db, pid, name, file_suffix) == 0) { + fprintf(stderr, + "FAIL: call record for \"%s\" in *%s was dropped " + "(expected: %s)\n", + name, file_suffix, why); + exit(1); + } + printf(" [PASS] %s.%s emitted (%s)\n", file_suffix, name, why); +} + +/// One assertion: a bare built-in call must still be filtered. +static void expectFiltered(sqlite3 *db, uint64_t pid, const char *name, + const char *file_suffix, const char *why) +{ + if (countCalls(db, pid, name, file_suffix) != 0) { + fprintf(stderr, + "FAIL: bare builtin \"%s\" in *%s was NOT filtered " + "(expected: %s)\n", + name, file_suffix, why); + exit(1); + } + printf(" [PASS] %s.%s filtered (%s)\n", file_suffix, name, why); +} + +int main() +{ + const std::string root = "/tmp/codescope_builtin_method_calls"; + std::filesystem::remove_all(root); + + // Go: a method named `copy` (a pre-declared builtin) called via a + // pointer receiver, plus a bare `len(x)` inside it. + writeFile(root + "/go/buf.go", "package go\n\n" + "type Buf struct{}\n\n" + "func (b *Buf) copy(src []int) int {\n" + "\treturn len(src)\n" + "}\n\n" + "func Use(b *Buf) int {\n" + "\treturn b.copy(nil)\n" + "}\n"); + + // Java: methods named `map` and `indexOf` (both in the JDK list). + writeFile(root + "/java/Box.java", + "class Box {\n" + " void map(int x) { }\n" + " int indexOf(char c) { return 0; }\n" + "}\n" + "class BoxUser {\n" + " int run(Box b, String s) {\n" + " b.map(1);\n" + " return s.indexOf('a');\n" + " }\n" + "}\n"); + + // Python: a method named `format` plus a bare `len(x)`. + writeFile(root + "/py/mod_a.py", "class B:\n" + " def format(self):\n" + " return 1\n\n" + "def use(b):\n" + " return b.format()\n\n" + "def count(xs):\n" + " return len(xs)\n"); + + // JS: a method named `map` (Array.prototype method name) plus a + // bare `parseInt(x)` builtin. + writeFile(root + "/js/app.js", "class Ctl {\n" + " map(x) { return x; }\n" + "}\n" + "export function run() {\n" + " const c = new Ctl();\n" + " return c.map(1);\n" + "}\n" + "export function toInt(s) {\n" + " return parseInt(s, 10);\n" + "}\n"); + + // C: a vtable-style struct whose callback field is named `free`. + writeFile(root + "/c/ops.c", "struct ops {\n" + " int (*free)(void *);\n" + "};\n\n" + "int use(struct ops *o) {\n" + " return o->free(0);\n" + "}\n"); + + // Locally-defined collisions: each file declares a name that is on its + // language's builtin list and then calls it WITHOUT a receiver, which is + // exactly the case the unqualified-call rule does not cover. Each file + // also makes one bare call to a builtin it does NOT define, as the + // negative control. + writeFile(root + "/py/own.py", "def format(x):\n" + " return x\n\n\n" + "def use():\n" + " return format(1)\n\n\n" + "def count(xs):\n" + " return len(xs)\n"); + + writeFile(root + "/js/own.js", + "function Map(x) { return x; }\n\n" + "export function use() { return Map(1); }\n\n" + "export function n(s) { return parseInt(s, 10); }\n"); + + writeFile(root + "/java/Own.java", + "class Own {\n" + " int format(int x) { return x; }\n" + " int use() { return format(1); }\n" + "}\n"); + + writeFile(root + "/c/own.c", "void free(void *p) { }\n\n" + "int use_own(void) { return free(0); }\n"); + + writeFile(root + "/rs/own.rs", "fn write(x: i32) -> i32 { x }\n\n" + "fn use_it() -> i32 { write(1) }\n"); + + const char *db_path = "/tmp/test_builtin_method_calls.db"; + unlink(db_path); + g_engine = engine_create(db_path); + check(g_engine != nullptr, "engine_init"); + uint64_t pid = + engine_create_project(g_engine, root.c_str(), "builtin-method"); + check(pid > 0, "create_project"); + + char *idx = engine_index_project(g_engine, pid, root.c_str(), nullptr); + check(idx != nullptr, "index_project null"); + check(strstr(idx, "\"ok\":true") != nullptr, "index_project ok"); + engine_free_string(idx); + + sqlite3 *db = nullptr; + check(sqlite3_open(db_path, &db) == SQLITE_OK, "sqlite_open"); + + // Sanity: the fixtures really were indexed, so an empty result can + // never be mistaken for "correctly filtered". + { + const char *sql = "SELECT COUNT(*) FROM semantic_records " + "WHERE project_id=?"; + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == + SQLITE_OK, + "prepare total record count"); + sqlite3_bind_int64(st, 1, static_cast(pid)); + int total = 0; + if (sqlite3_step(st) == SQLITE_ROW) + total = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + check(total > 20, "fixtures were not indexed"); + printf(" [INFO] %d semantic records indexed\n", total); + } + + // ── Go ──────────────────────────────────────────────────────── + expectEmitted(db, pid, "copy", "buf.go", + "method named like a Go builtin must keep its record"); + expectFiltered(db, pid, "len", "buf.go", + "bare builtin call must stay filtered"); + + // ── Java ───────────────────────────────────────────────────── + expectEmitted(db, pid, "map", "Box.java", + "method named like a Stream API method must be kept"); + expectEmitted(db, pid, "indexOf", "Box.java", + "method named like a String method must be kept"); + + // ── Python ─────────────────────────────────────────────────── + expectEmitted(db, pid, "format", "mod_a.py", + "attribute call named like a builtin must be kept"); + expectFiltered(db, pid, "len", "mod_a.py", + "bare builtin call must stay filtered"); + + // ── JavaScript ─────────────────────────────────────────────── + expectEmitted(db, pid, "map", "app.js", + "member call named like a builtin method must be kept"); + expectFiltered(db, pid, "parseInt", "app.js", + "bare builtin call must stay filtered"); + + // ── C (deliberate exception) ───────────────────────────────── + // Pinned on purpose: C's builtin list is stdlib names and its + // resolver has no receiver-type evidence for a field call, so + // emitting `o->free()` would re-open the false-positive flood. + expectFiltered(db, pid, "free", "ops.c", + "C field call on a stdlib-named field stays filtered " + "(deliberate)"); + + // ── Locally-defined collisions (bare calls) ────────────────── + expectEmitted(db, pid, "format", "own.py", + "bare call to a builtin-named function the file defines " + "must be kept"); + expectFiltered(db, pid, "len", "own.py", + "bare builtin the file does NOT define stays filtered"); + + expectEmitted(db, pid, "Map", "own.js", + "bare call to `function Map()` in the same file must be " + "kept"); + expectFiltered(db, pid, "parseInt", "own.js", + "bare builtin the file does NOT define stays filtered"); + + expectEmitted( + db, pid, "format", "Own.java", + "bare call to a format() method defined in the same class " + "must be kept"); + + expectEmitted(db, pid, "free", "own.c", + "bare call to free() defined in the same file must be " + "kept (distinct from the ops.c field case above)"); + + expectEmitted(db, pid, "write", "own.rs", + "bare call to write() defined in the same file must be " + "kept"); + + sqlite3_close(db); + engine_destroy(g_engine); + g_engine = nullptr; + std::filesystem::remove_all(root); + unlink(db_path); + printf("\nAll builtin method-call tests passed.\n"); + return 0; +} diff --git a/engine/tests/test_c_e2e.cpp b/engine/tests/test_c_e2e.cpp index 8fa51af..645ffa4 100644 --- a/engine/tests/test_c_e2e.cpp +++ b/engine/tests/test_c_e2e.cpp @@ -1,7 +1,8 @@ #include "test_e2e.h" -int main() { - const char* code = R"(int add(int a, int b) { +int main() +{ + const char *code = R"(int add(int a, int b) { return a + b; } @@ -25,12 +26,8 @@ int main() { } )"; - const char* defs[] = {"add", "multiply", "compute"}; - runE2eTest("c", code, "/tmp/test_math.c", - defs, 3, - "add", "multiply", - "add", - "compute", - "main", nullptr); - return 0; + const char *defs[] = { "add", "multiply", "compute" }; + runE2eTest("c", code, "/tmp/test_math.c", defs, 3, "add", "multiply", + "add", "compute", "main", nullptr); + return 0; } diff --git a/engine/tests/test_call_graph_accuracy.cpp b/engine/tests/test_call_graph_accuracy.cpp index cf11c55..ec793b9 100644 --- a/engine/tests/test_call_graph_accuracy.cpp +++ b/engine/tests/test_call_graph_accuracy.cpp @@ -43,6 +43,7 @@ #include #include #include +#include "test_engine_handle.h" // ─── Minimal JSON parser for ground_truth.json ─────────────────────── // The ground_truth format is fixed and simple: objects with string keys @@ -454,27 +455,31 @@ static FixtureResult runFixture(const std::string &fixture_dir, bool inject_fp, r.language.c_str()); unlink(db_path); - if (engine_init(db_path) != 0) { + g_engine = engine_create(db_path); + if (!g_engine) { fprintf(stderr, "FAIL: engine_init for %s\n", r.language.c_str()); r.fn = -1; return r; } - uint64_t pid = engine_create_project(tmp_dir.c_str(), + uint64_t pid = engine_create_project(g_engine, tmp_dir.c_str(), ("acc-" + r.language).c_str()); if (pid == 0) { fprintf(stderr, "FAIL: create_project for %s\n", r.language.c_str()); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; r.fn = -1; return r; } - char *idx = engine_index_project(pid, tmp_dir.c_str(), nullptr); + char *idx = + engine_index_project(g_engine, pid, tmp_dir.c_str(), nullptr); if (!idx || !strstr(idx, "\"ok\":true")) { fprintf(stderr, "FAIL: index_project for %s: %s\n", r.language.c_str(), idx ? idx : "(null)"); engine_free_string(idx); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; r.fn = -1; return r; } @@ -488,13 +493,15 @@ static FixtureResult runFixture(const std::string &fixture_dir, bool inject_fp, if (sqlite3_open(db_path, &db) != SQLITE_OK) { fprintf(stderr, "FAIL: sqlite3_open for %s\n", r.language.c_str()); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; r.fn = -1; return r; } std::set actual = enumerateActualCalls(db, pid); sqlite3_close(db); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; // Build ground-truth sets. std::set expected = buildEdgeSet(gt, "expected_calls"); diff --git a/engine/tests/test_call_graph_method.cpp b/engine/tests/test_call_graph_method.cpp index f9dd0c0..2070a83 100644 --- a/engine/tests/test_call_graph_method.cpp +++ b/engine/tests/test_call_graph_method.cpp @@ -33,6 +33,7 @@ #include #include #include +#include "test_engine_handle.h" static inline void check(bool cond, const char *msg) { @@ -132,13 +133,14 @@ def make_timeline(): char db_path[] = "/tmp/call_graph_method.db"; unlink(db_path); - int rc = engine_init(db_path); + g_engine = engine_create(db_path); + const int rc = g_engine != nullptr ? 0 : -1; check(rc == 0, "engine_init"); - uint64_t pid = engine_create_project(proj_dir, "method-proj"); + uint64_t pid = engine_create_project(g_engine, proj_dir, "method-proj"); check(pid > 0, "create_project"); - char *idx = engine_index_project(pid, proj_dir, nullptr); + char *idx = engine_index_project(g_engine, pid, proj_dir, nullptr); check(idx != nullptr, "index_project returns non-null"); check(strstr(idx, "\"ok\":true") != nullptr, "index_project ok"); engine_free_string(idx); @@ -146,13 +148,13 @@ def make_timeline(): // ── Part 1: C++ method call via field_expression ───────────── // Verify callees of AddPoints include "adder" (the method name, // extracted from the field_expression "a.adder"). - char *callees = engine_get_callees(pid, "AddPoints", nullptr); + char *callees = engine_get_callees(g_engine, pid, "AddPoints", nullptr); check(strstr(callees, "adder") != nullptr, "AddPoints should call adder (field_expression callee extraction)"); engine_free_string(callees); // Verify callers of adder include AddPoints. - char *callers = engine_get_callers(pid, "adder", nullptr); + char *callers = engine_get_callers(g_engine, pid, "adder", nullptr); check(strstr(callers, "AddPoints") != nullptr, "adder should be called by AddPoints (intra-file P1 edge)"); engine_free_string(callers); @@ -182,13 +184,14 @@ def make_timeline(): // ── Part 3: Python method call via attribute ───────────────── // Verify render calls _load_data (intra-class method call). - char *py_callees = engine_get_callees(pid, "render", nullptr); + char *py_callees = engine_get_callees(g_engine, pid, "render", nullptr); check(strstr(py_callees, "_load_data") != nullptr, "render should call _load_data (attribute callee extraction)"); engine_free_string(py_callees); // Verify _load_data is called by render. - char *py_callers = engine_get_callers(pid, "_load_data", nullptr); + char *py_callers = + engine_get_callers(g_engine, pid, "_load_data", nullptr); check(strstr(py_callers, "render") != nullptr, "_load_data should be called by render (Python intra-file P1 edge)"); engine_free_string(py_callers); @@ -240,16 +243,14 @@ def make_timeline(): "SELECT original_id FROM semantic_records " "WHERE project_id=? AND kind=0 AND name='make_timeline' " "LIMIT 1"; - check(sqlite3_prepare_v2(db, fn_sql, -1, &stmt, nullptr) == - SQLITE_OK, + check(sqlite3_prepare_v2(db, fn_sql, -1, &stmt, nullptr) == SQLITE_OK, "prepare make_timeline select"); sqlite3_bind_int64(stmt, 1, static_cast(pid)); int64_t make_timeline_id = -1; if (sqlite3_step(stmt) == SQLITE_ROW) make_timeline_id = sqlite3_column_int64(stmt, 0); sqlite3_finalize(stmt); - check(make_timeline_id > 0, - "make_timeline function record must exist"); + check(make_timeline_id > 0, "make_timeline function record must exist"); check(scatter_parent == make_timeline_id, "Scatter (nested call) parent_id must equal make_timeline " @@ -259,7 +260,8 @@ def make_timeline(): // include Scatter (the nested call). Before the fix, Scatter // was dropped from the reference table because its parent_id // pointed to a CallExpr record (not in _r2n), failing the JOIN. - char *nested_callees = engine_get_callees(pid, "make_timeline", nullptr); + char *nested_callees = + engine_get_callees(g_engine, pid, "make_timeline", nullptr); check(strstr(nested_callees, "Scatter") != nullptr, "make_timeline should call Scatter (nested call must " "appear in callees — was dropped before function_stack_ fix)"); @@ -267,7 +269,8 @@ def make_timeline(): sqlite3_close(db); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; // Cleanup temp dir std::filesystem::remove_all(proj_dir); diff --git a/engine/tests/test_call_graph_p1.cpp b/engine/tests/test_call_graph_p1.cpp index 371d214..9ae1953 100644 --- a/engine/tests/test_call_graph_p1.cpp +++ b/engine/tests/test_call_graph_p1.cpp @@ -22,6 +22,7 @@ #include #include #include +#include "test_engine_handle.h" static inline void check(bool cond, const char *msg) { @@ -53,7 +54,8 @@ int main() { char db_path[] = "/tmp/test_call_graph_p1.db"; unlink(db_path); - int rc = engine_init(db_path); + g_engine = engine_create(db_path); + const int rc = g_engine != nullptr ? 0 : -1; check(rc == 0, "engine_init"); const char *file_path = "/tmp/test_p1_call.c"; @@ -62,22 +64,22 @@ int main() { fwrite(code, 1, strlen(code), f); fclose(f); - uint64_t pid = engine_create_project("/tmp", "p1-test"); + uint64_t pid = engine_create_project(g_engine, "/tmp", "p1-test"); check(pid > 0, "create_project"); - char *idx = engine_index_file(pid, file_path); + char *idx = engine_index_file(g_engine, pid, file_path); check(strstr(idx, "\"ok\":true") != nullptr, "index_file ok"); engine_free_string(idx); // Verify callees of main include helper - char *callees = engine_get_callees(pid, "main", nullptr); + char *callees = engine_get_callees(g_engine, pid, "main", nullptr); printf("--- Callees of main ---\n%s\n", callees); check(strstr(callees, "helper") != nullptr, "main should call helper (P1 intra-file edge)"); engine_free_string(callees); // Verify callers of helper include main - char *callers = engine_get_callers(pid, "helper", nullptr); + char *callers = engine_get_callers(g_engine, pid, "helper", nullptr); printf("--- Callers of helper ---\n%s\n", callers); check(strstr(callers, "main") != nullptr, "helper should be called by main (P1 intra-file edge)"); @@ -85,7 +87,7 @@ int main() { // Verify total edges > 0 (graph_stats returns total_edges, not // total_call_edges — the edge count includes call edges). - char *stats = engine_get_graph_stats(pid); + char *stats = engine_get_graph_stats(g_engine, pid); printf("--- Graph Stats ---\n%s\n", stats); check(strstr(stats, "total_edges") != nullptr, "stats should include total_edges"); @@ -97,11 +99,11 @@ int main() { if (val) edge_count = atoi(val + 1); } - check(edge_count > 0, - "total_edges must be > 0 after P1 fix"); + check(edge_count > 0, "total_edges must be > 0 after P1 fix"); engine_free_string(stats); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; printf("\n=== P1 call graph test passed ===\n"); return 0; diff --git a/engine/tests/test_capability_drift.cpp b/engine/tests/test_capability_drift.cpp index 7b0a7c8..86379a7 100644 --- a/engine/tests/test_capability_drift.cpp +++ b/engine/tests/test_capability_drift.cpp @@ -17,7 +17,7 @@ #include "../src/store/store.h" #include "../src/verify/capability_drift.h" -#include +#include "test_check.h" #include #include #include @@ -36,12 +36,12 @@ static void insertEntity(store::GraphStore &store, uint64_t project_id, "start_col, end_row, end_col) " "VALUES (?,?,0,?,'',?,'cpp',0,0,0,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, file_path, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -53,11 +53,11 @@ static void insertCall(store::GraphStore &store, uint64_t project_id, const char *sql = "INSERT INTO relation (project_id, source_id, " "target_id, type) VALUES (?,?,?,1)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, source_id); sqlite3_bind_int64(stmt, 3, target_id); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -69,11 +69,11 @@ static void insertCapability(store::GraphStore &store, uint64_t project_id, const char *sql = "INSERT INTO capability (project_id, name, summary, " "source_kind) VALUES (?,?,?,'readme')"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 3, summary, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -85,8 +85,8 @@ static int countRows(store::GraphStore &store, uint64_t project_id, std::string sql = std::string("SELECT COUNT(*) FROM ") + table + " WHERE project_id=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) == + SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); int count = 0; if (sqlite3_step(stmt) == SQLITE_ROW) @@ -100,19 +100,20 @@ int main() unlink(kDbPath); store::GraphStore store; - assert(store.open(kDbPath)); + CHECK(store.open(kDbPath)); uint64_t project_id = store.createProject("/test", "test_cap_drift"); - assert(project_id > 0); + CHECK(project_id > 0); // ── Test 1: countImplementingEntities — entity with caller ────── { - insertEntity(store, project_id, 100, "parseCode", "/src/parser.cpp"); + insertEntity(store, project_id, 100, "parseCode", + "/src/parser.cpp"); insertEntity(store, project_id, 101, "caller", "/src/main.cpp"); insertCall(store, project_id, 101, 100); // caller → parseCode - assert(countImplementingEntities(store, project_id, "parseCode") == - 1); + CHECK(countImplementingEntities(store, project_id, + "parseCode") == 1); printf(" [PASS] countImplementingEntities: entity with caller = 1\n"); } @@ -122,21 +123,21 @@ int main() "/src/unused.cpp"); // No call edge targeting entity 200. - assert(countImplementingEntities(store, project_id, - "unusedFunc") == 0); + CHECK(countImplementingEntities(store, project_id, + "unusedFunc") == 0); printf(" [PASS] countImplementingEntities: entity without caller = 0\n"); } // ── Test 3: countImplementingEntities — empty name ────────────── { - assert(countImplementingEntities(store, project_id, "") == 0); + CHECK(countImplementingEntities(store, project_id, "") == 0); printf(" [PASS] countImplementingEntities: empty name = 0\n"); } // ── Test 4: countImplementingEntities — nonexistent name ──────── { - assert(countImplementingEntities(store, project_id, - "doesNotExist") == 0); + CHECK(countImplementingEntities(store, project_id, + "doesNotExist") == 0); printf(" [PASS] countImplementingEntities: nonexistent name = 0\n"); } @@ -149,7 +150,7 @@ int main() insertCapability(store, project_id, "parseCode", "Parses source code"); auto drifts = detectCapabilityDrift(store, project_id); - assert(drifts.empty()); + CHECK(drifts.empty()); printf(" [PASS] detectCapabilityDrift: capability with impl → no drift\n"); } @@ -162,10 +163,10 @@ int main() insertCapability(store, project_id, "nonexistentFeature", "Feature declared but not implemented"); auto drifts = detectCapabilityDrift(store, project_id); - assert(drifts.size() == 1); - assert(drifts[0].type == "CapabilityDrift"); - assert(drifts[0].severity == kDriftSeverityCapability); - assert(drifts[0].subject == "nonexistentFeature"); + CHECK(drifts.size() == 1); + CHECK(drifts[0].type == "CapabilityDrift"); + CHECK(drifts[0].severity == kDriftSeverityCapability); + CHECK(drifts[0].subject == "nonexistentFeature"); printf(" [PASS] detectCapabilityDrift: missing capability → 1 drift\n"); } @@ -176,7 +177,7 @@ int main() nullptr, nullptr, nullptr); auto drifts = detectCapabilityDrift(store, project_id); - assert(drifts.empty()); + CHECK(drifts.empty()); printf(" [PASS] detectCapabilityDrift: empty capability table → no drifts\n"); } @@ -194,28 +195,28 @@ int main() insertCapability(store, project_id, "unusedFunc", "no caller"); auto drifts = detectCapabilityDrift(store, project_id); - assert(drifts.size() == 2); + CHECK(drifts.size() == 2); bool has_ghost = false, has_unused = false; for (const auto &d : drifts) { - assert(d.type == "CapabilityDrift"); - assert(d.severity == kDriftSeverityCapability); + CHECK(d.type == "CapabilityDrift"); + CHECK(d.severity == kDriftSeverityCapability); if (d.subject == "ghostFeature") has_ghost = true; if (d.subject == "unusedFunc") has_unused = true; } - assert(has_ghost); - assert(has_unused); + CHECK(has_ghost); + CHECK(has_unused); printf(" [PASS] detectCapabilityDrift: mixed → 2 drifts (ghostFeature + unusedFunc)\n"); } // ── Test 9: detectCapabilityDrift — row count sanity check ────── // Verify that the row counts are as expected for sanity. { - assert(countRows(store, project_id, "entity") == 3); - assert(countRows(store, project_id, "relation") == 1); - assert(countRows(store, project_id, "capability") == 3); + CHECK(countRows(store, project_id, "entity") == 3); + CHECK(countRows(store, project_id, "relation") == 1); + CHECK(countRows(store, project_id, "capability") == 3); printf(" [PASS] detectCapabilityDrift: row counts verified\n"); } @@ -223,5 +224,5 @@ int main() unlink(kDbPath); printf("=== test_capability_drift PASSED ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_capability_plugin_lines.cpp b/engine/tests/test_capability_plugin_lines.cpp new file mode 100644 index 0000000..7a7901c --- /dev/null +++ b/engine/tests/test_capability_plugin_lines.cpp @@ -0,0 +1,99 @@ +// test_capability_plugin_lines: the capability pass must take its names from +// whole LINES, and must not duplicate what is already stored. +// +// Regression for two defects in CapabilityPlugin::build + insertCapability: +// * the line scan advanced on any '-' or '*' as well as '\n', so it restarted +// mid-line and named a capability after the fragment that follows the +// hyphen ("FastLookup" from "A thread-safe design - Supports fast lookup"). +// That name matches no code, so the capability-drift detector reported it as +// undocumented — drift invented by the extractor. +// * the table had no uniqueness rule and the INSERT was unguarded, so every +// index run appended another copy of every capability, double-counting in +// capability_state and in the drift output. + +#include "../src/model/plugin.h" +#include "../src/model/plugins/capability.h" +#include "../src/store/store.h" + +#include + +#include "test_check.h" +#include +#include +#include + +static const char *kDbPath = "/tmp/test_capability_plugin_lines.db"; + +/// Capability names stored for this project, in id order. +static std::vector capabilityNames(store::GraphStore &store, + uint64_t project_id) +{ + std::vector names; + sqlite3_stmt *stmt = nullptr; + const char *sql = + "SELECT name FROM capability WHERE project_id=? ORDER BY id"; + CHECK(sqlite3_prepare_v2(store.handle(), sql, -1, &stmt, nullptr) == + SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + while (sqlite3_step(stmt) == SQLITE_ROW) { + const char *n = reinterpret_cast( + sqlite3_column_text(stmt, 0)); + names.push_back(n ? n : ""); + } + sqlite3_finalize(stmt); + return names; +} + +int main() +{ + unlink(kDbPath); + + store::GraphStore store; + if (!store.open(kDbPath)) { + fprintf(stderr, "FAIL: cannot open store: %s\n", + store.error().c_str()); + return 1; + } + uint64_t pid = store.createProject("/tmp", "test_capability_lines"); + CHECK(pid > 0); + + model::ModelContext ctx; + model::DocumentInfo doc; + doc.file_path = "README.md"; + // Line 1 has a hyphen in the middle: the old scan restarted there. + // Line 2 is a plain bullet. + doc.content = "# Tool\n" + "A thread-safe design - Supports fast lookup\n" + "- Supports incremental indexing\n"; + ctx.documents.push_back(doc); + + model::CapabilityPlugin plugin(&store); + model::ModelResult r = plugin.build(pid, ctx); + CHECK(r.ok()); + CHECK(r.items_created == 2); + + std::vector names = capabilityNames(store, pid); + CHECK(names.size() == 2); + for (const auto &n : names) + printf(" capability: %s\n", n.c_str()); + + // The first name comes from the whole line, not from the fragment after the + // hyphen. ("FastLookup" was the old, fragment-derived name.) + CHECK(names[0].find("ThreadSafe") != std::string::npos); + CHECK(names[0] != "FastLookup"); + // The bullet's leading "- " is still stripped. + CHECK(names[1] == "IncrementalIndexing"); + + // Idempotent: a second pass over the same documents adds nothing. + model::ModelResult again = plugin.build(pid, ctx); + CHECK(again.ok()); + names = capabilityNames(store, pid); + CHECK(names.size() == 2); + printf(" second pass kept %zu capabilities (no duplicates)\n", + names.size()); + + store.close(); + unlink(kDbPath); + printf("=== test_capability_plugin_lines PASSED ===\n"); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_capability_prefix_floor.cpp b/engine/tests/test_capability_prefix_floor.cpp new file mode 100644 index 0000000..b3d6f15 --- /dev/null +++ b/engine/tests/test_capability_prefix_floor.cpp @@ -0,0 +1,147 @@ +// test_capability_prefix_floor.cpp — regression test for capability matching. +// +// Defect guarded: countImplementingEntities() and entitiesWithCallers() +// matched a capability name against an entity name with a BIDIRECTIONAL +// prefix LIKE and no length floor: +// +// LOWER(e.name) LIKE LOWER(cap) || '%' +// OR LOWER(cap) LIKE LOWER(e.name) || '%' +// +// The reverse direction degenerates when the entity name is short: with +// capability "GetNeighbors" any 1-3 character symbol (get, or a +// single-letter variable) satisfies `cap LIKE name||'%'`, so unrelated +// entities were counted as implementing the capability — a false positive +// that makes capability-drift verdicts lie. +// +// Rule now enforced: exact (case-insensitive) equality matches at any +// length; a PREFIX match requires BOTH sides to be at least +// kMinCapabilityPrefixLen (4) characters. That clears the very common code +// symbols of length <= 3 (get/set/add/map/run/put/len/i/x …) while keeping +// the intended behavior ("IncrementalIndex" implements +// "IncrementalIndexing"). +// +// Uses the explicit check() pattern: assert() is compiled out under +// NDEBUG, which would make these checks vacuous in a Release build. +// +// Exit code: 0 on success, 1 on any failed assertion. + +#include "../src/store/store.h" +#include "../src/verify/capability_drift.h" + +#include +#include +#include +#include +#include + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "FAIL: %s\n", msg); + exit(1); + } +} + +/// Insert an entity row with the given id, name, and file path. +static void insertEntity(store::GraphStore &store, uint64_t project_id, + int64_t id, const char *name, const char *file_path) +{ + sqlite3 *db = store.handle(); + const char *sql = "INSERT INTO entity (id, project_id, kind, name, " + "qualified_name, file_path, language, start_row, " + "start_col, end_row, end_col) " + "VALUES (?,?,0,?,'',?,'cpp',0,0,0,0)"; + sqlite3_stmt *stmt = nullptr; + check(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK, + "prepare insertEntity"); + sqlite3_bind_int64(stmt, 1, id); + sqlite3_bind_int64(stmt, 2, static_cast(project_id)); + sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 4, file_path, -1, SQLITE_TRANSIENT); + check(sqlite3_step(stmt) == SQLITE_DONE, "insertEntity step"); + sqlite3_finalize(stmt); +} + +/// Insert a call relation (type=1) from source_id to target_id. +static void insertCall(store::GraphStore &store, uint64_t project_id, + int64_t source_id, int64_t target_id) +{ + sqlite3 *db = store.handle(); + const char *sql = "INSERT INTO relation (project_id, source_id, " + "target_id, type) VALUES (?,?,?,1)"; + sqlite3_stmt *stmt = nullptr; + check(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK, + "prepare insertCall"); + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_int64(stmt, 2, source_id); + sqlite3_bind_int64(stmt, 3, target_id); + check(sqlite3_step(stmt) == SQLITE_DONE, "insertCall step"); + sqlite3_finalize(stmt); +} + +/// Assert the implementing-entity count for a capability name. +static void expectCount(store::GraphStore &store, uint64_t project_id, + const char *cap, int want, const char *why) +{ + int64_t got = verify::countImplementingEntities(store, project_id, cap); + if (got != want) { + fprintf(stderr, + "FAIL: countImplementingEntities(\"%s\") = %lld, " + "expected %d (%s)\n", + cap, static_cast(got), want, why); + exit(1); + } + printf(" [PASS] \"%s\" -> %d (%s)\n", cap, want, why); +} + +int main() +{ + const char *db_path = "/tmp/codescope_test_cap_prefix_floor.db"; + unlink(db_path); + + store::GraphStore store; + check(store.open(db_path), "open database"); + uint64_t project_id = store.createProject("/test", "cap_prefix_floor"); + check(project_id > 0, "createProject"); + + // A caller entity every target below can be reached from. + insertEntity(store, project_id, 1, "/src/main.cpp", "/src/main.cpp"); + + // ── Case 1: legit prefix (both sides >= 4) still counts ──────── + insertEntity(store, project_id, 10, "IncrementalIndex", + "/src/indexer.cpp"); + insertCall(store, project_id, 1, 10); + expectCount(store, project_id, "IncrementalIndexing", 1, + "legit prefix match must still count"); + + // ── Case 2: 3-char symbol must NOT match a longer phrase ─────── + // `get` is a plausible user symbol; "GetNeighbors" LIKE 'get%' is + // true, but a 3-char name is far more likely a coincidence. + insertEntity(store, project_id, 11, "get", "/src/util.c"); + insertCall(store, project_id, 1, 11); + expectCount(store, project_id, "GetNeighbors", 0, + "3-char symbol must not match by prefix"); + + // ── Case 3: single-letter symbol must NOT match either ───────── + insertEntity(store, project_id, 12, "i", "/src/loop.c"); + insertCall(store, project_id, 1, 12); + expectCount(store, project_id, "IncrementalIndexing", 1, + "1-char symbol must not change the count"); + + // ── Case 4: exact equality matches at any length ─────────────── + // A genuine short symbol is never lost — only the accidental prefix. + insertEntity(store, project_id, 13, "Run", "/src/run.cpp"); + insertCall(store, project_id, 1, 13); + expectCount(store, project_id, "Run", 1, + "exact match counts at any length"); + + // ── Case 5: dead code (no caller) never counts ───────────────── + insertEntity(store, project_id, 14, "MapIndexing", "/src/dead.cpp"); + expectCount(store, project_id, "MapIndexing", 0, + "dead code must not count as implementing"); + + store.close(); + unlink(db_path); + printf("\nAll capability prefix-floor tests passed.\n"); + return 0; +} diff --git a/engine/tests/test_capability_state_matches_verifier.cpp b/engine/tests/test_capability_state_matches_verifier.cpp new file mode 100644 index 0000000..5e3dbdb --- /dev/null +++ b/engine/tests/test_capability_state_matches_verifier.cpp @@ -0,0 +1,228 @@ +// test_capability_state_matches_verifier — the two capability views must agree. +// +// Defect guarded: build_project_state reports project_state.capability.total / +// verified from `capability_state` (model::StateBuilder::buildCapabilityState), +// while verify_claim / detect_capability_drift report the verdict of +// verify::CapabilityVerifier for the same DECLARED capabilities. The two used +// to answer with different rules — a naming heuristic with no call-graph +// evidence in the builder, a name match plus a caller/export requirement in the +// verifier — so one view called a capability Implemented while the other +// Contradicted it (measured on goagent: 19 vs 7; on this repository: 3 vs 3 +// contradicted). +// +// The rule now lives in verify::implementingEntitiesFor and the builder +// restates it in SQL (model/ cannot include verify/). Restating is only safe +// while the two stay equivalent, so this test is the equivalence check: for +// every declared capability it compares +// +// capability_state.state == implementingEntitiesFor(...).empty() +// ? "Declared" : "Implemented" +// +// and pins the cases where a hand-written restatement drifts: the arms that +// match two normalised spellings carry their own length floors, and a floor +// applied to the wrong name ("c_ac" → "cac") puts the builder back on the +// accidental-prefix path the floor exists to block. +// +// Exit code: 0 when every check passed, 1 otherwise. + +#include "../src/model/state_builder.h" +#include "../src/store/store.h" +#include "../src/verify/capability_verifier.h" + +#include "test_check.h" + +#include +#include +#include + +/// Insert a declared capability (the `capability` table, normally filled from +/// the project README by the knowledge builder). +static void insertCapability(store::GraphStore &store, uint64_t project_id, + const char *name) +{ + sqlite3 *db = store.handle(); + const char *sql = "INSERT INTO capability (project_id, name, source_kind) " + "VALUES (?,?, 'readme')"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_text(stmt, 2, name, -1, SQLITE_TRANSIENT); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); +} + +/// Insert a function entity. visibility=1 marks an exported/public entry point, +/// which counts as evidence on its own (an FFI export's callers live in the +/// other language and produce no edge here). +static void insertEntity(store::GraphStore &store, uint64_t project_id, + int64_t id, const char *name, int visibility = 0) +{ + sqlite3 *db = store.handle(); + const char *sql = "INSERT INTO entity (id, project_id, kind, name, " + "qualified_name, file_path, language, start_row, " + "start_col, end_row, end_col, visibility) " + "VALUES (?,?,0,?,'','/src/a.cpp','cpp',0,0,0,0,?)"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + sqlite3_bind_int64(stmt, 1, id); + sqlite3_bind_int64(stmt, 2, static_cast(project_id)); + sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); + sqlite3_bind_int(stmt, 4, visibility); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); +} + +/// Insert a call edge (relation type=1) from source_id to target_id — the +/// caller evidence both views require. +static void insertCall(store::GraphStore &store, uint64_t project_id, + int64_t source_id, int64_t target_id) +{ + sqlite3 *db = store.handle(); + const char *sql = "INSERT INTO relation (project_id, source_id, " + "target_id, type) VALUES (?,?,?,1)"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_int64(stmt, 2, source_id); + sqlite3_bind_int64(stmt, 3, target_id); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); +} + +/// The state the builder wrote for one capability ("Implemented" / +/// "Declared"); "(missing)" when it wrote no row, so a failure names the gap +/// instead of comparing against an empty string. +static std::string stateOf(store::GraphStore &store, uint64_t project_id, + const char *name) +{ + sqlite3 *db = store.handle(); + const char *sql = "SELECT state FROM capability_state " + "WHERE project_id=? AND name=?"; + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) != SQLITE_OK) + return "(prepare failed)"; + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_text(stmt, 2, name, -1, SQLITE_TRANSIENT); + std::string out = "(missing)"; + if (sqlite3_step(stmt) == SQLITE_ROW) { + const char *s = reinterpret_cast( + sqlite3_column_text(stmt, 0)); + out = s ? s : ""; + } + sqlite3_finalize(stmt); + return out; +} + +/// Scalar helper for the row-count assertions. +static int64_t scalar(store::GraphStore &store, const char *sql, + uint64_t project_id) +{ + sqlite3 *db = store.handle(); + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) != SQLITE_OK) + return -1; + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + int64_t n = -1; + if (sqlite3_step(stmt) == SQLITE_ROW) + n = sqlite3_column_int64(stmt, 0); + sqlite3_finalize(stmt); + return n; +} + +/// The one assertion this test exists for: the builder's row and the +/// verifier's rule must describe the same capability the same way. +static void expectAgreement(store::GraphStore &store, uint64_t project_id, + const char *capability) +{ + bool ok = false; + const std::vector ids = + verify::implementingEntitiesFor(&store, project_id, capability, + ok); + CHECK(ok); + const std::string want = ids.empty() ? "Declared" : "Implemented"; + const std::string got = stateOf(store, project_id, capability); + if (want != got) { + fprintf(stderr, + "FAIL: \"%s\" — capability_state said \"%s\" while " + "implementingEntitiesFor returned %zu entities " + "(\"%s\")\n", + capability, got.c_str(), ids.size(), want.c_str()); + } + CHECK(want == got); +} + +int main() +{ + const char *db_path = "/tmp/codescope_test_cap_state_agrees.db"; + unlink(db_path); + + store::GraphStore store; + CHECK(store.open(db_path)); + const uint64_t project_id = store.createProject("/test", "cap_state"); + CHECK(project_id > 0); + + // ── Declared capabilities ────────────────────────────────────── + // "IncrementalIndexing": legit bidirectional prefix of the code symbol + // "IncrementalIndex" — must stay Implemented. + // "Run": exact match at a length below the floor — must stay + // Implemented, because only the accidental PREFIX is rejected. + // "CacheIndex": its only near-match is "c_ac", whose normalised name is + // "cac" (3 characters). The de-underscore arm must floor the NORMALISED + // entity name, otherwise 'cacheindex' LIKE 'cac%' makes the builder + // report Implemented with no implementing entity at all. + insertCapability(store, project_id, "IncrementalIndexing"); + insertCapability(store, project_id, "Run"); + insertCapability(store, project_id, "CacheIndex"); + insertCapability(store, project_id, "TelemetryExport"); + + // A caller so every target below has the evidence both views require. + insertEntity(store, project_id, 1, "main"); + insertEntity(store, project_id, 10, "IncrementalIndex"); + insertCall(store, project_id, 1, 10); + insertEntity(store, project_id, 11, "Run"); + insertCall(store, project_id, 1, 11); + insertEntity(store, project_id, 12, "c_ac"); + insertCall(store, project_id, 1, 12); + // Name-matches "TelemetryExport" by prefix, has no caller and is not + // exported: a name match without evidence is not an implementation. + insertEntity(store, project_id, 13, "TelemetryExporter"); + + model::StateBuilder builder(&store, project_id); + CHECK(builder.buildCapabilityState() == 4); + + // 1. The snapshot mirrors the DECLARED set — no heuristic extra rows. + CHECK(scalar(store, + "SELECT COUNT(*) FROM capability_state WHERE project_id=?", + project_id) == 4); + + // 2. Every row agrees with the verifier's rule. + expectAgreement(store, project_id, "IncrementalIndexing"); + expectAgreement(store, project_id, "Run"); + expectAgreement(store, project_id, "CacheIndex"); + expectAgreement(store, project_id, "TelemetryExport"); + + // 3. Pinned verdicts, so the agreement above cannot pass vacuously by + // both sides going blank. + CHECK(stateOf(store, project_id, "IncrementalIndexing") == "Implemented"); + CHECK(stateOf(store, project_id, "Run") == "Implemented"); + CHECK(stateOf(store, project_id, "CacheIndex") == "Declared"); + CHECK(stateOf(store, project_id, "TelemetryExport") == "Declared"); + + // 4. A second build replaces the rows instead of appending a copy of + // every one (capability_state has no UNIQUE(project_id, name), so the + // old INSERT OR IGNORE had no conflict target). + CHECK(builder.buildCapabilityState() == 4); + CHECK(scalar(store, + "SELECT COUNT(*) FROM capability_state WHERE project_id=?", + project_id) == 4); + + store.close(); + unlink(db_path); + + if (checkFailures()) { + fprintf(stderr, "=== capability_state/verifier agreement: FAILED ===\n"); + return 1; + } + printf("=== capability_state/verifier agreement test passed ===\n"); + return 0; +} diff --git a/engine/tests/test_check.h b/engine/tests/test_check.h new file mode 100644 index 0000000..223883a --- /dev/null +++ b/engine/tests/test_check.h @@ -0,0 +1,120 @@ +#ifndef CODESCOPE_TEST_CHECK_H +#define CODESCOPE_TEST_CHECK_H + +// Shared non-vacuous check macro for the C++ engine tests. +// +// Why not : assert() is compiled out when NDEBUG is defined — i.e. +// in every Release / RelWithDebInfo build — so an assert-based test silently +// becomes a no-op that still "passes". That is the extreme form of the +// under-testing REVIEW_0.2.7.md TEST-3 describes, and it once masked a CI +// skip-list drift where 26 tests never really ran. +// +// CHECK() is always evaluated, and on failure it prints the source location, +// the expression and an optional message. Unlike assert() it does NOT abort: +// it records the failure and lets the test continue, so one broken expectation +// cannot hide the ones after it. +// +// Contract for a test's main(): the success path should end with +// +// return checkFailures() ? 1 : 0; +// +// so that a recorded failure becomes a non-zero exit code. That part is now +// also ENFORCED rather than asked for: reportFailure() registers an exit hook +// that exits non-zero if any check failed, because 18 of the engine tests +// printed their banner and returned without consulting the counter — a failing +// CHECK there printed the failure, exited 0, and `make test-engine` (which +// judges by exit code) called it a pass. +// +// Usage: +// #include "test_check.h" // instead of +// ... +// CHECK(pid > 0); // no message +// CHECK_MSG(rc == 0, "open"); // with a message +// ... +// return checkFailures() ? 1 : 0; // belt and braces; the hook covers it + +#include +#include + +namespace codescope_test +{ + +/// Failure counter for the current test binary. +/// +/// The function-local static lives in an inline function, so every translation +/// unit in one test binary shares the same counter. +inline int &failureCount() +{ + static int count = 0; + return count; +} + +/// Make a recorded failure outlive a main() that forgot to consult the +/// counter. +/// +/// Registered by the first failure, which is also the first moment the answer +/// matters: a test that records none exits with whatever its main returns (0), +/// and a test that records one cannot exit 0 by accident. The hook runs at +/// exit and uses std::_Exit on purpose — calling exit() from an atexit handler +/// is undefined, and the remaining handlers could flush or print against state +/// that is already being torn down. +inline void failOnRecordedFailure() +{ + static const bool registered = []() { + std::atexit([]() { + if (failureCount() == 0) + return; + // Flush before _Exit: _Exit skips the remaining handlers on + // purpose (calling exit() from a handler is undefined, and the + // rest could print or flush against state already being torn + // down), and under `make check` stdout is a fully buffered file, + // so the failing test's own output — the diagnosis — would + // otherwise be discarded exactly when it is needed. + std::fflush(nullptr); + std::_Exit(1); + }); + return true; + }(); + (void)registered; +} + +/// Record a failed check: print `file:line`, the expression and an optional +/// message, then bump the counter. +inline void reportFailure(const char *expr, const char *file, int line, + const char *message) +{ + failOnRecordedFailure(); + ++failureCount(); + if (message && *message) + std::fprintf(stderr, "CHECK FAILED: %s (%s) at %s:%d\n", expr, + message, file, line); + else + std::fprintf(stderr, "CHECK FAILED: %s at %s:%d\n", expr, file, + line); +} + +} // namespace codescope_test + +/// Evaluate `cond`; on failure record it and continue. +#define CHECK(cond) \ + do { \ + if (!(cond)) \ + ::codescope_test::reportFailure(#cond, __FILE__, \ + __LINE__, nullptr); \ + } while (0) + +/// CHECK() with a caller-supplied explanation. +#define CHECK_MSG(cond, message) \ + do { \ + if (!(cond)) \ + ::codescope_test::reportFailure(#cond, __FILE__, \ + __LINE__, (message)); \ + } while (0) + +/// Number of failed checks recorded so far (0 == success). +inline int checkFailures() +{ + return codescope_test::failureCount(); +} + +#endif // CODESCOPE_TEST_CHECK_H diff --git a/engine/tests/test_claim_parser.cpp b/engine/tests/test_claim_parser.cpp index 6130bb2..740ec85 100644 --- a/engine/tests/test_claim_parser.cpp +++ b/engine/tests/test_claim_parser.cpp @@ -3,7 +3,7 @@ // in engine/tests. #include "verify/claim_parser.h" -#include +#include "test_check.h" #include #include #include @@ -52,8 +52,8 @@ int main() bool has_contract = hasClaim(claims, ClaimType::ContractHolds, "ThreadSafe"); - assert(has_cap); - assert(has_contract); + CHECK(has_cap); + CHECK(has_contract); printf("Test 1 (supports + thread-safe): PASS\n"); } @@ -65,7 +65,7 @@ int main() bool has_cap = hasClaim(claims, ClaimType::CapabilityExists, "CallGraphAnalysis"); - assert(has_cap); + CHECK(has_cap); printf("Test 2 (implements): PASS\n"); } @@ -76,13 +76,37 @@ int main() "lock-free."; auto claims = parser.parse(text, "readme", "README.md"); - assert(hasClaim(claims, ClaimType::ContractHolds, - "MemorySafe")); - assert(hasClaim(claims, ClaimType::ContractHolds, "ZeroCopy")); - assert(hasClaim(claims, ClaimType::ContractHolds, "LockFree")); + CHECK(hasClaim(claims, ClaimType::ContractHolds, "MemorySafe")); + CHECK(hasClaim(claims, ClaimType::ContractHolds, "ZeroCopy")); + CHECK(hasClaim(claims, ClaimType::ContractHolds, "LockFree")); printf("Test 3 (memory-safe/zero-copy/lock-free): PASS\n"); } + // ── Test 3b: negation must NOT become a positive claim ─────── + // Regression: "not thread-safe" / "isn't thread-safe" previously + // produced ContractHolds(ThreadSafe) — polarity inversion that let + // ContractVerifier answer Supported on the README's denial. + // "cannot" is NOT a denial (it merely ends in the letters "not"): + // the keyword claim must still be emitted, or the `\bnot` anchor + // would over-suppress. + { + std::string neg1 = "This library is not thread-safe."; + std::string neg2 = "The buffer isn't thread-safe under load."; + std::string pos1 = "This design cannot thread-safe anything."; + std::string pos2 = "This library is thread-safe."; + + auto c1 = parser.parse(neg1, "readme", "README.md"); + auto c2 = parser.parse(neg2, "readme", "README.md"); + auto c3 = parser.parse(pos1, "readme", "README.md"); + auto c4 = parser.parse(pos2, "readme", "README.md"); + + CHECK(!hasClaim(c1, ClaimType::ContractHolds, "ThreadSafe")); + CHECK(!hasClaim(c2, ClaimType::ContractHolds, "ThreadSafe")); + CHECK(hasClaim(c3, ClaimType::ContractHolds, "ThreadSafe")); + CHECK(hasClaim(c4, ClaimType::ContractHolds, "ThreadSafe")); + printf("Test 3b (negation / cannot): PASS\n"); + } + // ── Test 4: architecture arrow chain (whitelisted) ─────────── { std::string text = @@ -91,7 +115,7 @@ int main() int arch_count = countByType(claims, ClaimType::ArchitectureFollows); - assert(arch_count >= 1); + CHECK(arch_count >= 1); // Verify the claim fields: subject=Controller, // predicate=flows_to, object=Service, scope=Repository. @@ -105,7 +129,7 @@ int main() break; } } - assert(found); + CHECK(found); printf("Test 4 (arrow chain): PASS\n"); } @@ -116,14 +140,14 @@ int main() int arch_count = countByType(claims, ClaimType::ArchitectureFollows); - assert(arch_count == 0); + CHECK(arch_count == 0); printf("Test 5 (non-layer arrow rejected): PASS\n"); } // ── Test 6: empty text produces no claims ──────────────────── { auto claims = parser.parse("", "readme", "README.md"); - assert(claims.empty()); + CHECK(claims.empty()); printf("Test 6 (empty text): PASS\n"); } @@ -133,14 +157,75 @@ int main() { std::string text = "supports XYZ"; auto claims = parser.parse(text, "ai_summary", "sum-42"); - assert(!claims.empty()); + CHECK(!claims.empty()); for (const auto &c : claims) { - assert(c.source_kind == "ai_summary"); - assert(c.source_ref == "sum-42"); + CHECK(c.source_kind == "ai_summary"); + CHECK(c.source_ref == "sum-42"); } printf("Test 7 (source stamping): PASS\n"); } + // ── Test 8: "has " existence claim ────────────────── + // "The project has a LeaderAgent class" must produce a + // CapabilityExists claim whose subject is the identity token + // "LeaderAgent" — the article and the trailing generic noun + // ("class") are not part of the name. + { + std::string text = "The project has a LeaderAgent class."; + auto claims = parser.parse(text, "ai_summary", "sum-8"); + CHECK(hasClaim(claims, ClaimType::CapabilityExists, + "LeaderAgent")); + CHECK(!hasClaim(claims, ClaimType::CapabilityExists, + "LeaderAgentClass")); + printf("Test 8 (has + trailing noun strip): PASS\n"); + } + + // ── Test 9: "should" behavioral claim ──────────────────────── + // "c_print should handle null input" must produce a + // FunctionImplements claim whose subject is the VERBATIM function + // name (PascalCase-mangling would break the entity lookup) and + // whose object is the behavior phrase. + { + std::string text = "c_print should handle null input."; + auto claims = parser.parse(text, "ai_summary", "sum-9"); + CHECK(hasClaim(claims, ClaimType::FunctionImplements, + "c_print")); + CHECK(!hasClaim(claims, ClaimType::FunctionImplements, + "CPrint")); + bool object_ok = false; + for (const auto &c : claims) { + if (c.type == ClaimType::FunctionImplements && + c.subject == "c_print" && + c.object.rfind("handle", 0) == 0) + object_ok = true; + } + CHECK(object_ok); + printf("Test 9 (should -> FunctionImplements): PASS\n"); + } + + // ── Test 10: negated has/should must NOT become positive ──── + // "has no X" / "doesn't have X" / "should never X" / "should not + // X" are denials; emitting a positive claim would invert the + // source polarity (the same failure mode Test 3b pins for the + // keyword contracts). + { + std::string t1 = "The project has no LeaderAgent class."; + std::string t2 = "The project doesn't have a LeaderAgent."; + std::string t3 = "c_print should never dereference null."; + std::string t4 = "c_print should not crash on empty input."; + auto c1 = parser.parse(t1, "ai_summary", "sum-10a"); + auto c2 = parser.parse(t2, "ai_summary", "sum-10b"); + auto c3 = parser.parse(t3, "ai_summary", "sum-10c"); + auto c4 = parser.parse(t4, "ai_summary", "sum-10d"); + CHECK(!hasClaim(c1, ClaimType::CapabilityExists, + "LeaderAgent")); + CHECK(!hasClaim(c2, ClaimType::CapabilityExists, + "LeaderAgent")); + CHECK(!hasClaim(c3, ClaimType::FunctionImplements, "c_print")); + CHECK(!hasClaim(c4, ClaimType::FunctionImplements, "c_print")); + printf("Test 10 (negated has/should suppressed): PASS\n"); + } + printf("\n=== test_claim_parser PASSED ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_communities.cpp b/engine/tests/test_communities.cpp new file mode 100644 index 0000000..ec6cce5 --- /dev/null +++ b/engine/tests/test_communities.cpp @@ -0,0 +1,210 @@ +// test_communities.cpp +// +// Regression test for community detection (QueryEngine::getCommunities, +// restored in query_communities.cpp after the Phase-0 cut left a stub). +// +// The fixture is two triangles joined by a single bridge edge: +// +// alpha ── bravo delta ── echo +// \ / \ / +// gamma ─────────────── foxtrot +// (bridge) +// +// plus three decoys that must NOT influence the result: +// * a self-loop (gamma → gamma): a recursive call carries no signal +// * an isolated entity: no CALLS edge, so it is not in any community +// * a References(0) edge between two other entities: only CALLS counts +// +// Asserts: +// 1. Two communities of three members each (the bridge must NOT collapse +// them into one giant community — that is the reason the propagation +// updates synchronously instead of sweeping sequentially). +// 2. The representative of each community is its highest-degree member. +// 3. inter_community_edges == 1 (the bridge). +// 4. The output is byte-identical across calls (determinism contract). +// 5. Members are omitted unless include_members is set, and are clamped. +// 6. max_communities / max_members are clamped and reported via +// "truncated". +// 7. A project with no CALLS edges yields an empty result, not an error. + +#include "../src/query/query_engine.h" +#include "../src/store/store.h" + +#include "test_check.h" +#include +#include +#include +#include + +static const char *kDbPath = "/tmp/codescope_test_communities.db"; + +/// Insert an entity row with an explicit id and the minimum required columns. +static void insertEntity(store::GraphStore &store, uint64_t project_id, + int64_t id, const char *name) +{ + sqlite3 *db = store.handle(); + const char *sql = "INSERT INTO entity (id, project_id, kind, name, " + "qualified_name, file_path, language, start_row, " + "start_col, end_row, end_col) " + "VALUES (?, ?, 0, ?, ?, ?, 'cpp', 1, 0, 10, 0)"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + std::string path = std::string("/t/") + name + ".cpp"; + sqlite3_bind_int64(stmt, 1, id); + sqlite3_bind_int64(stmt, 2, static_cast(project_id)); + sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 4, name, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 5, path.c_str(), -1, SQLITE_TRANSIENT); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); +} + +/// Insert a typed relation row (type=1 is CALLS). +static void insertRelation(store::GraphStore &store, uint64_t project_id, + int64_t source_id, int64_t target_id, int type) +{ + sqlite3 *db = store.handle(); + const char *sql = + "INSERT OR IGNORE INTO relation (project_id, source_id, " + "target_id, type) VALUES (?, ?, ?, ?)"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_int64(stmt, 2, source_id); + sqlite3_bind_int64(stmt, 3, target_id); + sqlite3_bind_int(stmt, 4, type); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); +} + +static bool contains(const std::string &haystack, const char *needle) +{ + return haystack.find(needle) != std::string::npos; +} + +int main() +{ + unlink(kDbPath); + unlink((std::string(kDbPath) + "-wal").c_str()); + unlink((std::string(kDbPath) + "-shm").c_str()); + + store::GraphStore store; + if (!store.open(kDbPath)) { + fprintf(stderr, "FAIL: store.open(%s)\n", kDbPath); + return 1; + } + uint64_t project_id = + store.createProject("/communities", "communities"); + CHECK(project_id > 0); + + // ── Fixture ───────────────────────────────────────────────── + insertEntity(store, project_id, 1, "alpha"); + insertEntity(store, project_id, 2, "bravo"); + insertEntity(store, project_id, 3, "gamma"); + insertEntity(store, project_id, 4, "delta"); + insertEntity(store, project_id, 5, "echo"); + insertEntity(store, project_id, 6, "foxtrot"); + insertEntity(store, project_id, 7, "isolated"); // no CALLS edge + insertEntity(store, project_id, 8, "refsource"); + insertEntity(store, project_id, 9, "reftarget"); + + // Triangle A + triangle B, joined by gamma → delta. + insertRelation(store, project_id, 1, 2, 1); + insertRelation(store, project_id, 2, 3, 1); + insertRelation(store, project_id, 1, 3, 1); + insertRelation(store, project_id, 3, 4, 1); // bridge + insertRelation(store, project_id, 4, 5, 1); + insertRelation(store, project_id, 5, 6, 1); + insertRelation(store, project_id, 4, 6, 1); + // Decoys. + insertRelation(store, project_id, 3, 3, 1); // self-loop + insertRelation(store, project_id, 8, 9, 0); // References — not CALLS + + query::QueryEngine engine(&store); + + // ── 1. Two communities, not one ───────────────────────────── + std::string out = engine.getCommunities(project_id, 10, 20, false); + printf(" [debug] summary = %s\n", out.c_str()); + CHECK(contains(out, "\"total_communities\":2") && + "bridge must not collapse the two triangles into one community"); + CHECK(contains(out, "\"returned_communities\":2")); + CHECK(contains(out, "\"inter_community_edges\":1") && + "exactly the bridge crosses communities"); + CHECK(contains(out, "\"truncated\":false")); + CHECK(contains(out, "\"approximation\":\"heuristic\"")); + // Both communities have three members. + CHECK(contains(out, "\"member_count\":3")); + // Highest-degree member of each triangle is the bridge endpoint. + CHECK(contains(out, + "\"id\":3,\"label\":\"gamma\",\"member_count\":3") && + "triangle A representative must be gamma (degree 3)"); + CHECK(contains(out, + "\"id\":4,\"label\":\"delta\",\"member_count\":3") && + "triangle B representative must be delta (degree 3)"); + // Decoys must not leak in. + CHECK(!contains(out, "isolated") && + "an isolated entity has no CALLS edge and must not appear"); + CHECK(!contains(out, "\"members\":[") && + "members must be omitted when include_members is false"); + CHECK(!contains(out, "refsource") && !contains(out, "reftarget") && + "a References(0) edge must not create a community"); + + // ── 2. Determinism ────────────────────────────────────────── + std::string out_again = + engine.getCommunities(project_id, 10, 20, false); + CHECK(out == out_again && + "getCommunities must be deterministic for identical input"); + + // ── 3. include_members ────────────────────────────────────── + std::string full = engine.getCommunities(project_id, 10, 20, true); + printf(" [debug] members = %s\n", full.c_str()); + CHECK(contains(full, "\"members\":[")); + CHECK(contains(full, "\"name\":\"alpha\"")); + CHECK(contains(full, "\"name\":\"foxtrot\"")); + CHECK(contains(full, "\"name\":\"isolated\"") == false); + + // ── 4. Member clamp ───────────────────────────────────────── + std::string clamped = engine.getCommunities(project_id, 2, 20, true); + printf(" [debug] clamped = %s\n", clamped.c_str()); + // Three members exist but only two may be emitted per community, so the + // response must say so instead of silently dropping them. + CHECK(contains(clamped, "\"truncated\":true") && + "member clamping must be reported via truncated"); + CHECK(!contains(clamped, "\"name\":\"gamma\"") && + "the third member of triangle A must be clipped"); + + // ── 5. Community clamp ────────────────────────────────────── + std::string one = engine.getCommunities(project_id, 10, 1, false); + printf(" [debug] one = %s\n", one.c_str()); + CHECK(contains(one, "\"total_communities\":2") && + "total_communities reports the real total"); + CHECK(contains(one, "\"returned_communities\":1")); + CHECK(contains(one, "\"truncated\":true")); + + // ── 6. Non-positive values mean "default", not "unlimited" ── + std::string defaults = engine.getCommunities(project_id, 0, 0, false); + CHECK(defaults == out && + "a non-positive limit must fall back to the default, not to an " + "unbounded response"); + + // ── 7. Empty graph yields an empty result, not an error ───── + uint64_t empty_project = store.createProject("/empty", "empty"); + std::string empty = engine.getCommunities(empty_project, 10, 20, false); + printf(" [debug] empty = %s\n", empty.c_str()); + CHECK(contains(empty, "\"total_communities\":0")); + CHECK(contains(empty, "\"communities\":[]")); + CHECK(!contains(empty, "\"error\"") && + "an empty graph is a valid answer, not an error"); + + store.close(); + unlink(kDbPath); + + printf("\n=== test_communities PASSED ===\n"); + printf("Community detection verified:\n"); + printf(" - two triangles joined by a bridge stay two communities\n"); + printf(" - representative is the highest-degree member\n"); + printf(" - self-loops / isolated nodes / non-CALLS edges are ignored\n"); + printf(" - output is deterministic and summary-first\n"); + printf(" - max_communities / max_members clamp and report truncation\n"); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_connected_components_ffi.cpp b/engine/tests/test_connected_components_ffi.cpp index 6432eac..3e42e12 100644 --- a/engine/tests/test_connected_components_ffi.cpp +++ b/engine/tests/test_connected_components_ffi.cpp @@ -13,11 +13,12 @@ // the FFI contract: valid JSON in, no crashes out. #include "../include/engine.h" -#include +#include "test_check.h" #include #include #include #include +#include "test_engine_handle.h" /// True if `haystack` contains `needle`. static bool contains(const char *haystack, const char *needle) @@ -36,22 +37,22 @@ int main() // ── Scenario 1: null/uninitialized store ────────────────────── // Do NOT call engine_init first — g_store is null. The FFI must // return error JSON instead of dereferencing the null pointer. - char *result = engine_find_connected_components(1); - assert(result != nullptr && - "null store must still return a non-null string"); - assert(looks_like_json(result) && "null store result must be JSON"); - assert(contains(result, "\"error\"") && - "null store result must contain an error field"); - assert(contains(result, "module=ffi") && - "null store error must carry the module tag"); - assert(contains(result, "method=engine_find_connected_components") && - "null store error must carry the method tag"); - assert(contains(result, "\"components\":[]") && - "null store result must contain an empty components array"); - assert(contains(result, "\"total\":0") && - "null store result must report total=0"); - assert(contains(result, "\"approximation\":\"heuristic\"") && - "null store result must carry the approximation marker"); + char *result = engine_find_connected_components(g_engine, 1); + CHECK(result != nullptr && + "null store must still return a non-null string"); + CHECK(looks_like_json(result) && "null store result must be JSON"); + CHECK(contains(result, "\"error\"") && + "null store result must contain an error field"); + CHECK(contains(result, "module=ffi") && + "null store error must carry the module tag"); + CHECK(contains(result, "method=engine_find_connected_components") && + "null store error must carry the method tag"); + CHECK(contains(result, "\"components\":[]") && + "null store result must contain an empty components array"); + CHECK(contains(result, "\"total\":0") && + "null store result must report total=0"); + CHECK(contains(result, "\"approximation\":\"heuristic\"") && + "null store result must carry the approximation marker"); engine_free_string(result); printf("[OK] null store returns tagged error JSON\n"); @@ -61,46 +62,48 @@ int main() unlink("/tmp/codescope_test_cc_ffi.db-wal"); unlink("/tmp/codescope_test_cc_ffi.db-shm"); - int rc = engine_init(db_path); - assert(rc == 0 && "engine_init should succeed"); + g_engine = engine_create(db_path); + const int rc = g_engine != nullptr ? 0 : -1; + CHECK(rc == 0 && "engine_init should succeed"); - uint64_t pid = engine_create_project("/tmp/test-cc", "test-cc"); - assert(pid > 0 && "engine_create_project should return a positive id"); + uint64_t pid = + engine_create_project(g_engine, "/tmp/test-cc", "test-cc"); + CHECK(pid > 0 && "engine_create_project should return a positive id"); - result = engine_find_connected_components(pid); - assert(result != nullptr && "initialized engine must return non-null"); - assert(looks_like_json(result) && "result must be JSON"); + result = engine_find_connected_components(g_engine, pid); + CHECK(result != nullptr && "initialized engine must return non-null"); + CHECK(looks_like_json(result) && "result must be JSON"); // No relation edges → empty components, but the envelope must be present. - assert(contains(result, "\"components\"") && - "result must contain a components field"); - assert(contains(result, "\"total\"") && - "result must contain a total field"); - assert(contains(result, "\"approximation\":\"heuristic\"") && - "result must carry the approximation marker"); - assert(contains(result, "name-matched call edges") && - "result must carry the explanatory note"); + CHECK(contains(result, "\"components\"") && + "result must contain a components field"); + CHECK(contains(result, "\"total\"") && + "result must contain a total field"); + CHECK(contains(result, "\"approximation\":\"heuristic\"") && + "result must carry the approximation marker"); + CHECK(contains(result, "name-matched call edges") && + "result must carry the explanatory note"); engine_free_string(result); printf("[OK] empty DB returns valid envelope JSON\n"); // ── Scenario 3: zero project_id (still initialized) ─────────── // project_id 0 does not exist; the inspector must not crash and // must still return the documented envelope. - result = engine_find_connected_components(0); - assert(result != nullptr && "zero project_id must return non-null"); - assert(looks_like_json(result) && - "zero project_id result must be JSON"); - assert(contains(result, "\"components\"") && - "zero project_id result must contain a components field"); - assert(contains(result, "\"total\"") && - "zero project_id result must contain a total field"); + result = engine_find_connected_components(g_engine, 0); + CHECK(result != nullptr && "zero project_id must return non-null"); + CHECK(looks_like_json(result) && "zero project_id result must be JSON"); + CHECK(contains(result, "\"components\"") && + "zero project_id result must contain a components field"); + CHECK(contains(result, "\"total\"") && + "zero project_id result must contain a total field"); engine_free_string(result); printf("[OK] zero project_id returns valid envelope JSON\n"); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; unlink(db_path); unlink("/tmp/codescope_test_cc_ffi.db-wal"); unlink("/tmp/codescope_test_cc_ffi.db-shm"); printf("\nAll connected_components FFI tests passed.\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_cpp_e2e.cpp b/engine/tests/test_cpp_e2e.cpp index ff9c6ac..43d83f8 100644 --- a/engine/tests/test_cpp_e2e.cpp +++ b/engine/tests/test_cpp_e2e.cpp @@ -1,7 +1,8 @@ #include "test_e2e.h" -int main() { - const char* code = R"(class Calculator { +int main() +{ + const char *code = R"(class Calculator { public: int add(int a, int b) { return a + b; @@ -29,12 +30,8 @@ int main() { } )"; - const char* defs[] = {"add", "Calculator", "compute"}; - runE2eTest("cpp", code, "/tmp/test_calc.cpp", - defs, 3, - "add", "multiply", - "add", - "compute", - "main", nullptr); - return 0; + const char *defs[] = { "add", "Calculator", "compute" }; + runE2eTest("cpp", code, "/tmp/test_calc.cpp", defs, 3, "add", + "multiply", "add", "compute", "main", nullptr); + return 0; } diff --git a/engine/tests/test_dispatch_order_determinism.cpp b/engine/tests/test_dispatch_order_determinism.cpp new file mode 100644 index 0000000..bae7e48 --- /dev/null +++ b/engine/tests/test_dispatch_order_determinism.cpp @@ -0,0 +1,271 @@ +// test_dispatch_order_determinism — the resolved graph must not depend on the +// order the records happen to sit in the database. +// +// Defect guarded: Step 8.1c resolves a field-chain receiver ("h.registry") by +// walking the candidate types of its first segment through the global struct +// field table, and the FIRST type whose chain resolves wins. Those candidate +// types are loaded per variable NAME — the same name is declared with different +// types in different files — and the loader had no ORDER BY, so "first" meant +// "whichever row the scan reached first", i.e. semantic_records.rowid order. +// That order is not a property of the records: the parallel scheduler +// re-assigns rowids when it merges the per-module worker databases, so it +// changes with the worker partitioning. Measured on goagent, with +// byte-identical entities, references and semantic_records: `-w 2`/`-w 4` +// resolved 7283 edges (684 of them `dispatch`) and `-w 6` resolved 7295 (696) — +// six call sites that the ambiguity gate abstained on in one ordering and that +// dispatch expanded in the other (`freeCapableAgents` → `Manager.Get` / +// `QueueRegistry.Get`). +// +// The fixture makes the order decisive in one project: +// iface.go Getter (interface, method Get) + Registry.Get implements it +// holder_a.go HolderA.registry is a Getter → the walk resolves the +// receiver to an INTERFACE → dispatch expansion to Registry.Get +// holder_b.go HolderB.registry is a *Registry → the walk resolves it to a +// CONCRETE type → the same call gets no dispatch edge +// Both files declare a parameter named `h`, so the walk's candidate list for +// `h` is {HolderA, HolderB} and its order decides which of the two outcomes the +// call site gets. The test indexes the fixture, then REORDERS those two rows +// (the same project, same records, rowids swapped — what a different module +// packing does to them) and re-runs the resolver: the graph must not move. +// +// The reorder is proven to be effective three ways: the row order is read back +// and asserted different, a marker edge injected before the rebuild must be +// gone after it (so a skipped rebuild cannot make the comparison vacuous), and +// the expected dispatch edge is pinned explicitly. +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include + +#include "test_check.h" +#include "test_engine_handle.h" + +namespace +{ + +/// Write `content` to `path`, failing the test if the file cannot be created. +void writeFile(const std::string &path, const char *content) +{ + FILE *f = fopen(path.c_str(), "w"); + CHECK(f != nullptr); + fputs(content, f); + fclose(f); +} + +/// Run `sql`, expecting success. +void execSql(sqlite3 *db, const char *sql) +{ + char *err = nullptr; + const int rc = sqlite3_exec(db, sql, nullptr, nullptr, &err); + if (rc != SQLITE_OK) { + fprintf(stderr, "FAIL: sqlite3_exec: %s\n %s\n", err ? err : "?", + sql); + if (err) + sqlite3_free(err); + } + CHECK(rc == SQLITE_OK); +} + +/// Every resolved call edge as one canonical line each — caller, callee and the +/// resolution kind, which is what differs between the two outcomes above. +std::string edgeFingerprint(sqlite3 *db) +{ + const char *sql = + "SELECT se.name || '>' || te.name || '|' || r.resolution_kind " + "FROM relation r " + "JOIN entity se ON se.id = r.source_id " + "JOIN entity te ON te.id = r.target_id " + "WHERE r.type = 1 ORDER BY 1"; + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + std::string out; + while (sqlite3_step(st) == SQLITE_ROW) { + const char *s = reinterpret_cast( + sqlite3_column_text(st, 0)); + if (s) + out += std::string(s) + "\n"; + } + sqlite3_finalize(st); + return out; +} + +/// Type of the FIRST record named `name` (kind is passed in so the fixture's +/// variable rows can be told apart from anything else). +std::string firstTypeOf(sqlite3 *db, const char *name, int kind) +{ + sqlite3_stmt *st = nullptr; + const char *sql = "SELECT type_name FROM semantic_records " + "WHERE name=? AND kind=? ORDER BY rowid"; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + sqlite3_bind_text(st, 1, name, -1, SQLITE_TRANSIENT); + sqlite3_bind_int(st, 2, kind); + std::string out; + if (sqlite3_step(st) == SQLITE_ROW) { + const char *s = reinterpret_cast( + sqlite3_column_text(st, 0)); + out = s ? s : ""; + } + sqlite3_finalize(st); + return out; +} + +} // namespace + +int main() +{ + const std::string proj_dir = "/tmp/dispatch_order_repro"; + std::filesystem::remove_all(proj_dir); + std::filesystem::create_directories(proj_dir + "/src"); + + writeFile(proj_dir + "/src/iface.go", + "package p\n" + "\n" + "type Getter interface {\n" + "\tGet(id string) error\n" + "}\n" + "\n" + "type Registry struct{}\n" + "\n" + "func (r *Registry) Get(id string) error { return nil }\n"); + writeFile(proj_dir + "/src/holders.go", + "package p\n" + "\n" + "type HolderA struct {\n" + "\tregistry Getter\n" + "}\n" + "\n" + "type HolderB struct {\n" + "\tregistry *Registry\n" + "}\n"); + // The call sites sit in files of their own: the Visitor resolves field + // types per file, so `h.registry` is only resolvable through the global + // struct-field table (Step 8.1c) when the struct is declared elsewhere. + writeFile(proj_dir + "/src/use_a.go", + "package p\n" + "\n" + "func useA(h HolderA, id string) error {\n" + "\treturn h.registry.Get(id)\n" + "}\n"); + writeFile(proj_dir + "/src/use_b.go", + "package p\n" + "\n" + "func useB(h HolderB, id string) error {\n" + "\treturn h.registry.Get(id)\n" + "}\n"); + + const std::string db_path = proj_dir + "/order.db"; + unlink(db_path.c_str()); + unlink((db_path + "-wal").c_str()); + unlink((db_path + "-shm").c_str()); + + g_engine = engine_create(db_path.c_str()); + CHECK(g_engine != nullptr); + const uint64_t pid = engine_create_project(g_engine, proj_dir.c_str(), + "dispatch-order"); + CHECK(pid > 0); + + const std::string request = "{\"paths\":[\"" + proj_dir + + "/src/iface.go\",\"" + proj_dir + + "/src/holders.go\",\"" + proj_dir + + "/src/use_a.go\",\"" + proj_dir + + "/src/use_b.go\"]}"; + char *res = engine_index_files(g_engine, pid, request.c_str(), 1); + CHECK(res != nullptr); + CHECK(strstr(res, "\"ok\":true") != nullptr); + engine_free_string(res); + // engine_destroy() joins the async knowledge builder before closing the + // store (engine_lifecycle.cpp), so reading the database directly afterwards + // is safe; every table asserted below is written synchronously by the index + // path anyway. + engine_destroy(g_engine); + g_engine = nullptr; + + sqlite3 *db = nullptr; + CHECK(sqlite3_open(db_path.c_str(), &db) == SQLITE_OK); + + // ── The fixture has to resolve something ────────────────────── + // `h` is declared as HolderA in one file and HolderB in the other, so the + // type table holds both spellings for one name: that is the ambiguity the + // walk used to settle by row order. + const std::string before_type = firstTypeOf(db, "h", 17); + CHECK(before_type == "HolderA" || before_type == "HolderB"); + const std::string before = edgeFingerprint(db); + // Which order the parse workers happened to write is not the point (and is + // itself what makes the graph flaky); the fixture has to have resolved + // SOMETHING, or the comparison below is vacuous. + CHECK(!before.empty()); + + // ── Reorder the records ─────────────────────────────────────── + // Negating every rowid reverses the order a scan sees, leaving the records + // themselves untouched — the same effect a different module packing has on + // the merged database. The whole table, not just the two `h` rows: the + // loaders read through a self-join, so which table drives the plan decides + // whose order reaches the candidate vectors. + execSql(db, "UPDATE semantic_records SET rowid = -rowid;"); + const std::string after_type = firstTypeOf(db, "h", 17); + CHECK(after_type != before_type); // the lever really moved the order + + // A marker edge that the rebuild must remove, so a skipped rebuild cannot + // make the comparison below pass for the wrong reason. + execSql(db, "INSERT INTO relation (project_id, source_id, target_id, type, " + "confidence, resolver, resolution_kind, reason) " + "SELECT 1, (SELECT id FROM entity WHERE name='useA' LIMIT 1), " + "(SELECT id FROM entity WHERE name='Get' LIMIT 1), 1, " + "0.1, 'test', 'probe-kind', 'probe';"); + execSql(db, "DELETE FROM project_readiness;"); + sqlite3_close(db); + + // ── Re-run the resolver on the reordered records ────────────── + g_engine = engine_create(db_path.c_str()); + CHECK(g_engine != nullptr); + char *enh = engine_enhance_project(g_engine, pid); + CHECK(enh != nullptr); + CHECK(strstr(enh, "\"ok\":true") != nullptr); + engine_free_string(enh); + engine_destroy(g_engine); + g_engine = nullptr; + + CHECK(sqlite3_open(db_path.c_str(), &db) == SQLITE_OK); + // The rebuild ran (the marker is gone) and the graph did not move. + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, + "SELECT COUNT(*) FROM relation " + "WHERE resolution_kind='probe'", + -1, &st, nullptr) == SQLITE_OK); + CHECK(sqlite3_step(st) == SQLITE_ROW); + const long long probe_left = sqlite3_column_int64(st, 0); + sqlite3_finalize(st); + CHECK(probe_left == 0); + + const std::string after = edgeFingerprint(db); + // The pinned outcome: with the candidate types ordered by name the walk + // resolves `h` to HolderA, whose `registry` is a Getter, so both call sites + // expand to the single implementation instead of falling to the ambiguity + // gate. Pinning it keeps the equality check below from passing on two + // equally empty graphs. + CHECK(after.find("useA>Get|dispatch") != std::string::npos); + CHECK(after.find("useB>Get|dispatch") != std::string::npos); + if (before != after) { + fprintf(stderr, + "FAIL: the resolved edges follow the record order.\n" + " before reorder (first `h` type = %s):\n%s" + " after reorder (first `h` type = %s):\n%s", + before_type.c_str(), before.c_str(), after_type.c_str(), + after.c_str()); + } + CHECK(before == after); + + sqlite3_close(db); + unlink(db_path.c_str()); + if (checkFailures()) { + fprintf(stderr, "=== dispatch order determinism: FAILED ===\n"); + return 1; + } + printf("=== dispatch order determinism test passed ===\n"); + return 0; +} diff --git a/engine/tests/test_documentation_drift.cpp b/engine/tests/test_documentation_drift.cpp index cf8c4c8..502eadc 100644 --- a/engine/tests/test_documentation_drift.cpp +++ b/engine/tests/test_documentation_drift.cpp @@ -15,7 +15,7 @@ #include "../src/store/store.h" #include "../src/verify/documentation_drift.h" -#include +#include "test_check.h" #include #include #include @@ -33,10 +33,10 @@ static void insertReadme(store::GraphStore &store, uint64_t project_id, "content, start_line, end_line) " "VALUES (?,0,'/README.md',?,0,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, content, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -50,12 +50,12 @@ static void insertEntity(store::GraphStore &store, uint64_t project_id, "start_col, end_row, end_col) " "VALUES (?,?,0,?,'','/test.cpp',?,0,0,0,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, language, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -74,49 +74,52 @@ int main() unlink(kDbPath); store::GraphStore store; - assert(store.open(kDbPath)); + CHECK(store.open(kDbPath)); uint64_t project_id = store.createProject("/test", "test_doc_drift"); - assert(project_id > 0); + CHECK(project_id > 0); - // ── Test 1: extractLanguageClaims — basic multi-language README ── + // ── Test 1: extractLanguageClaims — capability vs project claim ── + // "Supports X" says the tool can handle X. Only "Written in X" claims this + // project is written in X. Counting both made the detector report every + // language the tool parses as documented-but-absent from its own code. { - std::string readme = - "# My Project\n" - "Supports C++, Python, Rust, and Go.\n" - "Written in C++ and Rust primarily.\n"; + std::string readme = "# My Project\n" + "Supports C++, Python, Rust, and Go.\n" + "Written in C++ and Rust primarily.\n"; auto claims = extractLanguageClaims(readme); - assert(claims.size() >= 3); + CHECK(claims.size() == 2); const auto *cpp = findClaim(claims, "cpp"); - assert(cpp != nullptr); - assert(cpp->display == "C++"); - assert(cpp->mention_count >= 2); // "C++" appears twice - - const auto *py = findClaim(claims, "python"); - assert(py != nullptr); - assert(py->display == "Python"); - - const auto *go = findClaim(claims, "go"); - assert(go != nullptr); - assert(go->display == "Go"); + CHECK(cpp != nullptr); + CHECK(cpp->display == "C++"); + // Only the "Written in" mention is a claim; the "Supports" one is a + // capability statement about the tool. + CHECK(cpp->mention_count == 1); const auto *rust = findClaim(claims, "rust"); - assert(rust != nullptr); - assert(rust->display == "Rust"); + CHECK(rust != nullptr); + CHECK(rust->display == "Rust"); + CHECK(rust->mention_count == 1); + + // Python and Go are mentioned only as things the tool supports. + CHECK(findClaim(claims, "python") == nullptr); + CHECK(findClaim(claims, "go") == nullptr); - printf(" [PASS] extractLanguageClaims: detected C++, Python, Go, Rust\n"); + printf(" [PASS] extractLanguageClaims: capability list vetoed, project " + "claims (C++, Rust) kept\n"); } // ── Test 2: extractLanguageClaims — "Go" word-boundary ────────── // "Google" and "Going" should NOT trigger a Go claim, but standalone // "Go" should. { - std::string readme = "Powered by Google. Going forward, we use Go."; + std::string readme = + "Powered by Google. Going forward, we use Go."; auto claims = extractLanguageClaims(readme); const auto *go = findClaim(claims, "go"); - assert(go != nullptr); - assert(go->mention_count >= 1); // "Go" at end of sentence + CHECK(go != nullptr); + CHECK(go->mention_count >= 1); // "Go" at end of sentence printf(" [PASS] extractLanguageClaims: Go word-boundary (Google/Going excluded)\n"); } @@ -126,15 +129,15 @@ int main() std::string readme = "The cpp parser is fast."; auto claims = extractLanguageClaims(readme); const auto *cpp = findClaim(claims, "cpp"); - assert(cpp != nullptr); - assert(cpp->display == "C++"); + CHECK(cpp != nullptr); + CHECK(cpp->display == "C++"); printf(" [PASS] extractLanguageClaims: cpp alias -> C++\n"); } // ── Test 4: extractLanguageClaims — empty text ────────────────── { auto claims = extractLanguageClaims(""); - assert(claims.empty()); + CHECK(claims.empty()); printf(" [PASS] extractLanguageClaims: empty text -> no claims\n"); } @@ -142,10 +145,69 @@ int main() { std::string readme = "This is a project about databases."; auto claims = extractLanguageClaims(readme); - assert(claims.empty()); + CHECK(claims.empty()); printf(" [PASS] extractLanguageClaims: no languages -> no claims\n"); } + // ── Test 5a: extractLanguageClaims — a capability matrix is not a claim ── + // The shape this repository's own README has. Every row of a "Supported + // Languages" table says the tool can parse that language; none of them says + // THIS project is written in it. + { + std::string readme = "# Tool\n" + "### Supported Languages (3)\n" + "\n" + "| Language | Parser | Verified |\n" + "|----------|--------|----------|\n" + "| Python | yes | yes |\n" + "| Go | yes | yes |\n" + "| Rust | yes | yes |\n" + "\n" + "### Tech Stack\n" + "\n" + "| Layer | Technology |\n" + "|-------|------------|\n" + "| Core | C++ |\n"; + auto claims = extractLanguageClaims(readme); + CHECK(findClaim(claims, "python") == nullptr); + CHECK(findClaim(claims, "go") == nullptr); + CHECK(findClaim(claims, "rust") == nullptr); + // The Tech Stack row is a statement about this project, so it stays. + CHECK(findClaim(claims, "cpp") != nullptr); + printf(" [PASS] extractLanguageClaims: capability matrix vetoed, " + "tech-stack claim kept\n"); + } + + // ── Test 5b: extractLanguageClaims — a table about other projects ── + // A benchmark table's `Project` column lists third-party projects, so its + // Language column describes those projects, not this repository. + { + std::string readme = "# Benchmarks\n" + "\n" + "| Project | Language | Index Time |\n" + "|---------|----------|-----------:|\n" + "| tinygo | Go | 1.77 s |\n" + "| rustc | Rust | 38.9 s |\n"; + auto claims = extractLanguageClaims(readme); + CHECK(findClaim(claims, "go") == nullptr); + CHECK(findClaim(claims, "rust") == nullptr); + printf(" [PASS] extractLanguageClaims: third-party benchmark table " + "vetoed\n"); + } + + // ── Test 5c: "parser" in prose is not a capability cue ────────── + // The capability cues are for COLUMN NAMES and capability phrases. A + // sentence about this project's own parser must still count, otherwise the + // veto would be broad enough to hide real drift. + { + std::string readme = "The cpp parser is fast."; + auto claims = extractLanguageClaims(readme); + const auto *cpp = findClaim(claims, "cpp"); + CHECK(cpp != nullptr); + CHECK(cpp->mention_count == 1); + printf(" [PASS] extractLanguageClaims: prose 'parser' keeps the claim\n"); + } + // ── Test 6: countEntitiesByLanguage ───────────────────────────── { // Insert 3 C++ entities and 2 Python entities. @@ -155,10 +217,11 @@ int main() insertEntity(store, project_id, 200, "main", "python"); insertEntity(store, project_id, 201, "helper", "python"); - assert(countEntitiesByLanguage(store, project_id, "cpp") == 3); - assert(countEntitiesByLanguage(store, project_id, "python") == 2); - assert(countEntitiesByLanguage(store, project_id, "go") == 0); - assert(countEntitiesByLanguage(store, project_id, "rust") == 0); + CHECK(countEntitiesByLanguage(store, project_id, "cpp") == 3); + CHECK(countEntitiesByLanguage(store, project_id, "python") == + 2); + CHECK(countEntitiesByLanguage(store, project_id, "go") == 0); + CHECK(countEntitiesByLanguage(store, project_id, "rust") == 0); printf(" [PASS] countEntitiesByLanguage: cpp=3, python=2, go=0, rust=0\n"); } @@ -171,22 +234,25 @@ int main() sqlite3_exec(db, "DELETE FROM document WHERE project_id > 0", nullptr, nullptr, nullptr); + // A project claim, not a capability list: "Written in" binds the + // languages to this project. (A "Supports …" line would be vetoed and + // correctly produce zero claims and zero drifts.) insertReadme(store, project_id, - "Supports C++, Python, Go, and Rust."); + "Written in C++, Python, Go, and Rust."); auto drifts = detectDocumentationDrift(store, project_id); - assert(drifts.size() == 2); // Go + Rust missing + CHECK(drifts.size() == 2); // Go + Rust missing bool has_go = false, has_rust = false; for (const auto &d : drifts) { - assert(d.type == "DocumentationDrift"); - assert(d.severity == kDriftSeverityDoc); + CHECK(d.type == "DocumentationDrift"); + CHECK(d.severity == kDriftSeverityDoc); if (d.subject == "Go") has_go = true; if (d.subject == "Rust") has_rust = true; } - assert(has_go); - assert(has_rust); + CHECK(has_go); + CHECK(has_rust); printf(" [PASS] detectDocumentationDrift: Go + Rust missing (2 drifts)\n"); } @@ -198,8 +264,8 @@ int main() // Go still has zero entities. // Re-run detection — now only Go should be missing. auto drifts = detectDocumentationDrift(store, project_id); - assert(drifts.size() == 1); - assert(drifts[0].subject == "Go"); + CHECK(drifts.size() == 1); + CHECK(drifts[0].subject == "Go"); printf(" [PASS] detectDocumentationDrift: Rust added, only Go missing (1 drift)\n"); } @@ -210,7 +276,7 @@ int main() nullptr, nullptr, nullptr); // No README inserted → no claims → no drifts. auto drifts = detectDocumentationDrift(store, project_id); - assert(drifts.empty()); + CHECK(drifts.empty()); printf(" [PASS] detectDocumentationDrift: empty README -> no drifts\n"); } @@ -218,5 +284,5 @@ int main() unlink(kDbPath); printf("=== test_documentation_drift PASSED ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_domain_rules.cpp b/engine/tests/test_domain_rules.cpp index 0889786..93633a3 100644 --- a/engine/tests/test_domain_rules.cpp +++ b/engine/tests/test_domain_rules.cpp @@ -33,7 +33,7 @@ #include "../src/evidence/evidence_builder.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -61,13 +61,13 @@ static void insertFunction(GraphStore &store, uint64_t project_id, int64_t id, "start_row, start_col, end_row, end_col) " "VALUES (?,?,0,0,?,'',?,?,1,0,1000,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, file_path, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 5, language, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -86,7 +86,7 @@ static void insertFact(GraphStore &store, uint64_t project_id, " symbol, confidence, detail_json) " "VALUES (?,?,?,?,?,?,?,?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, static_cast(function_id)); sqlite3_bind_text(stmt, 3, category, -1, SQLITE_TRANSIENT); @@ -98,7 +98,7 @@ static void insertFact(GraphStore &store, uint64_t project_id, sqlite3_bind_text(stmt, 8, detail_json, -1, SQLITE_TRANSIENT); else sqlite3_bind_null(stmt, 8); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -120,11 +120,11 @@ static std::string detailJson(int line, const std::string &snippet, static std::string findRulesDir() { const char *candidates[] = { - "engine/src/evidence/rules", - "../src/evidence/rules", - "../../engine/src/evidence/rules", - "../../../engine/src/evidence/rules", - }; + "engine/src/evidence/rules", + "../src/evidence/rules", + "../../engine/src/evidence/rules", + "../../../engine/src/evidence/rules", + }; for (const char *cand : candidates) { std::error_code ec; if (!std::filesystem::is_directory(cand, ec)) @@ -166,7 +166,7 @@ int main() return 1; } uint64_t pid = store.createProject("/tmp", "test_domain_rules"); - assert(pid > 0); + CHECK(pid > 0); // ── Insert one function per test fact (different function_ids // so missing_match's per-function exclusion logic is exercised @@ -240,7 +240,7 @@ int main() printf("Loaded %zu rule sets\n", builder.ruleSets().size()); // ── Test 1: at least 10 rule sets (6 existing + 4 new) ─────── - assert(builder.ruleSets().size() >= 10); + CHECK(builder.ruleSets().size() >= 10); printf("Test 1 (rule sets >= 10, got %zu): PASS\n", builder.ruleSets().size()); @@ -251,17 +251,17 @@ int main() return true; return false; }; - assert(hasCategory("drift")); - assert(hasCategory("security")); - assert(hasCategory("concurrency")); - assert(hasCategory("test_quality")); + CHECK(hasCategory("drift")); + CHECK(hasCategory("security")); + CHECK(hasCategory("concurrency")); + CHECK(hasCategory("test_quality")); printf("Test 2 (drift/security/concurrency/test_quality " "categories present): PASS\n"); // ── Test 3: buildAll returns evidence from each new category ─ auto all = builder.buildAll(pid); printf("buildAll returned %zu evidence(s)\n", all.size()); - assert(!all.empty()); + CHECK(!all.empty()); size_t drift_n = countByCategory(all, "drift"); size_t security_n = countByCategory(all, "security"); @@ -271,10 +271,10 @@ int main() "test_quality=%zu\n", drift_n, security_n, concurrency_n, test_quality_n); - assert(drift_n >= 1); - assert(security_n >= 1); - assert(concurrency_n >= 1); - assert(test_quality_n >= 1); + CHECK(drift_n >= 1); + CHECK(security_n >= 1); + CHECK(concurrency_n >= 1); + CHECK(test_quality_n >= 1); printf("Test 3 (each new category produced >= 1 evidence): " "PASS\n"); @@ -282,9 +282,9 @@ int main() // concurrency evidence, and at least 1 item. // ────────────────────────────────────────────────────────── auto conc_evs = builder.buildByCategory(pid, "concurrency"); - assert(!conc_evs.empty()); + CHECK(!conc_evs.empty()); for (const auto &ev : conc_evs) - assert(ev.category == "concurrency"); + CHECK(ev.category == "concurrency"); printf("Test 4 (buildByCategory concurrency, %zu evidence, " "all category=concurrency): PASS\n", conc_evs.size()); @@ -297,12 +297,12 @@ int main() for (const auto &item : ev.items) { if (item.symbol == "m.Lock") { found_mutex = true; - assert(item.file == "/src/lock_leak.go"); - assert(item.line == 5); + CHECK(item.file == "/src/lock_leak.go"); + CHECK(item.line == 5); } } } - assert(found_mutex); + CHECK(found_mutex); printf("Test 5 (concurrency evidence contains m.Lock at " "/src/lock_leak.go:5): PASS\n"); @@ -314,8 +314,8 @@ int main() std::string("concurrency"), std::string("test_quality") }) { auto evs = builder.buildByCategory(pid, cat); for (const auto &ev : evs) - assert(ev.category == cat); - assert(!evs.empty()); + CHECK(ev.category == cat); + CHECK(!evs.empty()); } printf("Test 6 (buildByCategory isolates each new category): " "PASS\n"); @@ -326,7 +326,7 @@ int main() "mutex_without_unlock", "unwrap_in_non_test", "todo_accumulation" }) { auto evs = builder.buildByRule(pid, rule_name); - assert(!evs.empty()); + CHECK(!evs.empty()); } printf("Test 7 (buildByRule for each new rule returns " "evidence): PASS\n"); @@ -334,12 +334,12 @@ int main() // ── Test 8: buildByRule for unknown rule returns 0 ────────── { auto evs = builder.buildByRule(pid, "nonexistent_rule_xyz"); - assert(evs.empty()); + CHECK(evs.empty()); } printf("Test 8 (buildByRule unknown, 0 evidence): PASS\n"); store.close(); unlink(kDbPath); printf("\nAll domain_rules tests passed.\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_e2e.cpp b/engine/tests/test_e2e.cpp index ddf9aa0..2240b4f 100644 --- a/engine/tests/test_e2e.cpp +++ b/engine/tests/test_e2e.cpp @@ -1,7 +1,8 @@ #include "test_e2e.h" -int main() { - const char* code = R"PY( +int main() +{ + const char *code = R"PY( # calculator.py — A simple calculator module def add(a, b): @@ -44,12 +45,8 @@ if __name__ == "__main__": main() )PY"; - const char* defs[] = {"add", "Calculator", nullptr}; - runE2eTest("python", code, "/tmp/calculator.py", - defs, 2, - "add", "multiply", - "add", - "multiply", - "main", "Calculator"); - return 0; + const char *defs[] = { "add", "Calculator", nullptr }; + runE2eTest("python", code, "/tmp/calculator.py", defs, 2, "add", + "multiply", "add", "multiply", "main", "Calculator"); + return 0; } diff --git a/engine/tests/test_e2e.h b/engine/tests/test_e2e.h index 5c85517..fda57e9 100644 --- a/engine/tests/test_e2e.h +++ b/engine/tests/test_e2e.h @@ -10,21 +10,27 @@ #include #include #include +#include "test_engine_handle.h" // Check a condition and exit on failure -static inline void check(bool cond, const char* msg) { - if (!cond) { fprintf(stderr, "FAIL: %s\n", msg); exit(1); } +static inline void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "FAIL: %s\n", msg); + exit(1); + } } // Print a labeled JSON result -static inline void print_json(const char* label, const char* json) { - printf("\n--- %s (%zu chars) ---\n%s\n", label, strlen(json), json); +static inline void print_json(const char *label, const char *json) +{ + printf("\n--- %s (%zu chars) ---\n%s\n", label, strlen(json), json); } /** * Run the standard E2E test pipeline for a given language: - * init → create_project → index_file → find_definition × 3 → find_references - * → get_callers → get_callees → get_graph_stats → locate_by_name × 2 → shutdown + * create → create_project → index_file → find_definition × 3 → find_references + * → get_callers → get_callees → get_graph_stats → locate_by_name × 2 → destroy * * @param lang Language name (used for db file, project name) * @param code Source code string (embedded as-is) @@ -38,103 +44,107 @@ static inline void print_json(const char* label, const char* json) { * @param locate_name Symbol name to look up via locate_by_name * @param locate_extra Additional symbol name to locate (or nullptr) */ -static inline void runE2eTest( - const char* lang, const char* code, const char* file_path, - const char** def_names, int def_count, - const char* ref_name, const char* ref_caller, - const char* caller_name, - const char* callee_name, - const char* locate_name, - const char* locate_extra) +static inline void runE2eTest(const char *lang, const char *code, + const char *file_path, const char **def_names, + int def_count, const char *ref_name, + const char *ref_caller, const char *caller_name, + const char *callee_name, const char *locate_name, + const char *locate_extra) { - char db_path[64]; - snprintf(db_path, sizeof(db_path), "/tmp/test_%s.db", lang); - - unlink(db_path); - - int rc = engine_init(db_path); - check(rc == 0, "engine_init"); - - // Write test file - FILE* f = fopen(file_path, "w"); - check(f != nullptr, "fopen"); - fwrite(code, 1, strlen(code), f); - fclose(f); - - // Create project - char proj_name[64]; - snprintf(proj_name, sizeof(proj_name), "%s-test", lang); - uint64_t pid = engine_create_project("/tmp", proj_name); - check(pid > 0, "create_project"); - - // Index - char* result = engine_index_file(pid, file_path); - print_json("Index", result); - check(strstr(result, "\"ok\":true") != nullptr, "index_file ok"); - engine_free_string(result); - - // Find definitions - for (int i = 0; i < def_count; i++) { - char* def = engine_find_definition(pid, def_names[i], nullptr); - char label[128]; - snprintf(label, sizeof(label), "Definition: %s", def_names[i]); - print_json(label, def); - check(strstr(def, def_names[i]) != nullptr, "find_def"); - engine_free_string(def); - } - - // References - char* refs = engine_find_references(pid, ref_name, nullptr); - char ref_label[128]; - snprintf(ref_label, sizeof(ref_label), "References: %s", ref_name); - print_json(ref_label, refs); - check(strstr(refs, ref_caller) != nullptr, "find_refs has caller"); - engine_free_string(refs); - - // Callers - char* callers = engine_get_callers(pid, caller_name, nullptr); - char caller_label[128]; - snprintf(caller_label, sizeof(caller_label), "Callers: %s", caller_name); - print_json(caller_label, callers); - check(strstr(callers, "callers") != nullptr, "get_callers"); - engine_free_string(callers); - - // Callees - char* callees = engine_get_callees(pid, callee_name, nullptr); - char callee_label[128]; - snprintf(callee_label, sizeof(callee_label), "Callees: %s", callee_name); - print_json(callee_label, callees); - check(strstr(callees, "callees") != nullptr, "get_callees"); - engine_free_string(callees); - - // Stats - char* stats = engine_get_graph_stats(pid); - print_json("Graph Stats", stats); - check(strstr(stats, "total_nodes") != nullptr, "get_graph_stats"); - engine_free_string(stats); - - // Locate - char* locate = engine_locate_by_name(pid, locate_name); - char locate_label[128]; - snprintf(locate_label, sizeof(locate_label), "Locate: %s", locate_name); - print_json(locate_label, locate); - check(strstr(locate, locate_name) != nullptr, "locate"); - engine_free_string(locate); - - if (locate_extra) { - locate = engine_locate_by_name(pid, locate_extra); - snprintf(locate_label, sizeof(locate_label), "Locate: %s", locate_extra); - print_json(locate_label, locate); - check(strstr(locate, locate_extra) != nullptr, "locate extra"); - engine_free_string(locate); - } - - engine_shutdown(); - - printf("\n=== %s E2E test passed ===\n", lang); - } - - /** + char db_path[64]; + snprintf(db_path, sizeof(db_path), "/tmp/test_%s.db", lang); + + unlink(db_path); + + g_engine = engine_create(db_path); + const int rc = g_engine != nullptr ? 0 : -1; + check(rc == 0, "engine_init"); + + // Write test file + FILE *f = fopen(file_path, "w"); + check(f != nullptr, "fopen"); + fwrite(code, 1, strlen(code), f); + fclose(f); + + // Create project + char proj_name[64]; + snprintf(proj_name, sizeof(proj_name), "%s-test", lang); + uint64_t pid = engine_create_project(g_engine, "/tmp", proj_name); + check(pid > 0, "create_project"); + + // Index + char *result = engine_index_file(g_engine, pid, file_path); + print_json("Index", result); + check(strstr(result, "\"ok\":true") != nullptr, "index_file ok"); + engine_free_string(result); + + // Find definitions + for (int i = 0; i < def_count; i++) { + char *def = engine_find_definition(g_engine, pid, def_names[i], + nullptr); + char label[128]; + snprintf(label, sizeof(label), "Definition: %s", def_names[i]); + print_json(label, def); + check(strstr(def, def_names[i]) != nullptr, "find_def"); + engine_free_string(def); + } + + // References + char *refs = engine_find_references(g_engine, pid, ref_name, nullptr); + char ref_label[128]; + snprintf(ref_label, sizeof(ref_label), "References: %s", ref_name); + print_json(ref_label, refs); + check(strstr(refs, ref_caller) != nullptr, "find_refs has caller"); + engine_free_string(refs); + + // Callers + char *callers = engine_get_callers(g_engine, pid, caller_name, nullptr); + char caller_label[128]; + snprintf(caller_label, sizeof(caller_label), "Callers: %s", + caller_name); + print_json(caller_label, callers); + check(strstr(callers, "callers") != nullptr, "get_callers"); + engine_free_string(callers); + + // Callees + char *callees = engine_get_callees(g_engine, pid, callee_name, nullptr); + char callee_label[128]; + snprintf(callee_label, sizeof(callee_label), "Callees: %s", + callee_name); + print_json(callee_label, callees); + check(strstr(callees, "callees") != nullptr, "get_callees"); + engine_free_string(callees); + + // Stats + char *stats = engine_get_graph_stats(g_engine, pid); + print_json("Graph Stats", stats); + check(strstr(stats, "total_nodes") != nullptr, "get_graph_stats"); + engine_free_string(stats); + + // Locate + char *locate = engine_locate_by_name(g_engine, pid, locate_name); + char locate_label[128]; + snprintf(locate_label, sizeof(locate_label), "Locate: %s", locate_name); + print_json(locate_label, locate); + check(strstr(locate, locate_name) != nullptr, "locate"); + engine_free_string(locate); + + if (locate_extra) { + locate = engine_locate_by_name(g_engine, pid, locate_extra); + snprintf(locate_label, sizeof(locate_label), "Locate: %s", + locate_extra); + print_json(locate_label, locate); + check(strstr(locate, locate_extra) != nullptr, "locate extra"); + engine_free_string(locate); + } + + engine_destroy(g_engine); + g_engine = nullptr; + + printf("\n=== %s E2E test passed ===\n", lang); +} + +/** * Run FP-verification E2E: index code with builtin/FFI calls, then verify * that builtin functions do NOT create false call edges, and that language * visibility rules (e.g. Go unexported) are enforced. @@ -147,88 +157,97 @@ static inline void runE2eTest( * @param builtin_names Array of builtin function names that should NOT appear * as callees (nullptr-terminated) */ - static inline void runFPVerificationTest( - const char* lang, const char* code, const char* file_path, - const char* main_func, const char* legit_callee, - const char** builtin_names) - { - char db_path[64]; - snprintf(db_path, sizeof(db_path), "/tmp/test_fp_%s.db", lang); - - unlink(db_path); - - int rc = engine_init(db_path); - check(rc == 0, "engine_init"); - - // Write test file - FILE* f = fopen(file_path, "w"); - check(f != nullptr, "fopen"); - fwrite(code, 1, strlen(code), f); - fclose(f); - - // Create project - char proj_name[64]; - snprintf(proj_name, sizeof(proj_name), "%s-fp-test", lang); - uint64_t pid = engine_create_project("/tmp", proj_name); - check(pid > 0, "create_project"); - - // Index - char* result = engine_index_file(pid, file_path); - print_json("Index", result); - check(strstr(result, "\"ok\":true") != nullptr, "index_file ok"); - engine_free_string(result); - - // Enhance: build the call graph so callee lookups work - char* enh = engine_enhance_project(pid); - engine_free_string(enh); - - // Get callees of the main function — this is where we check for FPs - char* callees = engine_get_callees(pid, main_func, nullptr); - print_json("Callees of main function", callees); - - // 1. Legitimate callee must be present - check(strstr(callees, legit_callee) != nullptr, "legit callee present"); - - // 2. Builtin names must NOT appear as callees - for (int i = 0; builtin_names[i] != nullptr; i++) { - char buf[256]; - snprintf(buf, sizeof(buf), "\"name\":\"%s\"", builtin_names[i]); - // The callee should NOT include this builtin - if (strstr(callees, buf) != nullptr) { - fprintf(stderr, "FAIL: builtin '%s' should NOT appear in callees of mainFunc\n", - builtin_names[i]); - fprintf(stderr, " callees JSON: %s\n", callees); - exit(1); - } - printf(" [PASS] builtin '%s' correctly excluded from callees\n", builtin_names[i]); - } - engine_free_string(callees); - - // Get references to verify builtin calls don't create reference entries - for (int i = 0; builtin_names[i] != nullptr; i++) { - char* refs = engine_find_references(pid, builtin_names[i], nullptr); - // Builtin references should have zero results (no user func with that name) - if (strstr(refs, "\"total\":0") == nullptr && strstr(refs, "\"results\":[]") == nullptr) { - // If there ARE results, they must not include mainFunc as a caller - if (strstr(refs, "mainFunc") != nullptr) { - fprintf(stderr, "FAIL: builtin '%s' should NOT have references from mainFunc\n", - builtin_names[i]); - fprintf(stderr, " refs JSON: %s\n", refs); - exit(1); - } - } - printf(" [PASS] builtin '%s' reference check passed\n", builtin_names[i]); - engine_free_string(refs); - } - - // Stats check: total edges should be reasonable (no FP edges from builtins) - char* stats = engine_get_graph_stats(pid); - print_json("Graph Stats", stats); - check(strstr(stats, "total_nodes") != nullptr, "get_graph_stats"); - engine_free_string(stats); - - engine_shutdown(); - printf("\n=== %s FP verification test passed ===\n", lang); - } - - #endif // TEST_E2E_H +static inline void runFPVerificationTest(const char *lang, const char *code, + const char *file_path, + const char *main_func, + const char *legit_callee, + const char **builtin_names) +{ + char db_path[64]; + snprintf(db_path, sizeof(db_path), "/tmp/test_fp_%s.db", lang); + + unlink(db_path); + + g_engine = engine_create(db_path); + const int rc = g_engine != nullptr ? 0 : -1; + check(rc == 0, "engine_init"); + + // Write test file + FILE *f = fopen(file_path, "w"); + check(f != nullptr, "fopen"); + fwrite(code, 1, strlen(code), f); + fclose(f); + + // Create project + char proj_name[64]; + snprintf(proj_name, sizeof(proj_name), "%s-fp-test", lang); + uint64_t pid = engine_create_project(g_engine, "/tmp", proj_name); + check(pid > 0, "create_project"); + + // Index + char *result = engine_index_file(g_engine, pid, file_path); + print_json("Index", result); + check(strstr(result, "\"ok\":true") != nullptr, "index_file ok"); + engine_free_string(result); + + // Enhance: build the call graph so callee lookups work + char *enh = engine_enhance_project(g_engine, pid); + engine_free_string(enh); + + // Get callees of the main function — this is where we check for FPs + char *callees = engine_get_callees(g_engine, pid, main_func, nullptr); + print_json("Callees of main function", callees); + + // 1. Legitimate callee must be present + check(strstr(callees, legit_callee) != nullptr, "legit callee present"); + + // 2. Builtin names must NOT appear as callees + for (int i = 0; builtin_names[i] != nullptr; i++) { + char buf[256]; + snprintf(buf, sizeof(buf), "\"name\":\"%s\"", builtin_names[i]); + // The callee should NOT include this builtin + if (strstr(callees, buf) != nullptr) { + fprintf(stderr, + "FAIL: builtin '%s' should NOT appear in callees of mainFunc\n", + builtin_names[i]); + fprintf(stderr, " callees JSON: %s\n", callees); + exit(1); + } + printf(" [PASS] builtin '%s' correctly excluded from callees\n", + builtin_names[i]); + } + engine_free_string(callees); + + // Get references to verify builtin calls don't create reference entries + for (int i = 0; builtin_names[i] != nullptr; i++) { + char *refs = engine_find_references(g_engine, pid, + builtin_names[i], nullptr); + // Builtin references should have zero results (no user func with that name) + if (strstr(refs, "\"total\":0") == nullptr && + strstr(refs, "\"results\":[]") == nullptr) { + // If there ARE results, they must not include mainFunc as a caller + if (strstr(refs, "mainFunc") != nullptr) { + fprintf(stderr, + "FAIL: builtin '%s' should NOT have references from mainFunc\n", + builtin_names[i]); + fprintf(stderr, " refs JSON: %s\n", refs); + exit(1); + } + } + printf(" [PASS] builtin '%s' reference check passed\n", + builtin_names[i]); + engine_free_string(refs); + } + + // Stats check: total edges should be reasonable (no FP edges from builtins) + char *stats = engine_get_graph_stats(g_engine, pid); + print_json("Graph Stats", stats); + check(strstr(stats, "total_nodes") != nullptr, "get_graph_stats"); + engine_free_string(stats); + + engine_destroy(g_engine); + g_engine = nullptr; + printf("\n=== %s FP verification test passed ===\n", lang); +} + +#endif // TEST_E2E_H diff --git a/engine/tests/test_engine_handle.h b/engine/tests/test_engine_handle.h new file mode 100644 index 0000000..105cf20 --- /dev/null +++ b/engine/tests/test_engine_handle.h @@ -0,0 +1,27 @@ +#ifndef TEST_ENGINE_HANDLE_H +#define TEST_ENGINE_HANDLE_H + +// One engine instance handle per translation unit (TD-1 knife 3). +// +// The C ABI is handle-based: engine_create() returns the instance and every +// stateful entry point takes it as its first parameter. The suite used to call +// engine_init()/engine_shutdown(), which reached for process-global state; it +// now keeps the instance in `g_engine`, assigned by engine_create() and +// released by engine_destroy(). +// +// The include guard is load-bearing: a test file and the shared test_e2e.h +// harness both include this header, and one translation unit may define the +// object only once. Because the object is `static`, every translation unit +// gets its own instance — tests stay independent without sharing a handle. +// +// Usage: `g_engine = engine_create(db_path);` ... `engine_destroy(g_engine);`. +// engine_destroy() accepts null, so clearing the handle after releasing it +// makes a second shutdown a no-op instead of a double free. + +#include "../include/engine.h" + +/// Handle of the engine instance the current test drives. Null before the +/// first engine_create() and again after engine_destroy(). +static engine_t g_engine = nullptr; + +#endif // TEST_ENGINE_HANDLE_H diff --git a/engine/tests/test_engine_handles.cpp b/engine/tests/test_engine_handles.cpp new file mode 100644 index 0000000..b398193 --- /dev/null +++ b/engine/tests/test_engine_handles.cpp @@ -0,0 +1,279 @@ +// test_engine_handles.cpp — TD-1 (docs/REVIEW_0.2.7.md) knife 3. +// +// Knife 3 replaced the process-global engine state (engine_init()/ +// engine_shutdown() mutating a hidden singleton) with an explicit instance: +// engine_create() returns an `engine_t` handle, every stateful entry point takes +// it as its first parameter, and engine_destroy() releases it. This test is the +// evidence for the two things that change buys, and for the boundary contracts +// code_rules §4 requires of an FFI surface (null inputs, error conditions, +// ownership transfer, double-free prevention): +// +// 1. Rejected input and unopenable storage produce NULL instead of a +// half-built instance. +// 2. A null handle reaches every entry point as the documented +// "engine not initialized" envelope — the same answer the old code gave +// when engine_init() had not run — so a caller that ignores the NULL can +// never dereference it. +// 3. Two instances live in one process and keep separate stores: a symbol +// indexed into instance A is invisible to instance B, and vice versa. +// 4. engine_destroy(NULL) is a no-op, so a failed engine_create() (or an +// already-released handle) can be released without a guard, and the +// process stays usable afterwards. +// +// The test drives real files and real SQLite databases (code_rules §4: +// integration tests must use real dependencies) under /tmp, like the rest of +// the suite. + +#include "test_check.h" + +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace +{ + +/// Root of the fixture tree for one instance. Kept next to the DB so a failed +/// run leaves both behind for inspection instead of scattering files. +const char *kRootOne = "/tmp/codescope_handle_one"; +const char *kRootTwo = "/tmp/codescope_handle_two"; + +/// DB file of one instance. Distinct paths are what make the instances +/// genuinely separate stores rather than two handles onto one database. +const char *kDbOne = "/tmp/codescope_handle_one.db"; +const char *kDbTwo = "/tmp/codescope_handle_two.db"; + +/// Symbol that only instance one indexes, and the mirror for instance two. +const char *kSymbolOne = "symbol_only_in_instance_one"; +const char *kSymbolTwo = "symbol_only_in_instance_two"; + +/// Remove a database and its WAL sidecars so every run starts clean. +void removeDb(const char *path) +{ + std::string base(path); + unlink(base.c_str()); + unlink((base + "-wal").c_str()); + unlink((base + "-shm").c_str()); +} + +/// Write `content` to `path`, creating parent directories. +/// @return true when the file was written. +bool writeFile(const std::string &path, const std::string &content) +{ + FILE *f = fopen(path.c_str(), "w"); + if (!f) + return false; + const size_t written = fwrite(content.data(), 1, content.size(), f); + fclose(f); + return written == content.size(); +} + +/// Index one file and assert the engine reported success. +/// @param handle Instance to index into (may be null to test the +/// uninitialized path). +/// @param project_id Project to attach the file to. +/// @param path Source file to index. +/// @param expected_symbol Symbol the file defines; asserted to be findable +/// afterwards in the same instance. +void indexAndFind(engine_t handle, uint64_t project_id, const char *path, + const char *expected_symbol) +{ + char *indexed = engine_index_file(handle, project_id, path); + CHECK(indexed != nullptr); + if (indexed) { + CHECK(strstr(indexed, "\"ok\":true") != nullptr); + engine_free_string(indexed); + } + + char *found = engine_find_definition(handle, project_id, + expected_symbol, nullptr); + CHECK(found != nullptr); + if (found) { + CHECK_MSG(strstr(found, expected_symbol) != nullptr, + "the symbol the instance indexed must be findable"); + engine_free_string(found); + } +} + +/// Assert an entry point answers a null handle with the not-initialized +/// envelope instead of dereferencing it. +void checkNullHandleContract() +{ + // engine_create() rejects missing/empty input without allocating. + CHECK(engine_create(nullptr) == nullptr); + CHECK(engine_create("") == nullptr); + + // A path whose parent cannot exist: SQLite cannot create the file, so no + // instance is handed out. + CHECK(engine_create("/dev/null/codescope-cannot-exist.db") == nullptr); + + // Every stateful entry point must survive the null handle. Two opposite + // shapes are covered: a query that returns an envelope. + char *stats = engine_get_graph_stats(nullptr, 1); + CHECK(stats != nullptr); + if (stats) { + CHECK_MSG(strstr(stats, "\"ok\":false") != nullptr || + strstr(stats, "\"error\"") != nullptr, + "a null handle must produce an error envelope"); + engine_free_string(stats); + } + + // ... and a lookup that returns an empty result set plus an error field + // (the shape the MCP layer parses). + char *def = engine_find_definition(nullptr, 1, "anything", nullptr); + CHECK(def != nullptr); + if (def) { + CHECK_MSG( + strstr(def, "not initialized") != nullptr, + "the null-handle path must keep the documented message"); + engine_free_string(def); + } + + // A project-mutating call with a null handle must report failure, not + // create a project in some other instance. + CHECK(engine_create_project(nullptr, "/tmp", "no-instance") == 0); + + // engine_destroy() accepts NULL so callers do not need a guard; calling it + // twice is still a no-op, and the process keeps working afterwards. + engine_destroy(nullptr); + engine_destroy(nullptr); + char *after = engine_locate_by_name(nullptr, 1, "still-alive"); + CHECK(after != nullptr); + if (after) + engine_free_string(after); +} + +/// Create the fixture source tree for both instances. +/// @return true when both files were written. +bool writeFixtures() +{ + if (mkdir(kRootOne, 0755) != 0 && errno != EEXIST) + return false; + if (mkdir(kRootTwo, 0755) != 0 && errno != EEXIST) + return false; + + const std::string one = std::string(kRootOne) + "/one.py"; + const std::string two = std::string(kRootTwo) + "/two.py"; + + const std::string source_one = "def " + std::string(kSymbolOne) + + "():\n return 1\n\n\n" + "def caller_one():\n return " + + std::string(kSymbolOne) + "()\n"; + const std::string source_two = "def " + std::string(kSymbolTwo) + + "():\n return 2\n\n\n" + "def caller_two():\n return " + + std::string(kSymbolTwo) + "()\n"; + + return writeFile(one, source_one) && writeFile(two, source_two); +} + +/// Two live instances must not see each other's data. +void checkTwoInstancesAreIsolated() +{ + removeDb(kDbOne); + removeDb(kDbTwo); + + engine_t one = engine_create(kDbOne); + engine_t two = engine_create(kDbTwo); + CHECK_MSG(one != nullptr, "first instance must open its own DB"); + CHECK_MSG(two != nullptr, "second instance must open its own DB"); + CHECK_MSG(one != two, "the two handles must name different instances"); + if (!one || !two) { + engine_destroy(one); + engine_destroy(two); + return; + } + + // Each instance creates its own project row in its own store. + const uint64_t project_one = + engine_create_project(one, kRootTwo, "one"); + const uint64_t project_two = + engine_create_project(two, kRootTwo, "two"); + CHECK_MSG(project_one > 0, "project in instance one"); + CHECK_MSG(project_two > 0, "project in instance two"); + + indexAndFind(one, project_one, + (std::string(kRootOne) + "/one.py").c_str(), kSymbolOne); + indexAndFind(two, project_two, + (std::string(kRootTwo) + "/two.py").c_str(), kSymbolTwo); + + // The isolation assertion: instance one must not know instance two's + // symbol, and instance two must not know instance one's. + char *cross = + engine_find_definition(one, project_one, kSymbolTwo, nullptr); + CHECK(cross != nullptr); + if (cross) { + CHECK_MSG( + strstr(cross, kSymbolTwo) == nullptr, + "instance one must not resolve a symbol indexed into instance two"); + engine_free_string(cross); + } + cross = engine_find_definition(two, project_two, kSymbolOne, nullptr); + CHECK(cross != nullptr); + if (cross) { + CHECK_MSG( + strstr(cross, kSymbolOne) == nullptr, + "instance two must not resolve a symbol indexed into instance one"); + engine_free_string(cross); + } + + // Instance one's store does not contain instance two's project either: its + // graph stats must report nodes for its own project only. + char *stats = engine_get_graph_stats(one, project_one); + CHECK(stats != nullptr); + if (stats) { + CHECK_MSG( + strstr(stats, "\"total_nodes\":0") == nullptr, + "instance one must report the nodes it indexed itself"); + engine_free_string(stats); + } + + // Releasing one instance must leave the other fully usable (per-instance + // lifetime, not a process-wide switch). + engine_destroy(one); + + char *survivor = + engine_find_definition(two, project_two, kSymbolTwo, nullptr); + CHECK(survivor != nullptr); + if (survivor) { + CHECK_MSG( + strstr(survivor, kSymbolTwo) != nullptr, + "the surviving instance must keep working after engine_destroy(one)"); + engine_free_string(survivor); + } + + // Destroying the released handle again is a documented no-op. + engine_destroy(nullptr); + engine_destroy(two); + + removeDb(kDbOne); + removeDb(kDbTwo); +} + +} // namespace + +int main() +{ + printf("=== test_engine_handles (TD-1 knife 3) ===\n"); + + checkNullHandleContract(); + printf("null-handle contract: PASS\n"); + + if (!writeFixtures()) { + fprintf(stderr, "FAIL: could not write the fixture tree\n"); + return 1; + } + + checkTwoInstancesAreIsolated(); + printf("two-instance isolation: PASS\n"); + + printf("\n=== test_engine_handles %s ===\n", + checkFailures() ? "FAILED" : "PASSED"); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_enhance_e2e.cpp b/engine/tests/test_enhance_e2e.cpp index 4a46e59..656c1b3 100644 --- a/engine/tests/test_enhance_e2e.cpp +++ b/engine/tests/test_enhance_e2e.cpp @@ -4,7 +4,9 @@ #include #include #include +#include #include +#include "test_engine_handle.h" namespace fs = std::filesystem; @@ -45,6 +47,29 @@ static void write_file(const std::string &path, const char *content) fclose(f); } +// Count rows of a single table in the project DB. Opens a fresh read +// connection so it does not disturb the engine's handle. Returns -1 on error. +static int count_rows(const char *sql) +{ + sqlite3 *db = nullptr; + if (sqlite3_open(kDbPath, &db) != SQLITE_OK) { + sqlite3_close(db); + return -1; + } + sqlite3_stmt *st = nullptr; + int n = -1; + if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK) { + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + } + sqlite3_close(db); + return n; +} + +// count_rows lives above: the tests only ever read the project DB, because the +// engine holds its write lock for the whole run. + int main() { std::error_code ec; @@ -82,32 +107,69 @@ int helper(int x) { static int internal_impl(int x) { return x * 2; } + +// Capability-named function so buildCapabilityState emits a row +// (name LIKE 'Auth%'), used by the idempotency check below. +int AuthGuard(int x) { + return x + 1; +} )"); // ─── Step 1: init + create project ─── - check(engine_init(kDbPath) == 0, "engine_init"); - uint64_t pid = engine_create_project(kProjDir, "enhance_e2e"); + g_engine = engine_create(kDbPath); + check(g_engine != nullptr, "engine_init"); + uint64_t pid = engine_create_project(g_engine, kProjDir, "enhance_e2e"); check(pid > 0, "create_project"); printf("PASS: project_id=%llu\n", (unsigned long long)pid); // ─── Step 2: index (3 files) ─── - char *idx = engine_index_project(pid, kProjDir, NULL); + char *idx = engine_index_project(g_engine, pid, kProjDir, NULL); check(idx != nullptr, "index_project result"); check_json_has(idx, "\"ok\":true", "index_project ok"); printf("PASS: index ok — %s\n", idx); engine_free_string(idx); + // capability_state now mirrors the DECLARED capabilities (the + // `capability` table, filled from the project README by the knowledge + // builder) instead of entity names that merely look like one: the + // hard-coded prefix list (Auth%/Login%/JWT%/Config%/…) is gone, so + // AuthGuard on its own no longer produces a row, and build_project_state + // no longer reports a capability score that answers a different question + // than the drift tools. The assertion below is the invariant itself, so + // it holds for a fixture that declares capabilities and for one like + // this that declares none. + // ─── Step 3: enhance (first run) ─── // Note: index already builds the graph, so enhance may be a no-op. - char *enh = engine_enhance_project(pid); + char *enh = engine_enhance_project(g_engine, pid); check(enh != nullptr, "enhance_project result"); check(strstr(enh, "\"status\"") != nullptr, "enhance: has status field"); + // Honest reporting (2026-09-27 review D1-5): the success path + // must state ok:true + status:"ok" explicitly. A failed step now returns + // ok:false with failed_step instead of claiming success over a + // rolled-back graph. + check_json_has(enh, "\"ok\":true", "enhance: ok true on success"); + check_json_has(enh, "\"status\":\"ok\"", + "enhance: status ok on success"); printf("PASS: enhance run1 — %s\n", enh); engine_free_string(enh); + // Idempotency baseline (2026-09-27 review, second pass): + // capability_state and workflow_step are derived tables that a pre-fix + // build re-appended on every enhance (no UNIQUE / no delete-first), + // doubling capability/workflow counts per run. Capture the counts after + // the first enhance; Step 6 asserts a second enhance does not grow them. + int cap1 = count_rows("SELECT COUNT(*) FROM capability_state"); + int ws1 = count_rows("SELECT COUNT(*) FROM workflow_step"); + check(cap1 == count_rows("SELECT COUNT(*) FROM capability"), + "capability_state mirrors the declared capabilities"); + check(ws1 >= 1, "workflow_step must have >=1 row (main workflow)"); + printf("PASS: idempotency baseline — capability_state=%d workflow_step=%d\n", + cap1, ws1); + // ─── Step 4: check enhancement status ─── - char *st = engine_get_enhancement_status(pid); + char *st = engine_get_enhancement_status(g_engine, pid); check(st != nullptr, "enhancement_status result"); int total_st = 0, cg_st = 0, met_st = 0, emb_st = 0; sscanf(st, @@ -123,14 +185,15 @@ static int internal_impl(int x) { // ─── Step 5: cross-file caller/callee checks ─── // main() calls helper(int) via `helper(42)` — cross-file edge from buildGraph - char *callees_main = engine_get_callees(pid, "main", nullptr); + char *callees_main = engine_get_callees(g_engine, pid, "main", nullptr); check(callees_main != nullptr, "callees of main"); check_json_has(callees_main, "helper", "callees of main: has helper"); printf("PASS: callees of main (cross-file) ok\n%s\n", callees_main); engine_free_string(callees_main); // helper(int) calls internal_impl — same-file regex edge - char *callees_helper = engine_get_callees(pid, "helper", nullptr); + char *callees_helper = + engine_get_callees(g_engine, pid, "helper", nullptr); check(callees_helper != nullptr, "callees of helper(int)"); check_json_has(callees_helper, "internal_impl", "callees of helper: has internal_impl"); @@ -138,7 +201,8 @@ static int internal_impl(int x) { engine_free_string(callees_helper); // internal_impl is called by helper(int) - char *callers_impl = engine_get_callers(pid, "internal_impl", nullptr); + char *callers_impl = + engine_get_callers(g_engine, pid, "internal_impl", nullptr); check(callers_impl != nullptr, "callers of internal_impl"); check_json_has(callers_impl, "helper", "callers of internal_impl: has helper"); @@ -146,23 +210,31 @@ static int internal_impl(int x) { engine_free_string(callers_impl); // ─── Step 6: rerun idempotency ─── - char *enh2 = engine_enhance_project(pid); + char *enh2 = engine_enhance_project(g_engine, pid); check(enh2 != nullptr, "enhance_project rerun"); check_json_has(enh2, "\"status\"", "enhance rerun has status"); - char *st2 = engine_get_enhancement_status(pid); + char *st2 = engine_get_enhancement_status(g_engine, pid); check(st2 != nullptr, "status after rerun"); int cg_st2 = 0; sscanf(st2, "{\"total_symbols\":%d,\"callgraph_ready\":%d", &total_st, &cg_st2); // Idempotency: callgraph_ready must not decrease check(cg_st2 == cg_st, "rerun: callgraph_ready unchanged"); - printf("PASS: rerun idempotent — cg_ready %d→%d\n%s\n%s\n", - cg_st, cg_st2, enh2, st2); + // Idempotency: derived state tables must not grow on a second enhance. + int cap2 = count_rows("SELECT COUNT(*) FROM capability_state"); + int ws2 = count_rows("SELECT COUNT(*) FROM workflow_step"); + check(cap2 == cap1, + "rerun: capability_state must not grow (delete-first rebuild)"); + check(ws2 == ws1, + "rerun: workflow_step must not grow (delete-first rebuild)"); + printf("PASS: rerun idempotent — cg_ready %d→%d, capability_state %d→%d, " + "workflow_step %d→%d\n%s\n%s\n", + cg_st, cg_st2, cap1, cap2, ws1, ws2, enh2, st2); engine_free_string(enh2); engine_free_string(st2); // ─── Step 7: trace path (BFS on call_edges) ─── - char *trace = engine_trace_path(pid, "main", "internal_impl"); + char *trace = engine_trace_path(g_engine, pid, "main", "internal_impl"); check(trace != nullptr, "trace_path result"); check_json_has(trace, "\"path\"", "trace: has path"); check_json_has(trace, "main", "trace: contains main"); @@ -172,7 +244,7 @@ static int internal_impl(int x) { engine_free_string(trace); // ─── Step 8: overview (smoke check) ─── - char *overview = engine_get_project_overview(pid); + char *overview = engine_get_project_overview(g_engine, pid); check(overview != nullptr, "overview"); check_json_has(overview, "\"total_nodes\"", "overview: has total_nodes"); @@ -180,7 +252,8 @@ static int internal_impl(int x) { engine_free_string(overview); // ─── Cleanup ─── - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; fs::remove_all(kProjDir, ec); fs::remove(kDbPath, ec); diff --git a/engine/tests/test_evidence_builder.cpp b/engine/tests/test_evidence_builder.cpp index 3838e8a..5a6c2c4 100644 --- a/engine/tests/test_evidence_builder.cpp +++ b/engine/tests/test_evidence_builder.cpp @@ -29,7 +29,7 @@ #include "../src/evidence/evidence_builder.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -57,13 +57,13 @@ static void insertFunction(GraphStore &store, uint64_t project_id, int64_t id, "start_row, start_col, end_row, end_col) " "VALUES (?,?,0,0,?,'',?,?,1,0,1000,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, file_path, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 5, language, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -83,7 +83,7 @@ static void insertFact(GraphStore &store, uint64_t project_id, " symbol, confidence, detail_json) " "VALUES (?,?,?,?,?,?,?,?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, static_cast(function_id)); sqlite3_bind_text(stmt, 3, category, -1, SQLITE_TRANSIENT); @@ -95,7 +95,7 @@ static void insertFact(GraphStore &store, uint64_t project_id, sqlite3_bind_text(stmt, 8, detail_json, -1, SQLITE_TRANSIENT); else sqlite3_bind_null(stmt, 8); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -120,12 +120,12 @@ static std::string detailJson(int line, const std::string &snippet, static std::string findRulesDir() { const char *candidates[] = { - "engine/src/evidence/rules", - "../src/evidence/rules", - "../../engine/src/evidence/rules", - "../../../engine/src/evidence/rules", - }; - for (const char *cand : candidates) { + "engine/src/evidence/rules", + "../src/evidence/rules", + "../../engine/src/evidence/rules", + "../../../engine/src/evidence/rules", + }; + for (const char *cand : candidates) { std::error_code ec; if (!std::filesystem::is_directory(cand, ec)) continue; @@ -156,7 +156,7 @@ int main() return 1; } uint64_t pid = store.createProject("/tmp", "test_evidence_builder"); - assert(pid > 0); + CHECK(pid > 0); // ── Test 1: sync/mutex_without_defer_unlock (missing_match) ── // Function 1: lock WITHOUT defer_unlock → should appear in @@ -239,7 +239,7 @@ int main() // ── Run buildAll ───────────────────────────────────────────── auto all = builder.buildAll(pid); printf("buildAll returned %zu evidence(s)\n", all.size()); - assert(!all.empty()); + CHECK(!all.empty()); // Helper: find an evidence by substring in its title. Each // rule's title template is unique enough that a substring match @@ -260,11 +260,11 @@ int main() { const Evidence *ev = findByTitleContains("lock mutex without defer Unlock"); - assert(ev != nullptr); - assert(ev->items.size() == 1); - assert(ev->items[0].symbol == "m.Lock"); - assert(ev->items[0].line == 5); - assert(ev->items[0].file == "/src/sync_leak.go"); + CHECK(ev != nullptr); + CHECK(ev->items.size() == 1); + CHECK(ev->items[0].symbol == "m.Lock"); + CHECK(ev->items[0].line == 5); + CHECK(ev->items[0].file == "/src/sync_leak.go"); printf("Test 1 (mutex_without_defer_unlock, 1 item): " "PASS\n"); } @@ -274,10 +274,10 @@ int main() // should be emitted (one per surviving function). { auto cstring_evs = builder.buildByRule(pid, "cstring_leak"); - assert(cstring_evs.size() == 1); - assert(cstring_evs[0].items.size() == 1); - assert(cstring_evs[0].items[0].symbol == "C.CString"); - assert(cstring_evs[0].items[0].file == "/src/cgo_leak.go"); + CHECK(cstring_evs.size() == 1); + CHECK(cstring_evs[0].items.size() == 1); + CHECK(cstring_evs[0].items[0].symbol == "C.CString"); + CHECK(cstring_evs[0].items[0].file == "/src/cgo_leak.go"); printf("Test 2 (cstring_leak, 1 evidence 1 item): " "PASS\n"); } @@ -286,40 +286,40 @@ int main() { const Evidence *ev = findByTitleContains("bare except clause(s)"); - assert(ev != nullptr); - assert(ev->items.size() == 1); - assert(ev->items[0].symbol == "except"); + CHECK(ev != nullptr); + CHECK(ev->items.size() == 1); + CHECK(ev->items[0].symbol == "except"); printf("Test 3 (bare_except_collect, 1 item): PASS\n"); } // ── Verify: todo_collect has 1 item ────────────────────────── { const Evidence *ev = findByTitleContains("TODO marker(s)"); - assert(ev != nullptr); - assert(ev->items.size() == 1); - assert(ev->items[0].symbol == "TODO: implement"); + CHECK(ev != nullptr); + CHECK(ev->items.size() == 1); + CHECK(ev->items[0].symbol == "TODO: implement"); printf("Test 4 (todo_collect, 1 item): PASS\n"); } // ── Verify: unwrap_risk has 1 item (bonus) ─────────────────── { const Evidence *ev = findByTitleContains(".unwrap() call(s)"); - assert(ev != nullptr); - assert(ev->items.size() == 1); - assert(ev->items[0].symbol == "result.unwrap"); + CHECK(ev != nullptr); + CHECK(ev->items.size() == 1); + CHECK(ev->items[0].symbol == "result.unwrap"); printf("Test 5 (unwrap_risk, 1 item): PASS\n"); } // ── Test 6: buildByCategory("sync") returns only sync evidence ─ { auto sync_evs = builder.buildByCategory(pid, "sync"); - assert(!sync_evs.empty()); + CHECK(!sync_evs.empty()); for (const auto &ev : sync_evs) { - assert(ev.category == "sync"); + CHECK(ev.category == "sync"); } // The sync category has only the mutex_without_defer_unlock // rule, which produced 1 evidence. - assert(sync_evs.size() == 1); + CHECK(sync_evs.size() == 1); printf("Test 6 (buildByCategory sync, %zu evidence): " "PASS\n", sync_evs.size()); @@ -330,7 +330,7 @@ int main() // count assertion.) { auto evs = builder.buildByRule(pid, "cstring_leak"); - assert(evs.size() == 1); + CHECK(evs.size() == 1); printf("Test 7 (buildByRule cstring_leak, 1 evidence): " "PASS\n"); } @@ -338,7 +338,7 @@ int main() // ── Test 8: buildByRule with unknown name returns 0 ────────── { auto evs = builder.buildByRule(pid, "nonexistent_rule"); - assert(evs.empty()); + CHECK(evs.empty()); printf("Test 8 (buildByRule unknown, 0 evidence): " "PASS\n"); } @@ -346,7 +346,7 @@ int main() // ── Test 9: buildByCategory with no matching category ──────── { auto evs = builder.buildByCategory(pid, "nonexistent"); - assert(evs.empty()); + CHECK(evs.empty()); printf("Test 9 (buildByCategory unknown, 0 evidence): " "PASS\n"); } @@ -362,14 +362,32 @@ int main() // concurrency/mutex_without_unlock (1, lock without // defer_unlock in function 100) // test_quality/unwrap_in_non_test (1, unwrap inserted) - // test_quality/todo_accumulation (1, Count mode always emits - // 1 evidence) + // test_quality/todo_accumulation (1; the Count rule matches the + // inserted TODO — with no match it emits nothing, see Test 11) // Total = 5 + 5 = 10. - assert(all.size() == 10); + CHECK(all.size() == 10); printf("Test 10 (total evidence count == 10): PASS\n"); + // ── Test 11: Count mode with no matches emits nothing ────────── + // Regression: combineCount() pushed an Evidence with count=0, so a rule that + // found nothing still looked like an inspector that produced a result — + // inflating overall.inspectors_ran and the confidence derived from it. Every + // other combine mode returns no Evidence when nothing matches. + { + uint64_t empty_pid = store.createProject("/tmp/empty", + "test_evidence_empty"); + CHECK(empty_pid > 0); + auto none = builder.buildAll(empty_pid); + for (const auto &ev : none) { + printf(" unexpected evidence on an empty project: %s\n", + ev.title.c_str()); + } + CHECK(none.empty()); + printf("Test 11 (Count mode with 0 matches emits nothing): PASS\n"); + } + store.close(); unlink(kDbPath); printf("\nAll evidence_builder tests passed.\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_exclude_paths.cpp b/engine/tests/test_exclude_paths.cpp index 94f8f45..8c18f94 100644 --- a/engine/tests/test_exclude_paths.cpp +++ b/engine/tests/test_exclude_paths.cpp @@ -6,16 +6,18 @@ // .gitignore. Patterns are comma-separated globs matched against the // project-relative path via globMatch(). // -// Covers five scenarios: +// Covers six scenarios: // 1. Basic exclusion: "test/*,docs/*" skips test/foo.cpp, allows src/main.cpp. // 2. Whitespace trimming: " test/* , docs/* " is equivalent to the trimmed form. // 3. Unset env var: no user exclusions (only built-in defaults apply). // 4. Directory skipping: "vendor/*" skips the vendor/ directory entry. // 5. Non-default path: a custom dir not in the built-in skip list is excluded // only when the env var is set — isolates the env-var effect from defaults. +// 6. Escaped comma: "a\,b/**" is ONE pattern (a directory named "a,b"), while +// an unescaped comma still separates patterns (D2-7). #include "../src/filter_policy.h" -#include +#include "test_check.h" #include #include #include @@ -46,8 +48,8 @@ int main() // test for the env var. { FilterPolicy fp = makePolicyWithEnv("test/*,docs/*"); - assert(fp.shouldSkipEntry("test/foo.cpp", false)); - assert(!fp.shouldSkipEntry("src/main.cpp", false)); + CHECK(fp.shouldSkipEntry("test/foo.cpp", false)); + CHECK(!fp.shouldSkipEntry("src/main.cpp", false)); printf(" [PASS] basic: test/* skipped, src/main.cpp allowed\n"); } @@ -57,9 +59,9 @@ int main() // leading/trailing whitespace is trimmed from each pattern. { FilterPolicy fp = makePolicyWithEnv(" test/* , docs/* "); - assert(fp.shouldSkipEntry("test/foo.cpp", false)); - assert(fp.shouldSkipEntry("docs/bar.cpp", false)); - assert(!fp.shouldSkipEntry("src/main.cpp", false)); + CHECK(fp.shouldSkipEntry("test/foo.cpp", false)); + CHECK(fp.shouldSkipEntry("docs/bar.cpp", false)); + CHECK(!fp.shouldSkipEntry("src/main.cpp", false)); printf(" [PASS] whitespace-trimmed patterns work\n"); } @@ -71,9 +73,9 @@ int main() // - test/foo.cpp → true (test is in default skip_dirs) { FilterPolicy fp = makePolicyWithEnv(nullptr); - assert(!fp.shouldSkipEntry("src/main.cpp", false)); - assert(fp.shouldSkipEntry("test/foo.cpp", - false)); // default skip + CHECK(!fp.shouldSkipEntry("src/main.cpp", false)); + CHECK(fp.shouldSkipEntry("test/foo.cpp", + false)); // default skip printf(" [PASS] unset env var: only defaults apply\n"); } @@ -84,14 +86,14 @@ int main() // true regardless — but we verify the glob also matches the dir. { FilterPolicy fp = makePolicyWithEnv("vendor/*"); - assert(fp.shouldSkipEntry("vendor/", true)); + CHECK(fp.shouldSkipEntry("vendor/", true)); // Also verify a non-default directory is skipped only via env: // "custom_dir/" is not in the built-in skip list. FilterPolicy fp2 = makePolicyWithEnv("custom_dir/*"); - assert(fp2.shouldSkipEntry("custom_dir/", true)); + CHECK(fp2.shouldSkipEntry("custom_dir/", true)); // Without env var, custom_dir/ is NOT skipped. FilterPolicy fp3 = makePolicyWithEnv(nullptr); - assert(!fp3.shouldSkipEntry("custom_dir/", true)); + CHECK(!fp3.shouldSkipEntry("custom_dir/", true)); printf(" [PASS] directory: vendor/ + custom_dir/ skipped via env\n"); } @@ -107,19 +109,36 @@ int main() { // With env var FilterPolicy fp = makePolicyWithEnv("custom_skip/*"); - assert(fp.shouldSkipEntry("custom_skip/foo.cpp", false)); + CHECK(fp.shouldSkipEntry("custom_skip/foo.cpp", false)); // * does not cross '/' — nested path NOT matched - assert(!fp.shouldSkipEntry("custom_skip/sub/bar.cpp", false)); + CHECK(!fp.shouldSkipEntry("custom_skip/sub/bar.cpp", false)); // Without env var — same path is allowed FilterPolicy fp2 = makePolicyWithEnv(nullptr); - assert(!fp2.shouldSkipEntry("custom_skip/foo.cpp", false)); + CHECK(!fp2.shouldSkipEntry("custom_skip/foo.cpp", false)); printf(" [PASS] non-default path: custom_skip/* skipped only with env\n"); } + // ── 6. Escaped comma inside a pattern ──────────────────────── + // + // A directory whose name contains a literal comma ("a,b") yields the + // pattern "a,b/**", which the producer escapes as "a\,b/**" so the comma + // splitter does not tear it into "a\" + "b/**" (neither of which matches, + // so the file would silently NOT be excluded). + { + FilterPolicy fp = makePolicyWithEnv("a\\,b/**,src/*"); + // The escaped comma belongs to ONE pattern, so the path matches. + CHECK(fp.shouldSkipEntry("a,b/foo.cpp", false)); + // A neighbouring path is still not excluded. + CHECK(!fp.shouldSkipEntry("a/other.cpp", false)); + // The unescaped comma still separates the second pattern. + CHECK(fp.shouldSkipEntry("src/main.cpp", false)); + printf(" [PASS] escaped comma stays inside one pattern\n"); + } + // Clean up: unset the env var so it doesn't leak to other tests. unsetenv("CODESCOPE_EXCLUDE_PATHS"); printf("=== Exclude paths test passed ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_ffi_envelopes.cpp b/engine/tests/test_ffi_envelopes.cpp new file mode 100644 index 0000000..e0528d8 --- /dev/null +++ b/engine/tests/test_ffi_envelopes.cpp @@ -0,0 +1,145 @@ +// test_ffi_envelopes.cpp +// +// Smoke test for every FFI entry point that gained a try/catch wrapper (and +// for the two that only gained `catch (...)`). Two things are asserted: +// +// 1. Each entry point returns a heap-allocated JSON envelope — never null, +// never a bare non-JSON string — when handed degenerate input (empty +// strings, zero ids, a path that does not exist). Callers in the Rust +// server feed the result straight into serde_json, so a non-JSON return +// is a protocol break. +// 2. The process survives the whole sweep. That is the real point: these +// exports previously had no try/catch, so any C++ exception crossing the +// extern "C" boundary terminated the long-running MCP server. Reaching +// the end of this test means no export threw an uncaught exception and +// none of the wrappers was wired to the wrong implementation. +// +// It is deliberately NOT a behavioural test: the graph is empty, so most calls +// return an "empty"/"not ready" envelope. That is exactly what makes it stable. +// +// The exports are called with empty strings rather than nullptr: null-pointer +// validation is a separate contract (several of these already guard it, and a +// null dereference is a SIGSEGV that a try/catch cannot catch), so mixing the +// two concerns would make failures ambiguous. + +#include "../include/engine.h" + +#include +#include +#include +#include +#include "test_engine_handle.h" + +static const char *kDbPath = "/tmp/codescope_test_ffi_envelopes.db"; +/// A path that does not exist, so indexing has nothing to walk. +static const char *kMissingDir = "/tmp/codescope_no_such_dir_for_ffi_smoke"; + +static int g_checked = 0; +static int g_failures = 0; + +/// Assert an entry point returned a plausible JSON envelope, then release it. +static void expectEnvelope(const char *label, char *result) +{ + ++g_checked; + if (result == nullptr) { + fprintf(stderr, " FAIL %-38s null pointer\n", label); + ++g_failures; + return; + } + const std::string json(result); + engine_free_string(result); + if (json.empty() || (json[0] != '{' && json[0] != '[')) { + fprintf(stderr, " FAIL %-38s not a JSON envelope: %.70s\n", + label, json.c_str()); + ++g_failures; + return; + } + printf(" ok %-38s\n", label); +} + +int main() +{ + unlink(kDbPath); + unlink((std::string(kDbPath) + "-wal").c_str()); + unlink((std::string(kDbPath) + "-shm").c_str()); + + g_engine = engine_create(kDbPath); + if (!g_engine) { + fprintf(stderr, "FAIL: engine_init(%s)\n", kDbPath); + return 1; + } + // Project 0 is intentionally invalid: every call below must still answer + // with a JSON envelope rather than crash. + const uint64_t pid = 0; + + printf("FFI entry points:\n"); + + // ── Batch 2: the exports that gained a try/catch ──────────── + expectEnvelope("engine_index_project", + engine_index_project(g_engine, pid, kMissingDir, "")); + expectEnvelope("engine_index_files", + engine_index_files(g_engine, pid, "[]", 0)); + expectEnvelope("engine_scan_project", + engine_scan_project(g_engine, pid, kMissingDir, "")); + expectEnvelope("engine_get_module_tree", + engine_get_module_tree(g_engine, pid)); + expectEnvelope("engine_find_symbol", + engine_find_symbol(g_engine, pid, "")); + expectEnvelope("engine_enhance_project", + engine_enhance_project(g_engine, pid)); + expectEnvelope("engine_get_enhancement_status", + engine_get_enhancement_status(g_engine, pid)); + expectEnvelope("engine_unified_search", + engine_unified_search(g_engine, pid, "", 10)); + expectEnvelope("engine_find_callers_adaptive", + engine_find_callers_adaptive(g_engine, pid, "", "")); + expectEnvelope("engine_find_callees_adaptive", + engine_find_callees_adaptive(g_engine, pid, "", "")); + expectEnvelope("engine_find_callers_by_entity", + engine_find_callers_by_entity(g_engine, pid, 0)); + expectEnvelope("engine_find_callees_by_entity", + engine_find_callees_by_entity(g_engine, pid, 0)); + expectEnvelope("engine_get_entry_points_new", + engine_get_entry_points_new(g_engine, pid)); + expectEnvelope("engine_project_overview", + engine_project_overview(g_engine, pid)); + expectEnvelope("engine_trace_path", + engine_trace_path(g_engine, pid, "", "")); + expectEnvelope("engine_explore_function", + engine_explore_function(g_engine, pid, "", 1, "both")); + expectEnvelope("engine_build_context", + engine_build_context(g_engine, pid, "")); + expectEnvelope("engine_detect_ffi_boundaries", + engine_detect_ffi_boundaries(g_engine, pid)); + expectEnvelope("engine_build_project_state", + engine_build_project_state(g_engine, pid)); + expectEnvelope("engine_get_project_state", + engine_get_project_state(g_engine, pid)); + expectEnvelope("engine_build_evidence", + engine_build_evidence(g_engine, pid, "")); + // engine_verify_statement used to sit here; it was retired in favour of + // verify_claim, so the envelope check now covers engine_verify_claim — + // which this test did not cover at all before. + expectEnvelope("engine_verify_claim", + engine_verify_claim(g_engine, pid, "")); + + // ── The two that gained `catch (...)` ────────────────────── + expectEnvelope("engine_search_semantic", + engine_search_semantic(g_engine, pid, "", 10)); + expectEnvelope("engine_rebuild_csr", engine_rebuild_csr("", pid)); + + engine_destroy(g_engine); + g_engine = nullptr; + unlink(kDbPath); + unlink((std::string(kDbPath) + "-wal").c_str()); + unlink((std::string(kDbPath) + "-shm").c_str()); + + printf("\nchecked %d entry points, %d failure(s)\n", g_checked, + g_failures); + if (g_failures != 0) + return 1; + printf("\n=== test_ffi_envelopes PASSED ===\n"); + printf("Every patched FFI entry point returned a JSON envelope and the " + "process survived\n"); + return 0; +} diff --git a/engine/tests/test_filter_lang_fold.cpp b/engine/tests/test_filter_lang_fold.cpp new file mode 100644 index 0000000..ae1d0ea --- /dev/null +++ b/engine/tests/test_filter_lang_fold.cpp @@ -0,0 +1,56 @@ +// test_filter_lang_fold.cpp — language-filter alias folding and token +// trimming. Regression cover for: (a) a "c" filter must accept files the +// classifier labels "cpp" and the reverse (the two labels disagree for +// `.c`); (b) whitespace/CRLF-tainted filter tokens must fold like the bare +// label instead of silently matching nothing. + +#include "filter_policy.h" + +#include + +namespace +{ + +int fail(const char *msg) +{ + std::fprintf(stderr, "FAIL: %s\n", msg); + return 1; +} + +} // namespace + +int main() +{ + // Alias family: "c" accepts c-family labels both ways. + FilterPolicy p; + p.setLanguageFilter("c"); + if (!p.isLanguageAccepted("c")) + return fail("filter=c must accept label c"); + if (!p.isLanguageAccepted("cpp")) + return fail("filter=c must accept label cpp (alias family)"); + if (p.isLanguageAccepted("rust")) + return fail("filter=c must NOT accept label rust"); + if (p.isLanguageAccepted("")) + return fail("filter=c must NOT accept empty label"); + + FilterPolicy r; + r.setLanguageFilter("cpp"); + if (!r.isLanguageAccepted("c")) + return fail("filter=cpp must accept label c (alias family)"); + + // Token hygiene: CRLF/whitespace-tainted tokens fold like the bare + // label; a stray "\\r\\n" must not make the filter match nothing. + FilterPolicy q; + q.setLanguageFilter("c\r\n, python "); + if (!q.isLanguageAccepted("c")) + return fail("CRLF-tainted token must match label c"); + if (!q.isLanguageAccepted("cpp")) + return fail("CRLF-tainted token must match label cpp"); + if (!q.isLanguageAccepted("python")) + return fail("space-padded token must match label python"); + if (q.isLanguageAccepted("java")) + return fail("filter=c,python must NOT accept label java"); + + std::printf("=== test_filter_lang_fold PASSED ===\n"); + return 0; +} diff --git a/engine/tests/test_filter_policy_depth.cpp b/engine/tests/test_filter_policy_depth.cpp new file mode 100644 index 0000000..64b96a2 --- /dev/null +++ b/engine/tests/test_filter_policy_depth.cpp @@ -0,0 +1,178 @@ +// test_filter_policy_depth.cpp — regression test for dir-skip depth policy. +// +// Defect guarded: `scripts`, `hack`, `migrations`, `seeds`, `integration`, +// `locale(s)`, `i18n`, `l10n`, `assets`, `static`, `public`, `media` and +// `external` were skipped at ANY path depth. Those names are ordinary +// business directories outside the web-frontend convention +// (src/integration/, pkg/scripts/, db/migrations/, app/static/), so the +// skip silently dropped first-party source files — symbols and call edges +// the index then reported as non-existent, i.e. an undetectable false +// negative. The non-source junk they were meant to exclude (png/po/json/ +// css) is already rejected by the source-extension filter. +// +// Policy now enforced: those names are skipped only within the FIRST THREE +// path components (the depth at which the web conventions actually live), +// for every language; the universally non-source / vendor names (tests, +// docs, vendor, node_modules, ...) stay any-depth. +// +// The Java carve-out is unchanged and re-checked here: Java package +// namespaces still tolerate the any-depth names beyond the top 3 levels. +// +// Uses the explicit check() pattern: assert() is compiled out under +// NDEBUG, which would make these checks vacuous in a Release build. +// +// Exit code: 0 on success, 1 on any failed assertion. + +#include "../src/filter_policy.h" + +#include +#include +#include +#include + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "FAIL: %s\n", msg); + exit(1); + } +} + +static void expectPath(const FilterPolicy &p, const std::string &rel_path, + bool want_skip, const char *why) +{ + bool got = p.shouldSkipPath(rel_path, false); + if (got != want_skip) { + fprintf(stderr, + "FAIL: shouldSkipPath(\"%s\") = %s, expected %s " + "(%s)\n", + rel_path.c_str(), got ? "SKIP" : "INDEX", + want_skip ? "SKIP" : "INDEX", why); + exit(1); + } + printf(" [PASS] %-45s %s (%s)\n", rel_path.c_str(), + got ? "SKIP" : "INDEX", why); +} + +int main() +{ + FilterPolicy p; // NORMAL mode, no ignore files loaded + + printf("=== filter policy depth regression tests (NORMAL mode) ===\n"); + + // ── Business-plausible names: skipped at depth 1-3 only ──────── + expectPath(p, "public/index.js", true, + "root-level public/ is the web convention (depth 1)"); + expectPath(p, "web/public/app.js", true, "depth 2 still skipped"); + expectPath(p, "src/app/public/widget.js", true, + "depth 3 still skipped"); + expectPath(p, "src/features/deep/public/widget.js", false, + "depth 4 is first-party code — must be indexed"); + expectPath(p, "src/pkg/scripts/main.py", true, + "depth 3 still skipped (boundary)"); + expectPath(p, "src/pkg/tools/scripts/main.py", false, + "deep scripts/ is real code — must be indexed"); + expectPath(p, "src/a/b/integration/real.go", false, + "deep integration/ is real code — must be indexed"); + expectPath(p, "src/x/y/z/migrations/0001.sql.go", false, + "deep migrations/ is real code — must be indexed"); + expectPath(p, "migrations/0001_init.sql", true, + "root migrations/ still skipped (depth 1)"); + + // ── Universally non-source / vendor names: still any-depth ───── + expectPath(p, "src/deep/nested/tests/x_test.go", true, + "tests stays any-depth (documented policy)"); + expectPath(p, "src/deep/nested/vendor/lib.go", true, + "vendor stays any-depth"); + expectPath(p, "packages/app/node_modules/x/index.js", true, + "node_modules stays any-depth"); + expectPath(p, "src/deep/third_party/util.c", true, + "third_party stays any-depth"); + + // ── Extension filter still rejects the asset junk ─────────────── + // This is what made the depth relaxation safe: the formats that fill + // assets//media//public// never reach the parser anyway. + check(p.shouldSkipSuffix(".png"), + ".png must still be rejected by the extension filter"); + check(p.shouldSkipSuffix(".class"), + ".class must still be rejected by the extension filter"); + check(!p.shouldSkipSuffix(".go"), ".go must not be rejected"); + expectPath(p, "src/features/deep/assets/real.py", false, + "deep assets/ holding real code must be indexed"); + printf(" [PASS] extension filter still rejects asset formats\n"); + + // ── Java carve-out still protects package namespaces ──────────── + FilterPolicy java; + java.setLangContext("java"); + // samples at depth 4 is an org/... package component, not a docs + // folder: the Java carve-out must keep it indexed. + expectPath(java, "org/acme/store/samples/Widget.java", false, + "Java: samples beyond top 3 is a package component"); + // ... while the top-3 levels are still skipped for Java. + expectPath(java, "src/test/java/App.java", true, + "Java: src/test/ (depth 2) still skipped"); + expectPath(java, "org/acme/store/deep/public/App.java", false, + "Java: deep public/ is a package component too"); + + // ── The carve-out only helps if the policy LEARNS it is Java ─── + // The case above sets the context by hand; the defect was in the + // bootstrap. The server's module discovery queries the policy through + // engine_path_is_skipped, whose policy was built with no language context, + // so a Maven tree got the any-depth rule: spring-petclinic's whole source + // tree lives under src/main/java/org/springframework/samples/petclinic, the + // scheduler found 0 modules, index-parallel indexed nothing with "no source + // modules found", and `codescope discover` counted all 49 files for the same + // path. applyProjectLanguageContext is the one implementation both entry + // points now use. + { + const std::string root = "/tmp/filter_policy_java_bootstrap"; + std::filesystem::remove_all(root); + std::filesystem::create_directories( + root + "/src/main/java/org/acme/samples"); + std::filesystem::create_directories(root + "/src/test/java"); + for (const char *rel : { "src/main/java/org/acme/samples/Owner.java", + "src/test/java/OwnerTest.java" }) { + FILE *f = fopen((root + "/" + rel).c_str(), "w"); + check(f != nullptr, "fixture file must be creatable"); + fputs("class X {}\n", f); + fclose(f); + } + + // Before: no language context, so `samples` is skipped at any depth + // and the file that would reveal the language is unreachable. + FilterPolicy before; + expectPath(before, "src/main/java/org/acme/samples/Owner.java", true, + "without a language context, samples is skipped (the bug)"); + + FilterPolicy after; + applyProjectLanguageContext(after, root); + check(after.langContext() == "java", + "a tree containing a .java file must set the Java context"); + expectPath(after, "src/main/java/org/acme/samples/Owner.java", false, + "Java: the same path is package code, not a samples folder"); + expectPath(after, "src/test/java/OwnerTest.java", true, + "Java: src/test/ is still skipped at depth 2"); + + // A tree with no .java file must NOT be flipped: the carve-out only + // relaxes the names for Java, and other languages nest real test dirs. + const std::string go_root = root + "_go"; + std::filesystem::remove_all(go_root); + std::filesystem::create_directories(go_root + "/pkg/a/test"); + FILE *g = fopen((go_root + "/pkg/a/test/x.go").c_str(), "w"); + check(g != nullptr, "go fixture file must be creatable"); + fputs("package a\n", g); + fclose(g); + FilterPolicy go; + applyProjectLanguageContext(go, go_root); + check(go.langContext().empty(), + "a tree without .java must keep the default context"); + expectPath(go, "pkg/a/test/x.go", true, + "non-Java: test/ stays any-depth"); + + std::filesystem::remove_all(root); + std::filesystem::remove_all(go_root); + } + + printf("\nAll filter policy depth tests passed.\n"); + return 0; +} diff --git a/engine/tests/test_force_index_files.cpp b/engine/tests/test_force_index_files.cpp new file mode 100644 index 0000000..bf0403d --- /dev/null +++ b/engine/tests/test_force_index_files.cpp @@ -0,0 +1,122 @@ +// test_force_index_files.cpp — `force_index_files` must actually reach the +// graph, and the automatic index must keep ignoring test directories. +// +// The tool's documented contract (README) is "index these paths regardless of +// the default skip rules", and it honoured that at the parse stage: forcing +// this repository's `engine/tests` added 13174 `semantic_records` rows. The +// graph stage then threw the result away — `buildGraph` hard-coded a +// test/bench/spec path filter into the entity, reference and import inserts — +// so the tool answered {"ok":true,"files_indexed":115} while `entity` stayed +// at 2327, and a symbol defined only in a test file could never be found. +// +// What this test pins down: +// 1. the automatic project index still skips the `tests/` directory (the +// "AI only needs production code" policy is unchanged), and +// 2. the same file, forced with bypass_fail_fast=1, does produce an entity +// with the findable name. +// +// The request carries a FILE path because that is what the engine's +// file-list entry point takes; `h_force_index_files` walks the requested +// directories itself and passes the files it found. +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include + +#include "test_check.h" +#include "test_engine_handle.h" + +namespace +{ + +/// Run one scalar SQL query (COUNT(*)) against the test database. +long long scalarSql(sqlite3 *db, const char *sql) +{ + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + long long value = -1; + if (sqlite3_step(st) == SQLITE_ROW) + value = sqlite3_column_int64(st, 0); + sqlite3_finalize(st); + return value; +} + +/// Write `content` to `path`, failing the test if the file cannot be created. +void writeFile(const std::string &path, const char *content) +{ + FILE *f = fopen(path.c_str(), "w"); + CHECK(f != nullptr); + fputs(content, f); + fclose(f); +} + +} // namespace + +int main() +{ + const std::string proj_dir = "/tmp/force_index_files_repro"; + const std::string tests_dir = proj_dir + "/tests"; + std::filesystem::remove_all(proj_dir); + std::filesystem::create_directories(proj_dir + "/src"); + std::filesystem::create_directories(tests_dir); + + writeFile(proj_dir + "/src/keep.cpp", + "int keptSymbol()\n{\n\treturn 1;\n}\n"); + writeFile(tests_dir + "/extra.cpp", + "int onlyFromForcedTests()\n{\n\treturn 2;\n}\n"); + + char db[] = "/tmp/test_force_index_files.db"; + unlink(db); + g_engine = engine_create(db); + CHECK(g_engine != nullptr); + const uint64_t pid = + engine_create_project(g_engine, proj_dir.c_str(), "force-test"); + CHECK(pid > 0); + char *idx = + engine_index_project(g_engine, pid, proj_dir.c_str(), nullptr); + CHECK(idx != nullptr); + CHECK(strstr(idx, "\"ok\":true") != nullptr); + engine_free_string(idx); + usleep(500000); // let the async knowledge builder finish + + sqlite3 *db_h = nullptr; + CHECK(sqlite3_open(db, &db_h) == SQLITE_OK); + + // ── 1: the automatic index still ignores tests/ ────────────── + CHECK(scalarSql(db_h, "SELECT COUNT(*) FROM entity " + "WHERE file_path LIKE '%/tests/%'") == 0); + CHECK(scalarSql(db_h, + "SELECT COUNT(*) FROM entity WHERE name='keptSymbol'") == + 1); + CHECK(scalarSql(db_h, "SELECT COUNT(*) FROM semantic_records " + "WHERE file_path LIKE '%/tests/%'") == 0); + + // ── 2: forcing the very same file promotes it ──────────────── + // bypass_fail_fast=1 is exactly what the force_index_files handler + // passes (server/src/tools/indexing.rs). + const std::string request = + "{\"paths\":[\"" + tests_dir + "/extra.cpp\"]}"; + char *forced = engine_index_files(g_engine, pid, request.c_str(), 1); + CHECK(forced != nullptr); + CHECK(strstr(forced, "\"ok\":true") != nullptr); + engine_free_string(forced); + + CHECK(scalarSql( + db_h, + "SELECT COUNT(*) FROM entity WHERE name='onlyFromForcedTests'") == + 1); + CHECK(scalarSql(db_h, "SELECT COUNT(*) FROM entity " + "WHERE file_path LIKE '%/tests/%'") == 1); + + printf("=== force_index_files test passed ===\n"); + sqlite3_close(db_h); + engine_destroy(g_engine); + g_engine = nullptr; + unlink(db); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_fp_c.cpp b/engine/tests/test_fp_c.cpp index e00402b..8d2a2a2 100644 --- a/engine/tests/test_fp_c.cpp +++ b/engine/tests/test_fp_c.cpp @@ -1,8 +1,9 @@ #include "test_e2e.h" -int main() { - // C FP verification: builtins + stdlib + compiler intrinsics - const char* code = R"( +int main() +{ + // C FP verification: builtins + stdlib + compiler intrinsics + const char *code = R"( // User-defined function — should appear in callees int user_function(int x) { return x * 2; @@ -34,13 +35,10 @@ int wrapper(int x) { } )"; - const char* builtins[] = { - "printf", "fprintf", "snprintf", - "__builtin_expect", - nullptr - }; + const char *builtins[] = { "printf", "fprintf", "snprintf", + "__builtin_expect", nullptr }; - runFPVerificationTest("c", code, "/tmp/test_fp_c.c", - "mainFunc", "user_function", builtins); - return 0; + runFPVerificationTest("c", code, "/tmp/test_fp_c.c", "mainFunc", + "user_function", builtins); + return 0; } \ No newline at end of file diff --git a/engine/tests/test_fp_cpp.cpp b/engine/tests/test_fp_cpp.cpp index d46b3dd..18a3c82 100644 --- a/engine/tests/test_fp_cpp.cpp +++ b/engine/tests/test_fp_cpp.cpp @@ -1,10 +1,11 @@ #include "test_e2e.h" -int main() { - // C++ FP verification: builtins + stdlib + compiler intrinsics - // NOTE: use standalone functions, not class methods, for reliable - // call-edge detection in the old API translator. - const char* code = R"( +int main() +{ + // C++ FP verification: builtins + stdlib + compiler intrinsics + // NOTE: use standalone functions, not class methods, for reliable + // call-edge detection in the old API translator. + const char *code = R"( #include // User-defined function — should appear in callees @@ -28,12 +29,9 @@ int mainFunc() { } )"; - const char* builtins[] = { - "printf", "__builtin_expect", - nullptr - }; + const char *builtins[] = { "printf", "__builtin_expect", nullptr }; - runFPVerificationTest("cpp", code, "/tmp/test_fp_cpp.cpp", - "mainFunc", "user_function", builtins); - return 0; + runFPVerificationTest("cpp", code, "/tmp/test_fp_cpp.cpp", "mainFunc", + "user_function", builtins); + return 0; } \ No newline at end of file diff --git a/engine/tests/test_fp_go.cpp b/engine/tests/test_fp_go.cpp index 157893b..c568730 100644 --- a/engine/tests/test_fp_go.cpp +++ b/engine/tests/test_fp_go.cpp @@ -1,8 +1,9 @@ #include "test_e2e.h" -int main() { - // Go FP verification: builtins + unexported + FFI - const char* code = R"(package main +int main() +{ + // Go FP verification: builtins + unexported + FFI + const char *code = R"(package main import "fmt" @@ -53,13 +54,13 @@ func mainFunc() { } )"; - const char* builtins[] = { - "Len", "Append", "Copy", "Cap", "Make", "New", "Delete", - "Println", // fmt.Println is selector expression, not bare call - nullptr - }; + const char *builtins[] = { + "Len", "Append", "Copy", "Cap", "Make", "New", "Delete", + "Println", // fmt.Println is selector expression, not bare call + nullptr + }; - runFPVerificationTest("go", code, "/tmp/test_fp_go.go", - "mainFunc", "userFunction", builtins); - return 0; + runFPVerificationTest("go", code, "/tmp/test_fp_go.go", "mainFunc", + "userFunction", builtins); + return 0; } \ No newline at end of file diff --git a/engine/tests/test_fp_java.cpp b/engine/tests/test_fp_java.cpp index b1009aa..4f1ef23 100644 --- a/engine/tests/test_fp_java.cpp +++ b/engine/tests/test_fp_java.cpp @@ -1,10 +1,11 @@ #include "test_e2e.h" -int main() { - // Java FP verification: common JDK methods - // NOTE: use a standalone public class with a main method, not nested classes, - // for reliable call-edge detection in the old API translator. - const char* code = R"JAVA( +int main() +{ + // Java FP verification: common JDK methods + // NOTE: use a standalone public class with a main method, not nested classes, + // for reliable call-edge detection in the old API translator. + const char *code = R"JAVA( class Calculator { public int userFunction(int x) { return x * 2; @@ -23,12 +24,9 @@ class Calculator { } )JAVA"; - const char* builtins[] = { - "toString", "length", "charAt", - nullptr - }; + const char *builtins[] = { "toString", "length", "charAt", nullptr }; - runFPVerificationTest("java", code, "/tmp/Calculator.java", - "mainFunc", "userFunction", builtins); - return 0; + runFPVerificationTest("java", code, "/tmp/Calculator.java", "mainFunc", + "userFunction", builtins); + return 0; } \ No newline at end of file diff --git a/engine/tests/test_fp_js.cpp b/engine/tests/test_fp_js.cpp index ff3a821..b158530 100644 --- a/engine/tests/test_fp_js.cpp +++ b/engine/tests/test_fp_js.cpp @@ -1,8 +1,9 @@ #include "test_e2e.h" -int main() { - // JavaScript FP verification: builtins + global constructors - const char* code = R"( +int main() +{ + // JavaScript FP verification: builtins + global constructors + const char *code = R"( // User-defined function — should appear in callees function userFunction(x) { return x * 2; @@ -34,14 +35,14 @@ function mainFunc() { } )"; - const char* builtins[] = { - "String", "Number", "Boolean", "Array", "Object", - "parseInt", "parseFloat", "isNaN", "isFinite", - "encodeURIComponent", - nullptr - }; + const char *builtins[] = { "String", "Number", + "Boolean", "Array", + "Object", "parseInt", + "parseFloat", "isNaN", + "isFinite", "encodeURIComponent", + nullptr }; - runFPVerificationTest("js", code, "/tmp/test_fp_js.js", - "mainFunc", "userFunction", builtins); - return 0; + runFPVerificationTest("js", code, "/tmp/test_fp_js.js", "mainFunc", + "userFunction", builtins); + return 0; } \ No newline at end of file diff --git a/engine/tests/test_fp_python.cpp b/engine/tests/test_fp_python.cpp index 95aa962..b6bdce6 100644 --- a/engine/tests/test_fp_python.cpp +++ b/engine/tests/test_fp_python.cpp @@ -1,8 +1,9 @@ #include "test_e2e.h" -int main() { - // Python FP verification: builtins + dunder methods - const char* code = R"PY( +int main() +{ + // Python FP verification: builtins + dunder methods + const char *code = R"PY( # User-defined function — should appear in callees def user_function(x): return x * 2 @@ -37,14 +38,11 @@ def mainFunc(): _ = _private_helper(r1) )PY"; - const char* builtins[] = { - "len", "str", "int", "list", "dict", "range", - "print", "type", "bool", "float", "max", "min", - "sum", "sorted", - nullptr - }; + const char *builtins[] = { "len", "str", "int", "list", "dict", + "range", "print", "type", "bool", "float", + "max", "min", "sum", "sorted", nullptr }; - runFPVerificationTest("python", code, "/tmp/test_fp_python.py", - "mainFunc", "user_function", builtins); - return 0; + runFPVerificationTest("python", code, "/tmp/test_fp_python.py", + "mainFunc", "user_function", builtins); + return 0; } \ No newline at end of file diff --git a/engine/tests/test_fp_rust.cpp b/engine/tests/test_fp_rust.cpp index e2055b8..8f8f977 100644 --- a/engine/tests/test_fp_rust.cpp +++ b/engine/tests/test_fp_rust.cpp @@ -1,12 +1,13 @@ #include "test_e2e.h" -int main() { - // Rust FP verification: builtins + macros + visibility - // NOTE: The old API RustTranslator has issues with functions that have - // no parameters. Use a dummy parameter to work around this. - // The new API (RustVisitor) used by the project pipeline handles all - // patterns correctly — the builtin filtering is verified at project level. - const char* code = R"( +int main() +{ + // Rust FP verification: builtins + macros + visibility + // NOTE: The old API RustTranslator has issues with functions that have + // no parameters. Use a dummy parameter to work around this. + // The new API (RustVisitor) used by the project pipeline handles all + // patterns correctly — the builtin filtering is verified at project level. + const char *code = R"( fn add(x: i32, y: i32) -> i32 { x + y } @@ -31,11 +32,9 @@ fn mainFunc(_x: i32) -> i32 { } )"; - const char* builtins[] = { - nullptr - }; + const char *builtins[] = { nullptr }; - runFPVerificationTest("rust", code, "/tmp/test_fp_rust.rs", - "mainFunc", "user_function", builtins); - return 0; + runFPVerificationTest("rust", code, "/tmp/test_fp_rust.rs", "mainFunc", + "user_function", builtins); + return 0; } \ No newline at end of file diff --git a/engine/tests/test_fp_ts.cpp b/engine/tests/test_fp_ts.cpp index 3dc5bea..b6bfa86 100644 --- a/engine/tests/test_fp_ts.cpp +++ b/engine/tests/test_fp_ts.cpp @@ -1,8 +1,9 @@ #include "test_e2e.h" -int main() { - // TypeScript FP verification: builtins + type constructors - const char* code = R"TS( +int main() +{ + // TypeScript FP verification: builtins + type constructors + const char *code = R"TS( // User-defined function — should appear in callees function userFunction(x: number): number { return x * 2; @@ -25,13 +26,10 @@ function mainFunc(): number { } )TS"; - const char* builtins[] = { - "String", "Number", "Boolean", "Array", - "parseInt", "parseFloat", - nullptr - }; + const char *builtins[] = { "String", "Number", "Boolean", "Array", + "parseInt", "parseFloat", nullptr }; - runFPVerificationTest("ts", code, "/tmp/test_fp_ts.ts", - "mainFunc", "userFunction", builtins); - return 0; + runFPVerificationTest("ts", code, "/tmp/test_fp_ts.ts", "mainFunc", + "userFunction", builtins); + return 0; } \ No newline at end of file diff --git a/engine/tests/test_fts_rebuild.cpp b/engine/tests/test_fts_rebuild.cpp new file mode 100644 index 0000000..f88a860 --- /dev/null +++ b/engine/tests/test_fts_rebuild.cpp @@ -0,0 +1,130 @@ +// test_fts_rebuild.cpp — regression test for stale/ghost FTS rows after a +// file is re-indexed (2026-09-27 review, FTS non-idempotency). +// +// Before the fix: +// buildFTSFromGraph rebuilt code_fts / name_trgm with +// `INSERT OR IGNORE INTO … (rowid, …) SELECT e.id, …` and nothing ever +// deleted FTS rows. Re-indexing a file re-uses entity ids, so the OR IGNORE +// collided with the stale rowid: the old symbol stayed in code_fts (a ghost +// that searchUnifiedJson returns — its FTS branch does not JOIN entity) and +// the new symbol was silently dropped. `search` then returned deleted names +// and missed renamed ones. +// +// After the fix: +// buildFTSFromGraph deletes the project's FTS rows first, so a rebuild is +// idempotent: the renamed symbol is indexed and the old one is gone. +// +// This is a store-level test (no async knowledge-builder thread) so it is +// deterministic: it drives buildGraph + buildFTSFromGraph directly and reads +// code_fts back. + +#include "../include/engine.h" +#include "../src/ir/semantic_unit.h" +#include "../src/store/store.h" + +#include +#include +#include +#include +#include +#include + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "FAIL: %s\n", msg); + exit(1); + } +} + +// Count code_fts rows for a given name in a project. +static int ftsCount(store::GraphStore &store, uint64_t pid, const char *name) +{ + sqlite3 *db = store.handle(); + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2( + db, + "SELECT COUNT(*) FROM code_fts WHERE project_id=? AND name=?", + -1, &st, nullptr) == SQLITE_OK, + "prepare ftsCount"); + sqlite3_bind_int64(st, 1, static_cast(pid)); + sqlite3_bind_text(st, 2, name, -1, SQLITE_TRANSIENT); + int n = 0; + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + return n; +} + +static ir::Record fn(const char *name, const char *file, uint64_t id) +{ + ir::Record r; + r.id = id; + r.original_id = id; + r.kind = ir::RecordKind::Function; + r.name = name; + r.qualified_name = name; + r.file_path = file; + r.language = "cpp"; + r.loc = ir::SourceRange{ 1, 0, 2, 0 }; + return r; +} + +int main() +{ + const char *db_path = "/tmp/test_fts_rebuild.db"; + unlink(db_path); + + store::GraphStore store; + check(store.open(db_path), "open database"); + uint64_t pid = store.createProject("/proj", "fts-rebuild"); + check(pid > 0, "createProject"); + + const char *file = "/proj/pkg/a.cpp"; + + // Pass 1: index a file defining alphaFunc. + std::vector r1{ fn("alphaFunc", file, 1) }; + store.insertSemanticRecords(pid, file, r1); + check(store.buildGraph(pid, false), "buildGraph pass 1"); + store.buildFTSFromGraph(pid); + check(store.error().empty(), "buildFTSFromGraph pass 1 ok"); + check(ftsCount(store, pid, "alphaFunc") == 1, + "alphaFunc must be in code_fts after first index"); + + // Pass 2: re-index the same file, now defining betaFunc. This mirrors + // the incremental re-index path (insertFileResultBatch with + // is_reindex=true): delete the file's semantic_records and graph rows, + // insert the new record, rebuild the graph and FTS. + { + sqlite3_stmt *del = nullptr; + check(sqlite3_prepare_v2( + store.handle(), + "DELETE FROM semantic_records WHERE project_id=? " + "AND file_path=?", + -1, &del, nullptr) == SQLITE_OK, + "prepare delete semantic_records"); + sqlite3_bind_int64(del, 1, static_cast(pid)); + sqlite3_bind_text(del, 2, file, -1, SQLITE_TRANSIENT); + check(sqlite3_step(del) == SQLITE_DONE, + "delete semantic_records"); + sqlite3_finalize(del); + } + check(store.deleteGraphDataByFile(pid, file), "delete old file data"); + std::vector r2{ fn("betaFunc", file, 1) }; + store.insertSemanticRecords(pid, file, r2); + check(store.buildGraph(pid, false), "buildGraph pass 2"); + store.buildFTSFromGraph(pid); + check(store.error().empty(), "buildFTSFromGraph pass 2 ok"); + + // The renamed symbol must be searchable (dropped before the fix). + check(ftsCount(store, pid, "betaFunc") == 1, + "betaFunc must be in code_fts after re-index"); + // The old symbol must be gone (a ghost before the fix). + check(ftsCount(store, pid, "alphaFunc") == 0, + "alphaFunc must NOT remain in code_fts after rename"); + + store.close(); + unlink(db_path); + printf("\n=== FTS rebuild idempotency test passed ===\n"); + return 0; +} diff --git a/engine/tests/test_function_arity.cpp b/engine/tests/test_function_arity.cpp new file mode 100644 index 0000000..bcd943b --- /dev/null +++ b/engine/tests/test_function_arity.cpp @@ -0,0 +1,278 @@ +// test_function_arity — parameter counts must reach entity.param_count AND +// entity.arity, on the real index path, for the real grammars. +// +// Defect guarded: no Visitor emits RecordKind::Parameter, so the record-side +// walk in computeMetricsFromCST counted nothing and `param_count` was 0 on +// every function of every project (goagent: 6163 function entities, 0 with +// param_count > 0) while cyclomatic/lines/cognitive were correct. The same +// missing producer left `arity` at 0 — and `arity` is what the Resolver reads +// to weigh same-name overloads (factorSignatureMatch), so every candidate +// scored as "unknown arity" and the factor could neither prefer the overload +// whose parameter count matches the call site nor penalise the ones that do not +// (entity.arity was 0 for all 24604 goagent rows while reference.arity carried +// 16455 real call-site counts). +// +// The counts therefore come from the CST's own declaration — the function's +// `parameters` field — and are written to both the metric row (param_count) and +// the record (arity, which buildGraph copies into entity.arity). +// +// The fixture pins the traps, per language: +// C++ two_params(int,int) 2 / 2 +// no_params(void) 0 / 0 (`(void)` declares none) +// callback_param(int (*cb)(int,int),int) 2 / 2 (the callback's own +// list is not this function's) +// Go methodWithTwo declared 2 / arity 2 (the receiver +// is a separate field, not a param) +// plainNoParams 0 / 0 +// variadicSum(base int, rest ...int) 2 / 0 (a variadic tail accepts +// any number of extra arguments, +// so no single arity is right) +// Python py_two 2 / 2 +// py_default(a, b=1) 2 / 0 (a default value lets a +// call pass fewer than declared) +// Widget.render(self, canvas, scale) 3 / 2 (a call site passes the +// arguments only, never `self`) +// +// `declared` is reported by entity.param_count, `arity` by entity.arity and +// semantic_records.arity. Asserting both forms for every case is the point: the +// metric and the resolver input must come from one measurement, not drift apart. +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include + +#include "test_check.h" +#include "test_engine_handle.h" + +namespace +{ + +/// Write `content` to `path`, failing the test if the file cannot be created. +void writeFile(const std::string &path, const char *content) +{ + FILE *f = fopen(path.c_str(), "w"); + CHECK(f != nullptr); + fputs(content, f); + fclose(f); +} + +/// One scalar read of `sql` (one `?` placeholder) bound to `name`. +long long scalarForName(sqlite3 *db, const char *sql, const char *name) +{ + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + sqlite3_bind_text(st, 1, name, -1, SQLITE_TRANSIENT); + long long value = -1; + if (sqlite3_step(st) == SQLITE_ROW) + value = sqlite3_column_int64(st, 0); + sqlite3_finalize(st); + return value; +} + +/// entity.param_count — the published code metric. +long long declaredParams(sqlite3 *db, const char *name) +{ + return scalarForName(db, + "SELECT param_count FROM entity " + "WHERE kind IN (0,1) AND name=?", + name); +} + +/// entity.arity — what the Resolver compares a call site against. +long long entityArity(sqlite3 *db, const char *name) +{ + return scalarForName(db, "SELECT arity FROM entity " + "WHERE kind IN (0,1) AND name=?", + name); +} + +/// semantic_records.arity — the canonical fact buildGraph copies into +/// entity.arity. +long long recordArity(sqlite3 *db, const char *name) +{ + return scalarForName(db, "SELECT arity FROM semantic_records " + "WHERE kind IN (0,1) AND name=?", + name); +} + +/// Assert one measurement, naming the function and the column on failure. +void expect(long long got, long long want, const char *column, const char *name) +{ + if (got != want) + fprintf(stderr, "FAIL: %s of \"%s\" = %lld, expected %lld\n", + column, name, got, want); + CHECK(got == want); +} + +} // namespace + +int main() +{ + const std::string proj_dir = "/tmp/function_arity_repro"; + std::filesystem::remove_all(proj_dir); + std::filesystem::create_directories(proj_dir); + + writeFile(proj_dir + "/a.cpp", + "int two_params(int a, int b)\n" + "{\n" + "\treturn a + b;\n" + "}\n" + "\n" + "int no_params(void)\n" + "{\n" + "\treturn 0;\n" + "}\n" + "\n" + "int callback_param(int (*cb)(int a, int b), int x)\n" + "{\n" + "\treturn cb(x, 1);\n" + "}\n" + "\n" + "int with_default(int a, int b = 0)\n" + "{\n" + "\treturn a + b;\n" + "}\n"); + writeFile(proj_dir + "/b.go", + "package main\n" + "\n" + "type T struct{}\n" + "\n" + "func (t *T) methodWithTwo(a int, b int) int { return a + b }\n" + "\n" + "func plainNoParams() int { return 0 }\n" + "\n" + "func variadicSum(base int, rest ...int) int { return base }\n"); + writeFile(proj_dir + "/c.py", + "def py_two(a, b):\n" + " return a + b\n" + "\n" + "\n" + "def py_default(a, b=1):\n" + " return a\n" + "\n" + "\n" + "class Widget:\n" + " def render(self, canvas, scale):\n" + " return canvas\n"); + // C, and the exact shape the bounded declarator search exists for: the + // callback's own `(int a)` hangs one level below `takes_cb`'s declarator, and + // counting it would report three parameters for a two-parameter function. + writeFile(proj_dir + "/d.c", + "int takes_cb(int (*cb)(int a), int x)\n" + "{\n" + "\treturn cb(x);\n" + "}\n"); + // Python classmethod: `cls` is a receiver exactly like `self`, and only the + // enclosing-class check tells it apart from a free function whose first + // parameter happens to be called `cls`. + writeFile(proj_dir + "/e.py", + "class Entity:\n" + " @classmethod\n" + " def validate(cls, payload):\n" + " return payload\n"); + // Rust spells the receiver as its own node (`&self`) instead of a named + // first parameter, so it takes the self_parameter branch, not the name one. + writeFile(proj_dir + "/f.rs", + "struct Widget;\n" + "\n" + "impl Widget {\n" + " fn renderRust(&self, canvas: u32) -> u32 { canvas }\n" + "}\n" + "\n" + "fn rust_two(a: i32, b: i32) -> i32 { a + b }\n"); + // Java's parameter list is `formal_parameters`, a third node type. + writeFile(proj_dir + "/g.java", + "public class WidgetJ {\n" + " public int javaTwo(int canvas, int scale) {\n" + " return canvas + scale;\n" + " }\n" + "}\n"); + + char db[] = "/tmp/test_function_arity.db"; + unlink(db); + g_engine = engine_create(db); + CHECK(g_engine != nullptr); + const uint64_t pid = engine_create_project(g_engine, proj_dir.c_str(), + "function-arity"); + CHECK(pid > 0); + char *idx = + engine_index_project(g_engine, pid, proj_dir.c_str(), nullptr); + CHECK(idx != nullptr); + CHECK(strstr(idx, "\"ok\":true") != nullptr); + engine_free_string(idx); + + sqlite3 *db_h = nullptr; + CHECK(sqlite3_open(db, &db_h) == SQLITE_OK); + + // Every fixture function must exist, or the checks below would pass by + // measuring nothing. + struct Case { + const char *name; + long long declared; + long long arity; + }; + const Case cases[] = { + { "two_params", 2, 2 }, + { "no_params", 0, 0 }, + { "callback_param", 2, 2 }, + // A C++ default value leaves the argument count open: the metric stays + // the declared 2, but a call that supplies one argument is valid, so + // the resolver input is unknown. Without this the declaration was + // recorded as arity 2 and `with_default(1)` was PENALISED — on this + // repository that cost `policyFor -> loadIgnoreFile`, + // `indexProjectImpl -> loadGitignore` and every other call that omits a + // defaulted argument, which is how the omission was found: a rebuilt + // previous commit diffed against this one. + { "with_default", 2, 0 }, + { "methodWithTwo", 2, 2 }, + { "plainNoParams", 0, 0 }, + { "py_two", 2, 2 }, + // A variadic tail and a default value each let a VALID call supply a + // different number of arguments than the declaration lists, so the + // resolver input is unknown (0) while the metric stays the declared + // count. Without this, summarising `variadicSum(1)` as arity 2 would + // have penalised the one call that names it (goagent alone has 155 + // variadic functions). + { "variadicSum", 2, 0 }, + { "py_default", 2, 0 }, + // C: `cb`'s own parameter list must not be counted as `takes_cb`'s. + { "takes_cb", 2, 2 }, + // The per-language receiver branches, each reached by exactly one of + // these: Python's `cls` (name + class body), Rust's `&self` (its own + // node type) and Java's `formal_parameters` (a third list node type). + { "validate", 2, 1 }, + { "renderRust", 2, 1 }, + { "rust_two", 2, 2 }, + { "javaTwo", 2, 2 }, + }; + for (const Case &c : cases) { + expect(declaredParams(db_h, c.name), c.declared, "param_count", + c.name); + expect(entityArity(db_h, c.name), c.arity, "entity.arity", c.name); + expect(recordArity(db_h, c.name), c.arity, "record arity", c.name); + } + + // Python's receiver: 3 declared parameters, 2 supplied by a call site. + CHECK(declaredParams(db_h, "render") == 3); + CHECK(entityArity(db_h, "render") == 2); + CHECK(recordArity(db_h, "render") == 2); + + // Guard against a vacuous pass: the fixture has to have produced the + // function entities the assertions above read. + CHECK(scalarForName(db_h, + "SELECT COUNT(*) FROM entity WHERE kind IN (0,1) AND " + "name=?", + "two_params") == 1); + + printf("=== function arity/param_count test passed ===\n"); + sqlite3_close(db_h); + engine_destroy(g_engine); + g_engine = nullptr; + unlink(db); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_fuzzy_resolver.cpp b/engine/tests/test_fuzzy_resolver.cpp index 7e773c8..d3c1acb 100644 --- a/engine/tests/test_fuzzy_resolver.cpp +++ b/engine/tests/test_fuzzy_resolver.cpp @@ -11,7 +11,7 @@ #include "../src/resolver/fuzzy_resolver.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -30,11 +30,11 @@ static void insertEntity(store::GraphStore &store, uint64_t project_id, "start_col, end_row, end_col) " "VALUES (?,?,0,?,'','/test.cpp','cpp',0,0,0,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -44,10 +44,10 @@ int main() unlink(kDbPath); store::GraphStore store; - assert(store.open(kDbPath)); + CHECK(store.open(kDbPath)); uint64_t project_id = store.createProject("/test", "test_project"); - assert(project_id > 0); + CHECK(project_id > 0); // Insert entities with varied names for fuzzy matching: // 1: LoggerFactory — mixed case, good for prefix + suffix @@ -65,16 +65,16 @@ int main() { // "LOGGER" should match entity 3 "logger" (case-insensitive) auto results = resolver.resolve("LOGGER", 5); - assert(results.size() >= 1); - assert(results[0] == 3); + CHECK(results.size() >= 1); + CHECK(results[0] == 3); printf(" [PASS] case-insensitive: LOGGER -> logger(id=3)\n"); } { // "LOGGERFACTORY" should match entity 1 "LoggerFactory" auto results = resolver.resolve("LOGGERFACTORY", 5); - assert(results.size() >= 1); - assert(results[0] == 1); + CHECK(results.size() >= 1); + CHECK(results[0] == 1); printf(" [PASS] case-insensitive: LOGGERFACTORY -> LoggerFactory(id=1)\n"); } @@ -83,25 +83,25 @@ int main() // "LoggerF" (7 chars, >= 3) has no case-insensitive exact match, // so it falls through to prefix matching and finds "LoggerFactory". auto results = resolver.resolve("LoggerF", 5); - assert(results.size() >= 1); + CHECK(results.size() >= 1); // Entity 1 "LoggerFactory" starts with "LoggerF" bool found_1 = false; for (auto id : results) if (id == 1) found_1 = true; - assert(found_1); + CHECK(found_1); printf(" [PASS] prefix: LoggerF -> LoggerFactory(id=1)\n"); } { // "Use" (3 chars, exactly the minimum) should match "UserFactory" auto results = resolver.resolve("Use", 5); - assert(results.size() >= 1); + CHECK(results.size() >= 1); bool found_2 = false; for (auto id : results) if (id == 2) found_2 = true; - assert(found_2); + CHECK(found_2); printf(" [PASS] prefix: Use -> UserFactory(id=2)\n"); } @@ -109,7 +109,7 @@ int main() // "Au" (2 chars, < 3) should NOT trigger prefix match (too short). // Also no case-insensitive match for "Au" alone, so result is empty. auto results = resolver.resolve("Au", 5); - assert(results.empty()); + CHECK(results.empty()); printf(" [PASS] prefix rejected: Au (< 3 chars) -> empty\n"); } @@ -117,7 +117,7 @@ int main() { // "Factory" (7 chars) should match both "LoggerFactory" and "UserFactory" auto results = resolver.resolve("Factory", 10); - assert(results.size() >= 2); + CHECK(results.size() >= 2); bool found_1 = false, found_2 = false; for (auto id : results) { if (id == 1) @@ -125,8 +125,8 @@ int main() if (id == 2) found_2 = true; } - assert(found_1); - assert(found_2); + CHECK(found_1); + CHECK(found_2); printf(" [PASS] suffix: Factory -> LoggerFactory(id=1) + UserFactory(id=2)\n"); } @@ -134,21 +134,21 @@ int main() { // Empty name should return empty auto results = resolver.resolve("", 5); - assert(results.empty()); + CHECK(results.empty()); printf(" [PASS] empty name -> empty\n"); } { // Non-existent name should return empty auto results = resolver.resolve("NonexistentSymbol", 5); - assert(results.empty()); + CHECK(results.empty()); printf(" [PASS] non-existent -> empty\n"); } { // Limit is respected: request only 1 result for "Factory" auto results = resolver.resolve("Factory", 1); - assert(results.size() == 1); + CHECK(results.size() == 1); printf(" [PASS] limit=1 respected: Factory -> 1 result\n"); } @@ -156,5 +156,5 @@ int main() unlink(kDbPath); printf("=== FuzzyResolver test passed ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_gitignore_build_dirs.cpp b/engine/tests/test_gitignore_build_dirs.cpp new file mode 100644 index 0000000..4e2b28a --- /dev/null +++ b/engine/tests/test_gitignore_build_dirs.cpp @@ -0,0 +1,141 @@ +// test_gitignore_build_dirs.cpp — regression test for `.gitignore` directory +// pruning, written after a real-project index was found to contain entities and +// modules from build directories. +// +// Observed on this repository (fresh self-index, 2026-09-21): +// * 14 entities whose file_path is inside `build-cold` / `build-verify` / +// `build-release`(x2 CMake versions) / `build-tests`(x2) / +// `build-release-linux`, all of them CMake's compiler-probe files +// (`CMakeFiles//CompilerIdC*/CMakeCCompilerId.c` and `…CXX.cpp`); +// * 7 of the 43 rows in the `modules` table are those `…/CMakeFiles/` +// directories — i.e. the leak is user-visible through get_module_tree. +// +// `engine/build/` contributes zero, and the difference between that and +// `engine/build-cold/` is exactly what this file pins down: `build` is in +// FilterPolicy's hard-skip list while `build-*` is not, and the repository's +// `.gitignore` — which does list `**/build-*/` — is what is supposed to cover +// the rest. +// +// Uses the explicit check() pattern: assert() is compiled out under NDEBUG, +// which would make these checks vacuous in a Release build. +// +// Exit code: 0 on success, 1 on any failed assertion. + +#include "../src/filter_policy.h" + +#include +#include +#include +#include +#include + +static int g_failed = 0; + +static void check(bool cond, const std::string &what) +{ + if (cond) { + printf(" [PASS] %s\n", what.c_str()); + return; + } + fprintf(stderr, " [FAIL] %s\n", what.c_str()); + g_failed++; +} + +static void expectSkip(const FilterPolicy &p, const std::string &rel, + bool is_dir, bool want_skip, const char *why) +{ + bool got = p.shouldSkipEntry(rel, is_dir); + bool ok = (got == want_skip); + printf(" [%s] shouldSkipEntry(\"%s\", is_dir=%d) = %s, want %s (%s)\n", + ok ? "PASS" : "FAIL", rel.c_str(), is_dir ? 1 : 0, + got ? "SKIP" : "INDEX", want_skip ? "SKIP" : "INDEX", why); + if (!ok) + g_failed++; +} + +int main() +{ + std::error_code ec; + auto root = std::filesystem::temp_directory_path(ec) / + "codescope_gitignore_build_dirs"; + std::filesystem::remove_all(root, ec); + std::filesystem::create_directories(root, ec); + if (ec) { + fprintf(stderr, "FAIL: cannot create fixture %s: %s\n", + root.c_str(), ec.message().c_str()); + return 1; + } + + // The fixture mirrors this repository's shape: a .gitignore that ignores + // build trees by pattern, a source tree next to them, and the CMake + // compiler-probe files that a build directory legitimately contains. + const char *gitignore = "# fixture\n" + "**/build/\n" + "**/build-*/\n" + "node_modules/\n" + "*.o\n"; + { + std::ofstream f(root / ".gitignore"); + f << gitignore; + } + const char *probe = "CMakeFiles/4.4.3/CompilerIdC/CMakeCCompilerId.c"; + std::filesystem::create_directories( + root / "engine" / "build-cold" / + std::filesystem::path(probe).parent_path(), + ec); + std::filesystem::create_directories(root / "engine" / "src", ec); + { + std::ofstream f(root / "engine" / "build-cold" / probe); + f << "int main(void){return 0;}\n"; + std::ofstream g(root / "engine" / "src" / "app.cpp"); + g << "int main(){return 0;}\n"; + } + + FilterPolicy p; + bool loaded = p.loadGitignore(root.string()); + printf("=== .gitignore build-directory pruning ===\n"); + check(loaded, "fixture .gitignore loaded"); + + // The directory itself — this is the case that decides whether the walker + // ever descends into the tree at all. + expectSkip(p, "engine/build-cold", true, true, + "`**/build-*/` names the directory"); + expectSkip(p, "engine/build", true, true, + "`**/build/` (also hard-skipped)"); + expectSkip(p, "engine/build-release-linux", true, true, + "second build-* directory"); + + // The files inside it. Per gitignore semantics an ignored directory + // ignores everything under it, so these must be skipped even though the + // pattern ends at the directory name. + expectSkip(p, "engine/build-cold/" + std::string(probe), false, true, + "file inside an ignored directory"); + expectSkip(p, "engine/build-release-linux/" + std::string(probe), false, + true, "file inside a second ignored directory"); + + // Controls: the fix must not start skipping ordinary source. + expectSkip(p, "engine/src/app.cpp", false, false, "first-party source"); + expectSkip(p, "engine/src", true, false, "source directory"); + + // The parallel scheduler's case: a per-module worker scans + // `/` and hands the policy MODULE-relative paths, while the + // rules it loaded are anchored at the project root. The prefix puts the + // module component back, which is what lets a rule ending at the module + // name exclude the module's whole content. Without it, the module's own + // files were indexed (observed as a `build-x` module in the merged graph). + FilterPolicy mp; + check(mp.loadGitignore(root.string()), + "module worker: .gitignore loaded"); + mp.setScanPrefix("engine/"); + expectSkip(mp, "build-cold/" + std::string(probe), false, true, + "module-relative path, project-root rule"); + expectSkip(mp, "build-release-linux", true, true, + "module-relative directory"); + expectSkip(mp, "src/app.cpp", false, false, + "module-relative first-party source"); + + printf("=== %s (%d failure%s) ===\n", g_failed ? "FAILED" : "PASSED", + g_failed, g_failed == 1 ? "" : "s"); + std::filesystem::remove_all(root, ec); + return g_failed ? 1 : 0; +} diff --git a/engine/tests/test_go_e2e.cpp b/engine/tests/test_go_e2e.cpp index d5b85b9..f00b423 100644 --- a/engine/tests/test_go_e2e.cpp +++ b/engine/tests/test_go_e2e.cpp @@ -1,7 +1,8 @@ #include "test_e2e.h" -int main() { - const char* code = R"(package main +int main() +{ + const char *code = R"(package main import "fmt" @@ -29,12 +30,8 @@ func main() { } )"; - const char* defs[] = {"Calculator", "Add"}; - runE2eTest("go", code, "/tmp/calculator.go", - defs, 2, - "Add", "main", - "Add", - "main", - "main", nullptr); - return 0; + const char *defs[] = { "Calculator", "Add" }; + runE2eTest("go", code, "/tmp/calculator.go", defs, 2, "Add", "main", + "Add", "main", "main", nullptr); + return 0; } diff --git a/engine/tests/test_graph.cpp b/engine/tests/test_graph.cpp index 1af6bfe..047c1e7 100644 --- a/engine/tests/test_graph.cpp +++ b/engine/tests/test_graph.cpp @@ -1,99 +1,103 @@ #include "../src/ir/ir.h" #include "../src/graph/graph_builder.h" -#include +#include "test_check.h" #include #include -int main() { - using namespace ir; - using namespace graph; - - // Build a minimal IR: one file with a function and a call - auto* unit = new TranslationUnit(); - - auto* file = new Node(); - file->kind = NodeKind::TranslationUnit; - file->language = "python"; - file->file_path = "/test/test.py"; - file->name = "test.py"; - - auto* func = new Node(); - func->kind = NodeKind::FunctionDecl; - func->name = "compute"; - func->qualified_name = "compute"; - func->language = "python"; - func->file_path = "/test/test.py"; - func->loc = SourceLocation{1, 0, 4, 0}; - - auto* main_func = new Node(); - main_func->kind = NodeKind::FunctionDecl; - main_func->name = "main"; - main_func->qualified_name = "main"; - main_func->language = "python"; - main_func->file_path = "/test/test.py"; - main_func->loc = SourceLocation{6, 0, 9, 0}; - - auto* call = new Node(); - call->kind = NodeKind::CallExpr; - call->language = "python"; - call->file_path = "/test/test.py"; - call->loc = SourceLocation{7, 4, 7, 15}; - - // Build tree: file -> [func, main_func], main_func -> [call] - file->children.push_back(func); - file->children.push_back(main_func); - main_func->children.push_back(call); - - // Semantic edge: call → func - call->semantic_edges.emplace_back(func, Relation::CallTarget); - - unit->root = file; - std::function collect = [&](Node* n) { - unit->all_nodes.push_back(n); - for (auto* c : n->children) collect(c); - }; - collect(file); - unit->assignIds(); - - // ── Build graphs ─────────────────────────────────────────── - - GraphBuilder builder(1); // project_id = 1 - - auto symbol_graph = builder.buildSymbolGraph(unit); - auto call_graph = builder.buildCallGraph(unit); - - // ── Verify Symbol Graph ──────────────────────────────────── - - // Should have nodes: file, func, main_func - assert(symbol_graph.nodes.size() >= 2); - - bool has_func = false, has_main = false; - for (auto& n : symbol_graph.nodes) { - if (n.name == "compute") has_func = true; - if (n.name == "main") has_main = true; - } - assert(has_func); - assert(has_main); - printf("Symbol graph: %zu nodes, %zu edges\n", - symbol_graph.nodes.size(), symbol_graph.edges.size()); - - // ── Verify Call Graph ────────────────────────────────────── - - // Should have one edge: main → compute (Calls) - bool call_edge_found = false; - for (auto& e : call_graph.edges) { - if (e.type == EdgeType::Calls) { - call_edge_found = true; - break; - } - } - // Call edge should be in the call graph - printf("Call graph: %zu nodes, %zu edges (call edge found: %s)\n", - call_graph.nodes.size(), call_graph.edges.size(), - call_edge_found ? "yes" : "no"); - - printf("=== Graph builder test passed ===\n"); - - delete unit; - return 0; +int main() +{ + using namespace ir; + using namespace graph; + + // Build a minimal IR: one file with a function and a call + auto *unit = new TranslationUnit(); + + auto *file = new Node(); + file->kind = NodeKind::TranslationUnit; + file->language = "python"; + file->file_path = "/test/test.py"; + file->name = "test.py"; + + auto *func = new Node(); + func->kind = NodeKind::FunctionDecl; + func->name = "compute"; + func->qualified_name = "compute"; + func->language = "python"; + func->file_path = "/test/test.py"; + func->loc = SourceLocation{ 1, 0, 4, 0 }; + + auto *main_func = new Node(); + main_func->kind = NodeKind::FunctionDecl; + main_func->name = "main"; + main_func->qualified_name = "main"; + main_func->language = "python"; + main_func->file_path = "/test/test.py"; + main_func->loc = SourceLocation{ 6, 0, 9, 0 }; + + auto *call = new Node(); + call->kind = NodeKind::CallExpr; + call->language = "python"; + call->file_path = "/test/test.py"; + call->loc = SourceLocation{ 7, 4, 7, 15 }; + + // Build tree: file -> [func, main_func], main_func -> [call] + file->children.push_back(func); + file->children.push_back(main_func); + main_func->children.push_back(call); + + // Semantic edge: call → func + call->semantic_edges.emplace_back(func, Relation::CallTarget); + + unit->root = file; + std::function collect = [&](Node *n) { + unit->all_nodes.push_back(n); + for (auto *c : n->children) + collect(c); + }; + collect(file); + unit->assignIds(); + + // ── Build graphs ─────────────────────────────────────────── + + GraphBuilder builder(1); // project_id = 1 + + auto symbol_graph = builder.buildSymbolGraph(unit); + auto call_graph = builder.buildCallGraph(unit); + + // ── Verify Symbol Graph ──────────────────────────────────── + + // Should have nodes: file, func, main_func + CHECK(symbol_graph.nodes.size() >= 2); + + bool has_func = false, has_main = false; + for (auto &n : symbol_graph.nodes) { + if (n.name == "compute") + has_func = true; + if (n.name == "main") + has_main = true; + } + CHECK(has_func); + CHECK(has_main); + printf("Symbol graph: %zu nodes, %zu edges\n", + symbol_graph.nodes.size(), symbol_graph.edges.size()); + + // ── Verify Call Graph ────────────────────────────────────── + + // Should have one edge: main → compute (Calls) + bool call_edge_found = false; + for (auto &e : call_graph.edges) { + if (e.type == EdgeType::Calls) { + call_edge_found = true; + break; + } + } + // Call edge should be in the call graph + printf("Call graph: %zu nodes, %zu edges (call edge found: %s)\n", + call_graph.nodes.size(), call_graph.edges.size(), + call_edge_found ? "yes" : "no"); + + printf("=== Graph builder test passed ===\n"); + + delete unit; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_graph_call_precision.cpp b/engine/tests/test_graph_call_precision.cpp index e5a83fe..abd56ee 100644 --- a/engine/tests/test_graph_call_precision.cpp +++ b/engine/tests/test_graph_call_precision.cpp @@ -2,7 +2,7 @@ #include "../src/ir/semantic_unit.h" #include "../src/ir/semantic_emitter.h" #include "../src/graph/graph_builder.h" -#include +#include "test_check.h" #include // ── Tests for GraphBuilder layered callee resolution ───────────────────── @@ -53,9 +53,9 @@ static void test_qualified_name_prevents_false_edges() if (n.qualified_name == "ClassB::process") b_gid = n.id; } - assert(main_gid > 0); - assert(a_gid > 0); - assert(b_gid > 0); + CHECK(main_gid > 0); + CHECK(a_gid > 0); + CHECK(b_gid > 0); // Expect: edge main -> ClassA::process (YES) // edge main -> ClassB::process (NO — qualified_name prevents it) @@ -68,8 +68,8 @@ static void test_qualified_name_prevents_false_edges() edge_to_b = true; } } - assert(edge_to_a); - assert(!edge_to_b); + CHECK(edge_to_a); + CHECK(!edge_to_b); printf(" \u2713 test_qualified_name_prevents_false_edges: %zu call edges\n", call_g.edges.size()); @@ -111,9 +111,9 @@ static void test_arity_prevents_false_edges() if (n.name == "init" && n.ir_node_id == fn2) init2_gid = n.id; } - assert(main_gid > 0); - assert(init1_gid > 0); - assert(init2_gid > 0); + CHECK(main_gid > 0); + CHECK(init1_gid > 0); + CHECK(init2_gid > 0); // Expect: edge main -> init(arity=1) (YES — exact arity match) // edge main -> init(arity=2) (NO — arity mismatch) @@ -126,8 +126,8 @@ static void test_arity_prevents_false_edges() edge_to_init2 = true; } } - assert(edge_to_init1); - assert(!edge_to_init2); + CHECK(edge_to_init1); + CHECK(!edge_to_init2); printf(" \u2713 test_arity_prevents_false_edges: %zu call edges\n", call_g.edges.size()); @@ -155,7 +155,7 @@ static void test_ref_original_id_resolves_exactly() emitter.emitCall("handle", { 10, 4, 10, 15 }, main_fn); // Resolve the call precisely to fn_a via ref_original_id bool ok = unit.setCallReference(call_id, fn_a); - assert(ok); + CHECK(ok); GraphBuilder builder(1); auto sym_g = builder.buildSymbolGraph(unit); @@ -171,9 +171,9 @@ static void test_ref_original_id_resolves_exactly() if (n.name == "handle" && n.ir_node_id == fn_b) b_gid = n.id; } - assert(main_gid > 0); - assert(a_gid > 0); - assert(b_gid > 0); + CHECK(main_gid > 0); + CHECK(a_gid > 0); + CHECK(b_gid > 0); // Expect: edge main -> fn_a (YES — exact ref_original_id match) // edge main -> fn_b (NO — ref_original_id is authoritative) @@ -186,8 +186,8 @@ static void test_ref_original_id_resolves_exactly() edge_to_b = true; } } - assert(edge_to_a); - assert(!edge_to_b); + CHECK(edge_to_a); + CHECK(!edge_to_b); printf(" \u2713 test_ref_original_id_resolves_exactly: %zu call edges\n", call_g.edges.size()); @@ -224,8 +224,8 @@ static void test_bare_name_fallback_unique_candidate() if (n.name == "compute") compute_gid = n.id; } - assert(main_gid > 0); - assert(compute_gid > 0); + CHECK(main_gid > 0); + CHECK(compute_gid > 0); // Expect: edge main -> compute (name-only fallback works for unique name) bool edge_found = false; @@ -234,7 +234,7 @@ static void test_bare_name_fallback_unique_candidate() e.target_id == compute_gid) edge_found = true; } - assert(edge_found); + CHECK(edge_found); printf(" \u2713 test_bare_name_fallback_unique_candidate: %zu call edges\n", call_g.edges.size()); @@ -265,7 +265,7 @@ static void test_external_index_cross_file_call() if (n.name == "main") main_gid = n.id; } - assert(main_gid > 0); + CHECK(main_gid > 0); // Build an external CalleeIndex manually (simulating a cross-file index: // "helper" is defined in another file with a known graph node ID). @@ -283,7 +283,7 @@ static void test_external_index_cross_file_call() e.target_id == external_helper_gid) found = true; } - assert(found); + CHECK(found); printf(" \u2713 test_external_index_cross_file_call: %zu call edges\n", call_g.edges.size()); @@ -302,5 +302,5 @@ int main() test_external_index_cross_file_call(); printf("\n=== GraphBuilder call-edge precision tests passed ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_graph_query_hints.cpp b/engine/tests/test_graph_query_hints.cpp new file mode 100644 index 0000000..6b2a94f --- /dev/null +++ b/engine/tests/test_graph_query_hints.cpp @@ -0,0 +1,349 @@ +// test_graph_query_hints.cpp +// +// Pins three surfaces that only ever failed silently: +// +// 1. query::bareNameCandidates — the shared homonym probe used by +// trace_flow / codescope_trace. Must return "" for 0/1 matches (normal +// tracing proceeds) and a `{"ambiguous":true,"candidates":[…]}` fragment +// for 2+ (T5 finding #9). +// 2. graph_query's empty-result hint. A typed name that exists under a +// different entity kind must produce exactly ONE "hint" key (two probes +// must not emit duplicate JSON object names), and a node written without +// a type label (`:name`) must not hint at all — there the type filter is +// not why the result is empty (T5 finding #14 follow-up). +// 3. detect_ffi_boundaries.external_symbols — called-but-undefined symbols, +// i.e. one-sided FFI prototypes like `int rust_add(int,int);` that the +// name-prefix list cannot see (T5 finding #7). +// +// The database state is built with explicit SQL rather than by indexing a +// fixture, so each case tests exactly what it claims to. + +#include "../include/engine.h" +#include "../src/query/graph_query.h" +#include "../src/query/query_engine.h" +#include "../src/store/store.h" + +#include "test_check.h" +#include +#include +#include +#include +#include +#include +#include +#include "test_engine_handle.h" + +using namespace store; + +static const char *kDbPath = "/tmp/test_graph_query_hints.db"; + +/// Run a statement, failing loudly: this test builds its own state, so an +/// error here means the fixture is wrong rather than the code under test. +static void execOrDie(sqlite3 *db, const std::string &sql) +{ + char *err = nullptr; + if (sqlite3_exec(db, sql.c_str(), nullptr, nullptr, &err) != + SQLITE_OK) { + fprintf(stderr, "FAIL: sqlite3_exec: %s | sql=%s\n", + err ? err : "(no message)", sql.c_str()); + sqlite3_free(err); + exit(1); + } +} + +static void insertEntity(sqlite3 *db, uint64_t pid, int64_t id, int kind, + const std::string &name, const std::string &file, + int start_row) +{ + execOrDie(db, + "INSERT INTO entity (id, project_id, kind, name, " + "qualified_name, file_path, language, start_row, start_col, " + "end_row, end_col) VALUES (" + + std::to_string(id) + "," + std::to_string(pid) + "," + + std::to_string(kind) + ",'" + name + "','','" + file + + "','c'," + std::to_string(start_row) + ",0," + + std::to_string(start_row + 1) + ",0)"); +} + +static void insertCallRelation(sqlite3 *db, uint64_t pid, int64_t src, + int64_t tgt) +{ + execOrDie( + db, + "INSERT INTO relation (project_id, source_id, target_id, type) " + "VALUES (" + + std::to_string(pid) + "," + std::to_string(src) + "," + + std::to_string(tgt) + ",1)"); +} + +/// A CallExpr semantic record — the shape detect_ffi_boundaries reads to +/// find symbols that are called but have no definition entity. +static void insertCallExpr(sqlite3 *db, uint64_t pid, int64_t id, + const std::string &name, const std::string &file, + int start_row) +{ + execOrDie(db, "INSERT INTO semantic_records (original_id, project_id, " + "kind, name, file_path, language, start_row, end_row) " + "VALUES (" + + std::to_string(id) + "," + std::to_string(pid) + + ",9,'" + name + "','" + file + "','c'," + + std::to_string(start_row) + "," + + std::to_string(start_row) + ")"); +} + +/// Count occurrences of `"key":` in a JSON document — used to assert a key +/// appears exactly once, which a JSON parse alone cannot check (duplicate +/// object names are accepted by lenient parsers and the last one wins). +static int countKey(const std::string &json, const char *key) +{ + std::string needle = std::string("\"") + key + "\":"; + int n = 0; + size_t pos = 0; + while ((pos = json.find(needle, pos)) != std::string::npos) { + ++n; + pos += needle.size(); + } + return n; +} + +static bool contains(const std::string &hay, const char *needle) +{ + return hay.find(needle) != std::string::npos; +} + +int main() +{ + unlink(kDbPath); + unlink((std::string(kDbPath) + "-wal").c_str()); + unlink((std::string(kDbPath) + "-shm").c_str()); + + GraphStore store; + if (!store.open(kDbPath)) { + fprintf(stderr, "FAIL: cannot open store: %s\n", + store.error().c_str()); + return 1; + } + const uint64_t pid = store.createProject("/tmp", "graph-query-hints"); + CHECK(pid > 0); + sqlite3 *db = store.handle(); + CHECK(db != nullptr); + + // Fixture: + // 1 Widget kind=2 (class) src/widget.h + // 2 Widget kind=0 (constructor) src/widget.c + // 3 gadget kind=0 src/widget.c + // 4 other kind=0 src/other.c + // 1 -> 4 and 3 -> 4 are real Calls edges. + insertEntity(db, pid, 1, 2, "Widget", "src/widget.h", 10); + insertEntity(db, pid, 2, 0, "Widget", "src/widget.c", 20); + insertEntity(db, pid, 3, 0, "gadget", "src/widget.c", 30); + insertEntity(db, pid, 4, 0, "other", "src/other.c", 5); + insertCallRelation(db, pid, 1, 4); + insertCallRelation(db, pid, 3, 4); + // One-sided FFI: `extern_add` is called but never defined here. + insertCallExpr(db, pid, 100, "extern_add", "src/widget.c", 40); + + // ── Case 1: bareNameCandidates probe ──────────────────────── + { + std::string none = + query::bareNameCandidates(db, pid, "nonexistent"); + std::string one = query::bareNameCandidates(db, pid, "gadget"); + CHECK(none.empty() && + "0 matches must probe as unambiguous (caller reports " + "not-found)"); + CHECK(one.empty() && + "1 match must probe as unambiguous (caller traces " + "it)"); + printf("Test 1 (0/1 match -> empty probe): PASS\n"); + } + + // ── Case 2: 2 matches -> ambiguous + candidates ───────────── + { + std::string amb = query::bareNameCandidates(db, pid, "Widget"); + CHECK(contains(amb, "\"ambiguous\":true")); + CHECK(countKey(amb, "graph_node_id") == 2); + CHECK(contains(amb, "widget.h") && contains(amb, "widget.c")); + printf("Test 2 (2 matches -> ambiguous, 2 candidates): PASS\n"); + } + + // ── Case 3: typed mismatch -> exactly one hint ────────────── + // `gadget` exists only as kind 0, so `Class:gadget` resolves to + // nothing — the type filter is why the result is empty and the hint + // must fire once and name the kind it did find. + { + std::string q = query::executeGraphQuery( + pid, + "MATCH (Class:gadget)-[Calls]->(Function:other) " + "RETURN Class.name", + &store); + CHECK(q.find("\"total\":0") != std::string::npos); + CHECK(countKey(q, "hint") == 1 && + "a kind mismatch must emit exactly one hint key"); + CHECK(contains(q, "kind 0")); + printf("Test 3 (typed mismatch -> single hint): PASS\n"); + } + + // ── Case 4: both sides mismatch -> still one hint key ─────── + { + std::string q = query::executeGraphQuery( + pid, + "MATCH (Class:gadget)-[Calls]->(Class:other) " + "RETURN Class.name", + &store); + CHECK(q.find("\"total\":0") != std::string::npos); + CHECK(countKey(q, "hint") == 1 && + "two probes must merge into one hint key, never two"); + printf("Test 4 (both-sides mismatch -> one hint key): PASS\n"); + } + + // ── Case 5: untyped node (:name) -> no hint at all ────────── + // `:gadget` parses to name with an EMPTY type, so the type filter + // is not why the result is empty — hinting would be misleading. + { + std::string q = query::executeGraphQuery( + pid, + "MATCH (:gadget)-[Calls]->(:nosuch) RETURN gadget.name", + &store); + CHECK(q.find("\"total\":0") != std::string::npos); + CHECK(countKey(q, "hint") == 0 && + "an untyped node must not produce a kind-mismatch " + "hint"); + printf("Test 5 (untyped node -> no hint): PASS\n"); + } + + // ── Case 6: a query that matches -> no hint ───────────────── + { + std::string q = query::executeGraphQuery( + pid, + "MATCH (Function:gadget)-[Calls]->(Function:other) " + "RETURN Function.name", + &store); + CHECK(q.find("\"total\":1") != std::string::npos); + CHECK(countKey(q, "hint") == 0 && + "a matching query must not carry a hint"); + printf("Test 6 (matching query -> no hint): PASS\n"); + } + + // ── Test 8: LIMIT is honoured, unknown trailing text is rejected ── + // Regression: the DSL documented `LIMIT n` (the README's graph-query + // benchmarks instruct it on large graphs) and the parser ignored it — plus + // every other trailing token — silently, so `LIMIT 10` returned the whole + // result set and the MCP transport then replaced the oversized response + // with an error the caller could not act on. + { + insertEntity(db, pid, 9101, 0, "lq_caller", "/t/lq.c", 10); + for (int i = 0; i < 5; ++i) + insertEntity(db, pid, 9110 + i, 0, + "lq_callee" + std::to_string(i), "/t/lq.c", + 20 + i); + for (int i = 0; i < 5; ++i) + insertCallRelation(db, pid, 9101, 9110 + i); + + const std::string base = + "MATCH (Function:lq_caller)-[Calls]->(Function)"; + + const std::string all = + query::executeGraphQuery(pid, base.c_str(), &store); + CHECK(countKey(all, "source") == 5); + CHECK(!contains(all, "\"truncated\"")); + + const std::string two = query::executeGraphQuery( + pid, (base + " LIMIT 2").c_str(), &store); + CHECK(countKey(two, "source") == 2); + CHECK(contains(two, "\"total\":2")); + CHECK(contains(two, "\"truncated\":true")); + + // Keywords are case-insensitive, like MATCH. + const std::string three = query::executeGraphQuery( + pid, (base + " limit 3").c_str(), &store); + CHECK(countKey(three, "source") == 3); + CHECK(contains(three, "\"truncated\":true")); + + // A limit above the match count is not a truncation. + const std::string big = query::executeGraphQuery( + pid, (base + " LIMIT 99").c_str(), &store); + CHECK(countKey(big, "source") == 5); + CHECK(!contains(big, "\"truncated\"")); + + // `RETURN` stays accepted (the DSL's documented surface, used by the + // tests above) and may follow LIMIT. + const std::string ret = query::executeGraphQuery( + pid, (base + " LIMIT 2 RETURN Function.name").c_str(), + &store); + CHECK(countKey(ret, "source") == 2); + CHECK(contains(ret, "\"truncated\":true")); + + // A malformed clause — and any other trailing text — is an error, + // never a silent no-op. + for (const char *bad : { " LIMIT 0", " LIMIT abc", " LIMIT", + " GARBAGE", " LIMIT 1 LIMIT 2" }) { + const std::string r = query::executeGraphQuery( + pid, (base + bad).c_str(), &store); + CHECK(contains(r, "\"error\"")); + CHECK(countKey(r, "source") == 0); + } + printf("Test 8 (LIMIT honoured, unknown trailing text rejected): " + "PASS\n"); + } + + // ── Case 7: one-sided FFI appears as external_symbols ─────── + // engine_detect_ffi_boundaries runs against the GLOBAL FFI store, + // not a local GraphStore, so this case drives the real + // engine_init / engine_create_project lifecycle and seeds its DB + // through a second sqlite3 connection. + { + store.close(); + unlink(kDbPath); + + g_engine = engine_create(kDbPath); + if (!g_engine) { + fprintf(stderr, "FAIL: engine_init\n"); + return 1; + } + const uint64_t fpid = engine_create_project(g_engine, "/tmp", + "ffi-one-sided"); + CHECK(fpid > 0); + + sqlite3 *fdb = nullptr; + if (sqlite3_open(kDbPath, &fdb) != SQLITE_OK) { + fprintf(stderr, "FAIL: sqlite3_open for fixture: %s\n", + sqlite3_errmsg(fdb)); + engine_destroy(g_engine); + g_engine = nullptr; + return 1; + } + insertEntity(fdb, fpid, 1, 0, "run_c_side", "src/main.c", 1); + insertCallExpr(fdb, fpid, 100, "extern_add", "src/main.c", 40); + insertCallExpr(fdb, fpid, 101, "run_c_side", "src/main.c", 10); + sqlite3_close(fdb); + + char *raw = engine_detect_ffi_boundaries(g_engine, fpid); + CHECK(raw != nullptr); + std::string s(raw); + engine_free_string(raw); + engine_destroy(g_engine); + g_engine = nullptr; + + CHECK(contains(s, "\"external_symbols\":[")); + CHECK(contains(s, "extern_add")); + size_t ext = s.find("\"external_symbols\":"); + size_t orphan = s.find("\"orphan_symbols\":"); + CHECK(ext != std::string::npos && orphan != std::string::npos && + ext < orphan); + std::string ext_block = s.substr(ext, orphan - ext); + // `run_c_side` is defined here (entity kind=0), so the CallExpr + // to it must NOT be reported as external. + CHECK(contains(ext_block, "extern_add")); + CHECK(!contains(ext_block, "run_c_side")); + printf("Test 7 (one-sided FFI -> external_symbols): PASS\n"); + } + + unlink(kDbPath); + unlink((std::string(kDbPath) + "-wal").c_str()); + unlink((std::string(kDbPath) + "-shm").c_str()); + + printf("\n=== test_graph_query_hints PASSED ===\n"); + printf("Homonym probe, single-hint contract, untyped-node quiet, " + "external_symbols, LIMIT\n"); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_graph_semantic.cpp b/engine/tests/test_graph_semantic.cpp index 0c94261..a8cee61 100644 --- a/engine/tests/test_graph_semantic.cpp +++ b/engine/tests/test_graph_semantic.cpp @@ -2,7 +2,7 @@ #include "../src/ir/semantic_unit.h" #include "../src/ir/semantic_emitter.h" #include "../src/graph/graph_builder.h" -#include +#include "test_check.h" #include #include @@ -10,189 +10,254 @@ static void test_symbol_graph_from_semantic_unit() { - using namespace ir; - using namespace graph; - - SemanticUnit unit; - SemanticEmitter emitter(&unit); - unit.setFilePath("/test/app.js"); - unit.setLanguage("javascript"); - - SourceRange loc{0, 0, 10, 0}; - (void)loc; - - // Create: compute() at top level - uint64_t fn1 = emitter.emitFunction("compute", {1, 0, 3, 0}); - (void)fn1; - // Create: main() at top level - uint64_t fn2 = emitter.emitFunction("main", {5, 0, 9, 0}); - // Variable inside main - emitter.emitVariable("x", {6, 4, 6, 10}, fn2); - // var y inside main - emitter.emitVariable("y", {7, 4, 7, 10}, fn2); - - // Build symbol graph - GraphBuilder builder(1); - auto sym_g = builder.buildSymbolGraph(unit); - - // Verify nodes - bool has_compute = false, has_main = false, has_x = false, has_y = false; - for (auto &n : sym_g.nodes) { - if (n.name == "compute") { has_compute = true; assert(n.type == NodeType::Function); } - if (n.name == "main") { has_main = true; assert(n.type == NodeType::Function); } - if (n.name == "x") { has_x = true; assert(n.type == NodeType::Variable); } - if (n.name == "y") { has_y = true; assert(n.type == NodeType::Variable); } - } - assert(has_compute); - assert(has_main); - assert(has_x); - assert(has_y); - - // Verify Contains edges: main → x, main → y - uint64_t main_gid = 0, x_gid = 0; - for (auto &n : sym_g.nodes) { - if (n.name == "main") main_gid = n.id; - if (n.name == "x") x_gid = n.id; - } - bool contains_x = false; - for (auto &e : sym_g.edges) { - if (e.type == EdgeType::Contains && e.source_id == main_gid && e.target_id == x_gid) - contains_x = true; - } - assert(contains_x); - - printf(" ✓ test_symbol_graph_from_semantic_unit: %zu nodes, %zu edges\n", - sym_g.nodes.size(), sym_g.edges.size()); + using namespace ir; + using namespace graph; + + SemanticUnit unit; + SemanticEmitter emitter(&unit); + unit.setFilePath("/test/app.js"); + unit.setLanguage("javascript"); + + SourceRange loc{ 0, 0, 10, 0 }; + (void)loc; + + // Create: compute() at top level + uint64_t fn1 = emitter.emitFunction("compute", { 1, 0, 3, 0 }); + (void)fn1; + // Create: main() at top level + uint64_t fn2 = emitter.emitFunction("main", { 5, 0, 9, 0 }); + // Variable inside main + emitter.emitVariable("x", { 6, 4, 6, 10 }, fn2); + // var y inside main + emitter.emitVariable("y", { 7, 4, 7, 10 }, fn2); + + // Build symbol graph + GraphBuilder builder(1); + auto sym_g = builder.buildSymbolGraph(unit); + + // Verify nodes + bool has_compute = false, has_main = false, has_x = false, + has_y = false; + for (auto &n : sym_g.nodes) { + if (n.name == "compute") { + has_compute = true; + CHECK(n.type == NodeType::Function); + } + if (n.name == "main") { + has_main = true; + CHECK(n.type == NodeType::Function); + } + if (n.name == "x") { + has_x = true; + CHECK(n.type == NodeType::Variable); + } + if (n.name == "y") { + has_y = true; + CHECK(n.type == NodeType::Variable); + } + } + CHECK(has_compute); + CHECK(has_main); + CHECK(has_x); + CHECK(has_y); + + // Verify Contains edges: main → x, main → y + uint64_t main_gid = 0, x_gid = 0; + for (auto &n : sym_g.nodes) { + if (n.name == "main") + main_gid = n.id; + if (n.name == "x") + x_gid = n.id; + } + bool contains_x = false; + for (auto &e : sym_g.edges) { + if (e.type == EdgeType::Contains && e.source_id == main_gid && + e.target_id == x_gid) + contains_x = true; + } + CHECK(contains_x); + + printf(" ✓ test_symbol_graph_from_semantic_unit: %zu nodes, %zu edges\n", + sym_g.nodes.size(), sym_g.edges.size()); +} + +// ── Test 1b: the anonymous root container is not a node ──────── +// Every visitor emits an empty-named Variable at parent_id 0 so each record has +// a parent. It is not a symbol, and the bulk (entity) path never stores it — the +// entity table holds no nameless row — so a single-file graph that keeps it has +// a nameless ghost node the same file's bulk graph does not. The old guard +// (`recordKindToNodeType(kind) == NodeType::File`) could never hold for a +// Variable record, so the container was always kept. +static void test_anonymous_root_is_not_a_node() +{ + using namespace ir; + using namespace graph; + SemanticUnit unit; + SemanticEmitter emitter(&unit); + emitter.emitVariable("", { 0, 0, 0, 0 }, 0); // anonymous root container + emitter.emitFunction("solo", { 1, 0, 2, 0 }, 0); + + GraphBuilder builder(1); + auto g = builder.buildSymbolGraph(unit); + + size_t nameless = 0; + bool has_solo = false; + for (auto &n : g.nodes) { + if (n.name.empty()) + ++nameless; + if (n.name == "solo") + has_solo = true; + } + CHECK(nameless == 0); + CHECK(has_solo); + CHECK(g.nodes.size() == 1); + printf(" ✓ test_anonymous_root_is_not_a_node: %zu nodes\n", + g.nodes.size()); } // ── Test 2: CallGraph from SemanticUnit ─────────────────────── static void test_call_graph_from_semantic_unit() { - using namespace ir; - using namespace graph; - - SemanticUnit unit; - SemanticEmitter emitter(&unit); - unit.setFilePath("/test/app.js"); - unit.setLanguage("javascript"); - - // Define compute function - uint64_t fn1 = emitter.emitFunction("compute", {1, 0, 3, 0}); - (void)fn1; - // Define main function - uint64_t fn2 = emitter.emitFunction("main", {5, 0, 9, 0}); - // main calls compute - emitter.emitCall("compute", {6, 4, 6, 15}, fn2); - - // Build symbol graph first (call graph reuses its node mapping) - GraphBuilder builder(1); - auto sym_g = builder.buildSymbolGraph(unit); - auto call_g = builder.buildCallGraph(unit); - - // Verify call edge: main → compute - uint64_t main_gid = 0, compute_gid = 0; - for (auto &n : sym_g.nodes) { - if (n.name == "main") main_gid = n.id; - if (n.name == "compute") compute_gid = n.id; - } - assert(main_gid > 0); - assert(compute_gid > 0); - - bool call_found = false; - for (auto &e : call_g.edges) { - if (e.type == EdgeType::Calls && - e.source_id == main_gid && - e.target_id == compute_gid) - call_found = true; - } - assert(call_found); - - printf(" ✓ test_call_graph_from_semantic_unit: %zu nodes, %zu edges\n", - call_g.nodes.size(), call_g.edges.size()); + using namespace ir; + using namespace graph; + + SemanticUnit unit; + SemanticEmitter emitter(&unit); + unit.setFilePath("/test/app.js"); + unit.setLanguage("javascript"); + + // Define compute function + uint64_t fn1 = emitter.emitFunction("compute", { 1, 0, 3, 0 }); + (void)fn1; + // Define main function + uint64_t fn2 = emitter.emitFunction("main", { 5, 0, 9, 0 }); + // main calls compute + emitter.emitCall("compute", { 6, 4, 6, 15 }, fn2); + + // Build symbol graph first (call graph reuses its node mapping) + GraphBuilder builder(1); + auto sym_g = builder.buildSymbolGraph(unit); + auto call_g = builder.buildCallGraph(unit); + + // Verify call edge: main → compute + uint64_t main_gid = 0, compute_gid = 0; + for (auto &n : sym_g.nodes) { + if (n.name == "main") + main_gid = n.id; + if (n.name == "compute") + compute_gid = n.id; + } + CHECK(main_gid > 0); + CHECK(compute_gid > 0); + + bool call_found = false; + for (auto &e : call_g.edges) { + if (e.type == EdgeType::Calls && e.source_id == main_gid && + e.target_id == compute_gid) + call_found = true; + } + CHECK(call_found); + + printf(" ✓ test_call_graph_from_semantic_unit: %zu nodes, %zu edges\n", + call_g.nodes.size(), call_g.edges.size()); } // ── Test 3: Class + Method hierarchy from SemanticUnit ──────── static void test_class_method_hierarchy() { - using namespace ir; - using namespace graph; - - SemanticUnit unit; - SemanticEmitter emitter(&unit); - unit.setFilePath("/test/app.js"); - unit.setLanguage("javascript"); - - uint64_t cls = emitter.emitClass("Calculator", {0, 0, 10, 0}); - uint64_t m1 = emitter.emitMethod("add", {1, 4, 4, 8}, cls); - uint64_t m2 = emitter.emitMethod("sub", {5, 4, 8, 8}, cls); - (void)m2; - emitter.emitVariable("tmp", {2, 8, 2, 12}, m1); - - GraphBuilder builder(1); - auto sym_g = builder.buildSymbolGraph(unit); - - bool has_calc = false, has_add = false, has_sub = false; - for (auto &n : sym_g.nodes) { - if (n.name == "Calculator") { has_calc = true; assert(n.type == NodeType::Class); } - if (n.name == "add") { has_add = true; assert(n.type == NodeType::Method); } - if (n.name == "sub") { has_sub = true; assert(n.type == NodeType::Method); } - } - assert(has_calc); - assert(has_add); - assert(has_sub); - - // Verify Contains: Calculator → add, Calculator → sub - uint64_t calc_gid = 0, add_gid = 0; - for (auto &n : sym_g.nodes) { - if (n.name == "Calculator") calc_gid = n.id; - if (n.name == "add") add_gid = n.id; - } - bool contains_add = false; - for (auto &e : sym_g.edges) { - if (e.type == EdgeType::Contains && e.source_id == calc_gid && e.target_id == add_gid) - contains_add = true; - } - assert(contains_add); - - printf(" ✓ test_class_method_hierarchy: %zu nodes, %zu edges\n", - sym_g.nodes.size(), sym_g.edges.size()); + using namespace ir; + using namespace graph; + + SemanticUnit unit; + SemanticEmitter emitter(&unit); + unit.setFilePath("/test/app.js"); + unit.setLanguage("javascript"); + + uint64_t cls = emitter.emitClass("Calculator", { 0, 0, 10, 0 }); + uint64_t m1 = emitter.emitMethod("add", { 1, 4, 4, 8 }, cls); + uint64_t m2 = emitter.emitMethod("sub", { 5, 4, 8, 8 }, cls); + (void)m2; + emitter.emitVariable("tmp", { 2, 8, 2, 12 }, m1); + + GraphBuilder builder(1); + auto sym_g = builder.buildSymbolGraph(unit); + + bool has_calc = false, has_add = false, has_sub = false; + for (auto &n : sym_g.nodes) { + if (n.name == "Calculator") { + has_calc = true; + CHECK(n.type == NodeType::Class); + } + if (n.name == "add") { + has_add = true; + CHECK(n.type == NodeType::Method); + } + if (n.name == "sub") { + has_sub = true; + CHECK(n.type == NodeType::Method); + } + } + CHECK(has_calc); + CHECK(has_add); + CHECK(has_sub); + + // Verify Contains: Calculator → add, Calculator → sub + uint64_t calc_gid = 0, add_gid = 0; + for (auto &n : sym_g.nodes) { + if (n.name == "Calculator") + calc_gid = n.id; + if (n.name == "add") + add_gid = n.id; + } + bool contains_add = false; + for (auto &e : sym_g.edges) { + if (e.type == EdgeType::Contains && e.source_id == calc_gid && + e.target_id == add_gid) + contains_add = true; + } + CHECK(contains_add); + + printf(" ✓ test_class_method_hierarchy: %zu nodes, %zu edges\n", + sym_g.nodes.size(), sym_g.edges.size()); } // ── Test 4: Empty SemanticUnit produces empty graphs ───────── static void test_empty_semantic_unit() { - using namespace ir; - using namespace graph; + using namespace ir; + using namespace graph; - SemanticUnit unit; - unit.setFilePath("/test/empty.js"); - unit.setLanguage("javascript"); + SemanticUnit unit; + unit.setFilePath("/test/empty.js"); + unit.setLanguage("javascript"); - GraphBuilder builder(1); - auto sym_g = builder.buildSymbolGraph(unit); - auto call_g = builder.buildCallGraph(unit); + GraphBuilder builder(1); + auto sym_g = builder.buildSymbolGraph(unit); + auto call_g = builder.buildCallGraph(unit); - assert(sym_g.nodes.empty()); - assert(sym_g.edges.empty()); - assert(call_g.nodes.empty()); // no nodes from call graph alone - assert(call_g.edges.empty()); + CHECK(sym_g.nodes.empty()); + CHECK(sym_g.edges.empty()); + CHECK(call_g.nodes.empty()); // no nodes from call graph alone + CHECK(call_g.edges.empty()); - printf(" ✓ test_empty_semantic_unit\n"); + printf(" ✓ test_empty_semantic_unit\n"); } // ── Main ────────────────────────────────────────────────────── -int main() { - printf("GraphBuilder SemanticUnit tests:\n"); +int main() +{ + printf("GraphBuilder SemanticUnit tests:\n"); - test_symbol_graph_from_semantic_unit(); - test_call_graph_from_semantic_unit(); - test_class_method_hierarchy(); - test_empty_semantic_unit(); + test_symbol_graph_from_semantic_unit(); + test_anonymous_root_is_not_a_node(); + test_call_graph_from_semantic_unit(); + test_class_method_hierarchy(); + test_empty_semantic_unit(); - printf("\n=== GraphBuilder SemanticUnit tests passed ===\n"); - return 0; + printf("\n=== GraphBuilder SemanticUnit tests passed ===\n"); + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_homonym_filter.cpp b/engine/tests/test_homonym_filter.cpp index de6ad14..4ea07fe 100644 --- a/engine/tests/test_homonym_filter.cpp +++ b/engine/tests/test_homonym_filter.cpp @@ -29,6 +29,7 @@ #include #include #include +#include "test_engine_handle.h" /// Count occurrences of a substring in a JSON string. static int countOccurrences(const char *json, const char *needle) @@ -102,24 +103,28 @@ int main() const char *db_path = "/tmp/test_homonym_filter.db"; unlink(db_path); - if (engine_init(db_path) != 0) { + g_engine = engine_create(db_path); + if (!g_engine) { fprintf(stderr, "FAIL: engine_init\n"); return 1; } - uint64_t pid = engine_create_project(proj_dir, "homonym-test"); + uint64_t pid = + engine_create_project(g_engine, proj_dir, "homonym-test"); if (pid == 0) { fprintf(stderr, "FAIL: engine_create_project\n"); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; return 1; } - char *idx = engine_index_project(pid, proj_dir, nullptr); + char *idx = engine_index_project(g_engine, pid, proj_dir, nullptr); if (!idx || !strstr(idx, "\"ok\":true")) { fprintf(stderr, "FAIL: engine_index_project: %s\n", idx ? idx : "(null)"); engine_free_string(idx); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; return 1; } engine_free_string(idx); @@ -134,11 +139,14 @@ int main() // Step 7 semantics: two entities named "handler" exist, so the bare // name is ambiguous. The API must return ambiguous=true with a // candidate list instead of silently merging both callees. - char *callees_no_filter = engine_get_callees(pid, "handler", nullptr); + char *callees_no_filter = + engine_get_callees(g_engine, pid, "handler", nullptr); bool no_filter_ambiguous = - callees_no_filter && strstr(callees_no_filter, "\"ambiguous\":true"); - int candidates_count = countOccurrences( - callees_no_filter ? callees_no_filter : "", "\"graph_node_id\""); + callees_no_filter && + strstr(callees_no_filter, "\"ambiguous\":true"); + int candidates_count = + countOccurrences(callees_no_filter ? callees_no_filter : "", + "\"graph_node_id\""); int helperOne_no_filter = countOccurrences(callees_no_filter, "helperOne"); int helperTwo_no_filter = @@ -150,8 +158,8 @@ int main() // ── Test 2: callees("handler") WITH file_filter=first.go ───── // With filter, only first.go's handler is queried. Only helperOne // should appear; helperTwo must NOT appear. - char *callees_first = - engine_get_callees(pid, "handler", first_file.c_str()); + char *callees_first = engine_get_callees(g_engine, pid, "handler", + first_file.c_str()); int total_first = countTotal(callees_first); int helperOne_first = countOccurrences(callees_first, "helperOne"); int helperTwo_first = countOccurrences(callees_first, "helperTwo"); @@ -160,8 +168,8 @@ int main() engine_free_string(callees_first); // ── Test 3: callees("handler") WITH file_filter=second.go ──── - char *callees_second = - engine_get_callees(pid, "handler", second_file.c_str()); + char *callees_second = engine_get_callees(g_engine, pid, "handler", + second_file.c_str()); int total_second = countTotal(callees_second); int helperOne_second = countOccurrences(callees_second, "helperOne"); int helperTwo_second = countOccurrences(callees_second, "helperTwo"); @@ -234,12 +242,14 @@ int main() if (pass) { printf("\nPASS: homonym disambiguation works " "(ambiguous=%s candidates=%d first=%d second=%d)\n", - no_filter_ambiguous ? "true" : "false", - candidates_count, total_first, total_second); - engine_shutdown(); + no_filter_ambiguous ? "true" : "false", candidates_count, + total_first, total_second); + engine_destroy(g_engine); + g_engine = nullptr; return 0; } - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; return 1; } diff --git a/engine/tests/test_import_alias_resolution.cpp b/engine/tests/test_import_alias_resolution.cpp new file mode 100644 index 0000000..8018adb --- /dev/null +++ b/engine/tests/test_import_alias_resolution.cpp @@ -0,0 +1,167 @@ +// test_import_alias_resolution — `import { x as y }` must resolve to x. +// +// Defect guarded: a TypeScript/JavaScript aliased import binds a LOCAL name to +// an imported symbol, and only the local name appears at the call site. The +// visitor recorded the module binding (`emitImportBinding(y, "../lib/helper")`) +// but not which symbol `y` stands for, so a bare `y()` named no declaration at +// all and produced NO edge — a silent false negative for a shape every +// TypeScript codebase uses (`import { useState as useLocalState }`). +// +// The call is now emitted under the imported symbol, so it resolves like any +// other call to that symbol. The fixture puts the same symbol name in two +// modules on purpose: the call must reach the module it was imported FROM, which +// makes this the case the ImportModuleMatch factor was written for, and the +// assertion fails if the resolver guesses the other one. +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include + +#include "test_check.h" +#include "test_engine_handle.h" + +namespace +{ + +/// Write `content` to `path`, failing the test if the file cannot be created. +void writeFile(const std::string &path, const char *content) +{ + FILE *f = fopen(path.c_str(), "w"); + CHECK(f != nullptr); + fputs(content, f); + fclose(f); +} + +/// One scalar read; -1 when the query yields no row. +long long scalarSql(sqlite3 *db, const char *sql) +{ + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + long long value = -1; + if (sqlite3_step(st) == SQLITE_ROW) + value = sqlite3_column_int64(st, 0); + sqlite3_finalize(st); + return value; +} + +/// The file of the single callee `caller` reaches, or "(no edge)". +std::string calleeFile(sqlite3 *db, const char *caller) +{ + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2( + db, + "SELECT te.file_path FROM relation r " + "JOIN entity se ON se.id = r.source_id " + "JOIN entity te ON te.id = r.target_id " + "WHERE r.type = 1 AND se.name = ?", + -1, &st, nullptr) == SQLITE_OK); + sqlite3_bind_text(st, 1, caller, -1, SQLITE_TRANSIENT); + std::string out = "(no edge)"; + if (sqlite3_step(st) == SQLITE_ROW) { + const char *s = reinterpret_cast( + sqlite3_column_text(st, 0)); + if (s) + out = s; + } + sqlite3_finalize(st); + return out; +} + +/// The callee name recorded for `caller`'s call site, or "(none)". +std::string referenceName(sqlite3 *db, const char *caller) +{ + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2( + db, + "SELECT rf.name FROM reference rf JOIN entity e " + "ON e.id = rf.caller_id WHERE e.name = ? LIMIT 1", + -1, &st, nullptr) == SQLITE_OK); + sqlite3_bind_text(st, 1, caller, -1, SQLITE_TRANSIENT); + std::string out = "(none)"; + if (sqlite3_step(st) == SQLITE_ROW) { + const char *s = reinterpret_cast( + sqlite3_column_text(st, 0)); + if (s) + out = s; + } + sqlite3_finalize(st); + return out; +} + +} // namespace + +int main() +{ + const std::string proj_dir = "/tmp/import_alias_repro"; + std::filesystem::remove_all(proj_dir); + std::filesystem::create_directories(proj_dir + "/src/lib"); + std::filesystem::create_directories(proj_dir + "/src/util"); + + // The same exported symbol in two modules: the import decides which one the + // call means, so a resolver that ignores the import has a 50% chance of + // being wrong rather than merely unresolved. + writeFile(proj_dir + "/src/lib/helper.ts", + "export function pick(): number { return 1; }\n"); + writeFile(proj_dir + "/src/util/helper.ts", + "export function pick(): number { return 2; }\n"); + writeFile(proj_dir + "/src/use.ts", + "import { pick as pickLib } from '../lib/helper';\n" + "\n" + "export function use(): number { return pickLib(); }\n"); + + const std::string db_path = proj_dir + "/alias.db"; + unlink(db_path.c_str()); + unlink((db_path + "-wal").c_str()); + unlink((db_path + "-shm").c_str()); + + g_engine = engine_create(db_path.c_str()); + CHECK(g_engine != nullptr); + const uint64_t pid = engine_create_project(g_engine, proj_dir.c_str(), + "import-alias"); + CHECK(pid > 0); + const std::string request = "{\"paths\":[\"" + proj_dir + + "/src/lib/helper.ts\",\"" + proj_dir + + "/src/util/helper.ts\",\"" + proj_dir + + "/src/use.ts\"]}"; + char *res = engine_index_files(g_engine, pid, request.c_str(), 1); + CHECK(res != nullptr); + CHECK(strstr(res, "\"ok\":true") != nullptr); + engine_free_string(res); + // engine_destroy() joins the async knowledge builder before closing the + // store, and every row asserted below is written synchronously by the index + // path, so reading the database afterwards needs no sleep. + engine_destroy(g_engine); + g_engine = nullptr; + + sqlite3 *db = nullptr; + CHECK(sqlite3_open(db_path.c_str(), &db) == SQLITE_OK); + + // The fixture must really have parsed, and the ambiguity must exist: two + // same-named functions in different files. + CHECK(scalarSql(db, "SELECT COUNT(*) FROM entity WHERE name='pick'") == 2); + CHECK(scalarSql(db, "SELECT COUNT(*) FROM entity WHERE name='use'") == 1); + + // The call site is recorded as the IMPORTED symbol, not the local alias: the + // alias names nothing that can be declared, which is why the edge was lost. + CHECK(referenceName(db, "use") == "pick"); + + // One edge, to the module the symbol was imported from. + CHECK(calleeFile(db, "use") == proj_dir + "/src/lib/helper.ts"); + + sqlite3_close(db); + unlink(db_path.c_str()); + unlink((db_path + "-wal").c_str()); + unlink((db_path + "-shm").c_str()); + if (checkFailures()) { + fprintf(stderr, "=== import alias resolution: FAILED ===\n"); + return 1; + } + std::filesystem::remove_all(proj_dir); + printf("=== import alias resolution test passed ===\n"); + return 0; +} diff --git a/engine/tests/test_index_determinism.cpp b/engine/tests/test_index_determinism.cpp new file mode 100644 index 0000000..ffae86e --- /dev/null +++ b/engine/tests/test_index_determinism.cpp @@ -0,0 +1,381 @@ +// test_index_determinism.cpp — regression test for reproducible indexing. +// +// Defect guarded: buildGraph assigned entity ids with +// `ROW_NUMBER() OVER ()` over semantic_records, i.e. in the table's scan +// order — which is the row INSERT order and therefore depends on how the +// parse workers happened to finish. The same source tree indexed twice could +// produce different entity ids, and every downstream id (relation, +// type_ref, import, CSR adjacency) shifted with it. That broke +// reproducibility: snapshot comparisons, id-keyed regressions and any +// "index twice and diff" check were not stable. +// +// The test indexes one fixture into two independent databases and compares +// the full dumps: +// +// * entities — (file_path, kind, name, start_row, start_col) -> id +// * relations — (source tuple) -> (target tuple), type +// +// Both dumps must be byte-identical, which requires the id assignment to be +// a pure function of the input. +// +// To make the run reliably exercise the defect, the two runs use different +// parse-worker counts (1 and 8). The worker count decides the order in which +// settled files reach SQLite, i.e. the row order of semantic_records that +// `ROW_NUMBER() OVER ()` used to follow. Correct behaviour is that the +// worker count changes the speed but never the result. +// +// Uses the explicit check() pattern (the engine build defines NDEBUG). +// +// Exit code: 0 on success, 1 on any failed assertion. + +#include "../include/engine.h" +#include "../src/ir/semantic_unit.h" +#include "../src/store/store.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "test_engine_handle.h" + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "FAIL: %s\n", msg); + exit(1); + } +} + +/// Number of fixture translation units. Several files are required: the +/// row order in semantic_records depends on which parse worker finishes +/// first, so one file would not exercise the id-assignment order. +static const int kFixtureFiles = 6; + +/// Functions emitted per fixture file (helper, compute, entry, extra). +static const int kFuncsPerFile = 4; + +/// Lower bound on the entities/relations the fixture must produce, so an +/// empty (or failed) index cannot make the comparison pass vacuously. +static const long long kMinEntities = kFixtureFiles * kFuncsPerFile; +static const long long kMinRelations = 10; + +static void writeFile(const std::string &path, const std::string &content) +{ + std::filesystem::create_directories( + std::filesystem::path(path).parent_path()); + FILE *f = fopen(path.c_str(), "w"); + check(f != nullptr, "fopen fixture"); + fputs(content.c_str(), f); + fclose(f); +} + +/// Create the fixture tree: kFixtureFiles directories, each with one +/// translation unit declaring kFuncsPerFile functions that call each other. +static void buildFixture(const std::string &root) +{ + std::filesystem::remove_all(root); + for (int f = 0; f < kFixtureFiles; ++f) { + const std::string idx = std::to_string(f); + std::string body; + body += "namespace ns" + idx + " {\n"; + body += "int helper" + idx + "(int x) { return x + " + idx + + "; }\n"; + body += "int compute" + idx + "(int x) { return helper" + idx + + "(x) * 2; }\n"; + body += "int entry" + idx + "(int x) { return compute" + idx + + "(x); }\n"; + body += "int extra" + idx + "(int x) { return compute" + idx + + "(x) + 1; }\n"; + body += "} // namespace ns" + idx + "\n"; + writeFile(root + "/pkg" + idx + "/unit" + idx + ".cpp", body); + } +} + +/// Index `dir` into a fresh database using `workers` parse workers, then +/// shut the engine down so the on-disk state is final before inspection. +/// +/// The worker count is deliberately part of the test: it determines the +/// order in which settled files reach SQLite. Entity ids must not depend +/// on that order. +static uint64_t indexInto(const std::string &dir, const char *db_path, + int workers) +{ + // setenv() happens before any engine thread exists (the previous run + // was fully shut down), so it is not a concurrent write to environ. + std::string w = std::to_string(workers); + setenv("CODESCOPE_WORKERS", w.c_str(), 1); + + unlink(db_path); + g_engine = engine_create(db_path); + check(g_engine != nullptr, "engine_init"); + uint64_t pid = + engine_create_project(g_engine, dir.c_str(), "determinism"); + check(pid > 0, "create_project"); + char *idx = engine_index_project(g_engine, pid, dir.c_str(), nullptr); + check(idx != nullptr, "index_project null"); + check(strstr(idx, "\"ok\":true") != nullptr, "index_project ok"); + engine_free_string(idx); + engine_destroy(g_engine); + g_engine = nullptr; + + unsetenv("CODESCOPE_WORKERS"); + return pid; +} + +/// Read one TEXT column, mapping NULL to the empty string. +static std::string col(sqlite3_stmt *st, int i) +{ + const char *t = + reinterpret_cast(sqlite3_column_text(st, i)); + return t ? t : ""; +} + +/// Dump every entity as "file|kind|name|row|col|id", ordered by the +/// semantic key so only the id can make the two dumps differ. +static std::vector dumpEntities(const char *db_path, + uint64_t project_id) +{ + sqlite3 *db = nullptr; + check(sqlite3_open(db_path, &db) == SQLITE_OK, "sqlite_open entities"); + const char *sql = + "SELECT file_path, kind, name, start_row, start_col, " + "id FROM entity WHERE project_id=? " + "ORDER BY file_path, kind, name, start_row, start_col"; + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK, + "prepare dumpEntities"); + sqlite3_bind_int64(st, 1, static_cast(project_id)); + std::vector out; + while (sqlite3_step(st) == SQLITE_ROW) { + out.push_back(col(st, 0) + "|" + + std::to_string(sqlite3_column_int(st, 1)) + "|" + + col(st, 2) + "|" + + std::to_string(sqlite3_column_int(st, 3)) + "|" + + std::to_string(sqlite3_column_int(st, 4)) + "|" + + std::to_string(sqlite3_column_int64(st, 5))); + } + sqlite3_finalize(st); + sqlite3_close(db); + return out; +} + +/// Dump every relation as source tuple -> target tuple with its type. +static std::vector dumpRelations(const char *db_path, + uint64_t project_id) +{ + sqlite3 *db = nullptr; + check(sqlite3_open(db_path, &db) == SQLITE_OK, "sqlite_open relations"); + const char *sql = + "SELECT s.file_path, s.kind, s.name, s.start_row, " + " t.file_path, t.kind, t.name, t.start_row, r.type " + "FROM relation r " + "JOIN entity s ON s.id = r.source_id " + "JOIN entity t ON t.id = r.target_id " + "WHERE r.project_id=? " + "ORDER BY s.file_path, s.kind, s.name, s.start_row, " + " t.file_path, t.kind, t.name, t.start_row, r.type"; + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK, + "prepare dumpRelations"); + sqlite3_bind_int64(st, 1, static_cast(project_id)); + std::vector out; + while (sqlite3_step(st) == SQLITE_ROW) { + std::string line = col(st, 0) + "|" + + std::to_string(sqlite3_column_int(st, 1)) + + "|" + col(st, 2) + "|" + + std::to_string(sqlite3_column_int(st, 3)); + line += " -> "; + line += col(st, 4) + "|" + + std::to_string(sqlite3_column_int(st, 5)) + "|" + + col(st, 6) + "|" + + std::to_string(sqlite3_column_int(st, 7)); + line += " #" + std::to_string(sqlite3_column_int(st, 8)); + out.push_back(line); + } + sqlite3_finalize(st); + sqlite3_close(db); + return out; +} + +/// Report the first difference between two dumps (or an empty string). +static std::string firstDiff(const std::vector &a, + const std::vector &b) +{ + if (a.size() != b.size()) + return "size " + std::to_string(a.size()) + " vs " + + std::to_string(b.size()); + for (size_t i = 0; i < a.size(); ++i) + if (a[i] != b[i]) + return "[" + std::to_string(i) + "] " + a[i] + + " != " + b[i]; + return ""; +} + +// ── Controlled-order section ───────────────────────────────────── +// +// The end-to-end comparison above depends on the parse workers actually +// settling the files in different orders, which is timing dependent. This +// section removes that dependency: it drives GraphStore directly and +// inserts the same two files' records in BOTH orders, then asserts the +// resulting ids are identical and follow the semantic key (file_path) +// rather than the insertion order. That is the deterministic core of the +// guarantee. + +/// The single file used by the controlled-order section. +static const char *kOrderFile = "/order/only.cpp"; + +/// The two Function records inserted by the controlled-order section. Their +/// declaration positions are FIXED (Alpha at row 1, Axe at row 5), so the +/// record set is identical between the two runs and only the insert order +/// differs. +static std::vector fixedOrderRecords() +{ + std::vector recs; + auto make = [](const char *name, uint32_t row, uint64_t id) { + ir::Record r; + r.id = id; + r.original_id = id; + r.kind = ir::RecordKind::Function; + r.name = name; + r.qualified_name = name; + r.file_path = kOrderFile; + r.language = "cpp"; + r.loc = ir::SourceRange{ row, 0, row + 1, 0 }; + return r; + }; + recs.push_back(make("Alpha", 1, 1)); + recs.push_back(make("Axe", 5, 2)); + return recs; +} + +/// Build a map of entity name -> entity id for the project. +static std::map entityIds(store::GraphStore &store, + uint64_t project_id) +{ + sqlite3 *db = store.handle(); + const char *sql = "SELECT name, id FROM entity WHERE project_id=? " + "ORDER BY name"; + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK, + "prepare entityIds"); + sqlite3_bind_int64(st, 1, static_cast(project_id)); + std::map ids; + while (sqlite3_step(st) == SQLITE_ROW) { + const char *n = reinterpret_cast( + sqlite3_column_text(st, 0)); + ids[n ? n : ""] = + static_cast(sqlite3_column_int64(st, 1)); + } + sqlite3_finalize(st); + return ids; +} + +/// Insert the same two records forward or reversed and return the +/// resulting name -> id map. +/// +/// One file only: with several files SQLite can pick an index ordered by +/// file_path, which hides an unordered ROW_NUMBER. Within one file_path the +/// index ties and the scan falls back to rowid, i.e. the insert order — +/// exactly the dependency this test must expose. +static std::map runControlledOrder(bool reversed) +{ + const char *db_path = reversed ? + "/tmp/test_index_determinism_ord_rev.db" : + "/tmp/test_index_determinism_ord_fwd.db"; + unlink(db_path); + + store::GraphStore store; + check(store.open(db_path), "open controlled-order database"); + uint64_t pid = store.createProject("/order", "order"); + check(pid > 0, "createProject (controlled order)"); + + std::vector recs = fixedOrderRecords(); + if (reversed) + std::swap(recs[0], recs[1]); + store.insertSemanticRecords(pid, kOrderFile, recs); + + check(store.buildGraph(pid, false), "buildGraph (controlled order)"); + + std::map ids = entityIds(store, pid); + store.close(); + unlink(db_path); + return ids; +} + +int main() +{ + const std::string root = "/tmp/codescope_determinism"; + buildFixture(root); + + const char *db1 = "/tmp/test_index_determinism_1.db"; + const char *db2 = "/tmp/test_index_determinism_2.db"; + // Different worker counts: same input, different settling order. + uint64_t pid1 = indexInto(root, db1, 1); + uint64_t pid2 = indexInto(root, db2, 8); + + std::vector ent1 = dumpEntities(db1, pid1); + std::vector ent2 = dumpEntities(db2, pid2); + std::vector rel1 = dumpRelations(db1, pid1); + std::vector rel2 = dumpRelations(db2, pid2); + + printf(" [INFO] run 1: %zu entities, %zu relations\n", ent1.size(), + rel1.size()); + printf(" [INFO] run 2: %zu entities, %zu relations\n", ent2.size(), + rel2.size()); + + // Guard against a vacuous pass on an empty/failed index. + check(static_cast(ent1.size()) >= kMinEntities, + "fixture produced too few entities — index did not run"); + check(static_cast(rel1.size()) >= kMinRelations, + "fixture produced too few relations — resolution did not run"); + + // Entity ids must be a pure function of the input (this is the part + // ROW_NUMBER() OVER () broke). + std::string ent_diff = firstDiff(ent1, ent2); + if (!ent_diff.empty()) + fprintf(stderr, "FAIL: entity dump differs: %s\n", + ent_diff.c_str()); + check(ent_diff.empty(), + "entity ids are not reproducible: indexing the same input " + "twice produced different entities"); + printf(" [PASS] entity ids reproducible across two index runs\n"); + + std::string rel_diff = firstDiff(rel1, rel2); + if (!rel_diff.empty()) + fprintf(stderr, "FAIL: relation dump differs: %s\n", + rel_diff.c_str()); + check(rel_diff.empty(), + "call graph is not reproducible: indexing the same input " + "twice produced different relations"); + printf(" [PASS] call graph reproducible across two index runs\n"); + + // ── Controlled insertion order (deterministic, timing independent) ── + std::map ids_fwd = runControlledOrder(false); + std::map ids_rev = runControlledOrder(true); + + check(ids_fwd.size() == 2, + "controlled-order fixture lost entities (expected 2)"); + check(ids_fwd == ids_rev, + "entity ids depend on the semantic_records insertion order: " + "inserting the same two records in reverse order produced " + "different ids"); + // Alpha is declared at row 1 and Axe at row 5, so the semantic key + // (start_row) must give Alpha the lower id. + check(ids_fwd.at("Alpha") < ids_fwd.at("Axe"), + "entity ids do not follow the semantic key (start_row): " + "Alpha is declared first in the file and must own the lower id"); + printf(" [PASS] ids follow the semantic key, not the insertion " + "order\n"); + + unlink(db1); + unlink(db2); + std::filesystem::remove_all(root); + printf("\nAll index determinism tests passed.\n"); + return 0; +} diff --git a/engine/tests/test_index_metrics.cpp b/engine/tests/test_index_metrics.cpp index 3e70344..af69870 100644 --- a/engine/tests/test_index_metrics.cpp +++ b/engine/tests/test_index_metrics.cpp @@ -11,7 +11,7 @@ #include "../src/engine_index_metrics.h" #include "../src/ir/ir.h" -#include +#include "test_check.h" #include #include @@ -217,7 +217,7 @@ int main() // ── Test 1: null unit returns empty ──────────────────────────── { auto metrics = index_metrics::computeMetricsFromUnit(nullptr); - assert(metrics.empty()); + CHECK(metrics.empty()); printf(" [PASS] null unit -> empty metrics\n"); } @@ -226,24 +226,24 @@ int main() auto *unit = buildComplexFunctionUnit(); auto metrics = index_metrics::computeMetricsFromUnit(unit); - assert(metrics.size() == 1); + CHECK(metrics.size() == 1); const auto &m = metrics[0]; - assert(m.name == "complex_func"); - assert(m.line == 10); - assert(m.col == 0); - assert(m.lines == 21); // end_row(30) - start_row(10) + 1 - assert(m.param_count == 2); - assert(m.call_count == 1); - assert(m.branch_count == 1); - assert(m.loop_count == 1); - assert(m.cyclomatic == 3); // 1 + 1 branch + 1 loop + CHECK(m.name == "complex_func"); + CHECK(m.line == 10); + CHECK(m.col == 0); + CHECK(m.lines == 21); // end_row(30) - start_row(10) + 1 + CHECK(m.param_count == 2); + CHECK(m.call_count == 1); + CHECK(m.branch_count == 1); + CHECK(m.loop_count == 1); + CHECK(m.cyclomatic == 3); // 1 + 1 branch + 1 loop // nesting_depth is not computed by the IR path (computeMetricsFromUnit); // it is only computed by the CST path (computeMetricsFromCST). This is // a known limitation — cognitive complexity equals cyclomatic here. - assert(m.nesting_depth == 0); - assert(m.cognitive == 3); // cyclomatic + nesting_depth(0) - assert(m.is_stub == false); // has CallExpr + IfStmt + ForStmt + CHECK(m.nesting_depth == 0); + CHECK(m.cognitive == 3); // cyclomatic + nesting_depth(0) + CHECK(m.is_stub == false); // has CallExpr + IfStmt + ForStmt printf(" [PASS] complex_func: params=%d calls=%d branches=%d loops=%d cyclomatic=%d\n", m.param_count, m.call_count, m.branch_count, @@ -257,16 +257,16 @@ int main() auto *unit = buildStubFunctionUnit(); auto metrics = index_metrics::computeMetricsFromUnit(unit); - assert(metrics.size() == 1); + CHECK(metrics.size() == 1); const auto &m = metrics[0]; - assert(m.name == "stub_func"); - assert(m.param_count == 0); - assert(m.call_count == 0); - assert(m.branch_count == 0); - assert(m.loop_count == 0); - assert(m.cyclomatic == 1); // 1 + 0 + 0 - assert(m.is_stub == true); // no real statements + CHECK(m.name == "stub_func"); + CHECK(m.param_count == 0); + CHECK(m.call_count == 0); + CHECK(m.branch_count == 0); + CHECK(m.loop_count == 0); + CHECK(m.cyclomatic == 1); // 1 + 0 + 0 + CHECK(m.is_stub == true); // no real statements printf(" [PASS] stub_func: is_stub=%d cyclomatic=%d\n", m.is_stub, m.cyclomatic); @@ -279,19 +279,19 @@ int main() auto *unit = buildMultiFunctionUnit(); auto metrics = index_metrics::computeMetricsFromUnit(unit); - assert(metrics.size() == 2); + CHECK(metrics.size() == 2); // Function 1: one if-statement - assert(metrics[0].name == "func_with_branch"); - assert(metrics[0].branch_count == 1); - assert(metrics[0].loop_count == 0); - assert(metrics[0].cyclomatic == 2); // 1 + 1 + 0 + CHECK(metrics[0].name == "func_with_branch"); + CHECK(metrics[0].branch_count == 1); + CHECK(metrics[0].loop_count == 0); + CHECK(metrics[0].cyclomatic == 2); // 1 + 1 + 0 // Function 2: one while-statement (MethodDecl, not FunctionDecl) - assert(metrics[1].name == "method_with_loop"); - assert(metrics[1].branch_count == 0); - assert(metrics[1].loop_count == 1); - assert(metrics[1].cyclomatic == 2); // 1 + 0 + 1 + CHECK(metrics[1].name == "method_with_loop"); + CHECK(metrics[1].branch_count == 0); + CHECK(metrics[1].loop_count == 1); + CHECK(metrics[1].cyclomatic == 2); // 1 + 0 + 1 printf(" [PASS] multi-func: func_with_branch(cyclomatic=%d) + method_with_loop(cyclomatic=%d)\n", metrics[0].cyclomatic, metrics[1].cyclomatic); @@ -300,5 +300,5 @@ int main() } printf("=== index_metrics test passed ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_ir.cpp b/engine/tests/test_ir.cpp index 510834e..1d6ea14 100644 --- a/engine/tests/test_ir.cpp +++ b/engine/tests/test_ir.cpp @@ -1,64 +1,66 @@ #include "../src/ir/ir.h" -#include +#include "test_check.h" #include #include -int main() { - // Test: create a minimal IR tree manually and verify it - using namespace ir; +int main() +{ + // Test: create a minimal IR tree manually and verify it + using namespace ir; - auto* unit = new TranslationUnit(); + auto *unit = new TranslationUnit(); - auto* root = new Node(); - root->kind = NodeKind::TranslationUnit; - root->language = "python"; - root->file_path = "/test/test.py"; - root->name = "test_module"; + auto *root = new Node(); + root->kind = NodeKind::TranslationUnit; + root->language = "python"; + root->file_path = "/test/test.py"; + root->name = "test_module"; - auto* func = new Node(); - func->kind = NodeKind::FunctionDecl; - func->name = "hello"; - func->language = "python"; - func->file_path = "/test/test.py"; - func->loc = SourceLocation{1, 0, 3, 10}; + auto *func = new Node(); + func->kind = NodeKind::FunctionDecl; + func->name = "hello"; + func->language = "python"; + func->file_path = "/test/test.py"; + func->loc = SourceLocation{ 1, 0, 3, 10 }; - auto* call = new Node(); - call->kind = NodeKind::CallExpr; - call->language = "python"; - call->file_path = "/test/test.py"; - call->loc = SourceLocation{2, 4, 2, 12}; + auto *call = new Node(); + call->kind = NodeKind::CallExpr; + call->language = "python"; + call->file_path = "/test/test.py"; + call->loc = SourceLocation{ 2, 4, 2, 12 }; - // Build tree - root->children.push_back(func); - func->children.push_back(call); + // Build tree + root->children.push_back(func); + func->children.push_back(call); - // Add semantic edge: call -> func - call->semantic_edges.emplace_back(func, Relation::CallTarget); + // Add semantic edge: call -> func + call->semantic_edges.emplace_back(func, Relation::CallTarget); - unit->root = root; + unit->root = root; - // Assign IDs - std::function collect = [&](Node* n) { - unit->all_nodes.push_back(n); - for (auto* c : n->children) collect(c); - }; - collect(root); - unit->assignIds(); + // Assign IDs + std::function collect = [&](Node *n) { + unit->all_nodes.push_back(n); + for (auto *c : n->children) + collect(c); + }; + collect(root); + unit->assignIds(); - // Verify - assert(unit->root == root); - assert(unit->all_nodes.size() == 3); - assert(root->id == 0); - // After assignIds: root=0, func=1 (first child), call=2 (child of func) + // Verify + CHECK(unit->root == root); + CHECK(unit->all_nodes.size() == 3); + CHECK(root->id == 0); + // After assignIds: root=0, func=1 (first child), call=2 (child of func) - assert(func->name == "hello"); - assert(func->loc.start_row == 1); - assert(call->semantic_edges.size() == 1); - assert(call->semantic_edges[0].target == func); - assert(call->semantic_edges[0].relation == Relation::CallTarget); + CHECK(func->name == "hello"); + CHECK(func->loc.start_row == 1); + CHECK(call->semantic_edges.size() == 1); + CHECK(call->semantic_edges[0].target == func); + CHECK(call->semantic_edges[0].relation == Relation::CallTarget); - printf("=== IR test passed ===\n"); + printf("=== IR test passed ===\n"); - delete unit; - return 0; + delete unit; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_ir_deep_nesting.cpp b/engine/tests/test_ir_deep_nesting.cpp new file mode 100644 index 0000000..4c66e04 --- /dev/null +++ b/engine/tests/test_ir_deep_nesting.cpp @@ -0,0 +1,262 @@ +// test_ir_deep_nesting.cpp — regression test for the visitor recursion depth +// guards (bug: unbounded AST recursion → SIGSEGV) and for their truncation +// contract. +// +// Before the fix: +// the recursive walks had no depth limit. tree-sitter happily parses a +// pathologically deep AST — e.g. ~20k nested call expressions — and a walk +// then overflowed the native C++ stack and crashed with SIGSEGV. The FFI +// try/catch boundary cannot recover a stack overflow, so indexing an +// untrusted/generated file could kill the worker. +// +// After the fix: +// every recursive walk is bounded by `kMaxVisitDepth`: the traversal +// (`JsVisitor::visitChild`, the single guarded descent every recursive step +// goes through, including language handlers that recurse on their own), the +// defined-names pre-pass (`JsVisitor::collectDefinedNames`) and the C++ +// out-of-class-definition pre-scan (`CVisitor::collectOutOfClassDefs`). The +// first walk to reach the limit reports the file once on stderr, tagged +// [module=ir, method=…]; the others stay quiet for that file. Indexing +// completes with "ok":true and the file's top-level symbols are recorded. +// +// This test asserts both halves of that contract: a file well below the cap is +// indexed with no truncation report, and 20k-deep files (Python calls, Python +// parens, C++ parens) each survive and produce exactly one report. Without the +// guards the deep cases crash the process instead of failing an assertion. + +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include "test_engine_handle.h" + +static inline void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "FAIL: %s\n", msg); + exit(1); + } +} + +/// Depth well below kMaxVisitDepth: nothing may be dropped. +static constexpr int kShallowNestingDepth = 100; + +/// Depth chosen far above kMaxVisitDepth and into the range that reproduced +/// the SIGSEGV in the original report (~20k). If the guards were removed this +/// test would crash rather than fail an assertion. +static constexpr int kDefaultNestingDepth = 20000; + +/// The marker every depth guard prints when it truncates. +static const char *kTruncationMarker = "exceeded kMaxVisitDepth"; + +static int nestingDepth() +{ + // Overridable for local bisection while debugging the guards. + if (const char *env = getenv("IR_DEEP_NESTING_DEPTH")) { + int v = atoi(env); + if (v > 0) + return v; + } + return kDefaultNestingDepth; +} + +/// Shape of the generated Python expression. +enum class DeepShape { + /// `f(f(f(...f(0)...)))` — the shape from the original crash report. The + /// call handler consumes each argument itself, so this nests deeply in the + /// DEFINED-NAMES pre-pass (which walks every node) but not in the main + /// traversal. + NestedCalls, + /// `((((...0...))))` — plain expression nesting, which both walks descend + /// node by node. + NestedParens, +}; + +// Build a Python file whose `top()` returns `depth`-nested expressions, with a +// shallow top-level function so a symbol exists outside the deep subtree. +static std::string makeDeeplyNestedPython(int depth, DeepShape shape) +{ + std::string body; + body.reserve(static_cast(depth) * 4 + 64); + if (shape == DeepShape::NestedCalls) { + for (int i = 0; i < depth; i++) + body += "f("; + body += "0"; + for (int i = 0; i < depth; i++) + body += ")"; + } else { + body.append(static_cast(depth), '('); + body += "0"; + body.append(static_cast(depth), ')'); + } + + std::string src = "def f(x):\n return x\n\n"; + src += "def top():\n return "; + src += body; + src += "\n"; + return src; +} + +// Build a C++ file with the same plain expression nesting. Exercises the C++ +// pre-scans as well as the traversal. +static std::string makeDeeplyNestedCpp(int depth) +{ + std::string body; + body.reserve(static_cast(depth) * 2 + 64); + body.append(static_cast(depth), '('); + body += "0"; + body.append(static_cast(depth), ')'); + + std::string src = "int top() { return "; + src += body; + src += "; }\n"; + return src; +} + +/// Index one generated file into a fresh database and return everything the +/// engine wrote to stderr during the run. +/// +/// stderr (not the JSON result) is the observable that proves a guard engaged: +/// the truncation report is a one-time diagnostic, not part of the tool +/// response. The descriptor is restored before returning so a failing +/// assertion below is still visible on the terminal. +/// +/// \param source File contents to index. +/// \param file_name Name to write it under (extension selects the language). +/// \param proj_dir Scratch project directory (recreated). +/// \param db_path Database path (recreated). +/// \return Captured stderr of the whole index run. +static std::string indexCase(const std::string &source, + const std::string &file_name, + const std::string &proj_dir, + const std::string &db_path) +{ + std::filesystem::remove_all(proj_dir); + std::filesystem::create_directories(proj_dir); + + const std::string src_path = proj_dir + "/" + file_name; + FILE *f = fopen(src_path.c_str(), "w"); + check(f != nullptr, "fopen source file"); + check(fwrite(source.data(), 1, source.size(), f) == source.size(), + "write source file"); + fclose(f); + + unlink(db_path.c_str()); + unlink((db_path + "-wal").c_str()); + unlink((db_path + "-shm").c_str()); + + const std::string capture = db_path + ".stderr"; + fflush(stderr); + const int saved_fd = dup(fileno(stderr)); + check(saved_fd >= 0, "dup(stderr)"); + check(freopen(capture.c_str(), "w", stderr) != nullptr, + "freopen stderr to capture file"); + + g_engine = engine_create(db_path.c_str()); + check(g_engine != nullptr, "engine_init"); + uint64_t pid = engine_create_project(g_engine, proj_dir.c_str(), + "deep-nesting"); + check(pid > 0, "create_project"); + + // The assertion that matters most is implicit: this call must RETURN. + // Without the guards the process SIGSEGVs here and the test binary exits + // with a signal — a failure under `make test-engine`. + char *idx = + engine_index_project(g_engine, pid, proj_dir.c_str(), nullptr); + check(idx != nullptr, "index_project returns non-null (no crash)"); + check(strstr(idx, "\"ok\":true") != nullptr, + "index_project reports ok:true on deeply nested input"); + engine_free_string(idx); + + // Indexing still finished: the top-level function survived the walk (it is + // emitted before the deep subtree is truncated). + char *sym = engine_find_symbol(g_engine, pid, "top"); + check(sym != nullptr, "find_symbol(top) non-null"); + check(strstr(sym, "top") != nullptr, + "top-level function must still be indexed after truncation"); + engine_free_string(sym); + + // Join the async knowledge-builder thread before returning — a + // std::thread destroyed while joinable calls std::terminate. + engine_destroy(g_engine); + g_engine = nullptr; + + fflush(stderr); + dup2(saved_fd, fileno(stderr)); + close(saved_fd); + + std::string captured; + if (FILE *cap = fopen(capture.c_str(), "r")) { + char buf[4096]; + size_t n = 0; + while ((n = fread(buf, 1, sizeof buf, cap)) > 0) + captured.append(buf, n); + fclose(cap); + } + return captured; +} + +/// Count non-overlapping occurrences of `needle` in `haystack`. +static int countOccurrences(const std::string &haystack, const char *needle) +{ + int n = 0; + for (size_t pos = haystack.find(needle); pos != std::string::npos; + pos = haystack.find(needle, pos + 1)) + n++; + return n; +} + +int main() +{ + const int deep = nestingDepth(); + + // ── 1. Below the cap: no truncation, nothing dropped ───────── + const std::string shallow = + indexCase(makeDeeplyNestedPython(kShallowNestingDepth, + DeepShape::NestedCalls), + "deep.py", "/tmp/ir_deep_nesting_shallow", + "/tmp/test_ir_shallow_nesting.db"); + check(countOccurrences(shallow, kTruncationMarker) == 0, + "a file below kMaxVisitDepth must not be truncated"); + + // ── 2. Deeply nested Python CALLS (the original crash) ─────── + const std::string calls_err = indexCase( + makeDeeplyNestedPython(deep, DeepShape::NestedCalls), "deep.py", + "/tmp/ir_deep_nesting_repro", "/tmp/test_ir_deep_nesting.db"); + check(countOccurrences(calls_err, kTruncationMarker) == 1, + "a file far above kMaxVisitDepth must report the truncation once"); + + // ── 3. Deeply nested Python PARENS ─────────────────────────── + const std::string parens_err = + indexCase(makeDeeplyNestedPython(deep, DeepShape::NestedParens), + "deep.py", "/tmp/ir_deep_nesting_parens", + "/tmp/test_ir_deep_nesting_parens.db"); + check(countOccurrences(parens_err, kTruncationMarker) == 1, + "deep expression nesting must be bounded and reported once"); + // The report is shared per file (JsVisitor::depth_truncated_): whichever + // capped walk reaches the limit first reports the file and the others stay + // quiet for it, so a deep file yields exactly one diagnostic — never one + // per node, and never none. + check(parens_err.find("[module=ir, method=") != std::string::npos, + "the truncation report must carry its module/method tag"); + + // ── 4. Deeply nested C++ PARENS ────────────────────────────── + // Same shape through the C visitor: this exercises the C++ pre-scans + // (collectDefinedNames + collectOutOfClassDefs) in addition to the + // traversal, all of which recurse over the tree. + const std::string cpp_err = indexCase( + makeDeeplyNestedCpp(deep), "deep.cpp", + "/tmp/ir_deep_nesting_cpp", "/tmp/test_ir_deep_nesting_cpp.db"); + check(countOccurrences(cpp_err, kTruncationMarker) == 1, + "a deep C++ file must be bounded and reported once"); + + fprintf(stderr, + "PASS: deep-nesting guards held (shallow=%d, deep=%d, " + "py-calls/py-parens/cpp-parens)\n", + kShallowNestingDepth, deep); + return 0; +} diff --git a/engine/tests/test_ir_edge_coverage.cpp b/engine/tests/test_ir_edge_coverage.cpp new file mode 100644 index 0000000..4a69b0f --- /dev/null +++ b/engine/tests/test_ir_edge_coverage.cpp @@ -0,0 +1,470 @@ +/** + * Unit tests for the IR coverage gaps fixed for + * 2026-09-27 review D1-2 (README "Verified" column). + * + * Why this file exists: the per-language E2E harness (test_e2e.h) only asserts + * that the JSON contains the KEYS "callers"/"callees"/"total_nodes" — never + * that an edge exists. A translator that emitted zero call edges therefore + * passed the whole suite, which is how these gaps survived a green build. + * These cases assert the emitted RECORDS directly, at the visitor level: + * + * 1. Rust — macro_invocation emits a CallExpr; builtin macros do not. + * 2. Go — `a, b := ...` emits/defines the variables (LHS lives inside + * an expression_list, which the old scan never reached). + * 3. C++ — in-class method declarations, operator== and ~Point are + * recorded, and out-of-class definitions keep their scope. + * 4. Python — a chained callee `self.helper.compute()` resolves to the + * LAST segment ("compute"), not the receiver ("helper"). + * 5. Java — `implements` emits InterfaceImpl (names live in a type_list). + * 6. TS/TSX — `implements` emits InterfaceImpl; a self-closing JSX element's + * attribute expressions are visited. + * 7. C++/TS — a user function whose name collides with a builtin is not + * dropped by the builtin-name filter. + */ + +#include "../src/ir/semantic_unit.h" +#include "../src/ir/translators/cpp_visitor.h" +#include "../src/ir/translators/go_visitor.h" +#include "../src/ir/translators/java_visitor.h" +#include "../src/ir/translators/python_visitor.h" +#include "../src/ir/translators/rust_visitor.h" +#include "../src/ir/translators/ts_visitor.h" +#include "../src/ir/translators/tsx_visitor.h" +#include "grammar_loader.h" + +#include +#include +#include +#include + +#include + +// ── Harness ─────────────────────────────────────────────────── + +static int tests_run = 0; +static int tests_passed = 0; + +#define CHECK(cond, msg) \ + do { \ + tests_run++; \ + if (!(cond)) { \ + fprintf(stderr, "FAIL [%d]: %s\n", tests_run, msg); \ + exit(1); \ + } \ + tests_passed++; \ + } while (0) + +/// The grammar, from the library this test links (grammar_loader.h). The old +/// dlopen() of `/tree-sitter-.so` searched directories this +/// repository never writes, so `grammar loaded` failed on CI for every test +/// that used this loader. +static const TSLanguage *loadLanguage(const char *grammar) +{ + return testGrammar(grammar); +} + +/// Parse `code` with the given grammar and run `visitor` over it. +/// Ownership of the returned unit passes to the caller. +static ir::SemanticUnit *runVisitor(ir::JsVisitor &visitor, const char *grammar, + const char *code, const char *file_path) +{ + const TSLanguage *lang = loadLanguage(grammar); + CHECK(lang != nullptr, "grammar loaded"); + + TSParser *parser = ts_parser_new(); + ts_parser_set_language(parser, lang); + TSTree *tree = ts_parser_parse_string( + parser, nullptr, code, static_cast(strlen(code))); + ts_parser_delete(parser); + CHECK(tree != nullptr, "parse succeeded"); + + ir::SemanticUnit *unit = visitor.visit(tree, code, file_path); + ts_tree_delete(tree); + return unit; +} + +/// True when a record of `kind` is named `name`. +static bool hasNamed(const ir::SemanticUnit &unit, ir::RecordKind kind, + const std::string &name) +{ + for (size_t idx : unit.findRecordsByKind(kind)) { + if (unit.allRecords()[idx].name == name) + return true; + } + return false; +} + +/// Number of records of `kind` named `name`. +static int countNamed(const ir::SemanticUnit &unit, ir::RecordKind kind, + const std::string &name) +{ + int n = 0; + for (size_t idx : unit.findRecordsByKind(kind)) { + if (unit.allRecords()[idx].name == name) + n++; + } + return n; +} + +/// True when any record carries `qualified_name`. +static bool hasQualifiedName(const ir::SemanticUnit &unit, + const std::string &qname) +{ + for (const ir::Record &r : unit.allRecords()) { + if (r.qualified_name == qname) + return true; + } + return false; +} + +/// True when `impl_type` implements `iface` (an InterfaceImpl record). +static bool hasInterfaceImpl(const ir::SemanticUnit &unit, + const std::string &impl_type, + const std::string &iface) +{ + for (size_t idx : + unit.findRecordsByKind(ir::RecordKind::InterfaceImpl)) { + const ir::Record &r = unit.allRecords()[idx]; + if (r.name == impl_type && r.type_name == iface) + return true; + } + return false; +} + +// ── 1. Rust macro invocations ───────────────────────────────── + +static void test_rust_macro_invocation_emits_call() +{ + const char *code = "fn do_work() {}\n" + "macro_rules! do_work { () => {}; }\n" + "fn caller() { do_work!(); }\n"; + + ir::RustVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "rust", + code, "/test/macro.rs"); + + CHECK(hasNamed(*unit, ir::RecordKind::CallExpr, "do_work"), + "macro_invocation must emit a CallExpr (macro calls are calls)"); + + delete unit; + printf(" ✓ test_rust_macro_invocation_emits_call\n"); +} + +static void test_rust_builtin_macro_filtered() +{ + const char *code = "fn caller() { println!(\"hello\"); }\n"; + + ir::RustVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "rust", + code, "/test/builtin_macro.rs"); + + CHECK(!hasNamed(*unit, ir::RecordKind::CallExpr, "println"), + "builtin macro println! must not create a call record"); + + delete unit; + printf(" ✓ test_rust_builtin_macro_filtered\n"); +} + +// ── 2. Go short variable declarations ───────────────────────── + +static void test_go_short_var_defines_variables() +{ + // The LHS of `:=` is wrapped in an expression_list, so the previous + // direct-children scan emitted no Variable at all. + const char *code = "package main\n" + "func compute() int { return 1 }\n" + "func caller() {\n" + " a, b := compute(), compute()\n" + " _, c := compute(), compute()\n" + "}\n"; + + ir::GoVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "go", + code, "/test/shortvar.go"); + + CHECK(hasNamed(*unit, ir::RecordKind::Variable, "a"), + "`a, b := ...` must emit Variable a"); + CHECK(hasNamed(*unit, ir::RecordKind::Variable, "b"), + "`a, b := ...` must emit Variable b"); + CHECK(hasNamed(*unit, ir::RecordKind::Variable, "c"), + "`_, c := ...` must emit Variable c"); + CHECK(!hasNamed(*unit, ir::RecordKind::Variable, "_"), + "the blank identifier binds no symbol"); + + delete unit; + printf(" ✓ test_go_short_var_defines_variables\n"); +} + +static void test_go_blank_identifier_binds_nothing() +{ + // `_` is a discard, not a symbol, in every form — and discarding the name + // must not discard the initializer: its calls still belong to the + // enclosing function. Both halves regressed differently before: + // `_, x := f()` skipped the name but a plain `_, x = f()` emitted a `_` + // entity, and `var x = f()` dropped the call entirely. + const char *code = "package main\n" + "func compute() int { return 1 }\n" + "func caller() int {\n" + " _, c := compute(), compute()\n" + " var _ = compute()\n" + " var x = compute()\n" + " _, c = compute(), compute()\n" + " return c + x\n" + "}\n"; + + ir::GoVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "go", + code, "/test/blank.go"); + + CHECK(!hasNamed(*unit, ir::RecordKind::Variable, "_"), + "the blank identifier must not become a symbol"); + CHECK(countNamed(*unit, ir::RecordKind::CallExpr, "compute") == 6, + "every compute() call must be recorded — including the ones in a " + "blank-identifier or var-declaration initializer"); + + delete unit; + printf(" ✓ test_go_blank_identifier_binds_nothing\n"); +} + +// ── 3. C++ in-class declarations, operators, destructors ────── + +static void test_cpp_class_member_declarations() +{ + // Point's members are DECLARED only (no definition anywhere in the file). + // Line's members are declared in-class AND defined out-of-class. + const char *code = + "class Point {\n" + "public:\n" + " void foo();\n" + " bool operator==(const Point &o) const;\n" + " ~Point();\n" + "};\n" + "class Line {\n" + "public:\n" + " bool operator==(const Line &o) const;\n" + " ~Line();\n" + " int len;\n" + "};\n" + "bool Line::operator==(const Line &o) const { return len == o.len; }\n" + "Line::~Line() {}\n"; + + ir::CppVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "cpp", + code, "/test/members.cpp"); + + CHECK(hasNamed(*unit, ir::RecordKind::Method, "foo"), + "in-class declaration `void foo();` must emit a Method"); + CHECK(hasNamed(*unit, ir::RecordKind::Method, "operator=="), + "in-class `operator==` declaration must be named"); + CHECK(hasNamed(*unit, ir::RecordKind::Method, "~Point"), + "in-class destructor declaration must be named"); + CHECK(hasQualifiedName(*unit, "Line::operator=="), + "out-of-class definition must carry its scope (type_identifier, " + "not identifier)"); + CHECK(hasQualifiedName(*unit, "Line::~Line"), + "out-of-class destructor definition must carry its scope"); + // A declaration whose definition is in the SAME file must not be + // emitted a second time: two same-name/same-arity candidates make the + // Resolver's ambiguity gate abstain and the CALLS edge is lost + // (test_qualified_id_ast covers that edge). + CHECK(!hasNamed(*unit, ir::RecordKind::Method, "~Line"), + "declaration + same-file definition must not create a duplicate"); + + delete unit; + printf(" ✓ test_cpp_class_member_declarations\n"); +} + +static void test_cpp_local_function_decl_is_not_a_member() +{ + // A LOCAL function declaration inside a method body is not a class member. + // Dispatching `declaration` nodes on the enclosing class name (an earlier + // revision) also matched these and emitted a phantom `Point::helper`. + const char *code = "class Point {\n" + "public:\n" + " void run();\n" + " ~Point();\n" + "};\n" + "void Point::run() {\n" + " int helper(int);\n" + " (void)helper(1);\n" + "}\n" + "Point::~Point() {}\n"; + + ir::CppVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "cpp", + code, "/test/local.cpp"); + + CHECK(hasQualifiedName(*unit, "Point::run"), + "the out-of-class definition must carry its scope"); + CHECK(!hasNamed(*unit, ir::RecordKind::Method, "run"), + "a declaration whose definition is in the same file stays " + "deduplicated"); + CHECK(!hasNamed(*unit, ir::RecordKind::Method, "helper") && + !hasQualifiedName(*unit, "Point::helper"), + "a local function declaration must not become a class member"); + CHECK(!hasNamed(*unit, ir::RecordKind::Method, "~Point"), + "a declaration whose definition is in the same file stays " + "deduplicated"); + CHECK(hasQualifiedName(*unit, "Point::~Point"), + "the destructor definition must still be recorded"); + + delete unit; + printf(" ✓ test_cpp_local_function_decl_is_not_a_member\n"); +} + +// ── 4. Python chained attribute calls ───────────────────────── + +static void test_python_chained_call_name() +{ + // `self.helper.compute()` must resolve to "compute" — the old code + // returned the INNER receiver segment ("helper"). + const char *code = "class Worker:\n" + " def compute(self, x):\n" + " return x * 2\n" + " def run(self):\n" + " return self.helper.compute(10)\n"; + + ir::PythonVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "python", + code, + "/test/chained.py"); + + CHECK(hasNamed(*unit, ir::RecordKind::CallExpr, "compute"), + "chained call must resolve to the LAST segment (compute)"); + CHECK(!hasNamed(*unit, ir::RecordKind::CallExpr, "helper"), + "chained call must NOT resolve to the receiver (helper)"); + + delete unit; + printf(" ✓ test_python_chained_call_name\n"); +} + +// ── 5. Java implements ──────────────────────────────────────── + +static void test_java_implements_emits_interface_impl() +{ + const char *code = + "interface Drawable { void draw(); }\n" + "class Circle implements Drawable, Comparable {\n" + " public void draw() {}\n" + " public int compareTo(Circle o) { return 0; }\n" + "}\n"; + + ir::JavaVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "java", + code, "/test/Circle.java"); + + CHECK(hasInterfaceImpl(*unit, "Circle", "Drawable"), + "`implements Drawable` must emit InterfaceImpl (names live in " + "type_list)"); + CHECK(hasInterfaceImpl(*unit, "Circle", "Comparable"), + "generic implements clause records the base type"); + + delete unit; + printf(" ✓ test_java_implements_emits_interface_impl\n"); +} + +// ── 6. TypeScript / TSX ─────────────────────────────────────── + +static void test_ts_implements_emits_interface_impl() +{ + const char *code = + "interface Shape { area(): number; }\n" + "class Square implements Shape, Comparable {\n" + " area() { return 1; }\n" + " compareTo(o: Square) { return 0; }\n" + "}\n"; + + ir::TsVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "typescript", + code, + "/test/square.ts"); + + CHECK(hasInterfaceImpl(*unit, "Square", "Shape"), + "TS `implements Shape` must emit InterfaceImpl"); + CHECK(hasInterfaceImpl(*unit, "Square", "Comparable"), + "generic implements clause records the base type"); + + delete unit; + printf(" ✓ test_ts_implements_emits_interface_impl\n"); +} + +static void test_tsx_self_closing_attribute_call() +{ + const char *code = "function bar() { return 1; }\n" + "const el = ;\n"; + + ir::TsxVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "tsx", + code, "/test/elem.tsx"); + + CHECK(hasNamed(*unit, ir::RecordKind::CallExpr, "bar"), + "call inside a self-closing JSX attribute must be visited"); + // The tag name is not code: recursing wholesale would emit it. + CHECK(!hasNamed(*unit, ir::RecordKind::Variable, "Foo"), + "the JSX tag name must not become a Variable"); + + delete unit; + printf(" ✓ test_tsx_self_closing_attribute_call\n"); +} + +// ── 7. Builtin-name exemption (collectDefinedNames) ─────────── + +static void test_cpp_user_function_shadowing_builtin() +{ + // `free` is in isCBuiltin(); a function this file defines must survive + // that filter, which requires collectDefinedNames to have run. + const char *code = "void free(void *p) { (void)p; }\n" + "void caller(void *p) { free(p); }\n"; + + ir::CppVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "cpp", + code, "/test/free.cpp"); + + CHECK(hasNamed(*unit, ir::RecordKind::CallExpr, "free"), + "user-defined `free()` call must not be filtered as a builtin"); + + delete unit; + printf(" ✓ test_cpp_user_function_shadowing_builtin\n"); +} + +static void test_ts_user_function_shadowing_builtin() +{ + // `String` is in isJsBuiltin(); the file defines its own. + const char *code = "function String(x: number) { return x; }\n" + "function caller() { return String(1); }\n"; + + ir::TsVisitor visitor; + ir::SemanticUnit *unit = runVisitor(visitor, "typescript", + code, + "/test/shadow.ts"); + + CHECK(hasNamed(*unit, ir::RecordKind::CallExpr, "String"), + "user-defined `String()` call must not be filtered as a builtin"); + + delete unit; + printf(" ✓ test_ts_user_function_shadowing_builtin\n"); +} + +// ── Main ────────────────────────────────────────────────────── + +int main() +{ + printf("IR edge coverage tests (2026-09-27 review D1-2):\n"); + + test_rust_macro_invocation_emits_call(); + test_rust_builtin_macro_filtered(); + test_go_short_var_defines_variables(); + test_go_blank_identifier_binds_nothing(); + test_cpp_class_member_declarations(); + test_cpp_local_function_decl_is_not_a_member(); + test_python_chained_call_name(); + test_java_implements_emits_interface_impl(); + test_ts_implements_emits_interface_impl(); + test_tsx_self_closing_attribute_call(); + test_cpp_user_function_shadowing_builtin(); + test_ts_user_function_shadowing_builtin(); + + printf("\n=== ir_edge_coverage test passed (%d/%d) ===\n", tests_passed, + tests_run); + return 0; +} diff --git a/engine/tests/test_ir_edges_in_graph.cpp b/engine/tests/test_ir_edges_in_graph.cpp new file mode 100644 index 0000000..68dc537 --- /dev/null +++ b/engine/tests/test_ir_edges_in_graph.cpp @@ -0,0 +1,177 @@ +// test_ir_edges_in_graph.cpp — the IR fixes must reach the GRAPH, not only the +// IR records. +// +// test_ir_edge_coverage.cpp asserts that the visitors emit the right records +// (a Rust macro becomes a CallExpr, a Java `implements` becomes an +// InterfaceImpl, ...). The review that motivated those fixes +// (2026-09-27 review D1-2) was about CALL EDGES being lost, so a +// record-level assertion is one layer short: this test indexes real files +// end-to-end and queries `relation` for the edges those records are supposed to +// produce. +// +// Cases: +// 1. Rust: `do_work!()` must produce a CALLS edge caller → do_work. +// 2. Java: `implements Drawable` must let a call through the interface +// dispatch to the implementation (CALLS edge into Circle.draw). The +// InterfaceImpl record feeds the resolver's dispatch index; without it the +// call would have no candidate and no edge. + +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include +#include "test_engine_handle.h" + +namespace fs = std::filesystem; + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "\nFAIL: %s\n", msg); + exit(1); + } +} + +static void write_file(const std::string &path, const char *content) +{ + FILE *f = fopen(path.c_str(), "w"); + check(f != nullptr, "fopen"); + fputs(content, f); + fclose(f); +} + +/// Index `proj_dir` into a fresh database; on success `out_db` holds the path. +static uint64_t indexProject(const std::string &proj_dir, const char *db_path) +{ + unlink(db_path); + unlink((std::string(db_path) + "-wal").c_str()); + unlink((std::string(db_path) + "-shm").c_str()); + g_engine = engine_create(db_path); + check(g_engine != nullptr, "engine_init"); + uint64_t pid = + engine_create_project(g_engine, proj_dir.c_str(), "ir-edges"); + check(pid > 0, "create_project"); + char *idx = + engine_index_project(g_engine, pid, proj_dir.c_str(), nullptr); + check(idx != nullptr, "index_project result"); + check(strstr(idx, "\"ok\":true") != nullptr, "index_project ok"); + engine_free_string(idx); + engine_destroy(g_engine); + g_engine = nullptr; + return pid; +} + +/// Count `relation` rows of `edge_type` between two entity names. +static int edgeCount(const char *db_path, uint64_t pid, const char *src_name, + const char *tgt_name, int edge_type) +{ + sqlite3 *db = nullptr; + check(sqlite3_open_v2(db_path, &db, SQLITE_OPEN_READONLY, nullptr) == + SQLITE_OK, + "open db for edge count"); + sqlite3_stmt *st = nullptr; + const char *sql = + "SELECT COUNT(*) FROM relation r " + "JOIN entity e_s ON e_s.id=r.source_id AND e_s.project_id=r.project_id " + "JOIN entity e_t ON e_t.id=r.target_id AND e_t.project_id=r.project_id " + "WHERE r.project_id=? AND r.type=? AND e_s.name=? AND e_t.name=?"; + check(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK, + "prepare edge count"); + sqlite3_bind_int64(st, 1, static_cast(pid)); + sqlite3_bind_int(st, 2, edge_type); + sqlite3_bind_text(st, 3, src_name, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(st, 4, tgt_name, -1, SQLITE_TRANSIENT); + int n = 0; + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + sqlite3_close(db); + return n; +} + +/// Count `relation` rows of `edge_type` whose target entity is `tgt_name`. +static int incomingEdgeCount(const char *db_path, uint64_t pid, + const char *tgt_name, int edge_type) +{ + sqlite3 *db = nullptr; + check(sqlite3_open_v2(db_path, &db, SQLITE_OPEN_READONLY, nullptr) == + SQLITE_OK, + "open db for incoming count"); + char *sql = sqlite3_mprintf( + "SELECT COUNT(*) FROM relation r JOIN entity e_t ON " + "e_t.id=r.target_id AND e_t.project_id=r.project_id " + "WHERE r.project_id=%lld AND r.type=%d AND e_t.name=%Q", + static_cast(pid), edge_type, tgt_name); + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK, + "prepare incoming count"); + sqlite3_free(sql); + int n = 0; + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + sqlite3_close(db); + return n; +} + +/// EdgeType::Calls. +static constexpr int kCallsEdge = 1; + +static void test_rust_macro_call_becomes_an_edge() +{ + const std::string proj = "/tmp/ir_edges_rust"; + fs::remove_all(proj); + fs::create_directories(proj); + write_file(proj + "/macro.rs", "fn do_work() {}\n" + "fn caller() { do_work!(); }\n"); + const char *db = "/tmp/test_ir_edges_rust.db"; + const uint64_t pid = indexProject(proj, db); + + check(edgeCount(db, pid, "caller", "do_work", kCallsEdge) >= 1, + "a Rust macro invocation must produce a CALLS edge " + "caller -> do_work"); + printf(" ✓ rust macro invocation → CALLS edge\n"); +} + +static void test_java_implements_reaches_the_graph() +{ + const std::string proj = "/tmp/ir_edges_java"; + fs::remove_all(proj); + fs::create_directories(proj); + write_file(proj + "/Shape.java", + "interface Drawable {\n" + " void draw();\n" + "}\n" + "class Circle implements Drawable {\n" + " public void draw() { }\n" + "}\n" + "class App {\n" + " void use(Drawable d) { d.draw(); }\n" + "}\n"); + const char *db = "/tmp/test_ir_edges_java.db"; + const uint64_t pid = indexProject(proj, db); + + // The InterfaceImpl record feeds the resolver's dispatch index, so the call + // through the interface must reach the implementation. Without the record + // the call has no candidate and no edge exists. + check(incomingEdgeCount(db, pid, "draw", kCallsEdge) >= 1, + "an interface dispatch call must reach the implementing method " + "(needs the `implements` InterfaceImpl record)"); + printf(" ✓ java implements → dispatch CALLS edge\n"); +} + +int main() +{ + printf("IR edges in graph tests (2026-09-27 review D1-2):\n"); + + test_rust_macro_call_becomes_an_edge(); + test_java_implements_reaches_the_graph(); + + printf("\n=== ir_edges_in_graph test passed ===\n"); + return 0; +} diff --git a/engine/tests/test_java_e2e.cpp b/engine/tests/test_java_e2e.cpp index f09b5ec..d928880 100644 --- a/engine/tests/test_java_e2e.cpp +++ b/engine/tests/test_java_e2e.cpp @@ -1,7 +1,8 @@ #include "test_e2e.h" -int main() { - const char* code = R"(class Calculator { +int main() +{ + const char *code = R"(class Calculator { int add(int a, int b) { return a + b; } @@ -19,6 +20,14 @@ int main() { int b = add(x, y); return multiply(a, b); } + + String describe(int value) { + return format(value); + } + + String format(int value) { + return "v"; + } } class Main { @@ -29,12 +38,16 @@ class Main { } )"; - const char* defs[] = {"add", "Calculator", "multiply"}; - runE2eTest("java", code, "/tmp/TestApp.java", - defs, 3, - "add", "multiply", - "add", - "compute", - "main", nullptr); - return 0; + // `describe` and `format` return a CLASS type (`String`). This grammar + // spells a class type `identifier` too, so naming a declaration by its + // first `identifier` child named such methods after their return type: the + // entity used to be called `String`, and find_definition("describe") — the + // assertion below, via runE2eTest's def list — answered nothing. Primitives + // and `void` never showed it (their own node types), which is why the + // original three-definition fixture passed. + const char *defs[] = { "add", "Calculator", "multiply", "describe", + "format" }; + runE2eTest("java", code, "/tmp/TestApp.java", defs, 5, "format", + "describe", "format", "describe", "main", nullptr); + return 0; } diff --git a/engine/tests/test_js_e2e.cpp b/engine/tests/test_js_e2e.cpp index ab64aba..e9f7203 100644 --- a/engine/tests/test_js_e2e.cpp +++ b/engine/tests/test_js_e2e.cpp @@ -1,7 +1,8 @@ #include "test_e2e.h" -int main() { - const char* code = R"(function add(a, b) { +int main() +{ + const char *code = R"(function add(a, b) { return a + b; } @@ -27,12 +28,8 @@ function main() { main(); )"; - const char* defs[] = {"add", "multiply", "compute"}; - runE2eTest("js", code, "/tmp/test_math.js", - defs, 3, - "add", "multiply", - "add", - "compute", - "main", nullptr); - return 0; + const char *defs[] = { "add", "multiply", "compute" }; + runE2eTest("js", code, "/tmp/test_math.js", defs, 3, "add", "multiply", + "add", "compute", "main", nullptr); + return 0; } diff --git a/engine/tests/test_js_ts_call_facts.cpp b/engine/tests/test_js_ts_call_facts.cpp new file mode 100644 index 0000000..f8be51b --- /dev/null +++ b/engine/tests/test_js_ts_call_facts.cpp @@ -0,0 +1,199 @@ +// test_js_ts_call_facts.cpp — regression test for structured call facts in +// the JavaScript / TypeScript visitors. +// +// Defect guarded: JsVisitor::visitCallExpr never called +// SemanticEmitter::setCallFacts(), while the Go / Python / C / Rust / Java +// visitors did. JS/TS reference rows therefore carried empty +// receiver_type / qualified_target / import_alias evidence, which +// * disabled the resolver's fuzzy fallback (its has_evidence gate +// requires at least one structured field), and +// * left factorReceiverTypeMatch neutral for every candidate, so +// same-name methods could not be disambiguated. +// +// The test indexes a small JS + TS project and inspects the persisted +// semantic_records rows for the call sites. +// +// Boundary cases covered: +// * `this.method()` → receiver_type must be the enclosing class name +// * `ns.fn()` on a namespace import → import_alias must be the alias +// * `obj.method()` → receiver_text/qualified_target must name the object +// * a BARE call must leave every structured field empty (the visitor +// must not fabricate a receiver for a direct call) +// +// Exit code: 0 on success, 1 on any failed assertion. + +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include +#include "test_engine_handle.h" + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "FAIL: %s\n", msg); + exit(1); + } +} + +/// The four structured call-fact columns of one CallExpr row. +struct CallFact { + std::string qualified_target; + std::string receiver_text; + std::string receiver_type; + std::string import_alias; + bool found = false; +}; + +/// Fetch the CallExpr (kind=9) row named `name` whose file ends with +/// `file_suffix`. Returns found=false when no such row exists. +static CallFact getCallFact(sqlite3 *db, uint64_t project_id, const char *name, + const char *file_suffix) +{ + const char *sql = + "SELECT qualified_target, receiver_text, receiver_type, " + "import_alias FROM semantic_records " + "WHERE project_id=? AND kind=9 AND name=? AND file_path LIKE ? " + "ORDER BY rowid LIMIT 1"; + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK, + "prepare getCallFact"); + sqlite3_bind_int64(st, 1, static_cast(project_id)); + sqlite3_bind_text(st, 2, name, -1, SQLITE_TRANSIENT); + std::string pattern = std::string("%") + file_suffix; + sqlite3_bind_text(st, 3, pattern.c_str(), -1, SQLITE_TRANSIENT); + CallFact fact; + if (sqlite3_step(st) == SQLITE_ROW) { + auto text = [&](int col) -> std::string { + const char *t = reinterpret_cast( + sqlite3_column_text(st, col)); + return t ? t : ""; + }; + fact.qualified_target = text(0); + fact.receiver_text = text(1); + fact.receiver_type = text(2); + fact.import_alias = text(3); + fact.found = true; + } + sqlite3_finalize(st); + return fact; +} + +static void writeFile(const std::string &path, const char *content) +{ + std::filesystem::create_directories( + std::filesystem::path(path).parent_path()); + FILE *f = fopen(path.c_str(), "w"); + check(f != nullptr, "fopen fixture"); + fputs(content, f); + fclose(f); +} + +int main() +{ + const std::string root = "/tmp/codescope_js_ts_call_facts"; + std::filesystem::remove_all(root); + + // util.js — namespace-import target. + writeFile(root + "/src/util.js", + "export function greet() { return 'hi'; }\n"); + // main.js — namespace call, object call, and a bare call. + writeFile(root + "/src/main.js", + "import * as util from './util.js';\n" + "class Renderer { render() { return 1; } }\n" + "export function standalone() { return 0; }\n" + "export function main() {\n" + " const r = new Renderer();\n" + " r.render();\n" + " util.greet();\n" + " standalone();\n" + "}\n"); + // timeline.ts — `this.method()` receiver inference. + writeFile(root + "/src/timeline.ts", + "export class Timeline {\n" + " render(): number { return 1; }\n" + " draw(): number { return this.render(); }\n" + "}\n"); + + const char *db_path = "/tmp/test_js_ts_call_facts.db"; + unlink(db_path); + g_engine = engine_create(db_path); + check(g_engine != nullptr, "engine_init"); + uint64_t pid = engine_create_project(g_engine, root.c_str(), + "js-ts-call-facts"); + check(pid > 0, "create_project"); + + char *idx = engine_index_project(g_engine, pid, root.c_str(), nullptr); + check(idx != nullptr, "index_project null"); + check(strstr(idx, "\"ok\":true") != nullptr, "index_project ok"); + engine_free_string(idx); + + sqlite3 *db = nullptr; + check(sqlite3_open(db_path, &db) == SQLITE_OK, "sqlite_open"); + + // ── Case 1: namespace import call `util.greet()` ────────────── + { + CallFact f = getCallFact(db, pid, "greet", "main.js"); + check(f.found, "util.greet() call record missing"); + check(f.qualified_target == "util.greet", + "util.greet() qualified_target must be \"util.greet\""); + check(f.receiver_text == "util", + "util.greet() receiver_text must be \"util\""); + check(f.import_alias == "util", + "util.greet() import_alias must be \"util\""); + printf(" [PASS] util.greet(): qualified=%s alias=%s\n", + f.qualified_target.c_str(), f.import_alias.c_str()); + } + + // ── Case 2: object call `r.render()` ────────────────────────── + { + CallFact f = getCallFact(db, pid, "render", "main.js"); + check(f.found, "r.render() call record missing"); + check(f.receiver_text == "r", + "r.render() receiver_text must be \"r\""); + check(f.qualified_target == "r.render", + "r.render() qualified_target must be \"r.render\""); + printf(" [PASS] r.render(): receiver=%s qualified=%s\n", + f.receiver_text.c_str(), f.qualified_target.c_str()); + } + + // ── Case 3: TS `this.render()` → receiver_type = class ──────── + { + CallFact f = getCallFact(db, pid, "render", "timeline.ts"); + check(f.found, "this.render() call record missing"); + check(f.receiver_text == "this", + "this.render() receiver_text must be \"this\""); + check(f.receiver_type == "Timeline", + "this.render() receiver_type must be the enclosing class " + "\"Timeline\""); + printf(" [PASS] this.render(): receiver=%s type=%s\n", + f.receiver_text.c_str(), f.receiver_type.c_str()); + } + + // ── Case 4: bare call must not fabricate a receiver ─────────── + { + CallFact f = getCallFact(db, pid, "standalone", "main.js"); + check(f.found, "standalone() call record missing"); + check(f.qualified_target.empty(), + "bare call must not set qualified_target"); + check(f.receiver_text.empty(), + "bare call must not set receiver_text"); + check(f.receiver_type.empty(), + "bare call must not set receiver_type"); + check(f.import_alias.empty(), + "bare call must not set import_alias"); + printf(" [PASS] standalone(): no fabricated receiver\n"); + } + + sqlite3_close(db); + engine_destroy(g_engine); + g_engine = nullptr; + std::filesystem::remove_all(root); + printf("\nAll JS/TS call-fact tests passed.\n"); + return 0; +} diff --git a/engine/tests/test_js_visitor.cpp b/engine/tests/test_js_visitor.cpp index e15be8e..477ea87 100644 --- a/engine/tests/test_js_visitor.cpp +++ b/engine/tests/test_js_visitor.cpp @@ -13,70 +13,51 @@ #include "../src/ir/translators/js_visitor.h" #include "../src/ir/semantic_unit.h" +#include "grammar_loader.h" #include #include #include -#include #include #include // tree-sitter #include -// Load the JavaScript grammar dynamically +// The grammar is compiled into astgraph_engine, which this test links, so the +// language comes from the library rather than from a .so on disk +// (engine/tests/grammar_loader.h explains what the old lookup cost). static const TSLanguage *load_js_language() { - // Try GRAMMARS_DIR env, then default paths - const char *dirs[] = { - getenv("GRAMMARS_DIR"), - "../grammars", - "grammars", - "～/code/cppCode/CodeScope/grammars", - }; - for (auto d : dirs) { - if (!d) - continue; - std::string path = std::string(d) + - "/tree-sitter-javascript.so"; - void *handle = dlopen(path.c_str(), - RTLD_LAZY | RTLD_LOCAL); - if (handle) { - auto *fn = reinterpret_cast( - dlsym(handle, "tree_sitter_javascript")); - if (fn) - return fn(); - dlclose(handle); - } - } - return nullptr; + return testGrammar("javascript"); } static int tests_run = 0; static int tests_passed = 0; -#define CHECK(cond, msg) \ - do { \ - tests_run++; \ - if (!(cond)) { \ - fprintf(stderr, "FAIL [%d]: %s\n", tests_run, msg); \ - exit(1); \ - } \ - tests_passed++; \ +#define CHECK(cond, msg) \ + do { \ + tests_run++; \ + if (!(cond)) { \ + fprintf(stderr, "FAIL [%d]: %s\n", tests_run, msg); \ + exit(1); \ + } \ + tests_passed++; \ } while (0) -#define CHECK_EQ(a, b, msg) \ - do { \ - tests_run++; \ - if ((a) != (b)) { \ - fprintf(stderr, "FAIL [%d]: %s — expected %llu, " \ - "got %llu\n", \ - tests_run, msg, \ - static_cast(b), \ - static_cast(a)); \ - exit(1); \ - } \ - tests_passed++; \ +#define CHECK_EQ(a, b, msg) \ + do { \ + tests_run++; \ + if ((a) != (b)) { \ + fprintf(stderr, \ + "FAIL [%d]: %s — expected %llu, " \ + "got %llu\n", \ + tests_run, msg, \ + static_cast(b), \ + static_cast(a)); \ + exit(1); \ + } \ + tests_passed++; \ } while (0) // ── Helpers ─────────────────────────────────────────────────── @@ -86,8 +67,7 @@ static TSTree *parse(const char *source, const TSLanguage *lang) TSParser *parser = ts_parser_new(); ts_parser_set_language(parser, lang); TSTree *tree = ts_parser_parse_string( - parser, nullptr, source, - static_cast(strlen(source))); + parser, nullptr, source, static_cast(strlen(source))); ts_parser_delete(parser); return tree; } @@ -100,7 +80,7 @@ static size_t countKind(const ir::SemanticUnit &unit, ir::RecordKind kind) // Find first record by name static const ir::Record *findByName(const ir::SemanticUnit &unit, - const std::string &name) + const std::string &name) { size_t idx = unit.findRecordByName(name); if (idx == SIZE_MAX) @@ -136,11 +116,9 @@ static void test_simple_function() const ir::Record *add = findByName(*unit, "add"); CHECK(add != nullptr, "function 'add' found"); - CHECK(add->kind == ir::RecordKind::Function, - "record kind is Function"); + CHECK(add->kind == ir::RecordKind::Function, "record kind is Function"); CHECK(add->loc.start_row == 0, "add starts at row 0"); - CHECK(add->file_path == "/test/add.js", - "file path preserved"); + CHECK(add->file_path == "/test/add.js", "file path preserved"); ts_tree_delete(tree); delete unit; @@ -183,8 +161,7 @@ static void test_call_expression() // Verify containment: call should be a child of main const ir::Record *main_fn = findByName(*unit, "main"); CHECK(main_fn != nullptr, "main function found"); - CHECK_EQ(call->parent_id, main_fn->id, - "call is child of main"); + CHECK_EQ(call->parent_id, main_fn->id, "call is child of main"); ts_tree_delete(tree); delete unit; @@ -218,10 +195,8 @@ static void test_variable_declaration() // Both should be children of main const ir::Record *main_fn = findByName(*unit, "main"); CHECK(main_fn != nullptr, "main function found"); - CHECK_EQ(x->parent_id, main_fn->id, - "x is child of main"); - CHECK_EQ(y->parent_id, main_fn->id, - "y is child of main"); + CHECK_EQ(x->parent_id, main_fn->id, "x is child of main"); + CHECK_EQ(y->parent_id, main_fn->id, "y is child of main"); ts_tree_delete(tree); delete unit; @@ -268,8 +243,7 @@ static void test_multi_function() // r is a variable child of main const ir::Record *r = findByName(*unit, "r"); CHECK(r != nullptr, "variable 'r' found"); - CHECK_EQ(r->parent_id, main_fn->id, - "r is child of main"); + CHECK_EQ(r->parent_id, main_fn->id, "r is child of main"); ts_tree_delete(tree); delete unit; @@ -300,10 +274,8 @@ static void test_class_method() const ir::Record *add = findByName(*unit, "add"); CHECK(add != nullptr, "method 'add' found"); - CHECK(add->kind == ir::RecordKind::Method, - "add is a Method record"); - CHECK_EQ(add->parent_id, calc->id, - "add is child of Calculator"); + CHECK(add->kind == ir::RecordKind::Method, "add is a Method record"); + CHECK_EQ(add->parent_id, calc->id, "add is child of Calculator"); ts_tree_delete(tree); delete unit; @@ -347,8 +319,7 @@ static void test_import_export() } } CHECK(exp != nullptr, "export record found"); - CHECK_EQ(bar->parent_id, exp->id, - "bar is child of export record"); + CHECK_EQ(bar->parent_id, exp->id, "bar is child of export record"); ts_tree_delete(tree); delete unit; @@ -457,7 +428,7 @@ int main() test_member_expression(); test_empty_file(); - printf("\n=== js_visitor test passed (%d/%d) ===\n", - tests_passed, tests_run); + printf("\n=== js_visitor test passed (%d/%d) ===\n", tests_passed, + tests_run); return 0; } diff --git a/engine/tests/test_json_writer.cpp b/engine/tests/test_json_writer.cpp new file mode 100644 index 0000000..0f81eb5 --- /dev/null +++ b/engine/tests/test_json_writer.cpp @@ -0,0 +1,287 @@ +// test_json_writer: verify util::JsonWriter produces well-formed, correctly +// escaped JSON and fails closed on structural misuse. +// +// Why this exists: the engine previously built JSON by string concatenation +// in ~35 files, where a missing jsonEscape() call silently emitted invalid +// JSON and a missing comma produced unparseable output. JsonWriter makes +// escaping unconditional and separators automatic, so the negative controls +// below (unbalanced containers, values without keys, second top-level value) +// matter as much as the positive ones. + +#include "../src/util/json_writer.h" + +#include +#include +#include +#include +#include +#include +#include + +static int g_failures = 0; + +#define CHECK(cond, label) \ + do { \ + if (!(cond)) { \ + fprintf(stderr, "FAIL: %s (%s:%d)\n", label, __FILE__, \ + __LINE__); \ + g_failures++; \ + } \ + } while (0) + +#define CHECK_EQ(actual, expected, label) \ + do { \ + const std::string a_ = (actual); \ + const std::string e_ = (expected); \ + if (a_ != e_) { \ + fprintf(stderr, \ + "FAIL: %s\n expected: %s\n actual: %s\n", \ + label, e_.c_str(), a_.c_str()); \ + g_failures++; \ + } \ + } while (0) + +static void test_scalars() +{ + util::JsonWriter w; + w.value(42); + CHECK_EQ(w.str(), "42", "int value"); + CHECK(w.ok(), "int value ok"); + + w.reset(); + w.value(true); + CHECK_EQ(w.str(), "true", "bool true"); + + w.reset(); + w.value(false); + CHECK_EQ(w.str(), "false", "bool false"); + + w.reset(); + w.value(std::string("hi")); + CHECK_EQ(w.str(), "\"hi\"", "string value"); + + w.reset(); + w.nullValue(); + CHECK_EQ(w.str(), "null", "null value"); + + w.reset(); + w.value(static_cast(9223372036854775807LL)); + CHECK_EQ(w.str(), "9223372036854775807", "int64 max"); + + w.reset(); + w.value(static_cast(18446744073709551615ULL)); + CHECK_EQ(w.str(), "18446744073709551615", "uint64 max"); +} + +static void test_escaping() +{ + util::JsonWriter w; + w.value(std::string("a\"b\\c\nd\te\rc\x01")); + CHECK_EQ(w.str(), "\"a\\\"b\\\\c\\nd\\te\\rc\\u0001\"", + "string escaping"); +} + +static void test_object_and_array() +{ + util::JsonWriter w; + w.beginObject(); + w.key("name").value(std::string("a\"b")); + w.key("count").value(3); + w.key("ok").value(true); + w.key("ratio").value(0.5); + w.key("none").nullValue(); + w.key("items").beginArray(); + w.value(1); + w.value(2); + w.endArray(); + w.key("empty").beginObject().endObject(); + w.key("emptya").beginArray().endArray(); + w.endObject(); + CHECK(w.ok(), "nested build ok"); + CHECK_EQ(w.str(), + "{\"name\":\"a\\\"b\",\"count\":3,\"ok\":true," + "\"ratio\":0.5,\"none\":null,\"items\":[1,2]," + "\"empty\":{},\"emptya\":[]}", + "nested object/array"); +} + +static void test_top_level_array_of_objects() +{ + util::JsonWriter w; + w.beginArray(); + w.beginObject().key("id").value(1).endObject(); + w.beginObject().key("id").value(2).endObject(); + w.endArray(); + CHECK_EQ(w.str(), "[{\"id\":1},{\"id\":2}]", "array of objects"); +} + +static void test_double_matches_ostringstream() +{ + // Byte-compatibility with the string-concatenation code this replaces. + const double values[] = { 0.1, 1.0, + -0.5, 3.14159265358979, + 1000000.0, 0.000123456789, + 1e20 }; + for (double v : values) { + std::ostringstream ref; + ref.imbue(std::locale::classic()); + ref << v; + util::JsonWriter w; + w.value(v); + CHECK_EQ(w.str(), ref.str(), + "double format matches ostringstream"); + } +} + +static void test_non_finite_double_is_null() +{ + util::JsonWriter w; + w.value(std::nan("")); + CHECK_EQ(w.str(), "null", "NaN -> null"); + w.reset(); + w.value(std::numeric_limits::infinity()); + CHECK_EQ(w.str(), "null", "Inf -> null"); +} + +static void test_raw() +{ + util::JsonWriter w; + w.beginObject(); + w.key("nested").raw("{\"already\":\"json\"}"); + w.endObject(); + CHECK_EQ(w.str(), "{\"nested\":{\"already\":\"json\"}}", + "raw fragment"); +} + +static void test_structural_errors() +{ + { + util::JsonWriter w; + w.value(1); // top-level value + w.value(2); // second top-level value: error + CHECK(!w.ok(), "second top-level value is an error"); + } + { + util::JsonWriter w; + w.beginObject(); + w.value(1); // value without a key + CHECK(!w.ok(), "value without key is an error"); + } + { + util::JsonWriter w; + w.beginObject(); + w.key("a"); + w.key("b"); // two keys without a value between + CHECK(!w.ok(), "double key is an error"); + } + { + util::JsonWriter w; + w.beginObject(); + w.key("a"); // key left without a value + w.endObject(); + CHECK(!w.ok(), "object closed with pending key is an error"); + } + { + util::JsonWriter w; + w.beginObject(); + w.endArray(); // mismatched close + CHECK(!w.ok(), "endArray on object is an error"); + } + { + util::JsonWriter w; + w.key("a"); // key outside any object + CHECK(!w.ok(), "key outside object is an error"); + } +} + +static void test_frozen_escaper_contract() +{ + // Frozen bytes, not a round-trip through the function under test: this is + // the migration contract (the 35 hand-rolled escapers it replaced emitted + // exactly these sequences), so it must fail if jsonEscapeString changes. + struct Case { + const char *in; + const char *out; + }; + const Case cases[] = { + { "plain", "plain" }, + { "a\"b", "a\\\"b" }, + { "a\\b", "a\\\\b" }, + { "l1\nl2", "l1\\nl2" }, + { "t\tr", "t\\tr" }, + { "\b\f", "\\u0008\\u000c" }, + { "\x01\x1f", "\\u0001\\u001f" }, + { "\x7f", "\x7f" }, + { "\xE4\xBD\xA0", "\xE4\xBD\xA0" }, + { "\xFF\x80", "\xFF\x80" }, + }; + for (const Case &c : cases) { + CHECK_EQ("\"" + util::jsonEscapeString(std::string(c.in)) + + "\"", + "\"" + std::string(c.out) + "\"", + "jsonEscapeString frozen bytes"); + } +} + +static void test_key_escaping() +{ + util::JsonWriter w; + w.beginObject(); + w.key("a\"b\\c").value(1); + w.endObject(); + CHECK_EQ(w.str(), "{\"a\\\"b\\\\c\":1}", "keys are escaped too"); +} + +static void test_fail_closed_stops_emitting() +{ + util::JsonWriter w; + w.beginObject(); + w.key("a"); + w.endObject(); // pending key: error + CHECK(!w.ok(), "pending key is an error"); + const std::string at_error = w.str(); + w.key("b").value(1); // further use must not emit anything + CHECK_EQ(w.str(), at_error, "writer stops emitting after an error"); + CHECK(!w.error().empty(), "error() reports the first message"); +} + +static void test_error_envelopes() +{ + // The envelopes are the one JSON the Rust server parses on every failure, + // and the message comes from e.what(), i.e. it can contain anything. + CHECK_EQ(util::errorEnvelope("ffi", "engine_x", "plain"), + "{\"error\":\"[module=ffi, method=engine_x] plain\"}", + "error envelope shape"); + CHECK_EQ( + util::errorEnvelope("ffi", "engine_x", "a\"b\\c\nd"), + "{\"error\":\"[module=ffi, method=engine_x] a\\\"b\\\\c\\nd\"}", + "error envelope escapes the message"); + CHECK_EQ(util::okFalseEnvelope("ffi", "engine_y", "not initialized"), + "{\"ok\":false,\"error\":\"[module=ffi, method=engine_y] " + "not initialized\"}", + "ok:false envelope shape"); +} + +int main() +{ + test_scalars(); + test_escaping(); + test_object_and_array(); + test_top_level_array_of_objects(); + test_double_matches_ostringstream(); + test_non_finite_double_is_null(); + test_raw(); + test_structural_errors(); + test_frozen_escaper_contract(); + test_key_escaping(); + test_fail_closed_stops_emitting(); + test_error_envelopes(); + + if (g_failures != 0) { + fprintf(stderr, "\n=== test_json_writer: %d FAILURE(S) ===\n", + g_failures); + return 1; + } + fprintf(stderr, "=== test_json_writer: all checks passed ===\n"); + return 0; +} diff --git a/engine/tests/test_lsp_framing.cpp b/engine/tests/test_lsp_framing.cpp new file mode 100644 index 0000000..4e6eb4f --- /dev/null +++ b/engine/tests/test_lsp_framing.cpp @@ -0,0 +1,123 @@ +// Regression tests for LSP message framing and response matching. +// +// The defects these cover are invisible through the public API: they need a +// server that interleaves notifications with responses, or that splits a +// message across reads. The parsing rules are pure string handling, so they are +// driven directly. + +#include "../src/lsp/lsp_framing.h" + +#include "test_check.h" +#include +#include + +int main() +{ + // ── Test 1: a complete message is taken out of the buffer ─────── + { + const std::string response = + "{\"jsonrpc\":\"2.0\",\"id\":7,\"result\":{}}"; + std::string buffer = wrapLspFramed(response); + std::string body; + CHECK(takeNextLspMessage(buffer, body) == LspFraming::Message); + CHECK(body == response); + CHECK(buffer.empty()); + CHECK(isResponseFor(body, 7)); + // A response to a different request is not this call's answer. + CHECK(!isResponseFor(body, 8)); + printf(" [PASS] framing: complete message extracted and matched\n"); + } + + // ── Test 2: a message split across reads ──────────────────────── + { + const std::string response = + "{\"jsonrpc\":\"2.0\",\"id\":3,\"result\":{\"uri\":\"a\"}}"; + const std::string framed = wrapLspFramed(response); + // Header only, then part of the body, then the rest. + std::string buffer = framed.substr(0, 20); + std::string body; + CHECK(takeNextLspMessage(buffer, body) == LspFraming::NeedMore); + buffer += framed.substr(20, 10); + CHECK(takeNextLspMessage(buffer, body) == LspFraming::NeedMore); + buffer += framed.substr(30); + CHECK(takeNextLspMessage(buffer, body) == LspFraming::Message); + CHECK(body == response); + CHECK(buffer.empty()); + printf(" [PASS] framing: message reassembled across reads\n"); + } + + // ── Test 3: notification before the response (the reported bug) ── + // The old readResponse() returned the FIRST complete message, so this + // window/logMessage became the "definition result" the caller acted on, + // and the buffered bytes that followed it were discarded. + { + const std::string notification = + "{\"jsonrpc\":\"2.0\",\"method\":\"window/logMessage\"," + "\"params\":{\"type\":3,\"message\":\"indexing\"}}"; + const std::string response = + "{\"jsonrpc\":\"2.0\",\"id\":11,\"result\":{\"ok\":true}}"; + std::string buffer = + wrapLspFramed(notification) + wrapLspFramed(response); + std::string body; + + // The notification is framed correctly but is not an answer. + CHECK(takeNextLspMessage(buffer, body) == LspFraming::Message); + CHECK(body == notification); + CHECK(!isResponseFor(body, 11)); + + // The response that arrived in the same read is still there. + CHECK(takeNextLspMessage(buffer, body) == LspFraming::Message); + CHECK(body == response); + CHECK(isResponseFor(body, 11)); + CHECK(buffer.empty()); + printf(" [PASS] framing: notification skipped, response kept\n"); + } + + // ── Test 4: a notification alone never matches ────────────────── + { + std::string buffer = wrapLspFramed( + "{\"jsonrpc\":\"2.0\",\"method\":\"textDocument/" + "publishDiagnostics\",\"params\":{\"uri\":\"f\"}}"); + std::string body; + CHECK(takeNextLspMessage(buffer, body) == LspFraming::Message); + CHECK(!isResponseFor(body, 1)); + printf(" [PASS] framing: notification is never a response\n"); + } + + // ── Test 5: an unusable Content-Length cannot stall the caller ── + // Reporting a discarded frame as "need more bytes" would leave a caller + // waiting for data that is not coming, while a complete answer sits in the + // buffer behind the junk. + { + std::string buffer = "Content-Length: 0\r\n\r\n"; + const std::string tail = wrapLspFramed( + "{\"jsonrpc\":\"2.0\",\"id\":2,\"result\":null}"); + buffer += tail; + std::string body; + // The zero-length header is reported as consumed, not as "need more"... + CHECK(takeNextLspMessage(buffer, body) == LspFraming::Skipped); + CHECK(buffer == tail); + // ...so the message behind it is readable immediately. + CHECK(takeNextLspMessage(buffer, body) == LspFraming::Message); + CHECK(isResponseFor(body, 2)); + CHECK(buffer.empty()); + + std::string only_bad = "Content-Length: 0\r\n\r\n"; + CHECK(takeNextLspMessage(only_bad, body) == + LspFraming::Skipped); + CHECK(only_bad.empty()); + printf(" [PASS] framing: unusable header dropped, next message read\n"); + } + + // ── Test 6: a non-numeric id is not a response id ─────────────── + { + CHECK(!isResponseFor("{\"jsonrpc\":\"2.0\",\"id\":\"abc\"}", + 1)); + CHECK(!isResponseFor("{\"jsonrpc\":\"2.0\",\"params\":{}}", 1)); + CHECK(isResponseFor("{\"jsonrpc\":\"2.0\",\"id\": 42 }", 42)); + printf(" [PASS] framing: non-numeric / missing ids rejected\n"); + } + + printf("=== test_lsp_framing PASSED ===\n"); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_max_file_size.cpp b/engine/tests/test_max_file_size.cpp new file mode 100644 index 0000000..4259fd3 --- /dev/null +++ b/engine/tests/test_max_file_size.cpp @@ -0,0 +1,127 @@ +/** + * test_max_file_size: the documented CODESCOPE_MAX_FILE_SIZE default. + * + * README §3/§9 promise a 5 MB (5242880 byte) default cap, and files above it + * are skipped silently. The cap is a hard-coded constant + * (kMaxFileSize in engine_index_sched.h), so nothing but a test keeps the + * documentation and the code from drifting apart again + * (2026-09-27 review D2-1: the README claimed 10 MB in §3 and + * "(unset)" in §9 while the code used 5 MB). + * + * Cases: + * 1. Default (env unset): a 6 MB source file is skipped, a small one indexed. + * 2. CODESCOPE_MAX_FILE_SIZE raised: the same big file is indexed. + */ + +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include "test_engine_handle.h" + +namespace fs = std::filesystem; + +static const char *kProjDir = "/tmp/test_max_file_size_proj"; +static const char *kDbPath = "/tmp/test_max_file_size.db"; + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "\nFAIL: %s\n", msg); + std::error_code ec; + fs::remove_all(kProjDir, ec); + fs::remove(kDbPath, ec); + exit(1); + } +} + +static void write_file(const std::string &path, const std::string &content) +{ + FILE *f = fopen(path.c_str(), "w"); + check(f != nullptr, "fopen"); + fwrite(content.data(), 1, content.size(), f); + fclose(f); +} + +/// A valid C file just over `bytes` in size: one large block comment plus a +/// tiny function. Comment-heavy keeps the parse cheap (the point is the size +/// gate, not parse performance), unlike thousands of duplicate declarations. +static std::string oversized_source(size_t bytes) +{ + std::string code = "/*"; + code.append(bytes, 'x'); + code += "*/\nint big_symbol(void) { return 0; }\n"; + return code; +} + +/// Number of indexed source files reported by project_overview. +static int indexed_file_count(uint64_t pid) +{ + char *overview = engine_get_project_overview(g_engine, pid); + check(overview != nullptr, "project_overview result"); + const char *key = strstr(overview, "\"total_files\":"); + int files = -1; + if (key != nullptr) + files = atoi(key + strlen("\"total_files\":")); + engine_free_string(overview); + return files; +} + +/// Index kProjDir into `db_path` and return the reported file count. +static int index_and_count(const char *db_path) +{ + unlink(db_path); + g_engine = engine_create(db_path); + check(g_engine != nullptr, "engine_init"); + uint64_t pid = + engine_create_project(g_engine, kProjDir, "max-file-size"); + check(pid > 0, "create_project"); + + char *idx = engine_index_project(g_engine, pid, kProjDir, NULL); + check(idx != nullptr, "index_project result"); + check(strstr(idx, "\"ok\":true") != nullptr, "index_project ok"); + engine_free_string(idx); + + int files = indexed_file_count(pid); + engine_destroy(g_engine); + g_engine = nullptr; + return files; +} + +int main() +{ + std::error_code ec; + fs::remove_all(kProjDir, ec); + fs::create_directories(kProjDir); + fs::remove(kDbPath, ec); + + write_file(std::string(kProjDir) + "/small.c", + "int small_symbol(void) { return 0; }\n"); + // 6 MB: above the 5 MB (5242880 byte) default, below the raised cap below. + write_file(std::string(kProjDir) + "/big.c", + oversized_source(6u * 1024 * 1024)); + + // ── 1. Default cap (env unset) ─────────────────────────────── + unsetenv("CODESCOPE_MAX_FILE_SIZE"); + const int default_files = index_and_count(kDbPath); + printf("default cap: indexed %d file(s)\n", default_files); + check(default_files == 1, + "default 5 MB cap must index only the small file (big.c skipped)"); + + // ── 2. Raised cap: the same big file is indexed ────────────── + setenv("CODESCOPE_MAX_FILE_SIZE", "10000000", 1); + const int raised_files = index_and_count(kDbPath); + printf("raised cap: indexed %d file(s)\n", raised_files); + check(raised_files == 2, + "CODESCOPE_MAX_FILE_SIZE=10000000 must include the 6 MB file"); + unsetenv("CODESCOPE_MAX_FILE_SIZE"); + + fs::remove_all(kProjDir, ec); + fs::remove(kDbPath, ec); + printf("\n=== max file size test passed ===\n"); + return 0; +} diff --git a/engine/tests/test_membulk.cpp b/engine/tests/test_membulk.cpp index 2b4bc8b..c0bd69b 100644 --- a/engine/tests/test_membulk.cpp +++ b/engine/tests/test_membulk.cpp @@ -21,16 +21,19 @@ using namespace store; -static const char *tmpDir() { +static const char *tmpDir() +{ const char *d = getenv("TMPDIR"); return d ? d : "/tmp"; } -static std::string dbPath() { +static std::string dbPath() +{ return std::string(tmpDir()) + "/codescope_test_membulk.db"; } -static int countSemanticRecords(GraphStore &store, uint64_t project_id) { +static int countSemanticRecords(GraphStore &store, uint64_t project_id) +{ sqlite3_stmt *stmt = nullptr; std::string sql = "SELECT COUNT(*) FROM semantic_records WHERE project_id = " + @@ -45,7 +48,8 @@ static int countSemanticRecords(GraphStore &store, uint64_t project_id) { return count; } -static FileResult makeFakeResult(const std::string &path, int records) { +static FileResult makeFakeResult(const std::string &path, int records) +{ FileResult fr; fr.file_path = path; fr.language = "cpp"; @@ -64,19 +68,20 @@ static FileResult makeFakeResult(const std::string &path, int records) { static int g_failures = 0; -#define CHECK(cond, msg) \ - do { \ - if (!(cond)) { \ - fprintf(stderr, "FAIL: %s (%s:%d)\n", msg, __FILE__, \ - __LINE__); \ - ++g_failures; \ - } else { \ - fprintf(stderr, "ok: %s\n", msg); \ - } \ +#define CHECK(cond, msg) \ + do { \ + if (!(cond)) { \ + fprintf(stderr, "FAIL: %s (%s:%d)\n", msg, __FILE__, \ + __LINE__); \ + ++g_failures; \ + } else { \ + fprintf(stderr, "ok: %s\n", msg); \ + } \ } while (0) // Test 1: Empty aggregator flushes successfully and reports size 0. -static void testEmptyFlushReturnsTrue() { +static void testEmptyFlushReturnsTrue() +{ GraphStore store; unlink(dbPath().c_str()); CHECK(store.open(dbPath().c_str()), "open store"); @@ -91,7 +96,8 @@ static void testEmptyFlushReturnsTrue() { // Test 2: Single worker merges 4 FileResult objects, flush persists 4 rows // (1 record each). -static void testSingleWorkerMerge() { +static void testSingleWorkerMerge() +{ GraphStore store; unlink(dbPath().c_str()); CHECK(store.open(dbPath().c_str()), "open store"); @@ -107,12 +113,12 @@ static void testSingleWorkerMerge() { agg.mergeFrom(std::move(local)); CHECK(agg.size() == 4, "single worker size == 4"); CHECK(agg.flush(store, pid, false), "single worker flush returns true"); - CHECK(countSemanticRecords(store, pid) == 4, - "single worker -> 4 rows"); + CHECK(countSemanticRecords(store, pid) == 4, "single worker -> 4 rows"); } // Test 3: 8 threads each merge 16 FileResult objects (128 total). -static void testMultiWorkerConcurrentMerge() { +static void testMultiWorkerConcurrentMerge() +{ GraphStore store; unlink(dbPath().c_str()); CHECK(store.open(dbPath().c_str()), "open store"); @@ -150,7 +156,8 @@ static void testMultiWorkerConcurrentMerge() { // the error string is exposed. We simulate by flushing to a store that has // been closed (handle() becomes invalid) and capture stderr for // "module=store_membulk". -static void testFlushFailureRollsBackAndLogs() { +static void testFlushFailureRollsBackAndLogs() +{ GraphStore store; unlink(dbPath().c_str()); CHECK(store.open(dbPath().c_str()), "open store"); @@ -182,7 +189,8 @@ static void testFlushFailureRollsBackAndLogs() { } } -int main() { +int main() +{ testEmptyFlushReturnsTrue(); testSingleWorkerMerge(); testMultiWorkerConcurrentMerge(); diff --git a/engine/tests/test_membulk_parity.cpp b/engine/tests/test_membulk_parity.cpp index 01c9e30..9511d6d 100644 --- a/engine/tests/test_membulk_parity.cpp +++ b/engine/tests/test_membulk_parity.cpp @@ -14,10 +14,12 @@ #include #include "../include/engine.h" +#include "test_engine_handle.h" static std::string g_dir; -static void writeFile(const std::string &name, int id) { +static void writeFile(const std::string &name, int id) +{ std::string path = g_dir + "/" + name; FILE *f = fopen(path.c_str(), "w"); if (!f) { @@ -42,7 +44,8 @@ struct Counts { long nodes = 0, edges = 0, records = 0; }; -static Counts indexAndCount(bool force_streaming) { +static Counts indexAndCount(bool force_streaming) +{ if (force_streaming) setenv("CODESCOPE_FORCE_STREAMING", "1", 1); else @@ -51,16 +54,17 @@ static Counts indexAndCount(bool force_streaming) { std::string db = g_dir + "/parity.db"; unlink(db.c_str()); - if (engine_init(db.c_str()) != 0) { + g_engine = engine_create(db.c_str()); + if (!g_engine) { fprintf(stderr, "FAIL: engine_init\n"); exit(1); } - uint64_t pid = engine_create_project(g_dir.c_str(), "parity"); + uint64_t pid = engine_create_project(g_engine, g_dir.c_str(), "parity"); if (pid == 0) { fprintf(stderr, "FAIL: create_project\n"); exit(1); } - char *res = engine_index_project(pid, g_dir.c_str(), ""); + char *res = engine_index_project(g_engine, pid, g_dir.c_str(), ""); if (!res || strstr(res, "\"ok\":true") == nullptr) { fprintf(stderr, "FAIL: index_project: %s\n", res ? res : "(null)"); @@ -69,7 +73,7 @@ static Counts indexAndCount(bool force_streaming) { engine_free_string(res); Counts c; - char *stats = engine_get_graph_stats(pid); + char *stats = engine_get_graph_stats(g_engine, pid); if (stats) { // stats JSON contains total_nodes / total_edges. // We instead query the DB directly for robustness. @@ -79,7 +83,7 @@ static Counts indexAndCount(bool force_streaming) { // Re-open the DB file directly to count rows (engine API does not // expose raw counts; use sqlite3 via the engine's own handle is not // available here, so parse graph_stats JSON fields). - char *s2 = engine_get_graph_stats(pid); + char *s2 = engine_get_graph_stats(g_engine, pid); // graph_stats returns {"total_nodes":N,"total_edges":M,...} auto extract = [](const char *json, const char *key) -> long { std::string k = std::string("\"") + key + "\":"; @@ -100,11 +104,13 @@ static Counts indexAndCount(bool force_streaming) { // rely on nodes+edges equality which fully exercises both paths. c.records = c.nodes; - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; return c; } -int main() { +int main() +{ g_dir = "/tmp/codescope_membulk_parity"; std::filesystem::remove_all(g_dir); std::filesystem::create_directories(g_dir); @@ -128,9 +134,8 @@ int main() { if (membulk.nodes != streaming.nodes || membulk.edges != streaming.edges) { - fprintf(stderr, - "FAIL: parity mismatch between membulk and " - "streaming paths\n"); + fprintf(stderr, "FAIL: parity mismatch between membulk and " + "streaming paths\n"); return 1; } if (membulk.nodes <= 0) { diff --git a/engine/tests/test_metrics_persist.cpp b/engine/tests/test_metrics_persist.cpp new file mode 100644 index 0000000..4c56f77 --- /dev/null +++ b/engine/tests/test_metrics_persist.cpp @@ -0,0 +1,359 @@ +// test_metrics_persist.cpp — metric columns must survive a graph rebuild. +// +// Regression guard for the 2026-10-06 findings (verified on nine languages' +// real projects). Metric columns (cyclomatic / cognitive / nesting_depth / +// lines / ...) are derived data: parse workers stage them in _staged_metrics +// and resolveStagedMetrics() copies them onto entity rows. buildGraph(), in +// contrast, deletes the entity rows of every file it rebuilds and re-inserts +// them from semantic_records with a column list that has no metrics, so any +// rebuild after the metrics were resolved silently zeroed them: +// +// * parallel path (index-parallel): each module worker resolved its metrics +// correctly, but the post-index pass rebuilds the MERGED database, so every +// project measured 0 (one worker row was cyclomatic=4/lines=15, its merged +// counterpart 0/0), and analysis_progress.metrics / metrics_ready / every +// complexity field in tool output followed it to 0. +// * file-list path (force-index, force_index_files): metrics are staged but +// resolveStagedMetrics() is never called on that path — a goagent database +// kept 15160 unresolved staging rows and published 0 metrics. +// +// The assertions below are deliberately about VALUES, not just "some metric is +// non-zero": the rebuild must preserve the numbers the first index produced. +// +// Fix under test: GraphStore::snapshotMetricsForRebuild / +// applyMetricsAfterRebuild (engine/src/store/store_metrics.cpp), called from +// buildGraph around the delete/insert pair. + +#include "test_check.h" + +#include "../include/engine.h" +#include "test_engine_handle.h" + +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; + +namespace +{ + +/// Fixture root; both the sources and the database live here so a failed run +/// leaves everything behind for inspection. +const char *kProjDir = "/tmp/test_metrics_persist"; +const char *kDbPath = "/tmp/test_metrics_persist.db"; + +/// The fixture function must have a non-trivial cyclomatic value: two `if`s and +/// a loop give a count well above 1, so "cyclomatic == 0" can only mean the +/// metrics were lost, never that they were never computed. +const char *kMainCpp = R"(#include "helper.h" + +int classify(int value) { + int total = 0; + if (value > 0) { + total += value; + } + if (value % 2 == 0) { + for (int i = 0; i < value; ++i) { + total += helper(i); + } + } + return total; +} + +int main() { + return classify(4); +} +)"; + +const char *kHelperH = R"(#ifndef HELPER_H +#define HELPER_H +int helper(int x); +#endif +)"; + +const char *kHelperCpp = R"(#include "helper.h" + +int helper(int x) { + if (x < 0) { + return -x; + } + return x + 1; +} +)"; + +/// Write one fixture file, creating the fixture root on first use. +/// @param relative_path Path below the fixture root. +/// @param content File body. +/// @return true when the file was written. +bool writeFixture(const std::string &relative_path, const char *content) +{ + std::error_code ec; + fs::create_directories(kProjDir, ec); + const std::string path = std::string(kProjDir) + "/" + relative_path; + FILE *f = fopen(path.c_str(), "w"); + if (!f) + return false; + const size_t len = strlen(content); + const bool ok = fwrite(content, 1, len, f) == len; + fclose(f); + return ok; +} + +/// Remove the database and every SQLite sidecar so each run starts clean. +void removeDb() +{ + std::error_code ec; + fs::remove(kDbPath, ec); + fs::remove(std::string(kDbPath) + "-wal", ec); + fs::remove(std::string(kDbPath) + "-shm", ec); +} + +/// Open the engine's database on a second connection. +/// +/// The engine holds its own connection open, so this one needs a busy timeout +/// rather than an exclusive lock (same approach as test_metrics_readiness). +/// @return Open handle, or nullptr. +sqlite3 *openDbDirect() +{ + sqlite3 *db = nullptr; + if (sqlite3_open_v2(kDbPath, &db, + SQLITE_OPEN_READWRITE | SQLITE_OPEN_URI, + nullptr) != SQLITE_OK) { + if (db) + sqlite3_close(db); + return nullptr; + } + sqlite3_busy_timeout(db, 5000); + return db; +} + +/// Run a single-value integer query. +/// @param db Open direct connection. +/// @param sql Statement returning one integer. +/// @return First column of the first row, or -1 when the query fails. +long long scalar(sqlite3 *db, const char *sql) +{ + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) != SQLITE_OK) + return -1; + long long value = -1; + if (sqlite3_step(stmt) == SQLITE_ROW) + value = sqlite3_column_int64(stmt, 0); + sqlite3_finalize(stmt); + return value; +} + +/// `project_readiness.metrics_ready` for the only project in this database. +/// It is the flag the readiness reports are built from, and it must agree with +/// the metric rows those same reports count — the mismatch users saw as +/// `ready_features.metrics: false` beside real complexity numbers. +long long metricsReadyFlag(sqlite3 *db) +{ + return scalar( + db, + "SELECT COALESCE((SELECT metrics_ready FROM project_readiness" + " WHERE project_id = 1), -1)"); +} + +/// Run a statement that returns no rows (e.g. a DELETE). +/// @param db Open direct connection. +/// @param sql Statement to execute. +/// @return true when it completed. +bool execDirect(sqlite3 *db, const char *sql) +{ + return sqlite3_exec(db, sql, nullptr, nullptr, nullptr) == SQLITE_OK; +} + +/// Number of function/method entities carrying a resolved cyclomatic value. +/// This is the probe every readiness report uses +/// (`analysis_progress.metrics`, `metrics_ready`). +long long resolvedMetricRows(sqlite3 *db) +{ + return scalar( + db, + "SELECT COUNT(*) FROM entity WHERE kind IN (0,1) AND cyclomatic > 0"); +} + +/// Highest cyclomatic value in the database — the value-preservation probe. +long long maxCyclomatic(sqlite3 *db) +{ + return scalar(db, "SELECT COALESCE(MAX(cyclomatic), 0) FROM entity"); +} + +/// Rows left in the staging table. The fix consumes them, so a non-zero count +/// after an index run means a path that never resolves them was taken. +long long stagedRows(sqlite3 *db) +{ + return scalar(db, "SELECT COUNT(*) FROM _staged_metrics"); +} + +/// Free an engine string when it is non-null. +void freeIfPresent(char *json) +{ + if (json) + engine_free_string(json); +} + +} // namespace + +int main() +{ + printf("=== test_metrics_persist (metric columns across a rebuild) ===\n"); + removeDb(); + { + std::error_code ec; + fs::remove_all(kProjDir, ec); + } + if (!writeFixture("main.cpp", kMainCpp) || + !writeFixture("helper.h", kHelperH) || + !writeFixture("helper.cpp", kHelperCpp)) { + fprintf(stderr, "FAIL: could not write the fixture tree\n"); + return 1; + } + + g_engine = engine_create(kDbPath); + CHECK_MSG(g_engine != nullptr, + "engine instance must open its database"); + if (!g_engine) { + printf("\n=== test_metrics_persist FAILED ===\n"); + return 1; + } + + const uint64_t project_id = + engine_create_project(g_engine, kProjDir, "metrics_persist"); + CHECK_MSG(project_id > 0, "project row"); + + // ── 1. First index (project path) ─────────────────────────────────── + char *indexed = + engine_index_project(g_engine, project_id, kProjDir, nullptr); + CHECK_MSG(indexed != nullptr, + "engine_index_project returns an envelope"); + freeIfPresent(indexed); + + sqlite3 *db = openDbDirect(); + CHECK_MSG(db != nullptr, "direct database access"); + if (!db) { + engine_destroy(g_engine); + g_engine = nullptr; + printf("\n=== test_metrics_persist FAILED ===\n"); + return 1; + } + + const long long rows_after_first = resolvedMetricRows(db); + const long long max_after_first = maxCyclomatic(db); + CHECK_MSG( + rows_after_first > 0, + "the metrics producer must resolve cyclomatic onto entity rows"); + CHECK_MSG(max_after_first > 1, + "the fixture's branches must produce cyclomatic > 1"); + CHECK_MSG( + metricsReadyFlag(db) == 1, + "metrics_ready must be published wherever the metrics are applied"); + + // ── 2. Rebuild: this is where metrics used to be reset to 0 ───────── + // The fixture must actually change: re-indexing an unchanged tree + // short-circuits, which would make this step assert nothing (the + // falsification run showed the rebuild was never entered until the file + // changed). buildGraph deletes and re-inserts the rows of a changed file, + // which is exactly the step that dropped the metric columns. + const char *kMainCppChanged = R"(#include "helper.h" + +int classify(int value) { + int total = 1; + if (value > 0) { + total += value; + } + if (value % 2 == 0) { + for (int i = 0; i < value; ++i) { + total += helper(i); + } + } + return total; +} + +int main() { + return classify(4); +} +)"; + CHECK_MSG(writeFixture("main.cpp", kMainCppChanged), + "rewrite the fixture so the rebuild is not short-circuited"); + + char *rebuilt = + engine_index_project(g_engine, project_id, kProjDir, nullptr); + CHECK_MSG(rebuilt != nullptr, "second engine_index_project"); + freeIfPresent(rebuilt); + + const long long rows_after_rebuild = resolvedMetricRows(db); + const long long max_after_rebuild = maxCyclomatic(db); + CHECK_MSG( + rows_after_rebuild == rows_after_first, + "a rebuild must keep the same number of resolved metric rows"); + CHECK_MSG( + max_after_rebuild == max_after_first, + "a rebuild must preserve the metric VALUES, not just their presence"); + + // ── 3. File-list path: stages metrics, used to never resolve them ─── + const std::string changed = std::string(kProjDir) + "/helper.cpp"; + const std::string file_list = "[\"" + changed + "\"]"; + char *forced = + engine_index_files(g_engine, project_id, file_list.c_str(), 1); + CHECK_MSG(forced != nullptr, "engine_index_files returns an envelope"); + freeIfPresent(forced); + + CHECK_MSG( + resolvedMetricRows(db) == rows_after_first, + "the file-list path must leave metrics resolved (it used to stage them and never apply them)"); + CHECK_MSG(maxCyclomatic(db) == max_after_first, + "the file-list path must preserve the metric values"); + + // ── 4. Already-resolved database rebuilt with nothing staged ──────── + // This is the parallel path's merged database: the module workers already + // applied their metrics (so entity rows carry values) and consumed their + // staging rows, and the merge does not carry `_staged_metrics` across. The + // post-index pass then rebuilds that database, and buildGraph used to + // re-create every row without metrics and with nothing left to resolve — + // which is why 0 was published for every project checked. + CHECK_MSG(execDirect(db, "DELETE FROM _staged_metrics"), + "clear the staging table to emulate a merged database"); + CHECK_MSG(stagedRows(db) == 0, + "staging table starts empty for this case"); + + char *forced_again = + engine_index_files(g_engine, project_id, file_list.c_str(), 1); + CHECK_MSG(forced_again != nullptr, + "engine_index_files on a resolved database"); + freeIfPresent(forced_again); + + CHECK_MSG( + maxCyclomatic(db) == max_after_first, + "a rebuild with nothing staged must restore the snapshot's metric values"); + CHECK_MSG( + resolvedMetricRows(db) == rows_after_first, + "a rebuild with nothing staged must keep every resolved metric row"); + CHECK_MSG( + metricsReadyFlag(db) == 1, + "metrics_ready must survive a rebuild that preserved the metrics"); + + // ── 5. No staging rows may survive an index run ───────────────────── + // The fix consumes them in buildGraph; a leak here is what the goagent + // database showed as 15160 unresolved rows next to 0 published metrics. + CHECK_MSG(stagedRows(db) == 0, + "staged metric rows must be consumed, never left behind"); + + sqlite3_close(db); + engine_destroy(g_engine); + g_engine = nullptr; + removeDb(); + { + std::error_code ec; + fs::remove_all(kProjDir, ec); + } + + printf("\n=== test_metrics_persist %s ===\n", + checkFailures() ? "FAILED" : "PASSED"); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_metrics_readiness.cpp b/engine/tests/test_metrics_readiness.cpp index 4725026..ab21c11 100644 --- a/engine/tests/test_metrics_readiness.cpp +++ b/engine/tests/test_metrics_readiness.cpp @@ -33,6 +33,7 @@ #include #include #include +#include "test_engine_handle.h" namespace fs = std::filesystem; @@ -227,20 +228,22 @@ int helper(int x) { check(setenv("CODESCOPE_SKIP_ASYNC", "1", 1) == 0, "setenv CODESCOPE_SKIP_ASYNC=1"); - check(engine_init(kDbPath) == 0, "engine_init"); - uint64_t pid = engine_create_project(kProjDir, "metrics_readiness"); + g_engine = engine_create(kDbPath); + check(g_engine != nullptr, "engine_init"); + uint64_t pid = + engine_create_project(g_engine, kProjDir, "metrics_readiness"); check(pid > 0, "create_project"); printf("PASS: project_id=%llu\n", (unsigned long long)pid); // ─── Step 2: index ─────────────────────────────────────────── - char *idx = engine_index_project(pid, kProjDir, NULL); + char *idx = engine_index_project(g_engine, pid, kProjDir, NULL); check(idx != nullptr, "index_project result"); check(strstr(idx, "\"ok\":true") != nullptr, "index_project ok"); printf("PASS: index ok\n"); engine_free_string(idx); // ─── Step 3: run enhance so fts_ready is set ───────────────── - char *enh = engine_enhance_project(pid); + char *enh = engine_enhance_project(g_engine, pid); check(enh != nullptr, "enhance_project result"); check(strstr(enh, "\"status\"") != nullptr, "enhance: has status field"); @@ -251,7 +254,7 @@ int helper(int x) { // The original A20 guard: no hardcoded 0. v0.2.5 additionally expects // metrics_ready > 0 (real cyclomatic resolved onto entity) and // embedding_ready > 0 (DEEP mode wrote n-gram vectors). - char *st = engine_get_enhancement_status(pid); + char *st = engine_get_enhancement_status(g_engine, pid); check(st != nullptr, "enhancement_status result"); printf("PASS: enhancement_status — %s\n", st); @@ -289,7 +292,7 @@ int helper(int x) { // ─── Step 5: engine_get_complexity returns a REAL measurement ── // v0.2.5: complexity is restored — the entity has cyclomatic > 0, so // getComplexityJson returns a real integer, NOT the sunset null marker. - char *cplx = engine_get_complexity(pid, 1); + char *cplx = engine_get_complexity(g_engine, pid, 1); check(cplx != nullptr, "get_complexity result"); printf("PASS: get_complexity — %s\n", cplx); check_contains(cplx, "\"available\":true", @@ -301,7 +304,7 @@ int helper(int x) { engine_free_string(cplx); // ─── Step 6: engine_get_capabilities marks restored capabilities ── - char *caps = engine_get_capabilities(pid); + char *caps = engine_get_capabilities(g_engine, pid); check(caps != nullptr, "get_capabilities result"); printf("PASS: get_capabilities — %s\n", caps); check_contains(caps, @@ -310,8 +313,9 @@ int helper(int x) { check_contains(caps, "\"metrics\":{\"available\":true", "capabilities: metrics available=true (restored)"); // semantic_search: available=true + mode=ngram_hash (restored). - check_contains(caps, "\"semantic_search\":{\"available\":true", - "capabilities: semantic_search available=true (restored)"); + check_contains( + caps, "\"semantic_search\":{\"available\":true", + "capabilities: semantic_search available=true (restored)"); check_contains(caps, "\"mode\":\"ngram_hash\"", "capabilities: semantic_search mode=ngram_hash"); // FTS stays available (exact/prefix search alongside semantic). @@ -329,7 +333,8 @@ int helper(int x) { sqlite3 *db = open_db_direct(); check(db != nullptr, "open_db_direct for readiness check"); int64_t nv = count_node_vectors_direct(db, pid); - check(nv > 0, "DEEP index wrote node_vectors (restored producer)"); + check(nv > 0, + "DEEP index wrote node_vectors (restored producer)"); int vflag = read_vector_ready_flag(db, pid); check(vflag == 1, "vector_ready == 1 when node_vectors has rows (real readiness)"); @@ -343,10 +348,10 @@ int helper(int x) { // ─── Step 8: metrics_ready survives a re-run ───────────────── { - char *enh2 = engine_enhance_project(pid); + char *enh2 = engine_enhance_project(g_engine, pid); check(enh2 != nullptr, "enhance rerun"); engine_free_string(enh2); - char *st2 = engine_get_enhancement_status(pid); + char *st2 = engine_get_enhancement_status(g_engine, pid); check(st2 != nullptr, "status after rerun"); int met2 = -1; sscanf(st2, @@ -362,7 +367,7 @@ int helper(int x) { // ─── Step 9: FTS search still works ────────────────────────── // FTS remains the exact/prefix search path; semantic search is now // additive. Verify FTS still returns results for an exact name. - char *search = engine_unified_search(pid, "helper", 10); + char *search = engine_unified_search(g_engine, pid, "helper", 10); check(search != nullptr, "unified_search result"); printf("PASS: unified_search — %s\n", search); check(strstr(search, "\"results\"") != nullptr || @@ -387,7 +392,7 @@ int helper(int x) { check(nv_after_drop == 0, "staleness: node_vectors == 0 after drop"); - char *st3 = engine_get_enhancement_status(pid); + char *st3 = engine_get_enhancement_status(g_engine, pid); check(st3 != nullptr, "staleness: status after drop"); int emb3 = -1; sscanf(st3, @@ -407,7 +412,7 @@ int helper(int x) { // repopulate node_vectors, proving the producer is idempotent and the // full cycle works: build → ready → drop → 0 → rebuild → ready. { - idx = engine_index_project(pid, kProjDir, NULL); + idx = engine_index_project(g_engine, pid, kProjDir, NULL); check(idx != nullptr, "re-index result"); check(strstr(idx, "\"ok\":true") != nullptr, "re-index ok"); engine_free_string(idx); @@ -426,7 +431,8 @@ int helper(int x) { } // ─── Cleanup ───────────────────────────────────────────────── - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; std::error_code ec2; fs::remove_all(kProjDir, ec2); fs::remove(kDbPath, ec2); diff --git a/engine/tests/test_model_engine.cpp b/engine/tests/test_model_engine.cpp index 02d4f81..8468242 100644 --- a/engine/tests/test_model_engine.cpp +++ b/engine/tests/test_model_engine.cpp @@ -23,7 +23,7 @@ #include "../src/model/plugins/workflow.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -42,11 +42,11 @@ static void insertReadme(store::GraphStore &store, uint64_t project_id, const char *sql = "INSERT INTO document (project_id, type, file_path, " "content) VALUES (?, 0, ?, ?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, file_path.c_str(), -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 3, content.c_str(), -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -65,7 +65,7 @@ static void insertGraphNode(store::GraphStore &store, uint64_t project_id, "is_entry_point) " "VALUES (?, ?, 0, ?, ?, ?, '', '', '', ?, 0, 0, 0, ?, ?, '', 0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_int(stmt, 3, node_type); @@ -74,7 +74,7 @@ static void insertGraphNode(store::GraphStore &store, uint64_t project_id, sqlite3_bind_int(stmt, 6, start_row); sqlite3_bind_text(stmt, 7, file_path, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 8, language, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -90,13 +90,13 @@ static void insertEntity(store::GraphStore &store, uint64_t project_id, "start_col, end_row, end_col) " "VALUES (?,?,?,?,'',?,'',0,0,0,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_int(stmt, 3, kind); sqlite3_bind_text(stmt, 4, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 5, file_path, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -109,11 +109,11 @@ static void insertCallRelation(store::GraphStore &store, uint64_t project_id, const char *sql = "INSERT INTO relation (project_id, source_id, " "target_id, type) VALUES (?,?,?,1)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, source_id); sqlite3_bind_int64(stmt, 3, target_id); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -127,10 +127,10 @@ static void insertModuleScope(store::GraphStore &store, uint64_t project_id, const char *sql = "INSERT INTO scope (project_id, parent_id, kind, " "name, start_row, end_row) VALUES (?,0,1,?,0,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, name, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -143,8 +143,8 @@ static int countRows(store::GraphStore &store, uint64_t project_id, std::string sql = std::string("SELECT COUNT(*) FROM ") + table + " WHERE project_id=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) == + SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); int count = 0; if (sqlite3_step(stmt) == SQLITE_ROW) @@ -176,6 +176,67 @@ int main() "- This module is thread-safe\n" "- Feature: JWT authentication\n"); + // ── Negation / polarity regression (pass 3/5) ───────────────── + // "isn't thread-safe" must NOT become a positive ThreadSafe contract, + // and "cannot thread-safe" must STILL become one ("cannot" is not a + // denial — it merely ends in the letters "not"). Both guards live in + // ContractPlugin's ends_word / ends_suffix split. + // Counts are filtered by source_file: the positive README.md inserted + // above also yields a threadsafe contract, and that row is expected. + { + insertReadme(store, pid, "/tmp/NEG.md", + "The buffer isn't thread-safe under load.\n"); + ModelEngine me_neg(&store); + me_neg.addPlugin(std::make_unique(&store)); + me_neg.runAll(pid); + int neg_count = 0; + { + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2( + store.handle(), + "SELECT COUNT(*) FROM contract WHERE " + "project_id=? AND name='threadsafe' " + "AND source_file='/tmp/NEG.md'", + -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(stmt, 1, + static_cast(pid)); + if (sqlite3_step(stmt) == SQLITE_ROW) + neg_count = sqlite3_column_int(stmt, 0); + sqlite3_finalize(stmt); + } + } + CHECK(neg_count == 0 && + "an occurrence of \"isn't thread-safe\" in NEG.md must " + "not add a positive threadsafe contract"); + printf(" [PASS] ContractPlugin negation (isn't thread-safe)\n"); + + insertReadme(store, pid, "/tmp/CANNOT.md", + "This design cannot thread-safe anything.\n"); + ModelEngine me_can(&store); + me_can.addPlugin(std::make_unique(&store)); + me_can.runAll(pid); + int can_count = 0; + { + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2( + store.handle(), + "SELECT COUNT(*) FROM contract WHERE " + "project_id=? AND name='threadsafe' " + "AND source_file='/tmp/CANNOT.md'", + -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(stmt, 1, + static_cast(pid)); + if (sqlite3_step(stmt) == SQLITE_ROW) + can_count = sqlite3_column_int(stmt, 0); + sqlite3_finalize(stmt); + } + } + CHECK(can_count >= 1 && + "\"cannot thread-safe\" must still produce a threadsafe " + "contract (cannot is not a denial)"); + printf(" [PASS] ContractPlugin non-negation (cannot)\n"); + } + // Insert entities: main() is the entry point, Mutex is the // contract-enforcing entity, Helper is called by main. insertGraphNode(store, pid, 1, 0, "main", "/tmp/main.cpp", "cpp", 1); @@ -196,11 +257,11 @@ int main() me.addPlugin(std::make_unique(&store)); me.addPlugin(std::make_unique(&store)); auto names = me.pluginNames(); - assert(names.size() == 4); - assert(std::string(names[0]) == "Capability"); - assert(std::string(names[1]) == "Contract"); - assert(std::string(names[2]) == "Workflow"); - assert(std::string(names[3]) == "Architecture"); + CHECK(names.size() == 4); + CHECK(std::string(names[0]) == "Capability"); + CHECK(std::string(names[1]) == "Contract"); + CHECK(std::string(names[2]) == "Workflow"); + CHECK(std::string(names[3]) == "Architecture"); printf("Test 1 (plugin registration): PASS\n"); } @@ -212,50 +273,53 @@ int main() me.addPlugin(std::make_unique(&store)); me.addPlugin(std::make_unique(&store)); int64_t total = me.runAll(pid); - assert(total > 0); + CHECK(total > 0); // Verify capability table is populated. sqlite3 *db = store.handle(); sqlite3_stmt *stmt = nullptr; int cap_count = 0; - if (sqlite3_prepare_v2(db, "SELECT COUNT(*) FROM capability " - "WHERE project_id=?", + if (sqlite3_prepare_v2(db, + "SELECT COUNT(*) FROM capability " + "WHERE project_id=?", -1, &stmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64(stmt, 1, static_cast(pid)); if (sqlite3_step(stmt) == SQLITE_ROW) cap_count = sqlite3_column_int(stmt, 0); sqlite3_finalize(stmt); } - assert(cap_count > 0); + CHECK(cap_count > 0); printf("Test 2 (capability extraction: %d caps): PASS\n", cap_count); // Verify contract table is populated. int contract_count = 0; - if (sqlite3_prepare_v2(db, "SELECT COUNT(*) FROM contract " - "WHERE project_id=?", + if (sqlite3_prepare_v2(db, + "SELECT COUNT(*) FROM contract " + "WHERE project_id=?", -1, &stmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64(stmt, 1, static_cast(pid)); if (sqlite3_step(stmt) == SQLITE_ROW) contract_count = sqlite3_column_int(stmt, 0); sqlite3_finalize(stmt); } - assert(contract_count > 0); + CHECK(contract_count > 0); printf("Test 3 (contract extraction: %d contracts): PASS\n", contract_count); // Verify workflow table is populated (main is an entry point // with a callee, so WorkflowPlugin should detect it). int workflow_count = 0; - if (sqlite3_prepare_v2(db, "SELECT COUNT(*) FROM workflow " - "WHERE project_id=?", + if (sqlite3_prepare_v2(db, + "SELECT COUNT(*) FROM workflow " + "WHERE project_id=?", -1, &stmt, nullptr) == SQLITE_OK) { sqlite3_bind_int64(stmt, 1, static_cast(pid)); if (sqlite3_step(stmt) == SQLITE_ROW) workflow_count = sqlite3_column_int(stmt, 0); sqlite3_finalize(stmt); } - assert(workflow_count > 0); + CHECK(workflow_count > 0); printf("Test 4 (workflow extraction: %d workflows): PASS\n", workflow_count); } @@ -265,8 +329,8 @@ int main() ModelEngine me(&store); me.addPlugin(std::make_unique(&store)); ModelResult r = me.run("Capability", pid); - assert(r.ok()); - assert(r.plugin_name == "Capability"); + CHECK(r.ok()); + CHECK(r.plugin_name == "Capability"); printf("Test 5 (run by name): PASS\n"); } @@ -275,8 +339,8 @@ int main() ModelEngine me(&store); me.addPlugin(std::make_unique(&store)); ModelResult r = me.run("Nonexistent", pid); - assert(!r.ok()); - assert(r.error.find("not found") != std::string::npos); + CHECK(!r.ok()); + CHECK(r.error.find("not found") != std::string::npos); printf("Test 6 (run unknown plugin): PASS\n"); } @@ -291,23 +355,22 @@ int main() // Clear model tables from Test 2's runAll so counts reflect // only this test's output. sqlite3_exec(db, "DELETE FROM capability", nullptr, nullptr, - nullptr); + nullptr); sqlite3_exec(db, "DELETE FROM contract", nullptr, nullptr, - nullptr); + nullptr); sqlite3_exec(db, "DELETE FROM workflow", nullptr, nullptr, - nullptr); + nullptr); sqlite3_exec(db, "DELETE FROM workflow_step", nullptr, nullptr, - nullptr); + nullptr); sqlite3_exec(db, "DELETE FROM architecture_edge", nullptr, - nullptr, nullptr); + nullptr, nullptr); // Cross-module callee: Logger lives in /tmp/lib/, while main // lives in /tmp/main.cpp. The module scopes below let // ArchitecturePlugin detect this as a cross-module call. insertGraphNode(store, pid, 4, 0, "Logger", "/tmp/lib/logger.cpp", "cpp", 1); - insertEntity(store, pid, 4, 0, "Logger", - "/tmp/lib/logger.cpp"); + insertEntity(store, pid, 4, 0, "Logger", "/tmp/lib/logger.cpp"); // Module scopes: "/tmp/" matches /tmp/main.cpp, "/tmp/lib/" // matches /tmp/lib/logger.cpp. Both are prefixes, so Logger @@ -326,7 +389,7 @@ int main() me.addPlugin(std::make_unique(&store)); me.addPlugin(std::make_unique(&store)); int64_t total = me.runAll(pid); - assert(total > 0); + CHECK(total > 0); // All 4 plugins produced rows from the single shared // ModelContext populated by runAll. If the context had not @@ -335,10 +398,10 @@ int main() int contract_count = countRows(store, pid, "contract"); int workflow_count = countRows(store, pid, "workflow"); int arch_count = countRows(store, pid, "architecture_edge"); - assert(cap_count > 0); - assert(contract_count > 0); - assert(workflow_count > 0); - assert(arch_count > 0); + CHECK(cap_count > 0); + CHECK(contract_count > 0); + CHECK(workflow_count > 0); + CHECK(arch_count > 0); printf("Test 7 (shared ModelContext: cap=%d contract=%d " "workflow=%d arch_edge=%d): PASS\n", cap_count, contract_count, workflow_count, arch_count); @@ -346,5 +409,5 @@ int main() unlink(kDbPath); printf("\nAll model engine tests passed.\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_module_edge.cpp b/engine/tests/test_module_edge.cpp index f3e5121..45113fa 100644 --- a/engine/tests/test_module_edge.cpp +++ b/engine/tests/test_module_edge.cpp @@ -14,7 +14,7 @@ #include "../src/async_knowledge.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -38,14 +38,14 @@ static void insertEntity(store::GraphStore &store, uint64_t project_id, "VALUES (?,?,0,?,'',?,'cpp',0,0,0,0," "rtrim(?, replace(?, '/', 'x')))"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, file_path, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 5, file_path, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 6, file_path, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -57,12 +57,12 @@ static void insertRelation(store::GraphStore &store, uint64_t project_id, const char *sql = "INSERT INTO relation (project_id, source_id, " "target_id, type) VALUES (?,?,?,?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, source_id); sqlite3_bind_int64(stmt, 3, target_id); sqlite3_bind_int(stmt, 4, type); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -71,10 +71,11 @@ static int getEdgeCount(store::GraphStore &store, uint64_t project_id, const char *src_module, const char *tgt_module) { sqlite3 *db = store.handle(); - const char *sql = "SELECT edge_count FROM module_edge " - "WHERE project_id=? AND src_module=? AND tgt_module=?"; + const char *sql = + "SELECT edge_count FROM module_edge " + "WHERE project_id=? AND src_module=? AND tgt_module=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, src_module, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 3, tgt_module, -1, SQLITE_TRANSIENT); @@ -91,7 +92,7 @@ static int totalEdges(store::GraphStore &store, uint64_t project_id) sqlite3 *db = store.handle(); const char *sql = "SELECT COUNT(*) FROM module_edge WHERE project_id=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); int count = 0; if (sqlite3_step(stmt) == SQLITE_ROW) @@ -105,29 +106,34 @@ int main() unlink(kDbPath); store::GraphStore store; - assert(store.open(kDbPath)); + CHECK(store.open(kDbPath)); uint64_t project_id = store.createProject("/test", "test_module_edge"); - assert(project_id > 0); + CHECK(project_id > 0); // ── Test 1: cross-module edges populated ──────────────────────── { // Two modules: /src/engine/ and /src/store/ // Two call edges from engine to store. - insertEntity(store, project_id, 1, "parse", "/src/engine/parser.cpp"); - insertEntity(store, project_id, 2, "emit", "/src/engine/emitter.cpp"); - insertEntity(store, project_id, 3, "save", "/src/store/store.cpp"); - - insertRelation(store, project_id, 1, 3, 1); // parse → save (CALLS) - insertRelation(store, project_id, 2, 3, 1); // emit → save (CALLS) + insertEntity(store, project_id, 1, "parse", + "/src/engine/parser.cpp"); + insertEntity(store, project_id, 2, "emit", + "/src/engine/emitter.cpp"); + insertEntity(store, project_id, 3, "save", + "/src/store/store.cpp"); + + insertRelation(store, project_id, 1, 3, + 1); // parse → save (CALLS) + insertRelation(store, project_id, 2, 3, + 1); // emit → save (CALLS) int64_t rows = buildKnowledgeGraphSync(store, project_id); - assert(rows == 1); // one module pair: engine → store - assert(totalEdges(store, project_id) == 1); + CHECK(rows == 1); // one module pair: engine → store + CHECK(totalEdges(store, project_id) == 1); - int count = getEdgeCount(store, project_id, - "/src/engine/", "/src/store/"); - assert(count == 2); // two call edges aggregated + int count = getEdgeCount(store, project_id, "/src/engine/", + "/src/store/"); + CHECK(count == 2); // two call edges aggregated printf(" [PASS] cross-module edges populated (engine→store count=2)\n"); } @@ -156,10 +162,10 @@ int main() int64_t rows = buildKnowledgeGraphSync(store, project_id); // One module pair: /src/engine/ → /src/engine/ (self-loop). - assert(rows == 1); - int self_count = getEdgeCount(store, project_id, - "/src/engine/", "/src/engine/"); - assert(self_count == 1); + CHECK(rows == 1); + int self_count = getEdgeCount(store, project_id, "/src/engine/", + "/src/engine/"); + CHECK(self_count == 1); printf(" [PASS] same-file edges excluded; module self-loop allowed\n"); } @@ -174,8 +180,8 @@ int main() nullptr, nullptr, nullptr); int64_t rows = buildKnowledgeGraphSync(store, project_id); - assert(rows == 0); - assert(totalEdges(store, project_id) == 0); + CHECK(rows == 0); + CHECK(totalEdges(store, project_id) == 0); printf(" [PASS] empty relation table → 0 module_edge rows\n"); } @@ -192,11 +198,11 @@ int main() insertRelation(store, project_id, 20, 21, 1); buildKnowledgeGraphSync(store, project_id); - assert(totalEdges(store, project_id) == 1); + CHECK(totalEdges(store, project_id) == 1); // Re-run — should still have 1 row, not 2. buildKnowledgeGraphSync(store, project_id); - assert(totalEdges(store, project_id) == 1); + CHECK(totalEdges(store, project_id) == 1); printf(" [PASS] re-run replaces old rows (idempotent)\n"); } @@ -218,8 +224,8 @@ int main() insertRelation(store, project_id, 30, 31, 2); int64_t rows = buildKnowledgeGraphSync(store, project_id); - assert(rows == 0); - assert(totalEdges(store, project_id) == 0); + CHECK(rows == 0); + CHECK(totalEdges(store, project_id) == 0); printf(" [PASS] non-call relations (type=2) excluded\n"); } @@ -244,10 +250,13 @@ int main() insertRelation(store, project_id, 40, 42, 1); int64_t rows = buildKnowledgeGraphSync(store, project_id); - assert(rows == 3); // three distinct module pairs - assert(getEdgeCount(store, project_id, "/mod_a/", "/mod_b/") == 1); - assert(getEdgeCount(store, project_id, "/mod_b/", "/mod_c/") == 1); - assert(getEdgeCount(store, project_id, "/mod_a/", "/mod_c/") == 1); + CHECK(rows == 3); // three distinct module pairs + CHECK(getEdgeCount(store, project_id, "/mod_a/", "/mod_b/") == + 1); + CHECK(getEdgeCount(store, project_id, "/mod_b/", "/mod_c/") == + 1); + CHECK(getEdgeCount(store, project_id, "/mod_a/", "/mod_c/") == + 1); printf(" [PASS] multiple module pairs (A→B, B→C, A→C)\n"); } @@ -255,5 +264,5 @@ int main() unlink(kDbPath); printf("=== test_module_edge PASSED ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_module_path_column.cpp b/engine/tests/test_module_path_column.cpp index d2ea2c8..be3c271 100644 --- a/engine/tests/test_module_path_column.cpp +++ b/engine/tests/test_module_path_column.cpp @@ -18,7 +18,7 @@ #include "../src/ir/semantic_unit.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -28,9 +28,9 @@ using namespace ir; static const char *kDbPath = "/tmp/codescope_test_module_path.db"; /// Insert a single Function semantic record for the given file. -static void insertFunctionRecord(store::GraphStore &store, - uint64_t project_id, const char *file_path, - const char *name, uint64_t id) +static void insertFunctionRecord(store::GraphStore &store, uint64_t project_id, + const char *file_path, const char *name, + uint64_t id) { std::vector records; Record r; @@ -41,7 +41,7 @@ static void insertFunctionRecord(store::GraphStore &store, r.qualified_name = name; r.file_path = file_path; r.language = "cpp"; - r.loc = SourceRange{1, 0, 2, 0}; + r.loc = SourceRange{ 1, 0, 2, 0 }; records.push_back(r); store.insertSemanticRecords(project_id, file_path, records); } @@ -54,7 +54,7 @@ static std::string expectedModulePath(store::GraphStore &store, sqlite3 *db = store.handle(); const char *sql = "SELECT rtrim(?, replace(?, '/', 'x'))"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_text(stmt, 1, file_path.c_str(), -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 2, file_path.c_str(), -1, SQLITE_TRANSIENT); std::string result; @@ -73,14 +73,14 @@ struct EntityRow { std::string module_path; bool found = false; }; -static EntityRow getEntityByName(store::GraphStore &store, - uint64_t project_id, const char *name) +static EntityRow getEntityByName(store::GraphStore &store, uint64_t project_id, + const char *name) { sqlite3 *db = store.handle(); const char *sql = "SELECT file_path, module_path FROM entity " "WHERE project_id=? AND name=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, name, -1, SQLITE_TRANSIENT); EntityRow row; @@ -98,14 +98,14 @@ static EntityRow getEntityByName(store::GraphStore &store, } /// Fetch all module-level scope names (kind=1) for a project. -static std::vector -getModuleScopeNames(store::GraphStore &store, uint64_t project_id) +static std::vector getModuleScopeNames(store::GraphStore &store, + uint64_t project_id) { sqlite3 *db = store.handle(); const char *sql = "SELECT name FROM scope WHERE project_id=? AND kind=1"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); std::vector names; while (sqlite3_step(stmt) == SQLITE_ROW) { @@ -122,9 +122,9 @@ int main() unlink(kDbPath); store::GraphStore store; - assert(store.open(kDbPath)); + CHECK(store.open(kDbPath)); uint64_t pid = store.createProject("/test", "test_module_path"); - assert(pid > 0); + CHECK(pid > 0); // Insert semantic records for functions in three different files // spanning two directories. The file paths must NOT match the @@ -137,7 +137,7 @@ int main() // Run buildGraph — this populates entity.module_path via the // rtrim(file_path, replace(file_path, '/', 'x')) expression and // creates module scopes (kind=1) from module_path values. - assert(store.buildGraph(pid, true)); + CHECK(store.buildGraph(pid, true)); // ── Test 1: entity.module_path matches rtrim expression ────── // Each entity's stored module_path must equal both the hardcoded @@ -150,18 +150,18 @@ int main() const char *expected_module_path; }; const Case cases[] = { - {"GetUser", "/src/api/handler.cpp", "/src/api/"}, - {"PostUser", "/src/api/handler.cpp", "/src/api/"}, - {"TrimSpace", "/src/lib/util.cpp", "/src/lib/"}, + { "GetUser", "/src/api/handler.cpp", "/src/api/" }, + { "PostUser", "/src/api/handler.cpp", "/src/api/" }, + { "TrimSpace", "/src/lib/util.cpp", "/src/lib/" }, }; for (const auto &c : cases) { EntityRow row = getEntityByName(store, pid, c.name); - assert(row.found); - assert(row.file_path == c.file_path); + CHECK(row.found); + CHECK(row.file_path == c.file_path); std::string expected_sql = expectedModulePath(store, c.file_path); - assert(row.module_path == c.expected_module_path); - assert(row.module_path == expected_sql); + CHECK(row.module_path == c.expected_module_path); + CHECK(row.module_path == expected_sql); printf(" [PASS] %s: file=%s module_path=%s\n", c.name, c.file_path, c.expected_module_path); } @@ -174,7 +174,7 @@ int main() // the module_path values. { auto scope_names = getModuleScopeNames(store, pid); - assert(scope_names.size() == 2); + CHECK(scope_names.size() == 2); std::string expected_api = expectedModulePath(store, "/src/api/handler.cpp"); std::string expected_lib = @@ -186,8 +186,8 @@ int main() if (n == expected_lib) has_lib = true; } - assert(has_api); - assert(has_lib); + CHECK(has_api); + CHECK(has_lib); printf("Test 2 (scope names match module_path: %s, %s): PASS\n", expected_api.c_str(), expected_lib.c_str()); } @@ -195,5 +195,5 @@ int main() store.close(); unlink(kDbPath); printf("\nAll module_path column tests passed.\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_module_summary_internal_edges.cpp b/engine/tests/test_module_summary_internal_edges.cpp new file mode 100644 index 0000000..58510a1 --- /dev/null +++ b/engine/tests/test_module_summary_internal_edges.cpp @@ -0,0 +1,158 @@ +// test_module_summary_internal_edges.cpp — module_summary.internal_edges must +// report the module's real internal coupling. +// +// The column has existed since the canonical-schema migration with a +// `NOT NULL DEFAULT 0`, and buildModuleSummaries passed a literal 0 in its +// INSERT, so every module in every project reported zero internal edges +// whatever the call graph said — measured on goagent (114 of 131 modules have +// internal calls, 4533 edges in total) and on this repository (21 of 22, 1574). +// The new `intra` CTE counts call edges whose source and target entity both +// live in the module, and its result is joined on the module. +// +// Fixture (two modules, each with the three entities the aggregate requires): +// src/a.cpp alpha → beta (cross-file), recurse → recurse (self-loop) +// src/b.cpp beta → gamma +// other/c.cpp delta → alpha (cross-module, and unresolved without an import) +// so `src` has exactly 2 internal edges and `other` has 0. Self-loops are +// excluded on purpose, mirroring incoming_count/outgoing_count: an entity +// calling itself is not a dependency between two entities. +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include + +#include "test_check.h" +#include "test_engine_handle.h" + +namespace +{ + +/// Run one scalar SQL query (COUNT(*)) against the test database. +long long scalarSql(sqlite3 *db, const char *sql) +{ + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + long long value = -1; + if (sqlite3_step(st) == SQLITE_ROW) + value = sqlite3_column_int64(st, 0); + sqlite3_finalize(st); + return value; +} + +/// Write `content` to `path`, failing the test if the file cannot be created. +void writeFile(const std::string &path, const char *content) +{ + FILE *f = fopen(path.c_str(), "w"); + CHECK(f != nullptr); + fputs(content, f); + fclose(f); +} + +/// internal_edges of the module whose scope name ends with `suffix`. +long long internalEdgesOf(sqlite3 *db, const char *suffix) +{ + const std::string sql = + std::string("SELECT ms.internal_edges FROM module_summary ms " + "JOIN scope s ON s.id = ms.module_id WHERE s.name " + "LIKE '%") + + suffix + "'"; + return scalarSql(db, sql.c_str()); +} + +} // namespace + +int main() +{ + const std::string proj_dir = "/tmp/module_internal_edges_repro"; + std::filesystem::remove_all(proj_dir); + std::filesystem::create_directories(proj_dir + "/src"); + std::filesystem::create_directories(proj_dir + "/other"); + + writeFile(proj_dir + "/src/a.cpp", + "int beta();\n" + "\n" + "int alpha()\n" + "{\n" + "\treturn beta();\n" + "}\n" + "\n" + "int recurse(int n)\n" + "{\n" + "\treturn n > 0 ? recurse(n - 1) : 0;\n" + "}\n"); + writeFile(proj_dir + "/src/b.cpp", "int gamma();\n" + "\n" + "int beta()\n" + "{\n" + "\treturn gamma();\n" + "}\n" + "\n" + "int gamma()\n" + "{\n" + "\treturn 0;\n" + "}\n"); + writeFile(proj_dir + "/other/c.cpp", "int alpha();\n" + "\n" + "int delta()\n" + "{\n" + "\treturn alpha();\n" + "}\n" + "\n" + "int epsilon()\n" + "{\n" + "\treturn 0;\n" + "}\n" + "\n" + "int zeta()\n" + "{\n" + "\treturn 0;\n" + "}\n"); + + char db[] = "/tmp/test_module_summary_internal_edges.db"; + unlink(db); + g_engine = engine_create(db); + CHECK(g_engine != nullptr); + const uint64_t pid = engine_create_project(g_engine, proj_dir.c_str(), + "module-edges"); + CHECK(pid > 0); + char *idx = + engine_index_project(g_engine, pid, proj_dir.c_str(), nullptr); + CHECK(idx != nullptr); + CHECK(strstr(idx, "\"ok\":true") != nullptr); + engine_free_string(idx); + + sqlite3 *db_h = nullptr; + CHECK(sqlite3_open(db, &db_h) == SQLITE_OK); + + // module_summary is written by the async knowledge builder, so wait for it + // with a bounded poll: a failure has to be a failing assertion, not a + // flaky one, and the bound keeps a hung builder from hanging the suite. + int waited_ms = 0; + while (scalarSql(db_h, "SELECT COUNT(*) FROM module_summary") == 0 && + waited_ms < 5000) { + usleep(50000); + waited_ms += 50; + } + CHECK(scalarSql(db_h, "SELECT COUNT(*) FROM module_summary") > 0); + + // The two resolved call edges the fixture can support (alpha -> beta, + // beta -> gamma); without them the assertions below would pass for the + // wrong reason. + CHECK(scalarSql(db_h, "SELECT COUNT(*) FROM relation WHERE type=1") == + 2); + + CHECK(internalEdgesOf(db_h, "/src/") == 2); + CHECK(internalEdgesOf(db_h, "/other/") == 0); + + printf("=== module_summary internal_edges test passed ===\n"); + sqlite3_close(db_h); + engine_destroy(g_engine); + g_engine = nullptr; + unlink(db); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_parent_chain.cpp b/engine/tests/test_parent_chain.cpp index 2af6e1c..e75820a 100644 --- a/engine/tests/test_parent_chain.cpp +++ b/engine/tests/test_parent_chain.cpp @@ -19,6 +19,7 @@ #include #include #include +#include "test_engine_handle.h" static inline void check(bool cond, const char *msg) { @@ -33,8 +34,7 @@ int main() const char *proj_dir = "/tmp/parent_chain_repro"; std::filesystem::remove_all(proj_dir); std::filesystem::create_directories(proj_dir); - const std::string py_path = - std::string(proj_dir) + "/repro.py"; + const std::string py_path = std::string(proj_dir) + "/repro.py"; FILE *f = fopen(py_path.c_str(), "w"); check(f != nullptr, "fopen"); // Minimal shape of the reported bug: @@ -55,12 +55,14 @@ def create_evolution_timeline(): char db[] = "/tmp/test_parent_chain.db"; unlink(db); - check(engine_init(db) == 0, "engine_init"); + g_engine = engine_create(db); + check(g_engine != nullptr, "engine_init"); - uint64_t pid = engine_create_project(proj_dir, "parent-chain"); + uint64_t pid = + engine_create_project(g_engine, proj_dir, "parent-chain"); check(pid > 0, "create_project"); - char *idx = engine_index_project(pid, proj_dir, nullptr); + char *idx = engine_index_project(g_engine, pid, proj_dir, nullptr); check(idx != nullptr, "index_project null"); check(strstr(idx, "\"ok\":true") != nullptr, "index ok"); engine_free_string(idx); @@ -68,7 +70,7 @@ def create_evolution_timeline(): // Wait for async pipeline to finish. for (int i = 0; i < 50; i++) { usleep(100000); - char *stats = engine_get_graph_stats(pid); + char *stats = engine_get_graph_stats(g_engine, pid); if (stats && strstr(stats, "\"total_nodes\"")) { engine_free_string(stats); break; @@ -81,23 +83,20 @@ def create_evolution_timeline(): // ── Observation 1: callees of create_evolution_timeline ── // Expected: add_trace (intra-file edge via P1). // Before fix: only 0 or 1 edge, missing add_trace. - char *callees = engine_get_callees(pid, - "create_evolution_timeline", - nullptr); + char *callees = engine_get_callees( + g_engine, pid, "create_evolution_timeline", nullptr); check(callees != nullptr, "get_callees null"); - printf("--- callees(create_evolution_timeline) ---\n%s\n", - callees); + printf("--- callees(create_evolution_timeline) ---\n%s\n", callees); - bool has_add_trace = - strstr(callees, "add_trace") != nullptr; + bool has_add_trace = strstr(callees, "add_trace") != nullptr; int total_callees = 0; const char *p = strstr(callees, "\"total\":"); if (p) total_callees = atoi(p + 8); engine_free_string(callees); - printf("\nhas_add_trace = %d, total_callees = %d\n", - (int)has_add_trace, total_callees); + printf("\nhas_add_trace = %d, total_callees = %d\n", (int)has_add_trace, + total_callees); if (has_add_trace && total_callees >= 1) { printf("\nPASS: add_trace is a callee of " @@ -107,7 +106,8 @@ def create_evolution_timeline(): "is missing add_trace callee edge\n"); } - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; std::filesystem::remove_all(proj_dir); return 0; } diff --git a/engine/tests/test_parse_failures.cpp b/engine/tests/test_parse_failures.cpp new file mode 100644 index 0000000..270e0b2 --- /dev/null +++ b/engine/tests/test_parse_failures.cpp @@ -0,0 +1,628 @@ +// test_parse_failures.cpp — the parse_failures bookkeeping of every index path. +// +// Four defects are covered here, all found while checking what happens to a +// `.swift` file (detected by extension, but no Swift grammar is vendored): +// +// 1. memBulk — the path every project of <=2000 files takes — buffered parse +// failures and then RETURNED EARLY from the dispatcher, never reaching the +// streaming path's store::flushParseFailures(). `parse_failures` stayed +// empty, so an unparseable file was dropped with no record anywhere. +// 2. A registered-but-NULL grammar was handed to ts_parser_set_language, +// which yields a null tree recorded as "parse_null_tree" — the wrong +// reason, disguising an unsupported language as a broken file. +// 3. "language_missing" rows counted towards the permanent skip set +// (retry_max=1), so a file that only failed because its grammar was +// unavailable was skipped forever — even after the grammar was enabled. +// 4. engine_index_files — shared by the force_index_files tool and by the +// scheduler-driven worker/chunk paths — always applied the fail-fast +// skip, with a hard-coded threshold of 3 (three times the documented +// default), so a file the user explicitly asked to index was dropped +// silently: the opposite of that tool's "indexed regardless of the +// default skip rules" contract. The policy is now an explicit +// `bypass_fail_fast` argument (engine.h) that each caller states. +// +// Fixture: one valid C file, one 0-byte Python file (a genuine, permanent parse +// failure: "read_empty"), one .swift file and one .rb file (both detected but +// ungrammared). + +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include +#include "test_engine_handle.h" + +namespace fs = std::filesystem; + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "\nFAIL: %s\n", msg); + exit(1); + } +} + +/// Read (fail_reason, fail_count) for `file_path`; returns false when absent. +static bool failureOf(sqlite3 *db, uint64_t pid, const std::string &file_path, + std::string &reason, int &count) +{ + sqlite3_stmt *st = nullptr; + const char *sql = "SELECT fail_reason, fail_count FROM parse_failures " + "WHERE project_id=? AND file_path=?"; + check(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK, + "prepare failureOf"); + sqlite3_bind_int64(st, 1, static_cast(pid)); + sqlite3_bind_text(st, 2, file_path.c_str(), -1, SQLITE_TRANSIENT); + bool found = false; + if (sqlite3_step(st) == SQLITE_ROW) { + const char *r = reinterpret_cast( + sqlite3_column_text(st, 0)); + reason = r ? r : ""; + count = sqlite3_column_int(st, 1); + found = true; + } + sqlite3_finalize(st); + return found; +} + +static int entityCount(sqlite3 *db, uint64_t pid) +{ + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2( + db, "SELECT COUNT(*) FROM entity WHERE project_id=?", -1, + &st, nullptr) == SQLITE_OK, + "prepare entityCount"); + sqlite3_bind_int64(st, 1, static_cast(pid)); + int n = 0; + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + return n; +} + +/// Canonical (symlink-resolved) form of `path`, used by the identity tests to +/// spell a path the way the single-file entry points receive it. +/// \param path Existing file or directory to resolve. +/// \return The canonical spelling, or the test fails. +/// Uses std::filesystem rather than realpath()+free() so the test allocates +/// nothing by hand (plan/rules/code_rules.md §2). +static std::string canonicalPath(const std::string &path) +{ + std::error_code ec; + const fs::path canon = fs::canonical(path, ec); + check(!ec, "canonical path must resolve"); + return canon.string(); +} + +static int entityRowsForName(sqlite3 *db, uint64_t pid, const char *name) +{ + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2( + db, + "SELECT COUNT(*) FROM entity WHERE project_id=? " + "AND name=?", + -1, &st, nullptr) == SQLITE_OK, + "prepare entityRowsForName"); + sqlite3_bind_int64(st, 1, static_cast(pid)); + sqlite3_bind_text(st, 2, name, -1, SQLITE_TRANSIENT); + int n = 0; + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + return n; +} + +static int parseFailureRowCount(sqlite3 *db, uint64_t pid) +{ + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2( + db, + "SELECT COUNT(*) FROM parse_failures WHERE project_id=?", + -1, &st, nullptr) == SQLITE_OK, + "prepare parseFailureRowCount"); + sqlite3_bind_int64(st, 1, static_cast(pid)); + int n = 0; + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + return n; +} + +static void writeFile(const std::string &path, const char *content) +{ + FILE *f = fopen(path.c_str(), "w"); + check(f != nullptr, "fopen fixture"); + if (content) + fputs(content, f); + fclose(f); +} + +static void prepareProject(const std::string &dir) +{ + fs::remove_all(dir); + fs::create_directories(dir); + writeFile(dir + "/good.c", "int kept(void) { return 0; }\n"); + // 0 bytes: read returns empty → a real, permanent parse failure. + writeFile(dir + "/empty.py", nullptr); + // Detected as "swift", but no Swift grammar is vendored. + writeFile(dir + "/a.swift", + "class Foo { func bar() -> Int { return 1 } }\n"); + // A second detected-but-ungrammared extension (Kotlin/Ruby/Scala/Swift all + // behave identically): pins that the contract is per-language-availability, + // not a Swift special case. + writeFile(dir + "/b.rb", "def bar\n 1\nend\n"); +} + +static void removeDb(const std::string &db) +{ + unlink(db.c_str()); + unlink((db + "-wal").c_str()); + unlink((db + "-shm").c_str()); +} + +int main() +{ + const std::string dir = "/tmp/test_parse_failures_proj"; + const std::string db_path = "/tmp/test_parse_failures.db"; + const std::string swift_path = dir + "/a.swift"; + const std::string ruby_path = dir + "/b.rb"; + const std::string empty_path = dir + "/empty.py"; + const std::string good_path = dir + "/good.c"; + prepareProject(dir); + removeDb(db_path); + + // ── 1. memBulk (the default path): failures must be RECORDED ── + g_engine = engine_create(db_path.c_str()); + check(g_engine != nullptr, "engine_init"); + uint64_t pid = + engine_create_project(g_engine, dir.c_str(), "parse-failures"); + check(pid > 0, "create_project"); + + char *r1 = engine_index_project(g_engine, pid, dir.c_str(), nullptr); + check(r1 != nullptr && strstr(r1, "\"ok\":true") != nullptr, + "run 1 (membulk) reports ok:true"); + engine_free_string(r1); + engine_destroy(g_engine); + g_engine = nullptr; + + sqlite3 *db = nullptr; + check(sqlite3_open_v2(db_path.c_str(), &db, SQLITE_OPEN_READONLY, + nullptr) == SQLITE_OK, + "open result db"); + std::string reason; + int count = 0; + check(failureOf(db, pid, swift_path, reason, count), + "membulk must RECORD the unavailable-grammar file in " + "parse_failures (it used to be dropped silently)"); + check(reason == "language_missing", + "an unavailable grammar must be reported as language_missing"); + check(count == 1, "first failure has fail_count 1"); + check(failureOf(db, pid, ruby_path, reason, count), + "every detected-but-ungrammared extension must be recorded, not " + "just .swift"); + check(reason == "language_missing", + "ruby (no vendored grammar) is a language_missing failure too"); + check(failureOf(db, pid, empty_path, reason, count), + "the empty file must be recorded too"); + check(reason == "read_empty", "an empty file is a read_empty failure"); + const int entities1 = entityCount(db, pid); + check(entities1 >= 1, "the valid C file must still be indexed"); + sqlite3_close(db); + + // ── 2. A second run: language_missing is retried, read_empty is not ── + g_engine = engine_create(db_path.c_str()); + check(g_engine != nullptr, "engine_init (run 2)"); + pid = engine_create_project(g_engine, dir.c_str(), "parse-failures"); + check(pid > 0, "create_project (run 2)"); + char *r2 = engine_index_project(g_engine, pid, dir.c_str(), nullptr); + check(r2 != nullptr && strstr(r2, "\"ok\":true") != nullptr, + "run 2 reports ok:true"); + engine_free_string(r2); + engine_destroy(g_engine); + g_engine = nullptr; + + check(sqlite3_open_v2(db_path.c_str(), &db, SQLITE_OPEN_READONLY, + nullptr) == SQLITE_OK, + "reopen result db"); + check(failureOf(db, pid, swift_path, reason, count), + "the swift row must still exist after run 2"); + check(reason == "language_missing" && count == 2, + "an unavailable grammar must be RE-ATTEMPTED on every run " + "(fail_count grows) — a path-keyed skip set would pin it out " + "forever, even after the grammar is enabled"); + check(failureOf(db, pid, empty_path, reason, count), + "the empty row must still exist after run 2"); + check(count == 1, + "a genuine parse failure must stay skipped once the retry " + "threshold is reached (fail_count stays 1)"); + sqlite3_close(db); + + // ── 3. Streaming path: same reason, not "parse_null_tree" ───── + const std::string stream_db = "/tmp/test_parse_failures_stream.db"; + removeDb(stream_db); + setenv("CODESCOPE_FORCE_STREAMING", "1", 1); + g_engine = engine_create(stream_db.c_str()); + check(g_engine != nullptr, "engine_init (streaming)"); + uint64_t pid_s = engine_create_project(g_engine, dir.c_str(), + "parse-failures-s"); + check(pid_s > 0, "create_project (streaming)"); + char *r3 = engine_index_project(g_engine, pid_s, dir.c_str(), nullptr); + check(r3 != nullptr && strstr(r3, "\"ok\":true") != nullptr, + "streaming run reports ok:true"); + engine_free_string(r3); + engine_destroy(g_engine); + g_engine = nullptr; + unsetenv("CODESCOPE_FORCE_STREAMING"); + + check(sqlite3_open_v2(stream_db.c_str(), &db, SQLITE_OPEN_READONLY, + nullptr) == SQLITE_OK, + "open streaming db"); + check(failureOf(db, pid_s, swift_path, reason, count), + "the streaming path must record it as well"); + check(reason == "language_missing", + "the streaming path must report language_missing, not " + "parse_null_tree (the old behaviour handed nullptr to " + "ts_parser_set_language and mislabelled the result)"); + check(entityCount(db, pid_s) >= 1, + "the streaming path must still index the valid file"); + sqlite3_close(db); + + // ── 4. engine_index_files: the two fail-fast policies ──────── + // + // The entry point is shared by the force_index_files tool (which must + // index what the user named, "regardless of the default skip rules") and + // by the scheduler-driven worker/chunk paths (which must mirror the + // automatic project index). The caller states the policy through + // bypass_fail_fast; it used to be guessed from the environment with a + // hard-coded threshold of 3, so the tool's explicit request was dropped + // silently and the workers used a different threshold from the project + // path. + g_engine = engine_create(db_path.c_str()); + check(g_engine != nullptr, "engine_init (index_files)"); + check(engine_create_project(g_engine, dir.c_str(), "parse-failures") == + pid, + "index_files run reuses the same project"); + const std::string file_list = + "[\"" + empty_path + "\", \"" + swift_path + "\"]"; + + // 4a. bypass_fail_fast = 1 → every listed file is re-attempted. + char *rf = engine_index_files(g_engine, pid, file_list.c_str(), 1); + check(rf != nullptr && strstr(rf, "\"ok\":true") != nullptr, + "bypass run reports ok:true"); + engine_free_string(rf); + engine_destroy(g_engine); + g_engine = nullptr; + + check(sqlite3_open_v2(db_path.c_str(), &db, SQLITE_OPEN_READONLY, + nullptr) == SQLITE_OK, + "reopen result db (bypass)"); + check(failureOf(db, pid, empty_path, reason, count), + "the empty row must still exist after the bypass run"); + check(reason == "read_empty" && count == 2, + "bypass_fail_fast must RE-ATTEMPT a known parse failure " + "(fail_count 1 → 2); applying the fail-fast skip there silently " + "discarded the user's explicit request"); + check(failureOf(db, pid, swift_path, reason, count), + "the unavailable-grammar row must still exist after the bypass run"); + check(reason == "language_missing" && count == 3, + "the bypass path must re-attempt the language_missing file too"); + sqlite3_close(db); + + // 4b. bypass_fail_fast = 0 → the scheduler policy applies: the genuine + // parse failure is skipped, while the language_missing row stays exempt. + g_engine = engine_create(db_path.c_str()); + check(g_engine != nullptr, "engine_init (scheduler policy)"); + check(engine_create_project(g_engine, dir.c_str(), "parse-failures") == + pid, + "worker run reuses the same project"); + char *rw = engine_index_files(g_engine, pid, file_list.c_str(), 0); + check(rw != nullptr && strstr(rw, "\"ok\":true") != nullptr, + "worker-policy run reports ok:true"); + engine_free_string(rw); + engine_destroy(g_engine); + g_engine = nullptr; + + check(sqlite3_open_v2(db_path.c_str(), &db, SQLITE_OPEN_READONLY, + nullptr) == SQLITE_OK, + "reopen result db (worker policy)"); + check(failureOf(db, pid, empty_path, reason, count), + "the empty row must still exist after the worker-policy run"); + check(count == 2, + "the scheduler-driven callers must KEEP the fail-fast skip " + "(read_empty stays at 2), so index-parallel behaves like the " + "automatic project path"); + check(failureOf(db, pid, swift_path, reason, count), + "the unavailable-grammar row must still exist"); + check(reason == "language_missing" && count == 4, + "a missing grammar is exempt from the skip in BOTH policies " + "(2 → 3 → 4)"); + sqlite3_close(db); + + removeDb(db_path); + removeDb(stream_db); + fs::remove_all(dir); + + // ── 5. One file, ONE identity across the entry points ──────── + // + // The walk-based project index stores the spelling its caller passed (a + // relative root yields "proj/empty.py", "." yields "./empty.py"); the + // single-file entry points are handed ABSOLUTE canonicalised paths (the + // force_index_files walk calls std::fs::canonicalize). engine_index_paths.h + // exists to bridge that by reusing the stored spelling — but it looked in + // `entity` only, and a file that NEVER parses has no entity row, while + // those are exactly the files that accumulate parse_failures rows. The + // re-attempt then created a SECOND row, with its own fail_count, for the + // same file under the other spelling. + const std::string sp_dir = "/tmp/test_parse_failures_spelling"; + const std::string sp_db = "/tmp/test_parse_failures_spelling.db"; + // The spelling a relative-root walk stores. + const std::string sp_rel_file = "test_parse_failures_spelling/empty.py"; + fs::remove_all(sp_dir); + fs::create_directories(sp_dir); + writeFile(sp_dir + "/empty.py", nullptr); + removeDb(sp_db); + + const fs::path restore_cwd = fs::current_path(); + fs::current_path("/tmp"); + g_engine = engine_create(sp_db.c_str()); + check(g_engine != nullptr, "engine_init (spelling)"); + uint64_t pid_sp = engine_create_project( + g_engine, "test_parse_failures_spelling", "spelling"); + check(pid_sp > 0, "create_project (spelling)"); + char *rsp = engine_index_project( + g_engine, pid_sp, "test_parse_failures_spelling", nullptr); + check(rsp != nullptr && strstr(rsp, "\"ok\":true") != nullptr, + "spelling run: project index ok"); + engine_free_string(rsp); + engine_destroy(g_engine); + g_engine = nullptr; + fs::current_path(restore_cwd); + + // The absolute, canonical spelling the single-file callers receive. + const std::string canon = canonicalPath(sp_dir); + + // Reuse pid_sp as-is: engine_index_files needs only the project id, and + // re-creating the project here would resolve the relative root against the + // restored working directory, yielding a different project. + g_engine = engine_create(sp_db.c_str()); + check(g_engine != nullptr, "engine_init (spelling force)"); + const std::string sp_list = "[\"" + canon + "/empty.py\"]"; + char *rspf = engine_index_files(g_engine, pid_sp, sp_list.c_str(), 1); + check(rspf != nullptr && strstr(rspf, "\"ok\":true") != nullptr, + "spelling force run reports ok:true"); + engine_free_string(rspf); + engine_destroy(g_engine); + g_engine = nullptr; + + check(sqlite3_open_v2(sp_db.c_str(), &db, SQLITE_OPEN_READONLY, + nullptr) == SQLITE_OK, + "open spelling db"); + check(failureOf(db, pid_sp, sp_rel_file, reason, count), + "the file must stay under the spelling the project index used"); + check(reason == "read_empty" && count == 2, + "the re-attempt must land on the SAME row (fail_count 1 → 2), not " + "create a second identity for the absolute spelling"); + check(parseFailureRowCount(db, pid_sp) == 1, + "one file must have exactly one parse_failures row no matter which " + "entry point recorded it"); + sqlite3_close(db); + + removeDb(sp_db); + fs::remove_all(sp_dir); + + // ── 6. A SYMLINKED ANCESTOR must not split a file's identity ── + // + // The enumerable spellings cannot bridge this one: no spelling of the + // canonical "/private/tmp/…" yields the aliased "/tmp/…" the walk stored, + // because realpath only goes one way. The second pass narrows on the path + // tail and accepts a stored row only when its canonical form EQUALS the + // incoming path's, so this test covers both tables — a duplicated + // parse_failures row AND duplicated entity rows for a file that parses. + const std::string al_dir = "/tmp/test_pf_alias"; + const std::string al_alias = "/tmp/test_pf_alias_link"; + const std::string al_db = "/tmp/test_pf_alias.db"; + fs::remove_all(al_dir); + fs::remove(al_alias); + fs::remove_all(al_db); + fs::create_directories(al_dir); + writeFile(al_dir + "/empty.py", nullptr); + writeFile(al_dir + "/good.c", "int k(void) { return 0; }\n"); + // The alias the project will be indexed through. + std::error_code link_ec; + fs::create_symlink(al_dir, al_alias, link_ec); + check(!link_ec, "create the aliased project root"); + + g_engine = engine_create(al_db.c_str()); + check(g_engine != nullptr, "engine_init (alias)"); + uint64_t pid_al = + engine_create_project(g_engine, al_alias.c_str(), "alias"); + check(pid_al > 0, "create_project (alias)"); + // Indexed THROUGH the alias: the walk stores "/tmp/test_pf_alias_link/…", + // while projects.root_path records the canonical target. + char *ral = engine_index_project(g_engine, pid_al, al_alias.c_str(), + nullptr); + check(ral != nullptr && strstr(ral, "\"ok\":true") != nullptr, + "alias run: project index ok"); + engine_free_string(ral); + engine_destroy(g_engine); + g_engine = nullptr; + + check(sqlite3_open_v2(al_db.c_str(), &db, SQLITE_OPEN_READONLY, + nullptr) == SQLITE_OK, + "open alias db"); + check(failureOf(db, pid_al, al_alias + "/empty.py", reason, count), + "the failure must be recorded under the alias spelling"); + check(entityRowsForName(db, pid_al, "k") == 1, + "the symbol must start under the alias spelling"); + sqlite3_close(db); + + // Re-create the project record the way force_index_files sees it (the + // canonical root) and force-index the canonical paths. + const std::string canon_al = canonicalPath(al_dir); + + g_engine = engine_create(al_db.c_str()); + check(g_engine != nullptr, "engine_init (alias force)"); + const std::string al_list = + "[\"" + canon_al + "/empty.py\", \"" + canon_al + "/good.c\"]"; + char *ralf = engine_index_files(g_engine, pid_al, al_list.c_str(), 1); + check(ralf != nullptr && strstr(ralf, "\"ok\":true") != nullptr, + "alias force run reports ok:true"); + engine_free_string(ralf); + engine_destroy(g_engine); + g_engine = nullptr; + + check(sqlite3_open_v2(al_db.c_str(), &db, SQLITE_OPEN_READONLY, + nullptr) == SQLITE_OK, + "reopen alias db"); + check(failureOf(db, pid_al, al_alias + "/empty.py", reason, count), + "the unparseable file must stay under the alias spelling"); + check(reason == "read_empty" && count == 2, + "the force re-attempt must land on the alias row (1 → 2)"); + check(parseFailureRowCount(db, pid_al) == 1, + "a symlinked ancestor must not add a second parse_failures row"); + check(entityRowsForName(db, pid_al, "k") == 1, + "a symlinked ancestor must not duplicate the SYMBOL: the canonical " + "spelling has to resolve to the alias identity (find_symbol used to " + "answer with two rows for this one file)"); + sqlite3_close(db); + + removeDb(al_db); + fs::remove_all(al_dir); + fs::remove(al_alias); + + // ── 7. A relative spelling OUTSIDE the enumerated forms ────── + // + // The walk stores "/". For a root of "." the stored + // spelling is "./f.c" and for a plain name it is "proj/f.c" — both are + // enumerated as candidates. A root of "./proj" gives "./proj/f.c", which + // no candidate matches, so the second pass has to recognise it: the row's + // prefix must be a spelling of the project root that resolves to the + // (canonical) root from the current working directory. + const std::string rl_dir = "/tmp/test_pf_relfile"; + const std::string rl_db = "/tmp/test_pf_relfile.db"; + fs::remove_all(rl_dir); + fs::remove_all(rl_db); + fs::create_directories(rl_dir); + writeFile(rl_dir + "/empty.py", nullptr); + writeFile(rl_dir + "/good.c", "int k(void) { return 0; }\n"); + + const fs::path restore_cwd2 = fs::current_path(); + fs::current_path("/tmp"); + g_engine = engine_create(rl_db.c_str()); + check(g_engine != nullptr, "engine_init (relative root)"); + uint64_t pid_rl = + engine_create_project(g_engine, "./test_pf_relfile", "relfile"); + check(pid_rl > 0, "create_project (relative root)"); + char *rrl = engine_index_project(g_engine, pid_rl, "./test_pf_relfile", + nullptr); + check(rrl != nullptr && strstr(rrl, "\"ok\":true") != nullptr, + "relative-root run: project index ok"); + engine_free_string(rrl); + engine_destroy(g_engine); + g_engine = nullptr; + + check(sqlite3_open_v2(rl_db.c_str(), &db, SQLITE_OPEN_READONLY, + nullptr) == SQLITE_OK, + "open relative-root db"); + check(failureOf(db, pid_rl, "./test_pf_relfile/empty.py", reason, + count), + "the failure must be recorded under the './proj/…' spelling"); + check(entityRowsForName(db, pid_rl, "k") == 1, + "the symbol must start under the relative spelling"); + sqlite3_close(db); + + const std::string canon_rl = canonicalPath(rl_dir); + + // Force-index with absolute paths WHILE the working directory is still the + // one the project was indexed from — the condition that makes the relative + // row's prefix verifiable. + g_engine = engine_create(rl_db.c_str()); + check(g_engine != nullptr, "engine_init (relative force)"); + const std::string rl_list = + "[\"" + canon_rl + "/empty.py\", \"" + canon_rl + "/good.c\"]"; + char *rrlf = engine_index_files(g_engine, pid_rl, rl_list.c_str(), 1); + check(rrlf != nullptr && strstr(rrlf, "\"ok\":true") != nullptr, + "relative-root force run reports ok:true"); + engine_free_string(rrlf); + engine_destroy(g_engine); + g_engine = nullptr; + fs::current_path(restore_cwd2); + + check(sqlite3_open_v2(rl_db.c_str(), &db, SQLITE_OPEN_READONLY, + nullptr) == SQLITE_OK, + "reopen relative-root db"); + check(failureOf(db, pid_rl, "./test_pf_relfile/empty.py", reason, + count), + "the file must stay under the './proj/…' spelling"); + check(reason == "read_empty" && count == 2, + "a './proj/…' spelling must be recognised: the re-attempt has to " + "land on the existing row (1 → 2)"); + check(parseFailureRowCount(db, pid_rl) == 1, + "a './proj/…' spelling must not add a second parse_failures row"); + check(entityRowsForName(db, pid_rl, "k") == 1, + "…and must not duplicate the symbol either"); + sqlite3_close(db); + + removeDb(rl_db); + fs::remove_all(rl_dir); + + // ── 8. parse_failures must be READABLE and CLEARABLE ───────── + // + // The table is written by every index path and drives the fail-fast skip, + // yet nothing exported it: store::getParseFailuresJson and + // store::resetParseFailures had no caller at all, and the + // `codescope parse-failures` / `reset-failures` commands the comments and + // the README referred to did not exist. engine_get_parse_failures and + // engine_reset_parse_failures are that entry point. + const std::string pf_dir = "/tmp/test_pf_readable"; + const std::string pf_db = "/tmp/test_pf_readable.db"; + fs::remove_all(pf_dir); + removeDb(pf_db); + fs::create_directories(pf_dir); + writeFile(pf_dir + "/empty.py", nullptr); + + g_engine = engine_create(pf_db.c_str()); + check(g_engine != nullptr, "engine_init (parse-failures ffi)"); + uint64_t pid_pf = + engine_create_project(g_engine, pf_dir.c_str(), "readable"); + check(pid_pf > 0, "create_project (parse-failures ffi)"); + char *rpf = + engine_index_project(g_engine, pid_pf, pf_dir.c_str(), nullptr); + check(rpf != nullptr && strstr(rpf, "\"ok\":true") != nullptr, + "readable run: project index ok"); + engine_free_string(rpf); + + char *listed = engine_get_parse_failures(g_engine, pid_pf, 10); + check(listed != nullptr, "engine_get_parse_failures must return JSON"); + check(strstr(listed, "\"ok\":true") != nullptr, + "reading parse_failures must report ok:true"); + check(strstr(listed, "empty.py") != nullptr, + "the unparseable file must be listed"); + check(strstr(listed, "read_empty") != nullptr, + "…together with its failure reason"); + engine_free_string(listed); + + char *cleared = engine_reset_parse_failures(g_engine, pid_pf); + check(cleared != nullptr && strstr(cleared, "\"ok\":true") != nullptr, + "engine_reset_parse_failures must report ok:true"); + check(strstr(cleared, "\"removed\":1") != nullptr, + "the reset must report how many rows it removed"); + engine_free_string(cleared); + + char *after = engine_get_parse_failures(g_engine, pid_pf, 10); + check(after != nullptr && + strstr(after, "\"parse_failures\":[]") != nullptr, + "the table must be empty after the reset"); + engine_free_string(after); + engine_destroy(g_engine); + g_engine = nullptr; + removeDb(pf_db); + fs::remove_all(pf_dir); + + printf("\n=== parse failures test passed ===\n"); + return 0; +} diff --git a/engine/tests/test_project_id.cpp b/engine/tests/test_project_id.cpp index c6f56db..9d9b8af 100644 --- a/engine/tests/test_project_id.cpp +++ b/engine/tests/test_project_id.cpp @@ -16,6 +16,7 @@ #include #include #include +#include "test_engine_handle.h" static void check(bool cond, const char *msg) { @@ -35,8 +36,7 @@ int main() std::filesystem::remove_all(proj_a_dir); std::filesystem::create_directories(proj_a_dir); { - std::string fpath = - std::string(proj_a_dir) + "/hello.c"; + std::string fpath = std::string(proj_a_dir) + "/hello.c"; FILE *f = fopen(fpath.c_str(), "w"); check(f != nullptr, "fopen hello.c"); fputs("int add(int a, int b) { return a + b; }\n" @@ -45,14 +45,17 @@ int main() fclose(f); } - check(engine_init(db_path) == 0, "engine_init"); + g_engine = engine_create(db_path); + check(g_engine != nullptr, "engine_init"); // ── Test 1: createProject returns a valid id ── - uint64_t pid_a = engine_create_project(proj_a_dir, "project-a"); + uint64_t pid_a = + engine_create_project(g_engine, proj_a_dir, "project-a"); check(pid_a > 0, "create_project A"); // ── Test 2: createProject is idempotent ── - uint64_t pid_a_again = engine_create_project(proj_a_dir, "project-a"); + uint64_t pid_a_again = + engine_create_project(g_engine, proj_a_dir, "project-a"); check(pid_a == pid_a_again, "create_project idempotent: same root_path returns same id"); @@ -68,7 +71,8 @@ int main() "createProject after prior insert returns correct existing id"); // ── Index project A so it has graph_nodes ── - char *idx_result = engine_index_project(pid_a, proj_a_dir, nullptr); + char *idx_result = + engine_index_project(g_engine, pid_a, proj_a_dir, nullptr); check(idx_result != nullptr, "index_project A null"); check(strstr(idx_result, "\"ok\":true") != nullptr, "index_project A ok"); @@ -81,7 +85,7 @@ int main() usleep(200000); // ── Test 4: getProjectNodeCount returns > 0 for project A ── - uint64_t node_count_a = engine_get_project_node_count(pid_a); + uint64_t node_count_a = engine_get_project_node_count(g_engine, pid_a); check(node_count_a > 0, "get_project_node_count A > 0 (has indexed data)"); @@ -90,50 +94,48 @@ int main() std::filesystem::remove_all(proj_b_dir); std::filesystem::create_directories(proj_b_dir); - uint64_t pid_b = engine_create_project(proj_b_dir, "project-b-shell"); - check(pid_b > pid_a, - "project B has higher id than A (empty shell)"); - check(pid_b != pid_a, - "project B id differs from A"); + uint64_t pid_b = + engine_create_project(g_engine, proj_b_dir, "project-b-shell"); + check(pid_b > pid_a, "project B has higher id than A (empty shell)"); + check(pid_b != pid_a, "project B id differs from A"); // ── Test 6: getProjectNodeCount returns 0 for empty project B ── - uint64_t node_count_b = engine_get_project_node_count(pid_b); - check(node_count_b == 0, - "get_project_node_count B == 0 (empty shell)"); + uint64_t node_count_b = engine_get_project_node_count(g_engine, pid_b); + check(node_count_b == 0, "get_project_node_count B == 0 (empty shell)"); // ── Test 7: getLatestProjectId returns A (has data), not B // (empty shell with higher id). This is the core fix — // previously MAX(id) returned B, causing queries to read // empty data. ── - uint64_t latest = engine_get_latest_project_id(); + uint64_t latest = engine_get_latest_project_id(g_engine); check(latest == pid_a, "get_latest_project_id returns project with data (A), " "not empty shell (B)"); // ── Test 8: engine_get_project_id_by_path looks up by rootPath ── - uint64_t lookup_a = engine_get_project_id_by_path(proj_a_dir); + uint64_t lookup_a = engine_get_project_id_by_path(g_engine, proj_a_dir); check(lookup_a == pid_a, "get_project_id_by_path returns correct id for A"); - uint64_t lookup_b = engine_get_project_id_by_path(proj_b_dir); + uint64_t lookup_b = engine_get_project_id_by_path(g_engine, proj_b_dir); check(lookup_b == pid_b, "get_project_id_by_path returns correct id for B"); uint64_t lookup_none = - engine_get_project_id_by_path("/nonexistent/path"); + engine_get_project_id_by_path(g_engine, "/nonexistent/path"); check(lookup_none == 0, "get_project_id_by_path returns 0 for non-existent path"); // ── Test 9: createProject with a different root_path creates a // new project (not reusing A or B) ── - uint64_t pid_c = - engine_create_project("/tmp/test_project_id_c", "project-c"); - check(pid_c > pid_b, - "create_project C has id higher than B"); + uint64_t pid_c = engine_create_project( + g_engine, "/tmp/test_project_id_c", "project-c"); + check(pid_c > pid_b, "create_project C has id higher than B"); check(pid_c != pid_a && pid_c != pid_b, "create_project C is a new project"); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; // Cleanup std::filesystem::remove_all(proj_a_dir); diff --git a/engine/tests/test_project_state.cpp b/engine/tests/test_project_state.cpp index 9ac3867..15690a6 100644 --- a/engine/tests/test_project_state.cpp +++ b/engine/tests/test_project_state.cpp @@ -21,9 +21,10 @@ // 13. Cleanup: close store, unlink temp DB #include "../src/model/project_state_builder.h" +#include "../src/model/state_builder.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -38,9 +39,8 @@ static const char *kDbPath = "/tmp/test_project_state.db"; /// Insert a graph_node function row so semantic_fact.function_id FK /// is satisfied. -static void insertFunction(GraphStore &store, uint64_t project_id, - int64_t id, const char *name, - const char *file_path) +static void insertFunction(GraphStore &store, uint64_t project_id, int64_t id, + const char *name, const char *file_path) { sqlite3 *db = store.handle(); const char *sql = @@ -49,23 +49,22 @@ static void insertFunction(GraphStore &store, uint64_t project_id, "start_row, start_col, end_row, end_col) " "VALUES (?,?,0,0,?,'',?,'go',1,0,1000,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, file_path, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } /// Insert a semantic_fact row directly. The test controls the exact /// (category, primitive, kind) triple to exercise each rule. -static void -insertFact(GraphStore &store, uint64_t project_id, - uint64_t function_id, const char *category, - const char *primitive, const char *kind, const char *symbol, - double confidence, const char *detail_json) +static void insertFact(GraphStore &store, uint64_t project_id, + uint64_t function_id, const char *category, + const char *primitive, const char *kind, + const char *symbol, double confidence, + const char *detail_json) { sqlite3 *db = store.handle(); const char *sql = @@ -74,8 +73,7 @@ insertFact(GraphStore &store, uint64_t project_id, " symbol, confidence, detail_json) " "VALUES (?,?,?,?,?,?,?,?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, static_cast(function_id)); sqlite3_bind_text(stmt, 3, category, -1, SQLITE_TRANSIENT); @@ -84,38 +82,33 @@ insertFact(GraphStore &store, uint64_t project_id, sqlite3_bind_text(stmt, 6, symbol, -1, SQLITE_TRANSIENT); sqlite3_bind_double(stmt, 7, confidence); if (detail_json && *detail_json) - sqlite3_bind_text(stmt, 8, detail_json, -1, - SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 8, detail_json, -1, SQLITE_TRANSIENT); else sqlite3_bind_null(stmt, 8); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } /// Insert a capability_state row. -static void -insertCapabilityState(GraphStore &store, uint64_t project_id, - const char *name, const char *state) +static void insertCapabilityState(GraphStore &store, uint64_t project_id, + const char *name, const char *state) { sqlite3 *db = store.handle(); - const char *sql = - "INSERT INTO capability_state " - "(project_id, name, state, entities, evidence) " - "VALUES (?,?,?,'[]','[]')"; + const char *sql = "INSERT INTO capability_state " + "(project_id, name, state, entities, evidence) " + "VALUES (?,?,?,'[]','[]')"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 3, state, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } /// Insert an architecture_state row with N violations. -static void -insertArchitectureState(GraphStore &store, uint64_t project_id, - const char *layer, int violations) +static void insertArchitectureState(GraphStore &store, uint64_t project_id, + const char *layer, int violations) { sqlite3 *db = store.handle(); const char *sql = @@ -123,14 +116,13 @@ insertArchitectureState(GraphStore &store, uint64_t project_id, "(project_id, layer, violations, compliance, evidence) " "VALUES (?,?,?,?, '[]')"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, layer, -1, SQLITE_TRANSIENT); sqlite3_bind_int(stmt, 3, violations); double compliance = (violations > 0) ? 0.0 : 1.0; sqlite3_bind_double(stmt, 4, compliance); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -139,18 +131,18 @@ insertArchitectureState(GraphStore &store, uint64_t project_id, static std::string findRulesDir() { const char *candidates[] = { - "engine/src/evidence/rules", - "../src/evidence/rules", - "../../engine/src/evidence/rules", - "../../../engine/src/evidence/rules", - }; + "engine/src/evidence/rules", + "../src/evidence/rules", + "../../engine/src/evidence/rules", + "../../../engine/src/evidence/rules", + }; for (const char *cand : candidates) { std::error_code ec; if (!std::filesystem::is_directory(cand, ec)) continue; bool has_json = false; - for (const auto &entry : std::filesystem::directory_iterator( - cand, ec)) { + for (const auto &entry : + std::filesystem::directory_iterator(cand, ec)) { if (entry.is_regular_file() && entry.path().extension() == ".json") { has_json = true; @@ -167,21 +159,18 @@ static std::string findRulesDir() /// SemanticFactExtractor::buildDetailJson. static std::string detailJson(int line, const std::string &snippet) { - return "{\"line\":" + std::to_string(line) + - ",\"snippet\":\"" + snippet + - "\",\"related_symbol\":\"\"}"; + return "{\"line\":" + std::to_string(line) + ",\"snippet\":\"" + + snippet + "\",\"related_symbol\":\"\"}"; } /// Count project_state rows for a project. Used for idempotency check. -static int countProjectStateRows(GraphStore &store, - uint64_t project_id) +static int countProjectStateRows(GraphStore &store, uint64_t project_id) { sqlite3 *db = store.handle(); const char *sql = "SELECT COUNT(*) FROM project_state WHERE project_id=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); int count = 0; if (sqlite3_step(stmt) == SQLITE_ROW) @@ -192,15 +181,13 @@ static int countProjectStateRows(GraphStore &store, /// Read the persisted confidence for a project. Returns -1.0 if no /// row exists (so the caller can distinguish "missing" from "0.0"). -static double readPersistedConfidence(GraphStore &store, - uint64_t project_id) +static double readPersistedConfidence(GraphStore &store, uint64_t project_id) { sqlite3 *db = store.handle(); const char *sql = "SELECT confidence FROM project_state WHERE project_id=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); double out = -1.0; if (sqlite3_step(stmt) == SQLITE_ROW) @@ -210,20 +197,17 @@ static double readPersistedConfidence(GraphStore &store, } /// Read the persisted snapshot_json for a project. -static std::string readPersistedSnapshot(GraphStore &store, - uint64_t project_id) +static std::string readPersistedSnapshot(GraphStore &store, uint64_t project_id) { sqlite3 *db = store.handle(); const char *sql = "SELECT snapshot_json FROM project_state WHERE project_id=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); std::string out; if (sqlite3_step(stmt) == SQLITE_ROW) { - const unsigned char *t = - sqlite3_column_text(stmt, 0); + const unsigned char *t = sqlite3_column_text(stmt, 0); if (t) out = reinterpret_cast(t); } @@ -231,6 +215,57 @@ static std::string readPersistedSnapshot(GraphStore &store, return out; } +/// Insert an architecture_edge row: an ordinary cross-module dependency +/// (callee_module / caller_module hold module NAMES — see +/// StateBuilder::buildArchitectureState). +static void insertArchEdge(GraphStore &store, uint64_t project_id, + const char *callee_module, const char *caller_module) +{ + sqlite3 *db = store.handle(); + const char *sql = + "INSERT INTO architecture_edge " + "(project_id, caller_module, callee_module, entity_id) " + "VALUES (?,?,?,1)"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_text(stmt, 2, caller_module, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 3, callee_module, -1, SQLITE_TRANSIENT); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); +} + +/// Aggregate architecture_state for a project: row count, summed +/// violations, summed cross_module_edges, and the lowest compliance flag +/// (every row must sit at 1.0 for a project with no layer model). +struct ArchAgg { + int64_t rows = 0; + int64_t violations = 0; + int64_t cross_module_edges = 0; + double min_compliance = 1.0; +}; + +static ArchAgg readArchAgg(GraphStore &store, uint64_t project_id) +{ + sqlite3 *db = store.handle(); + const char *sql = "SELECT COUNT(*), COALESCE(SUM(violations), 0), " + " COALESCE(SUM(cross_module_edges), 0), " + " COALESCE(MIN(compliance), 1.0) " + "FROM architecture_state WHERE project_id=?"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + ArchAgg agg; + if (sqlite3_step(stmt) == SQLITE_ROW) { + agg.rows = sqlite3_column_int64(stmt, 0); + agg.violations = sqlite3_column_int64(stmt, 1); + agg.cross_module_edges = sqlite3_column_int64(stmt, 2); + agg.min_compliance = sqlite3_column_double(stmt, 3); + } + sqlite3_finalize(stmt); + return agg; +} + int main() { unlink(kDbPath); @@ -239,8 +274,7 @@ int main() // matter where the test binary is invoked from. std::string rules_dir = findRulesDir(); if (rules_dir.empty()) { - fprintf(stderr, - "FAIL: cannot find evidence rules directory\n"); + fprintf(stderr, "FAIL: cannot find evidence rules directory\n"); return 1; } setenv("CODESCOPE_RULES_DIR", rules_dir.c_str(), 1); @@ -252,9 +286,8 @@ int main() store.error().c_str()); return 1; } - uint64_t pid = - store.createProject("/tmp", "test_project_state"); - assert(pid > 0); + uint64_t pid = store.createProject("/tmp", "test_project_state"); + CHECK(pid > 0); // ── Insert graph_node functions (semantic_fact.function_id FK) insertFunction(store, pid, 100, "AcquireLeak", "/src/sync.go"); @@ -264,25 +297,21 @@ int main() // ── Insert semantic_fact rows for each category ───────────── // sync: mutex lock WITHOUT defer_unlock (1 issue) - insertFact(store, pid, 100, "sync", "mutex", "lock", "m.Lock", - 1.0, + insertFact(store, pid, 100, "sync", "mutex", "lock", "m.Lock", 1.0, detailJson(5, "m.Lock (/src/sync.go)").c_str()); // memory: cstring alloc WITHOUT free (1 issue per function) - insertFact(store, pid, 200, "memory", "cstring", "alloc", - "C.CString", 1.0, - detailJson(9, "C.CString (/src/cgo.go)").c_str()); + insertFact(store, pid, 200, "memory", "cstring", "alloc", "C.CString", + 1.0, detailJson(9, "C.CString (/src/cgo.go)").c_str()); // pattern: TODO marker (1 issue) insertFact(store, pid, 300, "pattern", "todo", "marker", "TODO: implement", 1.0, - detailJson(11, "TODO: implement (/src/todo.go)") - .c_str()); + detailJson(11, "TODO: implement (/src/todo.go)").c_str()); // framework: gin router detection (1 fact, no rule-level issue) - insertFact(store, pid, 400, "framework", "gin", "router", - "gin.New", 1.0, - detailJson(1, "gin.New (/src/api.go)").c_str()); + insertFact(store, pid, 400, "framework", "gin", "router", "gin.New", + 1.0, detailJson(1, "gin.New (/src/api.go)").c_str()); // ── Insert state rows ─────────────────────────────────────── insertCapabilityState(store, pid, "Auth", "Implemented"); @@ -293,14 +322,14 @@ int main() { ProjectStateBuilder builder(&store); bool ok = builder.build(pid); - assert(ok); + CHECK(ok); printf("Test 1 (build returned true): PASS\n"); } // ── Test 2: project_state has exactly 1 row ───────────────── { int count = countProjectStateRows(store, pid); - assert(count == 1); + CHECK(count == 1); printf("Test 2 (project_state rows == 1): PASS\n"); } @@ -308,7 +337,7 @@ int main() double confidence = -1.0; { confidence = readPersistedConfidence(store, pid); - assert(confidence >= 0.0 && confidence <= 1.0); + CHECK(confidence >= 0.0 && confidence <= 1.0); printf("Test 3 (confidence in [0,1]: %.4f): PASS\n", confidence); } @@ -317,23 +346,19 @@ int main() std::string snapshot; { snapshot = readPersistedSnapshot(store, pid); - assert(!snapshot.empty()); + CHECK(!snapshot.empty()); // Required top-level keys per plan §6.2. - assert(snapshot.find("\"overall\"") != std::string::npos); - assert(snapshot.find("\"sync\"") != std::string::npos); - assert(snapshot.find("\"memory\"") != std::string::npos); - assert(snapshot.find("\"pattern\"") != std::string::npos); + CHECK(snapshot.find("\"overall\"") != std::string::npos); + CHECK(snapshot.find("\"sync\"") != std::string::npos); + CHECK(snapshot.find("\"memory\"") != std::string::npos); + CHECK(snapshot.find("\"pattern\"") != std::string::npos); // Confidence field should be present in "overall". - assert(snapshot.find("\"confidence\"") != - std::string::npos); + CHECK(snapshot.find("\"confidence\"") != std::string::npos); // Architecture and capability should be present. - assert(snapshot.find("\"architecture\"") != - std::string::npos); - assert(snapshot.find("\"capability\"") != - std::string::npos); + CHECK(snapshot.find("\"architecture\"") != std::string::npos); + CHECK(snapshot.find("\"capability\"") != std::string::npos); // last_updated timestamp. - assert(snapshot.find("\"last_updated\"") != - std::string::npos); + CHECK(snapshot.find("\"last_updated\"") != std::string::npos); printf("Test 4 (snapshot contains required keys): " "PASS\n"); } @@ -342,8 +367,8 @@ int main() { ProjectStateBuilder builder(&store); std::string s = builder.getSnapshotJson(pid); - assert(!s.empty()); - assert(s == snapshot); + CHECK(!s.empty()); + CHECK(s == snapshot); printf("Test 5 (getSnapshotJson non-empty): PASS\n"); } @@ -351,22 +376,21 @@ int main() { ProjectStateBuilder builder(&store); double c = builder.getConfidence(pid); - assert(c >= 0.0 && c <= 1.0); - assert(c == confidence); - printf("Test 6 (getConfidence in [0,1]: %.4f): PASS\n", - c); + CHECK(c >= 0.0 && c <= 1.0); + CHECK(c == confidence); + printf("Test 6 (getConfidence in [0,1]: %.4f): PASS\n", c); } // ── Test 7: idempotency — build() twice → still 1 row ─────── { ProjectStateBuilder builder(&store); bool ok = builder.build(pid); - assert(ok); + CHECK(ok); int count = countProjectStateRows(store, pid); - assert(count == 1); + CHECK(count == 1); // Confidence should be unchanged (deterministic). double c = readPersistedConfidence(store, pid); - assert(c == confidence); + CHECK(c == confidence); printf("Test 7 (idempotency: still 1 row, same " "confidence): PASS\n"); } @@ -375,7 +399,7 @@ int main() { ProjectStateBuilder builder(&store); std::string s = builder.getSnapshotJson(pid + 999); - assert(s.empty()); + CHECK(s.empty()); printf("Test 8 (getSnapshotJson unknown project -> " "empty): PASS\n"); } @@ -384,13 +408,112 @@ int main() { ProjectStateBuilder builder(&store); double c = builder.getConfidence(pid + 999); - assert(c == 0.0); + CHECK(c == 0.0); printf("Test 9 (getConfidence unknown project -> " "0.0): PASS\n"); } + // ── Test 10: buildArchitectureState writes honest numbers ─── + // Regression for review #3: architecture_edge rows are ordinary + // cross-module dependencies (callee_module / caller_module hold module + // NAMES, and no layer model or direction test exists), so + // `violations` must stay 0, the count must land in + // cross_module_edges, and compliance must stay 1.0. The second run + // covers idempotency: the old INSERT OR IGNORE had no unique key to + // ignore on, so every rebuild appended a full copy of every row. + { + // Two module pairs: mod_a->mod_b (2 edges) + mod_b->mod_c. + insertArchEdge(store, pid, "mod_a", "mod_b"); + insertArchEdge(store, pid, "mod_a", "mod_b"); + insertArchEdge(store, pid, "mod_b", "mod_c"); + + StateBuilder sb(&store, pid); + int64_t n = sb.buildArchitectureState(); + CHECK(n == 2); // one row per (lower, upper) pair + ArchAgg agg = readArchAgg(store, pid); + CHECK(agg.rows == 2); + CHECK(agg.violations == 0); + CHECK(agg.cross_module_edges == 3); + CHECK(agg.min_compliance == 1.0); + + // Rebuild must replace, not append. (It also removes the + // hand-inserted violations=2 fixture row from step 5 — + // architecture_state is derived state.) + n = sb.buildArchitectureState(); + CHECK(n == 2); + agg = readArchAgg(store, pid); + CHECK(agg.rows == 2); + CHECK(agg.violations == 0); + CHECK(agg.cross_module_edges == 3); + printf("Test 10 (buildArchitectureState honest + " + "idempotent): PASS\n"); + } + + // ── Test 11: project_state.architecture reports both ──────── + { + ProjectStateBuilder builder(&store); + bool ok = builder.build(pid); + CHECK(ok); + std::string s = readPersistedSnapshot(store, pid); + CHECK(s.find("\"violations\":0,\"cross_module_edges\":3") != + std::string::npos); + printf("Test 11 (project_state.architecture reports " + "cross_module_edges): PASS\n"); + } + + // ── Test 12: migration repairs pre-fix rows on reopen ─────── + // Simulate what the pre-fix builder wrote: `violations` holds the + // cross-module count, compliance is 0.0, and duplicate rows from + // the old non-idempotent rebuild are stacked on the same layer key. + // Reopening runs createSchema + runSchemaMigrations, which must + // dedup AND move the count — even though the column already exists + // (the original migration only corrected rows in the same startup + // that added the column, so a downgrade/upgrade cycle never fixed + // them). + { + const char *legacy_sql = + "INSERT INTO architecture_state " + "(project_id, layer, violations, compliance, evidence) " + "VALUES (?, 'legacy->pair', 5, 0.0, '[]')"; + sqlite3_stmt *stmt = nullptr; + sqlite3 *db = store.handle(); + CHECK(sqlite3_prepare_v2(db, legacy_sql, -1, &stmt, nullptr) == + SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(pid)); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); + CHECK(sqlite3_prepare_v2(db, legacy_sql, -1, &stmt, nullptr) == + SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(pid)); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); + + store.close(); + CHECK(store.open(kDbPath)); + + db = store.handle(); + const char *q = "SELECT COUNT(*), " + " COALESCE(MAX(violations), 0), " + " COALESCE(MAX(cross_module_edges), 0), " + " COALESCE(MIN(compliance), 1.0) " + "FROM architecture_state " + "WHERE project_id=? AND layer='legacy->pair'"; + stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, q, -1, &stmt, nullptr) == + SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(pid)); + CHECK(sqlite3_step(stmt) == SQLITE_ROW); + CHECK(sqlite3_column_int64(stmt, 0) == 1); // deduped + CHECK(sqlite3_column_int64(stmt, 1) == 0); // moved out + CHECK(sqlite3_column_int64(stmt, 2) == 5); // preserved + CHECK(sqlite3_column_double(stmt, 3) == 1.0); // cleared + sqlite3_finalize(stmt); + printf("Test 12 (migration dedups + moves pre-fix rows " + "on reopen): PASS\n"); + } + store.close(); unlink(kDbPath); printf("\nAll project_state tests passed.\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_qualified_id_ast.cpp b/engine/tests/test_qualified_id_ast.cpp index c57583f..79db2b8 100644 --- a/engine/tests/test_qualified_id_ast.cpp +++ b/engine/tests/test_qualified_id_ast.cpp @@ -32,6 +32,7 @@ #include #include #include +#include "test_engine_handle.h" static inline void check(bool cond, const char *msg) { @@ -46,10 +47,9 @@ static inline void check(bool cond, const char *msg) static int recordKindByName(sqlite3 *db, int64_t pid, const char *name) { sqlite3_stmt *st = nullptr; - const char *sql = - "SELECT kind FROM semantic_records " - "WHERE project_id=? AND name=? " - "ORDER BY kind LIMIT 1"; + const char *sql = "SELECT kind FROM semantic_records " + "WHERE project_id=? AND name=? " + "ORDER BY kind LIMIT 1"; if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) != SQLITE_OK) return -1; sqlite3_bind_int64(st, 1, pid); @@ -121,14 +121,15 @@ int main() { // ── Index C++ fixture ─────────────────────────────────────── char cpp_db[] = "/tmp/test_qualified_id_cpp.db"; unlink(cpp_db); - check(engine_init(cpp_db) == 0, "engine_init cpp"); + g_engine = engine_create(cpp_db); + check(g_engine != nullptr, "engine_init cpp"); uint64_t cpp_pid = - engine_create_project(cpp_proj_dir, "cpp-qualified"); + engine_create_project(g_engine, cpp_proj_dir, "cpp-qualified"); check(cpp_pid > 0, "create_project cpp"); char *cpp_idx = - engine_index_project(cpp_pid, cpp_proj_dir, nullptr); + engine_index_project(g_engine, cpp_pid, cpp_proj_dir, nullptr); check(cpp_idx != nullptr, "index_project cpp returns non-null"); check(strstr(cpp_idx, "\"ok\":true") != nullptr, "index_project cpp ok"); @@ -140,48 +141,43 @@ int main() { // ── Assertion 1: buildCallEdgesSQL is a Function (kind=0) ── // Before the fix it was emitted as Variable (kind=6) because // extractName had no qualified_identifier branch. - int bces_kind = - recordKindByName(db, cpp_pid, "buildCallEdgesSQL"); - fprintf(stderr, "buildCallEdgesSQL kind = %d (expect 0)\n", - bces_kind); - check(bces_kind == 0, - "buildCallEdgesSQL must be Function (kind=0), " - "got kind=6 (Variable) before fix"); + int bces_kind = recordKindByName(db, cpp_pid, "buildCallEdgesSQL"); + fprintf(stderr, "buildCallEdgesSQL kind = %d (expect 0)\n", bces_kind); + check(bces_kind == 0, "buildCallEdgesSQL must be Function (kind=0), " + "got kind=6 (Variable) before fix"); // ── Assertion 2: buildGraph is a Function (kind=0) ───────── int bg_kind = recordKindByName(db, cpp_pid, "buildGraph"); - check(bg_kind == 0, - "buildGraph must be Function (kind=0)"); + check(bg_kind == 0, "buildGraph must be Function (kind=0)"); // ── Assertion 3: call edge buildGraph → buildCallEdgesSQL ── // This is the "this->buildCallEdgesSQL(pid)" call inside // buildGraph. Before the fix, resolveSymbol("buildCallEdgesSQL") // failed (never defineSymbol'd) → ref_original_id=0 → P1 // skipped → 0 edges. - int edge_bg_bces = edgeCount(db, cpp_pid, "buildGraph", - "buildCallEdgesSQL", 1); + int edge_bg_bces = + edgeCount(db, cpp_pid, "buildGraph", "buildCallEdgesSQL", 1); fprintf(stderr, "buildGraph -> buildCallEdgesSQL edges = %d\n", edge_bg_bces); - check(edge_bg_bces >= 1, - "buildGraph must call buildCallEdgesSQL " - "(this->method() intra-file edge)"); + check(edge_bg_bces >= 1, "buildGraph must call buildCallEdgesSQL " + "(this->method() intra-file edge)"); // ── Assertion 4: call edge main → buildGraph ──────────────── - int edge_main_bg = edgeCount(db, cpp_pid, "main", - "buildGraph", 1); + int edge_main_bg = edgeCount(db, cpp_pid, "main", "buildGraph", 1); check(edge_main_bg >= 1, "main must call buildGraph (gs.method() edge)"); // ── Assertion 5: engine_get_callees(buildGraph) ───────────── - char *callees_bg = engine_get_callees(cpp_pid, "buildGraph", nullptr); + char *callees_bg = + engine_get_callees(g_engine, cpp_pid, "buildGraph", nullptr); check(callees_bg != nullptr, "get_callees(buildGraph) non-null"); check(strstr(callees_bg, "buildCallEdgesSQL") != nullptr, "get_callees(buildGraph) must contain buildCallEdgesSQL"); engine_free_string(callees_bg); // ── Assertion 6: engine_get_callers(buildCallEdgesSQL) ────── - char *callers_bces = - engine_get_callers(cpp_pid, "buildCallEdgesSQL", nullptr); + char *callers_bces = engine_get_callers(g_engine, cpp_pid, + "buildCallEdgesSQL", nullptr); check(callers_bces != nullptr, "get_callers(buildCallEdgesSQL) non-null"); check(strstr(callers_bces, "buildGraph") != nullptr, @@ -189,7 +185,8 @@ int main() { engine_free_string(callers_bces); sqlite3_close(db); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; // ── Setup: Python fixture ─────────────────────────────────── // Exercises Bug 3-D for Python: self.fig.add_trace() should @@ -206,16 +203,22 @@ int main() { // For a positive intra-file edge test, use a chained // attribute call to a locally-defined method: // self.helper.compute() where compute is defined in class. + // + // This fixture USED to be the non-chained `self.compute(10)`, which + // passed even though extractAttributeName returned the INNER receiver + // segment ("helper") for a chained callee — so the test that existed to + // catch that bug could not catch it. Keep the real chained form: + // `self.helper.compute()` must resolve to "compute", and the edge + // run → compute below is the assertion that proves it. const char *py_code = R"( class Worker: def compute(self, x): return x * 2 def run(self): - # Chained attribute: self.helper is the attribute, - # but here we use a direct self.compute() to verify - # the simple attribute path still resolves. - return self.compute(10) + # Chained attribute callee: the resolved name must be the + # LAST segment ("compute"), not the receiver ("helper"). + return self.helper.compute(10) )"; f = fopen(py_path, "w"); check(f != nullptr, "fopen py"); @@ -225,38 +228,38 @@ class Worker: // ── Index Python fixture ──────────────────────────────────── char py_db[] = "/tmp/test_qualified_id_py.db"; unlink(py_db); - check(engine_init(py_db) == 0, "engine_init py"); + g_engine = engine_create(py_db); + check(g_engine != nullptr, "engine_init py"); uint64_t py_pid = - engine_create_project(py_proj_dir, "py-chained"); + engine_create_project(g_engine, py_proj_dir, "py-chained"); check(py_pid > 0, "create_project py"); char *py_idx = - engine_index_project(py_pid, py_proj_dir, nullptr); + engine_index_project(g_engine, py_pid, py_proj_dir, nullptr); check(py_idx != nullptr, "index_project py returns non-null"); - check(strstr(py_idx, "\"ok\":true") != nullptr, - "index_project py ok"); + check(strstr(py_idx, "\"ok\":true") != nullptr, "index_project py ok"); engine_free_string(py_idx); check(sqlite3_open(py_db, &db) == SQLITE_OK, "sqlite3_open py"); // ── Assertion 7: run → compute call edge exists ───────────── - int edge_run_compute = edgeCount(db, py_pid, "run", - "compute", 1); - fprintf(stderr, "run -> compute edges = %d\n", - edge_run_compute); + int edge_run_compute = edgeCount(db, py_pid, "run", "compute", 1); + fprintf(stderr, "run -> compute edges = %d\n", edge_run_compute); check(edge_run_compute >= 1, "run must call compute (self.method() intra-file edge)"); // ── Assertion 8: engine_get_callees(run) ──────────────────── - char *callees_run = engine_get_callees(py_pid, "run", nullptr); + char *callees_run = + engine_get_callees(g_engine, py_pid, "run", nullptr); check(callees_run != nullptr, "get_callees(run) non-null"); check(strstr(callees_run, "compute") != nullptr, "get_callees(run) must contain compute"); engine_free_string(callees_run); sqlite3_close(db); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; // ── Cleanup ───────────────────────────────────────────────── std::filesystem::remove_all(cpp_proj_dir); diff --git a/engine/tests/test_query_algorithms.cpp b/engine/tests/test_query_algorithms.cpp index e3420da..f165d4e 100644 --- a/engine/tests/test_query_algorithms.cpp +++ b/engine/tests/test_query_algorithms.cpp @@ -21,9 +21,10 @@ #include "../src/query/impact_analysis.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include +#include #include #include #include @@ -45,13 +46,13 @@ static void insertGraphNode(store::GraphStore &store, uint64_t project_id, "end_row, end_col) " "VALUES (?, ?, 0, ?, ?, ?, 'cpp', 0, 0, 0, 0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 5, file_path, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -66,11 +67,11 @@ static void insertCallEdge(store::GraphStore &store, uint64_t project_id, "INSERT INTO relation (project_id, source_id, target_id, type) " "VALUES (?, ?, ?, 1)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, source_id); sqlite3_bind_int64(stmt, 3, target_id); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -91,7 +92,7 @@ static void syncSQLite(store::GraphStore &store, uint64_t project_id) // adjacency) is the sole graph backend. Rebuild the CSR so the // freshly-inserted edges are visible to the CSR-based queries. bool ok = store.buildCSR(project_id); - assert(ok); + CHECK(ok); (void)project_id; } @@ -110,11 +111,11 @@ static void testShortestPathDirectEdge(store::GraphStore &store, std::string result = engine.findShortestPath(project_id, 1, 2); printf(" [debug] direct edge: %s\n", result.c_str()); - assert(jsonContains(result, "\"found\":true")); - assert(jsonContains(result, "\"hops\":1")); + CHECK(jsonContains(result, "\"found\":true")); + CHECK(jsonContains(result, "\"hops\":1")); // Path should be [1, 2] — check both node IDs appear in order. - assert(jsonContains(result, - "\"path\":[{\"node_id\":1},{\"node_id\":2}]")); + CHECK(jsonContains(result, + "\"path\":[{\"node_id\":1},{\"node_id\":2}]")); printf(" [PASS] findShortestPath: direct edge (1 hop)\n"); } @@ -132,11 +133,11 @@ static void testShortestPath2Hop(store::GraphStore &store, uint64_t project_id) std::string result = engine.findShortestPath(project_id, 10, 12); printf(" [debug] 2-hop: %s\n", result.c_str()); - assert(jsonContains(result, "\"found\":true")); - assert(jsonContains(result, "\"hops\":2")); - assert(jsonContains(result, - "\"path\":[{\"node_id\":10},{\"node_id\":11}," - "{\"node_id\":12}]")); + CHECK(jsonContains(result, "\"found\":true")); + CHECK(jsonContains(result, "\"hops\":2")); + CHECK(jsonContains(result, + "\"path\":[{\"node_id\":10},{\"node_id\":11}," + "{\"node_id\":12}]")); printf(" [PASS] findShortestPath: 2-hop path\n"); } @@ -156,11 +157,11 @@ static void testShortestPath3Hop(store::GraphStore &store, uint64_t project_id) std::string result = engine.findShortestPath(project_id, 20, 23); printf(" [debug] 3-hop: %s\n", result.c_str()); - assert(jsonContains(result, "\"found\":true")); - assert(jsonContains(result, "\"hops\":3")); - assert(jsonContains(result, - "\"path\":[{\"node_id\":20},{\"node_id\":21}," - "{\"node_id\":22},{\"node_id\":23}]")); + CHECK(jsonContains(result, "\"found\":true")); + CHECK(jsonContains(result, "\"hops\":3")); + CHECK(jsonContains(result, + "\"path\":[{\"node_id\":20},{\"node_id\":21}," + "{\"node_id\":22},{\"node_id\":23}]")); printf(" [PASS] findShortestPath: 3-hop path\n"); } @@ -180,10 +181,10 @@ static void testShortestPathNoPath(store::GraphStore &store, std::string result = engine.findShortestPath(project_id, 30, 33); printf(" [debug] no path: %s\n", result.c_str()); - assert(jsonContains(result, "\"found\":false")); - assert(jsonContains(result, "\"hops\":0")); + CHECK(jsonContains(result, "\"found\":false")); + CHECK(jsonContains(result, "\"hops\":0")); // Path should contain only the source node. - assert(jsonContains(result, "\"path\":[{\"node_id\":30}]")); + CHECK(jsonContains(result, "\"path\":[{\"node_id\":30}]")); printf(" [PASS] findShortestPath: no path (disconnected)\n"); } @@ -197,9 +198,9 @@ static void testShortestPathSelfToSelf(store::GraphStore &store, std::string result = engine.findShortestPath(project_id, 40, 40); printf(" [debug] self-to-self: %s\n", result.c_str()); - assert(jsonContains(result, "\"found\":true")); - assert(jsonContains(result, "\"hops\":0")); - assert(jsonContains(result, "\"path\":[{\"node_id\":40}]")); + CHECK(jsonContains(result, "\"found\":true")); + CHECK(jsonContains(result, "\"hops\":0")); + CHECK(jsonContains(result, "\"path\":[{\"node_id\":40}]")); printf(" [PASS] findShortestPath: self-to-self (0 hops)\n"); } @@ -224,9 +225,9 @@ static void testShortestPathDepthLimit(store::GraphStore &store, printf(" [debug] depth limit (11 hops, max 10): %s\n", result.c_str()); // 11 hops > kShortestPathMaxDepth (10) → no path found. - assert(jsonContains(result, "\"found\":false")); - assert(jsonContains(result, "\"hops\":0")); - assert(jsonContains(result, "\"path\":[{\"node_id\":50}]")); + CHECK(jsonContains(result, "\"found\":false")); + CHECK(jsonContains(result, "\"hops\":0")); + CHECK(jsonContains(result, "\"path\":[{\"node_id\":50}]")); printf(" [PASS] findShortestPath: depth limit reached\n"); } @@ -251,8 +252,8 @@ static void testShortestPathWithinDepthLimit(store::GraphStore &store, printf(" [debug] exactly 10 hops: %s\n", result.c_str()); // 10 hops == kShortestPathMaxDepth → should be found. - assert(jsonContains(result, "\"found\":true")); - assert(jsonContains(result, "\"hops\":10")); + CHECK(jsonContains(result, "\"found\":true")); + CHECK(jsonContains(result, "\"hops\":10")); printf(" [PASS] findShortestPath: exactly at depth limit (10 hops)\n"); } @@ -278,21 +279,21 @@ static void testImpact1Hop(store::GraphStore &store, uint64_t project_id) printf(" [debug] 1-hop impact: %s\n", result.c_str()); // Should find 1 modified node. - assert(jsonContains(result, "\"modified\":[{\"id\":101")); + CHECK(jsonContains(result, "\"modified\":[{\"id\":101")); // Should find caller at depth 1. - assert(jsonContains(result, "\"callers\":[")); - assert(jsonContains(result, "\"id\":100")); - assert(jsonContains(result, "\"depth\":1")); - assert(jsonContains(result, "\"caller_of\":\"modified_fn\"")); + CHECK(jsonContains(result, "\"callers\":[")); + CHECK(jsonContains(result, "\"id\":100")); + CHECK(jsonContains(result, "\"depth\":1")); + CHECK(jsonContains(result, "\"caller_of\":\"modified_fn\"")); // Should find callee at depth 1. - assert(jsonContains(result, "\"callees\":[")); - assert(jsonContains(result, "\"id\":102")); - assert(jsonContains(result, "\"depth\":1")); - assert(jsonContains(result, "\"callee_of\":\"modified_fn\"")); + CHECK(jsonContains(result, "\"callees\":[")); + CHECK(jsonContains(result, "\"id\":102")); + CHECK(jsonContains(result, "\"depth\":1")); + CHECK(jsonContains(result, "\"callee_of\":\"modified_fn\"")); // Total impacted = modified(1) + callers(1) + callees(1) = 3. - assert(jsonContains(result, "\"total_impacted\":3")); - assert(jsonContains(result, "\"max_depth\":3")); - assert(jsonContains(result, "\"approximation\":\"heuristic\"")); + CHECK(jsonContains(result, "\"total_impacted\":3")); + CHECK(jsonContains(result, "\"max_depth\":3")); + CHECK(jsonContains(result, "\"approximation\":\"heuristic\"")); printf(" [PASS] analyzeChangeImpact: 1-hop callers + callees\n"); } @@ -319,17 +320,17 @@ static void testImpact2Hop(store::GraphStore &store, uint64_t project_id) printf(" [debug] 2-hop impact: %s\n", result.c_str()); // Should find 1 modified node. - assert(jsonContains(result, "\"modified\":[{\"id\":202")); + CHECK(jsonContains(result, "\"modified\":[{\"id\":202")); // Callers: 201 at depth 1, 200 at depth 2. - assert(jsonContains(result, "\"id\":201")); - assert(jsonContains(result, "\"depth\":1")); - assert(jsonContains(result, "\"id\":200")); - assert(jsonContains(result, "\"depth\":2")); + CHECK(jsonContains(result, "\"id\":201")); + CHECK(jsonContains(result, "\"depth\":1")); + CHECK(jsonContains(result, "\"id\":200")); + CHECK(jsonContains(result, "\"depth\":2")); // Callees: 203 at depth 1, 204 at depth 2. - assert(jsonContains(result, "\"id\":203")); - assert(jsonContains(result, "\"id\":204")); + CHECK(jsonContains(result, "\"id\":203")); + CHECK(jsonContains(result, "\"id\":204")); // Total impacted = 1 (modified) + 2 (callers) + 2 (callees) = 5. - assert(jsonContains(result, "\"total_impacted\":5")); + CHECK(jsonContains(result, "\"total_impacted\":5")); printf(" [PASS] analyzeChangeImpact: 2-hop transitive impact\n"); } @@ -360,17 +361,17 @@ static void testImpact3Hop(store::GraphStore &store, uint64_t project_id) printf(" [debug] 3-hop impact: %s\n", result.c_str()); // Should find 1 modified node. - assert(jsonContains(result, "\"modified\":[{\"id\":303")); + CHECK(jsonContains(result, "\"modified\":[{\"id\":303")); // Upstream callers at depths 1, 2, 3. - assert(jsonContains(result, "\"id\":302")); - assert(jsonContains(result, "\"id\":301")); - assert(jsonContains(result, "\"id\":300")); + CHECK(jsonContains(result, "\"id\":302")); + CHECK(jsonContains(result, "\"id\":301")); + CHECK(jsonContains(result, "\"id\":300")); // Downstream callees at depths 1, 2, 3. - assert(jsonContains(result, "\"id\":304")); - assert(jsonContains(result, "\"id\":305")); - assert(jsonContains(result, "\"id\":306")); + CHECK(jsonContains(result, "\"id\":304")); + CHECK(jsonContains(result, "\"id\":305")); + CHECK(jsonContains(result, "\"id\":306")); // Total = 1 (modified) + 3 (callers) + 3 (callees) = 7. - assert(jsonContains(result, "\"total_impacted\":7")); + CHECK(jsonContains(result, "\"total_impacted\":7")); printf(" [PASS] analyzeChangeImpact: 3-hop transitive impact\n"); } @@ -400,26 +401,26 @@ static void testImpactDepthCap(store::GraphStore &store, uint64_t project_id) printf(" [debug] depth cap impact: %s\n", result.c_str()); // Modified node is 404. - assert(jsonContains(result, "\"modified\":[{\"id\":404")); + CHECK(jsonContains(result, "\"modified\":[{\"id\":404")); // Callers within depth 3: 403, 402, 401. Node 400 should NOT appear. - assert(jsonContains(result, "\"id\":403")); - assert(jsonContains(result, "\"id\":402")); - assert(jsonContains(result, "\"id\":401")); + CHECK(jsonContains(result, "\"id\":403")); + CHECK(jsonContains(result, "\"id\":402")); + CHECK(jsonContains(result, "\"id\":401")); // Node 400 is at depth 4 — beyond the cap. It must NOT be in the result. // We check that "id\":400" does not appear (would match 400 but not 400x). // Note: 401-409 all start with "40", so we check the exact ID pattern. - assert(!jsonContains(result, "\"id\":400,")); - assert(!jsonContains(result, "\"id\":400}")); + CHECK(!jsonContains(result, "\"id\":400,")); + CHECK(!jsonContains(result, "\"id\":400}")); // Callees within depth 3: 405, 406, 407. Nodes 408, 409 should NOT appear. - assert(jsonContains(result, "\"id\":405")); - assert(jsonContains(result, "\"id\":406")); - assert(jsonContains(result, "\"id\":407")); - assert(!jsonContains(result, "\"id\":408,")); - assert(!jsonContains(result, "\"id\":408}")); - assert(!jsonContains(result, "\"id\":409,")); - assert(!jsonContains(result, "\"id\":409}")); + CHECK(jsonContains(result, "\"id\":405")); + CHECK(jsonContains(result, "\"id\":406")); + CHECK(jsonContains(result, "\"id\":407")); + CHECK(!jsonContains(result, "\"id\":408,")); + CHECK(!jsonContains(result, "\"id\":408}")); + CHECK(!jsonContains(result, "\"id\":409,")); + CHECK(!jsonContains(result, "\"id\":409}")); // Total = 1 (modified) + 3 (callers) + 3 (callees) = 7. - assert(jsonContains(result, "\"total_impacted\":7")); + CHECK(jsonContains(result, "\"total_impacted\":7")); printf(" [PASS] analyzeChangeImpact: depth cap (3) enforced\n"); } @@ -435,11 +436,11 @@ static void testImpactDisconnectedNode(store::GraphStore &store, printf(" [debug] disconnected: %s\n", result.c_str()); // Should find 1 modified node, no callers, no callees. - assert(jsonContains(result, "\"modified\":[{\"id\":500")); - assert(jsonContains(result, "\"callers\":[]")); - assert(jsonContains(result, "\"callees\":[]")); + CHECK(jsonContains(result, "\"modified\":[{\"id\":500")); + CHECK(jsonContains(result, "\"callers\":[]")); + CHECK(jsonContains(result, "\"callees\":[]")); // Total = 1 (just the modified node). - assert(jsonContains(result, "\"total_impacted\":1")); + CHECK(jsonContains(result, "\"total_impacted\":1")); printf(" [PASS] analyzeChangeImpact: disconnected node\n"); } @@ -451,12 +452,12 @@ static void testImpactEmptyFileList(store::GraphStore &store, query::analyzeChangeImpact(project_id, &store, "[]"); printf(" [debug] empty file list: %s\n", result.c_str()); - assert(jsonContains(result, "\"error\":null")); - assert(jsonContains(result, "\"modified\":[]")); - assert(jsonContains(result, "\"callers\":[]")); - assert(jsonContains(result, "\"callees\":[]")); - assert(jsonContains(result, "\"total_impacted\":0")); - assert(jsonContains(result, "\"max_depth\":3")); + CHECK(jsonContains(result, "\"error\":null")); + CHECK(jsonContains(result, "\"modified\":[]")); + CHECK(jsonContains(result, "\"callers\":[]")); + CHECK(jsonContains(result, "\"callees\":[]")); + CHECK(jsonContains(result, "\"total_impacted\":0")); + CHECK(jsonContains(result, "\"max_depth\":3")); printf(" [PASS] analyzeChangeImpact: empty file list\n"); } @@ -486,27 +487,109 @@ static void testImpactMultipleModifiedFiles(store::GraphStore &store, printf(" [debug] multiple modified: %s\n", result.c_str()); // Should find 2 modified nodes. - assert(jsonContains(result, "\"id\":600")); - assert(jsonContains(result, "\"id\":601")); + CHECK(jsonContains(result, "\"id\":600")); + CHECK(jsonContains(result, "\"id\":601")); // 602 should be a callee at depth 1 (reached from 601). - assert(jsonContains(result, "\"id\":602")); - assert(jsonContains(result, "\"depth\":1")); + CHECK(jsonContains(result, "\"id\":602")); + CHECK(jsonContains(result, "\"depth\":1")); // 600 should not appear as a caller (it's a modified node, excluded). // 601 should not appear as a callee (it's a modified node, excluded). // Total = 2 (modified) + 0 (callers) + 1 (callee: 602) = 3. - assert(jsonContains(result, "\"total_impacted\":3")); + CHECK(jsonContains(result, "\"total_impacted\":3")); printf(" [PASS] analyzeChangeImpact: multiple modified files\n"); } +/// The modified-file list reaches analyzeChangeImpact as the CALLER spells it, +/// while entity.file_path holds the resolved absolute form. The lookup is an +/// exact `file_path IN (...)`, so a relative path or one that goes through a +/// symlink matched nothing and the tool answered `modified: []` — "nothing +/// changed" — for a file that had. The paths are now normalised the way the +/// store normalises a project root: anchored at the registered root when +/// relative, then weakly_canonical. +static void testImpactPathForms(store::GraphStore &store, uint64_t project_id) +{ + namespace fs = std::filesystem; + std::error_code ec; + + // ── Relative to the registered project root ────────────────── + // The fixture registers "/test", so this node is reachable by the + // relative path "src/rel.cpp" as well as its stored form. The ids are in a + // range no other case uses (the 3-hop case owns 300-306), because + // insertGraphNode() would otherwise fail its CHECK on the primary key and + // the lookups below would then report "nothing changed" for the wrong + // reason. + insertGraphNode(store, project_id, 700, "rel_fn", "/test/src/rel.cpp"); + syncSQLite(store, project_id); + CHECK(jsonContains(query::analyzeChangeImpact(project_id, &store, + "[\"/test/src/rel.cpp\"]"), + "\"id\":700")); + CHECK(jsonContains(query::analyzeChangeImpact(project_id, &store, + "[\"src/rel.cpp\"]"), + "\"id\":700")); + + // ── Through a symlink ──────────────────────────────────────── + // The node is stored under the RESOLVED directory, and the query names it + // through a link, which is what a caller sees when the project sits behind + // one (macOS /tmp is /private/tmp in exactly this way). + const fs::path base = fs::temp_directory_path(ec); + if (ec) { + printf(" [skip] symlink form: no temp directory\n"); + return; + } + const fs::path real_dir = base / "codescope_impact_real"; + const fs::path link = base / "codescope_impact_link"; + fs::remove_all(real_dir, ec); + fs::create_directories(real_dir, ec); + CHECK(!ec); + fs::remove(link, ec); + ec.clear(); + fs::create_directory_symlink(real_dir, link, ec); + if (ec) { + printf(" [skip] symlink form: cannot create a link\n"); + fs::remove_all(real_dir, ec); + return; + } + const std::string stored_dir = + fs::weakly_canonical(real_dir, ec).string(); + insertGraphNode(store, project_id, 701, "link_fn", + (stored_dir + "/b.cpp").c_str()); + syncSQLite(store, project_id); + const std::string via_link = (link / "b.cpp").string(); + CHECK(jsonContains(query::analyzeChangeImpact(project_id, &store, + ("[\"" + via_link + "\"]") + .c_str()), + "\"id\":701")); + + // The same spelling as a FILE FILTER, which findDefinition wraps in + // `file_path LIKE '%'||?||'%'`: an absolute path that reaches the project + // through a symlink matched nothing, so the query answered + // `{"results":[]}` for a symbol that exists (reproduced on a real index + // before the fix). Relative and partial filters must stay untouched — + // they are substrings by design, and resolving "b.cpp" against the CWD + // would break them — which is why only the absolute form is resolved. + query::QueryEngine engine(&store); + CHECK(engine.findDefinition(project_id, "link_fn", via_link.c_str()) + .find("\"name\":\"link_fn\"") != std::string::npos); + CHECK(engine.findDefinition(project_id, "link_fn", + (stored_dir + "/b.cpp").c_str()) + .find("\"name\":\"link_fn\"") != std::string::npos); + CHECK(engine.findDefinition(project_id, "link_fn", "b.cpp") + .find("\"name\":\"link_fn\"") != std::string::npos); + + fs::remove(link, ec); + fs::remove_all(real_dir, ec); + printf(" [PASS] analyzeChangeImpact: relative and symlinked paths\n"); +} + int main() { unlink(kDbPath); store::GraphStore store; - assert(store.open(kDbPath)); + CHECK(store.open(kDbPath)); uint64_t project_id = store.createProject("/test", "query_algos"); - assert(project_id > 0); + CHECK(project_id > 0); printf("=== findShortestPath tests ===\n"); testShortestPathDirectEdge(store, project_id); @@ -521,6 +604,7 @@ int main() testImpact1Hop(store, project_id); testImpact2Hop(store, project_id); testImpact3Hop(store, project_id); + testImpactPathForms(store, project_id); testImpactDepthCap(store, project_id); testImpactDisconnectedNode(store, project_id); testImpactEmptyFileList(store, project_id); @@ -529,6 +613,14 @@ int main() store.close(); unlink(kDbPath); + // The banner must follow the verdict: it used to be printed + // unconditionally, so a run with a failed CHECK announced PASSED and only + // the exit code disagreed — a caller reading the output (as this session + // did, on the way to a red gate) was told the opposite of the truth. + if (checkFailures()) { + printf("\n=== test_query_algorithms FAILED ===\n"); + return 1; + } printf("\n=== test_query_algorithms PASSED ===\n"); return 0; } diff --git a/engine/tests/test_readiness_canonical.cpp b/engine/tests/test_readiness_canonical.cpp new file mode 100644 index 0000000..4236e18 --- /dev/null +++ b/engine/tests/test_readiness_canonical.cpp @@ -0,0 +1,220 @@ +// test_readiness_canonical.cpp — readiness and overview answers must come from +// the canonical tables the pipeline actually writes. +// +// Regression (docs/REAL_PROJECT_TOOL_REPORT_2026-09-21.md, finding #4): +// GraphStore::getReadyRatio folded `graph_nodes._ready`, a table only the +// legacy engine_index_batch path populates. In a canonical DB it therefore +// returned 0.0 for every project, and the consumers drew the wrong conclusion +// from it — project_overview reported +// +// "callgraph_available": false, "ready_features": {"call_graph": false, …} +// "sample_call_edges": [] +// +// for a database in which find_callers/find_callees answered with hundreds of +// callers. verifier.h states the rule that was violated: the canonical fact +// layer (entity/relation) is the production source of truth, never graph_nodes. +// +// The fixture uses TWO projects on purpose: one whose only file contains a call, +// one whose only file does not. The positive case proves the ratio moves off 0, +// the negative case proves it is a measurement rather than a constant. + +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include "test_engine_handle.h" + +namespace fs = std::filesystem; + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "\nFAIL: %s\n", msg); + exit(1); + } +} + +static bool contains(const std::string &hay, const char *needle) +{ + return hay.find(needle) != std::string::npos; +} + +static void writeFile(const std::string &path, const char *content) +{ + FILE *f = fopen(path.c_str(), "w"); + if (!f) { + fprintf(stderr, "\nFAIL: cannot write %s\n", path.c_str()); + exit(1); + } + fputs(content, f); + fclose(f); +} + +static void removeDb(const std::string &db) +{ + unlink(db.c_str()); + unlink((db + "-wal").c_str()); + unlink((db + "-shm").c_str()); +} + +/// Index one directory and return the project id. +static uint64_t indexDir(const char *dir, const char *name) +{ + uint64_t pid = engine_create_project(g_engine, dir, name); + check(pid > 0, "create_project"); + char *r = engine_index_project(g_engine, pid, dir, nullptr); + check(r != nullptr, "index_project returns JSON"); + const std::string out(r); + engine_free_string(r); + check(contains(out, "\"ok\":true"), "index_project reports ok:true"); + return pid; +} + +int main() +{ + const std::string with_dir = "/tmp/test_readiness_with_calls"; + const std::string without_dir = "/tmp/test_readiness_without_calls"; + const std::string db_path = "/tmp/test_readiness_canonical.db"; + + fs::remove_all(with_dir); + fs::remove_all(without_dir); + removeDb(db_path); + fs::create_directories(with_dir); + fs::create_directories(without_dir); + + // One file with a real call edge (intra-file, so the resolver's P1 path + // produces it) and one file with none. + writeFile(with_dir + "/call.c", + "int callee(void) { return 1; }\n" + "int caller(void) { return callee(); }\n"); + writeFile(without_dir + "/lonely.c", + "int lonely(void) { return 0; }\n"); + // A file with no symbols at all: indexed (it is in `files`) but it + // produces no entity, which is what separates total_files from + // files_with_symbols below. + writeFile(with_dir + "/comment_only.c", "/* no symbols here */\n"); + + g_engine = engine_create(db_path.c_str()); + check(g_engine != nullptr, "engine_init"); + const uint64_t pid_calls = indexDir(with_dir.c_str(), "with-calls"); + const uint64_t pid_none = + indexDir(without_dir.c_str(), "without-calls"); + + // ── 1. project_overview: ready_features reflects the real call graph ── + char *ov = engine_project_overview(g_engine, pid_calls); + check(ov != nullptr, "engine_project_overview returns JSON"); + const std::string overview(ov); + engine_free_string(ov); + check(contains(overview, "\"ready_features\""), + "project_overview must carry ready_features"); + check(contains(overview, "\"call_graph\":true"), + "ready_features.call_graph must be true once the call graph exists " + "(it was false in every canonical DB, because the ratio folded the " + "empty graph_nodes table)"); + + // ── 2. build_context: the call graph is available and sampled ──────── + // An empty query maps to the \"general\" intent, the branch that reports + // callgraph_available and samples the edges. + char *bc = engine_build_context(g_engine, pid_calls, ""); + check(bc != nullptr, "engine_build_context returns JSON"); + const std::string context(bc); + engine_free_string(bc); + check(contains(context, "\"callgraph_available\":true"), + "a project with a call edge must report callgraph_available:true"); + check(contains(context, "sample_call_edges\":[{"), + "the sample edge list must not be empty when the call graph is " + "available (it used to read the empty graph_edges table)"); + check(contains(context, "caller") && contains(context, "callee"), + "the sample must name the caller and the callee"); + + // ── 3. the same question through the enhancement API agrees ───────── + char *st = engine_get_enhancement_status(g_engine, pid_calls); + check(st != nullptr, "engine_get_enhancement_status returns JSON"); + const std::string status(st); + engine_free_string(st); + int cg_ready = 0; + check(sscanf(status.c_str(), + "{\"total_symbols\":%*d,\"callgraph_ready\":%d", + &cg_ready) == 1, + "enhancement status parses"); + check(cg_ready > 0, + "engine_get_enhancement_status and project_overview must agree on " + "the call graph being ready"); + + // ── 3b. a SECOND project in the same DB is queryable ──────────────── + // Regression: buildGraph offset entity ids by a PROJECT-scoped MAX(id) and + // only for incremental rebuilds, so indexing a second project into a DB + // that already held one restarted at id 1, collided with the first + // project's ids, and INSERT OR IGNORE dropped every entity row of the new + // project — a project with semantic_records and zero entities, i.e. every + // tool answering "not found" right after a successful index. + char *fs = engine_find_symbol(g_engine, pid_none, "lonely"); + check(fs != nullptr, "engine_find_symbol returns JSON"); + const std::string found(fs); + engine_free_string(fs); + check(contains(found, "\"lonely\""), + "a project indexed into a database that already holds another " + "project must still have its entities (id collision made them " + "disappear silently)"); + + // ── 4. negative control: no call edge -> not available ────────────── + char *ov2 = engine_project_overview(g_engine, pid_none); + check(ov2 != nullptr, "overview for the call-free project"); + const std::string none(ov2); + engine_free_string(ov2); + check(contains(none, "\"call_graph\":false"), + "ready_features.call_graph must stay false without call edges — the " + "ratio is a measurement, not a constant"); + + char *bc2 = engine_build_context(g_engine, pid_none, ""); + check(bc2 != nullptr, "build_context for the call-free project"); + const std::string none_ctx(bc2); + engine_free_string(bc2); + check(contains(none_ctx, "\"callgraph_available\":false"), + "a project without call edges must NOT claim the call graph is " + "available"); + check(!contains(none_ctx, "sample_call_edges"), + "no sample edges may be reported when the call graph is unavailable"); + + // ── 5. get_graph_stats: total_files means files INDEXED ───────────── + // Regression (docs/REAL_PROJECT_TOOL_REPORT_2026-09-21.md, finding #5): + // total_files counted COUNT(DISTINCT file_path) FROM entity, i.e. only the + // files that produced a symbol, so a 1,579-file project reported 672. The + // fixture gives the two numbers different values on purpose: three indexed + // files, two of which carry symbols. + char *gs = engine_get_graph_stats(g_engine, 0); + check(gs != nullptr, "engine_get_graph_stats returns JSON"); + const std::string stats(gs); + engine_free_string(gs); + int total_files = -1, files_with_symbols = -1; + const char *tf = strstr(stats.c_str(), "\"total_files\":"); + const char *fws = strstr(stats.c_str(), "\"files_with_symbols\":"); + check(tf != nullptr, "get_graph_stats must report total_files"); + check(fws != nullptr, + "get_graph_stats must report files_with_symbols, so the old number " + "is not silently lost"); + if (tf) + total_files = atoi(tf + strlen("\"total_files\":")); + if (fws) + files_with_symbols = + atoi(fws + strlen("\"files_with_symbols\":")); + check(total_files == 3, + "total_files must count every indexed file (3 here: two with " + "symbols plus the comment-only one)"); + check(files_with_symbols == 2, + "files_with_symbols must count only the files that produced entities"); + + engine_destroy(g_engine); + g_engine = nullptr; + removeDb(db_path); + fs::remove_all(with_dir); + fs::remove_all(without_dir); + + printf("\n=== test_readiness_canonical PASSED ===\n"); + printf("canonical readiness ratio, overview agreement, negative control\n"); + return 0; +} diff --git a/engine/tests/test_readme_ingestion.cpp b/engine/tests/test_readme_ingestion.cpp index 48061fd..afe7de0 100644 --- a/engine/tests/test_readme_ingestion.cpp +++ b/engine/tests/test_readme_ingestion.cpp @@ -32,6 +32,7 @@ #include #include #include +#include "test_engine_handle.h" static inline void check(bool cond, const char *msg) { @@ -78,14 +79,16 @@ This project demonstrates README ingestion into the knowledge layer. // Init engine char db_path[] = "/tmp/test_readme_ingest.db"; unlink(db_path); - int rc = engine_init(db_path); + g_engine = engine_create(db_path); + const int rc = g_engine != nullptr ? 0 : -1; check(rc == 0, "engine_init"); - uint64_t pid = engine_create_project("/tmp", "readme-test"); + uint64_t pid = engine_create_project(g_engine, "/tmp", "readme-test"); check(pid > 0, "create_project"); // Index the project — this should ingest README.md - char *result = engine_index_project(pid, proj_dir.c_str(), nullptr); + char *result = + engine_index_project(g_engine, pid, proj_dir.c_str(), nullptr); check(result != nullptr, "index_project returns non-null"); printf("--- index_project result ---\n%s\n", result); check(strstr(result, "\"ok\":true") != nullptr, @@ -97,15 +100,15 @@ This project demonstrates README ingestion into the knowledge layer. // The async builder runs model plugins (CapabilityPlugin) which // read from document table and insert into capability table. // We use a short sleep + direct DB query. - std::this_thread::sleep_for( - std::chrono::milliseconds(500)); + std::this_thread::sleep_for(std::chrono::milliseconds(500)); // Query document table directly via SQLite sqlite3 *db = nullptr; check(sqlite3_open(db_path, &db) == SQLITE_OK, "sqlite3_open"); sqlite3_stmt *stmt = nullptr; - const char *doc_sql = "SELECT COUNT(*) FROM document WHERE project_id = ?"; + const char *doc_sql = + "SELECT COUNT(*) FROM document WHERE project_id = ?"; check(sqlite3_prepare_v2(db, doc_sql, -1, &stmt, nullptr) == SQLITE_OK, "prepare document count"); sqlite3_bind_int64(stmt, 1, static_cast(pid)); @@ -121,20 +124,21 @@ This project demonstrates README ingestion into the knowledge layer. // Verify the content matches what we wrote const char *content_sql = "SELECT content FROM document WHERE project_id = ? LIMIT 1"; - check(sqlite3_prepare_v2(db, content_sql, -1, &stmt, nullptr) == SQLITE_OK, + check(sqlite3_prepare_v2(db, content_sql, -1, &stmt, nullptr) == + SQLITE_OK, "prepare content select"); sqlite3_bind_int64(stmt, 1, static_cast(pid)); bool content_matches = false; if (sqlite3_step(stmt) == SQLITE_ROW) { - const char *content = - reinterpret_cast( - sqlite3_column_text(stmt, 0)); + const char *content = reinterpret_cast( + sqlite3_column_text(stmt, 0)); if (content) - content_matches = (strstr(content, "incremental indexing") != nullptr); + content_matches = + (strstr(content, "incremental indexing") != + nullptr); } sqlite3_finalize(stmt); - check(content_matches, - "document content must contain the README text"); + check(content_matches, "document content must contain the README text"); // Query capability table — CapabilityPlugin should have extracted // capabilities from the README lines matching "Supports ...". @@ -152,8 +156,113 @@ This project demonstrates README ingestion into the knowledge layer. check(cap_count > 0, "capability table must be non-empty after README ingestion + model build"); + // ── Project-root README ingested by the enhance pass ──────────────── + // + // The field case (2026-10-06): the walk ingests the README of a directory + // it scans, so a project whose index unit is a SUBDIRECTORY — every + // `index-parallel` module, and the whole file-list path — never had its + // root README ingested. `detect_documentation_drift` then answered + // "nothing claimed" for six of ten real projects that do have a README. + // Enhancing is the step every index path runs last, so it must ingest + // /README.md itself — and it must be idempotent, because the + // drift tools concatenate every README row of a project. + // + // Emulated by indexing only `sub/` of a second project: the walk cannot + // reach the root README, so any row that exists afterwards came from the + // enhance pass. + const std::string proj_dir2 = "/tmp/readme_ingest_enhance"; + std::filesystem::remove_all(proj_dir2); + std::filesystem::create_directories(proj_dir2 + "/sub"); + + FILE *readme2 = fopen((proj_dir2 + "/README.md").c_str(), "w"); + check(readme2 != nullptr, "fopen README of the second project"); + fputs("# Widget\n\nSupports Widgets for testing.\n", readme2); + fclose(readme2); + + FILE *src2 = fopen((proj_dir2 + "/sub/lib.c").c_str(), "w"); + check(src2 != nullptr, "fopen source of the second project"); + fputs("int widget(void) { return 1; }\n", src2); + fclose(src2); + + const uint64_t pid2 = engine_create_project(g_engine, proj_dir2.c_str(), + "readme-enhance"); + check(pid2 > 0, "create the second project"); + + // Index the subdirectory only: the walk never sees proj_dir2/README.md. + char *sub_result = engine_index_project( + g_engine, pid2, (proj_dir2 + "/sub").c_str(), nullptr); + check(sub_result != nullptr, "index_project on the subdirectory"); + engine_free_string(sub_result); + + sqlite3_stmt *count_stmt = nullptr; + const char *count_sql = + "SELECT COUNT(*) FROM document WHERE project_id = ?"; + check(sqlite3_prepare_v2(db, count_sql, -1, &count_stmt, nullptr) == + SQLITE_OK, + "prepare document count for the second project"); + sqlite3_bind_int64(count_stmt, 1, static_cast(pid2)); + int docs_before_enhance = -1; + if (sqlite3_step(count_stmt) == SQLITE_ROW) + docs_before_enhance = sqlite3_column_int(count_stmt, 0); + sqlite3_finalize(count_stmt); + check(docs_before_enhance == 0, + "scanning a subdirectory must not ingest the root README by itself"); + + // The enhance pass is what closes the gap. + char *enh2 = engine_enhance_project(g_engine, pid2); + check(enh2 != nullptr, "enhance the second project"); + engine_free_string(enh2); + + int docs_after_enhance = -1; + std::string docs_content; + { + sqlite3_stmt *content_stmt = nullptr; + const char *content_sql2 = "SELECT content FROM document " + "WHERE project_id = ?"; + check(sqlite3_prepare_v2(db, content_sql2, -1, &content_stmt, + nullptr) == SQLITE_OK, + "prepare content select for the second project"); + sqlite3_bind_int64(content_stmt, 1, static_cast(pid2)); + int rows = 0; + while (sqlite3_step(content_stmt) == SQLITE_ROW) { + ++rows; + const char *content = reinterpret_cast( + sqlite3_column_text(content_stmt, 0)); + if (content) + docs_content += content; + } + sqlite3_finalize(content_stmt); + docs_after_enhance = rows; + } + printf("--- second project documents after enhance: %d ---\n", + docs_after_enhance); + check(docs_after_enhance == 1, + "the enhance pass must ingest the project root README exactly once"); + check(docs_content.find("Supports Widgets") != std::string::npos, + "the ingested document must carry the root README content"); + + // Idempotent: a second enhance run must replace, not append. + char *enh3 = engine_enhance_project(g_engine, pid2); + check(enh3 != nullptr, "enhance the second project again"); + engine_free_string(enh3); + + sqlite3_stmt *again_stmt = nullptr; + check(sqlite3_prepare_v2(db, count_sql, -1, &again_stmt, nullptr) == + SQLITE_OK, + "prepare document count after the second enhance"); + sqlite3_bind_int64(again_stmt, 1, static_cast(pid2)); + int docs_after_second = -1; + if (sqlite3_step(again_stmt) == SQLITE_ROW) + docs_after_second = sqlite3_column_int(again_stmt, 0); + sqlite3_finalize(again_stmt); + check(docs_after_second == 1, + "re-running the enhance pass must not duplicate the README row"); + + std::filesystem::remove_all(proj_dir2); + sqlite3_close(db); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; // Cleanup std::filesystem::remove_all(proj_dir); diff --git a/engine/tests/test_receiver_type_canonical.cpp b/engine/tests/test_receiver_type_canonical.cpp new file mode 100644 index 0000000..fb319cd --- /dev/null +++ b/engine/tests/test_receiver_type_canonical.cpp @@ -0,0 +1,245 @@ +// test_receiver_type_canonical — one type, one spelling, every lookup. +// +// Defect guarded: a type name reaches the resolver in the spelling of its use +// site, while the tables it is looked up in (global_struct_fields_, +// global_var_types_, interface_impl_index_) are keyed by the bare name the +// declaration recorded. The field-chain walk (Step 8.1c) looked the RAW +// spelling up, so a parameter recorded as `*HolderA` never reached the entry +// keyed `HolderA`: the chain was abandoned and the walk moved on to the next +// same-named variable's type. On goagent that is how six `dispatch` call sites +// (`s.agents.Get(id)`) resolved through whichever same-named variable happened +// to be tried first, and why the result followed the module packing; in a +// fixture where the pointer spelling was the ONLY candidate the walk produced +// no edge at all while the value spelling produced two. The receiver-match +// factor had the same hole from the other side, building `prefix1` as +// `*Manager::`, a string no candidate's qualified_name can contain. +// +// Part 1 pins canonicalTypeName's boundaries (empty, sigil-only, pointers, +// references, package qualifiers, whitespace, trailing sigils, a Go slice). +// Part 2 drives the real index path over a Go fixture whose call sites can only +// resolve through that walk, in the three spellings that matter: a pointer +// parameter, a pointer to a struct whose FIELD is the pointer, and a type no +// declaration mentions (which must stay unresolved rather than guessed). +#include "../include/engine.h" +#include "../src/resolver/factors.h" + +#include +#include +#include +#include +#include +#include +#include + +#include "test_check.h" +#include "test_engine_handle.h" + +namespace +{ + +/// Assert one canonicalisation, naming both sides on failure. +void expectCanonical(const char *in, const char *want) +{ + const std::string got = resolver::canonicalTypeName(in); + if (got != want) { + fprintf(stderr, "FAIL: canonicalTypeName(\"%s\") = \"%s\", want \"%s\"\n", + in, got.c_str(), want); + } + CHECK(got == want); +} + +/// Write `content` to `path`, failing the test if the file cannot be created. +void writeFile(const std::string &path, const char *content) +{ + FILE *f = fopen(path.c_str(), "w"); + CHECK(f != nullptr); + fputs(content, f); + fclose(f); +} + +/// One scalar read. +long long scalarSql(sqlite3 *db, const char *sql) +{ + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + long long value = -1; + if (sqlite3_step(st) == SQLITE_ROW) + value = sqlite3_column_int64(st, 0); + sqlite3_finalize(st); + return value; +} + +/// The resolution kind of the edge `caller -> callee`, or "(no edge)". +std::string edgeKind(sqlite3 *db, const char *caller, const char *callee) +{ + const char *sql = + "SELECT r.resolution_kind FROM relation r " + "JOIN entity se ON se.id = r.source_id " + "JOIN entity te ON te.id = r.target_id " + "WHERE r.type = 1 AND se.name = ? AND te.name = ? " + "ORDER BY r.resolution_kind LIMIT 1"; + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + sqlite3_bind_text(st, 1, caller, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(st, 2, callee, -1, SQLITE_TRANSIENT); + std::string out = "(no edge)"; + if (sqlite3_step(st) == SQLITE_ROW) { + const char *s = reinterpret_cast( + sqlite3_column_text(st, 0)); + if (s) + out = s; + } + sqlite3_finalize(st); + return out; +} + +} // namespace + +int main() +{ + // ── Part 1: canonicalTypeName boundaries ────────────────────── + expectCanonical("", ""); + expectCanonical("*", ""); + expectCanonical("&", ""); + expectCanonical("***", ""); + expectCanonical("PluginBus", "PluginBus"); + expectCanonical("plugin_bus", "plugin_bus"); + expectCanonical("*PluginBus", "PluginBus"); + expectCanonical("&PluginBus", "PluginBus"); + expectCanonical("**PluginBus", "PluginBus"); + expectCanonical("PluginBus*", "PluginBus"); // pointer-suffix spelling + expectCanonical("PluginBus&", "PluginBus"); + expectCanonical("ares_runtime.PluginBus", "PluginBus"); + expectCanonical("*ares_runtime.PluginBus", "PluginBus"); + expectCanonical("a.b.C", "C"); + expectCanonical(".Leading", "Leading"); + expectCanonical(" *\tPluginBus ", "PluginBus"); + expectCanonical("ares_runtime::PluginBus", "PluginBus"); + // Template instantiations are UNKNOWN, not their base name: which type owns + // the method depends on the template (`std::vector` holds its own, + // `std::unique_ptr` forwards to `Store`). Stripping the argument list + // made `std::unique_ptr` into `unique_ptr`, which matches no + // declaration — on this repository that turned `policy->loadIgnoreFile(…)` + // from a resolved unique-name call into a lost edge and added false ones + // elsewhere (rebuilt-commit A/B: -85/+62, versus 0/0 once instantiations are + // unknown). + expectCanonical("std::vector", ""); + expectCanonical("Holder[T]", ""); + expectCanonical("pkg.Holder[T]", ""); + expectCanonical("std::map", ""); + expectCanonical("std::unique_ptr", ""); + expectCanonical("*std::unique_ptr", ""); + // Composite spellings carry punctuation a type name cannot contain, and they + // are what Go records for channels, slices, maps and function types. Mining + // them for a last segment is how `<-chan os.Signal` becomes `Signal` — a + // real symbol in most code bases — so they are reported as unknown instead. + expectCanonical("[]byte", ""); + expectCanonical("[]<-chan *Event", ""); + expectCanonical("map[string]int", ""); + expectCanonical("<-chan os.Signal", ""); + expectCanonical("chan<- *Event", ""); + expectCanonical("func(ctx context.Context, s string) (<-chan T, error)", ""); + expectCanonical("::", ""); + + // ── Part 2: the field-chain walk through the real index path ── + const std::string proj_dir = "/tmp/receiver_type_canonical_repro"; + std::filesystem::remove_all(proj_dir); + std::filesystem::create_directories(proj_dir + "/src"); + + writeFile(proj_dir + "/src/iface.go", + "package p\n" + "\n" + "type Getter interface {\n" + "\tGet(id string) error\n" + "}\n" + "\n" + "type Registry struct{}\n" + "\n" + "func (r *Registry) Get(id string) error { return nil }\n"); + writeFile(proj_dir + "/src/holders.go", + "package p\n" + "\n" + "type Holder struct {\n" + "\tregistry Getter\n" + "}\n" + "\n" + "type Outer struct {\n" + "\tinner *Holder\n" + "}\n"); + writeFile(proj_dir + "/src/use_ptr.go", + "package p\n" + "\n" + "func usePtr(h *Holder, id string) error {\n" + "\treturn h.registry.Get(id)\n" + "}\n"); + writeFile(proj_dir + "/src/use_nested.go", + "package p\n" + "\n" + "func useNested(o *Outer, id string) error {\n" + "\treturn o.inner.registry.Get(id)\n" + "}\n"); + writeFile(proj_dir + "/src/use_unknown.go", + "package p\n" + "\n" + "func useUnknown(u Undeclared, id string) error {\n" + "\treturn u.registry.Get(id)\n" + "}\n"); + + const std::string db_path = proj_dir + "/canonical.db"; + unlink(db_path.c_str()); + unlink((db_path + "-wal").c_str()); + unlink((db_path + "-shm").c_str()); + + g_engine = engine_create(db_path.c_str()); + CHECK(g_engine != nullptr); + const uint64_t pid = engine_create_project(g_engine, proj_dir.c_str(), + "receiver-canonical"); + CHECK(pid > 0); + + const std::string request = "{\"paths\":[\"" + proj_dir + + "/src/iface.go\",\"" + proj_dir + + "/src/holders.go\",\"" + proj_dir + + "/src/use_ptr.go\",\"" + proj_dir + + "/src/use_nested.go\",\"" + proj_dir + + "/src/use_unknown.go\"]}"; + char *res = engine_index_files(g_engine, pid, request.c_str(), 1); + CHECK(res != nullptr); + CHECK(strstr(res, "\"ok\":true") != nullptr); + engine_free_string(res); + // No wait needed before reading the database directly: engine_destroy() + // joins the async knowledge builder (engine_lifecycle.cpp) before it closes + // the store, and everything asserted below — relation, entity, + // semantic_records — is written synchronously by the index path, not by the + // builder. A fixed sleep here would only add a race against slow CI. + engine_destroy(g_engine); + g_engine = nullptr; + + sqlite3 *db = nullptr; + CHECK(sqlite3_open(db_path.c_str(), &db) == SQLITE_OK); + + // The call sites below are only meaningful while the name is ambiguous — + // the interface method and its implementation are both called `Get` — and + // while the fixture really parsed (the three call sites exist). + CHECK(scalarSql(db, "SELECT COUNT(*) FROM entity WHERE name='Get'") >= 2); + CHECK(scalarSql(db, "SELECT COUNT(*) FROM entity WHERE name IN " + "('usePtr','useNested','useUnknown')") == 3); + + // `h *Holder`: the chain is `h` → *Holder → registry → Getter, so every + // lookup depends on stripping the pointer sigil before hitting a table. + CHECK(edgeKind(db, "usePtr", "Get") == "dispatch"); + // `o *Outer.inner` → *Holder → registry → Getter: the sigil appears twice, + // once per segment. + CHECK(edgeKind(db, "useNested", "Get") == "dispatch"); + // `u Undeclared` resolves to nothing: no table has that struct, so the walk + // must abstain instead of falling through to another variable's type. + CHECK(edgeKind(db, "useUnknown", "Get") == "(no edge)"); + + sqlite3_close(db); + unlink(db_path.c_str()); + if (checkFailures()) { + fprintf(stderr, "=== receiver type canonicalisation: FAILED ===\n"); + return 1; + } + printf("=== receiver type canonicalisation test passed ===\n"); + return 0; +} diff --git a/engine/tests/test_resolution_kind.cpp b/engine/tests/test_resolution_kind.cpp new file mode 100644 index 0000000..fd79d5c --- /dev/null +++ b/engine/tests/test_resolution_kind.cpp @@ -0,0 +1,278 @@ +// test_resolution_kind: `relation.resolution_kind` must name the evidence that +// DECIDED the match, not the first non-empty reference field. +// +// The kind used to be read off "which reference field is non-empty" +// (receiver_type > qualified_target > import_alias > name), which answers a +// different question: a call carrying a `receiver_type` field but resolved by +// name/namespace evidence — the receiver evidence contributing nothing to the +// winning score — was still labelled `receiver_type`. Per-kind accuracy audits +// group by this column, so all of them inherited that skew. +// +// The label now comes from the winning candidate's largest positive factor +// contribution (Candidate::deciding_factor, recorded by applyConstraints while +// it accumulates the weighted score), with an exact qualified-name hit checked +// explicitly because it is not a scoring factor. This test asserts both +// directions: the label is NOT `receiver_type` when the receiver evidence did +// not decide the match, and IS `receiver_type` when it did. + +#include "../src/resolver/pipeline.h" +#include "../src/store/store.h" + +#include + +#include "test_check.h" +#include +#include +#include + +using namespace resolver; + +static const char *kDbPath = "/tmp/test_resolution_kind.db"; + +/// Insert an entity row. `qualified_name` carries the receiver evidence a +/// method call can match against ("Box.Beta" for receiver_type "Box"). +static void insertEntity(store::GraphStore &store, uint64_t project_id, + int64_t id, const char *name, + const char *qualified_name, const char *file_path, + const char *language = "cpp") +{ + sqlite3 *db = store.handle(); + const char *sql = "INSERT INTO entity (id, project_id, kind, name, " + "qualified_name, file_path, language, start_row, " + "start_col, end_row, end_col) " + "VALUES (?,?,0,?,?,?,?,0,0,0,0)"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + sqlite3_bind_int64(stmt, 1, id); + sqlite3_bind_int64(stmt, 2, static_cast(project_id)); + sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 4, qualified_name, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 5, file_path, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 6, language, -1, SQLITE_TRANSIENT); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); +} + +/// Insert a semantic_records InterfaceImpl row (kind=20): `impl` implements +/// `iface`. This is what pipeline_load.cpp reads to build +/// interface_impl_index_, i.e. what drives the dispatch expansion. +static void insertInterfaceImpl(store::GraphStore &store, uint64_t project_id, + const char *impl, const char *iface) +{ + sqlite3 *db = store.handle(); + // original_id and file_path are NOT NULL in semantic_records; the dispatch + // index reads only name/type_name, so they carry placeholder values here. + const char *sql = + "INSERT INTO semantic_records (project_id, original_id, kind, " + "name, type_name, file_path) " + "VALUES (?,1,20,?,?,'/fixture/iface_impl')"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_text(stmt, 2, impl, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 3, iface, -1, SQLITE_TRANSIENT); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); +} + +/// Insert a reference row: caller_id calls `name`, carrying `receiver_type`. +static void insertReference(store::GraphStore &store, uint64_t project_id, + int64_t caller_id, const char *name, + const char *receiver_type, + const char *qualified_target = "") +{ + sqlite3 *db = store.handle(); + const char *sql = "INSERT INTO reference (project_id, caller_id, name, " + "arity, call_kind, start_row, start_col, " + "receiver_type, call_site_file, qualified_target) " + "VALUES (?,?,?,0,0,0,0,?,?,?)"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_int64(stmt, 2, caller_id); + sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 4, receiver_type, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 5, "/src/app/main.cpp", -1, SQLITE_TRANSIENT); + sqlite3_bind_text(stmt, 6, qualified_target, -1, SQLITE_TRANSIENT); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); +} + +/// The call edge targeting `target_id`: its resolution_kind and its reason. +static bool callEdgeTo(store::GraphStore &store, uint64_t project_id, + int64_t target_id, std::string &kind, + std::string &reason) +{ + sqlite3 *db = store.handle(); + const char *sql = "SELECT resolution_kind, reason FROM relation " + "WHERE project_id=? AND type=1 AND target_id=?"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_int64(stmt, 2, target_id); + bool found = false; + if (sqlite3_step(stmt) == SQLITE_ROW) { + const char *k = reinterpret_cast( + sqlite3_column_text(stmt, 0)); + const char *r = reinterpret_cast( + sqlite3_column_text(stmt, 1)); + kind = k ? k : ""; + reason = r ? r : ""; + found = true; + } + sqlite3_finalize(stmt); + return found; +} + +int main() +{ + unlink(kDbPath); + + store::GraphStore store; + if (!store.open(kDbPath)) { + fprintf(stderr, "FAIL: cannot open store: %s\n", + store.error().c_str()); + return 1; + } + uint64_t pid = store.createProject("/test", "test_resolution_kind"); + CHECK(pid > 0); + + // ── Case A: receiver evidence is present but does NOT decide ── + // Caller and candidate 2 share a directory (namespace/module evidence), + // candidate 3 does not. The reference carries receiver_type="Unrelated", + // which matches neither candidate's qualified_name — so the receiver factor + // contributes nothing decisive even though the field is non-empty. + insertEntity(store, pid, 1, "main", "", "/src/app/main.cpp"); + insertEntity(store, pid, 2, "Alpha", "", "/src/app/other.cpp"); + insertEntity(store, pid, 3, "Alpha", "", "/src/other/other.cpp"); + insertReference(store, pid, 1, "Alpha", "Unrelated"); + + // ── Case B: qualified_target present but NOT the evidence that decided ── + // Same shape as Case A for a different field of the old priority chain: + // the reference carries a qualified_target, but it matches neither + // candidate's qualified_name, so the old rule's "qualified" label would be + // just as invented. + insertEntity(store, pid, 4, "Beta", "", "/src/app/beta.cpp"); + insertEntity(store, pid, 5, "Beta", "", "/src/other/beta.cpp"); + insertReference(store, pid, 1, "Beta", "", "Unrelated.Thing"); + + // ── Case D: a fuzzy candidate is not labelled "exact_local" ─── + // The fast path runs whenever the candidate set holds one same-directory + // entity — including when that set came from the fuzzy fallback, whose name + // only matches by prefix ("DeltaThi" -> "DeltaThing"). The label said + // "exact_local" regardless, i.e. it claimed an exactness the match never + // had. + insertEntity(store, pid, 6, "DeltaThing", "", "/src/app/delta.cpp"); + insertReference(store, pid, 1, "DeltaThi", "Whatever"); + + // ── Case E: dispatch expansion must respect the language filter ── + // "Iface" is implemented twice — once in C++ and once in Python — and the + // call site is C++. The expansion selects targets by qualified_name prefix + // only, so without a language filter the .py implementation became a CALLS + // target of a .cpp call site. + insertEntity(store, pid, 7, "Do", "CppImpl.Do", + "/src/app/cpp_impl.cpp"); + insertEntity(store, pid, 8, "Do", "PyImpl.Do", "/src/app/py_impl.py", + "python"); + insertInterfaceImpl(store, pid, "CppImpl", "Iface"); + insertInterfaceImpl(store, pid, "PyImpl", "Iface"); + insertReference(store, pid, 1, "Do", "Iface"); + + // ── Case C: the receiver mapping itself ─────────────────────── + // ReceiverMatch is the one factor the audits care most about, and whether + // it can ever be the largest contributor in a real fixture depends on the + // weights, so its mapping is asserted directly. + CHECK(resolutionKindFromFactor("ReceiverMatch") == "receiver_type"); + + ResolverPipeline pipe(&store, pid); + int64_t resolved = pipe.run(); + printf("resolved %lld call edge(s)\n", (long long)resolved); + // Diagnostics: every edge the pipeline wrote, so a failure here says which + // case abstained instead of only that something did. + { + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(store.handle(), + "SELECT target_id, confidence, " + "resolution_kind, reason FROM relation " + "WHERE project_id=? AND type=1", + -1, &st, nullptr) == SQLITE_OK); + sqlite3_bind_int64(st, 1, static_cast(pid)); + while (sqlite3_step(st) == SQLITE_ROW) { + printf(" edge -> target=%lld conf=%.3f kind=%s reason=%s\n", + (long long)sqlite3_column_int64(st, 0), + sqlite3_column_double(st, 1), + reinterpret_cast( + sqlite3_column_text(st, 2)), + reinterpret_cast( + sqlite3_column_text(st, 3))); + } + sqlite3_finalize(st); + } + CHECK(resolved >= 2); + + // ── Case A assertions ───────────────────────────────────────── + std::string kind_a; + std::string reason_a; + if (!callEdgeTo(store, pid, 2, kind_a, reason_a)) { + fprintf(stderr, + "FAIL: Case A did not resolve to the same-directory " + "candidate (entity 2)\n"); + return 1; + } + printf("Case A: target=2 kind=%s reason=%s\n", kind_a.c_str(), + reason_a.c_str()); + // The regression: this edge used to be labelled `receiver_type` purely + // because the reference's receiver_type field was non-empty. + CHECK(kind_a != "receiver_type"); + // The label must name the evidence that decided it. + CHECK(reason_a.find("decided_by=") != std::string::npos); + + // ── Case B assertions ───────────────────────────────────────── + std::string kind_b; + std::string reason_b; + if (!callEdgeTo(store, pid, 4, kind_b, reason_b)) { + fprintf(stderr, + "FAIL: Case B did not resolve to the same-directory " + "candidate (entity 4)\n"); + return 1; + } + printf("Case B: target=4 kind=%s reason=%s\n", kind_b.c_str(), + reason_b.c_str()); + CHECK(kind_b != "qualified"); + CHECK(reason_b.find("decided_by=") != std::string::npos); + + // ── Case D assertions ───────────────────────────────────────── + std::string kind_d; + std::string reason_d; + if (!callEdgeTo(store, pid, 6, kind_d, reason_d)) { + fprintf(stderr, + "FAIL: Case D did not resolve the fuzzy candidate\n"); + return 1; + } + printf("Case D: target=6 kind=%s reason=%s\n", kind_d.c_str(), + reason_d.c_str()); + // The name matched by prefix, so "exact_local" would be a claim the match + // never earned. + CHECK(kind_d == "fuzzy_local"); + + // ── Case E assertions ───────────────────────────────────────── + std::string kind_e; + std::string reason_e; + if (!callEdgeTo(store, pid, 7, kind_e, reason_e)) { + fprintf(stderr, + "FAIL: Case E lost the same-language dispatch target\n"); + return 1; + } + printf("Case E: target=7 kind=%s reason=%s\n", kind_e.c_str(), + reason_e.c_str()); + CHECK(kind_e == "dispatch"); + // The Python implementation must not be a target of a C++ call site. + std::string kind_py; + std::string reason_py; + CHECK(!callEdgeTo(store, pid, 8, kind_py, reason_py)); + + store.close(); + unlink(kDbPath); + printf("=== test_resolution_kind PASSED ===\n"); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_resolve_strategy.cpp b/engine/tests/test_resolve_strategy.cpp index f46f170..6450f62 100644 --- a/engine/tests/test_resolve_strategy.cpp +++ b/engine/tests/test_resolve_strategy.cpp @@ -14,6 +14,7 @@ #include #include #include +#include "test_engine_handle.h" static inline void check(bool cond, const char *msg) { @@ -28,8 +29,7 @@ int main() const char *proj_dir = "/tmp/resolve_strategy_repro"; std::filesystem::remove_all(proj_dir); std::filesystem::create_directories(proj_dir); - const std::string py_path = - std::string(proj_dir) + "/repro.py"; + const std::string py_path = std::string(proj_dir) + "/repro.py"; FILE *f = fopen(py_path.c_str(), "w"); check(f != nullptr, "fopen"); // Test two scenarios: @@ -53,12 +53,14 @@ class Worker: char db[] = "/tmp/test_resolve_strategy.db"; unlink(db); - check(engine_init(db) == 0, "engine_init"); + g_engine = engine_create(db); + check(g_engine != nullptr, "engine_init"); - uint64_t pid = engine_create_project(proj_dir, "resolve-test"); + uint64_t pid = + engine_create_project(g_engine, proj_dir, "resolve-test"); check(pid > 0, "create_project"); - char *idx = engine_index_project(pid, proj_dir, nullptr); + char *idx = engine_index_project(g_engine, pid, proj_dir, nullptr); check(idx != nullptr, "index_project null"); check(strstr(idx, "\"ok\":true") != nullptr, "index ok"); engine_free_string(idx); @@ -94,9 +96,8 @@ class Worker: int row = sqlite3_column_int(st, 5); printf("id=%lld name=%s parent_id=%lld ref_oid=%lld " "strategy=%s row=%d\n", - (long long)id, name ? name : "(null)", - (long long)par, (long long)ref, - rs ? rs : "", row); + (long long)id, name ? name : "(null)", (long long)par, + (long long)ref, rs ? rs : "", row); } sqlite3_finalize(st); @@ -114,10 +115,13 @@ class Worker: SQLITE_OK, "prepare verify"); sqlite3_bind_int64(s2, 1, pid); - bool found_load = false, found_add = false, found_unknown = false; + bool found_load = false, found_add = false, + found_unknown = false; while (sqlite3_step(s2) == SQLITE_ROW) { - const char *n = (const char *)sqlite3_column_text(s2, 0); - const char *rs = (const char *)sqlite3_column_text(s2, 1); + const char *n = + (const char *)sqlite3_column_text(s2, 0); + const char *rs = + (const char *)sqlite3_column_text(s2, 1); if (!n || !rs) continue; if (strcmp(n, "_load_data") == 0) { @@ -144,7 +148,8 @@ class Worker: } sqlite3_close(db_h); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; std::filesystem::remove_all(proj_dir); printf("\n=== PASS: resolve_strategy correctly set ===\n"); return 0; diff --git a/engine/tests/test_resolver_fuzzy_cache.cpp b/engine/tests/test_resolver_fuzzy_cache.cpp index a0f2407..9350df1 100644 --- a/engine/tests/test_resolver_fuzzy_cache.cpp +++ b/engine/tests/test_resolver_fuzzy_cache.cpp @@ -19,7 +19,7 @@ #include "../src/resolver/pipeline.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -38,12 +38,12 @@ static void insertEntity(store::GraphStore &store, uint64_t project_id, "start_col, end_row, end_col) " "VALUES (?,?,0,?,'',?,'cpp',0,0,0,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, file_path, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -62,12 +62,12 @@ static void insertReference(store::GraphStore &store, uint64_t project_id, "receiver_type) " "VALUES (?,?,?,0,0,0,0,?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, caller_id); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, receiver_type, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -78,7 +78,7 @@ static int countCallRelations(store::GraphStore &store, uint64_t project_id) const char *sql = "SELECT COUNT(*) FROM relation WHERE project_id=? AND type=1"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); int count = 0; if (sqlite3_step(stmt) == SQLITE_ROW) @@ -92,9 +92,10 @@ int main() unlink(kDbPath); store::GraphStore store; - assert(store.open(kDbPath)); - uint64_t pid = store.createProject("/test", "test_resolver_fuzzy_cache"); - assert(pid > 0); + CHECK(store.open(kDbPath)); + uint64_t pid = + store.createProject("/test", "test_resolver_fuzzy_cache"); + CHECK(pid > 0); // Caller "main" and callee "RealFunc" live in the same directory // (/src/app/) so factorImportMatch returns 1.0 (same-directory @@ -119,7 +120,7 @@ int main() // ── Test 1: miss cache is empty before run() ───────────────── { ResolverPipeline pipe(&store, pid); - assert(pipe.fuzzyMissCacheSize() == 0); + CHECK(pipe.fuzzyMissCacheSize() == 0); printf("Test 1 (miss cache empty before run): PASS\n"); } @@ -128,11 +129,11 @@ int main() ResolverPipeline pipe(&store, pid); int64_t resolved = pipe.run(); // GhostFunc was a fuzzy miss -> cached exactly once (set dedup). - assert(pipe.fuzzyMissCacheSize() == 1); + CHECK(pipe.fuzzyMissCacheSize() == 1); // RealFunc resolved via exact entity_index hit; the budget // mechanism only gates the fuzzy fallback, so exact matches // are unaffected. - assert(resolved >= 1); + CHECK(resolved >= 1); printf("Test 2 (miss cache size=%zu, resolved=%lld): PASS\n", pipe.fuzzyMissCacheSize(), (long long)resolved); } @@ -140,7 +141,7 @@ int main() // ── Test 3: resolved relation persisted to the relation table ── { int rel_count = countCallRelations(store, pid); - assert(rel_count >= 1); + CHECK(rel_count >= 1); printf("Test 3 (resolved relation persisted: %d): PASS\n", rel_count); } @@ -153,13 +154,13 @@ int main() { ResolverPipeline pipe(&store, pid); int64_t resolved = pipe.run(); - assert(resolved >= 1); - assert(pipe.fuzzyMissCacheSize() == 1); + CHECK(resolved >= 1); + CHECK(pipe.fuzzyMissCacheSize() == 1); printf("Test 4 (exact match survives fuzzy budget): PASS\n"); } store.close(); unlink(kDbPath); printf("\nAll resolver fuzzy cache tests passed.\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_resolver_language_consistency.cpp b/engine/tests/test_resolver_language_consistency.cpp new file mode 100644 index 0000000..4bfd3a0 --- /dev/null +++ b/engine/tests/test_resolver_language_consistency.cpp @@ -0,0 +1,412 @@ +// test_resolver_language_consistency — one call shape, every language, one +// outcome class. +// +// test_graph_call_precision.cpp pins the layered matching on hand-built +// SemanticUnits that merely carry a language LABEL, and test_ir_edge_coverage +// pins what each VISITOR emits as records. Neither covers the property that +// matters here: the visitors disagree about what they record (Go writes a +// receiver, Python a `self` parameter, C++ a declarator, Rust an impl block, +// Java/TS a formal parameter list, and each records its own type spelling), and +// all of them feed ONE resolver. A shape the resolver handles for Go can +// therefore be guessed or dropped for Python without any test noticing. +// +// This test indexes the same shape in six languages and asserts the same +// OUTCOME CLASS for each: +// +// S1 a call through a parameter whose declared type owns the method, while +// an unrelated type declares a method of the SAME NAME (the decoy). The +// call must land on the parameter's type — never on the decoy — or the +// receiver evidence is being ignored in favour of the name. +// S2 a call whose receiver type no declaration mentions and whose method +// name no declaration defines. There is nothing to resolve it to, so +// there must be NO edge; inventing one is a false positive that every +// language would inherit from the shared name fallback. +// +// The languages are Go, C++, Java, Python, TypeScript and Rust. JavaScript is +// absent on purpose: it has no type annotations, so S1 cannot be spelled +// without types at all (`test_js_ts_call_facts.cpp` covers its call facts). +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "test_check.h" +#include "test_engine_handle.h" + +namespace +{ + +/// Write `content` to `path`, failing the test if the file cannot be created. +void writeFile(const std::string &path, const char *content) +{ + FILE *f = fopen(path.c_str(), "w"); + CHECK(f != nullptr); + fputs(content, f); + fclose(f); +} + +/// One scalar read; -1 when the query yields no row. +long long scalarSql(sqlite3 *db, const char *sql) +{ + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + long long value = -1; + if (sqlite3_step(st) == SQLITE_ROW) + value = sqlite3_column_int64(st, 0); + sqlite3_finalize(st); + return value; +} + +/// The resolution kind of the edge `caller -> callee`, or "(no edge)". +std::string edgeKind(sqlite3 *db, const char *caller, const char *callee) +{ + const char *sql = + "SELECT r.resolution_kind FROM relation r " + "JOIN entity se ON se.id = r.source_id " + "JOIN entity te ON te.id = r.target_id " + "WHERE r.type = 1 AND se.name = ? AND te.name = ? " + "ORDER BY r.resolution_kind LIMIT 1"; + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + sqlite3_bind_text(st, 1, caller, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(st, 2, callee, -1, SQLITE_TRANSIENT); + std::string out = "(no edge)"; + if (sqlite3_step(st) == SQLITE_ROW) { + const char *s = reinterpret_cast( + sqlite3_column_text(st, 0)); + if (s) + out = s; + } + sqlite3_finalize(st); + return out; +} + +/// The resolution kind of the edge from `caller` to the method `method` that +/// `owner` declares, or "(no edge)". +/// +/// The decoy carries the SAME method name, so a lookup by name alone cannot +/// tell the two methods apart; `entity.qualified_name` is the declaration's +/// owning type plus the method (`CppOwner::Method`, `JavaOwner.method`), so a +/// prefix match identifies the owner in every language without assuming a +/// separator. +std::string edgeKindInType(sqlite3 *db, const char *caller, + const char *method, const char *owner) +{ + const char *sql = + "SELECT r.resolution_kind FROM relation r " + "JOIN entity se ON se.id = r.source_id " + "JOIN entity te ON te.id = r.target_id " + "WHERE r.type = 1 AND se.name = ? AND te.name = ? " + "AND te.qualified_name LIKE ? || '%' " + "ORDER BY r.resolution_kind LIMIT 1"; + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + sqlite3_bind_text(st, 1, caller, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(st, 2, method, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(st, 3, owner, -1, SQLITE_TRANSIENT); + std::string out = "(no edge)"; + if (sqlite3_step(st) == SQLITE_ROW) { + const char *s = reinterpret_cast( + sqlite3_column_text(st, 0)); + if (s) + out = s; + } + sqlite3_finalize(st); + return out; +} + +/// The receiver type the visitors recorded for `caller`'s call, or "(none)". +/// Printed as a diagnostic: the whole S1 result follows from this one field. +std::string receiverType(sqlite3 *db, const char *caller) +{ + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, + "SELECT rf.receiver_type FROM reference rf " + "JOIN entity c ON c.id = rf.caller_id " + "WHERE c.name = ? LIMIT 1", + -1, &st, nullptr) == SQLITE_OK); + sqlite3_bind_text(st, 1, caller, -1, SQLITE_TRANSIENT); + std::string out = "(none)"; + if (sqlite3_step(st) == SQLITE_ROW) { + const char *s = reinterpret_cast( + sqlite3_column_text(st, 0)); + if (s && *s) + out = s; + } + sqlite3_finalize(st); + return out; +} + +/// Number of call edges leaving `caller`. +long long outgoingCallEdges(sqlite3 *db, const char *caller) +{ + return scalarSql(db, + ("SELECT COUNT(*) FROM relation r JOIN entity se " + "ON se.id = r.source_id WHERE r.type = 1 AND se.name='" + + std::string(caller) + "'") + .c_str()); +} + +/// One language's expectations. The names are unique per language so one +/// project-wide database can hold all six fixtures without ambiguity, and the +/// S1 decoy is the sibling method that must never win. +/// +/// `s1_gap` records whether S1 is a KNOWN unresolved case for this language +/// today, rather than an expectation of the language. Measured on this fixture: +/// the six visitors agree on the call fact itself (all six write +/// `reference(name='method', call_kind=1, receiver_text='o')`) but not on the +/// receiver's TYPE — `reference.receiver_type` is `*GoOwner` / `JavaOwner` / +/// `PyOwner` for Go, Java and Python, and EMPTY for C++, Rust and TypeScript, +/// whose visitors record neither a `receiver_type` on the call nor a +/// variable-type record to derive it from. Without that evidence the two +/// same-named methods cannot be told apart, so the resolver abstains and the +/// dependency is LOST (a false negative, not a wrong edge). +/// +/// The assertions below treat the gap as a ratchet: the strong invariants must +/// hold for every language, the resolved set must be exactly the languages with +/// receiver-type evidence, and the moment a visitor starts recording it the +/// test fails on the count so the flag — and this comment — get updated instead +/// of the improvement passing unnoticed. +struct LanguageCase { + const char *language; // printed in the summary table + const char *file; // fixture path, relative to the project root + const char *s1_caller; // calls through a typed parameter + const char *s1_owner; // type whose method the call must reach + const char *s1_method; // that method's name (the decoy's name too) + const char *s1_decoy; // unrelated type declaring the same method name + const char *s2_caller; // calls a method nothing declares + const char *s2_callee; // the name that must NOT resolve + bool s1_gap; // S1 unresolved today: no receiver type recorded +}; + +} // namespace + +int main() +{ + const std::string proj_dir = "/tmp/resolver_language_consistency"; + std::filesystem::remove_all(proj_dir); + std::filesystem::create_directories(proj_dir + "/src"); + + // ── Go ──────────────────────────────────────────────────────────── + writeFile(proj_dir + "/src/lang.go", + "package p\n" + "\n" + "type GoOwner struct{}\n" + "\n" + "func (o *GoOwner) Method() int { return 1 }\n" + "\n" + "type GoDecoy struct{}\n" + "\n" + "func (d *GoDecoy) Method() int { return 2 }\n" + "\n" + "func goS1(o *GoOwner) int { return o.Method() }\n" + "\n" + "func goS2(u GoUndeclared) int { return u.computeAbsent() }\n"); + // ── C++ ─────────────────────────────────────────────────────────── + writeFile(proj_dir + "/src/lang.cpp", + "struct CppOwner {\n" + "\tint Method() { return 1; }\n" + "};\n" + "\n" + "struct CppDecoy {\n" + "\tint Method() { return 2; }\n" + "};\n" + "\n" + "int cppS1(CppOwner *o) { return o->Method(); }\n" + "\n" + "int cppS2(CppUndeclared u) { return u.computeAbsent(); }\n"); + // ── Java ────────────────────────────────────────────────────────── + writeFile(proj_dir + "/src/Lang.java", + "class JavaOwner {\n" + "\tint method() { return 1; }\n" + "}\n" + "\n" + "class JavaDecoy {\n" + "\tint method() { return 2; }\n" + "}\n" + "\n" + "class JavaUse {\n" + "\tint javaS1(JavaOwner o) { return o.method(); }\n" + "\n" + "\tint javaS2(JavaUndeclared u) { return u.computeAbsent(); }\n" + "}\n"); + // ── Python ──────────────────────────────────────────────────────── + writeFile(proj_dir + "/src/lang.py", + "class PyOwner:\n" + " def method(self):\n" + " return 1\n" + "\n" + "\n" + "class PyDecoy:\n" + " def method(self):\n" + " return 2\n" + "\n" + "\n" + "def pyS1(o: PyOwner):\n" + " return o.method()\n" + "\n" + "\n" + "def pyS2(u: PyUndeclared):\n" + " return u.compute_absent()\n"); + // ── TypeScript ──────────────────────────────────────────────────── + writeFile(proj_dir + "/src/lang.ts", + "class TsOwner {\n" + "\tmethod(): number { return 1; }\n" + "}\n" + "\n" + "class TsDecoy {\n" + "\tmethod(): number { return 2; }\n" + "}\n" + "\n" + "function tsS1(o: TsOwner): number { return o.method(); }\n" + "\n" + "function tsS2(u: TsUndeclared): number { return u.computeAbsent(); }\n"); + // ── Rust ────────────────────────────────────────────────────────── + writeFile(proj_dir + "/src/lang.rs", + "struct RsOwner;\n" + "\n" + "struct RsDecoy;\n" + "\n" + "impl RsOwner {\n" + "\tfn method(&self) -> i32 { 1 }\n" + "}\n" + "\n" + "impl RsDecoy {\n" + "\tfn method(&self) -> i32 { 2 }\n" + "}\n" + "\n" + "fn rsS1(o: &RsOwner) -> i32 { o.method() }\n" + "\n" + "fn rsS2(u: RsUndeclared) -> i32 { u.compute_absent() }\n"); + + const std::vector cases = { + { "go", "src/lang.go", "goS1", "GoOwner", "Method", "GoDecoy", + "goS2", "computeAbsent", false }, + { "c++", "src/lang.cpp", "cppS1", "CppOwner", "Method", "CppDecoy", + "cppS2", "computeAbsent", false }, + { "java", "src/Lang.java", "javaS1", "JavaOwner", "method", + "JavaDecoy", "javaS2", "computeAbsent", false }, + { "python", "src/lang.py", "pyS1", "PyOwner", "method", "PyDecoy", + "pyS2", "compute_absent", false }, + { "typescript", "src/lang.ts", "tsS1", "TsOwner", "method", + "TsDecoy", "tsS2", "computeAbsent", false }, + { "rust", "src/lang.rs", "rsS1", "RsOwner", "method", "RsDecoy", + "rsS2", "compute_absent", false }, + }; + + const std::string db_path = proj_dir + "/consistency.db"; + unlink(db_path.c_str()); + unlink((db_path + "-wal").c_str()); + unlink((db_path + "-shm").c_str()); + + g_engine = engine_create(db_path.c_str()); + CHECK(g_engine != nullptr); + const uint64_t pid = engine_create_project(g_engine, proj_dir.c_str(), + "language-consistency"); + CHECK(pid > 0); + + std::string request = "{\"paths\":["; + for (size_t i = 0; i < cases.size(); ++i) { + if (i) + request += ","; + request += "\"" + proj_dir + "/" + cases[i].file + "\""; + } + request += "]}"; + char *res = engine_index_files(g_engine, pid, request.c_str(), 1); + CHECK(res != nullptr); + CHECK(strstr(res, "\"ok\":true") != nullptr); + engine_free_string(res); + // engine_destroy() joins the async knowledge builder before closing the + // store, and every row asserted below is written synchronously by the + // index path, so reading the database afterwards needs no sleep. + engine_destroy(g_engine); + g_engine = nullptr; + + sqlite3 *db = nullptr; + CHECK(sqlite3_open(db_path.c_str(), &db) == SQLITE_OK); + + printf(" %-11s %-12s %-14s %-14s %-12s\n", "language", "receiver_type", + "S1 -> owner", "S1 -> decoy", "S2 -> none"); + size_t s1_resolved = 0; + for (const LanguageCase &c : cases) { + // The fixture must really have parsed: without the caller entity the + // assertions below would measure nothing. + CHECK(scalarSql(db, ("SELECT COUNT(*) FROM entity WHERE name='" + + std::string(c.s1_caller) + "'").c_str()) == 1); + CHECK(scalarSql(db, ("SELECT COUNT(*) FROM entity WHERE name='" + + std::string(c.s2_caller) + "'").c_str()) == 1); + // Both methods of S1 must exist, or "no edge to the decoy" would hold + // because the decoy was never recorded rather than because the + // resolver rejected it. Counted by file and name rather than by + // qualified_name: Rust and TypeScript record the bare method name with + // no owner at all, so only the file and name are comparable. + CHECK(scalarSql(db, ("SELECT COUNT(*) FROM entity WHERE kind IN (0,1) " + "AND name='" + + std::string(c.s1_method) + + "' AND file_path LIKE '%" + c.file + "'") + .c_str()) == 2); + + // S2: nothing declares that receiver type or that method name, so + // there is no evidence to resolve it with. + const std::string absent = + edgeKind(db, c.s2_caller, c.s2_callee); + CHECK(absent == "(no edge)"); + + std::string owner = "(no edge)"; + std::string decoy = "(no edge)"; + const long long outgoing = outgoingCallEdges(db, c.s1_caller); + if (c.s1_gap) { + // No receiver type reaches the resolver, so it must abstain + // rather than pick one of the two same-named methods. + CHECK(outgoing == 0); + } else { + // S1: the parameter's type owns the method... + owner = edgeKindInType(db, c.s1_caller, c.s1_method, + c.s1_owner); + // ...and the unrelated same-named method must not be reached. + decoy = edgeKindInType(db, c.s1_caller, c.s1_method, + c.s1_decoy); + CHECK(outgoing == 1); + CHECK(owner != "(no edge)"); + // An edge to the decoy means the resolver preferred the name over + // the receiver type the language recorded. + CHECK(decoy == "(no edge)"); + ++s1_resolved; + } + + printf(" %-11s %-12s %-14s %-14s %-12s\n", c.language, + receiverType(db, c.s1_caller).c_str(), owner.c_str(), + decoy.c_str(), absent.c_str()); + } + + size_t expected_resolved = 0; + for (const LanguageCase &c : cases) + if (!c.s1_gap) + ++expected_resolved; + CHECK(s1_resolved == expected_resolved); + if (expected_resolved != cases.size()) { + printf(" note: %zu of %zu languages resolve S1; the rest record no " + "receiver type (see the s1_gap comment)\n", + s1_resolved, cases.size()); + } + + sqlite3_close(db); + // The fixture directory (sources and database) is left behind on failure so + // the recorded qualified names and the missing edges can be inspected. + if (checkFailures()) { + fprintf(stderr, + "=== resolver language consistency: FAILED ===\n" + " fixture kept at %s\n", + proj_dir.c_str()); + return 1; + } + std::filesystem::remove_all(proj_dir); + printf("=== resolver language consistency test passed ===\n"); + return 0; +} diff --git a/engine/tests/test_resolver_language_filter.cpp b/engine/tests/test_resolver_language_filter.cpp new file mode 100644 index 0000000..1190c75 --- /dev/null +++ b/engine/tests/test_resolver_language_filter.cpp @@ -0,0 +1,374 @@ +// test_resolver_language_filter.cpp — regression tests for the resolver's +// language hard filter. +// +// Guards three defects fixed together: +// +// 1. The single-candidate fast path in ResolverPipeline::run() did not +// apply the language hard filter that the main candidate loop applies. +// A lone same-directory candidate in another language therefore +// produced a cross-language CALLS edge with confidence 0.85 +// (a .cpp call site resolved to a .py entity). +// +// 2. The main loop compared raw language labels, but the path-based +// classifier reports ".c" as "cpp" while the C visitor labels the +// translation unit "c". Every legitimate C call site was rejected +// whenever the fast path did not apply. C and C++ must be treated as +// one language family. +// +// 3. The same mismatch in the JS/TS family: the path classifier reports +// ".tsx" as "typescript" while the tsx visitor labels the entities it +// defines "tsx", so a TSX call site could never resolve to a TSX +// entity. Found on real projects rather than in theory: AIScope and +// PolyScope both produced ZERO call edges (37 and 9 references +// respectively named locally-defined symbols). javascript/typescript/ +// tsx form one family. +// +// Both scenarios are checked end-to-end: source files are written to disk, +// indexed through the FFI, and the resulting `relation` rows are inspected +// directly in SQLite. +// +// Boundary cases covered: +// - a single cross-language candidate that shares the caller's directory +// (must NOT produce an edge) +// - a single cross-file same-language-family candidate with a forward +// declaration in the caller's own file (must produce an edge) +// +// Exit code: 0 on success, 1 on any failed assertion. + +#include "../include/engine.h" +#include "../src/resolver/factors.h" + +#include +#include +#include +#include +#include +#include +#include +#include "test_engine_handle.h" + +using namespace resolver; + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "FAIL: %s\n", msg); + exit(1); + } +} + +/// Count CALLS (relation.type=1) rows, optionally filtered by the source +/// and target entity names. Empty filter strings mean "no filter". +static int countCalls(sqlite3 *db, uint64_t project_id, const char *source_name, + const char *target_name) +{ + const char *sql = "SELECT COUNT(*) FROM relation r " + "JOIN entity src ON src.id = r.source_id " + "JOIN entity tgt ON tgt.id = r.target_id " + "WHERE r.project_id=? AND r.type=1 " + "AND (? = '' OR src.name = ?) " + "AND (? = '' OR tgt.name = ?)"; + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK, + "prepare countCalls"); + sqlite3_bind_int64(st, 1, static_cast(project_id)); + sqlite3_bind_text(st, 2, source_name, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(st, 3, source_name, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(st, 4, target_name, -1, SQLITE_TRANSIENT); + sqlite3_bind_text(st, 5, target_name, -1, SQLITE_TRANSIENT); + int count = -1; + if (sqlite3_step(st) == SQLITE_ROW) + count = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + return count; +} + +/// Count CALLS edges whose target lives in a file with the given suffix. +static int countCallsToFileSuffix(sqlite3 *db, uint64_t project_id, + const char *suffix) +{ + const char *sql = "SELECT COUNT(*) FROM relation r " + "JOIN entity tgt ON tgt.id = r.target_id " + "WHERE r.project_id=? AND r.type=1 " + "AND tgt.file_path LIKE ?"; + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK, + "prepare countCallsToFileSuffix"); + sqlite3_bind_int64(st, 1, static_cast(project_id)); + std::string pattern = std::string("%") + suffix; + sqlite3_bind_text(st, 2, pattern.c_str(), -1, SQLITE_TRANSIENT); + int count = -1; + if (sqlite3_step(st) == SQLITE_ROW) + count = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + return count; +} + +/// Write `content` to `path`, creating parent directories. +static void writeFile(const std::string &path, const char *content) +{ + std::filesystem::create_directories( + std::filesystem::path(path).parent_path()); + FILE *f = fopen(path.c_str(), "w"); + check(f != nullptr, "fopen fixture"); + fputs(content, f); + fclose(f); +} + +/// Index one directory and leave the engine open for SQL inspection. +static void indexDir(const char *db_path, const std::string &dir, + uint64_t *out_pid) +{ + unlink(db_path); + g_engine = engine_create(db_path); + check(g_engine != nullptr, "engine_init"); + uint64_t pid = + engine_create_project(g_engine, dir.c_str(), "lang-filter"); + check(pid > 0, "create_project"); + char *idx = engine_index_project(g_engine, pid, dir.c_str(), nullptr); + check(idx != nullptr, "index_project null"); + check(strstr(idx, "\"ok\":true") != nullptr, "index_project ok"); + engine_free_string(idx); + *out_pid = pid; +} + +// ── Scenario 1: cross-language single candidate must be rejected ── +static void testCrossLanguageCandidateRejected() +{ + const std::string root = "/tmp/codescope_langfilter_mix"; + std::filesystem::remove_all(root); + // The C++ caller and the Python candidate share one directory, so the + // fast path's same_dir gate is satisfied — only the language rule can + // reject it. + writeFile(root + "/unit/caller.cpp", "void caller() { helper(); }\n"); + writeFile(root + "/unit/helper.py", "def helper():\n return 1\n"); + + const char *db_path = "/tmp/test_resolver_lang_filter_mix.db"; + uint64_t pid = 0; + indexDir(db_path, root, &pid); + + sqlite3 *db = nullptr; + check(sqlite3_open(db_path, &db) == SQLITE_OK, "sqlite_open"); + int to_python = countCallsToFileSuffix(db, pid, ".py"); + int caller_calls = countCalls(db, pid, "caller", ""); + sqlite3_close(db); + + check(to_python == 0, + "cross-language edge: a C++ call site must not resolve to a " + "Python entity"); + check(caller_calls == 0, + "caller() must have no CALLS edges (its only candidate is " + "Python)"); + + engine_destroy(g_engine); + g_engine = nullptr; + std::filesystem::remove_all(root); + printf(" [PASS] cross-language single candidate rejected\n"); +} + +// ── Scenario 2: C-family candidate must still resolve ── +static void testCFamilyCandidateAccepted() +{ + const std::string root = "/tmp/codescope_langfilter_cfam"; + std::filesystem::remove_all(root); + // `main` in a.c calls helper(), which is defined in b.c. The caller's + // language is classified "cpp" from the path while the entity is + // labelled "c" — the C family rule must let this resolve. + writeFile(root + "/src/a.c", + "void helper(void);\n" + "int main(void) { helper(); return 0; }\n"); + writeFile(root + "/src/b.c", "void helper(void) { }\n"); + + const char *db_path = "/tmp/test_resolver_lang_filter_cfam.db"; + uint64_t pid = 0; + indexDir(db_path, root, &pid); + + sqlite3 *db = nullptr; + check(sqlite3_open(db_path, &db) == SQLITE_OK, "sqlite_open"); + int main_to_helper = countCalls(db, pid, "main", "helper"); + sqlite3_close(db); + + check(main_to_helper >= 1, + "C call edge lost: main -> helper must resolve across files " + "inside the C family"); + + engine_destroy(g_engine); + g_engine = nullptr; + std::filesystem::remove_all(root); + printf(" [PASS] C-family candidate accepted (main -> helper)\n"); +} + +// ── Scenario 3: the JS/TS family must resolve across .tsx files ── +// +// A `.tsx` call site is classified "typescript" from its path while the +// entity it calls is labelled "tsx" by the visitor, so the two labels never +// matched. Two real projects (AIScope, PolyScope) produced zero call edges +// because of it. +static void testJsTsFamilyCandidateAccepted() +{ + const std::string root = "/tmp/codescope_langfilter_jsts"; + std::filesystem::remove_all(root); + writeFile(root + "/web/Widget.tsx", + "export function widgetHelper() { return 1; }\n"); + writeFile(root + "/web/App.tsx", + "import { widgetHelper } from './Widget';\n" + "export function App() { return widgetHelper(); }\n"); + + const char *db_path = "/tmp/test_resolver_lang_filter_jsts.db"; + uint64_t pid = 0; + indexDir(db_path, root, &pid); + + sqlite3 *db = nullptr; + check(sqlite3_open(db_path, &db) == SQLITE_OK, "sqlite_open"); + int app_to_helper = countCalls(db, pid, "App", "widgetHelper"); + int to_tsx = countCallsToFileSuffix(db, pid, ".tsx"); + sqlite3_close(db); + + check(app_to_helper >= 1, + "JS/TS family edge lost: a .tsx call site must resolve to a .tsx " + "entity (path label 'typescript' vs visitor label 'tsx')"); + check(to_tsx >= 1, "no CALLS edge into the .tsx file at all"); + + engine_destroy(g_engine); + g_engine = nullptr; + std::filesystem::remove_all(root); + printf(" [PASS] JS/TS family candidate accepted (App -> widgetHelper)\n"); +} + +// ── The relative-import matcher itself ── +// +// This is the pure part of the ImportModuleMatch factor, and it is tested +// directly rather than through a fixture because the fixture would pass for the +// wrong reason: with the current `import` table the factor cannot fire at all. +// The table records the module request in `target_path` (`../lib/Widget`) but +// its `alias` column holds the module's LAST SEGMENT, not the name the module +// was bound to, so a bare call to `widgetHelper` cannot be linked back to its +// module. Tested here: the matcher, which is what the factor will use once that +// link exists. +static void testRelativeImportMatcher() +{ + using resolver::relativeImportMatchesFile; + const std::string dir = "/p/app/src"; + check(relativeImportMatchesFile(dir, "./helper", + "/p/app/src/helper.ts"), + "same-directory ./helper"); + check(relativeImportMatchesFile(dir, "../lib/Widget", + "/p/app/lib/Widget.tsx"), + "../lib/Widget resolves to the .tsx file"); + check(relativeImportMatchesFile(dir, "../lib/Widget", + "/p/app/lib/Widget.ts"), + "the match is extension-agnostic"); + check(relativeImportMatchesFile(dir, "./utils/index.ts", + "/p/app/src/utils/index.ts"), + "an explicit file specifier"); + check(relativeImportMatchesFile(dir, "./utils", + "/p/app/src/utils/index.ts"), + "./utils may name a directory entry point"); + // Negative controls: anything that names no project file must not match, + // or the factor would invent evidence. + check(!relativeImportMatchesFile(dir, "react", "/p/app/src/react.ts"), + "a bare package name is never path evidence"); + check(!relativeImportMatchesFile(dir, "@/lib/utils", + "/p/app/src/lib/utils.ts"), + "a path ALIAS is not resolvable without tsconfig paths"); + check(!relativeImportMatchesFile(dir, "../other/Decoy", + "/p/app/lib/Widget.tsx"), + "a different module must not match"); + // With an unknown caller directory a relative specifier cannot be turned + // into a path at all: answering yes would be a guess. + check(!relativeImportMatchesFile("", "./helper", "/p/helper.ts"), + "no caller directory means no path evidence"); + printf(" [PASS] relative import matcher (5 positive, 4 negative)\n"); +} + +// ── Scenario 4: a RELATIVE import must beat a same-named decoy ── +// +// The end-to-end half of the ImportModuleMatch evidence (the matcher itself is +// tested above). TWO candidates share the name in different directories, so +// this measures the import evidence rather than "a lone candidate wins on any +// positive score" — the first version of this scenario had one candidate and +// stayed green even with the factor disabled, i.e. it proved nothing. +static void testRelativeImportBeatsDecoy() +{ + const std::string root = "/tmp/codescope_langfilter_importpath"; + std::filesystem::remove_all(root); + // The decoy lives in `app/decoy/`, which sorts BEFORE `app/lib/`: entity + // ids follow the scan order, and the resolver keeps the first candidate + // when scores are equal (`c.total_score > best_score` is a strict + // comparison). So without the import evidence the decoy wins, which is + // what makes this scenario measure the evidence rather than the tie-break. + // Two earlier versions of this fixture passed with the evidence disabled — + // one had a single candidate, the other had a decoy that sorted last. + // The decoy sits in `app/decoy/lib/` so the module's last segment ("lib") + // appears in BOTH candidates' paths: the pre-existing import heuristic + // matches on that segment and is therefore tied, while resolving the + // specifier to a path picks exactly one. Its directory also sorts first, so + // a tie would hand the edge to the decoy. + writeFile(root + "/app/decoy/lib/Decoy.tsx", + "export function widgetHelper() { return 2; }\n"); + writeFile(root + "/app/lib/Widget.tsx", + "export function widgetHelper() { return 1; }\n"); + // A DIRECTORY import on purpose: the module's last segment ("lib") matches + // no file stem, so the pre-existing heuristic that compares a candidate's + // file name with the import's last segment (`../lib/Widget` → `Widget`) + // cannot resolve this — only matching the specifier against the candidate's + // PATH can, which is what ImportModuleMatch does. With `../lib/Widget` as + // the specifier this scenario passed even with the new evidence disabled. + writeFile(root + "/app/src/App.tsx", + "import { widgetHelper } from '../lib';\n" + "export function App() { return widgetHelper(); }\n"); + + const char *db_path = "/tmp/test_resolver_lang_filter_importpath.db"; + uint64_t pid = 0; + indexDir(db_path, root, &pid); + + sqlite3 *db = nullptr; + check(sqlite3_open(db_path, &db) == SQLITE_OK, "sqlite_open"); + int to_imported = countCallsToFileSuffix(db, pid, "lib/Widget.tsx"); + int to_decoy = countCallsToFileSuffix(db, pid, "decoy/lib/Decoy.tsx"); + int app_calls_helper = countCalls(db, pid, "App", "widgetHelper"); + sqlite3_close(db); + + check(to_imported >= 1, + "the relative import must win over a same-named decoy in another " + "directory: App() imports widgetHelper from ../lib/Widget"); + check(to_decoy == 0, "the decoy definition must not be chosen"); + check(app_calls_helper == 1, "exactly one widgetHelper call edge"); + + engine_destroy(g_engine); + g_engine = nullptr; + std::filesystem::remove_all(root); + printf(" [PASS] relative import beats a same-named decoy\n"); +} + +// ── The family matrix, including the pairs that must stay incompatible ── +static void testLanguageFamilyMatrix() +{ + check(languagesCompatible("c", "cpp"), "C family"); + check(languagesCompatible("javascript", "typescript"), "js/ts"); + check(languagesCompatible("typescript", "tsx"), "ts/tsx"); + check(languagesCompatible("javascript", "tsx"), "js/tsx"); + // Negative controls: unrelated languages must remain incompatible, or + // the filter stops being a filter. + check(!languagesCompatible("python", "rust"), "python/rust"); + check(!languagesCompatible("python", "typescript"), "python/ts"); + check(!languagesCompatible("go", "typescript"), "go/ts"); + check(!languagesCompatible("rust", "tsx"), "rust/tsx"); + check(!languagesCompatible("java", "javascript"), "java/js"); + // An unknown label never filters anything. + check(languagesCompatible("", "rust"), "empty is always compatible"); + printf(" [PASS] language family matrix (4 compatible, 5 rejected)\n"); +} + +int main() +{ + printf("=== resolver language filter regression tests ===\n"); + testCrossLanguageCandidateRejected(); + testCFamilyCandidateAccepted(); + testJsTsFamilyCandidateAccepted(); + testRelativeImportMatcher(); + testRelativeImportBeatsDecoy(); + testLanguageFamilyMatrix(); + printf("\nAll resolver language filter tests passed.\n"); + return 0; +} diff --git a/engine/tests/test_resolver_name_gate.cpp b/engine/tests/test_resolver_name_gate.cpp new file mode 100644 index 0000000..fa72650 --- /dev/null +++ b/engine/tests/test_resolver_name_gate.cpp @@ -0,0 +1,184 @@ +// test_resolver_name_gate.cpp — the scored path must not resolve a reference +// to a same-directory function that only looks similar. +// +// Regression lock for the false edges measured on two real projects: a +// `count()` reference on a std::chrono duration (a method the parser had +// already marked `external`) resolved to the unrelated `countLines`, and on +// this repository that one target collected 10 callers from 10 different +// functions although its only real caller is one. +// +// The dominant evidence was ImportMatch (weight 0.80), which fired on +// location alone: `caller_dir == cand_dir` gave every candidate 1.0, so with a +// couple of fuzzy candidates present the winner cleared +// kFuzzyResolutionThreshold (0.55) without its name ever being compared. Two +// same-directory fuzzy candidates are therefore part of the fixture: with only +// one, the single-candidate fast path in run() would answer first and this +// test would not reach the scorer it exists to pin down. +// +// Deliberately NOT covered here: that fast path's own lone-candidate +// acceptance is by design (Case D of test_resolution_kind asserts +// "DeltaThi" -> "DeltaThing"), and it keeps working. +#include "../include/engine.h" + +#include +#include +#include +#include +#include +#include +#include + +#include "test_check.h" +#include "test_engine_handle.h" + +namespace +{ + +/// Run one scalar SQL query (COUNT(*)) against the test database. +long long scalarSql(sqlite3 *db, const char *sql) +{ + sqlite3_stmt *st = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &st, nullptr) == SQLITE_OK); + long long value = -1; + if (sqlite3_step(st) == SQLITE_ROW) + value = sqlite3_column_int64(st, 0); + sqlite3_finalize(st); + return value; +} + +} // namespace + +int main() +{ + const char *proj_dir = "/tmp/resolver_name_gate_repro"; + std::filesystem::remove_all(proj_dir); + std::filesystem::create_directories(std::string(proj_dir) + "/src"); + + // `src/unrelated.cpp` holds a same-directory function whose name merely + // starts with "count"; the caller below never calls it. The second + // candidate sits in another directory so the winner is not a tie — the + // same-directory one used to win on the ImportMatch bonus alone. + { + FILE *f = fopen( + (std::string(proj_dir) + "/src/unrelated.cpp").c_str(), + "w"); + CHECK(f != nullptr); + fputs("#include \n" + "\n" + "int countLines(const std::string &content)\n" + "{\n" + "\treturn static_cast(content.size());\n" + "}\n" + "\n" + "long long time_since_epoch_ms()\n" + "{\n" + "\treturn 0;\n" + "}\n", + f); + fclose(f); + } + std::filesystem::create_directories(std::string(proj_dir) + "/other"); + { + FILE *f = + fopen((std::string(proj_dir) + "/other/unrelated2.cpp") + .c_str(), + "w"); + CHECK(f != nullptr); + fputs("#include \n" + "\n" + "int countWords(const std::string &content)\n" + "{\n" + "\treturn static_cast(content.size());\n" + "}\n", + f); + fclose(f); + } + + // `src/worker.cpp` lives in the SAME directory and references `count` (a + // std::chrono duration member, i.e. the parser's `external` strategy), + // plus one genuine same-file call that must keep resolving. + { + FILE *f = fopen( + (std::string(proj_dir) + "/src/worker.cpp").c_str(), + "w"); + CHECK(f != nullptr); + fputs("#include \n" + "\n" + "static long long helper()\n" + "{\n" + "\treturn 1;\n" + "}\n" + "\n" + "long long elapsedMs()\n" + "{\n" + "\tauto t = std::chrono::steady_clock::now();\n" + "\treturn std::chrono::duration_cast(\n" + "\t\t t.time_since_epoch())\n" + "\t\t.count();\n" + "}\n" + "\n" + "long long stamp()\n" + "{\n" + "\treturn helper();\n" + "}\n", + f); + fclose(f); + } + + char db[] = "/tmp/test_resolver_name_gate.db"; + unlink(db); + g_engine = engine_create(db); + CHECK(g_engine != nullptr); + const uint64_t pid = + engine_create_project(g_engine, proj_dir, "name-gate"); + CHECK(pid > 0); + char *idx = engine_index_project(g_engine, pid, proj_dir, nullptr); + CHECK(idx != nullptr); + CHECK(strstr(idx, "\"ok\":true") != nullptr); + engine_free_string(idx); + usleep(500000); // let the async knowledge builder finish + + sqlite3 *db_h = nullptr; + CHECK(sqlite3_open(db, &db_h) == SQLITE_OK); + + // The reference itself must exist, or this test would pass for the wrong + // reason: a parse miss instead of a resolver abstention. + CHECK(scalarSql(db_h, + "SELECT COUNT(*) FROM reference WHERE name='count'") > + 0); + + // Neither unrelated same-directory function may collect a caller. + CHECK(scalarSql(db_h, + "SELECT COUNT(*) FROM relation r JOIN entity te " + "ON te.id=r.target_id WHERE te.name='countLines'") == + 0); + CHECK(scalarSql(db_h, + "SELECT COUNT(*) FROM relation r JOIN entity te " + "ON te.id=r.target_id WHERE te.name='countWords'") == + 0); + + // A receiver-carrying reference must not be fuzzy-resolved either: the + // parser records `t.time_since_epoch()` as a method call (call_kind=1), + // and a same-directory FREE function is not that member. Measured on two + // real projects every such edge was false (Instant::now -> now_ms, + // os.Stat -> modTime, math.Pow -> powFunc, Command::new -> the variable + // `expected`), while the genuine calls at those sites still resolve. + CHECK(scalarSql(db_h, "SELECT COUNT(*) FROM relation r JOIN entity te " + "ON te.id=r.target_id " + "WHERE te.name='time_since_epoch_ms'") == 0); + + // ...while the genuine call is still resolved: the fix must narrow the + // resolver, not silence it. + CHECK(scalarSql( + db_h, + "SELECT COUNT(*) FROM relation r JOIN entity se " + "ON se.id=r.source_id JOIN entity te ON te.id=r.target_id " + "WHERE se.name='stamp' AND te.name='helper'") == 1); + + printf("=== resolver name-gate test passed ===\n"); + sqlite3_close(db_h); + engine_destroy(g_engine); + g_engine = nullptr; + unlink(db); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_rules_dir_resolution.cpp b/engine/tests/test_rules_dir_resolution.cpp new file mode 100644 index 0000000..1e990b6 --- /dev/null +++ b/engine/tests/test_rules_dir_resolution.cpp @@ -0,0 +1,107 @@ +// test_rules_dir_resolution: pin evidence::resolveRulesDir() so the +// rule directory is found independently of the process working +// directory. +// +// Regression covered: engine_build_evidence / ProjectStateBuilder used +// to fall back to the CWD-relative "engine/src/evidence/rules", which +// only exists when running from the repository root. The MCP/CLI +// vehicles run inside the target project, so build_evidence silently +// loaded zero rules and returned [] (T5 finding #5). +// +// Test flow: +// 1. chdir into a fresh empty temp dir (so every relative candidate +// except the build-time absolute default is absent) +// 2. resolveRulesDir() must return a directory containing *.json +// 3. $CODESCOPE_RULES_DIR must win over the built-in default +// 4. restore CWD and clear the override + +#include "../src/evidence/rule.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace +{ + +bool dirHasJson(const std::string &dir) +{ + std::error_code ec; + if (!std::filesystem::is_directory(dir, ec)) + return false; + for (const auto &entry : std::filesystem::directory_iterator(dir, ec)) { + if (ec) + break; + if (entry.is_regular_file() && + entry.path().extension() == ".json") + return true; + } + return false; +} + +} // namespace + +int main() +{ + // Remember the real CWD so the test can restore it. + char original_cwd[4096]; + if (!getcwd(original_cwd, sizeof(original_cwd))) { + fprintf(stderr, "FAIL: getcwd failed\n"); + return 1; + } + + // 1. A fresh empty directory: no engine/src/evidence/rules under it, + // so the CWD-relative fallback cannot succeed. + std::filesystem::path foreign = std::filesystem::temp_directory_path() / + "codescope_rules_dir_test"; + std::error_code ec; + std::filesystem::remove_all(foreign, ec); + std::filesystem::create_directories(foreign, ec); + if (chdir(foreign.c_str()) != 0) { + fprintf(stderr, "FAIL: chdir(%s) failed\n", foreign.c_str()); + return 1; + } + + // 2. The build-time default (or the env override, if the operator + // set one before running the test) must still resolve. + std::string resolved = evidence::resolveRulesDir(); + if (resolved.empty() || !dirHasJson(resolved)) { + fprintf(stderr, + "FAIL: resolveRulesDir() from foreign CWD returned " + "'%s' (expected a rules directory with *.json)\n", + resolved.c_str()); + chdir(original_cwd); + return 1; + } + printf("Test 1 (resolveRulesDir from foreign CWD = %s): PASS\n", + resolved.c_str()); + + // 3. $CODESCOPE_RULES_DIR must win over every other candidate. + // Point it at the directory step 2 found (it is known-good). + setenv("CODESCOPE_RULES_DIR", resolved.c_str(), 1); + std::string overridden = evidence::resolveRulesDir(); + if (overridden != resolved) { + fprintf(stderr, + "FAIL: $CODESCOPE_RULES_DIR ignored — got '%s', " + "expected '%s'\n", + overridden.c_str(), resolved.c_str()); + unsetenv("CODESCOPE_RULES_DIR"); + chdir(original_cwd); + return 1; + } + printf("Test 2 ($CODESCOPE_RULES_DIR override honoured): PASS\n"); + unsetenv("CODESCOPE_RULES_DIR"); + + // 4. Restore CWD and clean up the temp dir. + if (chdir(original_cwd) != 0) + fprintf(stderr, "warning: chdir back to %s failed\n", + original_cwd); + std::filesystem::remove_all(foreign, ec); + + printf("=== test_rules_dir_resolution PASSED ===\n"); + return 0; +} diff --git a/engine/tests/test_rust_e2e.cpp b/engine/tests/test_rust_e2e.cpp index f569c60..cda94e5 100644 --- a/engine/tests/test_rust_e2e.cpp +++ b/engine/tests/test_rust_e2e.cpp @@ -1,7 +1,8 @@ #include "test_e2e.h" -int main() { - const char* code = R"(fn add(a: i32, b: i32) -> i32 { +int main() +{ + const char *code = R"(fn add(a: i32, b: i32) -> i32 { a + b } @@ -25,12 +26,8 @@ fn main() { } )"; - const char* defs[] = {"add", "multiply", "compute"}; - runE2eTest("rust", code, "/tmp/test_math.rs", - defs, 3, - "add", "multiply", - "add", - "compute", - "main", nullptr); - return 0; + const char *defs[] = { "add", "multiply", "compute" }; + runE2eTest("rust", code, "/tmp/test_math.rs", defs, 3, "add", + "multiply", "add", "compute", "main", nullptr); + return 0; } diff --git a/engine/tests/test_schema_reopen.cpp b/engine/tests/test_schema_reopen.cpp new file mode 100644 index 0000000..a2a631c --- /dev/null +++ b/engine/tests/test_schema_reopen.cpp @@ -0,0 +1,203 @@ +// test_schema_reopen.cpp — regression test for createSchema() on an +// already-initialised database. +// +// Defect guarded: the "create type_info table if missing" migration block +// finalised the WRONG statement handle on the branch taken when the table +// already exists — it called sqlite3_finalize() on the `probe` handle that +// the earlier semantic_records migration had already finalised (a +// use-after-free), and leaked the probe2 handle it had just prepared. +// Every reopen of an existing database took that branch. +// +// Two observable consequences are asserted, so the test is meaningful even +// in a Release build where assert() is compiled out: +// +// 1. The schema stays intact and writable across repeated reopens. +// 2. The number of outstanding prepared statements does NOT grow with +// each reopen — the leaked probe2 handle used to accumulate. The +// count is compared against the value after the first reopen so the +// test is independent of whatever baseline statements open() caches. +// +// Uses the explicit check() pattern instead of assert() because the +// default engine build defines NDEBUG. +// +// Exit code: 0 on success, 1 on any failed assertion. + +#include "../src/store/store.h" + +#include +#include +#include +#include +#include + +static const char *kDbPath = "/tmp/codescope_test_schema_reopen.db"; + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "FAIL: %s\n", msg); + exit(1); + } +} + +/// Remove the database and its WAL side files so the first open below is +/// guaranteed to create the schema from scratch (a leftover -wal could +/// otherwise make the "fresh" open behave like a reopen). +static void removeDbFiles() +{ + unlink(kDbPath); + std::string wal = std::string(kDbPath) + "-wal"; + std::string shm = std::string(kDbPath) + "-shm"; + unlink(wal.c_str()); + unlink(shm.c_str()); +} + +/// Run a scalar COUNT(*) query against an open store. +static long long scalarCount(store::GraphStore &store, const char *table) +{ + sqlite3 *db = store.handle(); + std::string sql = std::string("SELECT COUNT(*) FROM ") + table; + sqlite3_stmt *stmt = nullptr; + check(sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) == + SQLITE_OK, + "prepare scalarCount"); + long long n = -1; + if (sqlite3_step(stmt) == SQLITE_ROW) + n = sqlite3_column_int64(stmt, 0); + sqlite3_finalize(stmt); + return n; +} + +/// True when `table` has a column named `column`. +static bool hasColumn(store::GraphStore &store, const char *table, + const char *column) +{ + sqlite3 *db = store.handle(); + std::string sql = std::string("PRAGMA table_info(") + table + ")"; + sqlite3_stmt *stmt = nullptr; + check(sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) == + SQLITE_OK, + "prepare hasColumn"); + bool found = false; + while (sqlite3_step(stmt) == SQLITE_ROW) { + const char *name = reinterpret_cast( + sqlite3_column_text(stmt, 1)); + if (name && std::string(name) == column) + found = true; + } + sqlite3_finalize(stmt); + return found; +} + +/// Number of prepared statements that have not been finalised yet. +/// A createSchema() that leaks a handle makes this grow per reopen. +static int outstandingStatements(store::GraphStore &store) +{ + sqlite3 *db = store.handle(); + int n = 0; + for (sqlite3_stmt *st = sqlite3_next_stmt(db, nullptr); st != nullptr; + st = sqlite3_next_stmt(db, st)) + ++n; + return n; +} + +/// Upper bound on the prepared statements a correct open() may leave +/// outstanding. GraphStore::open() intentionally caches a small, fixed +/// number of hot-path statements; every probe createSchema() prepares must +/// be finalised. The type_info "table already exists" branch used to leak +/// one handle per open, which pushed the count above this bound — so this +/// absolute check catches leaks that a fresh-vs-reopen comparison cannot +/// (that bug leaked on the very first open too, because type_info is +/// created by the main schema DDL before the migration block runs). +/// Raising this limit requires documenting why another statement is +/// cached. +static const int kMaxCachedStatements = 3; + +int main() +{ + removeDbFiles(); + + // Outstanding prepared statements after a FRESH open. This is the + // reference for the reopen cycles: createSchema() must not leak a + // handle on the "table already exists" branches, so a reopen has to + // leave exactly the same number of statements behind as the first, + // table-creating open. (The buggy version leaked one per reopen.) + int fresh_stmts = -1; + + // ── Cycle 1: fresh database — creates every table ───────────── + { + store::GraphStore store; + check(store.open(kDbPath), "open fresh database"); + check(store.createProject("/test", "reopen") > 0, + "createProject"); + check(hasColumn(store, "semantic_records", "receiver_type"), + "semantic_records.receiver_type missing"); + check(hasColumn(store, "reference", "call_site_file"), + "reference.call_site_file missing"); + check(hasColumn(store, "relation", "resolution_kind"), + "relation.resolution_kind missing"); + fresh_stmts = outstandingStatements(store); + printf(" [INFO] outstanding statements after fresh open: %d\n", + fresh_stmts); + check(fresh_stmts <= kMaxCachedStatements, + "createSchema leaked a prepared statement on open() " + "(outstanding statement count above the documented " + "cache bound)"); + // close() returns void; a failed close would surface as a + // failed open in the next cycle. + store.close(); + } + + // ── Cycles 2..4: reopen — every "already exists" branch runs, + // including the type_info branch that used a dangling handle. + for (int cycle = 2; cycle <= 4; cycle++) { + store::GraphStore store; + check(store.open(kDbPath), "reopen database"); + + check(hasColumn(store, "semantic_records", "qualified_target"), + "semantic_records.qualified_target lost on reopen"); + check(hasColumn(store, "reference", "import_alias"), + "reference.import_alias lost on reopen"); + check(hasColumn(store, "relation", "confidence"), + "relation.confidence lost on reopen"); + check(scalarCount(store, "type_info") >= 0, + "type_info not queryable after reopen"); + check(scalarCount(store, "semantic_records") >= 0, + "semantic_records not queryable after reopen"); + + int stmts = outstandingStatements(store); + printf(" [INFO] outstanding statements after reopen %d: %d " + "(fresh baseline %d)\n", + cycle, stmts, fresh_stmts); + check(stmts == fresh_stmts, + "prepared statements leaked across reopens: " + "createSchema did not finalize a handle on the " + "\"table already exists\" branch"); + store.close(); + printf(" [PASS] reopen cycle %d succeeded\n", cycle); + } + + // ── Cycle 5: type_info must still be writable after reopens ─── + { + store::GraphStore store; + check(store.open(kDbPath), "reopen for write"); + uint64_t pid = store.createProject("/test2", "reopen2"); + check(pid > 0, "createProject (reopen2)"); + std::string ins = + "INSERT INTO type_info " + "(project_id, name, qualified_name, kind, file_path, " + " language) VALUES (" + + std::to_string(pid) + + ", 'ReopenProbe', 'ReopenProbe', 0, " + "'/test2/x.cpp', 'cpp')"; + check(store.exec(ins.c_str()), "insert into type_info"); + check(scalarCount(store, "type_info") >= 1, + "type_info row not persisted"); + store.close(); + printf(" [PASS] type_info still writable after reopens\n"); + } + + removeDbFiles(); + printf("\nAll schema-reopen tests passed.\n"); + return 0; +} diff --git a/engine/tests/test_scope_dup_migration.cpp b/engine/tests/test_scope_dup_migration.cpp new file mode 100644 index 0000000..80cbcda --- /dev/null +++ b/engine/tests/test_scope_dup_migration.cpp @@ -0,0 +1,212 @@ +// test_scope_dup_migration.cpp — the scope de-duplication migration. +// +// A pre-fix buildGraph accumulated duplicate scope rows: +// - module scopes (kind=1) because `INSERT OR IGNORE` had no UNIQUE key to +// ignore on, so every pass re-appended every module row; +// - function scopes (kind=2) because those older passes could append a scope +// row per (entity, matched module row) and again on later passes. +// The schema migration (store_schema_migrations.cpp) repairs such databases: +// +// 1) repoint kind=2 rows whose parent is a doomed duplicate to the surviving +// (lowest-id) module scope, +// 2) delete the duplicate kind=1 rows, +// 3) delete kind=2 rows that are identical in every inserted column — +// (project_id, parent_id, name, start_row, end_row). +// +// The third step must be precise: several entities in one module legitimately +// share a name (one `init` per file) and differ only by their source range, so +// a name-only grouping would delete distinct scopes. This test builds a dirty +// database with raw SQL, reopens it so the migration runs, asserts both the +// collapse and the survival of legitimate rows, and checks that a second open +// is a no-op. + +#include "../include/engine.h" +#include "../src/store/store.h" + +#include +#include +#include +#include +#include + +static const char *kDbPath = "/tmp/test_scope_dup_migration.db"; + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "\nFAIL: %s\n", msg); + unlink(kDbPath); + exit(1); + } +} + +/// Run a statement on `db`, failing the test on error. +static void run(sqlite3 *db, const std::string &sql) +{ + char *err = nullptr; + if (sqlite3_exec(db, sql.c_str(), nullptr, nullptr, &err) != + SQLITE_OK) { + fprintf(stderr, "\nFAIL: sqlite3_exec: %s | sql=%s\n", + err ? err : "(null)", sql.c_str()); + sqlite3_free(err); + unlink(kDbPath); + exit(1); + } +} + +/// Scalar query helper. +static int scalar(sqlite3 *db, const std::string &sql) +{ + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2(db, sql.c_str(), -1, &st, nullptr) == + SQLITE_OK, + "prepare scalar"); + int n = -1; + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + return n; +} + +/// Insert one scope row; returns its new id. +static int64_t insertScope(sqlite3 *db, uint64_t pid, int64_t parent_id, + int kind, const char *name, int start_row, + int end_row) +{ + const std::string sql = + "INSERT INTO scope (project_id, parent_id, kind, name, start_row, " + "end_row) VALUES (" + + std::to_string(pid) + "," + std::to_string(parent_id) + "," + + std::to_string(kind) + ",'" + name + "'," + + std::to_string(start_row) + "," + std::to_string(end_row) + ")"; + run(db, sql); + return sqlite3_last_insert_rowid(db); +} + +static int countKind(sqlite3 *db, uint64_t pid, int kind) +{ + return scalar(db, "SELECT COUNT(*) FROM scope WHERE project_id=" + + std::to_string(pid) + + " AND kind=" + std::to_string(kind)); +} + +/// COUNT(DISTINCT ) over a project's rows of a kind. +static int countDistinctOf(sqlite3 *db, uint64_t pid, int kind, + const char *cols) +{ + return scalar( + db, + "SELECT COUNT(*) FROM (SELECT DISTINCT " + std::string(cols) + + " FROM scope WHERE project_id=" + std::to_string(pid) + + " AND kind=" + std::to_string(kind) + ")"); +} + +int main() +{ + unlink(kDbPath); + uint64_t pid = 0; + // Kept across the blocks: the surviving module scope id (asserted after + // the migration run). + int64_t moda1 = 0; + + // ── Build a pre-fix-shaped dirty database ──────────────────── + { + store::GraphStore store; + check(store.open(kDbPath), "open database"); + pid = store.createProject("/proj", "scope-dup"); + check(pid > 0, "createProject"); + + sqlite3 *db = store.handle(); + // Two module scopes for "moda" (the pre-fix duplication) + one for + // "modb". + moda1 = insertScope(db, pid, 0, 1, "moda", 0, 0); + const int64_t moda2 = insertScope(db, pid, 0, 1, "moda", 0, 0); + const int64_t modb1 = insertScope(db, pid, 0, 1, "modb", 0, 0); + check(moda1 != moda2, + "duplicate module scopes have distinct ids"); + + // Function scopes. Rows (moda1,a1,10,12), (moda2,a1,10,12) and + // (moda2,a1,10,12) are the SAME logical scope reached through two + // parent ids plus one exact repeat; (moda1,a1,30,33) is a DIFFERENT + // entity that happens to share the module and the name — it must + // survive. Same for the duplicated (modb1,b1,5,7). + insertScope(db, pid, moda1, 2, "a1", 10, 12); + insertScope(db, pid, moda2, 2, "a1", 10, 12); + insertScope(db, pid, moda2, 2, "a1", 10, 12); + insertScope(db, pid, moda1, 2, "a1", 30, 33); + insertScope(db, pid, moda1, 2, "a2", 50, 52); + insertScope(db, pid, modb1, 2, "b1", 5, 7); + insertScope(db, pid, modb1, 2, "b1", 5, 7); + + const int dirty1 = countKind(db, pid, 1); + const int dirty2 = countKind(db, pid, 2); + fprintf(stderr, + "dirty: kind1=%d (expect 3), kind2=%d (expect 7)\n", + dirty1, dirty2); + check(dirty1 == 3, + "fixture must contain duplicate module scopes"); + check(dirty2 == 7, + "fixture must contain duplicate function scopes"); + // 5 distinct (parent, name, range) tuples among the 7 rows: two of + // them are repeats, and (moda2,a1,10,12) will merge into + // (moda1,a1,10,12) once the duplicate parent is collapsed. + check(countDistinctOf(db, pid, 2, + "parent_id,name,start_row,end_row") == 5, + "fixture holds 5 distinct function scopes"); + store.close(); + } + + // ── Reopen: the migration must collapse both kinds ─────────── + { + store::GraphStore store; + check(store.open(kDbPath), "reopen database"); + sqlite3 *db = store.handle(); + + const int modules = countKind(db, pid, 1); + const int funcs = countKind(db, pid, 2); + fprintf(stderr, + "migrated: kind1=%d (expect 2), kind2=%d (expect 4)\n", + modules, funcs); + check(modules == 2, + "migration must keep exactly one module scope per name"); + check(funcs == 4, + "migration must collapse the 3 surplus function rows"); + check(countDistinctOf(db, pid, 2, + "parent_id,name,start_row,end_row") == + funcs, + "no fully-identical function scope may survive"); + // The legitimate same-name/different-range row must NOT be removed: + // (moda,a1) legitimately has two rows, (modb,b1) one. + check(countDistinctOf(db, pid, 2, "parent_id,name") < funcs, + "a name shared by two entities must survive as two rows"); + check(scalar(db, "SELECT COUNT(*) FROM scope WHERE kind=2 AND " + "name='a1' AND start_row=30") == 1, + "the distinct 'a1' at row 30 must survive"); + // No child may be left pointing at a deleted module scope. + check(scalar(db, "SELECT COUNT(*) FROM scope WHERE kind=2 AND " + "parent_id NOT IN (SELECT id FROM scope WHERE " + "kind=1)") == 0, + "every function scope must point at a surviving module " + "scope"); + // The surviving module scope is the lowest id. + check(scalar(db, "SELECT id FROM scope WHERE kind=1 AND " + "name='moda'") == moda1, + "migration must keep the lowest-id module scope"); + store.close(); + } + + // ── Reopen again: the migration is idempotent ──────────────── + { + store::GraphStore store; + check(store.open(kDbPath), "reopen database a second time"); + check(countKind(store.handle(), pid, 1) == 2, + "second migration pass must not change module scopes"); + check(countKind(store.handle(), pid, 2) == 4, + "second migration pass must not change function scopes"); + store.close(); + } + + unlink(kDbPath); + printf("\n=== scope duplicate migration test passed ===\n"); + return 0; +} diff --git a/engine/tests/test_scope_idempotency.cpp b/engine/tests/test_scope_idempotency.cpp new file mode 100644 index 0000000..7117e1f --- /dev/null +++ b/engine/tests/test_scope_idempotency.cpp @@ -0,0 +1,245 @@ +// test_scope_idempotency.cpp — regression test for duplicate module scopes +// (scope.kind=1) on repeated buildGraph. +// +// Before the fix: +// buildGraph populated module scopes with `INSERT OR IGNORE INTO scope`, +// but `scope` has no UNIQUE key on (project_id, kind, name) — kind=2 +// function scopes legitimately repeat a name — so OR IGNORE had no conflict +// target and every buildGraph re-appended one kind=1 row per module. +// buildGraph runs more than once per project (the parallel indexer merges +// per-module worker DBs, then the post-index enhance runs buildGraph again), +// so each module ended up with ≥2 kind=1 rows. buildModuleSummaries GROUP BYs +// scope.id, so the duplicates doubled module_summary rows and dead_code +// counts, and inflated project_overview.total_modules (a project with 2 +// module directories reported 4 modules). +// +// After the fix: +// The insert carries a NOT EXISTS guard, so a module scope is inserted at +// most once per (project_id, name). Calling buildGraph any number of times +// leaves exactly one kind=1 row per module. +// +// This test inserts two files in two different directories (one entity each) +// and calls a full buildGraph three times, asserting the module-scope count +// stays at 2 and the function-scope count at 2, with no exact duplicate rows. + +#include "../include/engine.h" +#include "../src/ir/semantic_unit.h" +#include "../src/store/store.h" + +#include +#include +#include +#include +#include +#include + +static void check(bool cond, const char *msg) +{ + if (!cond) { + fprintf(stderr, "FAIL: %s\n", msg); + exit(1); + } +} + +// Count scope rows of a given kind for a project. +static int scopeCount(store::GraphStore &store, uint64_t pid, int kind) +{ + sqlite3 *db = store.handle(); + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2( + db, + "SELECT COUNT(*) FROM scope WHERE project_id=? AND kind=?", + -1, &st, nullptr) == SQLITE_OK, + "prepare scopeCount"); + sqlite3_bind_int64(st, 1, static_cast(pid)); + sqlite3_bind_int(st, 2, kind); + int n = 0; + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + return n; +} + +// Count DISTINCT (parent_id, name) tuples of a kind. The scope tables must +// never hold an EXACT duplicate row, so this must equal scopeCount(). +static int distinctScopeCount(store::GraphStore &store, uint64_t pid, int kind) +{ + sqlite3 *db = store.handle(); + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2( + db, + "SELECT COUNT(*) FROM (SELECT DISTINCT parent_id, name " + "FROM scope WHERE project_id=? AND kind=?)", + -1, &st, nullptr) == SQLITE_OK, + "prepare distinctScopeCount"); + sqlite3_bind_int64(st, 1, static_cast(pid)); + sqlite3_bind_int(st, 2, kind); + int n = 0; + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + return n; +} + +// Name of the module scope (kind=1) created for `file_path`. module_path is +// the directory portion of the file path; its exact spelling is not part of +// this test's contract, so it is read back instead of hard-coded. Must be +// called while the file's entity rows still exist. +static std::string scopeNameForFile(store::GraphStore &store, uint64_t pid, + const char *file_path) +{ + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2( + store.handle(), + "SELECT name FROM scope WHERE project_id=? AND kind=1 " + "AND name = (SELECT module_path FROM entity WHERE " + "project_id=? AND file_path=?) LIMIT 1", + -1, &st, nullptr) == SQLITE_OK, + "prepare scopeNameForFile"); + sqlite3_bind_int64(st, 1, static_cast(pid)); + sqlite3_bind_int64(st, 2, static_cast(pid)); + sqlite3_bind_text(st, 3, file_path, -1, SQLITE_TRANSIENT); + std::string name; + if (sqlite3_step(st) == SQLITE_ROW) { + const unsigned char *text = sqlite3_column_text(st, 0); + if (text) + name = reinterpret_cast(text); + } + sqlite3_finalize(st); + return name; +} + +// Count module scopes (kind=1) named `name`. +static int countModuleNamed(store::GraphStore &store, uint64_t pid, + const std::string &name) +{ + sqlite3_stmt *st = nullptr; + check(sqlite3_prepare_v2(store.handle(), + "SELECT COUNT(*) FROM scope WHERE project_id=? " + "AND kind=1 AND name=?", + -1, &st, nullptr) == SQLITE_OK, + "prepare countModuleNamed"); + sqlite3_bind_int64(st, 1, static_cast(pid)); + sqlite3_bind_text(st, 2, name.c_str(), -1, SQLITE_TRANSIENT); + int n = 0; + if (sqlite3_step(st) == SQLITE_ROW) + n = sqlite3_column_int(st, 0); + sqlite3_finalize(st); + return n; +} + +// One Function record in the given file. +static ir::Record fn(const char *name, const char *file, uint64_t id) +{ + ir::Record r; + r.id = id; + r.original_id = id; + r.kind = ir::RecordKind::Function; + r.name = name; + r.qualified_name = name; + r.file_path = file; + r.language = "cpp"; + r.loc = ir::SourceRange{ 1, 0, 2, 0 }; + return r; +} + +int main() +{ + const char *db_path = "/tmp/test_scope_idempotency.db"; + unlink(db_path); + + store::GraphStore store; + check(store.open(db_path), "open database"); + uint64_t pid = store.createProject("/proj", "scope-idem"); + check(pid > 0, "createProject"); + + // Two files in two distinct module directories → two module scopes. + const char *file_a = "/proj/moda/a.cpp"; + const char *file_b = "/proj/modb/b.cpp"; + std::vector ra{ fn("a", file_a, 1) }; + std::vector rb{ fn("b", file_b, 2) }; + store.insertSemanticRecords(pid, file_a, ra); + store.insertSemanticRecords(pid, file_b, rb); + + // Three full rebuilds, each of which must leave the scope table unchanged: + // 2 module scopes (one per directory) and 3 function scopes (one per + // entity). Before the fix every pass appended another module row, and + // each duplicated module row dragged a duplicated function row with it — + // the kind=2 insert stores parent_id = the matched kind=1 id, so it + // re-derives one function scope per (entity, module-scope row). + for (int pass = 1; pass <= 3; pass++) { + check(store.buildGraph(pid, false), "buildGraph pass"); + const int modules = scopeCount(store, pid, 1); + const int funcs = scopeCount(store, pid, 2); + fprintf(stderr, + "pass %d: module scopes=%d (expect 2), function scopes=%d " + "(expect 2)\n", + pass, modules, funcs); + check(modules == 2, + "repeated buildGraph must not duplicate module scopes"); + // The fixture declares one entity per file (two entities total), so + // buildGraph derives exactly one function scope per module. + check(funcs == 2, + "repeated buildGraph must not duplicate function scopes"); + // A duplicated function scope is always an EXACT duplicate (same + // parent_id and name) because it is re-derived from the same parent, + // which is what buildModuleSummaries/dead-code consumers join on. + check(distinctScopeCount(store, pid, 2) == funcs, + "function scopes must not contain exact (parent_id, name) " + "duplicates"); + } + + // ── A module that vanishes must not leave its scope behind ─── + // + // buildGraph never deletes a kind=1 row (it is shared by every file in the + // directory), so before the orphan sweep a removed directory kept + // inflating project_overview.total_modules and module_summary. Mirror what + // GraphStore::cleanupStaleFiles does for a file that no longer exists on + // disk: drop its records, then rebuild. + check(scopeCount(store, pid, 1) == 2, "precondition: two modules"); + // Read both module names while their entity rows still exist: the sweep is + // expected to keep moda's and drop modb's. + const std::string name_a = scopeNameForFile(store, pid, file_a); + const std::string name_b = scopeNameForFile(store, pid, file_b); + check(!name_a.empty() && !name_b.empty() && name_a != name_b, + "fixture must define two distinct module scopes"); + { + const std::string del = + "DELETE FROM semantic_records WHERE project_id=" + + std::to_string(pid) + " AND file_path='" + file_b + "'"; + const std::string del_entity = + "DELETE FROM entity WHERE project_id=" + + std::to_string(pid) + " AND file_path='" + file_b + "'"; + char *err = nullptr; + for (const std::string &sql : { del, del_entity }) { + if (sqlite3_exec(store.handle(), sql.c_str(), nullptr, + nullptr, &err) != SQLITE_OK) { + fprintf(stderr, "FAIL: %s | sql=%s\n", + err ? err : "(null)", sql.c_str()); + sqlite3_free(err); + exit(1); + } + } + } + + check(store.buildGraph(pid, false), "buildGraph after module removal"); + fprintf(stderr, + "after removing modb: module scopes=%d (expect 1), function " + "scopes=%d (expect 1)\n", + scopeCount(store, pid, 1), scopeCount(store, pid, 2)); + check(scopeCount(store, pid, 1) == 1, + "the vanished module's scope must be swept"); + check(scopeCount(store, pid, 2) == 1, + "the vanished module's function scopes must be swept with it"); + check(countModuleNamed(store, pid, name_a) == 1, + "the surviving module's scope must be kept"); + check(countModuleNamed(store, pid, name_b) == 0, + "the vanished module's scope must be gone"); + check(scopeCount(store, pid, 2) == distinctScopeCount(store, pid, 2), + "no function scope may be left dangling"); + + store.close(); + unlink(db_path); + printf("\n=== scope idempotency test passed ===\n"); + return 0; +} diff --git a/engine/tests/test_self_inspect.cpp b/engine/tests/test_self_inspect.cpp index 87e914a..f5208dd 100644 --- a/engine/tests/test_self_inspect.cpp +++ b/engine/tests/test_self_inspect.cpp @@ -13,6 +13,7 @@ #include #include #include +#include "test_engine_handle.h" static inline void check(bool cond, const char *msg) { @@ -35,11 +36,8 @@ int main() if (const char *env = std::getenv("CODESCOPE_ENGINE_SRC")) { self_dir = env; } else { - const char *candidates[] = { - "engine/src", - "../src", - "../engine/src", - nullptr}; + const char *candidates[] = { "engine/src", "../src", + "../engine/src", nullptr }; for (int i = 0; candidates[i]; ++i) { if (access(candidates[i], F_OK) == 0) { self_dir = candidates[i]; @@ -60,14 +58,16 @@ int main() char db_path[] = "/tmp/test_self_inspect.db"; unlink(db_path); - int rc = engine_init(db_path); + g_engine = engine_create(db_path); + const int rc = g_engine != nullptr ? 0 : -1; check(rc == 0, "engine_init"); - uint64_t pid = engine_create_project("/tmp", "self-inspect"); + uint64_t pid = engine_create_project(g_engine, "/tmp", "self-inspect"); check(pid > 0, "create_project"); // Index the engine/src directory (CodeScope's own C++ source) - char *result = engine_index_project(pid, self_dir.c_str(), nullptr); + char *result = + engine_index_project(g_engine, pid, self_dir.c_str(), nullptr); check(result != nullptr, "index_project returns non-null"); printf("--- Self-inspect index result ---\n%s\n", result); check(strstr(result, "\"ok\":true") != nullptr, @@ -121,17 +121,17 @@ int main() // An orphan is a function that neither calls nor is called. // Query: functions that appear in NO call edge (neither as // source nor as target). - const char *orphan_sql = - "SELECT COUNT(*) FROM graph_nodes gn " - "WHERE gn.project_id = ? " - "AND gn.node_type = 0 " - "AND gn.id NOT IN (" - " SELECT source_node_id FROM graph_edges " - " WHERE project_id = ? AND edge_type = 1) " - "AND gn.id NOT IN (" - " SELECT target_node_id FROM graph_edges " - " WHERE project_id = ? AND edge_type = 1)"; - check(sqlite3_prepare_v2(db, orphan_sql, -1, &stmt, nullptr) == SQLITE_OK, + const char *orphan_sql = "SELECT COUNT(*) FROM graph_nodes gn " + "WHERE gn.project_id = ? " + "AND gn.node_type = 0 " + "AND gn.id NOT IN (" + " SELECT source_node_id FROM graph_edges " + " WHERE project_id = ? AND edge_type = 1) " + "AND gn.id NOT IN (" + " SELECT target_node_id FROM graph_edges " + " WHERE project_id = ? AND edge_type = 1)"; + check(sqlite3_prepare_v2(db, orphan_sql, -1, &stmt, nullptr) == + SQLITE_OK, "prepare orphan count"); sqlite3_bind_int64(stmt, 1, static_cast(pid)); sqlite3_bind_int64(stmt, 2, static_cast(pid)); @@ -171,10 +171,10 @@ int main() // ── Check 5: sample some known functions to verify edges ── // engine_get_callees on a known function - char *callees = engine_get_callees(pid, "buildCallEdgesSQL", nullptr); + char *callees = + engine_get_callees(g_engine, pid, "buildCallEdgesSQL", nullptr); if (callees) { - printf("--- Callees of buildCallEdgesSQL ---\n%s\n", - callees); + printf("--- Callees of buildCallEdgesSQL ---\n%s\n", callees); // This function should have some callees after P1 fix check(strstr(callees, "callees") != nullptr, "get_callees should return valid JSON"); @@ -182,10 +182,10 @@ int main() } // ── Check 6: find_callers on a known function ── - char *callers = engine_get_callers(pid, "buildCallEdgesSQL", nullptr); + char *callers = + engine_get_callers(g_engine, pid, "buildCallEdgesSQL", nullptr); if (callers) { - printf("--- Callers of buildCallEdgesSQL ---\n%s\n", - callers); + printf("--- Callers of buildCallEdgesSQL ---\n%s\n", callers); check(strstr(callers, "callers") != nullptr, "get_callers should return valid JSON"); engine_free_string(callers); @@ -195,15 +195,13 @@ int main() printf("\n=== Self-inspect summary ===\n"); printf("Total nodes: %lld\n", (long long)total_nodes); printf("Functions: %lld\n", (long long)func_count); - printf("Call graph edges: %lld (P1 fix)\n", - (long long)call_edges); - printf("Documents: %lld (verify fix)\n", - (long long)doc_count); - printf("Orphan functions: %lld\n", - (long long)orphan_count); + printf("Call graph edges: %lld (P1 fix)\n", (long long)call_edges); + printf("Documents: %lld (verify fix)\n", (long long)doc_count); + printf("Orphan functions: %lld\n", (long long)orphan_count); sqlite3_close(db); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; printf("\n=== Self-inspect test passed ===\n"); return 0; diff --git a/engine/tests/test_semantic_fact_extractor.cpp b/engine/tests/test_semantic_fact_extractor.cpp index ec02179..ed81eaf 100644 --- a/engine/tests/test_semantic_fact_extractor.cpp +++ b/engine/tests/test_semantic_fact_extractor.cpp @@ -10,11 +10,12 @@ // 4. pattern/todo — function with a TODO comment // 5. framework/gin — import of gin-gonic/gin // 6. ffi/extern_call — function with extern "C" qualified_name +// 7. ffi/extern_call — cross-language Calls edge (cpp → rust) // // Test flow: // 1. Open a temp GraphStore at /tmp/test_semantic_fact.db // 2. createSchema + createProject -// 3. Insert graph_nodes (functions) for each test case +// 3. Insert entity (functions) for each test case // 4. Insert semantic_records rows simulating each pattern // 5. Insert one import row for the framework test // 6. Run SemanticFactExtractor::extractAll() inside a transaction @@ -24,7 +25,7 @@ #include "../src/model/semantic_fact_extractor.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -35,28 +36,51 @@ using namespace store; static const char *kDbPath = "/tmp/test_semantic_fact.db"; -/// Insert a graph_node function row with the given id, name, file_path, +/// Insert a Function entity (kind=0) with the given id, name, file_path and /// language. start_row/end_row are wide enough (1..1000) to enclose any /// semantic_records inserted below. -static void insertFunction(GraphStore &store, uint64_t project_id, - int64_t id, const char *name, - const char *file_path, const char *language) +/// +/// `entity` is the canonical table: the production indexing path +/// (engine_index_project → buildGraph) writes `entity` and never touches the +/// deprecated `graph_nodes`, so the extractor must resolve the enclosing +/// function from here. Seeding `graph_nodes` instead is what let the extractor +/// return zero facts in production while this test stayed green. +static void insertFunction(GraphStore &store, uint64_t project_id, int64_t id, + const char *name, const char *file_path, + const char *language) { sqlite3 *db = store.handle(); - const char *sql = - "INSERT INTO graph_nodes (id, project_id, ir_node_id, " - "node_type, name, qualified_name, file_path, language, " - "start_row, start_col, end_row, end_col) " - "VALUES (?,?,0,0,?,'',?,?,1,0,1000,0)"; + const char *sql = "INSERT INTO entity (id, project_id, kind, name, " + "qualified_name, file_path, language, " + "start_row, start_col, end_row, end_col) " + "VALUES (?,?,0,?,'',?,?,1,0,1000,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, file_path, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 5, language, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); + sqlite3_finalize(stmt); +} + +/// Insert a Calls relation (type=1) between two entities. Used by the +/// cross-language FFI case, which finds callees whose caller's language +/// differs from their own. +static void insertCallRelation(GraphStore &store, uint64_t project_id, + int64_t source_id, int64_t target_id) +{ + sqlite3 *db = store.handle(); + const char *sql = "INSERT OR IGNORE INTO relation " + "(project_id, source_id, target_id, type) " + "VALUES (?,?,?,1)"; + sqlite3_stmt *stmt = nullptr; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + sqlite3_bind_int64(stmt, 2, source_id); + sqlite3_bind_int64(stmt, 3, target_id); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -65,8 +89,8 @@ static void insertFunction(GraphStore &store, uint64_t project_id, /// 1..1000 range so the enclosing-function JOIN matches. static void insertCallRecord(GraphStore &store, uint64_t project_id, const char *name, const char *qualified_name, - const char *file_path, - const char *language, int start_row) + const char *file_path, const char *language, + int start_row) { sqlite3 *db = store.handle(); const char *sql = "INSERT INTO semantic_records " @@ -74,8 +98,7 @@ static void insertCallRecord(GraphStore &store, uint64_t project_id, " qualified_name, file_path, language, start_row) " "VALUES (?,?,9,?,?,?,?,?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, 1); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); @@ -83,7 +106,7 @@ static void insertCallRecord(GraphStore &store, uint64_t project_id, sqlite3_bind_text(stmt, 5, file_path, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 6, language, -1, SQLITE_TRANSIENT); sqlite3_bind_int(stmt, 7, start_row); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -98,15 +121,14 @@ static void insertCommentRecord(GraphStore &store, uint64_t project_id, " qualified_name, file_path, language, start_row) " "VALUES (?,?,14,?,'',?,?,?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, 1); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, text, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, file_path, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 5, language, -1, SQLITE_TRANSIENT); sqlite3_bind_int(stmt, 6, start_row); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -122,32 +144,29 @@ static void insertExceptRecord(GraphStore &store, uint64_t project_id, " qualified_name, file_path, language, start_row) " "VALUES (?,?,9,'except','',?,'python',?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, 2); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, file_path, -1, SQLITE_TRANSIENT); sqlite3_bind_int(stmt, 4, start_row); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } /// Insert an import row mapping target_path → file_path. static void insertImportRow(GraphStore &store, uint64_t project_id, - const char *target_path, - const char *file_path) + const char *target_path, const char *file_path) { sqlite3 *db = store.handle(); const char *sql = "INSERT INTO import (project_id, " "source_scope_id, target_path, alias, file_path) " "VALUES (?,0,?,'',?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, target_path, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 3, file_path, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -162,8 +181,7 @@ static int countFacts(GraphStore &store, uint64_t project_id, "WHERE project_id=? AND category=? " " AND primitive=? AND kind=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, category, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 3, primitive, -1, SQLITE_TRANSIENT); @@ -182,8 +200,7 @@ static int totalFacts(GraphStore &store, uint64_t project_id) const char *sql = "SELECT COUNT(*) FROM semantic_fact WHERE project_id=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); int count = 0; if (sqlite3_step(stmt) == SQLITE_ROW) @@ -202,9 +219,8 @@ int main() store.error().c_str()); return 1; } - uint64_t pid = - store.createProject("/tmp", "test_semantic_fact"); - assert(pid > 0); + uint64_t pid = store.createProject("/tmp", "test_semantic_fact"); + CHECK(pid > 0); // ── Test 1: sync/mutex/lock ──────────────────────────────── // Go function containing a m.Lock() call. The extractor should @@ -212,12 +228,19 @@ int main() insertFunction(store, pid, 10, "Acquire", "/src/sync.go", "go"); insertCallRecord(store, pid, "m.Lock", "sync.Mutex.Lock", "/src/sync.go", "go", 5); + // Regression: the RWMutex forms. The previous inline guard required a '.' + // immediately before "Lock", so `rw.RLock()` never entered the lock branch: + // it produced no fact at all and the rwmutex_usage rule stayed empty. The + // unlock half was not even selected by the query. + insertCallRecord(store, pid, "rw.RLock", "sync.RWMutex.RLock", + "/src/sync.go", "go", 6); + insertCallRecord(store, pid, "rw.RUnlock", "sync.RWMutex.RUnlock", + "/src/sync.go", "go", 7); // ── Test 2: error/bare_except ────────────────────────────── // Python function with `except:` (bare). The Python visitor // emits a record with name='except', qualified_name=''. - insertFunction(store, pid, 20, "RiskY", "/src/risky.py", - "python"); + insertFunction(store, pid, 20, "RiskY", "/src/risky.py", "python"); insertExceptRecord(store, pid, "/src/risky.py", 7); // ── Test 3: memory/cstring/alloc ─────────────────────────── @@ -225,93 +248,103 @@ int main() // function — the extractor still emits the alloc fact; matching // alloc/free pairs is a Phase 4 verifier concern. insertFunction(store, pid, 30, "ToString", "/src/cgo.go", "go"); - insertCallRecord(store, pid, "C.CString", "C.CString", - "/src/cgo.go", "go", 9); + insertCallRecord(store, pid, "C.CString", "C.CString", "/src/cgo.go", + "go", 9); // ── Test 4: pattern/todo ─────────────────────────────────── // A Comment record (kind=14) with text containing TODO. The // extractor should emit pattern/todo. insertFunction(store, pid, 40, "Stub", "/src/stub.go", "go"); - insertCommentRecord(store, pid, "TODO: implement this", - "/src/stub.go", "go", 11); + insertCommentRecord(store, pid, "TODO: implement this", "/src/stub.go", + "go", 11); // ── Test 5: framework/gin ────────────────────────────────── // An import of gin-gonic/gin. The extractor attaches the fact // to one function in the same file as the import. insertFunction(store, pid, 50, "Handler", "/src/main.go", "go"); - insertImportRow(store, pid, - "github.com/gin-gonic/gin", "/src/main.go"); + insertImportRow(store, pid, "github.com/gin-gonic/gin", "/src/main.go"); // ── Test 6: ffi/extern_call ──────────────────────────────── // A C++ function whose qualified_name contains 'extern "C"'. - insertFunction(store, pid, 60, "CBindings", - "/src/bindings.cpp", "cpp"); + insertFunction(store, pid, 60, "CBindings", "/src/bindings.cpp", "cpp"); insertCallRecord(store, pid, "register_callback", - "extern \"C\" register_callback", - "/src/bindings.cpp", "cpp", 15); + "extern \"C\" register_callback", "/src/bindings.cpp", + "cpp", 15); + + // ── Test 7: ffi/extern_call via a cross-language Calls edge ── + // A Rust callee called from C++ code. No `extern "C"` text is + // involved: this exercises the second FFI query, which traverses + // Calls edges (relation type=1) between differing languages. + insertFunction(store, pid, 70, "rust_entry", "/src/lib.rs", "rust"); + insertFunction(store, pid, 71, "call_rust", "/src/bridge.cpp", "cpp"); + insertCallRelation(store, pid, 71, 70); // ── Run the extractor inside a transaction ───────────────── { - assert(store.beginTransaction()); + CHECK(store.beginTransaction()); SemanticFactExtractor ex(&store); int64_t n = ex.extractAll(pid); - assert(store.commitTransaction()); - assert(n >= 6); // at least one fact per test case above - printf("extractAll returned %lld facts\n", - (long long)n); + CHECK(store.commitTransaction()); + CHECK(n >= 6); // at least one fact per test case above + printf("extractAll returned %lld facts\n", (long long)n); } // ── Assertions ───────────────────────────────────────────── int total = totalFacts(store, pid); - assert(total >= 6); + CHECK(total >= 6); printf("total semantic_fact rows: %d\n", total); // Test 1: sync/mutex/lock - assert(countFacts(store, pid, "sync", "mutex", "lock") == 1); + CHECK(countFacts(store, pid, "sync", "mutex", "lock") == 1); printf("Test 1 (sync/mutex/lock): PASS\n"); + // Test 1b: the RWMutex forms must be classified, not dropped. Before the + // fix these produced no fact at all (the lock) or were never selected (the + // unlock), which left the rwmutex_usage rule permanently empty. + CHECK(countFacts(store, pid, "sync", "rwmutex", "lock") == 1); + CHECK(countFacts(store, pid, "sync", "rwmutex", "defer_unlock") == 1); + printf("Test 1b (sync/rwmutex lock + unlock): PASS\n"); + // Test 2: error/bare_except - assert(countFacts(store, pid, "error", "bare_except", - "suppression") == 1); + CHECK(countFacts(store, pid, "error", "bare_except", "suppression") == + 1); printf("Test 2 (error/bare_except): PASS\n"); // Test 3: memory/cstring/alloc - assert(countFacts(store, pid, "memory", "cstring", - "alloc") == 1); + CHECK(countFacts(store, pid, "memory", "cstring", "alloc") == 1); printf("Test 3 (memory/cstring/alloc): PASS\n"); // Test 4: pattern/todo - assert(countFacts(store, pid, "pattern", "todo", - "marker") == 1); + CHECK(countFacts(store, pid, "pattern", "todo", "marker") == 1); printf("Test 4 (pattern/todo): PASS\n"); // Test 5: framework/gin - assert(countFacts(store, pid, "framework", "gin", - "router") == 1); + CHECK(countFacts(store, pid, "framework", "gin", "router") == 1); printf("Test 5 (framework/gin): PASS\n"); - // Test 6: ffi/extern_call - assert(countFacts(store, pid, "ffi", "extern_call", - "call") == 1); - printf("Test 6 (ffi/extern_call): PASS\n"); + // Test 6: ffi/extern_call — one from the extern "C" qualified + // name (CBindings) and one from the cross-language Calls edge + // (rust_entry), so exactly two. + CHECK(countFacts(store, pid, "ffi", "extern_call", "call") == 2); + printf("Test 6+7 (ffi/extern_call, incl. cross-language): " + "PASS\n"); // ── Idempotency: re-running extractAll clears then re-inserts ── // The clear-before-extract contract means the total fact count // stays the same after a second run (no duplicates accumulate). { int before = totalFacts(store, pid); - assert(store.beginTransaction()); + CHECK(store.beginTransaction()); SemanticFactExtractor ex(&store); ex.extractAll(pid); - assert(store.commitTransaction()); + CHECK(store.commitTransaction()); int after = totalFacts(store, pid); - assert(after == before); - printf("Test 7 (idempotency: %d == %d): PASS\n", before, - after); + CHECK(after == before); + printf("Test 7 (idempotency: %d == %d): PASS\n", before, after); } store.close(); unlink(kDbPath); printf("\nAll semantic_fact_extractor tests passed.\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_semantic_fact_production.cpp b/engine/tests/test_semantic_fact_production.cpp new file mode 100644 index 0000000..15d8a8d --- /dev/null +++ b/engine/tests/test_semantic_fact_production.cpp @@ -0,0 +1,177 @@ +// test_semantic_fact_production.cpp +// +// Regression test for the defect that made the v0.3 evidence pipeline return +// zero facts on the production path: SemanticFactExtractor resolved the +// enclosing function through the deprecated `graph_nodes` table, which +// engine_index_project → buildGraph never populates (its only writer, +// `engine_index_batch`, is not bound by the server). The companion test +// test_semantic_fact_extractor seeds `graph_nodes` by hand, so it could not +// observe the breakage — which is precisely why it went unnoticed. +// +// This test therefore indexes a real source directory through the real engine +// entry point, then runs the extractor against the resulting database and +// requires facts to appear *while `graph_nodes` is still empty*. That is the +// production condition, and asserting it (rather than assuming it) is what +// makes the regression detectable. +// +// Fixture (written at runtime, no repository files): +// /tmp/codescope_prod_fact_src/sample.cpp — a function whose body carries a +// TODO comment. +// Expected: at least one pattern/todo/marker fact, with entity rows > 0 and +// graph_nodes == 0. + +#include "../include/engine.h" + +#include "../src/model/semantic_fact_extractor.h" +#include "../src/store/store.h" + +#include "test_check.h" +#include +#include +#include +#include +#include +#include +#include +#include "test_engine_handle.h" + +namespace fs = std::filesystem; + +static const char *kDbPath = "/tmp/codescope_test_fact_production.db"; +static const char *kSrcDir = "/tmp/codescope_prod_fact_src"; + +/// Run a single-parameter COUNT query against `db_path` and return the value, +/// or -1 on failure. Kept deliberately dumb so a schema problem surfaces as a +/// failing assertion instead of an empty result being read as "0 facts". +static int64_t scalarCount(const char *db_path, const char *sql, + uint64_t project_id) +{ + sqlite3 *db = nullptr; + if (sqlite3_open(db_path, &db) != SQLITE_OK) { + fprintf(stderr, "FAIL: cannot open %s: %s\n", db_path, + sqlite3_errmsg(db)); + sqlite3_close(db); + return -1; + } + sqlite3_stmt *stmt = nullptr; + int64_t value = -1; + if (sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + if (sqlite3_step(stmt) == SQLITE_ROW) + value = sqlite3_column_int64(stmt, 0); + sqlite3_finalize(stmt); + } else { + fprintf(stderr, "FAIL: prepare: %s\n", sqlite3_errmsg(db)); + } + sqlite3_close(db); + return value; +} + +int main() +{ + // ── Fixture ───────────────────────────────────────────────── + unlink(kDbPath); + unlink((std::string(kDbPath) + "-wal").c_str()); + unlink((std::string(kDbPath) + "-shm").c_str()); + fs::remove_all(kSrcDir); + fs::create_directories(kSrcDir); + const std::string source_path = std::string(kSrcDir) + "/sample.cpp"; + { + std::ofstream out(source_path); + out << "#include \n" + "std::mutex g_mu;\n" + "int compute(int x) {\n" + " // TODO: replace with real validation\n" + " g_mu.lock();\n" + " return x + 1;\n" + "}\n" + "void helper() { compute(1); }\n" + "int main() { helper(); return 0; }\n"; + } + CHECK(fs::exists(source_path)); + + // ── Index through the real production entry point ─────────── + g_engine = engine_create(kDbPath); + if (!g_engine) { + fprintf(stderr, "FAIL: engine_init(%s)\n", kDbPath); + return 1; + } + const uint64_t pid = + engine_create_project(g_engine, kSrcDir, "fact-production"); + if (pid == 0) { + fprintf(stderr, "FAIL: engine_create_project\n"); + engine_destroy(g_engine); + g_engine = nullptr; + return 1; + } + char *index_result = + engine_index_project(g_engine, pid, kSrcDir, nullptr); + if (index_result == nullptr) { + fprintf(stderr, "FAIL: engine_index_project returned null\n"); + engine_destroy(g_engine); + g_engine = nullptr; + return 1; + } + printf(" [debug] index = %s\n", index_result); + engine_free_string(index_result); + // Also joins the background knowledge builder, leaving the DB quiescent. + engine_destroy(g_engine); + g_engine = nullptr; + + // ── Assert the production condition this test exists to cover ── + const int64_t entities = scalarCount( + kDbPath, "SELECT COUNT(*) FROM entity WHERE project_id=?", pid); + const int64_t graph_nodes = scalarCount( + kDbPath, "SELECT COUNT(*) FROM graph_nodes WHERE project_id=?", + pid); + printf(" [debug] entity=%lld graph_nodes=%lld\n", (long long)entities, + (long long)graph_nodes); + CHECK(entities > 0 && + "the fixture must produce entity rows (indexing broken?)"); + CHECK(graph_nodes == 0 && + "the canonical indexing path must not populate graph_nodes; if " + "that changed, this test's premise needs revisiting"); + + // ── Run the extractor against the indexed database ────────── + { + store::GraphStore store; + if (!store.open(kDbPath)) { + fprintf(stderr, "FAIL: cannot reopen %s: %s\n", kDbPath, + store.error().c_str()); + return 1; + } + CHECK(store.beginTransaction()); + model::SemanticFactExtractor extractor(&store); + const int64_t extracted = extractor.extractAll(pid); + CHECK(store.commitTransaction()); + printf(" [debug] extractAll = %lld\n", (long long)extracted); + CHECK(extracted > 0 && + "extracting zero facts from a real index is the " + "regression this test guards (#1)"); + store.close(); + } + + // ── The fact must be the one the fixture planted ──────────── + const int64_t todo_facts = scalarCount( + kDbPath, + "SELECT COUNT(*) FROM semantic_fact WHERE project_id=? " + "AND category='pattern' AND primitive='todo' " + "AND kind='marker'", + pid); + printf(" [debug] pattern/todo/marker = %lld\n", (long long)todo_facts); + CHECK(todo_facts >= 1 && + "the TODO inside compute() must yield a pattern/todo fact " + "without any graph_nodes rows"); + + // ── Cleanup ──────────────────────────────────────────────── + unlink(kDbPath); + unlink((std::string(kDbPath) + "-wal").c_str()); + unlink((std::string(kDbPath) + "-shm").c_str()); + fs::remove_all(kSrcDir); + + printf("\n=== test_semantic_fact_production PASSED ===\n"); + printf("Evidence pipeline verified on the production path:\n"); + printf(" - facts are extracted with graph_nodes empty\n"); + printf(" - the planted TODO yields pattern/todo/marker\n"); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_semantic_unit.cpp b/engine/tests/test_semantic_unit.cpp index a031e49..ea4a406 100644 --- a/engine/tests/test_semantic_unit.cpp +++ b/engine/tests/test_semantic_unit.cpp @@ -19,28 +19,29 @@ static int tests_run = 0; static int tests_passed = 0; -#define CHECK(cond, msg) \ - do { \ - tests_run++; \ - if (!(cond)) { \ - fprintf(stderr, "FAIL [%d]: %s\n", tests_run, msg); \ - exit(1); \ - } \ - tests_passed++; \ +#define CHECK(cond, msg) \ + do { \ + tests_run++; \ + if (!(cond)) { \ + fprintf(stderr, "FAIL [%d]: %s\n", tests_run, msg); \ + exit(1); \ + } \ + tests_passed++; \ } while (0) -#define CHECK_EQ(a, b, msg) \ - do { \ - tests_run++; \ - if ((a) != (b)) { \ - fprintf(stderr, "FAIL [%d]: %s — expected %llu, " \ - "got %llu\n", \ - tests_run, msg, \ - static_cast(b), \ - static_cast(a)); \ - exit(1); \ - } \ - tests_passed++; \ +#define CHECK_EQ(a, b, msg) \ + do { \ + tests_run++; \ + if ((a) != (b)) { \ + fprintf(stderr, \ + "FAIL [%d]: %s — expected %llu, " \ + "got %llu\n", \ + tests_run, msg, \ + static_cast(b), \ + static_cast(a)); \ + exit(1); \ + } \ + tests_passed++; \ } while (0) // ── Test: Empty SemanticUnit ────────────────────────────────── @@ -62,12 +63,10 @@ static void test_add_records() unit.setLanguage("typescript"); ir::SourceRange loc = { 0, 0, 5, 0 }; - uint64_t id1 = unit.addRecord(ir::RecordKind::Function, - "foo", 0, loc); - uint64_t id2 = unit.addRecord(ir::RecordKind::Function, - "bar", 0, loc); - uint64_t id3 = unit.addRecord(ir::RecordKind::CallExpr, - "baz", id1, loc); + uint64_t id1 = unit.addRecord(ir::RecordKind::Function, "foo", 0, loc); + uint64_t id2 = unit.addRecord(ir::RecordKind::Function, "bar", 0, loc); + uint64_t id3 = + unit.addRecord(ir::RecordKind::CallExpr, "baz", id1, loc); CHECK_EQ(unit.size(), 3ULL, "3 records added"); CHECK_EQ(id1, 1ULL, "first record id == 1"); @@ -81,10 +80,8 @@ static void test_add_records() CHECK(rec3 != nullptr, "getRecord(id3) non-null"); CHECK_EQ(rec1->parent_id, 0ULL, "foo is top-level"); CHECK_EQ(rec3->parent_id, id1, "baz has foo as parent"); - CHECK(rec1->file_path == "/test/file.ts", - "file path propagated"); - CHECK(rec1->language == "typescript", - "language propagated"); + CHECK(rec1->file_path == "/test/file.ts", "file path propagated"); + CHECK(rec1->language == "typescript", "language propagated"); } // ── Test: Query methods ─────────────────────────────────────── @@ -97,13 +94,12 @@ static void test_queries() ir::SourceRange loc2 = { 3, 0, 5, 0 }; ir::SourceRange loc3 = { 6, 0, 6, 10 }; - uint64_t fn1 = unit.addRecord(ir::RecordKind::Function, - "compute", 0, loc1); - uint64_t fn2 = unit.addRecord(ir::RecordKind::Function, - "add", 0, loc2); + uint64_t fn1 = + unit.addRecord(ir::RecordKind::Function, "compute", 0, loc1); + uint64_t fn2 = unit.addRecord(ir::RecordKind::Function, "add", 0, loc2); (void)fn2; - uint64_t call = unit.addRecord(ir::RecordKind::CallExpr, - "add", fn1, loc3); + uint64_t call = + unit.addRecord(ir::RecordKind::CallExpr, "add", fn1, loc3); (void)call; // findByName @@ -124,8 +120,8 @@ static void test_queries() // getChildren auto children = unit.getChildren(fn1); CHECK_EQ(children.size(), 1ULL, "compute has 1 child"); - CHECK_EQ(unit.allRecords()[children[0]].kind, - ir::RecordKind::CallExpr, "compute's child is a call"); + CHECK_EQ(unit.allRecords()[children[0]].kind, ir::RecordKind::CallExpr, + "compute's child is a call"); } // ── Test: Emitter interface ─────────────────────────────────── @@ -151,14 +147,10 @@ static void test_emitter() // Verify containment auto children = unit.getChildren(fn); CHECK_EQ(children.size(), 2ULL, "main has 2 children"); - CHECK(unit.getRecord(call) != nullptr, - "getRecord(call) non-null"); - CHECK(unit.getRecord(var) != nullptr, - "getRecord(var) non-null"); - CHECK_EQ(unit.getRecord(call)->parent_id, fn, - "call parent is main"); - CHECK_EQ(unit.getRecord(var)->parent_id, fn, - "variable parent is main"); + CHECK(unit.getRecord(call) != nullptr, "getRecord(call) non-null"); + CHECK(unit.getRecord(var) != nullptr, "getRecord(var) non-null"); + CHECK_EQ(unit.getRecord(call)->parent_id, fn, "call parent is main"); + CHECK_EQ(unit.getRecord(var)->parent_id, fn, "variable parent is main"); // Verify location preservation CHECK_EQ(unit.getRecord(call)->loc.start_row, 5U, @@ -210,7 +202,7 @@ int main() test_emitter(); test_memory_size(); - printf("=== semantic_unit test passed (%d/%d)\n", - tests_passed, tests_run); + printf("=== semantic_unit test passed (%d/%d)\n", tests_passed, + tests_run); return 0; } diff --git a/engine/tests/test_state_builder_batch.cpp b/engine/tests/test_state_builder_batch.cpp index 0d3daf2..a05c03b 100644 --- a/engine/tests/test_state_builder_batch.cpp +++ b/engine/tests/test_state_builder_batch.cpp @@ -18,7 +18,7 @@ #include "../src/model/state_builder.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -49,14 +49,14 @@ static void insertEntity(store::GraphStore &store, uint64_t project_id, "start_col, end_row, end_col, module_path, visibility) " "VALUES (?,?,0,?,'',?,'cpp',0,0,0,0,?,?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, file_path, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 5, module_path, -1, SQLITE_TRANSIENT); sqlite3_bind_int(stmt, 6, visibility); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -68,10 +68,10 @@ static void insertModuleScope(store::GraphStore &store, uint64_t project_id, const char *sql = "INSERT INTO scope (project_id, parent_id, kind, " "name, start_row, end_row) VALUES (?,0,1,?,0,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, name, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -83,11 +83,11 @@ static void insertCallRelation(store::GraphStore &store, uint64_t project_id, const char *sql = "INSERT INTO relation (project_id, source_id, " "target_id, type) VALUES (?,?,?,1)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, source_id); sqlite3_bind_int64(stmt, 3, target_id); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -98,7 +98,7 @@ static int countModuleSummaries(store::GraphStore &store, uint64_t project_id) const char *sql = "SELECT COUNT(*) FROM module_summary WHERE project_id=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); int count = 0; if (sqlite3_step(stmt) == SQLITE_ROW) @@ -115,7 +115,7 @@ static int64_t getModuleScopeId(store::GraphStore &store, const char *sql = "SELECT id FROM scope WHERE project_id=? AND kind=1 AND name=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_text(stmt, 2, name, -1, SQLITE_TRANSIENT); int64_t id = 0; @@ -142,7 +142,7 @@ getModuleSummary(store::GraphStore &store, uint64_t project_id, "SELECT incoming_count, outgoing_count, dead_entities, role " "FROM module_summary WHERE project_id=? AND module_id=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, module_id); ModuleSummaryRow row; @@ -171,7 +171,7 @@ int main() return 1; } uint64_t pid = store.createProject(tmpDir(), "test_state_builder"); - assert(pid > 0); + CHECK(pid > 0); // Module /src/api/ with 6 entities (meets HAVING total >= 3). // All entities set visibility=1 so pub_count > 0 for the @@ -217,7 +217,7 @@ int main() { StateBuilder sb(&store, pid); int64_t total = sb.buildAll(); - assert(total > 0); + CHECK(total > 0); printf("Test 1 (buildAll returned %lld items): PASS\n", (long long)total); } @@ -225,24 +225,24 @@ int main() // ── Test 2: module_summary has one row per qualifying module ─ { int count = countModuleSummaries(store, pid); - assert(count == 2); + CHECK(count == 2); printf("Test 2 (module_summary rows = %d): PASS\n", count); } // ── Test 3: api module content is correct ─────────────────── { int64_t api_scope_id = getModuleScopeId(store, pid, "/src/api/"); - assert(api_scope_id > 0); + CHECK(api_scope_id > 0); auto row = getModuleSummary(store, pid, api_scope_id); - assert(row.found); + CHECK(row.found); // 4 sources (1→2, 3→2, 4→5, 6→5), 2 targets (2,5) - assert(row.incoming == 4); - assert(row.outgoing == 2); + CHECK(row.incoming == 4); + CHECK(row.outgoing == 2); // 6 entities, only entities 2 and 5 are targets -> dead = 4. - assert(row.dead == 4); + CHECK(row.dead == 4); // Multi-signal classifier: pub_count=6, incoming=4 >= 2*outgoing=4, // incoming>=3, utilization=0.33>=0.3 -> role = 'api'. - assert(row.role == "api"); + CHECK(row.role == "api"); printf("Test 3 (api module incoming=%d outgoing=%d dead=%d " "role=%s): PASS\n", row.incoming, row.outgoing, row.dead, row.role.c_str()); @@ -254,16 +254,16 @@ int main() // Was 'infra' under v0.2.2 because utility required utilization>=0.5. { int64_t lib_scope_id = getModuleScopeId(store, pid, "/src/lib/"); - assert(lib_scope_id > 0); + CHECK(lib_scope_id > 0); auto row = getModuleSummary(store, pid, lib_scope_id); - assert(row.found); + CHECK(row.found); // Trim(7) -> Split(8): 1 source, 1 target - assert(row.incoming == 1); - assert(row.outgoing == 1); + CHECK(row.incoming == 1); + CHECK(row.outgoing == 1); // 3 entities, only entity 8 is a target -> dead = 2. - assert(row.dead == 2); + CHECK(row.dead == 2); // v0.2.1: utility rule fires (outgoing<=5, pub>0, util>=0.05). - assert(row.role == "utility"); + CHECK(row.role == "utility"); printf("Test 4 (lib module incoming=%d outgoing=%d dead=%d " "role=%s): PASS\n", row.incoming, row.outgoing, row.dead, row.role.c_str()); @@ -274,14 +274,47 @@ int main() // transaction. If it had rolled back, module_summary would be empty. { store.close(); - assert(store.open(kDbPath.c_str())); + CHECK(store.open(kDbPath.c_str())); int count = countModuleSummaries(store, pid); - assert(count == 2); + CHECK(count == 2); printf("Test 5 (rows persist after close/reopen: %d): PASS\n", count); } + // ── Test 6: a second buildAll must REPLACE, not append ── + // Regression: module_summary has no UNIQUE(project_id, + // module_id), so INSERT OR REPLACE never conflicted and + // every model-build pass appended another full set of rows. + // SUM(dead_entities) then grew past the real entity count + // (T5 finding #4: dead_code.entities > dead_code.total). + { + StateBuilder sb(&store, pid); + int64_t total = sb.buildAll(); + CHECK(total > 0); + int count = countModuleSummaries(store, pid); + CHECK(count == 2); + int64_t sum_dead = 0; + { + sqlite3 *db = store.handle(); + sqlite3_stmt *stmt = nullptr; + const char *sql = + "SELECT COALESCE(SUM(dead_entities),0) " + "FROM module_summary WHERE project_id=?"; + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == + SQLITE_OK); + sqlite3_bind_int64(stmt, 1, (sqlite3_int64)pid); + if (sqlite3_step(stmt) == SQLITE_ROW) + sum_dead = sqlite3_column_int64(stmt, 0); + sqlite3_finalize(stmt); + } + // api(4) + lib(2) = 6 — unchanged by the rebuild. + CHECK(sum_dead == 6); + printf("Test 6 (second buildAll replaces rows: " + "count=%d sum_dead=%lld): PASS\n", + count, (long long)sum_dead); + } + unlink(kDbPath.c_str()); printf("\nAll state builder batch tests passed.\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_step11_go_smoke.cpp b/engine/tests/test_step11_go_smoke.cpp index 8618999..1850328 100644 --- a/engine/tests/test_step11_go_smoke.cpp +++ b/engine/tests/test_step11_go_smoke.cpp @@ -45,6 +45,7 @@ #include #include #include +#include "test_engine_handle.h" // Tiny test helper: abort with a labeled message. Centralized so every // failure prints the layer that broke, which is the whole point of a @@ -78,10 +79,9 @@ static int64_t scalarInt(sqlite3 *db, const std::string &sql) // are kind 0 (free function) or 1 (method). Returns 0 if not found. static int64_t findEntityId(sqlite3 *db, uint64_t pid, const char *name) { - std::string sql = - "SELECT id FROM entity WHERE project_id=" + - std::to_string(pid) + " AND name='" + name + - "' AND kind IN (0,1) LIMIT 1"; + std::string sql = "SELECT id FROM entity WHERE project_id=" + + std::to_string(pid) + " AND name='" + name + + "' AND kind IN (0,1) LIMIT 1"; return scalarInt(db, sql); } @@ -127,14 +127,16 @@ int main() char db_path[] = "/tmp/test_step11_go_smoke.db"; unlink(db_path); - if (engine_init(db_path) != 0) + g_engine = engine_create(db_path); + if (!g_engine) fail("engine", "engine_init"); - uint64_t pid = engine_create_project(proj_dir, "step11-go-smoke"); + uint64_t pid = + engine_create_project(g_engine, proj_dir, "step11-go-smoke"); if (pid == 0) fail("engine", "engine_create_project"); - char *idx = engine_index_project(pid, proj_dir, nullptr); + char *idx = engine_index_project(g_engine, pid, proj_dir, nullptr); if (!idx || !strstr(idx, "\"ok\":true")) { fail("engine", "index_project did not return ok"); } @@ -165,12 +167,12 @@ int main() // engine_find_symbol must locate the callee, and engine_search_code // must hit the call text. If search misses, the discovery layer is // broken and no downstream query can recover. - char *sym = engine_find_symbol(pid, "multiply"); + char *sym = engine_find_symbol(g_engine, pid, "multiply"); if (!sym || !strstr(sym, "multiply")) fail("L1/search", "engine_find_symbol did not return multiply"); engine_free_string(sym); - char *code = engine_search_code(pid, "multiply", 10); + char *code = engine_search_code(g_engine, pid, "multiply", 10); // FTS (code search) depends on the async FTS build which may not // have completed yet. This is a secondary check — the critical // layers are L2-L5 below (reference, relation, SQLite, API). @@ -189,8 +191,9 @@ int main() // callee bare name. std::string ref_sql = "SELECT COUNT(*) FROM reference WHERE project_id=" + - std::to_string(pid) + " AND caller_id=" + - std::to_string(compute_id) + " AND name='multiply'"; + std::to_string(pid) + + " AND caller_id=" + std::to_string(compute_id) + + " AND name='multiply'"; int64_t ref_count = scalarInt(db, ref_sql); if (ref_count <= 0) fail("L2/reference", @@ -202,9 +205,9 @@ int main() // compute to multiply. This is the canonical call-graph edge. std::string rel_sql = "SELECT COUNT(*) FROM relation WHERE project_id=" + - std::to_string(pid) + " AND type=1 AND source_id=" + - std::to_string(compute_id) + " AND target_id=" + - std::to_string(multiply_id); + std::to_string(pid) + + " AND type=1 AND source_id=" + std::to_string(compute_id) + + " AND target_id=" + std::to_string(multiply_id); int64_t rel_count = scalarInt(db, rel_sql); if (rel_count <= 0) fail("L3/relation", @@ -216,7 +219,7 @@ int main() // explicit edge_type=1 filter (Step 1). If the graph compiler failed // to compile the relation into SQLite, this returns empty even // though L3 passed — exactly the A13 "no fallback" gap. - char *callers = engine_get_callers(pid, "multiply", nullptr); + char *callers = engine_get_callers(g_engine, pid, "multiply", nullptr); if (!callers || !strstr(callers, "compute")) fail("L4", "engine_get_callers(multiply) did not return compute — " @@ -226,8 +229,8 @@ int main() // ── L5: adaptive API ───────────────────────────────────────── // engine_find_callers_adaptive is the MCP-facing entry point. It // must agree with the direct SQLite query. - char *adaptive = - engine_find_callers_adaptive(pid, "multiply", nullptr); + char *adaptive = engine_find_callers_adaptive(g_engine, pid, "multiply", + nullptr); if (!adaptive || !strstr(adaptive, "compute")) fail("L5/api", "engine_find_callers_adaptive(multiply) did not return " @@ -240,29 +243,28 @@ int main() // caller set is non-empty and contains multiply; contamination is // additionally guarded by the typed_relation_query counter-example // test (this smoke test focuses on the positive control). - char *add_callers = engine_get_callers(pid, "add", nullptr); + char *add_callers = engine_get_callers(g_engine, pid, "add", nullptr); if (!add_callers || !strstr(add_callers, "multiply")) - fail("L4", - "engine_get_callers(add) did not return multiply"); + fail("L4", "engine_get_callers(add) did not return multiply"); engine_free_string(add_callers); // ── Step 1 invariant: no duplicate typed relations ─────────── // The UNIQUE(project_id, source_id, target_id, type) index must // guarantee zero duplicate typed edges. - int64_t dup_count = scalarInt( - db, - "SELECT COUNT(*) FROM relation r1 WHERE EXISTS (" - " SELECT 1 FROM relation r2 WHERE" - " r2.project_id=r1.project_id AND" - " r2.source_id=r1.source_id AND" - " r2.target_id=r1.target_id AND" - " r2.type=r1.type AND r2.id +#include +#include +#include +#include + +#include + +static int tests_run = 0; + +#define CHECK(cond, msg) \ + do { \ + tests_run++; \ + if (!(cond)) { \ + fprintf(stderr, "FAIL [%d]: %s\n", tests_run, msg); \ + exit(1); \ + } \ + } while (0) + +/// The grammar, from the library this test links (grammar_loader.h). The old +/// dlopen() searched directories this repository never writes — and the .so +/// files it looked for exist only as untracked leftovers — so `grammar loaded` +/// failed on CI for every test that used this loader. +static const TSLanguage *loadLanguage(const char *grammar) +{ + return testGrammar(grammar); +} + +/// Deepest node level in `node`'s subtree, so the fixture can assert it really +/// nests deeper than the bound the test is about (a fixture that collapsed +/// would make the test pass for the wrong reason). +static int treeDepth(TSNode node) +{ + const uint32_t n = ts_node_child_count(node); + int deepest = 0; + for (uint32_t i = 0; i < n; ++i) { + const int d = treeDepth(ts_node_child(node, i)); + if (d > deepest) + deepest = d; + } + return deepest + 1; +} + +/// Parse `code` and return its CST root depth. +static int parseDepth(const char *grammar, const std::string &code) +{ + const TSLanguage *lang = loadLanguage(grammar); + CHECK(lang != nullptr, "grammar loaded for the depth probe"); + TSParser *parser = ts_parser_new(); + ts_parser_set_language(parser, lang); + TSTree *tree = + ts_parser_parse_string(parser, nullptr, code.c_str(), + static_cast(code.size())); + ts_parser_delete(parser); + CHECK(tree != nullptr, "probe parse succeeded"); + const int depth = treeDepth(ts_tree_root_node(tree)); + ts_tree_delete(tree); + return depth; +} + +/// Read a whole file, or "" when it cannot be opened. +static std::string readFile(const char *path) +{ + std::string out; + FILE *f = fopen(path, "r"); + if (!f) + return out; + char buf[4096]; + size_t n = 0; + while ((n = fread(buf, 1, sizeof(buf), f)) > 0) + out.append(buf, n); + fclose(f); + return out; +} + +/// Run the legacy translator for `grammar` over `code`, with stderr captured. +/// \param produced Set to whether translate() returned a unit. +/// \return Everything the translator wrote to stderr. +static std::string translateCapturing(const char *grammar, + const std::string &code, + const char *file_path, bool *produced) +{ + const TSLanguage *lang = loadLanguage(grammar); + CHECK(lang != nullptr, "grammar loaded"); + + TSParser *parser = ts_parser_new(); + ts_parser_set_language(parser, lang); + TSTree *tree = + ts_parser_parse_string(parser, nullptr, code.c_str(), + static_cast(code.size())); + ts_parser_delete(parser); + CHECK(tree != nullptr, "parse succeeded"); + + const char *log = "/tmp/test_translator_depth_guard.stderr"; + fflush(stderr); + const int saved = dup(fileno(stderr)); + CHECK(saved >= 0, "stderr duplicated"); + CHECK(freopen(log, "w", stderr) != nullptr, "stderr redirected"); + + auto translator = ir::createTranslator(grammar); + CHECK(translator != nullptr, "translator created"); + ir::TranslationUnit *unit = + translator->translate(tree, code.c_str(), file_path); + *produced = unit != nullptr; + delete unit; + + fflush(stderr); + dup2(saved, fileno(stderr)); + close(saved); + ts_tree_delete(tree); + + return readFile(log); +} + +/// `opening` × n + `body` + `closing` × n, so the CST nests n levels deep. +static std::string nest(const char *opening, const char *body, + const char *closing, int n) +{ + std::string code; + for (int i = 0; i < n; ++i) + code += opening; + code += body; + for (int i = 0; i < n; ++i) + code += closing; + return code; +} + +static int countOccurrences(const std::string &hay, const std::string &needle) +{ + int n = 0; + size_t pos = hay.find(needle); + while (pos != std::string::npos) { + ++n; + pos = hay.find(needle, pos + needle.size()); + } + return n; +} + +int main() +{ + const std::string truncated = "kMaxTranslateDepth"; + + // ── 1. Ordinary code is not truncated ─────────────────────────────── + { + bool produced = false; + const std::string err = translateCapturing( + "c", "int f(void) { return 1; }\n", + "/t/shallow.c", &produced); + CHECK(produced, "a normal file must still translate"); + CHECK(countOccurrences(err, truncated) == 0, + "the recursion bound must not fire on ordinary code"); + printf("Test 1 (normal file untruncated): PASS\n"); + } + + // ── 2. Pathological nesting is bounded and reported once ──────────── + { + // Nested BLOCKS, not nested parentheses: the translator only + // descends into the container types it knows (compound_statement, + // statements, declarations), so a paren chain never reaches the + // recursion this bound protects — and could not overflow it. + const std::string code = std::string("int f(void) { ") + + nest("{", "int x = 0;", "}", 900) + + " }\n"; + bool produced = false; + const std::string err = translateCapturing( + "c", code, "/t/deep.c", &produced); + CHECK(parseDepth("c", code) > + ir::kMaxTranslateDepth, + "fixture check: the file must nest deeper than the " + "bound, or this test would pass vacuously"); + CHECK(produced, + "a pathologically nested file must still translate (no " + "stack overflow)"); + CHECK(countOccurrences(err, truncated) == 1, + "the truncation must be reported exactly once per file"); + printf("Test 2 (deep C file bounded + reported once): PASS\n"); + } + + // ── 3. The bound is wired per translator ──────────────────────────── + { + // Python has no braces: nest `if` statements by indentation. + std::string code; + for (int i = 0; i < 900; ++i) + code += std::string(i, '\t') + "if True:\n"; + code += std::string(900, '\t') + "pass\n"; + bool produced = false; + const std::string err = + translateCapturing("python", code, "/t/deep.py", + &produced); + CHECK(produced, "the python translator must also complete"); + CHECK(countOccurrences(err, truncated) == 1, + "the python translator must apply the same bound (the guard " + "belongs to every translator, not just C)"); + printf("Test 3 (bound wired per translator): PASS\n"); + } + + printf("\n=== test_translator_depth_guard PASSED (%d assertions) ===\n", + tests_run); + return 0; +} diff --git a/engine/tests/test_trigram_search.cpp b/engine/tests/test_trigram_search.cpp index 4cda053..5648325 100644 --- a/engine/tests/test_trigram_search.cpp +++ b/engine/tests/test_trigram_search.cpp @@ -14,7 +14,7 @@ // SQLITE_INTERRUPT; clearing the deadline restores normal operation. #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -35,11 +35,11 @@ static void insertGraphNode(GraphStore &store, uint64_t project_id, int64_t id, "language, start_row, start_col, end_row, end_col) " "VALUES (?,?,0,0,?,'','/test.cpp','cpp',0,0,0,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); // Also insert into entity (graph_nodes is deprecated, buildFTSFromGraph // now reads from entity). @@ -49,7 +49,8 @@ static void insertGraphNode(GraphStore &store, uint64_t project_id, int64_t id, "language, start_row, start_col, end_row, end_col, module_path) " "VALUES (?,?,0,?,'','/test.cpp','cpp',0,0,0,0,'')"; sqlite3_stmt *estmt = nullptr; - if (sqlite3_prepare_v2(db, entity_sql, -1, &estmt, nullptr) == SQLITE_OK) { + if (sqlite3_prepare_v2(db, entity_sql, -1, &estmt, nullptr) == + SQLITE_OK) { sqlite3_bind_int64(estmt, 1, id); sqlite3_bind_int64(estmt, 2, static_cast(project_id)); sqlite3_bind_text(estmt, 3, name, -1, SQLITE_TRANSIENT); @@ -82,10 +83,10 @@ int main() unlink(kDbPath); GraphStore store; - assert(store.open(kDbPath)); + CHECK(store.open(kDbPath)); uint64_t project_id = store.createProject("/test", "test_trigram"); - assert(project_id > 0); + CHECK(project_id > 0); // ── 1. isTrigramAvailable before build ───────────────────────── // @@ -94,7 +95,7 @@ int main() // isTrigramAvailable() must return false. { bool avail = store.isTrigramAvailable(); - assert(!avail); + CHECK(!avail); printf(" [PASS] isTrigramAvailable: false before buildFTSFromGraph\n"); } @@ -112,7 +113,7 @@ int main() { store.buildFTSFromGraph(project_id); bool avail = store.isTrigramAvailable(); - assert(avail); + CHECK(avail); printf(" [PASS] isTrigramAvailable: true after buildFTSFromGraph\n"); } @@ -125,8 +126,8 @@ int main() { std::string json = store.searchGraphFallback(project_id, "gerFa", 20); - assert(jsonContains(json, "LoggerFactory")); - assert(!jsonContains(json, "UserFactory")); + CHECK(jsonContains(json, "LoggerFactory")); + CHECK(!jsonContains(json, "UserFactory")); printf(" [PASS] searchGraphFallback: 'gerFa' -> LoggerFactory\n"); } @@ -135,8 +136,8 @@ int main() { std::string json = store.searchGraphFallback(project_id, "ory", 20); - assert(jsonContains(json, "LoggerFactory")); - assert(jsonContains(json, "UserFactory")); + CHECK(jsonContains(json, "LoggerFactory")); + CHECK(jsonContains(json, "UserFactory")); printf(" [PASS] searchGraphFallback: 'ory' -> LoggerFactory + UserFactory\n"); } @@ -150,8 +151,8 @@ int main() { std::string json = store.searchGraphFallback(project_id, "ab", 20); - assert(jsonContains(json, "\"note\"")); - assert(jsonContains(json, "AbstractHandler")); + CHECK(jsonContains(json, "\"note\"")); + CHECK(jsonContains(json, "AbstractHandler")); printf(" [PASS] searchGraphFallback: 'ab' (short) -> LIKE fallback + note\n"); } @@ -164,8 +165,8 @@ int main() { std::string json = store.searchUnifiedJson(project_id, "Factory", 20); - assert(jsonContains(json, "LoggerFactory") || - jsonContains(json, "UserFactory")); + CHECK(jsonContains(json, "LoggerFactory") || + jsonContains(json, "UserFactory")); printf(" [PASS] searchUnifiedJson: 'Factory' -> results\n"); } @@ -184,7 +185,7 @@ int main() { // Bulk-insert 50000 nodes in a single transaction for speed. constexpr int kBulkNodeCount = 50000; - assert(store.exec("BEGIN")); + CHECK(store.exec("BEGIN")); sqlite3 *db = store.handle(); const char *sql = "INSERT INTO graph_nodes (id, project_id, ir_node_id, " @@ -192,8 +193,8 @@ int main() "language, start_row, start_col, end_row, end_col) " "VALUES (?,?,0,0,?,'','/bulk.cpp','cpp',0,0,0,0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == - SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == + SQLITE_OK); // Start IDs at 1000 to avoid collision with the 4 nodes above. const int64_t kIdOffset = 1000; char name_buf[64]; @@ -206,11 +207,11 @@ int main() sqlite3_bind_text(stmt, 3, name_buf, -1, SQLITE_TRANSIENT); int rc = sqlite3_step(stmt); - assert(rc == SQLITE_DONE); + CHECK(rc == SQLITE_DONE); sqlite3_reset(stmt); } sqlite3_finalize(stmt); - assert(store.exec("COMMIT")); + CHECK(store.exec("COMMIT")); // Arm a 1ms deadline, then sleep 2ms so the deadline is // definitely in the past when the query starts. The progress @@ -226,14 +227,14 @@ int main() "SELECT COUNT(*) FROM graph_nodes WHERE project_id=" + std::to_string(project_id) + " AND name LIKE '%zzzzz%'"; bool ok = store.exec(slow_sql.c_str()); - assert(!ok); // must fail (interrupted) - assert(containsCI(store.error(), "interrupt")); + CHECK(!ok); // must fail (interrupted) + CHECK(containsCI(store.error(), "interrupt")); printf(" [PASS] setQueryDeadline(1ms): query interrupted\n"); // Clear the deadline and verify the same query succeeds. store.clearQueryDeadline(); bool ok2 = store.exec(slow_sql.c_str()); - assert(ok2); // must succeed now + CHECK(ok2); // must succeed now printf(" [PASS] clearQueryDeadline: query succeeds again\n"); // Verify searchGraphFallback works normally after clearing @@ -244,8 +245,8 @@ int main() // into graph_nodes for the LIKE-scan timeout test above). std::string json = store.searchGraphFallback(project_id, "Factory", 10); - assert(jsonContains(json, "LoggerFactory") || - jsonContains(json, "UserFactory")); + CHECK(jsonContains(json, "LoggerFactory") || + jsonContains(json, "UserFactory")); printf(" [PASS] searchGraphFallback normal after deadline cleared\n"); } @@ -253,5 +254,5 @@ int main() unlink(kDbPath); printf("=== Trigram search test passed ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_ts_e2e.cpp b/engine/tests/test_ts_e2e.cpp index 37b30d3..3ea635d 100644 --- a/engine/tests/test_ts_e2e.cpp +++ b/engine/tests/test_ts_e2e.cpp @@ -1,7 +1,8 @@ #include "test_e2e.h" -int main() { - const char* code = R"(function add(a: number, b: number): number { +int main() +{ + const char *code = R"(function add(a: number, b: number): number { return a + b; } @@ -27,12 +28,8 @@ function main(): void { main(); )"; - const char* defs[] = {"add", "multiply", "compute"}; - runE2eTest("ts", code, "/tmp/test_math.ts", - defs, 3, - "add", "multiply", - "add", - "compute", - "main", nullptr); - return 0; + const char *defs[] = { "add", "multiply", "compute" }; + runE2eTest("ts", code, "/tmp/test_math.ts", defs, 3, "add", "multiply", + "add", "compute", "main", nullptr); + return 0; } diff --git a/engine/tests/test_ts_visitor.cpp b/engine/tests/test_ts_visitor.cpp index 28c6a52..aa94b6f 100644 --- a/engine/tests/test_ts_visitor.cpp +++ b/engine/tests/test_ts_visitor.cpp @@ -14,69 +14,51 @@ #include "../src/ir/translators/ts_visitor.h" #include "../src/ir/semantic_unit.h" +#include "grammar_loader.h" #include #include #include -#include #include #include // tree-sitter #include -// Load the TypeScript grammar dynamically +// The grammar is compiled into astgraph_engine, which this test links, so the +// language comes from the library rather than from a .so on disk +// (engine/tests/grammar_loader.h explains what the old lookup cost). static const TSLanguage *load_ts_language() { - const char *dirs[] = { - getenv("GRAMMARS_DIR"), - "../grammars", - "grammars", - "～/code/cppCode/CodeScope/grammars", - }; - for (auto d : dirs) { - if (!d) - continue; - std::string path = std::string(d) + - "/tree-sitter-typescript.so"; - void *handle = dlopen(path.c_str(), - RTLD_LAZY | RTLD_LOCAL); - if (handle) { - auto *fn = reinterpret_cast( - dlsym(handle, "tree_sitter_typescript")); - if (fn) - return fn(); - dlclose(handle); - } - } - return nullptr; + return testGrammar("typescript"); } static int tests_run = 0; static int tests_passed = 0; -#define CHECK(cond, msg) \ - do { \ - tests_run++; \ - if (!(cond)) { \ - fprintf(stderr, "FAIL [%d]: %s\n", tests_run, msg); \ - exit(1); \ - } \ - tests_passed++; \ +#define CHECK(cond, msg) \ + do { \ + tests_run++; \ + if (!(cond)) { \ + fprintf(stderr, "FAIL [%d]: %s\n", tests_run, msg); \ + exit(1); \ + } \ + tests_passed++; \ } while (0) -#define CHECK_EQ(a, b, msg) \ - do { \ - tests_run++; \ - if ((a) != (b)) { \ - fprintf(stderr, "FAIL [%d]: %s — expected %llu, " \ - "got %llu\n", \ - tests_run, msg, \ - static_cast(b), \ - static_cast(a)); \ - exit(1); \ - } \ - tests_passed++; \ +#define CHECK_EQ(a, b, msg) \ + do { \ + tests_run++; \ + if ((a) != (b)) { \ + fprintf(stderr, \ + "FAIL [%d]: %s — expected %llu, " \ + "got %llu\n", \ + tests_run, msg, \ + static_cast(b), \ + static_cast(a)); \ + exit(1); \ + } \ + tests_passed++; \ } while (0) // ── Helpers ─────────────────────────────────────────────────── @@ -86,8 +68,7 @@ static TSTree *parse(const char *source, const TSLanguage *lang) TSParser *parser = ts_parser_new(); ts_parser_set_language(parser, lang); TSTree *tree = ts_parser_parse_string( - parser, nullptr, source, - static_cast(strlen(source))); + parser, nullptr, source, static_cast(strlen(source))); ts_parser_delete(parser); return tree; } @@ -98,7 +79,7 @@ static size_t countKind(const ir::SemanticUnit &unit, ir::RecordKind kind) } static const ir::Record *findByName(const ir::SemanticUnit &unit, - const std::string &name) + const std::string &name) { size_t idx = unit.findRecordByName(name); if (idx == SIZE_MAX) @@ -122,8 +103,8 @@ static void test_interface_declaration() CHECK(tree != nullptr, "parse succeeded"); ir::TsVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/interface.ts"); + ir::SemanticUnit *unit = + visitor.visit(tree, code, "/test/interface.ts"); size_t ifaces = countKind(*unit, ir::RecordKind::Interface); CHECK_EQ(ifaces, 1ULL, "1 interface found"); @@ -133,10 +114,8 @@ static void test_interface_declaration() CHECK(user->kind == ir::RecordKind::Interface, "record kind is Interface"); CHECK_EQ(user->parent_id, 0ULL, "User is top-level"); - CHECK(user->file_path == "/test/interface.ts", - "file path preserved"); - CHECK(unit->language() == "typescript", - "language is typescript"); + CHECK(user->file_path == "/test/interface.ts", "file path preserved"); + CHECK(unit->language() == "typescript", "language is typescript"); ts_tree_delete(tree); delete unit; @@ -156,8 +135,7 @@ static void test_type_alias_declaration() CHECK(tree != nullptr, "parse succeeded"); ir::TsVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/alias.ts"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/alias.ts"); size_t aliases = countKind(*unit, ir::RecordKind::TypeAlias); CHECK_EQ(aliases, 1ULL, "1 type alias found"); @@ -189,16 +167,14 @@ static void test_enum_declaration() CHECK(tree != nullptr, "parse succeeded"); ir::TsVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/enum.ts"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/enum.ts"); size_t enums = countKind(*unit, ir::RecordKind::Enum); CHECK_EQ(enums, 1ULL, "1 enum found"); const ir::Record *color = findByName(*unit, "Color"); CHECK(color != nullptr, "enum 'Color' found"); - CHECK(color->kind == ir::RecordKind::Enum, - "record kind is Enum"); + CHECK(color->kind == ir::RecordKind::Enum, "record kind is Enum"); ts_tree_delete(tree); delete unit; @@ -220,8 +196,7 @@ static void test_ts_function() CHECK(tree != nullptr, "parse succeeded"); ir::TsVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/function.ts"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/function.ts"); size_t funcs = countKind(*unit, ir::RecordKind::Function); CHECK_EQ(funcs, 1ULL, "1 function found"); @@ -251,18 +226,15 @@ static void test_ts_class_method() CHECK(tree != nullptr, "parse succeeded"); ir::TsVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/class.ts"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/class.ts"); const ir::Record *calc = findByName(*unit, "Calculator"); CHECK(calc != nullptr, "class 'Calculator' found"); const ir::Record *add = findByName(*unit, "add"); CHECK(add != nullptr, "method 'add' found"); - CHECK(add->kind == ir::RecordKind::Method, - "add is a Method record"); - CHECK_EQ(add->parent_id, calc->id, - "add is child of Calculator"); + CHECK(add->kind == ir::RecordKind::Method, "add is a Method record"); + CHECK_EQ(add->parent_id, calc->id, "add is child of Calculator"); ts_tree_delete(tree); delete unit; @@ -284,8 +256,7 @@ static void test_ts_call_expression() CHECK(tree != nullptr, "parse succeeded"); ir::TsVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/call.ts"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/call.ts"); size_t calls = countKind(*unit, ir::RecordKind::CallExpr); CHECK_EQ(calls, 1ULL, "1 call expression found"); @@ -298,8 +269,7 @@ static void test_ts_call_expression() } } CHECK(call != nullptr, "call record exists"); - CHECK(call->name == "compute", - "call name is 'compute'"); + CHECK(call->name == "compute", "call name is 'compute'"); ts_tree_delete(tree); delete unit; @@ -320,8 +290,7 @@ static void test_ts_import_export() CHECK(tree != nullptr, "parse succeeded"); ir::TsVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/import.ts"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/import.ts"); size_t imports = countKind(*unit, ir::RecordKind::Import); CHECK_EQ(imports, 1ULL, "1 import found"); @@ -358,15 +327,13 @@ static void test_mixed_ts_constructs() CHECK(tree != nullptr, "parse succeeded"); ir::TsVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/mixed.ts"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/mixed.ts"); CHECK_EQ(countKind(*unit, ir::RecordKind::Interface), 1ULL, "1 interface"); CHECK_EQ(countKind(*unit, ir::RecordKind::TypeAlias), 1ULL, "1 type alias"); - CHECK_EQ(countKind(*unit, ir::RecordKind::Enum), 1ULL, - "1 enum"); + CHECK_EQ(countKind(*unit, ir::RecordKind::Enum), 1ULL, "1 enum"); CHECK_EQ(countKind(*unit, ir::RecordKind::Function), 1ULL, "1 function"); @@ -382,14 +349,14 @@ static void test_ts_arrow_function() const TSLanguage *lang = load_ts_language(); CHECK(lang != nullptr, "TypeScript grammar loaded"); - const char *code = "const add = (a: number, b: number): number => a + b;\n"; + const char *code = + "const add = (a: number, b: number): number => a + b;\n"; TSTree *tree = parse(code, lang); CHECK(tree != nullptr, "parse succeeded"); ir::TsVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/arrow.ts"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/arrow.ts"); size_t funcs = countKind(*unit, ir::RecordKind::Function); CHECK_EQ(funcs, 1ULL, "1 arrow function found"); @@ -417,8 +384,7 @@ static void test_ts_empty_file() CHECK(tree != nullptr, "parse succeeded"); ir::TsVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/empty.ts"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/empty.ts"); CHECK(unit != nullptr, "visit returned unit"); CHECK(!unit->empty(), "unit is not empty"); @@ -429,6 +395,82 @@ static void test_ts_empty_file() // ── Main ────────────────────────────────────────────────────── +// ── Test: type-annotation extraction ───────────────────────── + +/// First node of `type` in the tree, reported through `out`. +static bool findFirstOfType(TSNode node, const char *type, TSNode &out) +{ + if (strcmp(ts_node_type(node), type) == 0) { + out = node; + return true; + } + const uint32_t count = ts_node_child_count(node); + for (uint32_t i = 0; i < count; i++) { + if (findFirstOfType(ts_node_child(node, i), type, out)) + return true; + } + return false; +} + +/// Type extracted from the first annotation in `snippet`. +static std::string annotationType(const TSLanguage *lang, const char *snippet) +{ + TSTree *tree = parse(snippet, lang); + if (!tree) + return ""; + TSNode ann; + std::string got = ""; + if (findFirstOfType(ts_tree_root_node(tree), "type_annotation", ann)) { + // The extractor is protected; this probe exposes it without widening + // the production contract. + struct Probe : ir::TsVisitor { + using ir::TsVisitor::extractTsTypeAnnotation; + }; + Probe probe; + // nodeText() reads the visitor's source, which visit() sets: without + // it the extractor sees an empty source and every case returns "". + ir::SemanticUnit *unit = + probe.visit(tree, snippet, "/test/annotation.ts"); + got = probe.extractTsTypeAnnotation(ann); + delete unit; + } + printf(" [debug] %-48s -> '%s'\n", snippet, got.c_str()); + ts_tree_delete(tree); + return got; +} + +static void test_type_annotation_extraction() +{ + const TSLanguage *lang = load_ts_language(); + CHECK(lang != nullptr, "TypeScript grammar loaded"); + + // Plain, generic, array and union annotations keep their base type name. + CHECK(annotationType(lang, "const a: Foo = x;\n") == "Foo", + "plain class type"); + CHECK(annotationType(lang, "const b: Array = x;\n") == "Array", + "generic type keeps its base"); + CHECK(annotationType(lang, "const c: Foo[] = x;\n") == "Foo", + "array element type"); + CHECK(annotationType(lang, "const d: Foo | null = x;\n") == "Foo", + "union takes the first member"); + CHECK(annotationType(lang, "const e: Map = x;\n") == "Map", + "map generic keeps its base"); + CHECK(annotationType(lang, "const f: Foo.Bar = x;\n") == "Foo.Bar", + "qualified type name"); + + // Shapes the extractor does not model report UNKNOWN instead of a fragment + // of the annotation. An object type literal used to yield its first member + // ("id: Foo"), which no declaration can be keyed by: the receiver type could + // never match a candidate, and every tool printed it as if it were a name. + CHECK(annotationType(lang, "const g: { id: Foo; n: number }[] = x;\n") == + "", + "object type literal reports unknown"); + CHECK(annotationType(lang, "const h: (a: number) => void = f;\n") == "", + "function type reports unknown"); + + printf(" ✓ test_type_annotation_extraction\n"); +} + int main() { printf("TsVisitor tests:\n"); @@ -443,8 +485,9 @@ int main() test_mixed_ts_constructs(); test_ts_arrow_function(); test_ts_empty_file(); + test_type_annotation_extraction(); - printf("\n=== ts_visitor test passed (%d/%d) ===\n", - tests_passed, tests_run); + printf("\n=== ts_visitor test passed (%d/%d) ===\n", tests_passed, + tests_run); return 0; } diff --git a/engine/tests/test_tsx_visitor.cpp b/engine/tests/test_tsx_visitor.cpp index 5990d2e..b1a6532 100644 --- a/engine/tests/test_tsx_visitor.cpp +++ b/engine/tests/test_tsx_visitor.cpp @@ -12,83 +12,66 @@ #include "../src/ir/translators/tsx_visitor.h" #include "../src/ir/semantic_unit.h" +#include "grammar_loader.h" #include #include #include -#include #include #include // tree-sitter #include -// Load the TSX grammar dynamically +// The grammar is compiled into astgraph_engine, which this test links, so the +// language comes from the library rather than from a .so on disk +// (engine/tests/grammar_loader.h explains what the old lookup cost). static const TSLanguage *load_tsx_language() { - const char *dirs[] = { - getenv("GRAMMARS_DIR"), - "../grammars", - "grammars", - "～/code/cppCode/CodeScope/grammars", - }; - for (auto d : dirs) { - if (!d) - continue; - std::string path = std::string(d) + - "/tree-sitter-tsx.so"; - void *handle = dlopen(path.c_str(), - RTLD_LAZY | RTLD_LOCAL); - if (handle) { - auto *fn = reinterpret_cast( - dlsym(handle, "tree_sitter_tsx")); - if (fn) - return fn(); - dlclose(handle); - } - } - return nullptr; + return testGrammar("tsx"); } static int tests_run = 0; static int tests_passed = 0; -#define CHECK(cond, msg) \ - do { \ - tests_run++; \ - if (!(cond)) { \ - fprintf(stderr, "FAIL [%d]: %s\n", tests_run, msg); \ - exit(1); \ - } \ - tests_passed++; \ +#define CHECK(cond, msg) \ + do { \ + tests_run++; \ + if (!(cond)) { \ + fprintf(stderr, "FAIL [%d]: %s\n", tests_run, msg); \ + exit(1); \ + } \ + tests_passed++; \ } while (0) -#define CHECK_EQ(a, b, msg) \ - do { \ - tests_run++; \ - if ((a) != (b)) { \ - fprintf(stderr, "FAIL [%d]: %s — expected %llu, " \ - "got %llu\n", \ - tests_run, msg, \ - static_cast(b), \ - static_cast(a)); \ - exit(1); \ - } \ - tests_passed++; \ +#define CHECK_EQ(a, b, msg) \ + do { \ + tests_run++; \ + if ((a) != (b)) { \ + fprintf(stderr, \ + "FAIL [%d]: %s — expected %llu, " \ + "got %llu\n", \ + tests_run, msg, \ + static_cast(b), \ + static_cast(a)); \ + exit(1); \ + } \ + tests_passed++; \ } while (0) -#define CHECK_GE(a, b, msg) \ - do { \ - tests_run++; \ - if ((a) < (b)) { \ - fprintf(stderr, "FAIL [%d]: %s — expected >= %llu, " \ - "got %llu\n", \ - tests_run, msg, \ - static_cast(b), \ - static_cast(a)); \ - exit(1); \ - } \ - tests_passed++; \ +#define CHECK_GE(a, b, msg) \ + do { \ + tests_run++; \ + if ((a) < (b)) { \ + fprintf(stderr, \ + "FAIL [%d]: %s — expected >= %llu, " \ + "got %llu\n", \ + tests_run, msg, \ + static_cast(b), \ + static_cast(a)); \ + exit(1); \ + } \ + tests_passed++; \ } while (0) // ── Helpers ─────────────────────────────────────────────────── @@ -98,8 +81,7 @@ static TSTree *parse(const char *source, const TSLanguage *lang) TSParser *parser = ts_parser_new(); ts_parser_set_language(parser, lang); TSTree *tree = ts_parser_parse_string( - parser, nullptr, source, - static_cast(strlen(source))); + parser, nullptr, source, static_cast(strlen(source))); ts_parser_delete(parser); return tree; } @@ -110,7 +92,7 @@ static size_t countKind(const ir::SemanticUnit &unit, ir::RecordKind kind) } static const ir::Record *findByName(const ir::SemanticUnit &unit, - const std::string &name) + const std::string &name) { size_t idx = unit.findRecordByName(name); if (idx == SIZE_MAX) @@ -131,17 +113,14 @@ static void test_jsx_self_closing_skipped() CHECK(tree != nullptr, "parse succeeded"); ir::TsxVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/elem.tsx"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/elem.tsx"); // Should only have the variable record (el), no JSX noise const ir::Record *el = findByName(*unit, "el"); CHECK(el != nullptr, "variable 'el' found"); - CHECK(el->kind == ir::RecordKind::Variable, - "el is a Variable record"); + CHECK(el->kind == ir::RecordKind::Variable, "el is a Variable record"); - CHECK(unit->language() == "tsx", - "language is tsx"); + CHECK(unit->language() == "tsx", "language is tsx"); ts_tree_delete(tree); delete unit; @@ -163,14 +142,12 @@ static void test_jsx_element_skipped() CHECK(tree != nullptr, "parse succeeded"); ir::TsxVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/jsx.tsx"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/jsx.tsx"); // Only the variable 'el' should be emitted const ir::Record *el = findByName(*unit, "el"); CHECK(el != nullptr, "variable 'el' found"); - CHECK(el->kind == ir::RecordKind::Variable, - "el is a Variable record"); + CHECK(el->kind == ir::RecordKind::Variable, "el is a Variable record"); // No extra records from JSX ts_tree_delete(tree); @@ -193,8 +170,7 @@ static void test_jsx_expression_recurses() CHECK(tree != nullptr, "parse succeeded"); ir::TsxVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/app.tsx"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/app.tsx"); // Should have a Function (App) size_t funcs = countKind(*unit, ir::RecordKind::Function); @@ -227,8 +203,7 @@ static void test_tsx_with_interface() CHECK(tree != nullptr, "parse succeeded"); ir::TsxVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/greet.tsx"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/greet.tsx"); // Interface should be detected size_t ifaces = countKind(*unit, ir::RecordKind::Interface); @@ -239,8 +214,7 @@ static void test_tsx_with_interface() CHECK_EQ(funcs, 1ULL, "1 function found"); // Language should be tsx - CHECK(unit->language() == "tsx", - "language is tsx"); + CHECK(unit->language() == "tsx", "language is tsx"); ts_tree_delete(tree); delete unit; @@ -263,8 +237,7 @@ static void test_tsx_member_call() CHECK(tree != nullptr, "parse succeeded"); ir::TsxVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/app.tsx"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/app.tsx"); // Import should be detected size_t imports = countKind(*unit, ir::RecordKind::Import); @@ -296,8 +269,7 @@ static void test_tsx_empty_file() CHECK(tree != nullptr, "parse succeeded"); ir::TsxVisitor visitor; - ir::SemanticUnit *unit = visitor.visit(tree, code, - "/test/empty.tsx"); + ir::SemanticUnit *unit = visitor.visit(tree, code, "/test/empty.tsx"); CHECK(unit != nullptr, "visit returned unit"); CHECK(!unit->empty(), "unit is not empty"); @@ -319,7 +291,7 @@ int main() test_tsx_member_call(); test_tsx_empty_file(); - printf("\n=== tsx_visitor test passed (%d/%d) ===\n", - tests_passed, tests_run); + printf("\n=== tsx_visitor test passed (%d/%d) ===\n", tests_passed, + tests_run); return 0; } diff --git a/engine/tests/test_type_extraction.cpp b/engine/tests/test_type_extraction.cpp index 18777f7..56f8b5d 100644 --- a/engine/tests/test_type_extraction.cpp +++ b/engine/tests/test_type_extraction.cpp @@ -1,11 +1,13 @@ #include "test_e2e.h" +#include "test_engine_handle.h" // E2E test for type extraction: verify that the Go visitor correctly // emits TypeRef records for variables, parameters, and return types. // These records are stored in semantic_records and then processed by // the buildGraph pipeline into type_info and type_ref tables. -int main() { - const char* code = R"(package main +int main() +{ + const char *code = R"(package main type User struct { Name string @@ -27,52 +29,54 @@ func processUsers() { } )"; - char db_path[64]; - snprintf(db_path, sizeof(db_path), "/tmp/test_type_go.db"); - unlink(db_path); + char db_path[64]; + snprintf(db_path, sizeof(db_path), "/tmp/test_type_go.db"); + unlink(db_path); - int rc = engine_init(db_path); - check(rc == 0, "engine_init"); + g_engine = engine_create(db_path); + const int rc = g_engine != nullptr ? 0 : -1; + check(rc == 0, "engine_init"); - // Write test file - FILE* f = fopen("/tmp/test_type_go.go", "w"); - check(f != nullptr, "fopen"); - fwrite(code, 1, strlen(code), f); - fclose(f); + // Write test file + FILE *f = fopen("/tmp/test_type_go.go", "w"); + check(f != nullptr, "fopen"); + fwrite(code, 1, strlen(code), f); + fclose(f); - // Create project - uint64_t pid = engine_create_project("/tmp", "type-go-test"); - check(pid > 0, "create_project"); + // Create project + uint64_t pid = engine_create_project(g_engine, "/tmp", "type-go-test"); + check(pid > 0, "create_project"); - // Index - char* result = engine_index_file(pid, "/tmp/test_type_go.go"); - print_json("Index", result); - check(strstr(result, "\"ok\":true") != nullptr, "index_file ok"); - engine_free_string(result); + // Index + char *result = engine_index_file(g_engine, pid, "/tmp/test_type_go.go"); + print_json("Index", result); + check(strstr(result, "\"ok\":true") != nullptr, "index_file ok"); + engine_free_string(result); - // Verify definitions exist - char* def = engine_find_definition(pid, "User", nullptr); - print_json("Definition: User", def); - check(strstr(def, "User") != nullptr, "find_def User"); - engine_free_string(def); + // Verify definitions exist + char *def = engine_find_definition(g_engine, pid, "User", nullptr); + print_json("Definition: User", def); + check(strstr(def, "User") != nullptr, "find_def User"); + engine_free_string(def); - def = engine_find_definition(pid, "NewUser", nullptr); - print_json("Definition: NewUser", def); - check(strstr(def, "NewUser") != nullptr, "find_def NewUser"); - engine_free_string(def); + def = engine_find_definition(g_engine, pid, "NewUser", nullptr); + print_json("Definition: NewUser", def); + check(strstr(def, "NewUser") != nullptr, "find_def NewUser"); + engine_free_string(def); - // Check graph stats - char* stats = engine_get_graph_stats(pid); - print_json("Graph Stats", stats); - check(strstr(stats, "total_nodes") != nullptr, "get_graph_stats"); - engine_free_string(stats); + // Check graph stats + char *stats = engine_get_graph_stats(g_engine, pid); + print_json("Graph Stats", stats); + check(strstr(stats, "total_nodes") != nullptr, "get_graph_stats"); + engine_free_string(stats); - // NOTE: The type_info and type_ref tables are populated by the project - // pipeline (buildGraph in store_graph.cpp), not by the single-file - // index path. This test verifies the single-file index path works. - // Full pipeline verification requires indexing a multi-file project. + // NOTE: The type_info and type_ref tables are populated by the project + // pipeline (buildGraph in store_graph.cpp), not by the single-file + // index path. This test verifies the single-file index path works. + // Full pipeline verification requires indexing a multi-file project. - engine_shutdown(); - printf("\n=== type extraction test passed ===\n"); - return 0; + engine_destroy(g_engine); + g_engine = nullptr; + printf("\n=== type extraction test passed ===\n"); + return 0; } \ No newline at end of file diff --git a/engine/tests/test_typed_relation_query.cpp b/engine/tests/test_typed_relation_query.cpp index 504fcff..244c09d 100644 --- a/engine/tests/test_typed_relation_query.cpp +++ b/engine/tests/test_typed_relation_query.cpp @@ -24,7 +24,7 @@ #include "../src/query/query_engine.h" #include "../src/store/store.h" -#include +#include "test_check.h" #include #include #include @@ -44,13 +44,13 @@ static void insertEntity(store::GraphStore &store, uint64_t project_id, "start_col, end_row, end_col) " "VALUES (?, ?, 0, ?, ?, ?, 'go', 1, 0, 10, 0)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, id); sqlite3_bind_int64(stmt, 2, static_cast(project_id)); sqlite3_bind_text(stmt, 3, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 4, name, -1, SQLITE_TRANSIENT); sqlite3_bind_text(stmt, 5, file_path, -1, SQLITE_TRANSIENT); - assert(sqlite3_step(stmt) == SQLITE_DONE); + CHECK(sqlite3_step(stmt) == SQLITE_DONE); sqlite3_finalize(stmt); } @@ -66,7 +66,7 @@ static bool insertRelation(store::GraphStore &store, uint64_t project_id, "INSERT OR IGNORE INTO relation (project_id, source_id, " "target_id, type) VALUES (?, ?, ?, ?)"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, source_id); sqlite3_bind_int64(stmt, 3, target_id); @@ -74,7 +74,7 @@ static bool insertRelation(store::GraphStore &store, uint64_t project_id, int rc = sqlite3_step(stmt); int changes = sqlite3_changes(db); sqlite3_finalize(stmt); - assert(rc == SQLITE_DONE); + CHECK(rc == SQLITE_DONE); return changes > 0; } @@ -88,7 +88,7 @@ static int countRelations(store::GraphStore &store, uint64_t project_id, "SELECT COUNT(*) FROM relation WHERE project_id=? AND " "source_id=? AND target_id=? AND type=?"; sqlite3_stmt *stmt = nullptr; - assert(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr) == SQLITE_OK); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); sqlite3_bind_int64(stmt, 2, source_id); sqlite3_bind_int64(stmt, 3, target_id); @@ -136,7 +136,7 @@ int main() } uint64_t project_id = store.createProject("/typed-rel", "typed-rel"); - assert(project_id > 0); + CHECK(project_id > 0); // ── Insert two entities sharing a single source→target pair ── insertEntity(store, project_id, 1, "caller", "/t/a.go"); @@ -146,15 +146,15 @@ int main() // References(0), Calls(1), Defines(2), Contains(3). The unique // index is on (project_id, source_id, target_id, type), so all four // coexist — they differ only by `type`. - assert(insertRelation( + CHECK(insertRelation( store, project_id, 1, 2, graph::relationTypeToInt(graph::EdgeType::References))); - assert(insertRelation(store, project_id, 1, 2, - graph::relationTypeToInt(graph::EdgeType::Calls))); - assert(insertRelation( + CHECK(insertRelation(store, project_id, 1, 2, + graph::relationTypeToInt(graph::EdgeType::Calls))); + CHECK(insertRelation( store, project_id, 1, 2, graph::relationTypeToInt(graph::EdgeType::Defines))); - assert(insertRelation( + CHECK(insertRelation( store, project_id, 1, 2, graph::relationTypeToInt(graph::EdgeType::Contains))); @@ -162,11 +162,11 @@ int main() bool dup_inserted = insertRelation( store, project_id, 1, 2, graph::relationTypeToInt(graph::EdgeType::Calls)); - assert(!dup_inserted && - "duplicate Calls(1) relation must be rejected by the unique " - "index (INSERT OR IGNORE should report 0 changes)"); - assert(countRelations(store, project_id, 1, 2, 1) == 1 && - "exactly one Calls(1) relation must exist for the pair"); + CHECK(!dup_inserted && + "duplicate Calls(1) relation must be rejected by the unique " + "index (INSERT OR IGNORE should report 0 changes)"); + CHECK(countRelations(store, project_id, 1, 2, 1) == 1 && + "exactly one Calls(1) relation must exist for the pair"); // ── Query boundary: callees of "caller" must be ONLY "callee" ── // Before Step 1 the query matched `CALLS|RELATES` and returned all @@ -175,41 +175,41 @@ int main() std::string callees = engine.getCallees(project_id, "caller", nullptr); printf(" [debug] getCallees(caller) = %s\n", callees.c_str()); - assert(callees.find("callee") != std::string::npos && - "getCallees(caller) must contain the callee (Calls edge)"); + CHECK(callees.find("callee") != std::string::npos && + "getCallees(caller) must contain the callee (Calls edge)"); // "callee" must appear exactly once — no duplicate CALLS edges and // no References/Defines/Contains leakage. int callee_hits = countOccurrences(callees, "\"name\":\"callee\""); - assert(callee_hits == 1 && - "getCallees(caller) must return callee exactly once (no " - "duplicate typed edges, no non-Calls contamination)"); + CHECK(callee_hits == 1 && + "getCallees(caller) must return callee exactly once (no " + "duplicate typed edges, no non-Calls contamination)"); std::string callers = engine.getCallers(project_id, "callee", nullptr); printf(" [debug] getCallers(callee) = %s\n", callers.c_str()); - assert(callers.find("caller") != std::string::npos && - "getCallers(callee) must contain the caller (Calls edge)"); + CHECK(callers.find("caller") != std::string::npos && + "getCallers(callee) must contain the caller (Calls edge)"); int caller_hits = countOccurrences(callers, "\"name\":\"caller\""); - assert(caller_hits == 1 && - "getCallers(callee) must return caller exactly once"); + CHECK(caller_hits == 1 && + "getCallers(callee) must return caller exactly once"); // ── Verify total counts in the JSON match the deduped edge set ── // total should be 1 for both directions (only the Calls edge). - assert(callees.find("\"total\":1") != std::string::npos && - "getCallees total must be 1 (single Calls edge)"); - assert(callers.find("\"total\":1") != std::string::npos && - "getCallers total must be 1 (single Calls edge)"); + CHECK(callees.find("\"total\":1") != std::string::npos && + "getCallees total must be 1 (single Calls edge)"); + CHECK(callers.find("\"total\":1") != std::string::npos && + "getCallers total must be 1 (single Calls edge)"); // ── Verify SQLite relation layer is unaffected by the query ── // All four typed relations still exist in SQLite; only the query // boundary filters to Calls(1). - assert(countRelations(store, project_id, 1, 2, 0) == 1 && - "References(0) relation preserved in SQLite"); - assert(countRelations(store, project_id, 1, 2, 1) == 1 && - "Calls(1) relation preserved in SQLite (deduped)"); - assert(countRelations(store, project_id, 1, 2, 2) == 1 && - "Defines(2) relation preserved in SQLite"); - assert(countRelations(store, project_id, 1, 2, 3) == 1 && - "Contains(3) relation preserved in SQLite"); + CHECK(countRelations(store, project_id, 1, 2, 0) == 1 && + "References(0) relation preserved in SQLite"); + CHECK(countRelations(store, project_id, 1, 2, 1) == 1 && + "Calls(1) relation preserved in SQLite (deduped)"); + CHECK(countRelations(store, project_id, 1, 2, 2) == 1 && + "Defines(2) relation preserved in SQLite"); + CHECK(countRelations(store, project_id, 1, 2, 3) == 1 && + "Contains(3) relation preserved in SQLite"); store.close(); unlink(kDbPath); @@ -221,5 +221,5 @@ int main() printf(" - UNIQUE(project_id, source_id, target_id, type) rejects " "duplicate Calls\n"); printf(" - defensive result-layer dedup collapses stale duplicates\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_verifier_claim_coverage.cpp b/engine/tests/test_verifier_claim_coverage.cpp index 83e3e8f..3866e41 100644 --- a/engine/tests/test_verifier_claim_coverage.cpp +++ b/engine/tests/test_verifier_claim_coverage.cpp @@ -21,7 +21,7 @@ #include "verify/claim.h" #include "verify/registry.h" -#include +#include "test_check.h" #include #include #include @@ -30,6 +30,7 @@ #include #include #include +#include "test_engine_handle.h" // ─── Fixture: a tiny Go project with known call graph ──────────────── // main -> compute -> multiply -> add @@ -42,7 +43,7 @@ static void writeFixture(const std::string &dir) std::filesystem::create_directories(dir); FILE *f = fopen((dir + "/main.go").c_str(), "w"); - assert(f != nullptr); + CHECK(f != nullptr); fputs("package main\n\n" "func add(a, b int) int { return a + b }\n" "func multiply(a, b int) int {\n" @@ -62,7 +63,7 @@ static void writeFixture(const std::string &dir) fclose(f); FILE *g = fopen((dir + "/controller.go").c_str(), "w"); - assert(g != nullptr); + CHECK(g != nullptr); // A Controller-named function so ArchitectureVerifier detects a // "Controller" layer member via the name suffix rule. fputs("package main\n\n" @@ -79,15 +80,16 @@ static void writeFixture(const std::string &dir) static uint64_t indexFixture(const char *db_path, const char *proj_dir, const char *proj_name) { - if (engine_init(db_path) != 0) { + g_engine = engine_create(db_path); + if (!g_engine) { fprintf(stderr, "FAIL: engine_init failed\n"); exit(1); } - uint64_t pid = engine_create_project(proj_dir, proj_name); - assert(pid > 0); - char *idx = engine_index_project(pid, proj_dir, nullptr); - assert(idx != nullptr); - assert(strstr(idx, "\"ok\":true") != nullptr); + uint64_t pid = engine_create_project(g_engine, proj_dir, proj_name); + CHECK(pid > 0); + char *idx = engine_index_project(g_engine, pid, proj_dir, nullptr); + CHECK(idx != nullptr); + CHECK(strstr(idx, "\"ok\":true") != nullptr); engine_free_string(idx); usleep(200000); return pid; @@ -97,11 +99,11 @@ static uint64_t indexFixture(const char *db_path, const char *proj_dir, // succeeded (no lifecycle error codes). static char *verifyClaimOk(uint64_t pid, const std::string &claim_json) { - char *out = engine_verify_claim(pid, claim_json.c_str()); - assert(out != nullptr); - assert(strstr(out, "registry_empty") == nullptr); - assert(strstr(out, "claim_type_unsupported") == nullptr); - assert(strstr(out, "verifier_execution_failed") == nullptr); + char *out = engine_verify_claim(g_engine, pid, claim_json.c_str()); + CHECK(out != nullptr); + CHECK(strstr(out, "registry_empty") == nullptr); + CHECK(strstr(out, "claim_type_unsupported") == nullptr); + CHECK(strstr(out, "verifier_execution_failed") == nullptr); return out; } @@ -167,17 +169,17 @@ int main() } // 4 public types total. - assert(public_types.size() == 4); + CHECK(public_types.size() == 4); // All 4 must be supported (Step 9.3 added FunctionImplements). - assert(supported.size() == 4); - assert(unsupported.empty()); + CHECK(supported.size() == 4); + CHECK(unsupported.empty()); // Sanity: each public type has a stable wire name. for (auto t : public_types) { const char *name = verify::claimTypeWireName(t); - assert(name != nullptr); - assert(*name != '\0'); - assert(strcmp(name, "unknown") != 0); + CHECK(name != nullptr); + CHECK(*name != '\0'); + CHECK(strcmp(name, "unknown") != 0); } printf("Test 1 (100%% claim type coverage): PASS\n"); @@ -223,11 +225,11 @@ int main() char *r = verifyClaimOk(pid, c.claim_json); std::string verdict = extractVerdict(r); std::string verifier = extractVerifier(r); - assert(!verdict.empty()); - assert(!verifier.empty()); - assert(verdict == "Supported" || - verdict == "Contradicted" || - verdict == "Unknown"); + CHECK(!verdict.empty()); + CHECK(!verifier.empty()); + CHECK(verdict == "Supported" || + verdict == "Contradicted" || + verdict == "Unknown"); printf(" %s -> %s via %s\n", c.wire_name, verdict.c_str(), verifier.c_str()); engine_free_string(r); @@ -249,12 +251,12 @@ int main() verifyClaimOk(pid, "{\"type\":\"function_implements\"," "\"subject\":\"compute\"}"); std::string v_supported = extractVerdict(r_supported); - assert(v_supported == "Supported"); - assert(strstr(r_supported, "FunctionImplementsVerifier") != - nullptr); + CHECK(v_supported == "Supported"); + CHECK(strstr(r_supported, "FunctionImplementsVerifier") != + nullptr); // Downgraded confidence: structural check only. - assert(strstr(r_supported, "\"confidence\":0.55") != nullptr || - strstr(r_supported, "\"confidence\": 0.55") != nullptr); + CHECK(strstr(r_supported, "\"confidence\":0.55") != nullptr || + strstr(r_supported, "\"confidence\": 0.55") != nullptr); engine_free_string(r_supported); // Contradicted: `nonexistent_function` does not exist. @@ -262,7 +264,7 @@ int main() pid, "{\"type\":\"function_implements\"," "\"subject\":\"nonexistent_function_xyz\"}"); std::string v_contradicted = extractVerdict(r_contradicted); - assert(v_contradicted == "Contradicted"); + CHECK(v_contradicted == "Contradicted"); engine_free_string(r_contradicted); printf("Test 3 (FunctionImplementsVerifier ground truth): " @@ -280,8 +282,8 @@ int main() pid, "{\"type\":\"capability_exists\"," "\"subject\":\"NonExistentCapability\"}"); std::string v = extractVerdict(r); - assert(v == "Contradicted"); - assert(strstr(r, "CapabilityVerifier") != nullptr); + CHECK(v == "Contradicted"); + CHECK(strstr(r, "CapabilityVerifier") != nullptr); engine_free_string(r); printf("Test 4 (CapabilityVerifier ground truth): PASS\n"); } @@ -299,8 +301,8 @@ int main() std::string v = extractVerdict(r); // No contract declared in the fixture → Unknown is the safe // verdict (we cannot contradict an undeclared claim). - assert(v == "Unknown"); - assert(strstr(r, "ContractVerifier") != nullptr); + CHECK(v == "Unknown"); + CHECK(strstr(r, "ContractVerifier") != nullptr); engine_free_string(r); printf("Test 5 (ContractVerifier ground truth): PASS\n"); } @@ -317,8 +319,8 @@ int main() "\"object\":\"Service\"," "\"scope\":\"Repository\"}"); std::string v = extractVerdict(r); - assert(v == "Unknown"); - assert(strstr(r, "ArchitectureVerifier") != nullptr); + CHECK(v == "Unknown"); + CHECK(strstr(r, "ArchitectureVerifier") != nullptr); engine_free_string(r); printf("Test 6 (ArchitectureVerifier ground truth): PASS\n"); } @@ -338,21 +340,22 @@ int main() const char *empty_dir = "/tmp/test_verifier_coverage_empty"; std::filesystem::remove_all(empty_dir); std::filesystem::create_directories(empty_dir); - uint64_t empty_pid = - engine_create_project(empty_dir, "empty-no-index"); - assert(empty_pid > 0); - assert(empty_pid != pid); // must be a new project - char *r = engine_verify_claim( - empty_pid, "{\"type\":\"function_implements\"," - "\"subject\":\"compute\"}"); - assert(r != nullptr); + uint64_t empty_pid = engine_create_project(g_engine, empty_dir, + "empty-no-index"); + CHECK(empty_pid > 0); + CHECK(empty_pid != pid); // must be a new project + char *r = + engine_verify_claim(g_engine, empty_pid, + "{\"type\":\"function_implements\"," + "\"subject\":\"compute\"}"); + CHECK(r != nullptr); std::string v = extractVerdict(r); - assert(v == "Unknown"); + CHECK(v == "Unknown"); // The error_code field must be present and tagged // evidence_backend_not_ready so callers can distinguish // "no evidence yet" from a normal Unknown verdict. - assert(strstr(r, "evidence_backend_not_ready") != nullptr); - assert(strstr(r, "evidence backend not ready") != nullptr); + CHECK(strstr(r, "evidence_backend_not_ready") != nullptr); + CHECK(strstr(r, "evidence backend not ready") != nullptr); engine_free_string(r); printf("Test 7 (evidence backend not ready -> Unknown): " "PASS\n"); @@ -368,10 +371,11 @@ int main() { // claim_type_unsupported: unknown type string. char *r_unknown = - engine_verify_claim(pid, "{\"type\":\"bogus_type\"," - "\"subject\":\"foo\"}"); - assert(r_unknown != nullptr); - assert(strstr(r_unknown, "claim_type_unsupported") != nullptr); + engine_verify_claim(g_engine, pid, + "{\"type\":\"bogus_type\"," + "\"subject\":\"foo\"}"); + CHECK(r_unknown != nullptr); + CHECK(strstr(r_unknown, "claim_type_unsupported") != nullptr); engine_free_string(r_unknown); // registry_empty: after engine_shutdown(), the registry is @@ -386,19 +390,20 @@ int main() probe.subject = "test"; verify::Verifier *matched = verify::VerifierRegistry::instance().match(probe); - assert(matched == nullptr); - assert(verify::VerifierRegistry::instance().verifier_count() == - 0); + CHECK(matched == nullptr); + CHECK(verify::VerifierRegistry::instance().verifier_count() == + 0); // Restore the registry for any subsequent tests. verify::VerifierRegistry::instance().ensureDefaultVerifiers( nullptr, 0); - assert(verify::VerifierRegistry::instance().verifier_count() >= - 4); + CHECK(verify::VerifierRegistry::instance().verifier_count() >= + 4); printf("Test 8 (distinct error codes): PASS\n"); } - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; printf("\n=== test_verifier_claim_coverage PASSED ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_verifier_evidence_gates.cpp b/engine/tests/test_verifier_evidence_gates.cpp new file mode 100644 index 0000000..38bcdbf --- /dev/null +++ b/engine/tests/test_verifier_evidence_gates.cpp @@ -0,0 +1,343 @@ +// test_verifier_evidence_gates.cpp +// +// Pins two evidence rules that were added to the verifiers. +// +// 1. Drift detectors must not draw a conclusion from an empty backend. +// Before the gate, a project with a declared capability but nothing indexed +// reported every capability as a severity-2 CapabilityDrift ("declared in +// the README but not implemented"), because countImplementingEntities() +// returns 0 when the entity table is empty. findOrphanModules() had the +// same shape: its `NOT EXISTS (import …)` test is vacuously true when the +// import table is empty, so every module with >=10 entities was reported as +// an orphan. Both are hard conclusions drawn from missing evidence. +// +// 2. ContractVerifier(ThreadSafe) must +// (a) not read a lock-shaped but unrelated name (Block / BlockStore / +// Clock / Deadlock / Unlock) as synchronisation evidence — an +// unanchored `%lock%` pattern made a project with a `Block` class +// report as thread-safe — and +// (b) answer Unknown rather than Contradicted when nothing matches, since a +// name pattern that fails to match is not evidence that the code is +// unsafe (Rust's std::sync, Java `synchronized`, an OS lock or a +// message-passing design are all invisible to the pattern). +// +// The database state is built with explicit SQL rather than by indexing a +// fixture, so each case tests exactly what it claims to. + +#include "../src/store/store.h" +#include "../src/verify/capability_drift.h" +#include "../src/verify/claim.h" +#include "../src/verify/contract_verifier.h" +#include "../src/verify/dead_code_inspector.h" + +#include "test_check.h" +#include +#include +#include +#include +#include +#include + +using namespace store; + +static const char *kDbPath = "/tmp/test_verifier_evidence_gates.db"; + +/// Run a statement, failing loudly: this test builds its own state, so an +/// error here means the fixture is wrong rather than the code under test. +static void execOrDie(sqlite3 *db, const std::string &sql) +{ + char *err = nullptr; + if (sqlite3_exec(db, sql.c_str(), nullptr, nullptr, &err) != + SQLITE_OK) { + fprintf(stderr, "FAIL: sqlite3_exec: %s | sql=%s\n", + err ? err : "(no message)", sql.c_str()); + sqlite3_free(err); + exit(1); + } +} + +static void insertEntity(sqlite3 *db, uint64_t pid, int64_t id, int kind, + const std::string &name, const std::string &file, + int start_row) +{ + execOrDie(db, + "INSERT INTO entity (id, project_id, kind, name, " + "qualified_name, file_path, language, start_row, start_col, " + "end_row, end_col) VALUES (" + + std::to_string(id) + "," + std::to_string(pid) + "," + + std::to_string(kind) + ",'" + name + "','','" + file + + "','go'," + std::to_string(start_row) + ",0," + + std::to_string(start_row + 1) + ",0)"); +} + +static void insertCallRelation(sqlite3 *db, uint64_t pid, int64_t src, + int64_t tgt) +{ + execOrDie( + db, + "INSERT INTO relation (project_id, source_id, target_id, type) " + "VALUES (" + + std::to_string(pid) + "," + std::to_string(src) + "," + + std::to_string(tgt) + ",1)"); +} + +static void clearGraph(sqlite3 *db) +{ + execOrDie(db, "DELETE FROM entity"); + execOrDie(db, "DELETE FROM relation"); + execOrDie(db, "DELETE FROM import"); +} + +/// Minimal graph so that evidence_backend_ready() reports "ready": at least +/// one entity row and one relation row. +static void seedReadyBackend(sqlite3 *db, uint64_t pid) +{ + insertEntity(db, pid, 1, 0, "helper", "src/main.go", 1); + insertCallRelation(db, pid, 1, 1); +} + +/// Count the orphan-module findings ("DeadModule") in an inspect() bundle. +/// inspect() also runs the orphan-function / architecture-drift / component +/// checks, so the type has to be filtered rather than the total counted. +static size_t countDeadModules(const std::vector &all) +{ + size_t n = 0; + for (const auto &f : all) + if (f.type == "DeadModule") + ++n; + return n; +} + +/// Count the dead-function findings ("DeadFunction") in an inspect() bundle. +/// Mirrors countDeadModules: inspect() emits several finding types and only +/// this one is the orphan-function check's output. +static size_t countDeadFunctions(const std::vector &all) +{ + size_t n = 0; + for (const auto &f : all) + if (f.type == "DeadFunction") + ++n; + return n; +} + +/// Count the architecture-drift findings ("ArchitectureDrift") in an +/// inspect() bundle. Mirrors countDeadModules / countDeadFunctions. +static size_t countArchitectureDrift(const std::vector &all) +{ + size_t n = 0; + for (const auto &f : all) + if (f.type == "ArchitectureDrift") + ++n; + return n; +} + +static std::string verdictOf(const verify::EvidenceRecord &rec) +{ + switch (rec.verdict) { + case verify::Verdict::Supported: + return "Supported"; + case verify::Verdict::Contradicted: + return "Contradicted"; + case verify::Verdict::Unknown: + return "Unknown"; + } + return "?"; +} + +int main() +{ + unlink(kDbPath); + unlink((std::string(kDbPath) + "-wal").c_str()); + unlink((std::string(kDbPath) + "-shm").c_str()); + + GraphStore store; + if (!store.open(kDbPath)) { + fprintf(stderr, "FAIL: cannot open store: %s\n", + store.error().c_str()); + return 1; + } + const uint64_t pid = store.createProject("/tmp", "verifier-gates"); + CHECK(pid > 0); + sqlite3 *db = store.handle(); + CHECK(db != nullptr); + // ThreadSafe must be a *declared* contract, otherwise verify() returns + // Unknown at the "no contract declared" step and every assertion below + // would pass for the wrong reason. + execOrDie(db, + "INSERT INTO contract (project_id, name, origin) VALUES (" + + std::to_string(pid) + ",'threadsafe','readme')"); + + verify::ContractVerifier verifier(&store, pid); + verify::Claim claim; + claim.type = verify::ClaimType::ContractHolds; + claim.subject = "ThreadSafe"; + claim.predicate = "implemented_by"; + claim.scope = "repository"; + claim.source_kind = "manual"; + + // ── Case 1: a `Block` class is not synchronisation evidence ── + // `%lock%` matched it before the patterns were anchored. + clearGraph(db); + insertEntity(db, pid, 1, 2, "Block", "src/block.go", 1); + insertEntity(db, pid, 2, 0, "helper", "src/main.go", 2); + insertCallRelation(db, pid, 2, 1); + { + const std::string v = verdictOf(verifier.verify(claim)); + printf(" [debug] Block-only -> %s\n", v.c_str()); + CHECK(v == "Unknown" && + "a class named Block must not be read as a lock"); + } + + // ── Case 2: real synchronisation evidence still supports it ── + insertEntity(db, pid, 3, 5, "mutex", "src/main.go", 3); + { + const std::string v = verdictOf(verifier.verify(claim)); + printf(" [debug] with mutex -> %s\n", v.c_str()); + CHECK(v == "Supported" && + "an actual mutex entity must still support ThreadSafe"); + } + + // ── Case 3: no synchronisation name at all -> Unknown ── + // The previous code returned Contradicted ("not thread safe") here. + clearGraph(db); + seedReadyBackend(db, pid); + { + const std::string v = verdictOf(verifier.verify(claim)); + printf(" [debug] no sync primitive -> %s\n", v.c_str()); + CHECK(v == "Unknown" && + "absence of a matching name is not evidence of being " + "unsafe; Contradicted would over-claim"); + } + + // ── Case 4: capability drift with an empty backend ── + // A declared capability plus no indexed entity rows must yield no drift + // conclusion (previously one severity-2 CapabilityDrift). + clearGraph(db); + execOrDie( + db, + "INSERT INTO capability (project_id, name, summary, source_kind) " + "VALUES (" + + std::to_string(pid) + ",'FancyFeature','','readme')"); + { + const std::vector drifts = + verify::detectCapabilityDrift(store, pid); + printf(" [debug] capability drifts -> %zu (empty backend)\n", + drifts.size()); + CHECK(drifts.empty() && + "an empty backend must not produce capability drift"); + } + + // ── Case 5: orphan modules with entity rows but no imports ── + // The `NOT EXISTS (import …)` test is vacuously true without imports, so + // a scope with >=10 entities used to be reported as an orphan module. + clearGraph(db); + execOrDie(db, "INSERT INTO scope (project_id, parent_id, kind, name) " + "VALUES (" + + std::to_string(pid) + ",0,1,'src/')"); + for (int64_t i = 1; i <= 10; ++i) + insertEntity(db, pid, i, 0, "f" + std::to_string(i), + "src/f" + std::to_string(i) + ".go", i); + insertCallRelation(db, pid, 1, 2); + { + verify::DeadCodeInspector inspector(&store, pid); + const size_t orphans = countDeadModules(inspector.inspect()); + printf(" [debug] orphan modules -> %zu (no import rows)\n", + orphans); + CHECK(orphans == 0 && + "modules must not be called orphaned when the import " + "table is empty"); + } + + // ── Case 6: orphan detection still works with real import data ── + // Guards against "fixed" by disabling the query: a module with entities + // and no importing file is still reported once imports exist. + // Must run BEFORE Case 5b: 5b's clearGraph drops the 10 entities this + // case needs to clear the >=10-entity threshold. + execOrDie( + db, + "INSERT INTO import (project_id, file_path, target_path, alias, " + "source_scope_id) VALUES (" + + std::to_string(pid) + + ",'other/consumer.go','x/y','',0)"); + { + verify::DeadCodeInspector inspector(&store, pid); + const size_t orphans = countDeadModules(inspector.inspect()); + printf(" [debug] orphan modules -> %zu (imports present)\n", + orphans); + CHECK(orphans > 0 && + "with import rows present the orphan query must still " + "report src/"); + } + + // ── Case 5b: dead functions with an empty relation table ── + // findOrphanFunctions's `NOT EXISTS (relation …)` test is vacuously + // true for every non-public, non-entry-point function when relation + // is empty, so a pre-index project would emit mass DeadFunction + // findings at 0.90 confidence. The relation gate must suppress them. + clearGraph(db); + for (int64_t i = 1; i <= 5; ++i) + insertEntity(db, pid, i, 0, "fn" + std::to_string(i), + "src/fn" + std::to_string(i) + ".go", i); + { + verify::DeadCodeInspector inspector(&store, pid); + const auto all = inspector.inspect(); + const size_t dead = countDeadFunctions(all); + printf(" [debug] dead functions -> %zu (no relation rows)\n", + dead); + CHECK(dead == 0 && + "empty relation table must not produce DeadFunction " + "findings (vacuous NOT EXISTS)"); + } + + // ── Case 7: nested scope prefixes are not a boundary crossing ── + // A file under `src/ir/` prefix-matches BOTH the child scope and its + // ancestor `src/`, so an intra-module call is emitted as the pair + // ('src/','src/ir/') and read back as architecture drift. Parent and + // child modules must be excluded; a true sibling pair must survive. + clearGraph(db); + execOrDie(db, "DELETE FROM scope"); + execOrDie( + db, + "INSERT INTO scope (project_id, parent_id, kind, name) VALUES (" + + std::to_string(pid) + ",0,1,'src/')"); + execOrDie( + db, + "INSERT INTO scope (project_id, parent_id, kind, name) VALUES (" + + std::to_string(pid) + ",0,1,'src/ir/')"); + execOrDie( + db, + "INSERT INTO scope (project_id, parent_id, kind, name) VALUES (" + + std::to_string(pid) + ",0,1,'src/verify/')"); + // 1 -> 2 lives entirely inside src/ir/ (both files prefix-match src/ + // and src/ir/): must NOT be drift. + insertEntity(db, pid, 1, 0, "a", "src/ir/a.go", 1); + insertEntity(db, pid, 2, 0, "b", "src/ir/b.go", 1); + // 3 (src/ir/) -> 4 (src/verify/) is a genuine sibling crossing. + insertEntity(db, pid, 3, 0, "c", "src/ir/c.go", 1); + insertEntity(db, pid, 4, 0, "d", "src/verify/d.go", 1); + insertCallRelation(db, pid, 1, 2); + insertCallRelation(db, pid, 3, 4); + { + verify::DeadCodeInspector inspector(&store, pid); + const auto all = inspector.inspect(); + const size_t drift = countArchitectureDrift(all); + printf(" [debug] architecture drift -> %zu " + "(1 nested pair + 1 sibling pair)\n", + drift); + // Only the src/ir/ -> src/verify/ sibling crossing may survive; + // the src/ir/-internal call must not surface as 'src/' <-> 'src/ir/'. + CHECK(drift == 1 && + "parent/child scope pairs must not be reported as " + "architecture drift, but a sibling crossing must be"); + } + + store.close(); + unlink(kDbPath); + unlink((std::string(kDbPath) + "-wal").c_str()); + unlink((std::string(kDbPath) + "-shm").c_str()); + + printf("\n=== test_verifier_evidence_gates PASSED ===\n"); + printf("Drift detectors gated; ThreadSafe no longer reads Block as a " + "lock and no longer contradicts on absence\n"); + return checkFailures() ? 1 : 0; +} diff --git a/engine/tests/test_verifier_ground_truth.cpp b/engine/tests/test_verifier_ground_truth.cpp index 5fd5eaf..d262831 100644 --- a/engine/tests/test_verifier_ground_truth.cpp +++ b/engine/tests/test_verifier_ground_truth.cpp @@ -23,13 +23,14 @@ #include "verify/claim.h" #include "verify/registry.h" -#include +#include "test_check.h" #include #include #include #include #include #include +#include "test_engine_handle.h" // ─── Fixture ───────────────────────────────────────────────────────── // A Go project with a known call chain plus an isolated function: @@ -43,7 +44,7 @@ static void writeFixture(const std::string &dir) std::filesystem::create_directories(dir); FILE *f = fopen((dir + "/main.go").c_str(), "w"); - assert(f != nullptr); + CHECK(f != nullptr); fputs("package main\n\n" "func add(a, b int) int { return a + b }\n" "func multiply(a, b int) int {\n" @@ -66,15 +67,16 @@ static void writeFixture(const std::string &dir) static uint64_t indexFixture(const char *db_path, const char *proj_dir, const char *proj_name) { - if (engine_init(db_path) != 0) { + g_engine = engine_create(db_path); + if (!g_engine) { fprintf(stderr, "FAIL: engine_init failed\n"); exit(1); } - uint64_t pid = engine_create_project(proj_dir, proj_name); - assert(pid > 0); - char *idx = engine_index_project(pid, proj_dir, nullptr); - assert(idx != nullptr); - assert(strstr(idx, "\"ok\":true") != nullptr); + uint64_t pid = engine_create_project(g_engine, proj_dir, proj_name); + CHECK(pid > 0); + char *idx = engine_index_project(g_engine, pid, proj_dir, nullptr); + CHECK(idx != nullptr); + CHECK(strstr(idx, "\"ok\":true") != nullptr); engine_free_string(idx); usleep(200000); return pid; @@ -83,11 +85,11 @@ static uint64_t indexFixture(const char *db_path, const char *proj_dir, // Helper: run verify_claim and assert dispatch succeeded. static char *verifyClaimOk(uint64_t pid, const std::string &claim_json) { - char *out = engine_verify_claim(pid, claim_json.c_str()); - assert(out != nullptr); - assert(strstr(out, "registry_empty") == nullptr); - assert(strstr(out, "claim_type_unsupported") == nullptr); - assert(strstr(out, "verifier_execution_failed") == nullptr); + char *out = engine_verify_claim(g_engine, pid, claim_json.c_str()); + CHECK(out != nullptr); + CHECK(strstr(out, "registry_empty") == nullptr); + CHECK(strstr(out, "claim_type_unsupported") == nullptr); + CHECK(strstr(out, "verifier_execution_failed") == nullptr); return out; } @@ -141,22 +143,23 @@ int main() // claim types in supported_claim_types, and evidence_backend_ready= // true for an indexed project. { - char *status = engine_get_verifier_registry_status(pid); - assert(status != nullptr); - assert(strstr(status, "\"registry_empty\":false") != nullptr); - assert(strstr(status, "\"verifier_count\":4") != nullptr); - assert(strstr(status, "\"capability_exists\"") != nullptr); - assert(strstr(status, "\"contract_holds\"") != nullptr); - assert(strstr(status, "\"architecture_follows\"") != nullptr); - assert(strstr(status, "\"function_implements\"") != nullptr); - assert(strstr(status, "\"unsupported_claim_types\":[]") != - nullptr); - assert(strstr(status, "\"evidence_backend_ready\":true") != - nullptr); + char *status = + engine_get_verifier_registry_status(g_engine, pid); + CHECK(status != nullptr); + CHECK(strstr(status, "\"registry_empty\":false") != nullptr); + CHECK(strstr(status, "\"verifier_count\":4") != nullptr); + CHECK(strstr(status, "\"capability_exists\"") != nullptr); + CHECK(strstr(status, "\"contract_holds\"") != nullptr); + CHECK(strstr(status, "\"architecture_follows\"") != nullptr); + CHECK(strstr(status, "\"function_implements\"") != nullptr); + CHECK(strstr(status, "\"unsupported_claim_types\":[]") != + nullptr); + CHECK(strstr(status, "\"evidence_backend_ready\":true") != + nullptr); // entity_count and relation_count must be > 0 for an indexed // project. - assert(strstr(status, "\"entity_count\":0") == nullptr); - assert(strstr(status, "\"relation_count\":0") == nullptr); + CHECK(strstr(status, "\"entity_count\":0") == nullptr); + CHECK(strstr(status, "\"relation_count\":0") == nullptr); engine_free_string(status); printf("Test 1 (introspection API healthy): PASS\n"); } @@ -171,13 +174,13 @@ int main() char *r = verifyClaimOk(pid, "{\"type\":\"function_implements\"," "\"subject\":\"compute\"}"); - assert(extractVerdict(r) == "Supported"); - assert(strstr(r, "FunctionImplementsVerifier") != nullptr); + CHECK(extractVerdict(r) == "Supported"); + CHECK(strstr(r, "FunctionImplementsVerifier") != nullptr); // Downgraded confidence: structural check only. - assert(strstr(r, "\"confidence\":0.55") != nullptr || - strstr(r, "\"confidence\": 0.55") != nullptr); + CHECK(strstr(r, "\"confidence\":0.55") != nullptr || + strstr(r, "\"confidence\": 0.55") != nullptr); int facts = countEvidenceFacts(r); - assert(facts >= 2); // at least 1 entity + 1 relation + CHECK(facts >= 2); // at least 1 entity + 1 relation engine_free_string(r); printf("Test 2 (FunctionImplements Supported, low confidence): PASS\n"); } @@ -193,11 +196,11 @@ int main() "{\"type\":\"function_implements\"," "\"subject\":\"orphanFunc\"}"); std::string v = extractVerdict(r); - assert(v == "Unknown"); - assert(strstr(r, "isolated") != nullptr); + CHECK(v == "Unknown"); + CHECK(strstr(r, "isolated") != nullptr); // Entity facts present (the function exists), but the detail // must mention "no callers/callees". - assert(strstr(r, "no callers/callees") != nullptr); + CHECK(strstr(r, "no callers/callees") != nullptr); engine_free_string(r); printf("Test 3 (FunctionImplements isolated -> Unknown): PASS\n"); } @@ -209,11 +212,11 @@ int main() char *r = verifyClaimOk(pid, "{\"type\":\"function_implements\"," "\"subject\":\"does_not_exist_xyz\"}"); - assert(extractVerdict(r) == "Contradicted"); - assert(strstr(r, "not found in canonical entity table") != - nullptr); + CHECK(extractVerdict(r) == "Contradicted"); + CHECK(strstr(r, "not found in canonical entity table") != + nullptr); int facts = countEvidenceFacts(r); - assert(facts == 0); + CHECK(facts == 0); engine_free_string(r); printf("Test 4 (FunctionImplements non-existent -> Contradicted): PASS\n"); } @@ -229,19 +232,20 @@ int main() const char *empty_dir = "/tmp/test_verifier_groundtruth_empty"; std::filesystem::remove_all(empty_dir); std::filesystem::create_directories(empty_dir); - uint64_t empty_pid = - engine_create_project(empty_dir, "empty-no-index"); - assert(empty_pid > 0); - assert(empty_pid != pid); + uint64_t empty_pid = engine_create_project(g_engine, empty_dir, + "empty-no-index"); + CHECK(empty_pid > 0); + CHECK(empty_pid != pid); // Introspection API must report backend NOT ready for the // empty project. - char *status = engine_get_verifier_registry_status(empty_pid); - assert(status != nullptr); - assert(strstr(status, "\"evidence_backend_ready\":false") != - nullptr); - assert(strstr(status, "\"entity_count\":0") != nullptr); - assert(strstr(status, "\"relation_count\":0") != nullptr); + char *status = engine_get_verifier_registry_status(g_engine, + empty_pid); + CHECK(status != nullptr); + CHECK(strstr(status, "\"evidence_backend_ready\":false") != + nullptr); + CHECK(strstr(status, "\"entity_count\":0") != nullptr); + CHECK(strstr(status, "\"relation_count\":0") != nullptr); engine_free_string(status); // Each claim type must return Unknown + backend-not-ready. @@ -265,12 +269,12 @@ int main() "\"subject\":\"compute\"}" }, }; for (const auto &c : cases) { - char *r = engine_verify_claim(empty_pid, + char *r = engine_verify_claim(g_engine, empty_pid, c.claim_json.c_str()); - assert(r != nullptr); - assert(extractVerdict(r) == "Unknown"); - assert(strstr(r, "evidence backend not ready") != - nullptr); + CHECK(r != nullptr); + CHECK(extractVerdict(r) == "Unknown"); + CHECK(strstr(r, "evidence backend not ready") != + nullptr); engine_free_string(r); } printf("Test 5 (backend not ready -> Unknown all types): PASS\n"); @@ -282,19 +286,20 @@ int main() // counts=0). This lets callers check registry health without a // project context. { - char *status = engine_get_verifier_registry_status(0); - assert(status != nullptr); - assert(strstr(status, "\"registry_empty\":false") != nullptr); - assert(strstr(status, "\"verifier_count\":4") != nullptr); - assert(strstr(status, "\"evidence_backend_ready\":false") != - nullptr); - assert(strstr(status, "\"entity_count\":0") != nullptr); - assert(strstr(status, "\"relation_count\":0") != nullptr); + char *status = engine_get_verifier_registry_status(g_engine, 0); + CHECK(status != nullptr); + CHECK(strstr(status, "\"registry_empty\":false") != nullptr); + CHECK(strstr(status, "\"verifier_count\":4") != nullptr); + CHECK(strstr(status, "\"evidence_backend_ready\":false") != + nullptr); + CHECK(strstr(status, "\"entity_count\":0") != nullptr); + CHECK(strstr(status, "\"relation_count\":0") != nullptr); engine_free_string(status); printf("Test 6 (introspection project_id=0): PASS\n"); } - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; printf("\n=== test_verifier_ground_truth PASSED ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_verifier_lifecycle.cpp b/engine/tests/test_verifier_lifecycle.cpp index 563f3ce..570e7bd 100644 --- a/engine/tests/test_verifier_lifecycle.cpp +++ b/engine/tests/test_verifier_lifecycle.cpp @@ -21,13 +21,14 @@ #include "verify/claim.h" #include "verify/registry.h" -#include +#include "test_check.h" #include #include #include #include #include #include +#include "test_engine_handle.h" // Helper: write a tiny Go project to `dir` so entity/relation tables get // populated after engine_index_project. The fixture defines add/multiply/ @@ -38,7 +39,7 @@ static void writeFixture(const std::string &dir) std::filesystem::create_directories(dir); FILE *f = fopen((dir + "/multi.go").c_str(), "w"); - assert(f != nullptr); + CHECK(f != nullptr); fputs("package main\n\n" "func add(a, b int) int { return a + b }\n" "func multiply(a, b int) int {\n" @@ -51,7 +52,7 @@ static void writeFixture(const std::string &dir) fclose(f); f = fopen((dir + "/main.go").c_str(), "w"); - assert(f != nullptr); + CHECK(f != nullptr); fputs("package main\n\n" "func main() {\n" " _ = compute(1, 2)\n" @@ -67,16 +68,17 @@ static void writeFixture(const std::string &dir) static uint64_t indexFixture(const char *db_path, const char *proj_dir, const char *proj_name) { - if (engine_init(db_path) != 0) { + g_engine = engine_create(db_path); + if (!g_engine) { fprintf(stderr, "FAIL: engine_init failed\n"); exit(1); } - uint64_t pid = engine_create_project(proj_dir, proj_name); - assert(pid > 0); + uint64_t pid = engine_create_project(g_engine, proj_dir, proj_name); + CHECK(pid > 0); - char *idx = engine_index_project(pid, proj_dir, nullptr); - assert(idx != nullptr); - assert(strstr(idx, "\"ok\":true") != nullptr); + char *idx = engine_index_project(g_engine, pid, proj_dir, nullptr); + CHECK(idx != nullptr); + CHECK(strstr(idx, "\"ok\":true") != nullptr); engine_free_string(idx); // Allow the synchronous graph build to settle. usleep(200000); @@ -88,13 +90,13 @@ static uint64_t indexFixture(const char *db_path, const char *proj_dir, // frees). The verdict itself is not asserted — only that dispatch worked. static char *assertDispatchOk(uint64_t pid, const std::string &claim_json) { - char *out = engine_verify_claim(pid, claim_json.c_str()); - assert(out != nullptr); + char *out = engine_verify_claim(g_engine, pid, claim_json.c_str()); + CHECK(out != nullptr); // Lifecycle error codes that indicate the registry is broken. None // of these should appear after a healthy init+create_project. - assert(strstr(out, "registry_empty") == nullptr); - assert(strstr(out, "claim_type_unsupported") == nullptr); - assert(strstr(out, "verifier_execution_failed") == nullptr); + CHECK(strstr(out, "registry_empty") == nullptr); + CHECK(strstr(out, "claim_type_unsupported") == nullptr); + CHECK(strstr(out, "verifier_execution_failed") == nullptr); return out; } @@ -104,9 +106,9 @@ static char *assertDispatchOk(uint64_t pid, const std::string &claim_json) static void assertRegistryHealthy() { auto ® = verify::VerifierRegistry::instance(); - assert(reg.verifier_count() >= 4); + CHECK(reg.verifier_count() >= 4); auto supported = reg.supported_claim_types(); - assert(supported.size() == 4); + CHECK(supported.size() == 4); } int main() @@ -153,11 +155,12 @@ int main() "\"subject\":\"compute\"}"); engine_free_string(r4); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; // After shutdown the registry MUST be empty so the // next init+ensureDefaultVerifiers re-populates it. - assert(verify::VerifierRegistry::instance() - .verifier_count() == 0); + CHECK(verify::VerifierRegistry::instance() + .verifier_count() == 0); } printf("Test 1 (3-cycle init/verify/shutdown): PASS\n"); } @@ -174,21 +177,23 @@ int main() // First init: create + index the project normally. uint64_t pid1 = indexFixture(db_path, proj_dir, "restore-proj"); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; // Second init: re-open the SAME db (project already exists). // Deliberately do NOT call engine_create_project — simulate a // restore/worker scenario. Use engine_get_latest_project_id to // recover the existing project_id. - assert(engine_init(db_path) == 0); - uint64_t pid2 = engine_get_latest_project_id(); - assert(pid2 == pid1); + g_engine = engine_create(db_path); + CHECK(g_engine != nullptr); + uint64_t pid2 = engine_get_latest_project_id(g_engine); + CHECK(pid2 == pid1); // Registry should be empty after init (engine_create_project // is what normally populates it). The first verify_claim call // must trigger ensureDefaultVerifiers() and dispatch. - assert(verify::VerifierRegistry::instance().verifier_count() == - 0); + CHECK(verify::VerifierRegistry::instance().verifier_count() == + 0); char *r = assertDispatchOk(pid2, "{\"type\":\"function_implements\"," @@ -198,7 +203,8 @@ int main() // After the first dispatch the registry should be healthy. assertRegistryHealthy(); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; printf("Test 2 (restore DB without create_project): PASS\n"); } @@ -214,21 +220,24 @@ int main() writeFixture(dir_a); writeFixture(dir_b); - assert(engine_init(db_path) == 0); - uint64_t pid_a = engine_create_project(dir_a.c_str(), "proj-a"); - assert(pid_a > 0); - char *idx_a = - engine_index_project(pid_a, dir_a.c_str(), nullptr); - assert(idx_a != nullptr && strstr(idx_a, "\"ok\":true")); + g_engine = engine_create(db_path); + CHECK(g_engine != nullptr); + uint64_t pid_a = engine_create_project(g_engine, dir_a.c_str(), + "proj-a"); + CHECK(pid_a > 0); + char *idx_a = engine_index_project(g_engine, pid_a, + dir_a.c_str(), nullptr); + CHECK(idx_a != nullptr && strstr(idx_a, "\"ok\":true")); engine_free_string(idx_a); usleep(150000); - uint64_t pid_b = engine_create_project(dir_b.c_str(), "proj-b"); - assert(pid_b > 0); - assert(pid_b != pid_a); - char *idx_b = - engine_index_project(pid_b, dir_b.c_str(), nullptr); - assert(idx_b != nullptr && strstr(idx_b, "\"ok\":true")); + uint64_t pid_b = engine_create_project(g_engine, dir_b.c_str(), + "proj-b"); + CHECK(pid_b > 0); + CHECK(pid_b != pid_a); + char *idx_b = engine_index_project(g_engine, pid_b, + dir_b.c_str(), nullptr); + CHECK(idx_b != nullptr && strstr(idx_b, "\"ok\":true")); engine_free_string(idx_b); usleep(150000); @@ -246,7 +255,8 @@ int main() "\"subject\":\"compute\"}"); engine_free_string(r_b); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; printf("Test 3 (multi-project sequential verify): PASS\n"); } @@ -259,16 +269,18 @@ int main() writeFixture(proj_dir); uint64_t pid = indexFixture(db_path, proj_dir, "unknown-claim"); char *r = engine_verify_claim( - pid, "{\"type\":\"nonexistent_claim_type\"," - "\"subject\":\"foo\"}"); - assert(r != nullptr); - assert(strstr(r, "claim_type_unsupported") != nullptr); - assert(strstr(r, "unknown claim type") != nullptr); + g_engine, pid, + "{\"type\":\"nonexistent_claim_type\"," + "\"subject\":\"foo\"}"); + CHECK(r != nullptr); + CHECK(strstr(r, "claim_type_unsupported") != nullptr); + CHECK(strstr(r, "unknown claim type") != nullptr); engine_free_string(r); - engine_shutdown(); + engine_destroy(g_engine); + g_engine = nullptr; printf("Test 4 (unknown claim type -> input error): PASS\n"); } printf("\n=== test_verifier_lifecycle PASSED ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/engine/tests/test_verifier_registry.cpp b/engine/tests/test_verifier_registry.cpp index 6d3b680..73c8375 100644 --- a/engine/tests/test_verifier_registry.cpp +++ b/engine/tests/test_verifier_registry.cpp @@ -15,7 +15,7 @@ #include "verify/function_implements_verifier.h" #include "verify/registry.h" -#include +#include "test_check.h" #include #include #include @@ -63,11 +63,11 @@ int main() // ── Test 1: four verifiers registered ──────────────────────── auto names = reg.verifier_names(); - assert(names.size() == 4); - assert(names[0] == "CapabilityVerifier"); - assert(names[1] == "ContractVerifier"); - assert(names[2] == "ArchitectureVerifier"); - assert(names[3] == "FunctionImplementsVerifier"); + CHECK(names.size() == 4); + CHECK(names[0] == "CapabilityVerifier"); + CHECK(names[1] == "ContractVerifier"); + CHECK(names[2] == "ArchitectureVerifier"); + CHECK(names[3] == "FunctionImplementsVerifier"); printf("Test 1 (registry names): PASS\n"); // ── Test 2: CapabilityExists → CapabilityVerifier ──────────── @@ -76,9 +76,9 @@ int main() c.type = ClaimType::CapabilityExists; c.subject = "IncrementalIndex"; Verifier *v = reg.match(c); - assert(v != nullptr); - assert(v->name() == "CapabilityVerifier"); - assert(v->accepts(c) == true); + CHECK(v != nullptr); + CHECK(v->name() == "CapabilityVerifier"); + CHECK(v->accepts(c) == true); printf("Test 2 (CapabilityExists dispatch): PASS\n"); } @@ -88,9 +88,9 @@ int main() c.type = ClaimType::ContractHolds; c.subject = "ThreadSafe"; Verifier *v = reg.match(c); - assert(v != nullptr); - assert(v->name() == "ContractVerifier"); - assert(v->accepts(c) == true); + CHECK(v != nullptr); + CHECK(v->name() == "ContractVerifier"); + CHECK(v->accepts(c) == true); printf("Test 3 (ContractHolds dispatch): PASS\n"); } @@ -102,9 +102,9 @@ int main() c.object = "Service"; c.scope = "Repository"; Verifier *v = reg.match(c); - assert(v != nullptr); - assert(v->name() == "ArchitectureVerifier"); - assert(v->accepts(c) == true); + CHECK(v != nullptr); + CHECK(v->name() == "ArchitectureVerifier"); + CHECK(v->accepts(c) == true); printf("Test 4 (ArchitectureFollows dispatch): PASS\n"); } @@ -117,9 +117,9 @@ int main() c.type = ClaimType::FunctionImplements; c.subject = "foo"; Verifier *v = reg.match(c); - assert(v != nullptr); - assert(v->name() == "FunctionImplementsVerifier"); - assert(v->accepts(c) == true); + CHECK(v != nullptr); + CHECK(v->name() == "FunctionImplementsVerifier"); + CHECK(v->accepts(c) == true); printf("Test 5 (FunctionImplements dispatch): PASS\n"); } @@ -135,9 +135,9 @@ int main() Verifier *cap_v = reg.match(cap); Verifier *contract_v = reg.match(contract); - assert(cap_v != contract_v); - assert(cap_v->accepts(contract) == false); - assert(contract_v->accepts(cap) == false); + CHECK(cap_v != contract_v); + CHECK(cap_v->accepts(contract) == false); + CHECK(contract_v->accepts(cap) == false); printf("Test 6 (type-exclusive accepts): PASS\n"); } @@ -148,7 +148,7 @@ int main() { auto supported = reg.supported_claim_types(); auto all = all_public_claim_types(); - assert(supported.size() == all.size()); + CHECK(supported.size() == all.size()); // Verify each public type is reported as supported. for (ClaimType t : all) { bool found = false; @@ -158,24 +158,24 @@ int main() break; } } - assert(found); + CHECK(found); } printf("Test 7 (supported_claim_types covers all public): PASS\n"); } // ── Test 8: claimTypeWireName round-trips all public types ─── { - assert(std::string(claimTypeWireName( - ClaimType::CapabilityExists)) == - "capability_exists"); - assert(std::string(claimTypeWireName( - ClaimType::ContractHolds)) == "contract_holds"); - assert(std::string(claimTypeWireName( - ClaimType::ArchitectureFollows)) == - "architecture_follows"); - assert(std::string(claimTypeWireName( - ClaimType::FunctionImplements)) == - "function_implements"); + CHECK(std::string( + claimTypeWireName(ClaimType::CapabilityExists)) == + "capability_exists"); + CHECK(std::string(claimTypeWireName( + ClaimType::ContractHolds)) == "contract_holds"); + CHECK(std::string(claimTypeWireName( + ClaimType::ArchitectureFollows)) == + "architecture_follows"); + CHECK(std::string(claimTypeWireName( + ClaimType::FunctionImplements)) == + "function_implements"); printf("Test 8 (claimTypeWireName): PASS\n"); } @@ -187,7 +187,7 @@ int main() { size_t before = reg.verifier_count(); reg.ensureDefaultVerifiers(&g_store, pid); - assert(reg.verifier_count() == before); + CHECK(reg.verifier_count() == before); printf("Test 9 (ensureDefaultVerifiers idempotent): PASS\n"); } @@ -197,19 +197,19 @@ int main() // lifecycle contract: shutdown clears, init re-arms. { reg.clear(); - assert(reg.verifier_count() == 0); + CHECK(reg.verifier_count() == 0); reg.ensureDefaultVerifiers(&g_store, pid); - assert(reg.verifier_count() == 4); + CHECK(reg.verifier_count() == 4); // Dispatch still works after re-arm. Claim c; c.type = ClaimType::FunctionImplements; c.subject = "foo"; Verifier *v = reg.match(c); - assert(v != nullptr); - assert(v->name() == "FunctionImplementsVerifier"); + CHECK(v != nullptr); + CHECK(v->name() == "FunctionImplementsVerifier"); printf("Test 10 (clear -> ensureDefaultVerifiers re-populates): PASS\n"); } printf("\n=== test_verifier_registry PASSED ===\n"); - return 0; + return checkFailures() ? 1 : 0; } diff --git a/install.sh b/install.sh index 316626f..78fca85 100644 --- a/install.sh +++ b/install.sh @@ -67,10 +67,10 @@ echo "Add to PATH:" echo " export PATH=\"\$PATH:$INSTALL_DIR\"" echo "" echo "Then index a project:" -echo " codescope cli index_project '{\"project_path\":\"/path/to/project\"}'" +echo " codescope index-parallel /path/to/project" echo "" echo "Or start MCP server:" echo " codescope" echo "" -echo "Large project? Use the built-in multi-process scheduler:" -echo " codescope index-parallel /path/to/large/project # multi-process, dynamic worker dispatch" +echo "Large project? Tune the built-in multi-process scheduler:" +echo " codescope index-parallel /path/to/large/project --workers 8 --parallel 4" diff --git a/plan/next/DYNAMIC_SCHED_REDESIGN.md b/plan/next/DYNAMIC_SCHED_REDESIGN.md new file mode 100644 index 0000000..9c4a105 --- /dev/null +++ b/plan/next/DYNAMIC_SCHED_REDESIGN.md @@ -0,0 +1,597 @@ +# 动态调度重设计：chunk 级共享队列 + work-stealing + +> **Status**: Implemented — the chunk-level shared queue, work-stealing and static +> CPU policy specified here are what `server/src/scheduler/chunked.rs` runs, and +> those files cite this document by section number. +> **Author**: 调度层改造设计 +> **Date**: 2026-07-19 +> **Scope**: 取代方案 E 的实现现状（模块级分配 + CPU 核心动态借用），改为 chunk 级共享队列 + work-stealing + 静态 CPU。 +> **History**: 本文所取代的方案 E（`builtin-scheduler-design.md`）与它的关联审查 +> （`CODE_REVIEW_DYNAMIC_SCHED_2026-07-19.md`，DS-1~DS-6）在重设计落地后已从仓库移除， +> 下文对它们的引用只作为历史保留。 + +--- + +## 0. 阅读导引 + +| 你想看什么 | 跳到 | +|---|---| +| 为什么现在这套要换 | §1 | +| 新架构长啥样 | §3 | +| chunk 怎么切、队列怎么认领 | §4–§6 | +| 失败文件怎么处理（fail-fast） | §7.3 | +| 崩溃了怎么办 | §8 | +| 我要删哪些旧代码 | §10 | +| **分步改造计划 + 每步验收标准** | §11 | +| 跟原设计文档的偏差 | §13 | + +--- + +## 1. 动机与问题陈述 + +当前实现（方案 E 的落地）有两个**结构性**病灶，靠打补丁修不掉： + +### 1.1 大模块粒度太粗 +模块级分配（`mod.rs:643-653` 把整模块入 `queue`）。一个 5 万文件的模块只能**一个 worker 进程**串行跑（`run_module_worker` 单进程内多线程 parse），其余 worker 即使空闲也帮不上忙。CPU 利用率低、墙钟时间长。 + +### 1.2 CPU 核心动态借用复杂且会死锁（DS-1） +为缓解 1.1，又叠加一层"核心池动态借用"： +- 调度器侧 `SchedShm` 维护 `available_cores`（`shm.rs:76`），worker 前后 `claim_cores`/`release_cores`（`shm.rs:314,347`）。 +- engine 侧 `monitor_thread` 再用 `.detach()` 派 temp worker，经 `grab_cores`/`return_cores`（`engine_index_project.cpp:130,148,806-872`）从同一池子借额外核。 + +链路脆弱点（已验证）：`monitor_thread` 的 temp worker lambda（`engine_index_project.cpp:836-841`）调用 `parse_worker_fn(true)` 若抛异常 → `std::terminate` → worker 崩溃，而崩溃路径上的 `return_cores(1)`（`engine_index_project.cpp:838,846`）**不执行** → 借出的核心永远不归还。多次崩溃后 `available_cores` 归零，`mod.rs:687` 命中 `!has_available()` + `active==0 && queue非空` → **主循环永久 spin，无超时无看门狗**。 + +### 1.3 其他已确认缺陷（来自补审） +- **DS-2**：`dyn_config.rs` 整体死代码——`should_use_dynamic_sched`（`mod.rs:461`）只认 `"1"`，`DynSchedConfig::should_enable` 从未被调用。 +- **DS-3**：每轮 poll 用 `pgrep -P $$` + 每 PID 一次 `ps`（`mod.rs:673-674, get_child_pids:889-902`），100ms 轮询下每秒 ~150 次 fork，违背设计文档 §4.6"零 fork"承诺。 +- **DS-6**：quarantine retry（`mod.rs:780-803`）用静态 1 worker 绕过 `claim_cores`，破坏 CPU accounting。 + +### 1.4 设计命题 +> **任务级已经动态（work-stealing chunk 队列），CPU 级就不需要再动态。** +> 把"核心池借用"整条链路删掉，CPU 在 worker 启动时静态绑核；任务切到 chunk 粒度让空闲 worker 立即认领。复杂度与死锁面大幅缩小，DS-1~DS-6 从设计上消失。 + +--- + +## 2. 设计目标与原则 + +| 目标 | 说明 | 反模式（现状） | +|---|---|---| +| 大模块可并行 | 5 万文件切成 N chunk，任何空闲 worker 可认领 | 整模块独占 1 worker | +| CPU 静态 | worker 启动绑固定核，内部 thread pool 打满 | claim/release/grab/return 核心池 | +| 无 merge | 所有 worker 写同一共享 DB（WAL） | 每模块独立 DB + ID 重映射合并 | +| fail-fast | 单文件解析失败直接标记跳过，不重试空耗 | quarantine 二分重跑整模块 | +| 崩溃可恢复 | CLAIMED 超时不 DONE → 重置 PENDING 让他人接手 | 核心泄漏 + spin | +| 零 fork 监控 | RSS 走 /proc 或 worker 自报 | 每轮 pgrep + ps | + +--- + +## 3. 架构总览 + +``` +Scheduler (主进程) + ├─ Phase 0: 扫模块目录 → 目录聚簇 + 字节加权切分 → chunk A~Z + ├─ Phase 1: 把 ChunkState[] 写入共享内存 ChunkQueue (shm) + ├─ Phase 2: 按 CPU 核启动 W 个 worker（每个静态绑核，互斥无重叠） + │ + │ 每个 worker 循环： + │ claim_next() ──CAS──> 取一个 PENDING chunk + │ parse chunk 内文件 → 构建 IR/图（本 chunk 内实体+边） + │ stream 写共享 DB（WAL，独立连接+批量事务） + │ 失败文件 → file_status(FAILED) 标记，跳过，不重试 + │ 标 chunk DONE（或整块失败标 FAILED） + │ 回到 claim_next + │ + └─ Phase 3: 所有 chunk DONE → resolve 阶段（全局按符号建跨 chunk 边） + → 统一 flush → 输出 SUMMARY + +共享组件： + ChunkQueue (shm) —— chunk 列表 + 状态（无核心池字段） + SharedDB (WAL SQLite) —— graph_nodes/edges/entity/… + file_status +``` + +对比图见对话内「现状 vs 提议 调度架构对比」widget。 + +--- + +## 4. 数据结构 + +### 4.1 `ChunkState`（每个 chunk 一条，存于 shm 定长数组） + +```c +// 布局需 C++/Rust 两侧 repr(C) 一致（沿用 shm.rs 既有的 layout 约定） +#[repr(C)] +struct ChunkState { + status: AtomicU32, // 0=PENDING 1=CLAIMED 2=DONE 3=FAILED + claimer_id: AtomicU32, // worker 索引（0..W-1） + module_id: u32, // 所属模块（用于 quarantine 排除） + file_start: u32, // 文件列表中的起始下标 + file_count: u32, // 本 chunk 文件数 + total_bytes: u64, // 本 chunk 总字节（用于核算负载） + started_at_ms: AtomicU64, // 认领时刻（崩溃恢复超时判定） + finished_at_ms:AtomicU64, // 完成时刻 + failed_files: AtomicU32, // 本 chunk 内失败文件计数（fail-fast 统计） +} +``` + +> 不再含 `available_cores` / `total_cores` / `generation`（核心池字段），仅保留可选的 `mem_usage` 供 RSS 上限参考（可由 worker 自报，无需每轮 fork）。 + +### 4.2 `ChunkQueue`（shm 控制区） + +``` +[ HEADER: magic(u64) | version(u32) | chunk_count(u32) | _pad ] +[ ChunkState; chunk_count ] // 定长数组，容量 = 预计 chunk 数（含 10% 余量） +``` + +- **magic + version**：scheduler 崩溃重启时校验；version 单调递增，旧 shm 视为失效重建（§8.2）。 +- **无 lock-free 队列指针**：worker 用线性扫描 + CAS 找第一个 `PENDING`，chunk 数（~100-250）量级下扫描成本可忽略，远优于维护无锁 ring 的复杂度。 + +### 4.3 共享 DB schema 增量 + +新增一张表用于 fail-fast 标记与查询： + +```sql +CREATE TABLE file_status ( + file_path TEXT PRIMARY KEY, + chunk_id INTEGER, -- 对应 ChunkState 下标 + module_id INTEGER, + status INTEGER, -- 0=OK 1=FAILED 2=SKIPPED + error_code INTEGER, + error_msg TEXT, + marked_at_ms INTEGER +); +``` + +> 现有 13 张表（`merge.rs:54-124` 的 `TABLE_SPECS`）**不再需要 ID 重映射**——因为所有 worker 写同一个共享 DB，node_id 由单一连接序列分配，无跨 DB 冲突。 + +--- + +## 5. 分块算法（目录聚簇 + 字节加权切分） + +### 5.1 步骤 + +1. **扫模块**：对单个模块目录跑 `FilterPolicy::shouldSkipEntry`（真相源），得到 candidate 文件全列表（路径 + 字节数）。 +2. **目录聚簇**：按**前 2 层相对路径**分组（保语义局部性，跨 chunk 符号引用尽量少）。`node_modules`、`target`、`vendor` 等大目录单独标记为"巨型簇"，不与其他目录合并。 +3. **字节加权切分**：遍历簇，累积字节数，达到 `TARGET_BYTES`（建议 8–16 MB 源码，约对应 200–500 文件/块）即切一块；巨型簇强制切成多块（每块不超过 `MAX_BYTES`）。 +4. **输出 chunk list**：写入 `ChunkQueue.chunks[]`，`chunk_count` 置位。 + +### 5.2 伪代码 + +``` +fn plan_chunks(module_dir, files: Vec<(path, bytes)>) -> Vec { + clusters = group_by_prefix(files, depth=2) // 目录聚簇 + chunks = [] + cur = Chunk::new() + for cluster in clusters.sorted_by_size_desc() { // 大簇先切，利于均衡 + for f in cluster.files { + cur.push(f) + if cur.bytes >= TARGET_BYTES || f.is_last_in_cluster { + chunks.push(cur.take()); cur = Chunk::new() + } + } + } + if !cur.empty { chunks.push(cur) } + return chunks +} +``` + +### 5.3 参数 + +| 参数 | 建议值 | 过大 | 过小 | +|---|---|---|---| +| `TARGET_BYTES` | 8–16 MB | 单 chunk 偏载 | CAS 争抢开销 | +| `MAX_BYTES`（巨型簇） | 32 MB | 单 worker 卡死 | — | +| chunk 总数（5万文件） | 100–250 | — | 同上 | +| 切分深度 | 2 层 | 局部性差 | 块太碎 | + +--- + +## 6. 认领协议（CAS work-stealing 状态机） + +### 6.1 状态机 + +``` + claim_next(worker i) parse 完成 +PENDING ──────────────► CLAIMED(i) ─────────────────────► DONE + ▲ │ parse 全部失败 │ + │ └──────────────────────────────────► FAILED + │ 崩溃恢复（watchdog 超时） + └──────── CLAIMED(i) 且 (now - started_at) > TIMEOUT +``` + +- **PENDING → CLAIMED**：`claim_next` 线性扫描找首个 `PENDING`，对该 slot 做 `compare_exchange(PENDING, CLAIMED)`（带 worker_id），成功即认领。多 worker 并发靠 CAS 天然互斥，**无锁、无丢更新**（复用 `shm.rs:314` 既有的 CAS 写法）。 +- **CLAIMED → DONE**：chunk 内文件全部 parse+写库完成。 +- **CLAIMED → FAILED**：chunk 内**所有**文件均失败（极少；见 §7.3 阈值）。 +- **CLAIMED → PENDING（恢复）**：watchdog 判定 `started_at` 超时且未完成 → 重置 PENDING，由其他 worker 接手。 + +### 6.2 `claim_next` 不依赖"告诉 1" + +worker 2 认领 HJKLMN **不需要通知 worker 1**：它直接 CAS 把 HJKLMN 从 `PENDING` 改成 `CLAIMED(2)`。worker 1 下次 `claim_next` 或扫描进度时自然看到这些 slot 已被认领。这就是 work-stealing 的 pull 模型，比 push（worker 间消息）简单且容错。 + +--- + +## 7. 解析与 flush（两阶段 + fail-fast） + +### 7.1 Phase A：各 worker 独立 index 自己的 chunk + +- worker 启动时 `taskset`/`cgroups` 静态绑核（§9 移除核心池后，绑核逻辑移到 worker 拉起处）。 +- 每个 worker 一个 **独立 SQLite 连接** 写共享 DB：`PRAGMA busy_timeout=5000` + WAL，批量事务（每 ~500 文件 commit 一次）。 +- 本 chunk 内 **实体 + 本地边** 正常写 `graph_nodes`/`graph_edges`。 +- **跨 chunk 边**：index 阶段只建"本地实体"与"本地调用"；引用外部实体的调用边先写为 **pending 边**（或暂存待 resolve 阶段补全）。理由：避免 index 阶段跨 chunk 远程查符号，保持 worker 间零耦合。 + +### 7.2 Phase B：resolve 阶段（全局） + +所有 chunk `DONE` 后，跑现有 resolve pipeline（`resolver/pipeline.cpp`）——按符号全局建跨 chunk 边、消 pending。这就是现状的 `index → resolve` 两阶段，只是 index 并行度从"模块级"提到"chunk 级"。**resolve 阶段本身不并行或按符号前缀分片并行**（后续优化，先单线程确保正确）。 + +### 7.3 fail-fast：解析失败文件直接标记（用户明确要求） + +**原则：单文件解析失败立即标记跳过，不重试、不阻塞、不二分重跑。** + +- 单文件 parse 失败（原因：`std::bad_alloc` / tree-sitter 报错 / 读超时 / 编码错误 / 文件被删） → 立即 `INSERT OR REPLACE INTO file_status(... status=1, error_code, error_msg)` → 跳过该文件，chunk 内其余文件继续。 +- **chunk 不因此标 FAILED**：只有"整块文件 100% 失败"才标 `FAILED`（实际几乎不会发生，因为 chunk 内有多个文件）。 +- **阈值熔断**：单 chunk 失败率 > `FAIL_RATE_LIMIT`（建议 50%）→ 标该 chunk `FAILED`，整块进 quarantine（§9 改为"排除这些文件后一次性重跑"，而非二分）。 +- **绝不"一直等待重试"**：不重试、不设退避、不循环。失败即标记，时间立即释放给下一个文件/chunk。 +- 输出 SUMMARY 含 `failed_files` 计数与 `file_status` 表，供用户事后查看哪些文件被跳过。 + +> 对照现状：现有 `quarantine`（`mod.rs:780-803` + `quarantine.rs`）对失败模块做**二分重跑**——反复切文件列表、反复拉 worker，对"单个 malformed 文件"要重跑 log₂(N) 次。fail-fast 把它降为一趟标记 + 一趟排除重跑。 + +### 7.4 flush 统一 + +- worker 边处理边 **stream 写共享 DB**，无需 `merge.rs`。 +- 全部 chunk `DONE` + resolve 完成后，做一次 `PRAGMA wal_checkpoint(TRUNCATE)`（`merge.rs:634-657` 已有实现可复用）收尾成单一 DB 文件。 +- `merge.rs`（整文件 + 13 表 ID 重映射）**删除**（§10）。 + +--- + +## 8. 崩溃恢复 + +| 故障 | 现象 | 恢复 | +|---|---|---| +| worker 进程崩溃 | 其 CLAIMED 的 chunk `started_at` 超时 | scheduler watchdog 把该 chunk 重置 `PENDING`，其他 worker 接手（§6.1） | +| worker 写库中途崩 | 未提交事务 | SQLite WAL 回滚，该 chunk 文件需由接手 worker 重做（file_status 幂等：重做时 `INSERT OR REPLACE`） | +| scheduler 崩溃 | shm 残留 | 重启时校验 `magic`/递增 `version`，旧 shm 视为失效 → 重建队列（已完成的 chunk 凭 `file_status`/`DONE` 标记跳过） | +| 共享 DB 损坏 | WAL 未 checkpoint | 每个 worker 独立连接 + 事务边界，单 worker 崩溃不影响他人已提交数据 | + +> **关键简化**：现状 DS-1（核心泄漏→spin）在新设计下**不存在**——没有核心池、没有 grab/return，worker 崩溃只是 chunk 被重新认领，CPU 静态绑核不受影响。 + +--- + +## 9. quarantine 改造（fail-fast 版） + +1. 模块跑完，扫描 `file_status` 找 `status=1`（FAILED）的文件。 +2. 把这些路径写入 `CODESCOPE_EXCLUDE_PATHS`（engine 已有机制，`engine_index_project.cpp:88-91` 传入 `FilterPolicy`）。 +3. 模块用**静态 1 worker**（或按剩余文件重切 chunk）一次性重跑，**排除失败文件**。 +4. 不再二分（`quarantine::quarantine_module` 的 binary search 逻辑移除或降级为可选）。 +5. 重跑后再次检查 `file_status`：仍有失败则永久标记进 `quarantine_list` 并记入 SUMMARY，不无限重试。 + +> 收益：把"对单个坏文件重跑 log₂(N) 次"降为"标记 1 次 + 排除重跑 1 次"。 + +--- + +## 10. 移除清单（简化收益） + +| 删除 | 位置 | 原因 | +|---|---|---| +| `SchedShm` 核心池字段 + `claim_cores`/`release_cores`/`has_available`/`grab_cores`/`return_cores` | `shm.rs:74,314,347,427` + `engine_index_project.cpp:130,148` | 任务级已动态，CPU 级静态 | +| `monitor_thread` + `.detach()` + temp worker | `engine_index_project.cpp:806-872` | 核心泄漏与 DS-1 根源 | +| `dyn_config.rs`（整体） | `server/src/scheduler/dyn_config.rs` | DS-2 死代码 | +| `merge.rs`（ID 重映射整套） | `server/src/scheduler/merge.rs` | 共享 DB 无需合并 | +| `get_child_pids` + 每轮 `pgrep`/`ps` | `mod.rs:673-674,889-902` | DS-3 零 fork 承诺 | +| quarantine 二分逻辑 | `quarantine.rs` | §9 fail-fast | +| `should_use_dynamic_sched` 的环境位 | `mod.rs:461-466` | 新模型常驻，无需开关 | + +**净效果**：`server/src/scheduler/` 删除约 850 行（shm 核心池 + dyn_config + merge），`engine_index_project.cpp` 删除 monitor 相关约 70 行，新增 `chunk_plan` + `chunk_queue` 约 400 行。复杂度净降。 + +--- + +## 11. 开发任务分解与验收标准 + +> 每步**独立可验证**，前一步通过后再动下一步。验收标准均为可机检/可观测的具体断言。 + +### P0 — 分块器 `chunk_plan` +- **目标**：给定模块目录，产出 chunk 列表（文件下标范围 + 字节数）。 +- **文件**：新增 `server/src/scheduler/chunk_plan.rs`；复用 `discover`/FilterPolicy 文件发现（C++ 侧已有，Rust 侧走 `discover-files` 或等价）。 +- **改动**：实现 §5 算法（目录聚簇 depth=2 + 字节加权切分 + 巨型簇强制切分）。 +- **验收**： + 1. 对 rustc `src`（2,489 candidate 文件）分块，输出 chunk 数 ∈ [100, 250]。 + 2. 每块 `file_count` ∈ [50, 800] 且 `total_bytes` 中位数接近 `TARGET_BYTES`（偏差 < 2×）。 + 3. 同目录文件优先落在同一 chunk（跨 chunk 引用率 < 20%，可用抽样验证）。 + 4. 单测：构造 5 万文件合成树，确认无 chunk 为空、无文件遗漏、字节总和 == 总字节。 +- **规模**：~150 行 Rust。 + +### P1 — `ChunkQueue` 共享内存 + `claim_next` +- **目标**：chunk 列表写入 shm，worker 可无锁认领。 +- **文件**：新增 `server/src/scheduler/chunk_queue.rs`（替代 `shm.rs` 核心池部分）；`ChunkState` 结构 §4.1。 +- **改动**：`magic`/`version` 头 + 定长 `ChunkState[]`；`claim_next(worker_id) -> Option` 用 `compare_exchange(PENDING, CLAIMED)`。 +- **验收**： + 1. 100 并发 `claim_next`（或用 stress 线程）认领 100 chunk，结果恰好覆盖 0..99 各一次，无重复无遗漏。 + 2. `CLAIMED` 后 `claimer_id` 正确写入。 + 3. `version` 递增使旧映射失效（单测：open 旧 shm 应报 version mismatch）。 +- **规模**：~120 行 Rust。 + +### P2 — Worker 改造：吃 chunk 而非模块 +- **目标**：worker 启动后从队列认领 chunk，静态绑核，parse + 写共享 DB。 +- **文件**：`worker.rs`（`run_module_worker` 改为 `run_chunk_worker`）；`engine_index_project.cpp` 入口接收 chunk 文件列表而非整目录。 +- **改动**：worker 拉起时 `taskset -c ` 绑核（CPU 集由 scheduler 按 W 等分算好，无重叠）；内部 `parse_worker_fn` 循环 `claim_next → parse chunk → stream write → DONE → 继续`。 +- **验收**： + 1. 启动 W 个 worker，每个 `/proc//status Cpus_allowed` 落在各自不重叠的 CPU 集。 + 2. 所有 chunk `DONE` 后，共享 DB 中 `graph_nodes` + `graph_edges` 行数 == 单进程全量跑的行数（±1%，容许 fail-fast 跳过差异）。 + 3. 中途 `kill -9` 一个 worker，其余 worker 最终仍把所有 chunk 跑完（崩溃恢复 §8）。 +- **规模**：~200 行 Rust + ~60 行 C++。 + +### P3 — fail-fast 文件标记 +- **目标**：单文件解析失败立即标记，不重试。 +- **文件**：`engine_index_project.cpp` parse 循环；`server/src/scheduler/chunk_queue.rs` 的 `file_status` 写入（经 engine FFI 或 worker 直写）；DB 加 `file_status` 表（§4.3）。 +- **改动**：catch 单文件异常 → 写 `file_status(status=1, error_code, error_msg)` → `continue`；chunk 失败率 > `FAIL_RATE_LIMIT` 才标 `FAILED`。 +- **验收**： + 1. 注入 3 个 malformed UTF-8 的 `.rs` 文件进模块，跑索引，**不触发 quarantine 二分重跑**，总耗时接近无坏文件基线（偏差 < 5%）。 + 2. `file_status` 表中这 3 个文件 `status=1`，且 `graph_nodes` 不含这 3 文件实体。 + 3. SUMMARY 含 `failed_files: 3`。 + 4. 单测：构造 parse 抛异常的 mock 文件，确认 `failed_files` 计数准确、chunk 不误标 FAILED（只要非 100% 失败）。 +- **规模**：~80 行 C++ + ~40 行 Rust。 + +### P4 — Scheduler 主循环重写 +- **目标**：预分块 → 启动 worker → worker 自认领 → 全 DONE → resolve → flush。 +- **文件**：`mod.rs`（`index_parallel_dynamic` 重写为 `index_parallel_chunked`）；删除 `index_parallel_dynamic` 旧逻辑（`:489-884`）。 +- **改动**：删核心池轮询（`has_available`/`claim_cores`/`release_cores`/`POLL_INTERVAL` spin）；改为：P0 分块 → P1 写队列 → 按 CPU 核启 W 个 `run_chunk_worker` → `join` → resolve 阶段 → checkpoint flush → SUMMARY。 +- **验收**： + 1. `codescope index-parallel --workers 14` 跑通，5 万文件墙钟 ≤ 单进程基线 / (并行度 × 0.7)（即并行有效）。 + 2. 无 `available_cores` 死锁：连续跑 10 次大项目，无一次 hang。 + 3. 输出 SUMMARY 结构与现状兼容（`ok/modules/success/fail/total_nodes/...`）。 +- **规模**：~250 行 Rust（替换旧 ~400 行）。 + +### P5 — 删除旧组件 +- **目标**：落实 §10 移除清单。 +- **文件**：`shm.rs`（删核心池，或整文件删后由 `chunk_queue.rs` 替代）、`dyn_config.rs`、`merge.rs`、`engine_index_project.cpp:806-872` monitor、`mod.rs:673-674,889-902 get_child_pids`、`mod.rs:461-466 should_use_dynamic_sched`。 +- **改动**：逐一删除并修复编译引用（`grep -rn claim_cores|release_cores|merge_module_dbs|should_use_dynamic_sched` 应为空）。 +- **验收**： + 1. `cargo build` 零警告零错误（移除 dead_code 后）。 + 2. `grep -rn "merge_module_dbs\|claim_cores\|grab_cores\|monitor_thread\|should_use_dynamic_sched"` 全仓为 0 命中。 + 3. 二进制体积下降（删 ~850+70 行）。 +- **规模**：纯删除 ~920 行。 + +### P6 — quarantine fail-fast 改造 +- **目标**：失败模块"排除坏文件后一次性重跑"，非二分。 +- **文件**：`quarantine.rs`（重写 `quarantine_module`）；`mod.rs` P4 主循环接入。 +- **改动**：读 `file_status` 取失败文件 → 写 `CODESCOPE_EXCLUDE_PATHS` → 静态 1 worker 重跑（排除后）→ 再查 `file_status`，仍失败则永久标记进 quarantine_list。 +- **验收**： + 1. 含 1 个崩溃文件的模块：跑索引 → 该文件进 `file_status(FAILED)` → 重跑排除它 → 模块 `exit=0`，QUARANTINE 列表含该文件，总重跑次数 == 1（非二分多次）。 + 2. 重跑后的 DB 不含崩溃文件实体、不含其他文件损失。 +- **规模**：~120 行 Rust。 + +### P7 — resolve 阶段对齐 +- **目标**：全 chunk DONE 后全局建跨 chunk 边。 +- **文件**：`resolver/pipeline.cpp` 接入点；scheduler 在 chunk 全 DONE 后触发。 +- **改动**：pending 边在 resolve 阶段补全；resolve 输入为共享 DB（已含所有 chunk 实体）。 +- **验收**： + 1. 跨 chunk 调用（A chunk 函数调 B chunk 函数）在 `graph_edges` 中存在正确边。 + 2. 跨 chunk 边数与单进程全量基线一致（偏差 < 1%）。 +- **规模**：~60 行 Rust 接线 + 复用现有 resolver。 + +### P8 — benchmark 与回归 +- **目标**：性能与正确性门禁。 +- **文件**：`benchmarks/`、`tests/`。 +- **验收**： + 1. 大项目（≥5万文件）索引墙钟 ≤ 现状（模块级）基线 × 0.6（理想并行加速）。 + 2. 节点数/边数/文件数 == 单进程权威基线（允许 fail-fast 跳过差异，且差异文件须全部在 `file_status` 中可追溯）。 + 3. 内存峰值 ≤ 现状（无共享队列额外爆炸）。 + 4. 现有一组 benchmark 全绿。 + +--- + +## 12. 风险与缓解 + +| 风险 | 影响 | 缓解 | +|---|---|---| +| chunk 切太碎 → CAS 争抢 | 调度开销上升 | §5.3 参数下限（≥50 文件/块）；实测调 `TARGET_BYTES` | +| chunk 切太大 → 负载不均 | 长尾 worker | 目录聚簇 + 字节加权；巨型簇强制切分 | +| 跨 chunk 边在 index 阶段丢失 | resolve 阶段必须补全 | §7.1 pending 边 + §7.2 全局 resolve；P7 验收兜底 | +| worker 崩溃导致 chunk 文件重做 | 少量重复工作 | SQLite 事务边界 + `file_status` 幂等；watchdog 超时合理（建议 2× 单 chunk 平均耗时） | +| resolve 单线程瓶颈 | 大项目尾部慢 | 先单线程保正确；后续按符号前缀分片并行（不在本期） | +| 静态绑核在容器/CI 受限 | taskset 不可用 | 绑核失败降级为不绑（仍正确，仅少隔离收益），记 warning | + +--- + +## 13. 与 `builtin-scheduler-design.md` 的偏差 + +| 原设计（方案 E 落地） | 本设计 | 偏差原因 | +|---|---|---| +| §4.3 每模块 1 worker | chunk 级 work-stealing，多 worker 共享队列 | 解决大模块粒度粗（§1.1） | +| §4.5 每模块独立 DB + merge.rs ID 重映射 | 单一共享 DB（WAL），无 merge | 消除 ID 重映射复杂度与 bug 面 | +| §4.6 "零 fork" 用 /proc | worker 自报 RSS / 删监控 fork | 现状 DS-3 退回 pgrep+ps，本设计直接删除该路径 | +| CPU 核心动态借用（shm 核心池） | CPU 静态绑核 | 任务级已动态，CPU 级无需动态（§1.4），消除 DS-1 | +| quarantine 二分重跑 | fail-fast 标记 + 排除后一次性重跑 | 用户明确要求"失败文件直接标记，不空耗重试" | + +> 本设计**保留**原设计的核心理念：调度器只管调度、文件发现/解析/建图归 worker 内部 FilterPolicy（单一真相源，§4.1）。 + +--- + +## 14. 不在本期 scope + +- engine 内部 parse/buildGraph 性能优化（另见 `membulk_optimization_plan.md`）。 +- resolve 阶段按符号前缀并行化（P7 之后优化）。 +- UI/CLI 对 chunk 级进度的可视化（shm 已含 `started_at`/`finished_at`，可暴露给 MCP 工具）。 +- FilterPolicy skip 规则调整。 + +--- + +## 15. I/O 瓶颈优化：JSON/CSV 中间文件批量导入方案 + +> **Status**: Design / Proposal +> **Date**: 2026-07-19 +> **Scope**: 解决大模块（如 Linux 内核 `drivers`，31,708 文件）索引时 SQLite 写入成为瓶颈的问题。仿照 codebase-memory-mcp 的"先内存积累，后一次性批量写入"思路，但用临时 JSON 文件换内存，避免 OOM。 + +### 15.1 问题背景 + +实测大模块（如 `drivers`，31,708 文件，~15 万节点）的索引耗时分布： + +| 阶段 | 耗时 | 占比 | 说明 | +|------|------|------|------| +| 解析（tree-sitter） | ~20s | 9% | CPU 密集，8 核打满 | +| **SQLite 写入**（逐行 INSERT） | **~180s** | **84%** | 15 万次 INSERT，每次走 SQL 解析 + B-tree 再平衡 + WAL 刷盘 | +| 解析器（resolver） | ~10s | 5% | 内存操作 | +| buildGraph | ~5s | 2% | CSR 构建 | +| **合计** | **~215s** | | | + +**瓶颈不在解析，在写入。** 每个文件解析完立即 INSERT，SQLite 的 B-tree 页分裂 + WAL 追加成为主要开销。 + +### 15.2 方案对比 + +| 方案 | 原理 | 优点 | 缺点 | +|------|------|------|------| +| **A：当前方案** | 解析完一个文件就 INSERT | 内存低 | I/O 瓶颈严重 | +| **B：codebase-memory-mcp** | 全内存 graph_buffer，最后直接写 SQLite 页 | 写入极快（跳过 SQL 层） | 内存占用大，大项目可能 OOM | +| **C：JSON 中间文件 ← 本方案** | 解析写入 JSON 文件，解析完后批量导入 SQLite | 内存友好，磁盘换内存 | 多一次 JSON 序列化/反序列化 | + +### 15.3 JSON 中间文件方案设计 + +#### 15.3.1 流程 + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Phase 1: 解析 + 写 JSON（并行，每个 worker） │ +│ │ +│ 解析文件 → 序列化节点/边 → 追加到 /tmp/xxx_nodes.json │ +│ → 追加到 /tmp/xxx_edges.json │ +│ （fprintf 顺序追加，~GB/s 级写入，无 SQL 开销） │ +│ │ +│ Phase 2: 批量导入 SQLite（串行，一次完成） │ +│ │ +│ BEGIN TRANSACTION │ +│ 读 JSON → 预排序（按 qualified_name）→ 批量 INSERT │ +│ COMMIT │ +│ │ +│ Phase 3: 解析器 + buildGraph（同现有流程） │ +│ │ +│ resolver 从 SQLite 读回数据进行引用解析 │ +│ buildGraph 构建 CSR + 边 │ +└──────────────────────────────────────────────────────────────┘ +``` + +#### 15.3.2 JSON 格式 + +```json +// /tmp/codescope_bulk_drivers_nodes.json +// 每行一个 JSON 对象（JSON Lines 格式，无需完整数组） +{"id":1,"name":"find_pa","type":0,"file":"...","line":39,"col":0} +{"id":2,"name":"pal_init","type":0,"file":"...","line":66,"col":0} +... + +// /tmp/codescope_bulk_drivers_edges.json +{"id":1,"src":1,"tgt":2,"type":1,"file":"...","line":70} +{"id":2,"src":1,"tgt":3,"type":3,"file":"...","line":75} +... +``` + +选择 JSON Lines（每行一个对象）而非完整 JSON 数组，原因： +- 无需在内存中维护完整数组，逐行写入即可 +- 解析时也可逐行读，内存可控 +- 天然支持流式处理 + +#### 15.3.3 批量导入优化 + +```sql +-- 预排序 + 单事务批量导入 +BEGIN TRANSACTION; + +-- 关闭同步、WAL、页面缓存 +PRAGMA synchronous = OFF; +PRAGMA journal_mode = MEMORY; +PRAGMA cache_size = -1048576; -- 1 GB page cache + +-- 批量插入节点（预排序后 B-tree 构建更高效，减少页分裂） +INSERT INTO graph_nodes (id, name, node_type, file_path, ...) VALUES + (1, 'find_pa', 0, '...', ...), + (2, 'pal_init', 0, '...', ...), + ...; + +-- 批量插入边 +INSERT INTO graph_edges (id, source_id, target_id, edge_type, ...) VALUES + (1, 1, 2, 1, ...), + (2, 1, 3, 3, ...), + ...; + +COMMIT; + +-- 导入完成后恢复同步 +PRAGMA synchronous = NORMAL; +PRAGMA journal_mode = WAL; +``` + +关键优化点： +- **预排序**：写入 JSON 前按 `qualified_name` 排序，让 B-tree 页顺序写入，避免随机页分裂 +- **单事务**：整个模块一个事务，每行 INSERT 不需要独立事务开销 +- **关闭同步**：导入期间关闭 fsync，导入完成后恢复 +- **大 page cache**：给 SQLite 分配足够缓存，让 B-tree 构建在内存中完成 + +#### 15.3.4 解析器（resolver）数据回读 + +解析器需要从 SQLite 读回节点/边来解析引用（call graph、import 等）。有两种方式： + +**方式 A：导入完再跑解析器** +``` +Phase 1: 解析 + 写 JSON → 所有 worker 并行 +Phase 2: 批量导入 SQLite → 串行 +Phase 3: 解析器从 SQLite 读回 → 同现有流程 +``` + +优点：无需修改解析器逻辑 +缺点：解析器只能串行，失去并行机会 + +**方式 B：JSON 直接喂解析器** +``` +Phase 1: 解析 + 写 JSON → 所有 worker 并行 +Phase 1b: 解析器从 JSON 读回（内存中） → 并行，不经过 SQLite +Phase 2: 批量导入 SQLite（最终结果） → 串行 +``` + +优点：解析器可并行，不依赖 SQLite +缺点：需要修改解析器支持 JSON 输入 + +**建议：先走方式 A，验证批量导入效果后，再优化到方式 B。** + +### 15.4 混合策略 + +小模块用当前 `membulk` 路径（直接入库），大模块自动切到 JSON 中间文件路径： + +``` +阈值：candidate_files > 5000 && 预计节点数 > 50000 +``` + +```rust +fn should_use_bulk_json(files: u64, nodes_estimate: u64) -> bool { + // 小模块直接入库（membulk 路径） + // 大模块走 JSON 中间文件 + files > 5000 && nodes_estimate > 50000 +} +``` + +### 15.5 性能预估 + +以 `drivers` 模块（31,708 文件，~15 万节点）为例： + +| 阶段 | 当前（INSERT） | JSON 中间文件 | 加速比 | +|------|-------------|-------------|-------| +| 解析 | ~20s | ~20s | 1x | +| **写入** | **~180s** | **~0.5s** | **360x** | +| 批量导入 | — | ~5s | 新增 | +| 解析器 | ~10s | ~10s | 1x | +| buildGraph | ~5s | ~5s | 1x | +| **合计** | **~215s** | **~40.5s** | **~5.3x** | + +### 15.6 实现计划 + +| 步骤 | 内容 | 工作量 | +|------|------|--------| +| P1 | 在 `engine_index_project` 中添加 JSON Lines 写入路径 | ~2 天 | +| P2 | 实现批量导入器（预排序 + 单事务 + PRAGMA 优化） | ~1 天 | +| P3 | 集成到调度器：自动选择 membulk/JSON 路径 | ~0.5 天 | +| P4 | 解析器支持 JSON 回读（可选） | ~2 天 | +| P5 | 清理临时 JSON 文件 | ~0.2 天 | + +### 15.7 风险与注意事项 + +- **JSON 文件大小**：`drivers` 级别 ~30 MB，完全可接受 +- **磁盘空间**：临时 JSON 文件在 `/tmp`，模块处理完后自动清理 +- **断点续传**：JSON 写一半挂了，下次直接读 JSON 继续，无需重新解析 +- **内存**：读 JSON 回解析器时逐行读，内存可控，无 OOM 风险 +- **排序**：写入时按 `qualified_name` 或 `id` 排序，批量导入时 B-tree 构建更高效 diff --git a/plan/rules/code_rules.md b/plan/rules/code_rules.md new file mode 100644 index 0000000..e61efff --- /dev/null +++ b/plan/rules/code_rules.md @@ -0,0 +1,102 @@ +**Here is a clean, professional English Coding Standards document** for your project. + +--- + +**CodeScope Coding Standards** +**C++ & Rust** +**Version 1.0** + + +所有公共函数和方法必须有完整的注释，包括参数、返回值、异常等。 +所有的错误禁止静默处理，必须显示处理，而且有完整的错误追踪链，能定位到什么模块的，什么方法。 +禁止用git commit + + +FFI： 规范 + +绝对不要“单条符号调用一次 FFI”：如果 C++ 扫出一个符号就调用一次 Rust 导出的 Insert 接口，几千次 FFI 的上下文切换开销会直接毁掉 ms 级的目标。 + +块级传递（Chunking）或回调（Callback）： + +方案 A：C++ 扫描完一个文件，将该文件所有的 symbols 压入一个连续的 C-Style 结构体数组（POD struct array），一次性通过指针传给 Rust，由 Rust 侧的 rusqlite 开启事务进行 std::vec 批量插入。 + +方案 B：C++ 侧直接静态链接 sqlite3（或者通过相同的 libsqlite 动态链接），直接由 C++ 写入事实表。Rust 侧只负责传入 db_path 和任务指令，并负责读取展示。从你的 Mermaid 架构图来看，方案 B（C++ Core 直接写 FACTS 表）是最符合你当前设计的，也能压榨出极限的 I/O 性能。 + + + +### 1. General Principles + +- **File Size Limit**: No source file may exceed **1000 lines** (including blank lines and comments). Split files when necessary. +- **Comments**: All comments must be written in **English**. Use clear, concise language. Include purpose, parameters, return values, and any important invariants for complex functions. +- **Code Style**: + - **Rust**: Follow official `rustfmt` + `cargo clippy --all-targets -- -D warnings`. + - **C++**: Follow Google C++ Style Guide + `clang-format`. +- **Language Version**: Rust 2024 Edition, C++23. + +### 2. Memory Management (Strict Rules) + +**Rust**: +- Prefer safe abstractions: `Box`, `Vec`, `String`, `Rc/Arc`, `Cow`, etc. +- Avoid `unsafe` unless absolutely necessary (mainly FFI boundaries). +- Always use `reserve()` for `Vec`/`String` when size is predictable. +- All `unsafe` blocks must have a detailed safety comment explaining invariants. + +**C++**: +- **Never** use raw `new`/`delete` outside FFI boundaries. +- Use RAII: `std::unique_ptr`, `std::shared_ptr`, `std::vector`, `std::string`. +- All resources must be managed by RAII types. + +**General**: +- Prefer moving ownership over copying. +- Avoid global/static mutable state unless strictly necessary. + +### 3. FFI Boundaries (Most Critical) + +All cross-language boundaries (Rust ↔ C++) must follow these strict rules: + +- Use `extern "C"` and `#[no_mangle]` for FFI functions. +- **Ownership Transfer Rules**: + - Rust-allocated memory passed to C++ must be explicitly freed by C++ using a provided `free_xxx()` function. + - C++-allocated memory passed to Rust must be wrapped in `Box` or explicitly transferred. +- **Data Types**: + - Use `CString` / `CStr` for strings. + - Use opaque pointers (`*mut c_void` or boxed structs) for complex objects. + - Never pass raw Rust structs by value across FFI. +- **Error Handling**: + - FFI functions should return `int` error codes (0 = success). + - Use `#[repr(C)]` for error structs if needed. +- **Safety Documentation**: + - Every FFI function must have a comment block explaining memory ownership, lifetime, and thread safety. + +**Example (Rust side)**: +```rust +/// # Safety +/// The caller must ensure `ptr` is valid and call `free_xxx` when done. +#[no_mangle] +pub unsafe extern "C" fn create_xxx() -> *mut c_void { ... } +``` + +### 4. Testing Requirements + +- Every public function / module must have corresponding tests. +- **Boundary & Edge Cases** are **mandatory**: + - Null / empty inputs + - Maximum / minimum values + - Error conditions + - Concurrency / race conditions (for thread-safe code) + - FFI boundary cases (ownership transfer, double-free prevention) +- Use property-based testing where appropriate (`proptest` in Rust). +- Integration tests must use real dependencies (e.g., real PostgreSQL for storage modules). + +### 5. Additional Rules + +- **No magic numbers** — use named constants. +- **Error handling**: Prefer `Result` in Rust and `std::expected` / error codes in C++. +- **Dependencies**: Keep them minimal and well-documented. +- **Performance**: Profile critical paths. Use `cargo flamegraph` / `perf` regularly. + +--- + +**Enforcement**: +- Use CI to check file length, Clippy, clang-format, and test coverage. +- All new code must be reviewed against this standard. diff --git a/plan/rules/relation_contract.md b/plan/rules/relation_contract.md new file mode 100644 index 0000000..674ae4e --- /dev/null +++ b/plan/rules/relation_contract.md @@ -0,0 +1,120 @@ +# CodeScope Relation Type Contract + +> Status: **Authoritative** +> Owners: engine + store + query modules +> Last updated: 2026-07-31 + +## 1. Purpose + +This document is the single source of truth for how SQLite `relation.type` +integers map to LadybugDB `CALLS` / `RELATES` tables and to the +`graph::EdgeType` enum. Production code MUST NOT branch on raw integer +thresholds such as `rtype >= 4`. It MUST call the helpers declared in +`engine/src/graph/graph_types.h`. + +## 2. Canonical Type Table + +| `relation.type` | `graph::EdgeType` | LadybugDB table | Semantic | +|-----------------|-----------------------|-----------------|-------------------------------------------| +| 0 | `References` | RELATES | Non-call symbol reference | +| 1 | `Calls` | **CALLS** | Function / method invocation | +| 2 | `Defines` | RELATES | Definition relationship | +| 3 | `Contains` | RELATES | Structural containment (class → method) | +| 4 | `Imports` | RELATES | Import / include | +| 5 | `Inherits` | RELATES | Class inheritance | +| 6 | `UsesType` | RELATES | Entity uses a type | +| 7 | `HasType` | RELATES | Type definition has fields/methods | +| any other | (mapped to References)| RELATES | Unknown kinds — safe non-call fallback | + +The `RELATES` table retains its `edge_type` column so callers can +distinguish References from Defines, Contains, Imports, etc. + +## 3. C++ API + +Declared in `engine/src/graph/graph_types.h`: + +```cpp +namespace graph { + +EdgeType relationTypeFromInt(int rtype); +int relationTypeToInt(EdgeType type); + +bool isCallsEdge(int rtype); +bool isCallsEdge(EdgeType type); + +bool isRelatesEdge(int rtype); +bool isRelatesEdge(EdgeType type); + +} // namespace graph +``` + +- `isCallsEdge(rtype)` returns `true` iff `rtype == 1` (`EdgeType::Calls`). +- `isRelatesEdge(rtype)` is the logical complement. +- `relationTypeFromInt` maps unknown integers to `EdgeType::References` + so an out-of-range value can never accidentally pollute `CALLS`. + +## 4. LadybugDB Mapping Rules + +1. Only `EdgeType::Calls` is written to the `CALLS` table. +2. Every other typed relation is written to `RELATES` and keeps its + `edge_type` column. +3. The split point in `store_graph_compiler.cpp` MUST use + `graph::isCallsEdge(rtype)` — never a raw integer comparison. +4. `getCallers` / `getCallees` Cypher (Step 1 onward) MUST restrict to + `CALLS` and assert `r.edge_type = 1`. + +## 5. Legacy `graph_edges.edge_type` Numbering (DEPRECATED) + +The legacy `graph_edges` table uses a different numbering: +- `1` = `call_graph` +- `3` = `symbol_reference` + +The legacy fallback path in `store_graph_compiler.cpp` is being phased +out and only runs when `entity`/`relation` tables are empty. New code +MUST NOT use this numbering; it MUST go through the canonical +`relation.type` contract above. + +## 6. Accuracy Metric Definitions + +The accuracy benchmark (Step 2 onward) uses these definitions, recorded +here so they cannot drift between implementers: + +- **TP (True Positive)**: an actual CALLS edge that is in `expected_calls`. +- **FP (False Positive)**: an actual CALLS edge that is not in + `expected_calls` and not in `allowed_unresolved` / `external_calls`. +- **FN (False Negative)**: an `expected_calls` edge missing from actual. +- **Precision** = `TP / (TP + FP)`. +- **Recall** = `TP / (TP + FN)`. +- **F1** = `2 * P * R / (P + R)`. +- **contamination_rate** = non-Calls edges / total edges returned by + callers/callees. Must be 0%. +- **duplicate_rate** = duplicate `(project_id, source_id, target_id, + type)` rows / total `relation` rows. Must be 0%. +- **sqlite_ladybug_diff** = `|{CALLS in SQLite relation(type=1)} △ + {CALLS in LadybugDB}|`. Must be 0. + +## 7. Fixture Identity Rules + +To keep fixtures portable and immune to database ID drift, every +fixture entity is identified by the semantic tuple: + +``` +file_path + qualified_name + kind + start_row +``` + +Database IDs (`entity.id`, `relation.id`) MUST NOT appear in fixture +ground truth. Step 7 introduces a stable entity UID API; until then +the semantic tuple above is the canonical identity. + +## 8. Baseline Snapshot + +The baseline snapshot for `dev@eca4bd0` is generated by +`engine/tests/test_accuracy_baseline.cpp` (added in Step 0). It records: + +- `relation` row counts grouped by `type` (0–7). +- LadybugDB `CALLS` / `RELATES` row counts grouped by `edge_type`. +- Duplicate `(project_id, source_id, target_id, type)` count. +- Pass/fail status of the 13 existing accuracy tests. + +The baseline is intentionally recorded BEFORE the Step 1 query +tightening so subsequent steps can produce before/after diffs. diff --git a/plan/rules/skills.md b/plan/rules/skills.md new file mode 100644 index 0000000..638f31d --- /dev/null +++ b/plan/rules/skills.md @@ -0,0 +1,63 @@ +Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed. + +**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment. + +## 1. Think Before Coding + +**Don't assume. Don't hide confusion. Surface tradeoffs.** + +Before implementing: +- State your assumptions explicitly. If uncertain, ask. +- If multiple interpretations exist, present them - don't pick silently. +- If a simpler approach exists, say so. Push back when warranted. +- If something is unclear, stop. Name what's confusing. Ask. + +## 2. Simplicity First + +**Minimum code that solves the problem. Nothing speculative.** + +- No features beyond what was asked. +- No abstractions for single-use code. +- No "flexibility" or "configurability" that wasn't requested. +- No error handling for impossible scenarios. +- If you write 200 lines and it could be 50, rewrite it. + +Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify. + +## 3. Surgical Changes + +**Touch only what you must. Clean up only your own mess.** + +When editing existing code: +- Don't "improve" adjacent code, comments, or formatting. +- Don't refactor things that aren't broken. +- Match existing style, even if you'd do it differently. +- If you notice unrelated dead code, mention it - don't delete it. + +When your changes create orphans: +- Remove imports/variables/functions that YOUR changes made unused. +- Don't remove pre-existing dead code unless asked. + +The test: Every changed line should trace directly to the user's request. + +## 4. Goal-Driven Execution + +**Define success criteria. Loop until verified.** + +Transform tasks into verifiable goals: +- "Add validation" → "Write tests for invalid inputs, then make them pass" +- "Fix the bug" → "Write a test that reproduces it, then make it pass" +- "Refactor X" → "Ensure tests pass before and after" + +For multi-step tasks, state a brief plan: +``` +1. [Step] → verify: [check] +2. [Step] → verify: [check] +3. [Step] → verify: [check] +``` + +Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification. + +--- + +**These guidelines are working if:** fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes. diff --git a/codescope-index.sh b/scripts/legacy/codescope-index.sh similarity index 100% rename from codescope-index.sh rename to scripts/legacy/codescope-index.sh diff --git a/codescope-parallel.sh b/scripts/legacy/codescope-parallel.sh similarity index 100% rename from codescope-parallel.sh rename to scripts/legacy/codescope-parallel.sh diff --git a/server/.cargo/config.toml b/server/.cargo/config.toml index da8790f..ea7f670 100644 --- a/server/.cargo/config.toml +++ b/server/.cargo/config.toml @@ -13,8 +13,17 @@ jobs = -1 # Incremental compilation (enabled by default in dev profile) incremental = true -# Rustflags for faster builds -[target.'cfg(debug_assertions)'] +# Rustflags for faster builds. +# +# `-C target-cpu=native` is only sound when the target IS the host: on an Apple +# Silicon host cross-compiling to Windows it resolves to `apple-m3`, and +# rustc-LLVM then rejects every crate with "64-bit code requested on a subtarget +# that doesn't support it" — which is why +# `cargo build --release --target x86_64-pc-windows-gnu` (the macOS → Windows +# cross-compile job in `.github/workflows/dev.yml`) could never build. The flag +# is therefore kept out of Windows targets, which are the only ones this project +# cross-compiles to. +[target.'cfg(all(debug_assertions, not(target_os = "windows")))'] rustflags = ["-C", "target-cpu=native"] # Profile-specific optimizations diff --git a/server/Cargo.toml b/server/Cargo.toml index 0c0ca4a..a414383 100644 --- a/server/Cargo.toml +++ b/server/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "codescope" -version = "0.2.6" +version = "0.2.7" edition = "2024" [[bin]] @@ -13,3 +13,17 @@ serde_json = "1" walkdir = "2" once_cell = "1" libc = "0.2" + +# Windows-only: the shared-file mapping in src/scheduler/mapped_file.rs needs +# CreateFileMappingW / MapViewOfFile, which the `libc` crate does not expose on +# Windows. `walkdir` already pulls windows-sys in as a transitive dependency, so +# this adds no new crate to the lockfile — only the Win32 features we call. +[target.'cfg(windows)'.dependencies] +windows-sys = { version = "0.61", features = [ + "Win32_Foundation", + "Win32_Security", # CreateFileMappingW takes a *const SECURITY_ATTRIBUTES + "Win32_System_Memory", +] } + +[dev-dependencies] +proptest = "1.11.0" diff --git a/server/build.rs b/server/build.rs index 5af032f..721c4c8 100644 --- a/server/build.rs +++ b/server/build.rs @@ -118,14 +118,16 @@ fn main() { // Disable tests (they use POSIX APIs not available on Windows) cmake_args.push("-DBUILD_TESTS=OFF".to_string()); } - // Use Ninja generator if available (faster parallel builds) - if std::process::Command::new("ninja") - .arg("--version") - .output() - .is_ok() - { + // Use the generator the build tree already uses; only a fresh tree gets to + // pick one. Pushing `-G Ninja` merely because ninja is on PATH is a fatal + // CMake error when build_dir was previously configured with another + // generator ("does not match the generator used previously") — the same + // mismatch the Makefile had, triggered whenever ninja's presence changes + // (installed later, different PATH, restored CI cache). + if let Some(generator) = resolve_generator(&build_dir) { + eprintln!("build.rs [{}]: cmake generator = {}", target_os, generator); cmake_args.push("-G".to_string()); - cmake_args.push("Ninja".to_string()); + cmake_args.push(generator); } if !sdk_arg.is_empty() { cmake_args.push(sdk_arg); @@ -178,6 +180,38 @@ fn main() { } } +/// Resolve the CMake generator for `build_dir`. +/// +/// A generator is a property of the build tree: reconfiguring an existing +/// tree with a different `-G` fails outright. So the generator recorded in +/// the tree's `CMakeCache.txt` wins. Only a tree without a cache may pick +/// one — Ninja when it actually runs, otherwise `None` so CMake applies its +/// own platform default (`Unix Makefiles` on macOS/Linux, MSBuild/VS on +/// Windows). +fn resolve_generator(build_dir: &str) -> Option { + let cache = format!("{}/CMakeCache.txt", build_dir); + if let Ok(contents) = std::fs::read_to_string(&cache) { + for line in contents.lines() { + if let Some(value) = line.strip_prefix("CMAKE_GENERATOR:INTERNAL=") { + let value = value.trim(); + if !value.is_empty() { + return Some(value.to_string()); + } + } + } + } + let ninja_works = Command::new("ninja") + .arg("--version") + .output() + .map(|out| out.status.success()) + .unwrap_or(false); + if ninja_works { + Some("Ninja".to_string()) + } else { + None + } +} + /// Platform-default compiler detection. fn platform_default_compiler(target_os: &str) -> (String, String) { match target_os { diff --git a/server/src/discover.rs b/server/src/discover.rs index 4f386ec..0e8b689 100644 --- a/server/src/discover.rs +++ b/server/src/discover.rs @@ -6,7 +6,7 @@ //! tool (which is designed for project-overview UI display, not for //! dispatching workers). //! -//! See `builtin-scheduler-design.md` §4.2 for the rationale. The skip +//! The skip //! rules mirror the C++ `FilterPolicy::shouldSkipEntry` so the file list //! we emit matches what the worker will actually index. @@ -14,138 +14,72 @@ use serde_json::{Value, json}; use std::path::Path; use walkdir::WalkDir; -/// Built-in skip dirs matching `FilterPolicy::normal_skip_dirs_`. -/// Applied to EVERY path component (any depth) so nested `target/` and -/// `node_modules/` directories are pruned during the walk. -fn is_skip_dir(name: &str) -> bool { - let lower = name.to_lowercase(); - matches!( - lower.as_str(), - ".git" - | ".svn" - | ".hg" - | ".bzr" - | "node_modules" - | ".venv" - | "venv" - | "env" - | ".env" - | "__pycache__" - | ".pytest_cache" - | ".mypy_cache" - | ".cache" - | ".codescope" - | "target" - | "build" - | "dist" - | ".next" - | ".turbo" - | ".direnv" - | "vendor" - | "third_party" - | "third-party" - | ".bundle" - | ".gem" - | "go_pkg" - | "pkg" - ) +/// Whether the indexer would skip this path. +/// +/// Delegates to the engine's `FilterPolicy` — the single authority — instead +/// of keeping a copy of the skip rules. The copy is what let two defects +/// through in one review: a `.gitignore`d top-level directory was still +/// reported as a module (the copy read no ignore file at all, so the scheduler +/// dispatched a worker that indexed zero files), and the extension list below +/// advertised `.zig` while the engine has no Zig support. `root` is the +/// project root `rel_path` is relative to, which is the shape the indexer +/// passes, so project-anchored rules can match. +fn skipped_by_engine(root: &str, rel_path: &str, is_dir: bool) -> bool { + crate::ffi::path_is_skipped(root, rel_path, is_dir) } -/// Top-only skip dirs matching `FilterPolicy::top_only_skip_dirs_`. -/// Source-bearing dirs that are rarely the focus of analysis -/// (test/, docs/, bench/, examples/, ...). Matched ONLY against the -/// first path component (top-level dirs), so Java packages like -/// `org/springframework/samples/petclinic` are NOT falsely skipped. -fn is_top_only_skip_dir(name: &str) -> bool { - let lower = name.to_lowercase(); - matches!( - lower.as_str(), - "test" - | "tests" - | "docs" - | "doc" - | "documentation" - | "examples" - | "example" - | "samples" - | "sample" - | "scripts" - | "hack" - | "migrations" - | "seeds" - | "e2e" - | "integration" - | "locale" - | "locales" - | "i18n" - | "l10n" - | "assets" - | "static" - | "public" - | "media" - | "external" - | "vendored" - | "bench" - | "benchmarks" - ) +/// Path of `entry_path` relative to `root`, the form both the indexer and +/// `skipped_by_engine` expect. +fn rel_to(root: &Path, entry_path: &Path) -> String { + entry_path + .strip_prefix(root) + .unwrap_or(entry_path) + .to_string_lossy() + .to_string() } -/// Skip hidden dirs/files (those starting with `.`). -fn is_skip_prefix(name: &str) -> bool { - name.starts_with('.') +/// Countable source files, decided by the engine. +/// +/// Delegates to `FilterPolicy::detectLanguage`, so the server cannot count a +/// language the indexer cannot parse. The former hand-written list contained +/// `.zig`, `.rb`, `.kt`, `.cs`, `.vue` and others with no engine support; on a +/// 373-file Zig project the count said "source files" and the index contained +/// zero entities for them. +fn is_source_file(name: &str) -> bool { + crate::ffi::is_indexable_source(name) } -/// Countable source extensions matching `FilterPolicy::isSourceFile`. -/// Kept in sync with the C++ list so the scheduler's file count agrees -/// with what the worker will actually parse. -fn is_source_file(name: &str) -> bool { - let dot = match name.rfind('.') { - Some(d) => d, - None => return false, +/// Module name for the files that sit DIRECTLY in the target directory. +/// +/// They belong to no top-level directory, so a walk that only looks at +/// subdirectories drops them: a flat project (all sources in one directory) +/// reported `"modules":[]` and indexed nothing at all, and a project mixing +/// root files with subdirectories silently lost the root files. The scheduler +/// roots this module's worker at the project directory and excludes the +/// subdirectories, so the root files are indexed exactly once. +pub const ROOT_MODULE_NAME: &str = "."; + +/// Absolute paths of the source files that sit directly in `dir_path`, +/// sorted. Mirrors the shape of `discover_files` for one directory level and +/// is the companion to ROOT_MODULE_NAME. +pub fn root_source_files(dir_path: &str) -> Vec { + let mut out: Vec = Vec::new(); + let entries = match std::fs::read_dir(Path::new(dir_path)) { + Ok(e) => e, + Err(_) => return out, }; - let ext = name[dot..].to_lowercase(); - matches!( - ext.as_str(), - ".c" | ".h" - | ".cpp" - | ".hpp" - | ".cc" - | ".cxx" - | ".hh" - | ".hxx" - | ".rs" - | ".go" - | ".py" - | ".java" - | ".kt" - | ".kts" - | ".js" - | ".jsx" - | ".ts" - | ".tsx" - | ".swift" - | ".rb" - | ".php" - | ".cs" - | ".fs" - | ".scala" - | ".clj" - | ".cljs" - | ".ex" - | ".exs" - | ".erl" - | ".hrl" - | ".vue" - | ".svelte" - | ".mjs" - | ".cjs" - | ".mts" - | ".cts" - | ".d.ts" - | ".wasm" - | ".zig" - | ".mojo" - ) + for entry in entries.flatten() { + if entry.path().is_dir() { + continue; + } + let fname = entry.file_name().to_string_lossy().to_string(); + if skipped_by_engine(dir_path, &fname, false) || !is_source_file(&fname) { + continue; + } + out.push(entry.path().to_string_lossy().to_string()); + } + out.sort(); + out } /// Discover top-level modules and their source-file counts. @@ -161,7 +95,7 @@ fn is_source_file(name: &str) -> bool { /// /// Skip rules mirror `FilterPolicy` so the count matches what the worker /// will actually parse (modulo nested `test/`/`docs/` dirs which the -/// worker additionally skips — see `builtin-scheduler-design.md` §1.2). +/// worker additionally skips). /// The count is therefore an upper bound; the worker reports the true /// `discovery.candidate_files` in its stdout JSON, which the scheduler /// uses for the final summary (so reported numbers are always accurate). @@ -197,7 +131,7 @@ pub fn discover_modules(dir_path: &str) -> String { if !path.is_dir() { continue; } - if is_skip_dir(&name_str) || is_top_only_skip_dir(&name_str) || is_skip_prefix(&name_str) { + if skipped_by_engine(dir_path, &name_str, true) { continue; } @@ -210,12 +144,11 @@ pub fn discover_modules(dir_path: &str) -> String { // uses this as the worker-allocation weight. let mut bytes: u64 = 0; let walk = WalkDir::new(&path).into_iter().filter_entry(|e| { - let fname = e.file_name().to_string_lossy(); if e.depth() == 0 { return true; } if e.file_type().is_dir() { - !is_skip_dir(&fname) && !is_skip_prefix(&fname) + !skipped_by_engine(dir_path, &rel_to(Path::new(dir_path), e.path()), true) } else { true } @@ -223,7 +156,13 @@ pub fn discover_modules(dir_path: &str) -> String { for entry in walk.flatten() { if entry.file_type().is_file() { let fname = entry.file_name().to_string_lossy(); - if is_source_file(&fname) { + if is_source_file(&fname) + && !skipped_by_engine( + dir_path, + &rel_to(Path::new(dir_path), entry.path()), + false, + ) + { count += 1; if let Ok(meta) = entry.metadata() { bytes += meta.len(); @@ -237,6 +176,26 @@ pub fn discover_modules(dir_path: &str) -> String { } } + // Files directly in the target directory. The loop above only ever + // descends into subdirectories, so these were counted nowhere: a flat + // project produced an empty module list and indexed nothing, and a project + // with both root files and subdirectories lost the root files. They become + // the root module, whose worker restricts itself to this level. + let root_files = root_source_files(dir_path); + if !root_files.is_empty() { + let root_bytes: u64 = root_files + .iter() + .filter_map(|p| std::fs::metadata(p).ok()) + .map(|m| m.len()) + .sum(); + total_files += root_files.len() as u64; + modules.push(( + ROOT_MODULE_NAME.to_string(), + root_files.len() as u64, + root_bytes, + )); + } + // Sort by file count descending so the scheduler dispatches the // largest modules first (better CPU utilisation when `--parallel` // is smaller than the module count). @@ -281,26 +240,16 @@ pub fn discover_files(dir_path: &str) -> String { } let walk = WalkDir::new(root).into_iter().filter_entry(|e| { - let fname = e.file_name().to_string_lossy(); if e.depth() == 0 { return true; } if e.file_type().is_dir() { - // Apply both any-depth skip dirs and top-only skip dirs at - // depth 1. The C++ FilterPolicy checks `top_only_skip_dirs_` - // against the first 3 path components — we approximate by - // checking depth==1 here. Deeper nested test/docs dirs are - // also skipped to match worker behaviour (see design §1.2). - if is_skip_dir(&fname) || is_skip_prefix(&fname) { - return false; - } - if e.depth() == 1 && is_top_only_skip_dir(&fname) { - return false; - } - true - } else { - true + // The engine decides, so the depth-3 `top_only_skip_dirs_` rule is + // applied exactly as the worker applies it rather than being + // approximated at depth 1 as it used to be here. + return !skipped_by_engine(dir_path, &rel_to(root, e.path()), true); } + true }); let mut files: Vec = Vec::new(); @@ -309,7 +258,9 @@ pub fn discover_files(dir_path: &str) -> String { continue; } let fname = entry.file_name().to_string_lossy(); - if !is_source_file(&fname) { + if !is_source_file(&fname) + || skipped_by_engine(dir_path, &rel_to(root, entry.path()), false) + { continue; } // Use absolute path so the worker can resolve regardless of cwd. @@ -340,33 +291,153 @@ mod tests { } #[test] - fn test_is_skip_dir_matches_known_vendors() { - assert!(is_skip_dir("node_modules")); - assert!(is_skip_dir("target")); - assert!(is_skip_dir("vendor")); - assert!(is_skip_dir("third_party")); - assert!(!is_skip_dir("src")); - assert!(!is_skip_dir("engine")); + fn test_skipped_by_engine_matches_known_vendors() { + // These assertions used to run against a Rust copy of the C++ skip + // lists. They now run against the engine's FilterPolicy, i.e. the + // implementation the indexer itself uses — which is the point of the + // delegation: there is no second list left to drift. + let root = make_tmpdir("skip_vendors"); + let root = root.to_str().unwrap(); + for vendor in ["node_modules", "target", "vendor", "third_party"] { + assert!(skipped_by_engine(root, vendor, true), "{vendor} must skip"); + } + assert!(!skipped_by_engine(root, "src", true)); + assert!(!skipped_by_engine(root, "engine", true)); + // The depth-gated names are applied at the depth the indexer applies + // them (first three components), not approximated at depth 1. + assert!(skipped_by_engine(root, "test", true)); + assert!(skipped_by_engine(root, "docs", true)); + assert!(!skipped_by_engine(root, "src", true)); + assert!(!skipped_by_engine(root, "compiler", true)); } #[test] - fn test_is_top_only_skip_dir_matches_test_docs() { - assert!(is_top_only_skip_dir("test")); - assert!(is_top_only_skip_dir("docs")); - assert!(is_top_only_skip_dir("bench")); - assert!(!is_top_only_skip_dir("src")); - assert!(!is_top_only_skip_dir("compiler")); + fn test_discover_modules_reads_a_maven_layout() { + // spring-petclinic's shape: the whole source tree lives under + // src/main/java/org/.../samples/petclinic/. `samples` is a Java package + // component, not a docs folder, but the skip names are only relaxed to + // a top-only check for JAVA — and the policy behind + // engine_path_is_skipped was built with no language context, so it took + // the any-depth branch, reported `"modules":[]` and made + // `index-parallel` index nothing while claiming "no source modules + // found", whereas `codescope discover` counted every file. The policy + // now learns its language the same way the indexer's own walk does. + let root = make_tmpdir("maven_layout"); + let src = root.join("src/main/java/org/acme/samples/petclinic"); + fs::create_dir_all(&src).unwrap(); + fs::write(src.join("Owner.java"), "class Owner {}\n").unwrap(); + + let json = discover_modules(root.to_str().unwrap()); + let v: Value = serde_json::from_str(&json).unwrap(); + assert_eq!(v["ok"], true, "{json}"); + assert_eq!(v["total_files"], 1, "{json}"); + let modules = v["modules"].as_array().unwrap(); + assert_eq!(modules.len(), 1, "{json}"); + assert_eq!(modules[0]["name"], "src", "{json}"); + + // ... while a real Java test source stays excluded: the carve-out + // relaxes the names only beyond the first three path components. + let test_src = root.join("src/test/java"); + fs::create_dir_all(&test_src).unwrap(); + fs::write(test_src.join("OwnerTest.java"), "class OwnerTest {}\n").unwrap(); + let json = discover_modules(root.to_str().unwrap()); + let v: Value = serde_json::from_str(&json).unwrap(); + assert_eq!(v["total_files"], 1, "src/test must stay skipped: {json}"); } #[test] fn test_is_source_file_recognises_extensions() { assert!(is_source_file("foo.rs")); assert!(is_source_file("bar.cpp")); + // "foo.d.ts" resolves through the ".ts" entry: the extension is + // everything after the LAST dot. The ".d.ts" allow-list entry that + // used to sit in the list could never fire and has been removed — + // this assertion is what keeps the behaviour identical. + assert!(is_source_file("types.d.ts")); assert!(is_source_file("baz.go")); assert!(is_source_file("a.py")); assert!(!is_source_file("readme.md")); assert!(!is_source_file("Makefile")); assert!(!is_source_file("noext")); + // Counted only if the ENGINE can parse it. The list used to be + // hand-written and counted `.zig` (362 files in OmniScope) although + // Zig is deliberately unsupported — no mapping, no translator, and + // none planned — so the count advertised files the index could never + // contain. Ruby, which the same list contains, IS recognized: the + // point is to follow the engine, not to shorten a list. + assert!(!is_source_file("main.zig"), "no Zig support in the engine"); + assert!( + is_source_file("script.rb"), + "the engine does recognize Ruby" + ); + } + + #[test] + fn test_gitignored_top_level_dir_is_not_a_module() { + // Drift that the delegation removes: the module list was decided by a + // Rust copy of the skip rules that read no ignore file, so a + // `.gitignore`d top-level directory was reported as a module and the + // scheduler dispatched a worker for it that indexed zero files. + let dir = make_tmpdir("discover_modules_gitignore"); + fs::write(dir.join(".gitignore"), "**/build-*/\n").unwrap(); + fs::create_dir_all(dir.join("build-x/src")).unwrap(); + fs::write(dir.join("build-x/src/main.rs"), "fn main() {}").unwrap(); + fs::create_dir_all(dir.join("src")).unwrap(); + fs::write(dir.join("src/lib.rs"), "pub fn f() {}").unwrap(); + + let v: Value = serde_json::from_str(&discover_modules(dir.to_str().unwrap())).unwrap(); + assert_eq!(v["total_modules"], 1, "only src/ is a module: {v}"); + assert_eq!(v["modules"][0]["name"], "src"); + + let files: Value = serde_json::from_str(&discover_files(dir.to_str().unwrap())).unwrap(); + let list = files["files"].as_array().unwrap(); + assert_eq!(list.len(), 1, "only src/lib.rs is a candidate: {list:?}"); + } + + #[test] + fn test_discover_modules_counts_root_files() { + // Regression (#22): files directly in the target directory were counted + // nowhere. A flat project reported no modules at all and indexed + // nothing; a project mixing root files with subdirectories lost the + // root files. + let flat = make_tmpdir("discover_modules_flat"); + fs::write(flat.join("a.go"), "package main").unwrap(); + let v: Value = serde_json::from_str(&discover_modules(flat.to_str().unwrap())).unwrap(); + assert_eq!(v["total_modules"], 1, "a flat project has one module"); + assert_eq!(v["modules"][0]["name"], ROOT_MODULE_NAME); + assert_eq!(v["modules"][0]["files"], 1); + assert_eq!(v["total_files"], 1); + + let mixed = make_tmpdir("discover_modules_mixed"); + fs::write(mixed.join("top.go"), "package main").unwrap(); + fs::create_dir_all(mixed.join("sub")).unwrap(); + fs::write(mixed.join("sub/b.go"), "package sub").unwrap(); + let v: Value = serde_json::from_str(&discover_modules(mixed.to_str().unwrap())).unwrap(); + assert_eq!(v["total_modules"], 2, "root files + sub/ = two modules"); + assert_eq!( + v["total_files"], 2, + "the root file must be counted as well as sub/b.go" + ); + let names: Vec<&str> = v["modules"] + .as_array() + .unwrap() + .iter() + .map(|m| m["name"].as_str().unwrap()) + .collect(); + assert!(names.contains(&ROOT_MODULE_NAME)); + assert!(names.contains(&"sub")); + } + + #[test] + fn test_root_source_files_are_direct_children_only() { + let dir = make_tmpdir("root_source_files"); + fs::write(dir.join("a.go"), "package main").unwrap(); + fs::write(dir.join("notes.md"), "not source").unwrap(); + fs::create_dir_all(dir.join("sub")).unwrap(); + fs::write(dir.join("sub/b.go"), "package sub").unwrap(); + let files = root_source_files(dir.to_str().unwrap()); + assert_eq!(files.len(), 1); + assert!(files[0].ends_with("a.go")); } #[test] diff --git a/server/src/ffi/decls.rs b/server/src/ffi/decls.rs new file mode 100644 index 0000000..48ea90c --- /dev/null +++ b/server/src/ffi/decls.rs @@ -0,0 +1,751 @@ +// decls.rs — the raw `extern "C"` surface exported by the C++ engine. +// +// Split out of ffi/mod.rs: documenting every declaration with its ownership, +// lifetime and thread-safety contract pushed that file past the 1000-line limit +// (plan/rules/code_rules.md §1). The declarations are re-exported into +// `super` so the wrappers there call them unqualified. +// +// The wrappers in `super` are the API the server uses; these declarations are +// unsafe on purpose and every one carries a `# Safety` block stating +// +// * ownership — who frees what (engine strings come back owned by Rust and +// go through `take_string`; `engine_version` is static), +// * lifetime — inputs are borrowed for the call and never retained, +// * thread safety — the engine serialises on one store connection, so calls +// that mutate the store must not overlap an indexing call. +// +// * the instance — every declaration below takes the engine handle +// (`EngineHandle`) as its first parameter; it must name a live +// instance from `engine_create`, and stays valid until +// `engine_destroy` (TD-1 knife 3 removed the process-global +// state those calls used to reach for). +// +// Error convention (an accepted deviation from plan/rules/code_rules.md §3, +// "return int error codes"): the engine reports failures INSIDE the returned +// JSON envelope as {"ok":false,"error":"...","module":...,"method":...}, which +// is what the rule asks for in substance — an explicit error that carries the +// module/method chain — while `engine_create` (and the few other status +// functions) return an int because they transfer no payload. Changing all of +// these to `int` plus out-parameters is a cross-cutting rewrite of every +// engine export with no functional gain, so it is recorded here rather than +// done silently. + +use std::os::raw::c_char; + +/// Opaque engine instance — the C ABI's `engine_t` +/// (`typedef struct CodescopeEngine *engine_t;` in engine/include/engine.h). +/// +/// Rust never dereferences this type: it exists so an instance handle cannot be +/// confused with any other pointer. One instance is created per call to +/// [`engine_create`] and released exactly once with [`engine_destroy`] +/// (TD-1 knife 3 replaced the process-global `engine_init`/`engine_shutdown` +/// pair, whose state every entry point used to reach for implicitly). +#[repr(C)] +pub struct CodescopeEngine { + _private: [u8; 0], +} + +/// Handle to one engine instance (the C ABI's `engine_t`). +pub type EngineHandle = *mut CodescopeEngine; + +unsafe extern "C" { + /// # Safety + /// Returns an instance owned by the caller; release it with + /// [`engine_destroy`] exactly once and do not use it afterwards. + /// `db_path` is a NUL-terminated string borrowed for the call only. + /// Returns NULL when the path is rejected or the store cannot be opened + /// (the engine logs the reason to stderr). + pub fn engine_create(db_path: *const c_char) -> EngineHandle; + /// # Safety + /// `handle` must be a live instance from [`engine_create`], or NULL — NULL + /// is a documented no-op. The handle is invalid after this call, so + /// destroying it twice is undefined behaviour. Must not run while another + /// call on the same handle is in flight. + pub fn engine_destroy(handle: EngineHandle); + + /// # Safety + /// The returned pointer is a static engine string and must NOT be freed. + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + pub fn engine_version() -> *const c_char; + + /// # Safety + /// Returns a plain integer; no memory is transferred. + /// Input strings are NUL-terminated and borrowed for the call only. + /// Mutates engine/store state; must not run concurrently with another engine call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_create_project( + handle: EngineHandle, + root_path: *const c_char, + name: *const c_char, + ) -> u64; + /// # Safety + /// Returns a plain integer; no memory is transferred. + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_latest_project_id(handle: EngineHandle) -> u64; + /// # Safety + /// Returns a plain integer; no memory is transferred. + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_project_id_by_path(handle: EngineHandle, root_path: *const c_char) -> u64; + /// # Safety + /// Returns a plain integer; no memory is transferred. + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_project_node_count(handle: EngineHandle, project_id: u64) -> u64; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// Mutates engine/store state; must not run concurrently with another engine call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_index_file( + handle: EngineHandle, + project_id: u64, + file_path: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// Mutates engine/store state; must not run concurrently with another engine call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_index_project( + handle: EngineHandle, + project_id: u64, + dir_path: *const c_char, + language_filter: *const c_char, + ) -> *mut c_char; + /// Index an explicit list of files. + /// + /// # Safety + /// `file_list_json` must be a valid NUL-terminated C string that outlives + /// the call; it is not retained. The returned pointer is owned by Rust and + /// must be freed with `engine_free_string` (see [`take_string`]). The call + /// is not reentrant: the engine serialises it against other indexing calls + /// with its store guard, so callers must not run two of them concurrently. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_index_files( + handle: EngineHandle, + project_id: u64, + file_list_json: *const c_char, + bypass_fail_fast: std::os::raw::c_int, + ) -> *mut c_char; + + /// Read the project's parse_failures rows. + /// + /// # Safety + /// No pointers are passed in. The returned pointer is owned by Rust and + /// must be freed with `engine_free_string` (see [`take_string`]). Shares the + /// engine's single store connection, so it must not run concurrently with + /// an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_parse_failures( + handle: EngineHandle, + project_id: u64, + limit: std::os::raw::c_int, + ) -> *mut c_char; + + /// Delete every parse_failures row of the project. + /// + /// # Safety + /// No pointers are passed in. The returned pointer is owned by Rust and + /// must be freed with `engine_free_string` (see [`take_string`]). Shares the + /// engine's single store connection, so it must not run concurrently with + /// an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_reset_parse_failures(handle: EngineHandle, project_id: u64) -> *mut c_char; + + // Filter decisions, answered by the indexer's own FilterPolicy so the + // server never keeps a second copy of the rules (see + // engine/src/engine_filter_ffi.cpp for why both were exported). + /// # Safety + /// Returns a status code; no memory is transferred. + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + pub fn engine_path_is_skipped( + project_root: *const c_char, + rel_path: *const c_char, + is_dir: i32, + ) -> i32; + /// # Safety + /// Returns a status code; no memory is transferred. + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + pub fn engine_is_indexable_source(file_path: *const c_char) -> i32; + + // v0.2.5 (C2 fix): rebuild a project's CSR adjacency on the given DB + // (used after parallel merge, where local-id BLOBs would be dangling). + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// Mutates engine/store state; must not run concurrently with another engine call. + pub fn engine_rebuild_csr(db_path: *const c_char, project_id: u64) -> *mut c_char; + + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_find_definition( + handle: EngineHandle, + project_id: u64, + symbol_name: *const c_char, + file_filter: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_find_references( + handle: EngineHandle, + project_id: u64, + symbol_name: *const c_char, + file_filter: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_graph_stats(handle: EngineHandle, project_id: u64) -> *mut c_char; + + // ── Graph path + location queries ──────────────────────────── + // See engine_ffi.cpp for the C++ implementation. Each returns a + // heap-allocated JSON string that the caller MUST release via + // engine_free_string(). + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_find_shortest_path( + handle: EngineHandle, + project_id: u64, + source_id: u64, + target_id: u64, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_locate_by_name( + handle: EngineHandle, + project_id: u64, + name: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_find_connected_components(handle: EngineHandle, project_id: u64) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_communities( + handle: EngineHandle, + project_id: u64, + max_members: i32, + max_communities: i32, + include_members: i32, + ) -> *mut c_char; + + // ── Graph region + full export queries ──────────────────── + // See engine_ffi.cpp for the C++ implementations. Each returns a + // heap-allocated JSON string that the caller MUST release via + // engine_free_string(). + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_subgraph( + handle: EngineHandle, + project_id: u64, + center_node_id: u64, + radius: i32, + node_type_filter: *const c_char, + edge_type_filter: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_neighbors( + handle: EngineHandle, + project_id: u64, + node_id: u64, + edge_type_filter: i32, + radius: i32, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_graph_query( + handle: EngineHandle, + project_id: u64, + dsl_query: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_graph( + handle: EngineHandle, + project_id: u64, + node_offset: i64, + node_limit: i32, + edge_offset: i64, + edge_limit: i32, + node_type_filter: *const c_char, + edge_type_filter: *const c_char, + ) -> *mut c_char; + + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_search_code( + handle: EngineHandle, + project_id: u64, + query: *const c_char, + limit: i32, + ) -> *mut c_char; + + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_detect_changes( + handle: EngineHandle, + project_id: u64, + modified_files_json: *const c_char, + ) -> *mut c_char; + + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_verify_integrity( + handle: EngineHandle, + project_id: u64, + max_findings: i32, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_explain_symbol( + handle: EngineHandle, + project_id: u64, + symbol_name: *const c_char, + ) -> *mut c_char; + + // ── Knowledge + Evidence Layer (v0.3) ─────────────────────────── + // All three return a heap-allocated JSON string that the caller MUST + // release via engine_free_string(). See engine_verify_ffi.cpp for the + // C++ implementation and output shape documentation. + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_verify_claim( + handle: EngineHandle, + project_id: u64, + claim_json: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_verify_summary( + handle: EngineHandle, + project_id: u64, + text: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_explain_module( + handle: EngineHandle, + project_id: u64, + module_name: *const c_char, + ) -> *mut c_char; + + // ── Verify + Drift Layer (v0.4) ─────────────────────────────── + // See engine_verify_drift_ffi.cpp for the C++ implementation and + // output shape documentation. Each returns a heap-allocated JSON + // string that the caller MUST release via engine_free_string(). + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_verify_review( + handle: EngineHandle, + project_id: u64, + text: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_verify_reality( + handle: EngineHandle, + project_id: u64, + text: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_detect_drift(handle: EngineHandle, project_id: u64) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_detect_documentation_drift(handle: EngineHandle, project_id: u64) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_detect_capability_drift(handle: EngineHandle, project_id: u64) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_detect_architecture_drift(handle: EngineHandle, project_id: u64) -> *mut c_char; + + // ── v0.3 Evidence Pipeline ────────────────────────────────── + // See engine_evidence_ffi.cpp, engine_verify_planner_ffi.cpp, + // engine_project_state_ffi.cpp for the C++ implementations. Each + // returns a heap-allocated JSON string that the caller MUST release + // via engine_free_string(). + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// Mutates engine/store state; must not run concurrently with another engine call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_build_evidence( + handle: EngineHandle, + project_id: u64, + category_filter: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// Mutates engine/store state; must not run concurrently with another engine call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_build_project_state(handle: EngineHandle, project_id: u64) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_project_state(handle: EngineHandle, project_id: u64) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// Mutates engine/store state; must not run concurrently with another engine call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_enhance_project(handle: EngineHandle, project_id: u64) -> *mut c_char; + + // ── Verifier Registry introspection (Step 9.2) ───────────── + // See engine_verify_ffi.cpp for the C++ implementation. Returns a + // heap-allocated JSON string that the caller MUST release via + // engine_free_string(). Describes registry health, claim-type + // coverage, and evidence backend readiness. + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_verifier_registry_status( + handle: EngineHandle, + project_id: u64, + ) -> *mut c_char; + + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// Mutates engine/store state; must not run concurrently with another engine call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_build_fts(handle: EngineHandle, project_id: u64) -> *mut c_char; + + // ── Phase A: Fast Scan ──────────────────────────────────────── + + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_module_tree(handle: EngineHandle, project_id: u64) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_find_symbol( + handle: EngineHandle, + project_id: u64, + symbol_name: *const c_char, + ) -> *mut c_char; + + // ── Knowledge Graph direct query (v0.2.1) ──────────────────────── + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_knowledge_graph( + handle: EngineHandle, + project_id: u64, + table_name: *const c_char, + limit: i32, + ) -> *mut c_char; + + // ── Phase B: Background Enhancement ────────────────────────── + + // ── Phase C: Unified MCP Tools ─────────────────────────────── + + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_unified_search( + handle: EngineHandle, + project_id: u64, + query: *const c_char, + limit: i32, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_find_callers_adaptive( + handle: EngineHandle, + project_id: u64, + symbol_name: *const c_char, + file_filter: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_find_callees_adaptive( + handle: EngineHandle, + project_id: u64, + symbol_name: *const c_char, + file_filter: *const c_char, + ) -> *mut c_char; + // Step 7 (plan §7.2): entity-precise caller/callee queries. Unlike the + // bare-name APIs, these unambiguously target a single entity even when + // multiple entities share the same name. + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_find_callers_by_entity( + handle: EngineHandle, + project_id: u64, + entity_id: u64, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_find_callees_by_entity( + handle: EngineHandle, + project_id: u64, + entity_id: u64, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_entry_points_new(handle: EngineHandle, project_id: u64) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_type_info( + handle: EngineHandle, + project_id: u64, + type_name_filter: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_get_routes(handle: EngineHandle, project_id: u64) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_project_overview(handle: EngineHandle, project_id: u64) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_trace_path( + handle: EngineHandle, + project_id: u64, + from_name: *const c_char, + to_name: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// Input strings are NUL-terminated and borrowed for the call only. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_explore_function( + handle: EngineHandle, + project_id: u64, + function_name: *const c_char, + depth: i32, + direction: *const c_char, + ) -> *mut c_char; + /// # Safety + /// The returned pointer is engine-owned; release it with [`take_string`] + /// (i.e. `engine_free_string`). + /// No pointers are passed in. + /// The engine serialises calls on one store connection, so do not call + /// it concurrently with an indexing call. + /// `handle` must be a live instance from `engine_create` (engine.h). + pub fn engine_detect_ffi_boundaries(handle: EngineHandle, project_id: u64) -> *mut c_char; + + // ── Code Understanding (Phase C, missing bindings) ─────────── + + /// # Safety + /// Returns a status code; no memory is transferred. + /// No pointers are passed in. + /// Transfers ownership back to the engine; call it once per string the engine allocated. + pub fn engine_free_string(ptr: *mut c_char); +} diff --git a/server/src/ffi/mod.rs b/server/src/ffi/mod.rs index 898aefa..5b72857 100644 --- a/server/src/ffi/mod.rs +++ b/server/src/ffi/mod.rs @@ -1,176 +1,41 @@ use std::ffi::{CStr, CString}; use std::os::raw::c_char; +use std::ptr; +use std::sync::atomic::{AtomicPtr, Ordering}; // ── FFI bindings to the C++ engine ───────────────────────────── -unsafe extern "C" { - fn engine_init(db_path: *const c_char) -> i32; - fn engine_shutdown(); - - fn engine_version() -> *const c_char; - - fn engine_create_project(root_path: *const c_char, name: *const c_char) -> u64; - fn engine_get_latest_project_id() -> u64; - fn engine_get_project_id_by_path(root_path: *const c_char) -> u64; - fn engine_get_project_node_count(project_id: u64) -> u64; - fn engine_index_file(project_id: u64, file_path: *const c_char) -> *mut c_char; - fn engine_index_project( - project_id: u64, - dir_path: *const c_char, - language_filter: *const c_char, - ) -> *mut c_char; - fn engine_index_files(project_id: u64, file_list_json: *const c_char) -> *mut c_char; - - // v0.2.5 (C2 fix): rebuild a project's CSR adjacency on the given DB - // (used after parallel merge, where local-id BLOBs would be dangling). - fn engine_rebuild_csr(db_path: *const c_char, project_id: u64) -> *mut c_char; - - fn engine_find_definition( - project_id: u64, - symbol_name: *const c_char, - file_filter: *const c_char, - ) -> *mut c_char; - fn engine_find_references( - project_id: u64, - symbol_name: *const c_char, - file_filter: *const c_char, - ) -> *mut c_char; - fn engine_get_graph_stats(project_id: u64) -> *mut c_char; - - // ── Graph path + location queries ──────────────────────────── - // See engine_ffi.cpp for the C++ implementation. Each returns a - // heap-allocated JSON string that the caller MUST release via - // engine_free_string(). - fn engine_find_shortest_path(project_id: u64, source_id: u64, target_id: u64) -> *mut c_char; - fn engine_locate_by_name(project_id: u64, name: *const c_char) -> *mut c_char; - fn engine_find_connected_components(project_id: u64) -> *mut c_char; - - // ── Graph region + full export queries ──────────────────── - // See engine_ffi.cpp for the C++ implementations. Each returns a - // heap-allocated JSON string that the caller MUST release via - // engine_free_string(). - fn engine_get_subgraph( - project_id: u64, - center_node_id: u64, - radius: i32, - node_type_filter: *const c_char, - edge_type_filter: *const c_char, - ) -> *mut c_char; - fn engine_get_neighbors( - project_id: u64, - node_id: u64, - edge_type_filter: i32, - radius: i32, - ) -> *mut c_char; - fn engine_graph_query(project_id: u64, dsl_query: *const c_char) -> *mut c_char; - fn engine_get_graph( - project_id: u64, - node_offset: i64, - node_limit: i32, - edge_offset: i64, - edge_limit: i32, - node_type_filter: *const c_char, - edge_type_filter: *const c_char, - ) -> *mut c_char; - - fn engine_search_code(project_id: u64, query: *const c_char, limit: i32) -> *mut c_char; - - fn engine_detect_changes(project_id: u64, modified_files_json: *const c_char) -> *mut c_char; - - fn engine_verify_integrity(project_id: u64) -> *mut c_char; - fn engine_explain_symbol(project_id: u64, symbol_name: *const c_char) -> *mut c_char; - - // ── Knowledge + Evidence Layer (v0.3) ─────────────────────────── - // All three return a heap-allocated JSON string that the caller MUST - // release via engine_free_string(). See engine_verify_ffi.cpp for the - // C++ implementation and output shape documentation. - fn engine_verify_claim(project_id: u64, claim_json: *const c_char) -> *mut c_char; - fn engine_verify_summary(project_id: u64, text: *const c_char) -> *mut c_char; - fn engine_explain_module(project_id: u64, module_name: *const c_char) -> *mut c_char; - - // ── Verify + Drift Layer (v0.4) ─────────────────────────────── - // See engine_verify_drift_ffi.cpp for the C++ implementation and - // output shape documentation. Each returns a heap-allocated JSON - // string that the caller MUST release via engine_free_string(). - fn engine_verify_review(project_id: u64, text: *const c_char) -> *mut c_char; - fn engine_verify_reality(project_id: u64, text: *const c_char) -> *mut c_char; - fn engine_detect_drift(project_id: u64) -> *mut c_char; - fn engine_detect_documentation_drift(project_id: u64) -> *mut c_char; - fn engine_detect_capability_drift(project_id: u64) -> *mut c_char; - fn engine_detect_architecture_drift(project_id: u64) -> *mut c_char; - - // ── v0.3 Evidence Pipeline ────────────────────────────────── - // See engine_evidence_ffi.cpp, engine_verify_planner_ffi.cpp, - // engine_project_state_ffi.cpp for the C++ implementations. Each - // returns a heap-allocated JSON string that the caller MUST release - // via engine_free_string(). - fn engine_build_evidence(project_id: u64, category_filter: *const c_char) -> *mut c_char; - fn engine_verify_statement(project_id: u64, claim_text: *const c_char) -> *mut c_char; - fn engine_build_project_state(project_id: u64) -> *mut c_char; - fn engine_get_project_state(project_id: u64) -> *mut c_char; - fn engine_enhance_project(project_id: u64) -> *mut c_char; - - // ── Verifier Registry introspection (Step 9.2) ───────────── - // See engine_verify_ffi.cpp for the C++ implementation. Returns a - // heap-allocated JSON string that the caller MUST release via - // engine_free_string(). Describes registry health, claim-type - // coverage, and evidence backend readiness. - fn engine_get_verifier_registry_status(project_id: u64) -> *mut c_char; - - fn engine_build_fts(project_id: u64) -> *mut c_char; - - // ── Phase A: Fast Scan ──────────────────────────────────────── - - fn engine_get_module_tree(project_id: u64) -> *mut c_char; - fn engine_find_symbol(project_id: u64, symbol_name: *const c_char) -> *mut c_char; - - // ── Knowledge Graph direct query (v0.2.1) ──────────────────────── - fn engine_get_knowledge_graph( - project_id: u64, - table_name: *const c_char, - limit: i32, - ) -> *mut c_char; - - // ── Phase B: Background Enhancement ────────────────────────── - - // ── Phase C: Unified MCP Tools ─────────────────────────────── - - fn engine_unified_search(project_id: u64, query: *const c_char, limit: i32) -> *mut c_char; - fn engine_find_callers_adaptive( - project_id: u64, - symbol_name: *const c_char, - file_filter: *const c_char, - ) -> *mut c_char; - fn engine_find_callees_adaptive( - project_id: u64, - symbol_name: *const c_char, - file_filter: *const c_char, - ) -> *mut c_char; - // Step 7 (plan §7.2): entity-precise caller/callee queries. Unlike the - // bare-name APIs, these unambiguously target a single entity even when - // multiple entities share the same name. - fn engine_find_callers_by_entity(project_id: u64, entity_id: u64) -> *mut c_char; - fn engine_find_callees_by_entity(project_id: u64, entity_id: u64) -> *mut c_char; - fn engine_get_entry_points_new(project_id: u64) -> *mut c_char; - fn engine_get_type_info(project_id: u64, type_name_filter: *const c_char) -> *mut c_char; - fn engine_get_routes(project_id: u64) -> *mut c_char; - fn engine_project_overview(project_id: u64) -> *mut c_char; - fn engine_trace_path( - project_id: u64, - from_name: *const c_char, - to_name: *const c_char, - ) -> *mut c_char; - fn engine_explore_function( - project_id: u64, - function_name: *const c_char, - depth: i32, - direction: *const c_char, - ) -> *mut c_char; - fn engine_detect_ffi_boundaries(project_id: u64) -> *mut c_char; - - // ── Code Understanding (Phase C, missing bindings) ─────────── - - fn engine_free_string(ptr: *mut c_char); +// The raw engine declarations, re-exported so the wrappers below can call them +// unqualified. See decls.rs for their safety contract. +mod decls; +use decls::*; + +// ── Engine instance ownership ────────────────────────────────── + +/// The process's engine instance (TD-1 knife 3). +/// +/// The C ABI is handle-based: `engine_create()` returns the instance and every +/// stateful call takes it as its first parameter. The server is single-tenant — +/// one database per process, one dispatch thread — so the process owns exactly +/// one instance and keeps its handle here. Ownership is explicit: `init` +/// installs a handle (releasing any previous instance, which is what the index +/// path's re-initialisation does) and `shutdown` releases it. No other code +/// creates or frees an instance. +/// +/// This is the caller's ownership, not hidden engine state: the C++ side keeps +/// no instance pointer of its own. An atomic (rather than a `static mut`) makes +/// the handle safe to publish/take even though today's callers are +/// single-threaded. +static ENGINE: AtomicPtr = AtomicPtr::new(ptr::null_mut()); + +/// The handle passed to every stateful engine call. +/// +/// It is null when no instance is live. Every engine entry point reports that +/// as the same "engine not initialized" envelope it returned before TD-1, so a +/// call made outside an `init`/`shutdown` window degrades exactly as it used to +/// instead of dereferencing anything. +fn engine_handle() -> EngineHandle { + ENGINE.load(Ordering::Acquire) } // ── Safe wrapper ─────────────────────────────────────────────── @@ -212,12 +77,31 @@ fn take_string(ptr: *mut c_char) -> String { s } +/// Open (creating if needed) the engine's database and install the instance +/// that every other wrapper in this module drives. +/// +/// Returns 0 on success and a non-zero status on failure, matching the old +/// `engine_init` contract. A failed call installs no instance (the handle stays +/// null and every wrapper then reports "engine not initialized"), and any +/// previous instance is released either way — a re-initialisation must not +/// leave the old database open behind the caller's back. pub fn init(db_path: &str) -> i32 { - unsafe { engine_init(cstr(db_path).as_ptr()) } + let handle = unsafe { engine_create(cstr(db_path).as_ptr()) }; + let previous = ENGINE.swap(handle, Ordering::AcqRel); + if !previous.is_null() { + unsafe { engine_destroy(previous) }; + } + if handle.is_null() { -1 } else { 0 } } +/// Release the process's engine instance (closing its store and joining its +/// background knowledge builder). Safe to call when no instance is live, and +/// safe to call more than once. pub fn shutdown() { - unsafe { engine_shutdown() } + let handle = ENGINE.swap(ptr::null_mut(), Ordering::AcqRel); + if !handle.is_null() { + unsafe { engine_destroy(handle) }; + } } /// Rebuild a project's CSR adjacency on the given DB (C2 fix). @@ -250,42 +134,107 @@ pub fn version() -> String { } pub fn create_project(root_path: &str, name: &str) -> u64 { - unsafe { engine_create_project(cstr(root_path).as_ptr(), cstr(name).as_ptr()) } + unsafe { + engine_create_project( + engine_handle(), + cstr(root_path).as_ptr(), + cstr(name).as_ptr(), + ) + } } pub fn get_latest_project_id() -> u64 { - unsafe { engine_get_latest_project_id() } + unsafe { engine_get_latest_project_id(engine_handle()) } } /// Look up a project ID by its root_path without creating a new project. /// Returns 0 if no project matches. pub fn get_project_id_by_path(root_path: &str) -> u64 { - unsafe { engine_get_project_id_by_path(cstr(root_path).as_ptr()) } + unsafe { engine_get_project_id_by_path(engine_handle(), cstr(root_path).as_ptr()) } } /// Count graph_nodes for a project — 0 means no indexed data. pub fn get_project_node_count(project_id: u64) -> u64 { - unsafe { engine_get_project_node_count(project_id) } + unsafe { engine_get_project_node_count(engine_handle(), project_id) } +} + +/// Read the project's `parse_failures` rows as +/// `{"ok":true,"parse_failures":[…]}`. `limit <= 0` means the store default. +pub fn get_parse_failures(project_id: u64, limit: i32) -> String { + take_string(unsafe { engine_get_parse_failures(engine_handle(), project_id, limit) }) +} + +/// Clear the project's `parse_failures` rows; returns `{"ok":true,"removed":N}`. +pub fn reset_parse_failures(project_id: u64) -> String { + take_string(unsafe { engine_reset_parse_failures(engine_handle(), project_id) }) } pub fn index_file(project_id: u64, file_path: &str) -> String { - take_string(unsafe { engine_index_file(project_id, cstr(file_path).as_ptr()) }) + take_string(unsafe { engine_index_file(engine_handle(), project_id, cstr(file_path).as_ptr()) }) } pub fn index_project(project_id: u64, dir_path: &str, language_filter: *const c_char) -> String { take_string(unsafe { - engine_index_project(project_id, cstr(dir_path).as_ptr(), language_filter) + engine_index_project( + engine_handle(), + project_id, + cstr(dir_path).as_ptr(), + language_filter, + ) + }) +} + +/// Index an explicit list of files (JSON array of paths). +/// +/// `bypass_fail_fast` picks the fail-fast policy for the listed files: +/// `false` honours it like the automatic project path (a file whose parse has +/// failed `CODESCOPE_FAIL_RETRY_MAX` times is skipped — the scheduler-driven +/// worker paths want this), `true` always re-attempts every file +/// (`force_index_files`, whose contract is "index these paths regardless"). +/// See `engine_index_files` in `engine/include/engine.h`. +/// +/// # Safety +/// `file_list_json` is copied into a `CString` that lives for the duration of +/// the call; the returned pointer is owned by Rust and freed by [`take_string`]. +pub fn index_files(project_id: u64, file_list_json: &str, bypass_fail_fast: bool) -> String { + take_string(unsafe { + engine_index_files( + engine_handle(), + project_id, + cstr(file_list_json).as_ptr(), + if bypass_fail_fast { 1 } else { 0 }, + ) }) } -pub fn index_files(project_id: u64, file_list_json: &str) -> String { - take_string(unsafe { engine_index_files(project_id, cstr(file_list_json).as_ptr()) }) +/// Whether the engine's `FilterPolicy` would skip `rel_path`. +/// +/// `rel_path` must be relative to `project_root` — the same shape the indexer +/// passes — so a rule anchored there (`.gitignore`'s `**/build-*/`) can match. +/// This is the single authority the module discovery consults instead of +/// keeping its own copy of the skip lists. +pub fn path_is_skipped(project_root: &str, rel_path: &str, is_dir: bool) -> bool { + let root = cstr(project_root); + let rel = cstr(rel_path); + unsafe { engine_path_is_skipped(root.as_ptr(), rel.as_ptr(), if is_dir { 1 } else { 0 }) != 0 } +} + +/// Whether the engine recognizes a language for this path (by extension, or by +/// shebang for extensionless scripts). +/// +/// Replaces the server's own extension allow-list, which counted languages the +/// engine cannot parse — `.zig` among them, which is how a Zig project came to +/// report source files the index could never contain. +pub fn is_indexable_source(name_or_path: &str) -> bool { + let p = cstr(name_or_path); + unsafe { engine_is_indexable_source(p.as_ptr()) != 0 } } pub fn find_definition(project_id: u64, symbol_name: &str, file_filter: Option<&str>) -> String { let ff = file_filter.map(cstr); take_string(unsafe { engine_find_definition( + engine_handle(), project_id, cstr(symbol_name).as_ptr(), ff.as_ref().map_or(std::ptr::null(), |s| s.as_ptr()), @@ -297,6 +246,7 @@ pub fn find_references(project_id: u64, symbol_name: &str, file_filter: Option<& let ff = file_filter.map(cstr); take_string(unsafe { engine_find_references( + engine_handle(), project_id, cstr(symbol_name).as_ptr(), ff.as_ref().map_or(std::ptr::null(), |s| s.as_ptr()), @@ -305,7 +255,7 @@ pub fn find_references(project_id: u64, symbol_name: &str, file_filter: Option<& } pub fn get_graph_stats(project_id: u64) -> String { - take_string(unsafe { engine_get_graph_stats(project_id) }) + take_string(unsafe { engine_get_graph_stats(engine_handle(), project_id) }) } /// Find the shortest call-graph path between two graph nodes by ID. @@ -314,7 +264,9 @@ pub fn get_graph_stats(project_id: u64) -> String { /// (typically `{"path":[...],"error":"..."}`). When the engine is not /// initialized the C++ side returns `{"path":[],"error":"not initialized"}`. pub fn find_shortest_path(project_id: u64, source_id: u64, target_id: u64) -> String { - take_string(unsafe { engine_find_shortest_path(project_id, source_id, target_id) }) + take_string(unsafe { + engine_find_shortest_path(engine_handle(), project_id, source_id, target_id) + }) } /// Resolve a symbol name to its graph node location(s). @@ -323,7 +275,7 @@ pub fn find_shortest_path(project_id: u64, source_id: u64, target_id: u64) -> St /// The MCP layer uses the first `node_id` to translate a user-supplied /// symbol name into the integer ID required by `find_shortest_path`. pub fn locate_by_name(project_id: u64, name: &str) -> String { - take_string(unsafe { engine_locate_by_name(project_id, cstr(name).as_ptr()) }) + take_string(unsafe { engine_locate_by_name(engine_handle(), project_id, cstr(name).as_ptr()) }) } /// Find connected components in the call graph (heuristic, BFS over @@ -333,7 +285,34 @@ pub fn locate_by_name(project_id: u64, name: &str) -> String { /// "note":"..."}`. On error the JSON contains an "error" field tagged with /// module/method per code_rules.md. pub fn find_connected_components(project_id: u64) -> String { - take_string(unsafe { engine_find_connected_components(project_id) }) + take_string(unsafe { engine_find_connected_components(engine_handle(), project_id) }) +} + +/// Run label-propagation community detection over the CALLS graph. +/// +/// Returns JSON +/// `{"communities":[{"id":N,"label":"...","member_count":N[,"members":[...]]}], +/// "total_communities":N,"returned_communities":N, +/// "inter_community_edges":N,"truncated":bool,"approximation":"heuristic", +/// "note":"..."}` from `QueryEngine::getCommunities` (implemented in +/// `engine/src/query/query_communities.cpp`). Members are only included when +/// `include_members` is true; `max_members` / `max_communities` are clamped +/// engine-side, and a non-positive value means "use the default". +pub fn get_communities( + project_id: u64, + max_members: i32, + max_communities: i32, + include_members: bool, +) -> String { + take_string(unsafe { + engine_get_communities( + engine_handle(), + project_id, + max_members, + max_communities, + if include_members { 1 } else { 0 }, + ) + }) } /// Fetch a local region of the code graph centered on a node. @@ -353,6 +332,7 @@ pub fn get_subgraph( let et = edge_type_filter.map(cstr); take_string(unsafe { engine_get_subgraph( + engine_handle(), project_id, center_node_id, radius, @@ -368,7 +348,15 @@ pub fn get_subgraph( /// is reserved (currently 1 hop). Returns JSON `{"neighbors":[...],"total":N}` /// from `QueryEngine::getNeighbors`. pub fn get_neighbors(project_id: u64, node_id: u64, edge_type_filter: i32, radius: i32) -> String { - take_string(unsafe { engine_get_neighbors(project_id, node_id, edge_type_filter, radius) }) + take_string(unsafe { + engine_get_neighbors( + engine_handle(), + project_id, + node_id, + edge_type_filter, + radius, + ) + }) } /// Query the code graph with the Cypher-like DSL implemented by @@ -381,7 +369,9 @@ pub fn get_neighbors(project_id: u64, node_id: u64, edge_type_filter: i32, radiu /// `edgeType*min..max` (e.g. `MATCH (Function)-[Calls*1..3]->(Function)`). /// Returns JSON `{"results":[...],"total":N}`. pub fn graph_query(project_id: u64, dsl_query: &str) -> String { - take_string(unsafe { engine_graph_query(project_id, cstr(dsl_query).as_ptr()) }) + take_string(unsafe { + engine_graph_query(engine_handle(), project_id, cstr(dsl_query).as_ptr()) + }) } /// Export the complete code graph in paginated pages. @@ -405,6 +395,7 @@ pub fn get_graph( let et = edge_type_filter.map(cstr); take_string(unsafe { engine_get_graph( + engine_handle(), project_id, node_offset, node_limit, @@ -417,23 +408,33 @@ pub fn get_graph( } pub fn search_code(project_id: u64, query: &str, limit: i32) -> String { - take_string(unsafe { engine_search_code(project_id, cstr(query).as_ptr(), limit) }) + take_string(unsafe { + engine_search_code(engine_handle(), project_id, cstr(query).as_ptr(), limit) + }) } pub fn detect_changes(project_id: u64, modified_files_json: &str) -> String { - take_string(unsafe { engine_detect_changes(project_id, cstr(modified_files_json).as_ptr()) }) + take_string(unsafe { + engine_detect_changes( + engine_handle(), + project_id, + cstr(modified_files_json).as_ptr(), + ) + }) } pub fn build_fts(project_id: u64) -> String { - take_string(unsafe { engine_build_fts(project_id) }) + take_string(unsafe { engine_build_fts(engine_handle(), project_id) }) } -pub fn verify_integrity(project_id: u64) -> String { - take_string(unsafe { engine_verify_integrity(project_id) }) +pub fn verify_integrity(project_id: u64, max_findings: i32) -> String { + take_string(unsafe { engine_verify_integrity(engine_handle(), project_id, max_findings) }) } pub fn explain_symbol(project_id: u64, symbol_name: &str) -> String { - take_string(unsafe { engine_explain_symbol(project_id, cstr(symbol_name).as_ptr()) }) + take_string(unsafe { + engine_explain_symbol(engine_handle(), project_id, cstr(symbol_name).as_ptr()) + }) } // ── Knowledge + Evidence Layer (v0.3) ─────────────────────────── @@ -452,7 +453,9 @@ pub fn explain_symbol(project_id: u64, symbol_name: &str) -> String { /// Returns JSON with `claim_id`, `verdict`, `confidence`, `verifier`, /// `detail`, and `evidence_facts` fields. pub fn verify_claim(project_id: u64, claim_json: &str) -> String { - take_string(unsafe { engine_verify_claim(project_id, cstr(claim_json).as_ptr()) }) + take_string(unsafe { + engine_verify_claim(engine_handle(), project_id, cstr(claim_json).as_ptr()) + }) } /// Parse a natural-language summary into claims and verify each one. @@ -461,7 +464,7 @@ pub fn verify_claim(project_id: u64, claim_json: &str) -> String { /// Returns JSON with aggregated `verdicts`, `trust_score`, and a `claims` /// array describing each parsed claim + its verdict. pub fn verify_summary(project_id: u64, text: &str) -> String { - take_string(unsafe { engine_verify_summary(project_id, cstr(text).as_ptr()) }) + take_string(unsafe { engine_verify_summary(engine_handle(), project_id, cstr(text).as_ptr()) }) } /// Build a Knowledge Card for a named module. @@ -471,7 +474,9 @@ pub fn verify_summary(project_id: u64, text: &str) -> String { /// `findings`, and `integrity_score` fields. Falls back to deriving module /// info from the `files` table when the `modules` table is empty. pub fn explain_module(project_id: u64, module_name: &str) -> String { - take_string(unsafe { engine_explain_module(project_id, cstr(module_name).as_ptr()) }) + take_string(unsafe { + engine_explain_module(engine_handle(), project_id, cstr(module_name).as_ptr()) + }) } // ── Verify + Drift Layer (v0.4) ─────────────────────────────── @@ -487,7 +492,7 @@ pub fn explain_module(project_id: u64, module_name: &str) -> String { /// `source_kind="code_review"` so the evidence table can be filtered by /// origin. Output JSON shape is identical to `verify_summary`. pub fn verify_review(project_id: u64, text: &str) -> String { - take_string(unsafe { engine_verify_review(project_id, cstr(text).as_ptr()) }) + take_string(unsafe { engine_verify_review(engine_handle(), project_id, cstr(text).as_ptr()) }) } /// Verify a single AI statement about the current project reality. @@ -496,7 +501,7 @@ pub fn verify_review(project_id: u64, text: &str) -> String { /// `Supported`, `Contradicted`, `PartiallyVerified`, or `Unknown` plus /// a `confidence` score and the per-claim `results` array. pub fn verify_reality(project_id: u64, text: &str) -> String { - take_string(unsafe { engine_verify_reality(project_id, cstr(text).as_ptr()) }) + take_string(unsafe { engine_verify_reality(engine_handle(), project_id, cstr(text).as_ptr()) }) } // ── v0.3 Evidence Pipeline ─────────────────────────────────────── @@ -518,7 +523,7 @@ pub fn verify_reality(project_id: u64, text: &str) -> String { /// timing breakdowns. On error returns a JSON object with an "error" /// field tagged with module/method per code_rules.md. pub fn enhance_project(project_id: u64) -> String { - take_string(unsafe { engine_enhance_project(project_id) }) + take_string(unsafe { engine_enhance_project(engine_handle(), project_id) }) } /// Build evidence findings for a project by applying the rule set @@ -533,40 +538,25 @@ pub fn build_evidence(project_id: u64, category_filter: Option<&str>) -> String let cf = category_filter.map(cstr); take_string(unsafe { engine_build_evidence( + engine_handle(), project_id, cf.as_ref().map_or(std::ptr::null(), |s| s.as_ptr()), ) }) } -/// Verify a natural-language claim against the project's indexed -/// evidence. This is a thin wrapper over the structured verify_claim -/// path: the claim is parsed into an Intent by IntentParser, mapped to -/// a structured Claim (capability_question → capability_exists, -/// safety/pattern_question → contract_holds), and dispatched through -/// the same verify_one_claim core used by verify_claim. Intents that -/// match no known category return verdict Unknown with -/// error_code="intent_unrecognized" instead of silently guessing. -/// -/// Returns JSON with `verdict`, `confidence`, and verifier-specific -/// detail fields. On error returns a JSON object with an "error" field -/// tagged with module/method per code_rules.md. -pub fn verify_statement(project_id: u64, claim_text: &str) -> String { - take_string(unsafe { engine_verify_statement(project_id, cstr(claim_text).as_ptr()) }) -} - /// Build and persist the project state snapshot. Runs the full /// analysis pipeline (evidence aggregation + state queries + UPSERT /// into project_state) and returns the persisted snapshot JSON. pub fn build_project_state(project_id: u64) -> String { - take_string(unsafe { engine_build_project_state(project_id) }) + take_string(unsafe { engine_build_project_state(engine_handle(), project_id) }) } /// Get the persisted project state snapshot (without rebuilding). /// Returns the snapshot_json string for the project, or a JSON error /// object if no snapshot exists yet. pub fn get_project_state(project_id: u64) -> String { - take_string(unsafe { engine_get_project_state(project_id) }) + take_string(unsafe { engine_get_project_state(engine_handle(), project_id) }) } /// Inspect the VerifierRegistry health and claim-type coverage (Step 9.2). @@ -576,7 +566,7 @@ pub fn get_project_state(project_id: u64) -> String { /// evidence backend (entity/relation) has data for the given project. /// Pass `project_id = 0` to skip the evidence backend probe. pub fn get_verifier_registry_status(project_id: u64) -> String { - take_string(unsafe { engine_get_verifier_registry_status(project_id) }) + take_string(unsafe { engine_get_verifier_registry_status(engine_handle(), project_id) }) } /// Scan all declared capabilities and contracts for drift between @@ -587,7 +577,7 @@ pub fn get_verifier_registry_status(project_id: u64) -> String { /// detected drift is persisted as a `finding` row and returned in the /// JSON output. pub fn detect_drift(project_id: u64) -> String { - take_string(unsafe { engine_detect_drift(project_id) }) + take_string(unsafe { engine_detect_drift(engine_handle(), project_id) }) } /// Scan README for language support claims and cross-reference with @@ -597,7 +587,7 @@ pub fn detect_drift(project_id: u64) -> String { /// entities with that language exist in the entity table. Each detected drift /// is persisted as a `finding` row and returned in the JSON output. pub fn detect_documentation_drift(project_id: u64) -> String { - take_string(unsafe { engine_detect_documentation_drift(project_id) }) + take_string(unsafe { engine_detect_documentation_drift(engine_handle(), project_id) }) } /// Scan declared capabilities and cross-reference with actual implementing @@ -607,7 +597,7 @@ pub fn detect_documentation_drift(project_id: u64) -> String { /// implementing entity with callers exists. Each drift is persisted as a /// `finding` row and returned in the JSON output. pub fn detect_capability_drift(project_id: u64) -> String { - take_string(unsafe { engine_detect_capability_drift(project_id) }) + take_string(unsafe { engine_detect_capability_drift(engine_handle(), project_id) }) } /// Scan call edges for architecture layer violations (e.g. Repository @@ -617,13 +607,13 @@ pub fn detect_capability_drift(project_id: u64) -> String { /// layered flow Controller -> Service -> Repository. Each drift is /// persisted as a `finding` row and returned in the JSON output. pub fn detect_architecture_drift(project_id: u64) -> String { - take_string(unsafe { engine_detect_architecture_drift(project_id) }) + take_string(unsafe { engine_detect_architecture_drift(engine_handle(), project_id) }) } // ── Phase A: Fast Scan ──────────────────────────────────────── pub fn get_module_tree(project_id: u64) -> String { - take_string(unsafe { engine_get_module_tree(project_id) }) + take_string(unsafe { engine_get_module_tree(engine_handle(), project_id) }) } /// Direct-query a knowledge-layer table (v0.2.1). @@ -637,11 +627,20 @@ pub fn get_module_tree(project_id: u64) -> String { /// from the C++ `engine_get_knowledge_graph`. On error the JSON contains /// an `"error"` field tagged with module/method per code_rules.md. pub fn get_knowledge_graph(project_id: u64, table_name: &str, limit: i32) -> String { - take_string(unsafe { engine_get_knowledge_graph(project_id, cstr(table_name).as_ptr(), limit) }) + take_string(unsafe { + engine_get_knowledge_graph( + engine_handle(), + project_id, + cstr(table_name).as_ptr(), + limit, + ) + }) } pub fn find_symbol(project_id: u64, symbol_name: &str) -> String { - take_string(unsafe { engine_find_symbol(project_id, cstr(symbol_name).as_ptr()) }) + take_string(unsafe { + engine_find_symbol(engine_handle(), project_id, cstr(symbol_name).as_ptr()) + }) } // ── Phase B: Background Enhancement ─────────────────────────── @@ -649,7 +648,9 @@ pub fn find_symbol(project_id: u64, symbol_name: &str) -> String { // ── Phase C: Unified MCP Tools ─────────────────────────────── pub fn unified_search(project_id: u64, query: &str, limit: i32) -> String { - take_string(unsafe { engine_unified_search(project_id, cstr(query).as_ptr(), limit) }) + take_string(unsafe { + engine_unified_search(engine_handle(), project_id, cstr(query).as_ptr(), limit) + }) } pub fn find_callers_adaptive( @@ -659,7 +660,12 @@ pub fn find_callers_adaptive( ) -> String { let ff = file_filter.unwrap_or(""); take_string(unsafe { - engine_find_callers_adaptive(project_id, cstr(symbol_name).as_ptr(), cstr(ff).as_ptr()) + engine_find_callers_adaptive( + engine_handle(), + project_id, + cstr(symbol_name).as_ptr(), + cstr(ff).as_ptr(), + ) }) } @@ -670,7 +676,12 @@ pub fn find_callees_adaptive( ) -> String { let ff = file_filter.unwrap_or(""); take_string(unsafe { - engine_find_callees_adaptive(project_id, cstr(symbol_name).as_ptr(), cstr(ff).as_ptr()) + engine_find_callees_adaptive( + engine_handle(), + project_id, + cstr(symbol_name).as_ptr(), + cstr(ff).as_ptr(), + ) }) } @@ -678,37 +689,44 @@ pub fn find_callees_adaptive( /// The entity id is resolved to (name, file_path, start_row) in the /// engine, so the query never aggregates homonyms. pub fn find_callers_by_entity(project_id: u64, entity_id: u64) -> String { - take_string(unsafe { engine_find_callers_by_entity(project_id, entity_id) }) + take_string(unsafe { engine_find_callers_by_entity(engine_handle(), project_id, entity_id) }) } /// Step 7 (plan §7.2): find callees of a precise entity by its id. pub fn find_callees_by_entity(project_id: u64, entity_id: u64) -> String { - take_string(unsafe { engine_find_callees_by_entity(project_id, entity_id) }) + take_string(unsafe { engine_find_callees_by_entity(engine_handle(), project_id, entity_id) }) } pub fn get_entry_points_new(project_id: u64) -> String { - take_string(unsafe { engine_get_entry_points_new(project_id) }) + take_string(unsafe { engine_get_entry_points_new(engine_handle(), project_id) }) } pub fn get_type_info(project_id: u64, type_name_filter: &str) -> String { - take_string(unsafe { engine_get_type_info(project_id, cstr(type_name_filter).as_ptr()) }) + take_string(unsafe { + engine_get_type_info(engine_handle(), project_id, cstr(type_name_filter).as_ptr()) + }) } pub fn get_routes(project_id: u64) -> String { - take_string(unsafe { engine_get_routes(project_id) }) + take_string(unsafe { engine_get_routes(engine_handle(), project_id) }) } pub fn project_overview(project_id: u64) -> String { - take_string(unsafe { engine_project_overview(project_id) }) + take_string(unsafe { engine_project_overview(engine_handle(), project_id) }) } pub fn detect_ffi_boundaries(project_id: u64) -> String { - take_string(unsafe { engine_detect_ffi_boundaries(project_id) }) + take_string(unsafe { engine_detect_ffi_boundaries(engine_handle(), project_id) }) } pub fn trace_path(project_id: u64, from_name: &str, to_name: &str) -> String { take_string(unsafe { - engine_trace_path(project_id, cstr(from_name).as_ptr(), cstr(to_name).as_ptr()) + engine_trace_path( + engine_handle(), + project_id, + cstr(from_name).as_ptr(), + cstr(to_name).as_ptr(), + ) }) } @@ -720,6 +738,7 @@ pub fn explore_function( ) -> String { take_string(unsafe { engine_explore_function( + engine_handle(), project_id, cstr(function_name).as_ptr(), depth, diff --git a/server/src/main.rs b/server/src/main.rs index d83a5aa..d224f7b 100644 --- a/server/src/main.rs +++ b/server/src/main.rs @@ -1,23 +1,34 @@ mod discover; mod ffi; mod mcp; -#[cfg(not(windows))] mod scheduler; mod tools; -#[cfg(not(windows))] use crate::scheduler::chunk_queue; -// `Value`/`json` are used on non-Windows cfg branches but are otherwise -// referenced via fully-qualified `serde_json::` paths on Windows, so the -// import can appear unused when cross-compiling. Allow it to keep the -// Windows build warning-free. -#[allow(unused_imports)] use serde_json::{Value, json}; use std::env; use std::fs; use std::path::Path; +/// Resolve the database path used by the CLI-style entry points (`cli` and +/// `index-parallel`). +/// +/// `CODESCOPE_DB_PATH` wins; otherwise the cwd-relative `.codescope/` directory +/// is created and used — the same rule server mode applies when it is not given +/// a `--rootPath`. Extracted so `index-parallel` installs its finished database +/// at exactly the path the next command opens: the scheduler assembles its +/// output in a temp directory and only reported it as `main_db`, so the next +/// `get_graph_stats` answered 0 nodes for a project that had just been indexed +/// (measured twice on 2026-10-06). +fn resolve_cli_db_path() -> String { + let default_dir = ".codescope"; + if !Path::new(default_dir).exists() { + let _ = fs::create_dir_all(default_dir); + } + env::var("CODESCOPE_DB_PATH").unwrap_or_else(|_| format!("{}/codescope.db", default_dir)) +} + fn main() { // ── --version / -V: print version and exit ───────────────────── // Handled before any other argument processing so it works regardless @@ -43,12 +54,18 @@ fn main() { // ── discover-modules: codescope discover-modules ─── // Built-in scheduler entry: list top-level modules + their source-file // counts. Output schema is focused on scheduler use (no skipped_dirs/ - // skipped_files fields). See builtin-scheduler-design.md §4.2. + // skipped_files fields). // Exit code: 0 on success, 1 on missing dir. if args.len() >= 2 && args[1] == "discover-modules" { let dir_path = args.get(2).map(|s| s.as_str()).unwrap_or("."); let result = discover::discover_modules(dir_path); println!("{}", result); + let ok = serde_json::from_str::(&result) + .map(|v| v["ok"] == true) + .unwrap_or(false); + if !ok { + std::process::exit(1); + } return; } @@ -62,78 +79,253 @@ fn main() { let dir_path = args.get(2).map(|s| s.as_str()).unwrap_or("."); let result = discover::discover_files(dir_path); println!("{}", result); + let ok = serde_json::from_str::(&result) + .map(|v| v["ok"] == true) + .unwrap_or(false); + if !ok { + std::process::exit(1); + } return; } // ── index-parallel: codescope index-parallel [--workers N] [--parallel M] ── - // Built-in CPU-dynamic parallel indexer. Replaces codescope-parallel.sh. - // See builtin-scheduler-design.md §4. Dispatches one worker subprocess + // Built-in CPU-dynamic parallel indexer. Replaces the legacy + // scripts/legacy/codescope-parallel.sh. + // Dispatches one worker subprocess // per top-level module with proportional parse-worker allocation; failed // modules are quarantined via binary search. // Exit code: 0 on success (>=1 module indexed with nodes), 1 on failure. - // NOTE: This command uses Unix-specific shared memory (mmap) and is not - // available on Windows. Use `codescope index` instead on Windows. + // NOTE: the queue segment and the per-module DBs live in the platform + // temp directory and are shared through src/scheduler/mapped_file.rs, + // so this path runs on Windows as well as POSIX. if args.len() >= 2 && args[1] == "index-parallel" { - #[cfg(not(windows))] - { - let mut dir_path = ".".to_string(); - let mut total_workers: u32 = 0; - let mut parallel: u32 = 0; - - let mut i = 2; - while i < args.len() { - match args[i].as_str() { - "--workers" | "-w" => match args.get(i + 1) { - Some(v) => { - total_workers = v.parse().unwrap_or(0); - i += 2; - continue; - } - None => { - eprintln!("error: --workers requires a value"); - std::process::exit(2); - } - }, - "--parallel" | "-p" => match args.get(i + 1) { - Some(v) => { - parallel = v.parse().unwrap_or(0); - i += 2; - continue; - } - None => { - eprintln!("error: --parallel requires a value"); - std::process::exit(2); - } - }, - "--help" | "-h" => { - println!( - "Usage: codescope index-parallel [--workers N] [--parallel M]" - ); - println!(" Built-in CPU-dynamic parallel indexer."); - println!(" --workers N total parse-worker cores (default 8)"); - println!(" --parallel M max concurrent module workers (default 4)"); - return; + let mut dir_path = ".".to_string(); + let mut total_workers: u32 = 0; + let mut parallel: u32 = 0; + + let mut i = 2; + while i < args.len() { + match args[i].as_str() { + "--workers" | "-w" => match args.get(i + 1) { + Some(v) => { + total_workers = v.parse().unwrap_or(0); + i += 2; + continue; } - p => { - if !p.starts_with("--") { - dir_path = p.to_string(); - } - i += 1; + None => { + eprintln!("error: --workers requires a value"); + std::process::exit(2); } + }, + "--parallel" | "-p" => match args.get(i + 1) { + Some(v) => { + parallel = v.parse().unwrap_or(0); + i += 2; + continue; + } + None => { + eprintln!("error: --parallel requires a value"); + std::process::exit(2); + } + }, + "--help" | "-h" => { + println!("Usage: codescope index-parallel [--workers N] [--parallel M]"); + println!(" Built-in CPU-dynamic parallel indexer."); + println!(" --workers N total parse-worker cores (default 8)"); + println!(" --parallel M max concurrent module workers (default 4)"); + return; + } + p => { + if !p.starts_with("--") { + dir_path = p.to_string(); + } + i += 1; } } + } + + let result = scheduler::index_parallel(&dir_path, total_workers, parallel); + + // A scheduler-built DB is assembled by the merger from per-worker + // DBs, so it never receives the engine's post-index pass (knowledge + // layer, metrics, readiness flags, search index). Without this the + // CLI's own output looks complete while `search` silently degrades + // to its graph fallback — it cannot find functions at all — and + // `project_readiness` stays empty so `project_overview` reports no + // feature as ready. Measured: 0 rows in project_readiness and in + // code_fts/name_trgm before, 1 and 1793 after. The merge unifies + // the DB onto project id 1 (see merge_driver::unify_project), so + // the pass runs on that id here, while this process owns the DB. + let merged_db = serde_json::from_str::(&result) + .ok() + .filter(|v| v["ok"] == serde_json::Value::Bool(true)) + .and_then(|v| v["main_db"].as_str().map(|s| s.to_string())); + // Counts as they stand AFTER the pass below; the summary was built + // before it and would otherwise report a graph the caller cannot see. + let mut final_totals: Option<(u64, u64)> = None; + if let Some(main_db) = merged_db.as_deref() { + if ffi::init(main_db) == 0 { + let enhanced = ffi::enhance_project(1); + eprintln!( + "codescope: post-index pass on {}: {}", + main_db, + enhanced.chars().take(120).collect::() + ); + // Read the finished numbers while the engine still has the DB + // open (no second sqlite3 round trip). + if let Ok(v) = serde_json::from_str::(&ffi::get_graph_stats(1)) { + final_totals = v["total_nodes"].as_u64().zip(v["total_edges"].as_u64()); + } + ffi::shutdown(); + } else { + eprintln!( + "codescope: could not open {} for the post-index pass", + main_db + ); + } + } - let result = scheduler::index_parallel(&dir_path, total_workers, parallel); - println!("{}", result); - return; + // Install the finished database at the CLI path. + // + // The scheduler assembles its output in a temp directory and reports it + // as `main_db`; nothing copied it back, so `index-parallel` followed by + // any other command silently read an EMPTY database. The copy happens + // after the post-index pass released the database, so it is a plain + // file copy of a closed SQLite file. A stale WAL sidecar of the previous + // database must be removed first — SQLite would replay it against the + // new file and corrupt it. + if let Some(main_db) = merged_db.as_deref() { + let target = resolve_cli_db_path(); + if main_db != target { + for suffix in ["-wal", "-shm"] { + let sidecar = format!("{}{}", target, suffix); + if Path::new(&sidecar).exists() + && let Err(e) = fs::remove_file(&sidecar) + { + eprintln!("codescope: could not remove {}: {}", sidecar, e); + } + } + match fs::copy(main_db, &target) { + Ok(bytes) => eprintln!( + "codescope: index-parallel installed its database at {} ({} bytes)", + target, bytes + ), + Err(e) => eprintln!( + "codescope: FAILED to install the index at {}: {} — the result is only in {}", + target, e, main_db + ), + } + } } - #[cfg(windows)] - { + + // Report the graph the run actually produced. The scheduler's summary + // counts what the workers wrote; the post-index pass above then adds + // the knowledge layer's edges — a 185-file run summed 1427 edges while + // the DB it handed back answered 1850, so `index-parallel` disagreed + // with the next `get_graph_stats` call on its own output. Per-module + // numbers in `modules[]` still describe the workers. + let result = match ( + serde_json::from_str::(&result), + final_totals, + ) { + (Ok(mut v), Some((nodes, edges))) => { + v["total_nodes"] = nodes.into(); + v["total_edges"] = edges.into(); + v.to_string() + } + _ => result, + }; + + println!("{}", result); + return; + } + + // ── Parse-failure maintenance ──────────────────────────────── + // codescope parse-failures [--db ] [--limit ] + // codescope reset-failures [--db ] + // parse_failures lists the files the indexer could not parse, with the + // reason and the retry count that drives the fail-fast skip. These two + // subcommands are the only supported way to read and clear it — they are + // what the store comments and the README refer to. + if args.len() >= 2 && (args[1] == "parse-failures" || args[1] == "reset-failures") { + let reset = args[1] == "reset-failures"; + let mut db_path = std::env::var("CODESCOPE_DB_PATH") + .unwrap_or_else(|_| ".codescope/codescope.db".to_string()); + let mut limit: i32 = 100; + let mut i = 2; + while i < args.len() { + match args[i].as_str() { + "--db" => match args.get(i + 1) { + Some(v) => { + db_path = v.clone(); + i += 2; + continue; + } + None => { + eprintln!("error: --db requires a value"); + std::process::exit(2); + } + }, + "--limit" => match args.get(i + 1).map(|v| v.parse::()) { + Some(Ok(v)) => { + limit = v; + i += 2; + continue; + } + Some(Err(_)) => { + eprintln!("error: --limit requires an integer"); + std::process::exit(2); + } + None => { + eprintln!("error: --limit requires a value"); + std::process::exit(2); + } + }, + "--help" | "-h" => { + println!("Usage: codescope parse-failures [--db ] [--limit ]"); + println!(" codescope reset-failures [--db ]"); + println!( + " Read or clear parse_failures: the files the indexer could not parse," + ); + println!(" with their reason and retry count. Clearing them re-arms the"); + println!(" fail-fast skip for those files."); + return; + } + other => { + eprintln!("error: unexpected argument '{}' (try --help)", other); + std::process::exit(2); + } + } + } + if ffi::init(&db_path) != 0 { + eprintln!("parse-failures: engine init failed (db={})", db_path); + std::process::exit(1); + } + let pid = ffi::get_latest_project_id(); + if pid == 0 { eprintln!( - "error: index-parallel is not available on Windows. Use `codescope index` instead." + "parse-failures: no project in {} — index something first", + db_path ); + ffi::shutdown(); + std::process::exit(1); + } + let result = if reset { + ffi::reset_parse_failures(pid) + } else { + ffi::get_parse_failures(pid, limit) + }; + ffi::shutdown(); + println!("{}", result); + // An ok:false envelope must fail the process, or a scripted caller + // treats a broken read as an empty result. + let ok = serde_json::from_str::(&result) + .map(|v| v["ok"] == true) + .unwrap_or(false); + if !ok { std::process::exit(1); } + return; } // ── Force-index mode: codescope force-index [...] ─ @@ -210,8 +402,27 @@ fn main() { }); let result = tools::execute(pid, "force_index_files", &tool_args); println!("{}", result); + // Both an explicit ok:false AND an "ok:true, files_indexed:0" + // empty run must fail the process: exit 0 here makes shell/CI + // callers treat it as an empty success — the same hazard the + // worker --file-list path guards against below. The engine + // reports ok:true for an empty run (engine_index_files.cpp:144 + // jobs.empty() early return, and the writer path when every file + // fails to parse — empty read, unavailable grammar), so the ok flag + // alone is not a sufficient success predicate. Re-runs of a healthy + // index always report files_indexed > 0 (unchanged files are + // re-parsed and counted), so requiring it is safe. Note that this + // path applies NO fail-fast skip: a forced file is always + // re-attempted, so a previously failed file still counts as a parse + // attempt rather than being dropped silently. + let ok = serde_json::from_str::(&result) + .map(|v| v["ok"] == true && v["files_indexed"].as_u64().unwrap_or(0) > 0) + .unwrap_or(false); ffi::shutdown(); + if !ok { + std::process::exit(1); + } return; } @@ -230,21 +441,35 @@ fn main() { let project_id_arg = args.get(6).map(|s| s.as_str()).unwrap_or("0"); // Check for --file-list argument (path to JSON file containing file list) - let file_list = if args.len() >= 8 && args[7] == "--file-list" { - args.get(8).map(|s| { - // Read file list from the specified file - let path = s.as_str(); - match std::fs::read_to_string(path) { - Ok(content) => content, - Err(e) => { - eprintln!( - "codescope worker: failed to read file-list from {}: {}", - path, e - ); - String::new() + let file_list: Option = if args.len() >= 8 && args[7] == "--file-list" { + match args.get(8) { + Some(s) => { + // Read file list from the specified file. A read failure + // must fail the worker: substituting "" makes the engine + // return ok:false while we exit 0, which the scheduler + // counts as an empty success — and quarantine bisection + // treats exit 0 as healthy, so it can skip past a crasher. + let path = s.as_str(); + match std::fs::read_to_string(path) { + Ok(content) => Some(content), + Err(e) => { + eprintln!( + "codescope worker: failed to read file-list from {}: {} [module=scheduler, method=worker]", + path, e + ); + ffi::shutdown(); + std::process::exit(1); + } } } - }) + None => { + eprintln!( + "codescope worker: --file-list requires a path [module=scheduler, method=worker]" + ); + ffi::shutdown(); + std::process::exit(1); + } + } } else { None }; @@ -271,7 +496,9 @@ fn main() { ); let result = if let Some(files_json) = file_list { - ffi::index_files(pid, &files_json) + // Scheduler-driven worker: honour the fail-fast skip, exactly like + // ffi::index_project does for the whole-directory path. + ffi::index_files(pid, &files_json, false) } else if lang_filter.is_empty() { ffi::index_project(pid, dir_path, std::ptr::null()) } else { @@ -294,7 +521,6 @@ fn main() { // peer) via reset_all_stale, and exits when every chunk is // DONE/FAILED. Each worker owns its OWN DB — no shared-DB corruption. // Requires the `chunk_queue` module (see scheduler/chunk_queue.rs). - #[cfg(not(windows))] if args.len() >= 7 && args[1] == "chunk-worker" { let shm_path = args[2].as_str(); let worker_id: u32 = args[3].parse().unwrap_or(0); @@ -341,22 +567,19 @@ fn main() { } }; - // Forced project_id (mirrors the static path); falls back to a - // fresh project if 0. Each worker's DB is independent, so a single - // project per worker keeps ids consistent within the DB. - let pid = if project_id > 0 { - project_id - } else { - let new_pid = ffi::create_project(".", &format!("chunk-worker-{}", worker_id)); - if new_pid == 0 { - eprintln!( - "chunk-worker: failed to create project [module=scheduler, method=chunk_worker]" - ); - ffi::shutdown(); - std::process::exit(1); - } - new_pid - }; + // The scheduler always assigns a unique non-zero project_id per + // worker ((worker_id + 1)) and passes it on argv. A zero here means + // the invocation is broken; silently minting a fresh project would + // create a ghost project whose id the merge phase cannot map, so + // fail loudly instead of indexing into an orphan. + if project_id == 0 { + eprintln!( + "chunk-worker: project_id is 0 (must be assigned by the scheduler) [module=scheduler, method=chunk_worker]" + ); + ffi::shutdown(); + std::process::exit(1); + } + let pid = project_id; // Watchdog window for reclaiming orphaned chunks (crashed peer). // Set to match the scheduler's per-worker timeout so a chunk is @@ -371,6 +594,7 @@ fn main() { let mut total_edges: u64 = 0; let mut files_indexed: u64 = 0; let mut chunks_done: u32 = 0; + let mut chunks_failed: u32 = 0; loop { match queue.claim_next(worker_id) { @@ -380,6 +604,7 @@ fn main() { None => { queue.mark_failed(idx); chunks_done += 1; + chunks_failed += 1; continue; } }; @@ -398,10 +623,14 @@ fn main() { eprintln!("chunk-worker: serialize chunk {} failed: {}", idx, e); queue.mark_failed(idx); chunks_done += 1; + chunks_failed += 1; continue; } }; - let result = ffi::index_files(pid, &files_json); + // Chunk worker: same policy as the other scheduler paths — + // a chunk whose files keep failing must not be retried + // forever by every claimant. + let result = ffi::index_files(pid, &files_json, false); if let Ok(v) = serde_json::from_str::(&result) { if v["ok"] == true { total_nodes += v["total_nodes"].as_u64().unwrap_or(0); @@ -415,6 +644,7 @@ fn main() { idx, err ); queue.mark_failed(idx); + chunks_failed += 1; } } else { eprintln!( @@ -422,6 +652,7 @@ fn main() { idx ); queue.mark_failed(idx); + chunks_failed += 1; } chunks_done += 1; } @@ -440,19 +671,20 @@ fn main() { } let result_json = json!({ - "ok": true, + "ok": chunks_failed == 0, "worker_id": worker_id, "total_nodes": total_nodes, "total_edges": total_edges, "files_indexed": files_indexed, "chunks_done": chunks_done, + "chunks_failed": chunks_failed, }); println!("{}", result_json); ffi::shutdown(); eprintln!( - "chunk-worker {}: done (nodes={} edges={} files={} chunks={})", - worker_id, total_nodes, total_edges, files_indexed, chunks_done + "chunk-worker {}: done (nodes={} edges={} files={} chunks={} failed={})", + worker_id, total_nodes, total_edges, files_indexed, chunks_done, chunks_failed ); return; } @@ -466,13 +698,8 @@ fn main() { serde_json::Value::Null }; - // Use same persistent DB path as server mode - let default_dir = ".codescope"; - if !Path::new(default_dir).exists() { - let _ = fs::create_dir_all(default_dir); - } - let default_db = format!("{}/codescope.db", default_dir); - let db_path = env::var("CODESCOPE_DB_PATH").unwrap_or(default_db); + // Same persistent DB path as server mode. + let db_path = resolve_cli_db_path(); if ffi::init(&db_path) != 0 { eprintln!("codescope: engine init failed"); @@ -522,15 +749,7 @@ fn main() { eprintln!("codescope: failed to create {}: {}", codescope_dir, e); }); } - let db = format!("{}/codescope.db", codescope_dir); - // Set env var so worker subprocesses (spawned by tools::execute) - // inherit the correct DB path. - // Safety: this runs before any threads are spawned (server.run() - // starts later), so there is no data race on the environment. - unsafe { - env::set_var("CODESCOPE_DB_PATH", &db); - } - db + format!("{}/codescope.db", codescope_dir) } else { let default_dir = ".codescope"; let default_db = format!("{}/codescope.db", default_dir); @@ -540,6 +759,12 @@ fn main() { env::var("CODESCOPE_DB_PATH").unwrap_or(default_db) }; + // Publish the resolved path once, before any thread is spawned. Worker + // subprocesses receive it explicitly via `Command::env`, so there is no + // need to mutate the process environment (which is `unsafe` in Rust + // 2024). Tool handlers read it via `tools::db_path()`. + tools::set_db_path(db_path.clone()); + eprintln!("codescope: initializing with db={}", db_path); let rc = ffi::init(&db_path); diff --git a/server/src/mcp/protocol.rs b/server/src/mcp/protocol.rs index ef3703c..7b2c150 100644 --- a/server/src/mcp/protocol.rs +++ b/server/src/mcp/protocol.rs @@ -1,5 +1,17 @@ use serde::{Deserialize, Serialize}; +// ── Protocol version ─────────────────────────────────────────── + +/// The MCP revision this server speaks, declared once. +/// +/// The lifecycle requires the server to answer with the client's requested +/// version when it supports it, and with a version it does support otherwise. +/// This server supports exactly one revision, so "always answer this one" IS +/// that rule — the constant exists so the version is stated in one place, and +/// `handle_initialize` reads what the client asked for and logs a mismatch so +/// the compatibility decision is visible rather than implicit. +pub const SUPPORTED_PROTOCOL_VERSION: &str = "2024-11-05"; + // ── JSON-RPC 2.0 message types ───────────────────────────────── #[derive(Debug, Deserialize)] @@ -88,6 +100,14 @@ pub struct ListToolsResult { } #[derive(Debug, Serialize)] +// The tool-call result is the one response type that carried the protocol's +// naming rule by hand and got it wrong: `is_error` went out as snake_case, +// while MCP requires `isError` — so a spec-compliant client, which reads +// `isError`, saw NO failure flag at all, not even for results that were +// already being flagged. Every other multi-word field in this file is renamed +// explicitly (`inputSchema`, `listChanged`, `type`); this struct now states the +// rule where the field lives. +#[serde(rename_all = "camelCase")] pub struct CallToolResult { pub content: Vec, #[serde(skip_serializing_if = "Option::is_none")] @@ -106,10 +126,42 @@ mod tests { use super::*; use serde_json::json; + #[test] + fn test_call_tool_result_uses_the_wire_name_is_error() { + // MCP's CallToolResult field is `isError`. It was serialized as + // `is_error`, so a spec-compliant client reading `isError` saw no + // failure flag at all — including for results that were already + // flagged as failures. Pinned here because the payload is what clients + // actually parse, not the Rust field name. + let err = CallToolResult { + content: vec![TextContent { + content_type: "text", + text: "{\"error\":\"boom\"}".into(), + }], + is_error: Some(true), + }; + let v = serde_json::to_value(&err).expect("serialize"); + assert_eq!(v["isError"], serde_json::json!(true)); + assert!(v.get("is_error").is_none(), "snake_case must not appear"); + + // A successful result omits the field entirely (the client's default + // is "not an error"), so the wire shape is unchanged for successes. + let ok = CallToolResult { + content: vec![TextContent { + content_type: "text", + text: "{}".into(), + }], + is_error: None, + }; + let v = serde_json::to_value(&ok).expect("serialize"); + assert!(v.get("isError").is_none()); + assert!(v.get("is_error").is_none()); + } + #[test] fn test_initialize_result_serde() { let r = InitializeResult { - protocol_version: "2024-11-05".into(), + protocol_version: SUPPORTED_PROTOCOL_VERSION.into(), capabilities: ServerCapabilities { tools: ToolCapability { list_changed: true }, }, @@ -156,3 +208,48 @@ mod tests { assert_eq!(json["content"][0]["text"], "hello"); } } + +/// Property-based coverage of the wire contract (code_rules §4). +/// +/// The CHANGELOG records a long tail of malformed JSON produced by +/// concatenating payloads by hand; serde_json is now the single producer. These +/// properties pin the two invariants that hand-building repeatedly broke: the +/// emitted message is always parseable JSON, and arbitrary text (quotes, +/// backslashes, control characters, non-BMP code points) survives the round +/// trip byte-for-byte. +#[cfg(test)] +mod proptests { + use super::*; + use proptest::prelude::*; + + proptest! { + #[test] + fn call_tool_result_serializes_to_parseable_json(text in any::()) { + let result = CallToolResult { + content: vec![TextContent { + content_type: "text", + text: text.clone(), + }], + is_error: Some(true), + }; + let wire = serde_json::to_string(&result).expect("serialize"); + let parsed: serde_json::Value = + serde_json::from_str(&wire).expect("tool result must be parseable JSON"); + prop_assert_eq!(parsed["content"][0]["text"].as_str(), Some(text.as_str())); + prop_assert_eq!(parsed["isError"].as_bool(), Some(true)); + } + + #[test] + fn json_rpc_error_round_trips_arbitrary_messages(message in any::()) { + let err = JsonRpcError { + code: -32603, + message: message.clone(), + data: None, + }; + let wire = serde_json::to_string(&err).expect("serialize"); + let parsed: serde_json::Value = + serde_json::from_str(&wire).expect("error must be parseable JSON"); + prop_assert_eq!(parsed["message"].as_str(), Some(message.as_str())); + } + } +} diff --git a/server/src/mcp/server.rs b/server/src/mcp/server.rs index a2b6513..22f9291 100644 --- a/server/src/mcp/server.rs +++ b/server/src/mcp/server.rs @@ -169,8 +169,27 @@ impl Server { } } + // The client's requested revision, if it sent one. This server speaks + // exactly one version, so the answer is always + // SUPPORTED_PROTOCOL_VERSION — but a client that asked for something + // else deserves to see it acknowledged rather than silently ignored, + // and the log is where that decision is recorded. + if let Some(requested) = params + .as_ref() + .and_then(|p| p.get("protocolVersion")) + .and_then(|v| v.as_str()) + && requested != SUPPORTED_PROTOCOL_VERSION + { + eprintln!( + "initialize: client requested MCP protocol {}; this server speaks {} — replying with \ + the version it supports, the client decides whether to continue (see the MCP \ + lifecycle's version negotiation)", + requested, SUPPORTED_PROTOCOL_VERSION + ); + } + let result = InitializeResult { - protocol_version: "2024-11-05".to_string(), + protocol_version: SUPPORTED_PROTOCOL_VERSION.to_string(), capabilities: ServerCapabilities { tools: ToolCapability { list_changed: true }, }, @@ -208,8 +227,45 @@ impl Server { // ── Call Tool ─────────────────────────────────────────────── + /// Give an indexing call a project to write into when the session has none. + /// + /// Every row the engine writes carries a `project_id`, and the knowledge / + /// evidence layer is scoped by it: with `project_id == 0` the registry's + /// evidence probe is skipped (`entity_count: 0` next to a non-empty graph) + /// and `get_project_state` answers "project state not yet built". A client + /// that goes straight to `force_index_files` / `index_file` without + /// declaring a workspace root in `initialize` (`rootUri` / `roots`) used to + /// leave the whole index under project 0 — no `projects` row, invisible to + /// every project-scoped tool for the rest of the session. + /// + /// The CLI's `force-index` already bootstraps a project row before it + /// indexes; this is the same step for the MCP path, using the directory + /// being indexed as the project root so `get_project_id_by_path` can reuse + /// it on the next session. + fn ensure_project_for_indexing(&mut self, tool_name: &str, args: &serde_json::Value) { + if self.project_id != 0 { + return; + } + let Some(root) = indexing_root(tool_name, args) else { + return; + }; + let name = std::path::Path::new(&root) + .file_name() + .and_then(|n| n.to_str()) + .filter(|n| !n.is_empty()) + .unwrap_or("default"); + let pid = ffi::create_project(&root, name); + if pid > 0 { + self.project_id = pid; + eprintln!( + "codescope: created project {} (id={}) for {} [module=mcp, method=handle_call_tool]", + name, pid, tool_name + ); + } + } + fn handle_call_tool( - &self, + &mut self, params: Option, ) -> Result { let params = params.ok_or_else(|| JsonRpcError { @@ -229,16 +285,18 @@ impl Server { .cloned() .unwrap_or(serde_json::Value::Null); + self.ensure_project_for_indexing(tool_name, &tool_args); + let result = tools::execute(self.project_id, tool_name, &tool_args); // Note: background enhancement after `scan_project` is triggered inside // `tools::execute`, which is the natural owner for tool-specific logic. - // Determine if the result indicates an error (JSON with non-null "error" key) - let is_error = serde_json::from_str::(&result) - .ok() - .and_then(|v| v.get("error").cloned()) - .and_then(|e| if e.is_null() { None } else { Some(true) }); + let is_error = if tool_result_is_error(&result) { + Some(true) + } else { + None + }; let content = vec![TextContent { content_type: "text", @@ -292,3 +350,97 @@ fn json_response( }), } } + +/// Root directory an indexing tool is about to write, or `None` when the tool +/// does not index by path. Used only to bootstrap a project (see +/// [`Server::ensure_project_for_indexing`]). +/// +/// `force_index_files` carries `paths` (first entry wins) with a single-`path` +/// convenience form; `index_file` carries a file, whose directory becomes the +/// project root so repeated calls for files in one tree share a project. +fn indexing_root(tool_name: &str, args: &serde_json::Value) -> Option { + match tool_name { + "force_index_files" => args["paths"] + .as_array() + .and_then(|a| a.first()) + .and_then(|v| v.as_str()) + .or_else(|| args["path"].as_str()) + .map(str::to_string), + "index_file" => args["file_path"].as_str().map(|p| { + std::path::Path::new(p) + .parent() + .filter(|d| !d.as_os_str().is_empty()) + .map(|d| d.to_string_lossy().into_owned()) + .unwrap_or_else(|| p.to_string()) + }), + _ => None, + } +} + +/// Whether a tool result must be reported to the client as a failure. +/// +/// MCP has exactly one protocol-level failure signal (`isError`) while the +/// payloads carry their own verdict in the data, so this decides how the two +/// line up. The old rule was "the parsed result has a non-null `error` key", +/// which missed half of the codebase's own convention: the schedulers report +/// "this run did not produce a usable index" as a top-level `"ok": false` — +/// an empty project, a failed worker, or a failed merge, with the cause nested +/// under `merge.error` or in a `note` — so a client watching `isError` saw a +/// failed run as a successful call. Two real payloads were being reported that +/// way: `{"ok":false,...,"note":"no source modules found"}` (scheduler/mod.rs) +/// and the run summary `{"ok":complete,...,"merge":{"error":...}}`. +/// +/// Non-JSON is a failure too: every handler's contract is JSON out, so a +/// result that does not parse is a broken tool response rather than a success. +/// +/// Deliberately narrow: only a non-null TOP-LEVEL `error` and a top-level +/// `ok == false` count. A payload that merely mentions the word error +/// somewhere deeper is not a failure. +fn tool_result_is_error(result: &str) -> bool { + match serde_json::from_str::(result) { + Ok(v) => { + let error_set = v.get("error").map(|e| !e.is_null()).unwrap_or(false); + let not_ok = v + .get("ok") + .map(|o| o == &serde_json::Value::Bool(false)) + .unwrap_or(false); + error_set || not_ok + } + Err(_) => true, + } +} + +#[cfg(test)] +mod tests { + use super::tool_result_is_error; + + #[test] + fn test_tool_result_is_error_covers_both_failure_conventions() { + // Success shapes have to stay successes — `detect_changes` returns a + // literal null error on the happy path, and many tools return arrays. + assert!(!tool_result_is_error(r#"{"error":null,"modified":[]}"#)); + assert!(!tool_result_is_error( + r#"{"claim_id":1,"verdict":"Unknown"}"# + )); + assert!(!tool_result_is_error(r#"[{"category":"drift"}]"#)); + assert!(!tool_result_is_error(r#"{"ok":true,"files":[]}"#)); + // `"error"` as ordinary data deeper down is not a failure signal. + assert!(!tool_result_is_error(r#"{"files":[{"error":"parse"}]}"#)); + + // The convention that was being missed: a run that says it did not + // produce a usable index, with no top-level error key. + assert!(tool_result_is_error( + r#"{"ok":false,"complete":false,"modules":[],"note":"no source modules found"}"# + )); + assert!(tool_result_is_error( + r#"{"ok":false,"complete":false,"merge":{"merged":false,"error":"boom"},"fail":2}"# + )); + + // And the one the old rule did catch. + assert!(tool_result_is_error( + r#"{"error":"claim field is required"}"# + )); + // A result that is not JSON at all is a broken response, not a success. + assert!(tool_result_is_error("panic: index out of bounds")); + } +} diff --git a/server/src/mcp/transport.rs b/server/src/mcp/transport.rs index b9ef5ec..37adcfe 100644 --- a/server/src/mcp/transport.rs +++ b/server/src/mcp/transport.rs @@ -33,56 +33,30 @@ const MAX_DRAIN_BYTES: u64 = 16 << 20; // 16 MiB /// Returns the parsed Request, Eof on clean EOF, or ParseError on bad JSON. pub fn read_message() -> io::Result { let stdin = io::stdin(); - let mut line = String::new(); - // Limit line length to avoid unbounded memory consumption (DoS). - // take() ensures at most MAX_LINE_LEN+1 bytes are read into the - // buffer: MAX_LINE_LEN bytes of payload + 1 byte to detect - // overflow. If the line is longer, read_line returns without a - // trailing '\n' and we drain the remainder below (M8). - let mut reader = stdin.lock().take((MAX_LINE_LEN + 1) as u64); - let n = reader.read_line(&mut line)?; - if n == 0 { - return Ok(ReadResult::Eof); // clean EOF - } - // M8: If we hit the take limit without a trailing '\n', the line - // was longer than MAX_LINE_LEN. The take() reader is exhausted but - // the underlying stdin cursor still sits in the MIDDLE of that - // line — without draining, the next read_message() would resume - // from there and produce a cascade of ParseErrors, putting the - // JSON-RPC protocol permanently out of sync. Drop the take - // wrapper (releasing the stdin lock), drain stdin until the next - // '\n' or EOF, then report a parse error so the caller continues - // at a clean protocol boundary. - if !line.ends_with('\n') && n > MAX_LINE_LEN { - drop(reader); - drain_stdin_until_newline()?; - let error_resp = serde_json::json!({ - "jsonrpc": "2.0", - "id": null, - "error": { - "code": -32700, - "message": "Parse error", - "data": { - "detail": format!("line exceeds {} byte limit", MAX_LINE_LEN), - } - } - }); - let json = serde_json::to_string(&error_resp).unwrap_or_default(); - let stdout = io::stdout(); - let mut handle = stdout.lock(); - let _ = writeln!(handle, "{}", json); - let _ = handle.flush(); - return Ok(ReadResult::ParseError); - } - drop(reader); - let trimmed = line.trim(); - if trimmed.is_empty() { - return Ok(ReadResult::Eof); - } - match serde_json::from_str::(trimmed) { - Ok(req) => Ok(ReadResult::Msg(req)), - Err(e) => { - // JSON-RPC requires a Parse Error response (-32700) + loop { + let mut line = String::new(); + // Limit line length to avoid unbounded memory consumption (DoS). + // take() ensures at most MAX_LINE_LEN+1 bytes are read into the + // buffer: MAX_LINE_LEN bytes of payload + 1 byte to detect + // overflow. If the line is longer, read_line returns without a + // trailing '\n' and we drain the remainder below (M8). + let mut reader = stdin.lock().take((MAX_LINE_LEN + 1) as u64); + let n = reader.read_line(&mut line)?; + if n == 0 { + return Ok(ReadResult::Eof); // clean EOF + } + // M8: If we hit the take limit without a trailing '\n', the line + // was longer than MAX_LINE_LEN. The take() reader is exhausted but + // the underlying stdin cursor still sits in the MIDDLE of that + // line — without draining, the next read_message() would resume + // from there and produce a cascade of ParseErrors, putting the + // JSON-RPC protocol permanently out of sync. Drop the take + // wrapper (releasing the stdin lock), drain stdin until the next + // '\n' or EOF, then report a parse error so the caller continues + // at a clean protocol boundary. + if !line.ends_with('\n') && n > MAX_LINE_LEN { + drop(reader); + drain_stdin_until_newline()?; let error_resp = serde_json::json!({ "jsonrpc": "2.0", "id": null, @@ -90,8 +64,7 @@ pub fn read_message() -> io::Result { "code": -32700, "message": "Parse error", "data": { - "detail": format!("{}", e), - "raw": trimmed + "detail": format!("line exceeds {} byte limit", MAX_LINE_LEN), } } }); @@ -100,8 +73,41 @@ pub fn read_message() -> io::Result { let mut handle = stdout.lock(); let _ = writeln!(handle, "{}", json); let _ = handle.flush(); - Ok(ReadResult::ParseError) + return Ok(ReadResult::ParseError); + } + drop(reader); + let trimmed = line.trim(); + if trimmed.is_empty() { + // A blank line is NOT EOF. Only `n == 0` above is a clean + // disconnect. Treating an empty line as EOF meant a single stray + // newline — a separator-style client, or a trailing blank line — + // terminated the long-running server. Skip it and keep reading. + continue; } + return match serde_json::from_str::(trimmed) { + Ok(req) => Ok(ReadResult::Msg(req)), + Err(e) => { + // JSON-RPC requires a Parse Error response (-32700) + let error_resp = serde_json::json!({ + "jsonrpc": "2.0", + "id": null, + "error": { + "code": -32700, + "message": "Parse error", + "data": { + "detail": format!("{}", e), + "raw": trimmed + } + } + }); + let json = serde_json::to_string(&error_resp).unwrap_or_default(); + let stdout = io::stdout(); + let mut handle = stdout.lock(); + let _ = writeln!(handle, "{}", json); + let _ = handle.flush(); + Ok(ReadResult::ParseError) + } + }; } } @@ -122,12 +128,96 @@ fn drain_stdin_until_newline() -> io::Result<()> { Ok(()) } +/// Cap on a single outbound JSON-RPC message. Measured `get_graph` / +/// `graph_query` payloads reached 2–3 MiB on real projects (prior review +/// #30): a client with a smaller message cap fails the whole session, so +/// oversized tool results are replaced with an error the client can act +/// on (paginate with smaller limits) instead of a frame it cannot parse. +/// 1 MiB is comfortably above every normal tool result and below the +/// usual client caps. +const MAX_MESSAGE_BYTES: usize = 1 << 20; // 1 MiB + /// Writes a JSON-RPC response to stdout (one line per message). +/// Messages larger than [`MAX_MESSAGE_BYTES`] are replaced by an error +/// object (never truncated mid-JSON) so the client always receives a +/// parseable frame. The original `id` is preserved so the client can +/// still correlate the error with its request (JSON-RPC 2.0). pub fn write_message(msg: &serde_json::Value) -> io::Result<()> { let stdout = io::stdout(); let mut handle = stdout.lock(); let json = serde_json::to_string(msg)?; - writeln!(handle, "{}", json)?; + let out: String = if json.len() > MAX_MESSAGE_BYTES { + serde_json::to_string(&serde_json::json!({ + "jsonrpc": "2.0", + "id": msg.get("id").cloned().unwrap_or(serde_json::Value::Null), + "error": { + "code": -32000, + "message": format!( + "tool response {} bytes exceeds the {} byte message cap; \ + re-run with smaller node_limit/edge_limit or a narrower filter \ + [module=mcp, method=write_message]", + json.len(), + MAX_MESSAGE_BYTES + ), + } + }))? + } else { + json + }; + writeln!(handle, "{}", out)?; handle.flush()?; Ok(()) } + +#[cfg(test)] +mod tests { + use super::MAX_MESSAGE_BYTES; + + /// Regression: an oversized tool result must be replaced by a JSON-RPC + /// error that keeps the request `id`, not truncated mid-JSON and not + /// emitted without an `id` (the client could not correlate it). + #[test] + fn test_oversized_message_is_replaced_with_error_keeping_id() { + // Build a response whose serialized form exceeds the cap. + let big = "x".repeat(MAX_MESSAGE_BYTES + 1024); + let msg = serde_json::json!({ + "jsonrpc": "2.0", + "id": 42, + "result": { "payload": big }, + }); + let json = serde_json::to_string(&msg).unwrap(); + assert!(json.len() > MAX_MESSAGE_BYTES); + + // Mirror write_message's replacement branch (the IO write itself is + // not under test — stdout is not capturable here). + let out: String = if json.len() > MAX_MESSAGE_BYTES { + serde_json::to_string(&serde_json::json!({ + "jsonrpc": "2.0", + "id": msg.get("id").cloned().unwrap_or(serde_json::Value::Null), + "error": { + "code": -32000, + "message": "exceeds cap", + } + })) + .unwrap() + } else { + json + }; + + let v: serde_json::Value = serde_json::from_str(&out).unwrap(); + assert_eq!(v["id"], 42, "the request id must be preserved"); + assert!(v["error"].is_object(), "an error object must be present"); + assert!( + v["result"].is_null(), + "the oversized result must be dropped" + ); + } + + /// A normal-size message must pass through untouched. + #[test] + fn test_normal_message_is_not_replaced() { + let msg = serde_json::json!({"jsonrpc": "2.0", "id": 1, "result": 2}); + let json = serde_json::to_string(&msg).unwrap(); + assert!(json.len() < MAX_MESSAGE_BYTES); + } +} diff --git a/server/src/scheduler/chunk_plan.rs b/server/src/scheduler/chunk_plan.rs index 9f4e3c4..a8c8c1e 100644 --- a/server/src/scheduler/chunk_plan.rs +++ b/server/src/scheduler/chunk_plan.rs @@ -116,13 +116,17 @@ fn group_by_prefix(files: &[FileEntry], depth: usize) -> Vec<(String, Vec Vec { assert!(!files.is_empty(), "plan_chunks: empty file list"); assert!( @@ -151,8 +155,7 @@ pub fn plan_chunks(files: &[FileEntry], target_bytes: u64, max_bytes: u64) -> Ve .collect(); for (prefix, cluster_entries) in &clusters { - // Sort cluster entries by their original index so the chunk's file range - // is contiguous in the original ordering. + // Cluster entries ordered by their position in `files`. let mut sorted_entries: Vec<&FileEntry> = cluster_entries.iter().collect(); sorted_entries.sort_by_key(|e| { path_to_idx @@ -161,81 +164,52 @@ pub fn plan_chunks(files: &[FileEntry], target_bytes: u64, max_bytes: u64) -> Ve .unwrap_or(usize::MAX) }); - // Split large clusters that exceed max_bytes into multiple chunks. - let mut chunk_start: usize = 0; - let mut chunk_bytes: u64 = 0; - - for entry in &sorted_entries { - let idx = path_to_idx[entry.path.as_str()]; - - // If this single file exceeds max_bytes, it gets its own chunk. - // If adding this file would exceed max_bytes AND the current - // chunk already has content, finalise the current chunk first. - // The guard is `chunk_bytes > 0` (current chunk non-empty), - // NOT `!chunks.is_empty()`: the latter gates on whether ANY - // chunk has been emitted across the whole call, which has - // nothing to do with whether the current chunk should close. - if entry.size > max_bytes || (chunk_bytes > 0 && chunk_bytes + entry.size > max_bytes) { - // Finalise the current chunk. - let count = sorted_entries[chunk_start..] - .iter() - .position(|e| { - let ei = path_to_idx[e.path.as_str()]; - ei >= idx - }) - .unwrap_or(sorted_entries.len() - chunk_start); - if count > 0 { - chunks.push(Chunk { - file_start: path_to_idx[sorted_entries[chunk_start].path.as_str()], - file_count: count, - total_bytes: chunk_bytes, - module_id: 0, - dir_prefix: prefix.clone(), - }); + // A Chunk is a contiguous `(file_start, file_count)` range, so that + // range must contain the cluster's indices and nothing else. Grouping + // by prefix does NOT guarantee it: a cluster whose files are + // interleaved with another cluster's has a non-contiguous index set + // (e.g. `a/aa.rs`=0, `a/mm/x.rs`=1, `a/zz.rs`=2 gives cluster "a" the + // indices {0,2} while "a/mm" holds 1). Emitting `[min_index, count)` + // then covers a foreign file AND leaves the cluster's own later + // member uncovered — one file indexed twice, another never indexed. + // So walk the cluster in index order and cut a chunk at every gap. + let mut i = 0usize; + while i < sorted_entries.len() { + let start_idx = path_to_idx[sorted_entries[i].path.as_str()]; + let mut end = i; // exclusive + let mut bytes: u64 = 0; + + while end < sorted_entries.len() { + let entry = sorted_entries[end]; + let idx = path_to_idx[entry.path.as_str()]; + + // Gap: the next index belongs to another cluster, so the + // chunk has to end here. + if end > i && idx != path_to_idx[sorted_entries[end - 1].path.as_str()] + 1 { + break; } - chunk_start += count; - chunk_bytes = 0; - } - - chunk_bytes += entry.size; - - // Greedy cut: if we've reached target, finalise the chunk and reset. - if chunk_bytes >= target_bytes { - let chunk_end = chunk_start - + sorted_entries[chunk_start..] - .iter() - .position(|e| { - let ei = path_to_idx[e.path.as_str()]; - ei > idx - }) - .unwrap_or(sorted_entries.len() - chunk_start); - let count = chunk_end - chunk_start; - if count > 0 { - chunks.push(Chunk { - file_start: path_to_idx[sorted_entries[chunk_start].path.as_str()], - file_count: count, - total_bytes: chunk_bytes, - module_id: 0, - dir_prefix: prefix.clone(), - }); + // Hard ceiling. `end > i` keeps a single file larger than + // max_bytes in a chunk of its own rather than emitting an + // empty chunk (which would also stall the walk). + if end > i && bytes + entry.size > max_bytes { + break; + } + bytes += entry.size; + end += 1; + // Greedy cut once the target size is reached. + if bytes >= target_bytes { + break; } - chunk_start = chunk_end; - chunk_bytes = 0; } - } - // Flush remaining entries in this cluster. - if chunk_bytes > 0 { - let remaining = sorted_entries.len() - chunk_start; - if remaining > 0 { - chunks.push(Chunk { - file_start: path_to_idx[sorted_entries[chunk_start].path.as_str()], - file_count: remaining, - total_bytes: chunk_bytes, - module_id: 0, - dir_prefix: prefix.clone(), - }); - } + chunks.push(Chunk { + file_start: start_idx, + file_count: end - i, + total_bytes: bytes, + module_id: 0, + dir_prefix: prefix.clone(), + }); + i = end; } } @@ -359,6 +333,69 @@ mod tests { assert_eq!(groups[1].0, "tests"); } + #[test] + fn test_plan_chunks_interleaved_clusters_cover_each_file_once() { + // Regression: a directory cluster whose files are interleaved with + // another cluster's used to be emitted as a single + // `[min_index, count)` range. Sorting these by path gives + // a/aa.rs(0) a/mm/x.rs(1) a/zz.rs(2) + // so cluster "a" holds indices {0,2} while "a/mm" holds {1}. The old + // planner emitted {start:0, count:2} for "a" — covering index 0 AND 1 + // — plus {start:1, count:1} for "a/mm", so index 1 was indexed twice + // and index 2 never. This test pins "every file exactly once". + let files = sorted(vec![ + make_file("a/aa.rs", 10), + make_file("a/mm/x.rs", 10), + make_file("a/zz.rs", 10), + ]); + assert_eq!(files[0].path, "a/aa.rs"); + assert_eq!(files[1].path, "a/mm/x.rs"); + assert_eq!(files[2].path, "a/zz.rs"); + // Sanity: the two "a" files are genuinely non-adjacent, which is what + // makes the arrangement a counterexample. + assert_eq!(dir_prefix(&files[0].path, CLUSTER_DEPTH), "a"); + assert_eq!(dir_prefix(&files[1].path, CLUSTER_DEPTH), "a/mm"); + assert_eq!(dir_prefix(&files[2].path, CLUSTER_DEPTH), "a"); + + let chunks = plan_chunks(&files, TARGET_BYTES, MAX_BYTES); + + let mut covered = vec![0usize; files.len()]; + for c in &chunks { + for slot in covered.iter_mut().skip(c.file_start).take(c.file_count) { + *slot += 1; + } + } + assert_eq!( + covered, + vec![1, 1, 1], + "every file must be covered exactly once (got {:?} from {} chunk(s))", + covered, + chunks.len() + ); + + // Each chunk must cover files from a single cluster only. + for c in &chunks { + for f in &files[c.file_start..c.file_start + c.file_count] { + assert_eq!( + dir_prefix(&f.path, CLUSTER_DEPTH), + c.dir_prefix, + "chunk [{}..{}) claims prefix {} but covers {}", + c.file_start, + c.file_start + c.file_count, + c.dir_prefix, + f.path + ); + } + } + // "a/zz.rs" is separated from "a/aa.rs" by "a/mm/x.rs", so cluster "a" + // must be split rather than emitted as one range. + assert!( + chunks.len() >= 2, + "an interleaved cluster must be split, got {} chunk(s)", + chunks.len() + ); + } + #[test] fn test_plan_chunks_small_project_single_chunk() { let files = vec![ diff --git a/server/src/scheduler/chunk_queue.rs b/server/src/scheduler/chunk_queue.rs index 0a3e07d..9f66db0 100644 --- a/server/src/scheduler/chunk_queue.rs +++ b/server/src/scheduler/chunk_queue.rs @@ -23,20 +23,23 @@ //! This module is infrastructure for the chunk-level scheduler (P2-P4 //! of the redesign). The current module-level scheduler still uses //! `shm.rs`; both can coexist during the migration. +//! +//! The queue lives in a shared file mapped by +//! [`super::mapped_file::MappedFile`], so it works on POSIX and Windows +//! alike; nothing here calls `mmap`, `ftruncate` or `unlink` directly. #![allow(dead_code)] -use std::ffi::CString; -use std::os::unix::io::RawFd; +// Worker-facing operations live in a child module of THIS file so +// they can reach the private helpers and struct fields declared above +// (see plan/rules/code_rules.md 1000-line rule). +#[path = "chunk_queue_ops.rs"] +mod chunk_queue_ops; + use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; use std::time::{SystemTime, UNIX_EPOCH}; -use libc::{ - MAP_SHARED, O_CREAT, O_RDWR, PROT_READ, PROT_WRITE, S_IRUSR, S_IWUSR, c_void, close, ftruncate, - mmap, munmap, open, unlink, -}; -#[cfg(windows)] -const MAP_FAILED: *mut libc::c_void = !0 as *mut libc::c_void; +use super::mapped_file::MappedFile; /// Magic number stored in [`ChunkQueueHeader::magic`] — ASCII "CUNK" /// (Chunk Queue) in little-endian. Workers verify this on attach. @@ -162,23 +165,20 @@ pub struct ChunkQueueState { /// RAII wrapper around a `mmap`'d [`ChunkQueueState`]. /// -/// Owner (scheduler) creates via [`ChunkQueue::create`] and unlinks on -/// `Drop`. Workers attach via [`ChunkQueue::open`] and only `munmap` on -/// drop — they do not unlink. +/// Owner (scheduler) creates via [`ChunkQueue::create`]; unmapping, closing +/// and removing the file are [`super::mapped_file::MappedFile`]'s `Drop`. +/// Workers attach via [`ChunkQueue::open`] and only unmap. pub struct ChunkQueue { - path: String, - fd: RawFd, + map: MappedFile, ptr: *mut ChunkQueueState, - is_owner: bool, } // Manual Debug impl — `*mut ChunkQueueState` doesn't auto-derive Debug. impl std::fmt::Debug for ChunkQueue { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("ChunkQueue") - .field("path", &self.path) - .field("fd", &self.fd) - .field("is_owner", &self.is_owner) + .field("path", &self.map.path()) + .field("is_owner", &self.map.is_owner()) .finish_non_exhaustive() } } @@ -209,83 +209,27 @@ impl ChunkQueue { /// not consume the queue before all writes are done — caller's /// responsibility). pub fn create(path: &str, chunk_count: u32) -> Result { - let c_path = CString::new(path).map_err(|e| { - format!( - "path contains NUL byte: {} [module=scheduler, method=ChunkQueue::create]", - e - ) - })?; - - // SAFETY: c_path is a valid NUL-terminated CString; O_CREAT|O_RDWR - // creates the file if absent; mode 0600 restricts to owner. - let fd = unsafe { - open( - c_path.as_ptr(), - O_CREAT | O_RDWR, - (S_IRUSR | S_IWUSR) as libc::c_int, - ) - }; - if fd < 0 { - return Err(format!( - "open failed: {} [module=scheduler, method=ChunkQueue::create, path={}]", - std::io::Error::last_os_error(), - path, - )); - } - let size = std::mem::size_of::(); - // SAFETY: fd is valid; ftruncate sizes the file to exactly - // `size` bytes. Cast to i64 is safe because the struct is - // roughly 256 * 64 = ~16KB. - if unsafe { ftruncate(fd, size as i64) } != 0 { - let err = std::io::Error::last_os_error(); - // SAFETY: fd is valid; close it to leak no fd. - unsafe { close(fd) }; - return Err(format!( - "ftruncate failed: {} [module=scheduler, method=ChunkQueue::create]", - err - )); - } - - // SAFETY: fd is valid and points to a file we just sized. - // MAP_SHARED makes writes visible to other processes that map - // the same file. - let ptr = unsafe { - mmap( - std::ptr::null_mut(), - size, - PROT_READ | PROT_WRITE, - MAP_SHARED, - fd, - 0, - ) - }; - if ptr == libc::MAP_FAILED { - let err = std::io::Error::last_os_error(); - // SAFETY: fd is valid; close it to leak no fd. - unsafe { close(fd) }; - return Err(format!( - "mmap failed: {} [module=scheduler, method=ChunkQueue::create]", - err - )); - } + // The mapping layer sizes the file to `size` and zero-fills it, so the + // chunk array starts as all-zero — status PENDING with no claimer, once + // `claimer_id` is set below. + let map = MappedFile::create(path, size, "ChunkQueue::create")?; let clamped = std::cmp::min(chunk_count, MAX_CHUNKS as u32); - // SAFETY: ptr is a valid, properly-aligned pointer to memory - // returned by mmap and sized to size_of::(). - // The file was just created/truncated so the bytes are zero. - // No worker has opened the file yet (caller's responsibility), - // so exclusive mutable access here is safe. - let state = unsafe { &mut *(ptr as *mut ChunkQueueState) }; + // SAFETY: `map` is a valid, properly-aligned, all-zero view of + // `size_of::()` bytes and no worker has opened the + // file yet (callers create the queue before spawning workers), so this + // exclusive mutable borrow is sound. + let ptr = map.ptr() as *mut ChunkQueueState; + let state = unsafe { &mut *ptr }; state.header.magic = MAGIC; state.header.version = VERSION; state.header.chunk_count = clamped; state.header.reserved = [0u32; 13]; - // ChunkState fields are already zero from ftruncate — but - // `status` must be explicitly initialised because zero IS - // STATUS_PENDING (so it's fine) and `claimer_id` zero would - // look like worker 0 claimed it, so we set it to u32::MAX. + // `status` is already 0 == STATUS_PENDING and every timing field is 0, + // but `claimer_id` must be u32::MAX: a zero there would look like + // worker 0 owns the chunk. for slot in state.chunks.iter_mut() { slot.status.store(STATUS_PENDING, Ordering::Relaxed); slot.claimer_id.store(u32::MAX, Ordering::Relaxed); @@ -296,12 +240,7 @@ impl ChunkQueue { // until write_chunk populates them. } - Ok(Self { - path: path.to_string(), - fd, - ptr: ptr as *mut ChunkQueueState, - is_owner: true, - }) + Ok(Self { map, ptr }) } /// Worker opens an existing shm created by the scheduler. @@ -309,60 +248,22 @@ impl ChunkQueue { /// Verifies magic + version. Returns `Err` with a tagged message /// if `open`, `mmap`, or the magic/version check fails. pub fn open(path: &str) -> Result { - let c_path = CString::new(path).map_err(|e| { - format!( - "path contains NUL byte: {} [module=scheduler, method=ChunkQueue::open]", - e - ) - })?; - - // SAFETY: c_path is valid NUL-terminated; O_RDWR (no O_CREAT). - let fd = unsafe { open(c_path.as_ptr(), O_RDWR) }; - if fd < 0 { - return Err(format!( - "open failed: {} [module=scheduler, method=ChunkQueue::open, path={}]", - std::io::Error::last_os_error(), - path, - )); - } - let size = std::mem::size_of::(); - // SAFETY: fd is valid; the file was created and sized by - // ChunkQueue::create, so the mapping fits the file exactly. - let ptr = unsafe { - mmap( - std::ptr::null_mut(), - size, - PROT_READ | PROT_WRITE, - MAP_SHARED, - fd, - 0, - ) - }; - if ptr == libc::MAP_FAILED { - let err = std::io::Error::last_os_error(); - // SAFETY: fd is valid; close it to leak no fd. - unsafe { close(fd) }; - return Err(format!( - "mmap failed: {} [module=scheduler, method=ChunkQueue::open]", - err - )); - } - - // SAFETY: ptr is valid ChunkQueueState-aligned memory returned - // by mmap. We only read header.magic and header.version here - // (non-atomic u32); the scheduler writes them once in create() - // before any worker opens the file, so the read is race-free. - let state = unsafe { &*(ptr as *const ChunkQueueState) }; - // Cache the values BEFORE any munmap — reading through `state` - // after `munmap(ptr, size)` would be use-after-unmap. + // The owner created this file with the same size, so the mapping fits + // the queue exactly. + let map = MappedFile::open(path, size, "ChunkQueue::open")?; + + // SAFETY: `map` is a valid, properly-aligned view of `size_of::< + // ChunkQueueState>()` bytes. We only read the non-atomic header fields + // here; the scheduler wrote them once in create() before any worker + // attached, so the read is race-free. + let ptr = map.ptr() as *mut ChunkQueueState; + let state = unsafe { &*ptr }; let observed_magic = state.header.magic; let observed_version = state.header.version; if observed_magic != MAGIC { - // SAFETY: ptr was returned by mmap with `size` bytes. - unsafe { munmap(ptr, size) }; - // SAFETY: fd is valid; close it to leak no fd. - unsafe { close(fd) }; + // Dropping `map` unmaps and closes the segment; removing the file + // is the owner's job, so these error paths leak nothing. return Err(format!( "magic mismatch: 0x{:08x} (expected 0x{:08x}) \ [module=scheduler, method=ChunkQueue::open]", @@ -370,10 +271,6 @@ impl ChunkQueue { )); } if observed_version != VERSION { - // SAFETY: ptr was returned by mmap with `size` bytes. - unsafe { munmap(ptr, size) }; - // SAFETY: fd is valid; close it to leak no fd. - unsafe { close(fd) }; return Err(format!( "version mismatch: {} (expected {}) \ [module=scheduler, method=ChunkQueue::open]", @@ -381,12 +278,7 @@ impl ChunkQueue { )); } - Ok(Self { - path: path.to_string(), - fd, - ptr: ptr as *mut ChunkQueueState, - is_owner: false, - }) + Ok(Self { map, ptr }) } /// Scheduler writes a chunk's immutable metadata into slot `idx`. @@ -419,292 +311,8 @@ impl ChunkQueue { slot.init(module_id, file_start, file_count, total_bytes); Ok(()) } - - /// Worker atomically claims the next PENDING chunk via CAS. - /// - /// Linear-scans `chunks[0..chunk_count]` for the first `PENDING` - /// slot and CAS-es it to `CLAIMED` with `claimer_id = worker_id`. - /// Returns the claimed slot index, or `None` if no chunk is - /// pending. Multiple workers calling concurrently are serialised - /// by the CAS — losers simply retry the scan (§6.1). - /// - /// Records `started_at_ms` so [`ChunkQueue::reset_stale`] can - /// detect crashed workers. - pub fn claim_next(&self, worker_id: u32) -> Option { - // SAFETY: self.ptr is valid for the lifetime of self; reads via - // shared reference are safe because all mutable fields are atomic. - let state = unsafe { &*self.ptr }; - let count = state.header.chunk_count; - for i in 0..count { - let slot = &state.chunks[i as usize]; - // Acquire on success pairs with the Release store in - // mark_done/mark_failed/reset_stale so a claimer observes - // the full prior state of the slot (weak memory: Apple - // Silicon reorders). Relaxed on failure — a lost CAS just - // retries the scan and carries no cross-thread dependency. - match slot.status.compare_exchange( - STATUS_PENDING, - STATUS_CLAIMED, - Ordering::Acquire, - Ordering::Relaxed, - ) { - Ok(_) => { - slot.claimer_id.store(worker_id, Ordering::Relaxed); - slot.started_at_ms.store(now_ms(), Ordering::Relaxed); - return Some(i); - } - Err(_) => continue, - } - } - None - } - - /// Mark chunk `idx` as DONE. Records `finished_at_ms`. No-op if - /// the slot is out of range (caller bug — we don't panic). - pub fn mark_done(&self, idx: u32) { - // SAFETY: self.ptr is valid for the lifetime of self. - let state = unsafe { &*self.ptr }; - if idx >= state.header.chunk_count { - return; - } - let slot = &state.chunks[idx as usize]; - // Write finished_at BEFORE the status store so a Release/Acquire - // pair makes it (and the worker's parse side-effects) visible to - // any observer that reads status == DONE via an Acquire load. - slot.finished_at_ms.store(now_ms(), Ordering::Relaxed); - slot.status.store(STATUS_DONE, Ordering::Release); - } - - /// Mark chunk `idx` as FAILED. Records `finished_at_ms`. No-op if - /// the slot is out of range. - pub fn mark_failed(&self, idx: u32) { - // SAFETY: self.ptr is valid for the lifetime of self. - let state = unsafe { &*self.ptr }; - if idx >= state.header.chunk_count { - return; - } - let slot = &state.chunks[idx as usize]; - // Same Release discipline as mark_done: publish finished_at (and - // the worker's parse side-effects) before the status store so an - // Acquire reader that observes FAILED also sees the prior writes. - slot.finished_at_ms.store(now_ms(), Ordering::Relaxed); - slot.status.store(STATUS_FAILED, Ordering::Release); - } - - /// Increment the failed-file counter on chunk `idx`. - /// Called by the parse loop when a single file fails (§7.3). - /// No-op if the slot is out of range. - pub fn inc_failed_files(&self, idx: u32) { - // SAFETY: self.ptr is valid for the lifetime of self. - let state = unsafe { &*self.ptr }; - if idx >= state.header.chunk_count { - return; - } - let slot = &state.chunks[idx as usize]; - slot.failed_files.fetch_add(1, Ordering::Release); - } - - /// Reset a CLAIMED chunk back to PENDING if its worker has timed - /// out (crash recovery, §8). Returns true if the reset happened. - /// - /// `timeout_ms` is the maximum allowed gap between `started_at_ms` - /// and now. The CAS only succeeds if the slot is still CLAIMED — - /// if the worker raced and finished just before us, we leave its - /// DONE/FAILED state alone. - pub fn reset_stale(&self, idx: u32, timeout_ms: u64) -> bool { - // SAFETY: self.ptr is valid for the lifetime of self. - let state = unsafe { &*self.ptr }; - if idx >= state.header.chunk_count { - return false; - } - let slot = &state.chunks[idx as usize]; - let started = slot.started_at_ms.load(Ordering::Acquire); - if started == 0 { - return false; // never claimed - } - let elapsed = now_ms().saturating_sub(started); - if elapsed < timeout_ms { - return false; - } - // CAS CLAIMED → PENDING so we don't clobber a worker that just - // finished (race between watchdog and worker completion). - // Release on success publishes the recycle so the next claimer's - // Acquire CAS in claim_next observes a clean slot; Relaxed on - // failure (a lost race carries no cross-thread dependency). - match slot.status.compare_exchange( - STATUS_CLAIMED, - STATUS_PENDING, - Ordering::Release, - Ordering::Relaxed, - ) { - Ok(_) => { - slot.claimer_id.store(u32::MAX, Ordering::Relaxed); - slot.started_at_ms.store(0, Ordering::Relaxed); - true - } - Err(_) => false, - } - } - - /// Returns true if every chunk is in DONE or FAILED state. - /// Used by the scheduler's main loop to detect completion. - /// Scan every chunk and reclaim any `CLAIMED` chunk whose worker has - /// been silent longer than `timeout_ms` (orphaned by a crashed - /// worker). Returns the number of chunks reset to `PENDING`. - /// - /// Called by idle workers (when `claim_next` finds no `PENDING` chunk) - /// so a crash mid-chunk cannot permanently strand files: another - /// worker re-claims the orphaned chunk and re-indexes its files - /// (idempotent — the worker writes to its OWN per-worker DB, so no - /// duplicate rows appear in the final merge). See DYNAMIC_SCHED_REDESIGN.md §8. - pub fn reset_all_stale(&self, timeout_ms: u64) -> u32 { - let count = self.chunk_count(); - let mut reclaimed = 0u32; - for i in 0..count { - if self.reset_stale(i, timeout_ms) { - reclaimed += 1; - } - } - reclaimed - } - - pub fn is_complete(&self) -> bool { - // SAFETY: self.ptr is valid for the lifetime of self. - let state = unsafe { &*self.ptr }; - let count = state.header.chunk_count; - for i in 0..count { - // Acquire pairs with the Release store in mark_done/mark_failed: - // once the scheduler observes every chunk DONE/FAILED it gates - // the resolve phase, so it must see all of each worker's prior - // writes (weak memory ordering on Apple Silicon). - let s = state.chunks[i as usize].status.load(Ordering::Acquire); - if s != STATUS_DONE && s != STATUS_FAILED { - return false; - } - } - true - } - - /// Number of chunks currently in PENDING state. - pub fn pending_count(&self) -> u32 { - // SAFETY: self.ptr is valid for the lifetime of self. - let state = unsafe { &*self.ptr }; - let count = state.header.chunk_count; - let mut n = 0u32; - for i in 0..count { - if state.chunks[i as usize].status.load(Ordering::Acquire) == STATUS_PENDING { - n += 1; - } - } - n - } - - /// Number of chunks in DONE state. - pub fn done_count(&self) -> u32 { - // SAFETY: self.ptr is valid for the lifetime of self. - let state = unsafe { &*self.ptr }; - let count = state.header.chunk_count; - let mut n = 0u32; - for i in 0..count { - if state.chunks[i as usize].status.load(Ordering::Acquire) == STATUS_DONE { - n += 1; - } - } - n - } - - /// Number of chunks in FAILED state. - pub fn failed_count(&self) -> u32 { - // SAFETY: self.ptr is valid for the lifetime of self. - let state = unsafe { &*self.ptr }; - let count = state.header.chunk_count; - let mut n = 0u32; - for i in 0..count { - if state.chunks[i as usize].status.load(Ordering::Acquire) == STATUS_FAILED { - n += 1; - } - } - n - } - - /// Snapshot of chunk `idx`'s state. Returns `None` if out of range. - /// Useful for diagnostics and SUMMARY generation. - pub fn chunk_state(&self, idx: u32) -> Option { - // SAFETY: self.ptr is valid for the lifetime of self. - let state = unsafe { &*self.ptr }; - if idx >= state.header.chunk_count { - return None; - } - let slot = &state.chunks[idx as usize]; - // Acquire on status first; the remaining Relaxed loads are ordered - // after it in program order, so a snapshot that sees DONE/FAILED - // also observes the finished_at/failed_files written before the - // producer's Release store. - Some(ChunkStateSnapshot { - status: slot.status.load(Ordering::Acquire), - claimer_id: slot.claimer_id.load(Ordering::Relaxed), - module_id: slot.module_id, - file_start: slot.file_start, - file_count: slot.file_count, - total_bytes: slot.total_bytes, - started_at_ms: slot.started_at_ms.load(Ordering::Relaxed), - finished_at_ms: slot.finished_at_ms.load(Ordering::Relaxed), - failed_files: slot.failed_files.load(Ordering::Relaxed), - }) - } - - /// Number of valid chunks in the queue. - pub fn chunk_count(&self) -> u32 { - // SAFETY: self.ptr is valid for the lifetime of self. - let state = unsafe { &*self.ptr }; - state.header.chunk_count - } - - /// Filesystem path backing this shm segment. - pub fn path(&self) -> &str { - &self.path - } -} - -impl Drop for ChunkQueue { - fn drop(&mut self) { - let size = std::mem::size_of::(); - if !self.ptr.is_null() { - // SAFETY: self.ptr was returned by mmap with `size` bytes and - // has not been unmapped yet (Drop runs once per instance). - unsafe { - munmap(self.ptr as *mut c_void, size); - } - } - if self.fd >= 0 { - // SAFETY: self.fd is a valid open descriptor (or already - // closed, in which case close returns EBADF — harmless). - unsafe { - close(self.fd); - } - } - // Only the scheduler (owner) unlinks the file. Workers just - // munmap+close — the inode persists until the scheduler unlinks, - // and any in-flight mmap references stay valid (POSIX semantics). - if self.is_owner - && let Ok(c_path) = CString::new(self.path.clone()) - { - // SAFETY: c_path is a valid NUL-terminated CString. unlink - // removes the directory entry; existing mmap references - // remain valid until munmap (POSIX shared memory semantics). - unsafe { - unlink(c_path.as_ptr()); - } - } - } } -// SAFETY: ChunkQueue is Send for the same reasons as SchedShm: -// - `ptr` points to mmap'd memory (MAP_SHARED) that is process-global; -// moving the Rust handle across threads does not affect the underlying -// memory or its visibility to other processes. -// - `fd` (RawFd = i32), `path` (String), and `is_owner` (bool) are all -// Send by default. unsafe impl Send for ChunkQueue {} // SAFETY: ChunkQueue is Sync for the same reasons as SchedShm: @@ -720,7 +328,7 @@ unsafe impl Send for ChunkQueue {} // read-only afterwards. `write_chunk` completes before workers are // spawned (caller's responsibility), so there are no concurrent // reads during the writes. -// - `fd` and `is_owner` are only accessed in `Drop`, which takes +// - `map` (the `MappedFile`) is only touched by `Drop`, which takes // `&mut self` (exclusive access) — no concurrent access is possible. unsafe impl Sync for ChunkQueue {} @@ -740,9 +348,12 @@ mod tests { use std::sync::atomic::AtomicU64; /// Monotonic counter appended to test paths so parallel test - /// processes never collide on the same /tmp file. + /// processes never collide on the same temp file. static TEST_COUNTER: AtomicU64 = AtomicU64::new(0); + /// Build a unique path in the platform temp directory for one test + /// invocation. `temp_dir()` rather than a hardcoded `/tmp` because this + /// suite also runs on Windows hosts, where `/tmp` does not exist. fn unique_path() -> String { let pid = std::process::id(); let nanos = std::time::SystemTime::now() @@ -750,10 +361,10 @@ mod tests { .map(|d| d.as_nanos() as u64) .unwrap_or(0); let counter = TEST_COUNTER.fetch_add(1, Ordering::Relaxed); - format!( - "/tmp/codescope_chunkq_test_{}_{}_{}.shm", - pid, nanos, counter - ) + std::env::temp_dir() + .join(format!("codescope_chunkq_test_{pid}_{nanos}_{counter}.shm")) + .to_string_lossy() + .into_owned() } #[test] @@ -891,6 +502,105 @@ mod tests { assert_eq!(q.pending_count(), 0); } + #[test] + fn test_claim_publishes_start_time_with_the_claim() { + // Regression (#23): claim_next() used to publish CLAIMED with its CAS + // and store started_at_ms afterwards — two separate atomics, with no + // Release anywhere on the claim path. An observer (reset_stale, called + // by every idle chunk worker) could therefore read CLAIMED + + // started_at_ms == 0, judge the chunk abandoned ("as stale as it + // gets") and hand it to a second worker while the first was still + // parsing it. The merge copies each worker DB row-for-row, so the + // re-parsed files then appeared twice in the final index. + // + // Two invariants close it, both checked here while claimers and + // independent samplers race on a fresh queue per round: + // 1. a CLAIMED slot always carries a non-zero start time (the stamp + // is stored before the Release CAS — structural, this is the + // behavioural guard against a future reordering); + // 2. reset_stale must never recycle a chunk with a timeout no + // wall-clock reading can exceed. + use std::sync::atomic::AtomicBool; + + // create() clamps to MAX_CHUNKS, so this is the largest queue there is. + const CHUNKS: u32 = MAX_CHUNKS as u32; + const ROUNDS: usize = 8; + const CLAIMERS: usize = 4; + + let mut unstamped = 0u64; + let mut recycled = 0u64; + for _ in 0..ROUNDS { + let path = unique_path(); + let q = Arc::new(ChunkQueue::create(&path, CHUNKS).expect("create")); + for i in 0..CHUNKS { + q.write_chunk(i, 1, i, 10, 100_000).expect("write"); + } + + let bad_unstamped = Arc::new(AtomicU64::new(0)); + let bad_recycled = Arc::new(AtomicU64::new(0)); + let stop = Arc::new(AtomicBool::new(false)); + + let mut samplers = Vec::new(); + for _ in 0..2 { + let q = Arc::clone(&q); + let bad_unstamped = Arc::clone(&bad_unstamped); + let bad_recycled = Arc::clone(&bad_recycled); + let stop = Arc::clone(&stop); + samplers.push(std::thread::spawn(move || { + // SAFETY: q.ptr is valid for the queue's lifetime and + // every field read here is an atomic. + let state = unsafe { &*q.ptr }; + while !stop.load(Ordering::Relaxed) { + for i in 0..CHUNKS { + let slot = &state.chunks[i as usize]; + // Acquire pairs with the claim's Release CAS. + if slot.status.load(Ordering::Acquire) != STATUS_CLAIMED { + continue; + } + if slot.started_at_ms.load(Ordering::Acquire) == 0 { + bad_unstamped.fetch_add(1, Ordering::Relaxed); + } + if q.reset_stale(i, u64::MAX) { + bad_recycled.fetch_add(1, Ordering::Relaxed); + } + } + } + })); + } + + let mut claimers = Vec::new(); + for t in 0..CLAIMERS { + let q = Arc::clone(&q); + claimers.push(std::thread::spawn(move || { + // Leave the chunks CLAIMED: the publication of the claim + // is exactly what is under test. + while q.claim_next(t as u32).is_some() {} + })); + } + for h in claimers { + h.join().unwrap(); + } + stop.store(true, Ordering::Relaxed); + for h in samplers { + h.join().unwrap(); + } + + unstamped += bad_unstamped.load(Ordering::Relaxed); + recycled += bad_recycled.load(Ordering::Relaxed); + let _ = std::fs::remove_file(&path); + } + + assert_eq!( + unstamped, 0, + "a CLAIMED chunk was observable with started_at_ms == 0 — \ + reset_stale treats that as abandoned and duplicates the chunk" + ); + assert_eq!( + recycled, 0, + "reset_stale recycled a chunk under an unreachable timeout" + ); + } + #[test] fn test_mark_done_failed_and_completion() { let path = unique_path(); @@ -979,6 +689,77 @@ mod tests { assert_eq!(s.claimer_id, 9); } + #[test] + fn test_reset_stale_recovers_claim_without_timestamp() { + // Legacy state: a chunk that is CLAIMED with started_at_ms == 0. The + // current claim_next() stamps the start time BEFORE it publishes + // CLAIMED, so this build never leaves a claim unstamped; the state can + // only survive in an shm segment written by an older binary (whose + // worker could crash between the CAS and the stamp). It must still be + // reclaimed: reset_stale's old early return treated 0 as "never + // claimed", so the watchdog skipped such a slot forever and its files + // were never indexed. + let path = unique_path(); + let q = ChunkQueue::create(&path, 1).expect("create"); + q.write_chunk(0, 1, 0, 10, 100_000).expect("write"); + let _ = q.claim_next(0).expect("claim"); + assert_eq!(q.chunk_state(0).unwrap().status, STATUS_CLAIMED); + unsafe { + let state = &*q.ptr; + state.chunks[0].started_at_ms.store(0, Ordering::Relaxed); + } + + // Even a zero timeout must reclaim it: an unstamped claim is stale. + assert!( + q.reset_stale(0, 0), + "unstamped CLAIMED chunk must be reclaimable" + ); + let s = q.chunk_state(0).expect("snap"); + assert_eq!(s.status, STATUS_PENDING); + assert_eq!(s.started_at_ms, 0); + + // A PENDING slot with no claimer is still NOT stale — the fix must not + // have turned every idle chunk into a watchdog target. + assert!( + !q.reset_stale(0, 0), + "an unclaimed PENDING chunk is not stale" + ); + } + + #[test] + fn test_release_worker_chunks_returns_only_that_workers_chunks() { + // Regression (#15, option C): when a chunk worker dies, the merge drops + // its DB — including the chunks it had already marked DONE, because one + // worker owns one DB. `claimer_id` is the only record of which chunks + // those were, so release must hand back exactly that worker's chunks + // (whatever their status) and leave every other worker's alone. + let path = unique_path(); + let q = ChunkQueue::create(&path, 3).expect("create"); + for i in 0..3 { + q.write_chunk(i, 1, 0, 10, 100_000).expect("write"); + } + assert_eq!(q.claim_next(0), Some(0)); + assert_eq!(q.claim_next(1), Some(1)); + q.mark_done(0); // worker 0 finished chunk 0 + assert_eq!(q.claim_next(0), Some(2)); // ...and is mid-way through chunk 2 + q.mark_failed(1); // worker 1 failed chunk 1 + + // Worker 0 owns chunk 0 (DONE) and chunk 2 (CLAIMED) — both must come + // back, because both live in the DB that is about to be dropped. + assert_eq!(q.release_worker_chunks(0), 2); + for i in [0u32, 2] { + let s = q.chunk_state(i).expect("snap"); + assert_eq!(s.status, STATUS_PENDING, "chunk {i} must be re-claimable"); + assert_eq!(s.started_at_ms, 0, "chunk {i} must look never-claimed"); + } + // Worker 1's chunk is left exactly as it was. + assert_ne!(q.chunk_state(1).expect("snap").status, STATUS_PENDING); + // An id that owns nothing releases nothing. + assert_eq!(q.release_worker_chunks(9), 0); + // Releasing is only useful if the chunks can be picked up again. + assert_eq!(q.claim_next(5), Some(0)); + } + #[test] fn test_reset_stale_does_not_clobber_finished() { let path = unique_path(); diff --git a/server/src/scheduler/chunk_queue_ops.rs b/server/src/scheduler/chunk_queue_ops.rs new file mode 100644 index 0000000..fcb10af --- /dev/null +++ b/server/src/scheduler/chunk_queue_ops.rs @@ -0,0 +1,357 @@ +// chunk_queue_ops.rs — worker-facing queue operations. +// +// Split out of chunk_queue.rs (see plan/rules/code_rules.md 1000-line +// rule). A second `impl ChunkQueue` block: claiming, marking done/failed, +// stale-chunk recovery and the completion/progress counters all read the +// atomically-shared header, so they live together and stay apart from the +// mmap setup/teardown in the parent module. `use super::*` reaches the +// private struct fields and `now_ms` helper without widening their +// visibility crate-wide. + +use super::*; + +impl ChunkQueue { + /// Worker atomically claims the next PENDING chunk via CAS. + /// + /// Linear-scans `chunks[0..chunk_count]` for the first `PENDING` + /// slot and CAS-es it to `CLAIMED` with `claimer_id = worker_id`. + /// Returns the claimed slot index, or `None` if no chunk is + /// pending. Multiple workers calling concurrently are serialised + /// by the CAS — losers simply retry the scan (§6.1). + /// + /// The claim is published *together* with `started_at_ms`: the stamp + /// is stored before the status CAS, and the CAS carries `Release`, so + /// any observer that Acquire-loads `CLAIMED` necessarily sees a + /// non-zero start time. Storing the stamp *after* the CAS (as this used + /// to) left a window in which a live chunk was observable as + /// `CLAIMED` + `started_at_ms == 0`; [`ChunkQueue::reset_stale`] treats + /// that combination as infinitely stale and hands the chunk to a second + /// worker, which re-parses the same files and — because the merge copies + /// each worker DB row-for-row without content de-duplication — duplicates + /// those rows in the final index. That state can no longer be produced + /// here. + /// + /// The price is that a thread which loses the CAS may already have + /// stamped the slot it could not claim, pushing the owner's timestamp + /// later by the skew between stamp and CAS. The only effect is that the + /// watchdog may consider such a chunk slightly fresher than it is, which + /// delays — never doubles — its recovery. `claimer_id` is written only + /// after the CAS is won, so a loser can never overwrite the winner's + /// identity (that used to make `release_worker_chunks` miss the dead + /// worker's chunks, or release a live worker's chunks). + pub fn claim_next(&self, worker_id: u32) -> Option { + // SAFETY: self.ptr is valid for the lifetime of self; reads via + // shared reference are safe because all mutable fields are atomic. + let state = unsafe { &*self.ptr }; + let count = state.header.chunk_count; + let started = now_ms(); + for i in 0..count { + let slot = &state.chunks[i as usize]; + // Cheap non-atomic-skew pre-check: skip slots a CAS cannot win. + // It also keeps the stamp below off slots that are already + // claimed, which shrinks the "loser stamps the winner" window. + if slot.status.load(Ordering::Relaxed) != STATUS_PENDING { + continue; + } + // Stamp BEFORE publishing CLAIMED — see the doc comment. + slot.started_at_ms.store(started, Ordering::Relaxed); + // AcqRel on success: Release publishes the stamp above to + // whoever Acquires this slot afterwards (the watchdog in + // reset_stale), and Acquire pairs with the Release store in + // mark_done/mark_failed/reset_stale so the claimer observes the + // slot's full prior state (weak memory: Apple Silicon reorders). + // Relaxed on failure — a lost CAS just retries the scan and + // carries no cross-thread dependency. + match slot.status.compare_exchange( + STATUS_PENDING, + STATUS_CLAIMED, + Ordering::AcqRel, + Ordering::Relaxed, + ) { + Ok(_) => { + // Publish the owner only after winning the CAS. Writing + // it before let a loser clobber the winner's id, leaving + // a CLAIMED slot attributed to the wrong worker — so + // release_worker_chunks(winner) would miss its chunks + // (files dropped from the merge) or release the loser's + // own chunks while it is still alive (duplicate parse). + // No reader needs claimer_id concurrently: mark_done and + // reset_stale key off status, and release_worker_chunks + // runs after every worker thread has been joined. + slot.claimer_id.store(worker_id, Ordering::Relaxed); + return Some(i); + } + Err(_) => continue, + } + } + None + } + + /// Mark chunk `idx` as DONE. Records `finished_at_ms`. No-op if + /// the slot is out of range (caller bug — we don't panic). + pub fn mark_done(&self, idx: u32) { + // SAFETY: self.ptr is valid for the lifetime of self. + let state = unsafe { &*self.ptr }; + if idx >= state.header.chunk_count { + return; + } + let slot = &state.chunks[idx as usize]; + // Write finished_at BEFORE the status store so a Release/Acquire + // pair makes it (and the worker's parse side-effects) visible to + // any observer that reads status == DONE via an Acquire load. + slot.finished_at_ms.store(now_ms(), Ordering::Relaxed); + slot.status.store(STATUS_DONE, Ordering::Release); + } + + /// Mark chunk `idx` as FAILED. Records `finished_at_ms`. No-op if + /// the slot is out of range. + pub fn mark_failed(&self, idx: u32) { + // SAFETY: self.ptr is valid for the lifetime of self. + let state = unsafe { &*self.ptr }; + if idx >= state.header.chunk_count { + return; + } + let slot = &state.chunks[idx as usize]; + // Same Release discipline as mark_done: publish finished_at (and + // the worker's parse side-effects) before the status store so an + // Acquire reader that observes FAILED also sees the prior writes. + slot.finished_at_ms.store(now_ms(), Ordering::Relaxed); + slot.status.store(STATUS_FAILED, Ordering::Release); + } + + /// Increment the failed-file counter on chunk `idx`. + /// Called by the parse loop when a single file fails (§7.3). + /// No-op if the slot is out of range. + pub fn inc_failed_files(&self, idx: u32) { + // SAFETY: self.ptr is valid for the lifetime of self. + let state = unsafe { &*self.ptr }; + if idx >= state.header.chunk_count { + return; + } + let slot = &state.chunks[idx as usize]; + slot.failed_files.fetch_add(1, Ordering::Release); + } + + /// Reset a CLAIMED chunk back to PENDING if its worker has timed + /// out (crash recovery, §8). Returns true if the reset happened. + /// + /// `timeout_ms` is the maximum allowed gap between `started_at_ms` + /// and now. The CAS only succeeds if the slot is still CLAIMED — + /// if the worker raced and finished just before us, we leave its + /// DONE/FAILED state alone. + pub fn reset_stale(&self, idx: u32, timeout_ms: u64) -> bool { + // SAFETY: self.ptr is valid for the lifetime of self. + let state = unsafe { &*self.ptr }; + if idx >= state.header.chunk_count { + return false; + } + let slot = &state.chunks[idx as usize]; + let started = slot.started_at_ms.load(Ordering::Acquire); + if started == 0 { + // A PENDING slot carrying 0 has never been claimed and is not + // stale — nobody owns it. + // + // A CLAIMED slot carrying 0 predates the claim/stamp pairing in + // claim_next(): this build stores started_at_ms before it + // publishes CLAIMED, so a claim made by it is never observable + // without a start time. Such a slot is therefore residue from an + // shm segment created by an older binary (whose worker could crash + // between the CAS and the stamp). It still has to be reclaimed — + // returning false made the chunk unrecoverable and its files were + // never indexed — but the reclaim is not free: the merge copies + // every worker DB row-for-row, so a re-parsed chunk duplicates its + // rows rather than being absorbed by id collisions. + if slot.status.load(Ordering::Acquire) != STATUS_CLAIMED { + return false; + } + } + let elapsed = if started == 0 { + u64::MAX // claimed but never stamped: as stale as it gets + } else { + now_ms().saturating_sub(started) + }; + if elapsed < timeout_ms { + return false; + } + // CAS CLAIMED → PENDING so we don't clobber a worker that just + // finished (race between watchdog and worker completion). + // Release on success publishes the recycle so the next claimer's + // Acquire CAS in claim_next observes a clean slot; Relaxed on + // failure (a lost race carries no cross-thread dependency). + match slot.status.compare_exchange( + STATUS_CLAIMED, + STATUS_PENDING, + Ordering::Release, + Ordering::Relaxed, + ) { + Ok(_) => { + slot.claimer_id.store(u32::MAX, Ordering::Relaxed); + slot.started_at_ms.store(0, Ordering::Relaxed); + true + } + Err(_) => false, + } + } + + /// Return every chunk claimed by `worker_id` to PENDING so a replacement + /// worker can re-index them; returns how many were released. + /// + /// Used when a chunk worker dies or times out (see chunked.rs). Such a + /// worker owns ONE per-worker DB covering every chunk it processed — they + /// steal work — so dropping that DB (what the merge does for a failed + /// worker) throws away chunks it had already marked DONE. Nobody would + /// redo them and their files would be missing from the index with only + /// `complete: false` to show for it. + /// + /// `mark_done`/`mark_failed` leave `claimer_id` set, which is what makes + /// the queue a record of "which chunks did worker N finish" — the + /// information this release needs. Every status is released, not only + /// CLAIMED: a DONE chunk's rows live in the vanished DB, and a FAILED one + /// was judged by a worker that did not survive to report it. + /// + /// Called after every worker thread has been joined, so the stores below + /// have no concurrent reader. + pub fn release_worker_chunks(&self, worker_id: u32) -> u32 { + // SAFETY: self.ptr is valid for the lifetime of self. + let state = unsafe { &*self.ptr }; + let mut released = 0u32; + for i in 0..state.header.chunk_count { + let slot = &state.chunks[i as usize]; + // init() sets claimer_id = u32::MAX for a chunk that has never + // been claimed, so a real worker id can only match a real claim. + if slot.claimer_id.load(Ordering::Acquire) != worker_id { + continue; + } + slot.claimer_id.store(u32::MAX, Ordering::Relaxed); + slot.started_at_ms.store(0, Ordering::Relaxed); + slot.finished_at_ms.store(0, Ordering::Relaxed); + // Release publishes the cleared fields to whoever claims next. + slot.status.store(STATUS_PENDING, Ordering::Release); + released += 1; + } + released + } + + /// Scan every chunk and reclaim any `CLAIMED` chunk whose worker has + /// been silent longer than `timeout_ms` (orphaned by a crashed + /// worker). Returns the number of chunks reset to `PENDING`. + /// + /// Called by idle workers (when `claim_next` finds no `PENDING` chunk) + /// so a crash mid-chunk cannot permanently strand files: another + /// worker re-claims the orphaned chunk and re-indexes its files. + /// + /// Reclaiming is NOT free: the merge copies each worker DB row-for-row + /// with id offsets above the target's MAX(id), so `INSERT OR IGNORE` + /// never fires for a logically-identical row coming from another DB — a + /// chunk parsed twice yields duplicate entities. `timeout_ms` must + /// therefore be long enough that it can only elapse for a worker that is + /// really gone (main.rs defaults it to the per-worker timeout, 600 s). + /// See DYNAMIC_SCHED_REDESIGN.md §8. + pub fn reset_all_stale(&self, timeout_ms: u64) -> u32 { + let count = self.chunk_count(); + let mut reclaimed = 0u32; + for i in 0..count { + if self.reset_stale(i, timeout_ms) { + reclaimed += 1; + } + } + reclaimed + } + + pub fn is_complete(&self) -> bool { + // SAFETY: self.ptr is valid for the lifetime of self. + let state = unsafe { &*self.ptr }; + let count = state.header.chunk_count; + for i in 0..count { + // Acquire pairs with the Release store in mark_done/mark_failed: + // once the scheduler observes every chunk DONE/FAILED it gates + // the resolve phase, so it must see all of each worker's prior + // writes (weak memory ordering on Apple Silicon). + let s = state.chunks[i as usize].status.load(Ordering::Acquire); + if s != STATUS_DONE && s != STATUS_FAILED { + return false; + } + } + true + } + + /// Number of chunks currently in PENDING state. + pub fn pending_count(&self) -> u32 { + // SAFETY: self.ptr is valid for the lifetime of self. + let state = unsafe { &*self.ptr }; + let count = state.header.chunk_count; + let mut n = 0u32; + for i in 0..count { + if state.chunks[i as usize].status.load(Ordering::Acquire) == STATUS_PENDING { + n += 1; + } + } + n + } + + /// Number of chunks in DONE state. + pub fn done_count(&self) -> u32 { + // SAFETY: self.ptr is valid for the lifetime of self. + let state = unsafe { &*self.ptr }; + let count = state.header.chunk_count; + let mut n = 0u32; + for i in 0..count { + if state.chunks[i as usize].status.load(Ordering::Acquire) == STATUS_DONE { + n += 1; + } + } + n + } + + /// Number of chunks in FAILED state. + pub fn failed_count(&self) -> u32 { + // SAFETY: self.ptr is valid for the lifetime of self. + let state = unsafe { &*self.ptr }; + let count = state.header.chunk_count; + let mut n = 0u32; + for i in 0..count { + if state.chunks[i as usize].status.load(Ordering::Acquire) == STATUS_FAILED { + n += 1; + } + } + n + } + + /// Snapshot of chunk `idx`'s state. Returns `None` if out of range. + /// Useful for diagnostics and SUMMARY generation. + pub fn chunk_state(&self, idx: u32) -> Option { + // SAFETY: self.ptr is valid for the lifetime of self. + let state = unsafe { &*self.ptr }; + if idx >= state.header.chunk_count { + return None; + } + let slot = &state.chunks[idx as usize]; + // Acquire on status first; the remaining Relaxed loads are ordered + // after it in program order, so a snapshot that sees DONE/FAILED + // also observes the finished_at/failed_files written before the + // producer's Release store. + Some(ChunkStateSnapshot { + status: slot.status.load(Ordering::Acquire), + claimer_id: slot.claimer_id.load(Ordering::Relaxed), + module_id: slot.module_id, + file_start: slot.file_start, + file_count: slot.file_count, + total_bytes: slot.total_bytes, + started_at_ms: slot.started_at_ms.load(Ordering::Relaxed), + finished_at_ms: slot.finished_at_ms.load(Ordering::Relaxed), + failed_files: slot.failed_files.load(Ordering::Relaxed), + }) + } + + /// Number of valid chunks in the queue. + pub fn chunk_count(&self) -> u32 { + // SAFETY: self.ptr is valid for the lifetime of self. + let state = unsafe { &*self.ptr }; + state.header.chunk_count + } + + /// Filesystem path backing this shm segment. + pub fn path(&self) -> &str { + self.map.path() + } +} diff --git a/server/src/scheduler/chunked.rs b/server/src/scheduler/chunked.rs new file mode 100644 index 0000000..c942484 --- /dev/null +++ b/server/src/scheduler/chunked.rs @@ -0,0 +1,612 @@ +// chunked.rs — opt-in chunk-level parallel indexer with work-stealing. +// +// Split out of scheduler/mod.rs (see plan/rules/code_rules.md 1000-line +// rule). OPT-IN path: entered only when CODESCOPE_CPU_DYNAMIC / +// CODESCOPE_DYNAMIC_SCHED is explicitly set; the default stays the static +// proportional allocator in index_parallel. +// +// `use super::*` is deliberate rather than lazy: this is a child module of +// `scheduler`, so it inherits the parent's private helpers (error_json, +// rebuild_csr_all_projects, canonicalize_project_dir, resolve_self_exe) +// and the shared constants. Widening those to `pub` would advertise them +// to the whole crate for a split that is pure code motion. + +use super::*; + +/// Chunk-level parallel indexer with work-stealing (CPU-dynamic scheduling). +/// +/// OPT-IN path: entered only when `CODESCOPE_CPU_DYNAMIC` / +/// `CODESCOPE_DYNAMIC_SCHED` is explicitly set. The DEFAULT (no flag) is the +/// static proportional allocator in `index_parallel` — see the project's +/// "static by default" scheduling principle (DYNAMIC_SCHED_REDESIGN.md). +/// +/// Stability: each worker writes its OWN DB (no concurrent WAL writers), +/// uses a unique 1-based project_id, and the final merge reuses the proven +/// per-unit-DB + merge_module_dbs machinery from the static path. +/// Whether a chunked run counts as complete. +/// +/// The chunked path can recover, so the module path's rule (`run_complete()`: +/// no failed worker AND a successful merge) would understate it: a worker that +/// dies loses every chunk it owned (one DB per worker), and Phase 3b re-indexes +/// them with a replacement worker. A non-zero `fail` is therefore not proof of +/// missing data — the queue is the authority (`queue_complete` = every chunk +/// DONE or FAILED, where FAILED is the normal file-level outcome). +/// +/// A complete index is: the merge succeeded, every chunk reached a terminal +/// state, and a recovery round that ran did not itself fail. `fail` and +/// `recovered_chunks` stay in the response either way, so the deaths are +/// visible without being reported as missing data. +fn chunked_run_complete( + merged: bool, + retry_worker_attempted: bool, + retry_worker_failed: bool, + queue_complete: bool, + failed_chunks: u32, +) -> bool { + // A complete index is the merge plus a fully terminal queue with no + // FAILED chunks; the only thing that can invalidate a recovery that + // already ran is that recovery itself failing. `fail` (dead workers + // whose chunks a replacement re-indexed) is deliberately not part of + // the test: those chunks are DONE again under a live worker. FAILED + // chunks, by contrast, mean index_files never succeeded for their + // files — they are missing data, not a recovered death. + merged + && queue_complete + && failed_chunks == 0 + && !(retry_worker_attempted && retry_worker_failed) +} + +pub(super) fn index_parallel_chunked( + project_dir: &str, + total_workers: u32, + parallel: u32, +) -> String { + let start = Instant::now(); + let total_workers = if total_workers == 0 { + DEFAULT_TOTAL_WORKERS + } else { + total_workers + }; + let parallel = if parallel == 0 { + DEFAULT_PARALLEL + } else { + parallel + }; + // Cap parallel at total_workers (see Bug 9 fix). + let parallel = parallel.min(total_workers).max(1); + + let project_path_buf = match canonicalize_project_dir(project_dir) { + Ok(p) => p, + Err(e) => return error_json(&e, "scheduler", "index_parallel_chunked"), + }; + let project_path = project_path_buf.to_string_lossy().to_string(); + + let exe_path = match resolve_self_exe() { + Ok(p) => p, + Err(e) => return error_json(&e, "scheduler", "index_parallel_chunked"), + }; + let exe_str = exe_path.to_string_lossy().to_string(); + + let grammars_dir = + std::env::var("GRAMMARS_DIR").unwrap_or_else(|_| "engine/grammars".to_string()); + + let run_id = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0); + let prefix_env = std::env::var("CODESCOPE_DB_PREFIX"); + let keep_db = prefix_env.is_ok(); + let db_prefix = prefix_env.unwrap_or_else(|_| super::default_db_prefix("chunked", run_id)); + let _ = keep_db; // keep_db consumed in run_module_worker calls below + + // ── Phase 1: discover the GLOBAL file list ─────────────── + // One walk of the whole project yields every candidate source file. + // Unlike the static path (which plans per-module), the chunked path + // plans over the flat global list so chunks can span directory + // boundaries and balance by byte weight. + let discover_json = discover::discover_files(&project_path); + let discover_val: Value = match serde_json::from_str(&discover_json) { + Ok(v) => v, + Err(e) => { + return error_json( + &format!( + "discover_files parse failed: {} [module=scheduler, method=index_parallel_chunked]", + e + ), + "scheduler", + "index_parallel_chunked", + ); + } + }; + if discover_val["ok"] != true { + return discover_json; + } + let all_paths: Vec = discover_val["files"] + .as_array() + .map(|a| { + a.iter() + .filter_map(|v| v.as_str().map(|s| s.to_string())) + .collect() + }) + .unwrap_or_default(); + if all_paths.is_empty() { + // Same honesty as the static path (#23): a run that indexed + // nothing is not a success, and `complete` must be present. + return json!({ + "ok": false, + "complete": false, + "project_path": project_path, + "duration_ms": start.elapsed().as_millis() as u64, + "success": 0, + "fail": 0, + "total_nodes": 0, + "total_edges": 0, + "total_files_indexed": 0, + "modules": [], + "note": "no source files found" + }) + .to_string(); + } + + // Build FileEntry list (path + size), sorted by path — plan_chunks + // REQUIRES the input sorted by path (see chunk_plan.rs Invariant). + let mut files: Vec = all_paths + .iter() + .map(|p| { + let size = std::fs::metadata(p).map(|m| m.len()).unwrap_or(0); + chunk_plan::FileEntry { + path: p.clone(), + size, + } + }) + .collect(); + files.sort_by(|a, b| a.path.cmp(&b.path)); + + // ── Phase 2: plan chunks ──────────────────────────────── + let chunks = chunk_plan::plan_chunks(&files, chunk_plan::TARGET_BYTES, chunk_plan::MAX_BYTES); + // Refuse to run rather than silently truncate. The chunk queue is a + // fixed-size shared-memory array, so a plan with more chunks than the + // queue holds used to be cut down with only a stderr warning — the + // trailing files were never indexed while the result still reported + // ok=true, i.e. a silently incomplete index. Failing here makes the + // gap explicit and leaves the database untouched. + if chunks.len() > chunk_queue::MAX_CHUNKS { + return error_json( + &format!( + "plan_chunks produced {} chunks but the shared-memory queue holds at most {}; \ + refusing to index a partial project [module=scheduler, method=index_parallel_chunked]", + chunks.len(), + chunk_queue::MAX_CHUNKS + ), + "scheduler", + "index_parallel_chunked", + ); + } + let chunk_count = chunks.len() as u32; + if chunk_count == 0 { + return error_json( + "plan_chunks returned zero chunks [module=scheduler, method=index_parallel_chunked]", + "scheduler", + "index_parallel_chunked", + ); + } + + // Share the GLOBAL file list with workers via a temp JSON file. + // Workers slice it by each chunk's (file_start, file_count). + let files_json_path = format!("{}_chunk_files.json", db_prefix); + let files_json = serde_json::to_string(&all_paths).unwrap_or_default(); + if let Err(e) = std::fs::write(&files_json_path, &files_json) { + return error_json( + &format!( + "write chunk file list failed: {} [module=scheduler, method=index_parallel_chunked]", + e + ), + "scheduler", + "index_parallel_chunked", + ); + } + + // ── Phase 3: create + fill the chunk queue ────────────── + let shm_path = super::default_shm_path("chunked_sched"); + let _ = std::fs::remove_file(&shm_path); + let queue = match chunk_queue::ChunkQueue::create(&shm_path, chunk_count) { + Ok(q) => q, + Err(e) => return error_json(&e, "scheduler", "index_parallel_chunked"), + }; + for (idx, ch) in chunks.iter().take(chunk_count as usize).enumerate() { + if let Err(e) = queue.write_chunk( + idx as u32, + ch.module_id, + ch.file_start as u32, + ch.file_count as u32, + ch.total_bytes, + ) { + return error_json( + &format!( + "write_chunk {} failed: {} [module=scheduler, method=index_parallel_chunked]", + idx, e + ), + "scheduler", + "index_parallel_chunked", + ); + } + } + + eprintln!( + "scheduler: [chunked] project={} files={} chunks={} workers={} parallel={}", + project_path, + all_paths.len(), + chunk_count, + total_workers, + parallel + ); + + // ── Phase 3: spawn workers, each with its OWN DB ────────── + // Every chunk-worker writes to a unique per-worker DB + // (`{db_prefix}_chunk_{worker_id}.db`) with a unique project_id + // (`worker_id + 1`), so there are NO concurrent WAL writers on a + // shared DB — the original shared_db design corrupted the index + // under concurrency. After all workers exit, Phase 4 merges the + // per-worker DBs into the unified main DB via the SAME + // merge_module_dbs machinery as the static path. + let mut handles = Vec::new(); + let mut results: Vec = Vec::new(); + // Worker ids whose process failed, kept so their chunks can be released + // and re-attempted below (Phase 3b). + let mut failed_worker_ids: Vec = Vec::new(); + let active = Arc::new(AtomicU32::new(0)); + // Carries the worker id alongside the result so a failed worker's chunks + // can be released and re-attempted (Phase 3b). + let (tx, rx) = mpsc::channel::<(u32, ModuleResult)>(); + + for worker_id in 0..total_workers { + // Wait for a free slot if at concurrency cap. + while active.load(Ordering::SeqCst) >= parallel { + std::thread::sleep(POLL_INTERVAL); + } + active.fetch_add(1, Ordering::SeqCst); + + // CPU set for this worker (static binding, Linux only; empty + // elsewhere so run_chunk_worker skips taskset wrapping). + #[cfg(target_os = "linux")] + let cpu_set = { + let host_cores = std::thread::available_parallelism() + .map(|n| n.get() as u32) + .unwrap_or(1) + .max(1); + let cores_per_worker = (host_cores / total_workers.max(1)).max(1); + let cpu_idx = (worker_id * cores_per_worker) % host_cores; + format!("{}", cpu_idx) + }; + #[cfg(not(target_os = "linux"))] + let cpu_set: String = String::new(); + + // Per-worker DB + unique project_id. Mirrors the static path's + // (idx + 1) project_id scheme so merge's id-remap and project + // disambiguation behave identically. + let worker_db = format!("{}_chunk_{}.db", db_prefix, worker_id); + let project_id: u64 = (worker_id as u64) + 1; + + let active_clone = Arc::clone(&active); + let tx = tx.clone(); + let exe_str = exe_str.clone(); + let grammars_dir = grammars_dir.clone(); + let shm_path = shm_path.clone(); + let worker_db = worker_db.clone(); + let files_json_path = files_json_path.clone(); + + let handle = std::thread::spawn(move || { + // catch_unwind so a panicking worker cannot leak its slot in + // `active` (the dispatch loop's `while active >= parallel` would + // then wait forever) and cannot vanish from `results` (fail + // undercount). Same guard as the static and dynamic paths. + let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + worker::run_chunk_worker( + &exe_str, + &shm_path, + worker_id, + &cpu_set, + &worker_db, + &files_json_path, + project_id, + &grammars_dir, + ) + })); + let result = match result { + Ok(r) => r, + Err(_) => ModuleResult { + name: format!("chunk-worker-{}", worker_id), + exit_code: -5, + total_nodes: 0, + total_edges: 0, + files_indexed: 0, + candidate_files: 0, + time_parse_ms: 0, + duration_secs: 0, + workers: 1, + db_path: worker_db.clone(), + project_id, + error: Some( + "worker panicked [module=scheduler, method=index_parallel_chunked]" + .to_string(), + ), + }, + }; + let _ = tx.send((worker_id, result)); + active_clone.fetch_sub(1, Ordering::SeqCst); + }); + handles.push(handle); + } + + drop(tx); + + // Collect results, remembering which worker each came from: the retry + // round below releases exactly the chunks the failed workers own. + while let Ok((worker_id, r)) = rx.recv() { + // A worker that exited 0 but produced no parseable JSON, or an + // ok:false engine envelope, has error=Some. It owns chunks the queue + // already marked DONE, and its DB is dropped from the merge — so it + // must count as failed, or the run reports complete:true while its + // files are missing. `worker_succeeded` is the single predicate used + // by the static path too. + if !worker_succeeded(&r) { + failed_worker_ids.push(worker_id); + } + results.push(r); + } + + // Wait for all threads to finish. + for h in handles { + let _ = h.join(); + } + + // ── Phase 3b: recover the chunks of workers that died ───────── + // A chunk worker owns ONE DB covering every chunk it processed (they steal + // work), and the merge below drops the DB of a failed worker — so a crash + // or timeout discarded the chunks that worker had already marked DONE, and + // their files were missing from the index with nothing but `complete: + // false` to show for it (2026-09-18 review #15, option C). + // + // The queue still records which chunks belong to which worker (`mark_done` + // leaves `claimer_id` set), so they are released back to PENDING and ONE + // replacement worker re-indexes them. One round only: a deterministic + // crasher — a file that kills every worker that reads it — would kill the + // replacement the same way, and looping on it would turn a crash into a + // hang. Whatever the round does not recover is reported as before (`ok` and + // `complete` stay false, `fail` counts the workers). + let mut recovered_chunks = 0u32; + let mut retry_worker_failed = false; + let mut retry_worker_attempted = false; + if !failed_worker_ids.is_empty() { + for id in &failed_worker_ids { + recovered_chunks += queue.release_worker_chunks(*id); + } + // The release above only finds chunks the dead worker had CLAIMED. A + // worker can also die before claiming anything, which leaves chunks + // that are still PENDING and now have nobody to pick them up — so the + // queue, not the release count, decides whether work is missing. + if !queue.is_complete() { + retry_worker_attempted = true; + let retry_id = total_workers; + let retry_db = format!("{}_chunk_retry.db", db_prefix); + // Project ids are 1..=total_workers for the first round; the + // replacement takes the next one so the merge's id remap sees it as + // its own worker, exactly like the others. + let retry_project_id = (total_workers as u64) + 1; + eprintln!( + "scheduler: [chunked] {} chunk(s) released from {} failed worker(s); re-indexing as worker {}", + recovered_chunks, + failed_worker_ids.len(), + retry_id + ); + let retry_result = worker::run_chunk_worker( + &exe_str, + &shm_path, + retry_id, + "", // no CPU binding for the single recovery worker + &retry_db, + &files_json_path, + retry_project_id, + &grammars_dir, + ); + retry_worker_failed = !worker_succeeded(&retry_result); + eprintln!( + "scheduler: [chunked] recovery worker {}: exit={} files={} nodes={}", + retry_id, + retry_result.exit_code, + retry_result.files_indexed, + retry_result.total_nodes + ); + // Counted like any other worker: `success`/`fail` are derived from + // `results`, so a replacement that also dies makes the run + // incomplete rather than silently recovering on paper. + results.push(retry_result); + } + } + + // Both per-run temporaries are removed here, after every worker has exited + // and before the merge. Neither used to be removed at all, so a successful + // run leaked them too: the queue path embeds the PID + // (`/tmp/codescope_chunked_sched_.shm`), so the next run's + // remove-before-create can never clean up an earlier process's file and + // the chunk-file list accumulated in /tmp one JSON per run. Unlinking the + // shm while the mapping is alive is fine on POSIX (the name is what goes + // away, not the pages), and no code path reads either name after this + // point — the response reports `db_prefix`/`main_db`, not these. + let _ = std::fs::remove_file(&shm_path); + let _ = std::fs::remove_file(&files_json_path); + + // ── Phase 4: merge per-worker DBs into the unified main DB ── + // Mirror index_parallel Phase 6: ATTACH each worker DB to a fresh + // main DB and INSERT OR IGNORE. Per-worker project_ids are unique + // (worker_id + 1) and merge_module_dbs remaps ids to avoid + // cross-worker collisions (same as the static path). + let main_db = format!("{}_main.db", db_prefix); + // main.db is ALWAYS rebuilt from the module DBs below — keep_db only + // preserves the per-module DBs so workers can skip unchanged files. + // Keeping main.db too would double-count rows on INSERT OR IGNORE. + let _ = std::fs::remove_file(&main_db); + let _ = std::fs::remove_file(format!("{}-wal", main_db)); + let _ = std::fs::remove_file(format!("{}-shm", main_db)); + + let worker_db_paths: Vec = results + .iter() + .filter(|r| { + r.exit_code == 0 && r.error.is_none() && (r.total_nodes > 0 || r.files_indexed == 0) + }) + .map(|r| r.db_path.clone()) + .collect(); + + let merge_result = if worker_db_paths.is_empty() { + MergeResult { + merged: false, + main_db_path: main_db.clone(), + tables_merged: 0, + rows_merged: 0, + duration_ms: 0, + error: Some( + "no successful chunk-worker DBs to merge [module=scheduler, method=index_parallel_chunked]" + .to_string(), + ), + } + } else { + merge::merge_module_dbs(&main_db, &worker_db_paths, &project_path) + }; + + // v0.2.5 (C2 fix): parallel chunk workers deferred CSR construction; + // rebuild each project's CSR from the globally-remapped relation table. + if merge_result.merged { + rebuild_csr_all_projects(&main_db, "index_parallel_chunked"); + } + + // ── Phase 5: aggregate summary ──────────────────────────── + let success = results + .iter() + .filter(|r| { + r.exit_code == 0 && r.error.is_none() && (r.total_nodes > 0 || r.files_indexed == 0) + }) + .count(); + let fail = results.len() - success; + let failed_chunks = queue.failed_count(); + let total_nodes: u64 = results.iter().map(|r| r.total_nodes).sum(); + let total_edges: u64 = results.iter().map(|r| r.total_edges).sum(); + let total_files_indexed: u64 = results.iter().map(|r| r.files_indexed).sum(); + + let modules_json: Vec = results + .iter() + .map(|r| { + json!({ + "name": r.name, + "exit_code": r.exit_code, + "files_indexed": r.files_indexed, + "candidate_files": r.candidate_files, + "total_nodes": r.total_nodes, + "total_edges": r.total_edges, + "time_parse_ms": r.time_parse_ms, + "duration_secs": r.duration_secs, + "workers": r.workers, + "db_path": r.db_path, + "error": r.error, + }) + }) + .collect(); + + // "ok" means the run completed AND produced a consistent index — not just + // "at least one worker finished". See run_complete(). + let complete = chunked_run_complete( + merge_result.merged, + retry_worker_attempted, + retry_worker_failed, + queue.is_complete(), + failed_chunks, + ); + json!({ + "ok": complete, + "complete": complete, + "project_path": project_path, + "db_prefix": db_prefix, + "main_db": merge_result.main_db_path, + "merge": { + "merged": merge_result.merged, + "tables_merged": merge_result.tables_merged, + "rows_merged": merge_result.rows_merged, + "duration_ms": merge_result.duration_ms, + "error": merge_result.error, + }, + "total_workers": total_workers, + "parallel": parallel, + "sched_mode": "chunked", + "duration_ms": start.elapsed().as_millis() as u64, + "success": success, + "fail": fail, + // Chunks whose index_files failed (engine ok:false / bad JSON). + // Non-zero ⇒ files missing from the merged DB ⇒ complete: false. + "failed_chunks": failed_chunks, + // Chunks whose worker died and that a replacement re-indexed; a + // non-zero count with a failed retry_worker means files are still + // missing (the run says complete: false either way). + "recovered_chunks": recovered_chunks, + "retry_worker_failed": retry_worker_failed, + "total_nodes": total_nodes, + "total_edges": total_edges, + "total_files_indexed": total_files_indexed, + "modules": modules_json + }) + .to_string() +} + +#[cfg(test)] +mod tests { + use super::chunked_run_complete; + use serde_json::Value; + + #[test] + fn test_chunked_run_complete_accounts_for_recovery() { + // A healthy run: merge ok, every chunk terminal, no recovery needed. + assert!(chunked_run_complete(true, false, false, true, 0)); + // A failed merge is never complete, whatever the queue says. + assert!(!chunked_run_complete(false, false, false, true, 0)); + + // Failures whose chunks a replacement re-indexed: complete, with the + // deaths still visible in `fail` / `recovered_chunks`. + assert!(chunked_run_complete(true, true, false, true, 0)); + + // Every way the recovery can fall short stays incomplete: + assert!( + !chunked_run_complete(true, true, true, true, 0), + "the replacement died too" + ); + assert!( + !chunked_run_complete(true, true, false, false, 0), + "the queue still holds chunks nobody finished" + ); + assert!( + !chunked_run_complete(true, false, false, false, 0), + "a worker died before claiming anything and nothing recovered the work" + ); + // FAILED chunks mean index_files never succeeded for their files: + // those files are missing even though the queue is "terminal". + assert!( + !chunked_run_complete(true, false, false, true, 1), + "a FAILED chunk leaves its files out of the merged DB" + ); + } + + #[test] + fn test_chunked_empty_project_reports_incomplete() { + // Regression: the static path got #23 (ok:false + complete:false on + // empty discovery); the chunked path still returned ok:true with no + // complete field, so an empty index looked successful. + let dir = std::env::temp_dir().join("codescope_test_chunked_empty"); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + // No CODESCOPE_CPU_DYNAMIC needed — call the inner function directly. + let out = super::index_parallel_chunked(dir.to_str().unwrap(), 1, 1); + let v: Value = serde_json::from_str(&out).unwrap(); + if v["note"] == "no source files found" { + assert_eq!(v["ok"], false, "an empty index is not a success"); + assert_eq!(v["complete"], false, "complete must be present and false"); + } + let _ = std::fs::remove_dir_all(&dir); + } +} diff --git a/server/src/scheduler/dyn_config.rs b/server/src/scheduler/dyn_config.rs index a215721..1121cb6 100644 --- a/server/src/scheduler/dyn_config.rs +++ b/server/src/scheduler/dyn_config.rs @@ -78,9 +78,11 @@ impl DynSchedConfig { } } - /// Generate a shm path for this scheduler process. + /// Generate a shm path for this scheduler process (temp directory + + /// PID, so concurrent runs never collide; see + /// [`super::default_shm_path`]). pub fn default_shm_path() -> String { - format!("/tmp/codescope_sched_{}.shm", std::process::id()) + super::default_shm_path("sched") } } diff --git a/server/src/scheduler/mapped_file.rs b/server/src/scheduler/mapped_file.rs new file mode 100644 index 0000000..b287a88 --- /dev/null +++ b/server/src/scheduler/mapped_file.rs @@ -0,0 +1,370 @@ +//! Cross-platform, file-backed shared memory. +//! +//! Both schedulers need a `#[repr(C)]` state block that the scheduler process +//! and its worker subprocesses all observe: the dynamic scheduler's +//! [`crate::scheduler::shm::SchedState`] and the chunked scheduler's +//! [`crate::scheduler::chunk_queue::ChunkQueueState`]. Both used to open-code +//! the same POSIX sequence (`open` + `ftruncate` + `mmap(MAP_SHARED)`) and so +//! both failed to compile on Windows, where the `libc` crate exports no mapping +//! API at all. This module owns that sequence once, behind two backends: +//! +//! * **POSIX** — `open`/`ftruncate`/`mmap(MAP_SHARED)`, torn down with +//! `munmap`/`close`/`unlink`. +//! * **Windows** — a file-backed section object: `CreateFileW` (through +//! [`std::fs`]) + `SetEndOfFile` + `CreateFileMappingW` + `MapViewOfFile`, +//! torn down with `UnmapViewOfFile`/`CloseHandle`/`DeleteFileW`. +//! +//! The contract is identical on both platforms: +//! +//! * [`MappedFile::create`] sizes the file to exactly `len` and returns a +//! writable view whose bytes are **all zero**. +//! * [`MappedFile::open`] attaches to a file the owner already created; the +//! caller passes the same `len` it used for `create`. +//! * `Drop` unmaps and closes, and the owner additionally removes the file. +//! +//! Callers must keep the returned handle alive for as long as they read the +//! mapped memory, and must ensure every process that maps the same file agrees +//! on the layout (`#[repr(C)]` fields, atomics for anything mutated after +//! `create`). + +/// Method tag appended to error strings, e.g. `SchedShm::create`, so messages +/// keep the `[module=scheduler, method=…]` shape callers' tests and log greps +/// expect. Kept as a parameter rather than a constant because the same mapping +/// code backs two differently named queues. +pub struct MappedFile { + path: String, + ptr: *mut u8, + len: usize, + owner: bool, + handle: imp::Handle, +} + +impl MappedFile { + /// Create (or re-truncate) `path` to exactly `len` bytes and map it + /// writable. The mapped bytes are guaranteed to be zero. + pub fn create(path: &str, len: usize, method: &str) -> Result { + let (ptr, handle) = imp::map(path, len, true, method)?; + Ok(Self { + path: path.to_string(), + ptr, + len, + owner: true, + handle, + }) + } + + /// Attach to a file previously created with [`MappedFile::create`]. + pub fn open(path: &str, len: usize, method: &str) -> Result { + let (ptr, handle) = imp::map(path, len, false, method)?; + Ok(Self { + path: path.to_string(), + ptr, + len, + owner: false, + handle, + }) + } + + /// Base address of the mapping. Cast to the caller's `#[repr(C)]` type. + pub fn ptr(&self) -> *mut u8 { + self.ptr + } + + /// Path backing the mapping. + pub fn path(&self) -> &str { + &self.path + } + + /// True when this handle created the file (and so removes it on drop). + pub fn is_owner(&self) -> bool { + self.owner + } +} + +impl Drop for MappedFile { + fn drop(&mut self) { + if !self.ptr.is_null() { + // SAFETY: ptr/len are exactly what this mapping returned, and Drop + // runs once, so the view is unmapped exactly once. + unsafe { imp::unmap(self.ptr, self.len) }; + self.ptr = std::ptr::null_mut(); + } + // SAFETY: handle is owned by this MappedFile and released once; on the + // Unix backend this closes the descriptor, on Windows the section. + unsafe { imp::close(&mut self.handle) }; + if self.owner { + imp::remove(&self.path); + } + } +} + +// SAFETY: `MappedFile` owns a process-wide mapping of a shared file. Moving it +// between threads grants no exclusive access — every reader observes the same +// bytes, and all mutable fields in the mapped structs are atomics. The raw +// pointer and the backend handle are only ever passed to the OS calls that +// created them, all of which are thread-safe. +unsafe impl Send for MappedFile {} +// SAFETY: see above — reads through the raw pointer are of atomic fields, and +// the handle is touched only by `Drop`, which takes `&mut self`. +unsafe impl Sync for MappedFile {} + +#[cfg(unix)] +mod imp { + use std::ffi::{CString, c_void}; + use std::io; + use std::os::unix::io::RawFd; + + // NOTE: libc's `close` is intentionally not imported — this module defines + // its own `close` for the mapping handle, so the libc one is called with an + // explicit path below. + use libc::{ + MAP_SHARED, O_CREAT, O_RDWR, PROT_READ, PROT_WRITE, S_IRUSR, S_IWUSR, ftruncate, mmap, + munmap, open, unlink, + }; + + /// Open descriptor of the backing file. `-1` means "already closed", which + /// makes the release path idempotent. + pub(super) struct Handle(RawFd); + + pub(super) fn map( + path: &str, + len: usize, + create: bool, + method: &str, + ) -> Result<(*mut u8, Handle), String> { + let c_path = CString::new(path).map_err(|e| { + format!("path contains NUL byte: {e} [module=scheduler, method={method}]") + })?; + + // SAFETY: c_path is a valid NUL-terminated CString. O_CREAT|O_RDWR + // creates the file when the caller owns it; mode 0600 restricts it to + // the owner. The mode is cast to c_int because open(2) is variadic and + // macOS types S_IRUSR as u16. + let fd = unsafe { + if create { + open( + c_path.as_ptr(), + O_CREAT | O_RDWR, + (S_IRUSR | S_IWUSR) as libc::c_int, + ) + } else { + open(c_path.as_ptr(), O_RDWR) + } + }; + if fd < 0 { + return Err(format!( + "open failed: {} [module=scheduler, method={method}, path={path}]", + io::Error::last_os_error(), + )); + } + + if create { + // ftruncate both sizes the file and — when extending — zero-fills + // the new bytes, which is what lets the callers treat a fresh + // mapping as all-zero. The struct sizes here are a few hundred + // bytes to ~16 KB, well inside off_t range. + // SAFETY: fd is a valid descriptor owned by this frame. + if unsafe { ftruncate(fd, len as i64) } != 0 { + let err = io::Error::last_os_error(); + // SAFETY: fd is valid and owned here; release it on error. + unsafe { libc::close(fd) }; + return Err(format!( + "ftruncate failed: {err} [module=scheduler, method={method}]" + )); + } + } + + // SAFETY: fd is valid and the file is at least `len` bytes (just + // truncated by us, or created by the owner). MAP_SHARED makes writes + // visible to every other process mapping the same file, which is the + // whole point: PROT_READ|PROT_WRITE allows the atomic updates. + let ptr = unsafe { + mmap( + std::ptr::null_mut(), + len, + PROT_READ | PROT_WRITE, + MAP_SHARED, + fd, + 0, + ) + }; + if ptr == libc::MAP_FAILED { + let err = io::Error::last_os_error(); + // SAFETY: fd is valid and owned here; release it on error. + unsafe { libc::close(fd) }; + return Err(format!( + "mmap failed: {err} [module=scheduler, method={method}]" + )); + } + + Ok((ptr as *mut u8, Handle(fd))) + } + + pub(super) unsafe fn unmap(ptr: *mut u8, len: usize) { + // SAFETY: caller guarantees ptr/len came from `mmap` above and that the + // view is unmapped exactly once. + unsafe { munmap(ptr as *mut c_void, len) }; + } + + pub(super) unsafe fn close(handle: &mut Handle) { + if handle.0 >= 0 { + // SAFETY: the descriptor is owned by the caller and closed once; + // the -1 sentinel makes a second call a no-op. + unsafe { libc::close(handle.0) }; + handle.0 = -1; + } + } + + pub(super) fn remove(path: &str) { + if let Ok(c_path) = CString::new(path) { + // SAFETY: c_path is a valid NUL-terminated CString. Unlinking while + // another process still maps the file is harmless: existing + // mappings stay valid until they are unmapped (POSIX semantics), + // and the inode is freed with the last mapping. + unsafe { unlink(c_path.as_ptr()) }; + } + } +} + +#[cfg(windows)] +mod imp { + use std::fs::{File, OpenOptions}; + use std::io::Write; + use std::os::windows::io::AsRawHandle; + + use windows_sys::Win32::Foundation::{CloseHandle, HANDLE}; + use windows_sys::Win32::System::Memory::{ + CreateFileMappingW, FILE_MAP_ALL_ACCESS, MapViewOfFile, PAGE_READWRITE, UnmapViewOfFile, + }; + + /// Owns the backing file (which keeps the section object valid) plus the + /// section handle itself. `mapping.is_null()` means "already released", so + /// the release path is idempotent; dropping `file` closes the last handle. + pub(super) struct Handle { + file: Option, + mapping: HANDLE, + } + + pub(super) fn map( + path: &str, + len: usize, + create: bool, + method: &str, + ) -> Result<(*mut u8, Handle), String> { + let mut options = OpenOptions::new(); + options.read(true).write(true); + if create { + options.create(true); + } + let file = options.open(path).map_err(|e| { + format!("open failed: {e} [module=scheduler, method={method}, path={path}]") + })?; + + if create { + // SetEndOfFile (behind `set_len`) sizes the file but leaves the + // bytes of an *extended* region undefined on Windows, unlike POSIX + // ftruncate. Write the zeros explicitly so `create` offers the same + // guarantee on both platforms: a fresh mapping reads as all-zero, + // which is what the callers' "reserved/zeroed fields" comments + // (and their magic/version writes) rely on. + file.set_len(len as u64).map_err(|e| { + format!("ftruncate failed: {e} [module=scheduler, method={method}]") + })?; + let zeros = vec![0u8; len]; + let mut writer = &file; + writer.write_all(&zeros).map_err(|e| { + format!("zero-fill failed: {e} [module=scheduler, method={method}]") + })?; + writer.flush().map_err(|e| { + format!("zero-fill flush failed: {e} [module=scheduler, method={method}]") + })?; + } + + // The section size is passed as two u32 halves. Callers map a single + // struct, so anything above 4 GiB means a layout mistake — refuse it + // rather than silently mapping a truncated (and therefore + // out-of-bounds) view. + let size_low = u32::try_from(len).map_err(|_| { + format!( + "mapping too large for a section object: {len} bytes \ + [module=scheduler, method={method}]" + ) + })?; + + // SAFETY: the file handle is valid for as long as `file` is alive and + // is only borrowed for the duration of the call; a null name gives the + // section no name in the object namespace (workers reach it through the + // shared file path, not through a named section). + let mapping = unsafe { + CreateFileMappingW( + file.as_raw_handle() as HANDLE, + std::ptr::null(), + PAGE_READWRITE, + 0, + size_low, + std::ptr::null(), + ) + }; + if mapping.is_null() { + return Err(format!( + "mmap failed: {} [module=scheduler, method={method}]", + std::io::Error::last_os_error(), + )); + } + + // SAFETY: `mapping` is a live section handle object; FILE_MAP_ALL_ACCESS + // matches the PAGE_READWRITE protection it was created with. + let view = unsafe { MapViewOfFile(mapping, FILE_MAP_ALL_ACCESS, 0, 0, len) }; + if view.Value.is_null() { + let err = std::io::Error::last_os_error(); + // SAFETY: mapping was created above and is owned by this frame. + unsafe { CloseHandle(mapping) }; + return Err(format!( + "mmap failed: {err} [module=scheduler, method={method}]" + )); + } + + Ok(( + view.Value as *mut u8, + Handle { + file: Some(file), + mapping, + }, + )) + } + + pub(super) unsafe fn unmap(ptr: *mut u8, _len: usize) { + // SAFETY: caller guarantees ptr came from `MapViewOfFile` above and that + // the view is unmapped exactly once. + unsafe { + UnmapViewOfFile( + windows_sys::Win32::System::Memory::MEMORY_MAPPED_VIEW_ADDRESS { + Value: ptr as *mut core::ffi::c_void, + }, + ) + }; + } + + pub(super) unsafe fn close(handle: &mut Handle) { + if !handle.mapping.is_null() { + // SAFETY: the section handle is owned here and released once; the + // null sentinel makes a second call a no-op. + unsafe { CloseHandle(handle.mapping) }; + handle.mapping = std::ptr::null_mut(); + } + // Closing the file handle releases the file itself. Order matters: the + // view and the section must be gone first (Windows refuses to delete a + // file that still has an open handle or a live section). + handle.file = None; + } + + pub(super) fn remove(path: &str) { + // Best effort, and deliberately after both handles were released: if a + // worker has not exited yet the delete fails with a sharing violation + // and the file stays in the temp directory. That is the Windows + // counterpart of the POSIX path's "stale segment" case (a killed run + // leaves the file behind too), and the names are unique per PID, so the + // next run never collides with it. + let _ = std::fs::remove_file(path); + } +} diff --git a/server/src/scheduler/merge.rs b/server/src/scheduler/merge.rs index 68e1647..3f87f03 100644 --- a/server/src/scheduler/merge.rs +++ b/server/src/scheduler/merge.rs @@ -23,6 +23,18 @@ //! //! Uses the sqlite3 CLI (no new Rust deps). +// The merge is split across two child modules (1000-line rule). Both are +// children of THIS module so they can reach the private TableSpec +// definitions, the SQL builders and the SQLite helpers declared below +// without any of them becoming crate-visible. +#[path = "merge_driver.rs"] +mod merge_driver; +#[path = "merge_fetch.rs"] +mod merge_fetch; + +pub(super) use merge_driver::merge_module_dbs; +use merge_fetch::{check_module_schema_consistency, fetch_all_columns_excluding_rowid}; + use std::process::{Command, Stdio}; use std::time::Instant; @@ -326,498 +338,6 @@ fn fetch_columns_excluding_rowid(db_path: &str, table_name: &str) -> Result Result, String> { - if tables.is_empty() { - return Ok(std::collections::HashMap::new()); - } - // One query: for each table emit a pragma_table_info scan, tagged with - // the table name so we can group columns back to their table. Column - // `rowid` (the INTEGER PRIMARY KEY AUTOINCREMENT alias) is skipped here - // so it isn't copied on INSERT — same contract as fetch_columns_excluding_rowid. - let mut sql = String::new(); - for (i, t) in tables.iter().enumerate() { - if i > 0 { - sql.push_str(" UNION ALL "); - } - // cid/name are the only fields we need; type/notnull/dflt/pk ignored. - sql.push_str(&format!( - "SELECT '{t}' AS tbl, cid, name FROM pragma_table_info('{t}') WHERE name != 'rowid'", - t = t - )); - } - sql.push_str(" ORDER BY tbl, cid;"); - - let output = Command::new("sqlite3") - .arg(db_path) - .arg(&sql) - .stdin(Stdio::null()) - .stdout(Stdio::piped()) - .stderr(Stdio::piped()) - .output() - .map_err(|e| { - format!( - "spawn: {} [module=scheduler, method=fetch_all_columns_excluding_rowid]", - e - ) - })?; - if !output.status.success() { - let stderr = String::from_utf8_lossy(&output.stderr).to_string(); - return Err(format!( - "sqlite3 exit={}: {} [module=scheduler, method=fetch_all_columns_excluding_rowid]", - output.status.code().unwrap_or(-1), - stderr - )); - } - - // Output rows are `table_name|cid|column_name` (sqlite3 default '|' - // separator). Group columns per table in cid order. - let stdout = String::from_utf8_lossy(&output.stdout).to_string(); - let mut out: std::collections::HashMap<&'static str, String> = std::collections::HashMap::new(); - for line in stdout.lines() { - if line.is_empty() { - continue; - } - let mut fields = line.split('|'); - let tbl = fields.next().unwrap_or("").trim(); - fields.next(); // cid - let name = fields.next().unwrap_or("").trim(); - if tbl.is_empty() || name.is_empty() { - continue; - } - // Resolve the static &'static str table name from the requested list. - // Columns are joined with ", " — EXACTLY matching the format of - // fetch_columns_excluding_rowid, because remap_table_cols later - // splits on ", " to rebuild the inline SELECT column list. A bare - // comma here would collapse all columns into one split element and - // silently corrupt the id-remap INSERT. - if let Some(slot) = tables.iter().find(|t| **t == tbl) { - let entry = out.entry(slot).or_default(); - if !entry.is_empty() { - entry.push_str(", "); - } - entry.push_str(name); - } - } - Ok(out) -} - -/// Merge per-module DBs into a single main DB using the sqlite3 CLI. -/// -/// See module docs for the strategy (schema-preserving copy + id remap -/// for modules i > 0). project_ids are NOT remapped — the scheduler -/// assigns unique project_ids to each worker. -pub(super) fn merge_module_dbs(main_db: &str, module_db_paths: &[String]) -> MergeResult { - let start = Instant::now(); - // v0.6 (perf): per-phase timers so the merge cost can be attributed to - // WAL checkpointing, schema introspection, or the final sqlite3 exec. - // `t_checkpoint` stays at `start`; the rest are re-stamped at each phase - // boundary, so the initial `= start` here is just a safe default. - let t_checkpoint = start; - #[allow(unused_assignments)] - let mut t_schema = start; - #[allow(unused_assignments)] - let mut t_schema_read = start; - #[allow(unused_assignments)] - let mut t_columns = start; - #[allow(unused_assignments)] - let mut t_sqlite = start; - - if module_db_paths.is_empty() { - return MergeResult { - merged: false, - main_db_path: main_db.to_string(), - tables_merged: 0, - rows_merged: 0, - duration_ms: 0, - error: Some( - "no module DBs to merge [module=scheduler, method=merge_module_dbs]".to_string(), - ), - }; - } - - // ── Step 1: checkpoint WAL on each module DB ──────────────── - // The worker process exits immediately after writing, but the WAL - // file may still have pending data not yet checkpointed to the - // main DB file. If we ATTACH such a DB, sqlite3 may fail with - // "database is locked" when it tries to recover the WAL. - // After checkpoint, remove the -wal and -shm sidecar files: safe - // after a TRUNCATE checkpoint (WAL is empty, SHM recreated on - // next access), prevents stale shared-memory lock errors. - for db_path in module_db_paths { - if let Err(e) = checkpoint_module_db(db_path) { - return MergeResult { - merged: false, - main_db_path: main_db.to_string(), - tables_merged: 0, - rows_merged: 0, - duration_ms: start.elapsed().as_millis() as u64, - error: Some(format!( - "wal_checkpoint failed for {}: {} [module=scheduler, method=merge_module_dbs]", - db_path, e - )), - }; - } - let _ = std::fs::remove_file(format!("{}-wal", db_path)); - let _ = std::fs::remove_file(format!("{}-shm", db_path)); - } - t_schema = Instant::now(); - - // ── Step 2: read schema + table list from first module DB ── - // This single sqlite3 call returns both the CREATE TABLE statements - // (preserving PRIMARY KEY / UNIQUE / AUTOINCREMENT constraints — - // `CREATE TABLE AS SELECT ... WHERE 0` would drop them, breaking - // `INSERT OR IGNORE` deduplication) AND the set of tables that - // actually exist in the first module DB. The existing-tables set - // lets us skip TABLE_SPECS entries missing from the main DB (e.g. - // adjacency/adjacency_rev when async was skipped) in the final - // COUNT(*) query and the actual_tables_merged count — without a - // second sqlite3 round-trip. - let (schema_sql, main_db_existing_tables) = match read_schema_and_tables(&module_db_paths[0]) { - Ok(pair) => pair, - Err(e) => { - return MergeResult { - merged: false, - main_db_path: main_db.to_string(), - tables_merged: 0, - rows_merged: 0, - duration_ms: start.elapsed().as_millis() as u64, - error: Some(format!( - "read schema failed: {} [module=scheduler, method=merge_module_dbs]", - e - )), - }; - } - }; - let actual_tables_merged: u32 = TABLE_SPECS - .iter() - .filter(|s| main_db_existing_tables.contains(s.name)) - .count() as u32; - t_schema_read = Instant::now(); - - // ── Step 2b: fetch column lists for skip_rowid tables ────── - // Dynamically query column names (excluding `rowid`) from module - // 0's schema so future ALTER TABLE ADD COLUMN migrations are - // automatically picked up. A hardcoded column list would silently - // lose data for new columns (SQLite fills them with DEFAULT - // instead of the worker-written value on INSERT-with-fewer-cols). - // All modules share the same schema, so one fetch from module 0 - // covers all modules. - // v0.6 (perf): fetch ALL needed column lists in a single sqlite3 spawn - // instead of one spawn per table. On a large module DB each spawn + open - // costs ~59ms, so the old per-table loop (~13 spawns) dominated the merge - // (~770ms of the observed ~1.4s). Column order from PRAGMA table_info - // matches SELECT *, so inline SELECTs stay byte-identical. - let mut cols_to_fetch: Vec<&'static str> = Vec::new(); - for spec in TABLE_SPECS { - if main_db_existing_tables.contains(spec.name) - && (spec.skip_rowid || !spec.remap_cols.is_empty()) - { - cols_to_fetch.push(spec.name); - } - } - let all_cols = match fetch_all_columns_excluding_rowid(&module_db_paths[0], &cols_to_fetch) { - Ok(m) => m, - Err(e) => { - return MergeResult { - merged: false, - main_db_path: main_db.to_string(), - tables_merged: 0, - rows_merged: 0, - duration_ms: start.elapsed().as_millis() as u64, - error: Some(format!( - "fetch_all_columns_excluding_rowid failed: {} [module=scheduler, method=merge_module_dbs]", - e - )), - }; - } - }; - let mut skip_rowid_cols: std::collections::HashMap<&'static str, String> = - std::collections::HashMap::new(); - let mut remap_table_cols: std::collections::HashMap<&'static str, Vec> = - std::collections::HashMap::new(); - for spec in TABLE_SPECS { - if !main_db_existing_tables.contains(spec.name) { - continue; - } - if let Some(cols) = all_cols.get(spec.name) { - if spec.skip_rowid { - skip_rowid_cols.insert(spec.name, cols.clone()); - } else if !spec.remap_cols.is_empty() { - remap_table_cols - .insert(spec.name, cols.split(", ").map(|s| s.to_string()).collect()); - } - } - } - - t_columns = Instant::now(); - - // ── Step 3: build merge SQL script ────────────────────────── - // MEMORY journal mode avoids WAL mutex contention with the WAL- - // mode attached module DBs. busy_timeout=10000 waits for any - // lingering OS lock from worker cleanup. foreign_keys=OFF because - // we insert in parent-first order with id remapping; FK checks - // would reject intermediate states. - let mut sql = String::new(); - sql.push_str("PRAGMA busy_timeout=10000;\n"); - sql.push_str("PRAGMA journal_mode=MEMORY;\n"); - sql.push_str("PRAGMA synchronous=OFF;\n"); - sql.push_str("PRAGMA foreign_keys=OFF;\n"); - - // ── Step 4: merge each module ─────────────────────────────── - for (i, db_path) in module_db_paths.iter().enumerate() { - let alias = format!("m{}", i); - // Each module is merged in its OWN transaction; the DB is - // DETACHed only AFTER COMMIT (see H4 in module docs) because - // DETACH inside an open transaction fails with "database mN - // is locked" when a temp table was built FROM the attached DB. - sql.push_str("BEGIN;\n"); - if i == 0 { - // Module 0 lays down the schema (read from its - // sqlite_master) so the INSERTs below have a target. - sql.push_str(&schema_sql); - sql.push('\n'); - } - // Escape single quotes by doubling them (' -> ''). Module DB - // paths embed the module name, which comes from a directory - // name; a dir like `O'Brien` or `it's` would terminate the - // SQL string literal early and break the ATTACH (merge fails - // for the whole module). SQLite follows the SQL standard: - // `''` inside a `'...'` literal is a single quote. Alias is - // generated as `m{i}` so it never needs escaping. See H3. - let escaped_db_path = db_path.replace('\'', "''"); - sql.push_str(&format!( - "ATTACH DATABASE '{}' AS {};\n", - escaped_db_path, alias - )); - - // v0.6 (perf): every worker builds the full schema via - // createSchema() (CREATE TABLE IF NOT EXISTS is idempotent), so all - // modules share the SAME table set as module 0. We reuse - // `main_db_existing_tables` instead of spawning `list_tables_in_db` - // once per module — each such spawn opens a large module DB (~59ms), - // so for N modules this removes N expensive sqlite3 processes. The - // only tables a module may lack (adjacency/adjacency_rev, created by - // the async pass the worker skips via CODESCOPE_SKIP_ASYNC=1) are - // consistently absent from EVERY module, including module 0, so the - // shared set is still accurate. `SELECT * FROM {a}.{t}` for a table - // missing from module 0 is skipped because the loop guards on - // main_db_existing_tables. - let existing_tables = &main_db_existing_tables; - - if i == 0 { - // Module 0: INSERT OR IGNORE directly. project_ids are - // already unique (scheduler passes 1, 2, 3, ... to each - // worker), and id collisions don't exist yet (first - // module). - for spec in TABLE_SPECS { - if !existing_tables.contains(spec.name) { - continue; - } - let cols = skip_rowid_cols.get(spec.name).map(|s| s.as_str()); - sql.push_str(&build_insert_sql(spec, &alias, cols)); - } - // (Module 0's DB is detached after COMMIT — see the - // DETACH at the end of the loop body, outside the txn.) - } else { - // Module i > 0: use temp tables to remap ids so they - // don't collide with previously-merged modules. - // - // Build _offsets temp table with MAX(id) for each table - // in the main DB (before this module is merged). These - // are the per-table offsets added to incoming ids. - sql.push_str("CREATE TEMP TABLE _offsets AS SELECT\n"); - let mut first = true; - for tbl in OFFSET_TABLES { - if !first { - sql.push_str(",\n"); - } - sql.push_str(&format!( - "(SELECT COALESCE(MAX(id), 0) FROM {}) AS _{}_offset", - tbl, tbl - )); - first = false; - } - sql.push_str(";\n"); - - // For each table: copy to temp, remap ids, INSERT OR IGNORE. - for spec in TABLE_SPECS { - if !existing_tables.contains(spec.name) { - continue; - } - if spec.remap_cols.is_empty() { - // No id columns to remap (e.g., file_scan_state - // whose PK is (project_id, file_path), or - // semantic_records which uses skip_rowid). - let cols = skip_rowid_cols.get(spec.name).map(|s| s.as_str()); - sql.push_str(&build_insert_sql(spec, &alias, cols)); - continue; - } - - // v0.6 (perf): inline the id offsets into a single SELECT - // (see build_remap_insert_sql) — avoids the old CREATE TEMP - // TABLE + per-column UPDATE + INSERT + DROP round-trip per - // table. INSERT OR IGNORE dedupes on PK/UNIQUE constraints. - let cols = remap_table_cols - .get(spec.name) - .expect("remap_table_cols must be populated for remap tables"); - sql.push_str(&build_remap_insert_sql(spec, &alias, cols)); - } - - sql.push_str("DROP TABLE _offsets;\n"); - } - - // Commit this module's merge, THEN detach (DETACH must be - // outside the transaction — see H4 note at loop top). This - // keeps live attachments at <=1 and scales past SQLite's - // 10-attached-DB limit regardless of module/worker count. - sql.push_str("COMMIT;\n"); - sql.push_str(&format!("DETACH DATABASE {};\n", alias)); - } - - // Count total rows across all merged tables for reporting. Only - // sum tables that exist in the main DB (whose schema comes from - // module 0). Skipping this guard would fail with "no such table" - // if any TABLE_SPECS entry is missing from the main DB (e.g. - // adjacency/adjacency_rev when the async pass was skipped). - sql.push_str("SELECT ("); - let mut first = true; - for spec in TABLE_SPECS { - if !main_db_existing_tables.contains(spec.name) { - continue; - } - if !first { - sql.push_str(" + "); - } - sql.push_str(&format!("(SELECT COUNT(*) FROM {})", spec.name)); - first = false; - } - if first { - // No tables exist — emit a literal 0 so the SELECT is still valid. - sql.push('0'); - } - sql.push_str(");\n"); - - // ── Step 5: run the merge via sqlite3 CLI ─────────────────── - t_sqlite = Instant::now(); - let mut cmd = Command::new("sqlite3"); - cmd.arg(main_db) - .stdin(Stdio::piped()) - .stdout(Stdio::piped()) - .stderr(Stdio::piped()); - - let child = match cmd.spawn() { - Ok(c) => c, - Err(e) => { - return MergeResult { - merged: false, - main_db_path: main_db.to_string(), - tables_merged: 0, - rows_merged: 0, - duration_ms: start.elapsed().as_millis() as u64, - error: Some(format!( - "sqlite3 spawn failed: {} [module=scheduler, method=merge_module_dbs]", - e - )), - }; - } - }; - - // Write SQL to stdin and close it so sqlite3 processes the script. - use std::io::Write; - let mut child = child; - if let Some(mut stdin) = child.stdin.take() - && stdin.write_all(sql.as_bytes()).is_err() - { - // Continue — the error will surface as a non-zero exit code. - } - drop(child.stdin.take()); // close stdin to signal EOF - - let output = match child.wait_with_output() { - Ok(o) => o, - Err(e) => { - return MergeResult { - merged: false, - main_db_path: main_db.to_string(), - tables_merged: 0, - rows_merged: 0, - duration_ms: start.elapsed().as_millis() as u64, - error: Some(format!( - "sqlite3 wait failed: {} [module=scheduler, method=merge_module_dbs]", - e - )), - }; - } - }; - - if !output.status.success() { - let stderr = String::from_utf8_lossy(&output.stderr).to_string(); - return MergeResult { - merged: false, - main_db_path: main_db.to_string(), - tables_merged: 0, - rows_merged: 0, - duration_ms: start.elapsed().as_millis() as u64, - error: Some(format!( - "sqlite3 exit={} stderr_tail={:?} [module=scheduler, method=merge_module_dbs]", - output.status.code().unwrap_or(-1), - stderr.lines().last().unwrap_or("") - )), - }; - } - - // Parse the total row count from the last stdout line. - let stdout = String::from_utf8_lossy(&output.stdout).to_string(); - let rows_merged = stdout - .lines() - .last() - .and_then(|l| l.trim().parse::().ok()) - .unwrap_or(0); - - // v0.6 (perf): attribute merge time so a large-project merge (rust: - // 3.3s / 4.86M rows) can be targeted: WAL checkpointing of the module - // DBs, schema introspection, column introspection, or the final sqlite3 - // exec (schema DDL + id-remap INSERTs + row COUNT). - eprintln!( - "[scheduler] merge_module_dbs: checkpoint={}ms schema_read={}ms \ - columns={}ms sql_build={}ms sqlite_exec={}ms total={}ms \ - rows={} tables={} [module=scheduler, method=merge_module_dbs]", - t_schema.duration_since(t_checkpoint).as_millis(), - t_schema_read.duration_since(t_schema).as_millis(), - t_columns.duration_since(t_schema_read).as_millis(), - t_sqlite.duration_since(t_columns).as_millis(), - t_sqlite.elapsed().as_millis(), - start.elapsed().as_millis(), - rows_merged, - actual_tables_merged, - ); - - MergeResult { - merged: true, - main_db_path: main_db.to_string(), - tables_merged: actual_tables_merged, - rows_merged, - duration_ms: start.elapsed().as_millis() as u64, - error: None, - } -} - /// Read CREATE TABLE statements AND the set of existing tables from a /// DB's sqlite_master in a single sqlite3 round-trip. /// @@ -1014,7 +534,10 @@ mod tests { /// don't need data). fn make_test_db(suffix: &str, table_sql: &str) -> String { let pid = std::process::id(); - let path = format!("/tmp/codescope_merge_test_{}_{}.db", pid, suffix); + let path = std::env::temp_dir() + .join(format!("codescope_merge_test_{pid}_{suffix}.db")) + .to_string_lossy() + .into_owned(); let _ = std::fs::remove_file(&path); let sql = format!("CREATE TABLE t ({});", table_sql); let status = Command::new("sqlite3") @@ -1225,8 +748,14 @@ mod tests { // to the old CREATE TEMP TABLE + UPDATE approach. Guards against // silent precision loss when module i>0 ids collide with module 0. let pid = std::process::id(); - let main = format!("/tmp/codescope_remap_test_{main}.db", main = pid); - let m1 = format!("/tmp/codescope_remap_test_{m1}_m1.db", m1 = pid); + let main = std::env::temp_dir() + .join(format!("codescope_remap_test_{pid}.db")) + .to_string_lossy() + .into_owned(); + let m1 = std::env::temp_dir() + .join(format!("codescope_remap_test_{pid}_m1.db")) + .to_string_lossy() + .into_owned(); let _ = std::fs::remove_file(&main); let _ = std::fs::remove_file(&m1); diff --git a/server/src/scheduler/merge_driver.rs b/server/src/scheduler/merge_driver.rs new file mode 100644 index 0000000..d37e50e --- /dev/null +++ b/server/src/scheduler/merge_driver.rs @@ -0,0 +1,663 @@ +// merge_driver.rs — the per-unit DB merge driver. +// +// Split out of merge.rs (see plan/rules/code_rules.md 1000-line rule). +// merge_module_dbs is the entry point the parallel scheduler calls after +// every worker has written its own DB: it copies the unit schemas across, +// INSERT..SELECTs each table with row ids remapped so a unit's local ids +// cannot collide with the main DB's, and reports rows merged. +// +// It is a sibling of merge_fetch.rs (which discovers each unit DB's real +// column list) and uses `use super::*` to reach merge.rs's TableSpec +// definitions and SQL builders without widening their visibility. + +// pub(crate) rather than pub(super): merge.rs re-exports this to its own +// parent (the scheduler), so the item must be visible there too — a +// re-export can never widen beyond the item's own visibility. +use super::*; + +/// Merge per-module DBs into a single main DB using the sqlite3 CLI. +/// +/// See module docs for the strategy (schema-preserving copy + id remap +/// for modules i > 0). project_ids are NOT remapped — the scheduler +/// assigns unique project_ids to each worker. +/// Make the merged DB describe exactly ONE project. +/// +/// Every module/worker DB carries its own `projects` row and its own +/// project_id (`worker_id + 1`), and the merge keeps those ids — its remap is +/// for entity/relation ids. So the merged file used to contain N partial +/// "projects" and no row for the directory that was actually indexed: +/// CodeScope's 1788 entities landed as project 1 → 1346 and project 2 → 442, +/// goagent's 24987 as 1 → 22517, 2 → 1298, 3 → 969, 4 → 160, 5 → …. A query +/// with `project_id=1` therefore saw a *partial* project and reported it as +/// the whole thing, and a consumer that resolves a project by path — the MCP +/// server — found no row at all and silently re-indexed from scratch. +/// +/// The rows are rewritten to one id (1, the first worker's) and the `projects` +/// row for the indexed directory is written. Tables are discovered by asking +/// which ones have a `project_id` column (39 of them here), not by a hardcoded +/// list, so a new table cannot quietly keep stale ids. +/// +/// @param main_db The merged DB (already built from the module DBs). +/// @param project_path The directory that was indexed — stored so +/// `get_project_id_by_path` can adopt this DB. +/// @return Ok(()) when the DB now describes one project, Err(summary) else. +fn unify_project(main_db: &str, project_path: &str) -> Result<(), String> { + let list_sql = "SELECT m.name FROM sqlite_master m WHERE m.type='table' \ + AND m.name NOT LIKE 'sqlite_%' AND EXISTS (SELECT 1 FROM \ + pragma_table_info(m.name) WHERE name='project_id') ORDER BY m.name;"; + let output = Command::new("sqlite3") + .arg(main_db) + .arg(list_sql) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .map_err(|e| { + format!( + "unify_project spawn failed: {} [module=scheduler, method=unify_project]", + e + ) + })?; + if !output.status.success() { + return Err(format!( + "unify_project table discovery failed: {} [module=scheduler, method=unify_project]", + String::from_utf8_lossy(&output.stderr).trim() + )); + } + let tables: Vec = String::from_utf8_lossy(&output.stdout) + .lines() + .map(|l| l.trim().to_string()) + .filter(|l| !l.is_empty()) + .collect(); + if tables.is_empty() { + return Err( + "unify_project found no project-scoped tables — the merged DB does not \ + look like an index [module=scheduler, method=unify_project]" + .to_string(), + ); + } + + // Quote identifiers and string literals: table names come from sqlite_master + // and the path from the caller, so neither may be pasted in raw. + let quote_ident = |s: &str| format!("\"{}\"", s.replace('"', "\"\"")); + let quote_str = |s: &str| format!("'{}'", s.replace('\'', "''")); + let name = project_path + .trim_end_matches('/') + .rsplit('/') + .next() + .unwrap_or(project_path); + + let mut sql = String::from("BEGIN;\n"); + // The merged DB is built from the module DBs' data tables, so it may not + // have `projects` at all (the engine creates it on open, which has not + // happened yet). DDL here is byte-identical to the engine's, so whichever + // comes first the schema is the same. Then any per-worker rows go: the + // merged DB gets exactly one. + sql.push_str( + "CREATE TABLE IF NOT EXISTS projects (\n\ + id INTEGER PRIMARY KEY AUTOINCREMENT,\n\ + root_path TEXT NOT NULL UNIQUE,\n\ + name TEXT NOT NULL,\n\ + created_at TEXT DEFAULT (datetime('now'))\n\ + );\n", + ); + sql.push_str("DELETE FROM projects;\n"); + sql.push_str(&format!( + "INSERT INTO projects (id, root_path, name) VALUES (1, {}, {});\n", + quote_str(project_path), + quote_str(name) + )); + for t in &tables { + sql.push_str(&format!( + "UPDATE {} SET project_id=1 WHERE project_id<>1;\n", + quote_ident(t) + )); + } + sql.push_str("COMMIT;\n"); + + let output = Command::new("sqlite3") + .arg(main_db) + .arg(&sql) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .map_err(|e| { + format!( + "unify_project exec failed: {} [module=scheduler, method=unify_project]", + e + ) + })?; + if !output.status.success() { + return Err(format!( + "unify_project could not unify {} tables: {} [module=scheduler, method=unify_project]", + tables.len(), + String::from_utf8_lossy(&output.stderr).trim() + )); + } + Ok(()) +} + +pub(crate) fn merge_module_dbs( + main_db: &str, + module_db_paths: &[String], + project_path: &str, +) -> MergeResult { + let start = Instant::now(); + // v0.6 (perf): per-phase timers so the merge cost can be attributed to + // WAL checkpointing, schema introspection, or the final sqlite3 exec. + // `t_checkpoint` stays at `start`; the rest are re-stamped at each phase + // boundary, so the initial `= start` here is just a safe default. + let t_checkpoint = start; + #[allow(unused_assignments)] + let mut t_schema = start; + #[allow(unused_assignments)] + let mut t_schema_read = start; + #[allow(unused_assignments)] + let mut t_columns = start; + #[allow(unused_assignments)] + let mut t_sqlite = start; + + if module_db_paths.is_empty() { + return MergeResult { + merged: false, + main_db_path: main_db.to_string(), + tables_merged: 0, + rows_merged: 0, + duration_ms: 0, + error: Some( + "no module DBs to merge [module=scheduler, method=merge_module_dbs]".to_string(), + ), + }; + } + + // ── Step 1: checkpoint WAL on each module DB ──────────────── + // The worker process exits immediately after writing, but the WAL + // file may still have pending data not yet checkpointed to the + // main DB file. If we ATTACH such a DB, sqlite3 may fail with + // "database is locked" when it tries to recover the WAL. + // After checkpoint, remove the -wal and -shm sidecar files: safe + // after a TRUNCATE checkpoint (WAL is empty, SHM recreated on + // next access), prevents stale shared-memory lock errors. + for db_path in module_db_paths { + if let Err(e) = checkpoint_module_db(db_path) { + return MergeResult { + merged: false, + main_db_path: main_db.to_string(), + tables_merged: 0, + rows_merged: 0, + duration_ms: start.elapsed().as_millis() as u64, + error: Some(format!( + "wal_checkpoint failed for {}: {} [module=scheduler, method=merge_module_dbs]", + db_path, e + )), + }; + } + let _ = std::fs::remove_file(format!("{}-wal", db_path)); + let _ = std::fs::remove_file(format!("{}-shm", db_path)); + } + t_schema = Instant::now(); + + // ── Step 2: read schema + table list from first module DB ── + // This single sqlite3 call returns both the CREATE TABLE statements + // (preserving PRIMARY KEY / UNIQUE / AUTOINCREMENT constraints — + // `CREATE TABLE AS SELECT ... WHERE 0` would drop them, breaking + // `INSERT OR IGNORE` deduplication) AND the set of tables that + // actually exist in the first module DB. The existing-tables set + // lets us skip TABLE_SPECS entries missing from the main DB (e.g. + // adjacency/adjacency_rev when async was skipped) in the final + // COUNT(*) query and the actual_tables_merged count — without a + // second sqlite3 round-trip. + let (schema_sql, main_db_existing_tables) = match read_schema_and_tables(&module_db_paths[0]) { + Ok(pair) => pair, + Err(e) => { + return MergeResult { + merged: false, + main_db_path: main_db.to_string(), + tables_merged: 0, + rows_merged: 0, + duration_ms: start.elapsed().as_millis() as u64, + error: Some(format!( + "read schema failed: {} [module=scheduler, method=merge_module_dbs]", + e + )), + }; + } + }; + let actual_tables_merged: u32 = TABLE_SPECS + .iter() + .filter(|s| main_db_existing_tables.contains(s.name)) + .count() as u32; + t_schema_read = Instant::now(); + + // ── Step 2b: fetch column lists for skip_rowid tables ────── + // Dynamically query column names (excluding `rowid`) from module + // 0's schema so future ALTER TABLE ADD COLUMN migrations are + // automatically picked up. A hardcoded column list would silently + // lose data for new columns (SQLite fills them with DEFAULT + // instead of the worker-written value on INSERT-with-fewer-cols). + // All modules share the same schema, so one fetch from module 0 + // covers all modules. + // v0.6 (perf): fetch ALL needed column lists in a single sqlite3 spawn + // instead of one spawn per table. On a large module DB each spawn + open + // costs ~59ms, so the old per-table loop (~13 spawns) dominated the merge + // (~770ms of the observed ~1.4s). Column order from PRAGMA table_info + // matches SELECT *, so inline SELECTs stay byte-identical. + // Every table present in the main DB is fetched, not only the + // skip_rowid/remap ones: the same fetch feeds the schema-consistency check + // below, which has to compare EVERY table the merge copies — a plain + // `SELECT *` table whose modules disagree aborts the script mid-way, after + // earlier modules were committed. Still one spawn: the query already + // batches all tables with UNION ALL. + let mut cols_to_fetch: Vec<&'static str> = Vec::new(); + for spec in TABLE_SPECS { + if main_db_existing_tables.contains(spec.name) { + cols_to_fetch.push(spec.name); + } + } + let all_cols = match fetch_all_columns_excluding_rowid(&module_db_paths[0], &cols_to_fetch) { + Ok(m) => m, + Err(e) => { + return MergeResult { + merged: false, + main_db_path: main_db.to_string(), + tables_merged: 0, + rows_merged: 0, + duration_ms: start.elapsed().as_millis() as u64, + error: Some(format!( + "fetch_all_columns_excluding_rowid failed: {} [module=scheduler, method=merge_module_dbs]", + e + )), + }; + } + }; + // Fail BEFORE writing anything if the module DBs disagree about the schema. + // The merge generates its column lists from module 0 and applies them to + // every module, so a module left over from another build (or preserved by + // an incremental run) either loses its extra column silently or aborts the + // script with earlier modules already committed. See + // check_module_schema_consistency. + if let Err(e) = check_module_schema_consistency(module_db_paths, &all_cols) { + return MergeResult { + merged: false, + main_db_path: main_db.to_string(), + tables_merged: 0, + rows_merged: 0, + duration_ms: start.elapsed().as_millis() as u64, + error: Some(e), + }; + } + + let mut skip_rowid_cols: std::collections::HashMap<&'static str, String> = + std::collections::HashMap::new(); + let mut remap_table_cols: std::collections::HashMap<&'static str, Vec> = + std::collections::HashMap::new(); + for spec in TABLE_SPECS { + if !main_db_existing_tables.contains(spec.name) { + continue; + } + if let Some(cols) = all_cols.get(spec.name) { + if spec.skip_rowid { + skip_rowid_cols.insert(spec.name, cols.clone()); + } else if !spec.remap_cols.is_empty() { + remap_table_cols + .insert(spec.name, cols.split(", ").map(|s| s.to_string()).collect()); + } + } + } + + t_columns = Instant::now(); + + // ── Step 3: build merge SQL script ────────────────────────── + // MEMORY journal mode avoids WAL mutex contention with the WAL- + // mode attached module DBs. busy_timeout=10000 waits for any + // lingering OS lock from worker cleanup. foreign_keys=OFF because + // we insert in parent-first order with id remapping; FK checks + // would reject intermediate states. + let mut sql = String::new(); + sql.push_str("PRAGMA busy_timeout=10000;\n"); + sql.push_str("PRAGMA journal_mode=MEMORY;\n"); + sql.push_str("PRAGMA synchronous=OFF;\n"); + sql.push_str("PRAGMA foreign_keys=OFF;\n"); + + // ── Step 4: merge each module ─────────────────────────────── + for (i, db_path) in module_db_paths.iter().enumerate() { + let alias = format!("m{}", i); + // Each module is merged in its OWN transaction; the DB is + // DETACHed only AFTER COMMIT (see H4 in module docs) because + // DETACH inside an open transaction fails with "database mN + // is locked" when a temp table was built FROM the attached DB. + sql.push_str("BEGIN;\n"); + if i == 0 { + // Module 0 lays down the schema (read from its + // sqlite_master) so the INSERTs below have a target. + sql.push_str(&schema_sql); + sql.push('\n'); + } + // Escape single quotes by doubling them (' -> ''). Module DB + // paths embed the module name, which comes from a directory + // name; a dir like `O'Brien` or `it's` would terminate the + // SQL string literal early and break the ATTACH (merge fails + // for the whole module). SQLite follows the SQL standard: + // `''` inside a `'...'` literal is a single quote. Alias is + // generated as `m{i}` so it never needs escaping. See H3. + let escaped_db_path = db_path.replace('\'', "''"); + sql.push_str(&format!( + "ATTACH DATABASE '{}' AS {};\n", + escaped_db_path, alias + )); + + // v0.6 (perf): every worker builds the full schema via + // createSchema() (CREATE TABLE IF NOT EXISTS is idempotent), so all + // modules share the SAME table set as module 0. We reuse + // `main_db_existing_tables` instead of spawning `list_tables_in_db` + // once per module — each such spawn opens a large module DB (~59ms), + // so for N modules this removes N expensive sqlite3 processes. The + // only tables a module may lack (adjacency/adjacency_rev, created by + // the async pass the worker skips via CODESCOPE_SKIP_ASYNC=1) are + // consistently absent from EVERY module, including module 0, so the + // shared set is still accurate. `SELECT * FROM {a}.{t}` for a table + // missing from module 0 is skipped because the loop guards on + // main_db_existing_tables. + let existing_tables = &main_db_existing_tables; + + if i == 0 { + // Module 0: INSERT OR IGNORE directly. project_ids are + // already unique (scheduler passes 1, 2, 3, ... to each + // worker), and id collisions don't exist yet (first + // module). + for spec in TABLE_SPECS { + if !existing_tables.contains(spec.name) { + continue; + } + let cols = skip_rowid_cols.get(spec.name).map(|s| s.as_str()); + sql.push_str(&build_insert_sql(spec, &alias, cols)); + } + // (Module 0's DB is detached after COMMIT — see the + // DETACH at the end of the loop body, outside the txn.) + } else { + // Module i > 0: use temp tables to remap ids so they + // don't collide with previously-merged modules. + // + // Build _offsets temp table with MAX(id) for each table + // in the main DB (before this module is merged). These + // are the per-table offsets added to incoming ids. + sql.push_str("CREATE TEMP TABLE _offsets AS SELECT\n"); + let mut first = true; + for tbl in OFFSET_TABLES { + if !first { + sql.push_str(",\n"); + } + sql.push_str(&format!( + "(SELECT COALESCE(MAX(id), 0) FROM {}) AS _{}_offset", + tbl, tbl + )); + first = false; + } + sql.push_str(";\n"); + + // For each table: copy to temp, remap ids, INSERT OR IGNORE. + for spec in TABLE_SPECS { + if !existing_tables.contains(spec.name) { + continue; + } + if spec.remap_cols.is_empty() { + // No id columns to remap (e.g., file_scan_state + // whose PK is (project_id, file_path), or + // semantic_records which uses skip_rowid). + let cols = skip_rowid_cols.get(spec.name).map(|s| s.as_str()); + sql.push_str(&build_insert_sql(spec, &alias, cols)); + continue; + } + + // v0.6 (perf): inline the id offsets into a single SELECT + // (see build_remap_insert_sql) — avoids the old CREATE TEMP + // TABLE + per-column UPDATE + INSERT + DROP round-trip per + // table. INSERT OR IGNORE dedupes on PK/UNIQUE constraints. + let cols = remap_table_cols + .get(spec.name) + .expect("remap_table_cols must be populated for remap tables"); + sql.push_str(&build_remap_insert_sql(spec, &alias, cols)); + } + + sql.push_str("DROP TABLE _offsets;\n"); + } + + // Commit this module's merge, THEN detach (DETACH must be + // outside the transaction — see H4 note at loop top). This + // keeps live attachments at <=1 and scales past SQLite's + // 10-attached-DB limit regardless of module/worker count. + sql.push_str("COMMIT;\n"); + sql.push_str(&format!("DETACH DATABASE {};\n", alias)); + } + + // Count total rows across all merged tables for reporting. Only + // sum tables that exist in the main DB (whose schema comes from + // module 0). Skipping this guard would fail with "no such table" + // if any TABLE_SPECS entry is missing from the main DB (e.g. + // adjacency/adjacency_rev when the async pass was skipped). + sql.push_str("SELECT ("); + let mut first = true; + for spec in TABLE_SPECS { + if !main_db_existing_tables.contains(spec.name) { + continue; + } + if !first { + sql.push_str(" + "); + } + sql.push_str(&format!("(SELECT COUNT(*) FROM {})", spec.name)); + first = false; + } + if first { + // No tables exist — emit a literal 0 so the SELECT is still valid. + sql.push('0'); + } + sql.push_str(");\n"); + + // ── Step 5: run the merge via sqlite3 CLI ─────────────────── + t_sqlite = Instant::now(); + let mut cmd = Command::new("sqlite3"); + cmd.arg(main_db) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + + let child = match cmd.spawn() { + Ok(c) => c, + Err(e) => { + return MergeResult { + merged: false, + main_db_path: main_db.to_string(), + tables_merged: 0, + rows_merged: 0, + duration_ms: start.elapsed().as_millis() as u64, + error: Some(format!( + "sqlite3 spawn failed: {} [module=scheduler, method=merge_module_dbs]", + e + )), + }; + } + }; + + // Write SQL to stdin and close it so sqlite3 processes the script. + // A failed write MUST fail the merge: sqlite3 can consume only a prefix + // of the script and still exit 0, which would commit a partial merge and + // report merged:true (code_rules §5: no silent error handling). + use std::io::Write; + let mut child = child; + if let Some(mut stdin) = child.stdin.take() + && let Err(e) = stdin.write_all(sql.as_bytes()) + { + drop(stdin); // close the pipe + let _ = child.wait(); + return MergeResult { + merged: false, + main_db_path: main_db.to_string(), + tables_merged: 0, + rows_merged: 0, + duration_ms: start.elapsed().as_millis() as u64, + error: Some(format!( + "sqlite3 stdin write failed: {} [module=scheduler, method=merge_module_dbs]", + e + )), + }; + } + // stdin is dropped at the end of the `if let` above, closing the pipe so + // sqlite3 sees EOF. + + let output = match child.wait_with_output() { + Ok(o) => o, + Err(e) => { + return MergeResult { + merged: false, + main_db_path: main_db.to_string(), + tables_merged: 0, + rows_merged: 0, + duration_ms: start.elapsed().as_millis() as u64, + error: Some(format!( + "sqlite3 wait failed: {} [module=scheduler, method=merge_module_dbs]", + e + )), + }; + } + }; + + if !output.status.success() { + let stderr = String::from_utf8_lossy(&output.stderr).to_string(); + return MergeResult { + merged: false, + main_db_path: main_db.to_string(), + tables_merged: 0, + rows_merged: 0, + duration_ms: start.elapsed().as_millis() as u64, + error: Some(format!( + "sqlite3 exit={} stderr_tail={:?} [module=scheduler, method=merge_module_dbs]", + output.status.code().unwrap_or(-1), + stderr.lines().last().unwrap_or("") + )), + }; + } + + // Parse the total row count from the last stdout line. + let stdout = String::from_utf8_lossy(&output.stdout).to_string(); + let rows_merged = stdout + .lines() + .last() + .and_then(|l| l.trim().parse::().ok()) + .unwrap_or(0); + + // The merged file must describe ONE project before anyone reads it: the + // per-worker project_ids would otherwise leave it looking like N partial + // projects with no way to find the one that was indexed. See unify_project. + if let Err(e) = unify_project(main_db, project_path) { + return MergeResult { + merged: false, + main_db_path: main_db.to_string(), + tables_merged: actual_tables_merged, + rows_merged, + duration_ms: start.elapsed().as_millis() as u64, + error: Some(e), + }; + } + + // v0.6 (perf): attribute merge time so a large-project merge (rust: + // 3.3s / 4.86M rows) can be targeted: WAL checkpointing of the module + // DBs, schema introspection, column introspection, or the final sqlite3 + // exec (schema DDL + id-remap INSERTs + row COUNT). + eprintln!( + "[scheduler] merge_module_dbs: checkpoint={}ms schema_read={}ms \ + columns={}ms sql_build={}ms sqlite_exec={}ms total={}ms \ + rows={} tables={} [module=scheduler, method=merge_module_dbs]", + t_schema.duration_since(t_checkpoint).as_millis(), + t_schema_read.duration_since(t_schema).as_millis(), + t_columns.duration_since(t_schema_read).as_millis(), + t_sqlite.duration_since(t_columns).as_millis(), + t_sqlite.elapsed().as_millis(), + start.elapsed().as_millis(), + rows_merged, + actual_tables_merged, + ); + + MergeResult { + merged: true, + main_db_path: main_db.to_string(), + tables_merged: actual_tables_merged, + rows_merged, + duration_ms: start.elapsed().as_millis() as u64, + error: None, + } +} + +#[cfg(test)] +mod tests { + use super::unify_project; + use std::process::{Command, Stdio}; + + fn sqlite(db: &str, sql: &str) -> String { + let out = Command::new("sqlite3") + .arg(db) + .arg(sql) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .unwrap(); + assert!( + out.status.success(), + "sqlite3 failed: {}", + String::from_utf8_lossy(&out.stderr) + ); + String::from_utf8_lossy(&out.stdout).trim().to_string() + } + + #[test] + fn test_unify_project_makes_one_project_of_many_workers() { + // Regression (#26): the merge kept each worker's own project_id, so the + // merged DB looked like N partial projects with no row for the indexed + // directory — a query with project_id=1 saw only the first worker's + // rows, and a consumer resolving the project by path found nothing. + let path = std::env::temp_dir() + .join(format!("codescope_unify_{}.db", std::process::id())) + .to_string_lossy() + .into_owned(); + let _ = std::fs::remove_file(&path); + sqlite( + &path, + "CREATE TABLE projects (id INTEGER PRIMARY KEY, root_path TEXT UNIQUE, name TEXT); + CREATE TABLE entity (id INTEGER PRIMARY KEY, project_id INTEGER, name TEXT); + CREATE TABLE relation (id INTEGER PRIMARY KEY, project_id INTEGER); + CREATE TABLE unrelated (id INTEGER PRIMARY KEY, note TEXT);", + ); + // Three "workers", each with its own projects row and project_id. + sqlite( + &path, + "INSERT INTO projects VALUES (1,'/w/1','w1'),(2,'/w/2','w2'),(3,'/w/3','w3'); + INSERT INTO entity VALUES (1,1,'a'),(2,2,'b'),(3,2,'c'),(4,3,'d'); + INSERT INTO relation VALUES (1,2),(2,3); + INSERT INTO unrelated VALUES (1,'x');", + ); + + unify_project(&path, "/real/project").expect("unify"); + + assert_eq!(sqlite(&path, "SELECT COUNT(*) FROM projects;"), "1"); + assert_eq!(sqlite(&path, "SELECT id FROM projects;"), "1"); + assert_eq!( + sqlite(&path, "SELECT root_path FROM projects;"), + "/real/project" + ); + assert_eq!(sqlite(&path, "SELECT name FROM projects;"), "project"); + // Every project-scoped row now belongs to it, and nothing was lost. + assert_eq!( + sqlite(&path, "SELECT COUNT(DISTINCT project_id) FROM entity;"), + "1" + ); + assert_eq!(sqlite(&path, "SELECT COUNT(*) FROM entity;"), "4"); + assert_eq!( + sqlite(&path, "SELECT COUNT(DISTINCT project_id) FROM relation;"), + "1" + ); + // A table without project_id is left alone. + assert_eq!(sqlite(&path, "SELECT COUNT(*) FROM unrelated;"), "1"); + let _ = std::fs::remove_file(&path); + } +} diff --git a/server/src/scheduler/merge_fetch.rs b/server/src/scheduler/merge_fetch.rs new file mode 100644 index 0000000..66b0083 --- /dev/null +++ b/server/src/scheduler/merge_fetch.rs @@ -0,0 +1,310 @@ +// merge_fetch.rs — single-spawn column discovery across module DBs. +// +// Split out of merge.rs (see plan/rules/code_rules.md 1000-line rule). +// The per-unit merge needs each module DB's real column list before it can +// build an INSERT..SELECT; asking PRAGMA table_info once per table cost one +// sqlite3 spawn per table per module. This collapses it into ONE spawn by +// UNION-ing `pragma_table_info` table-valued function calls for every +// requested table. + +use super::*; + +/// Fetch the non-rowid column lists for many tables in a SINGLE sqlite3 +/// spawn, instead of one spawn per table. +/// +/// v0.6 (perf): `fetch_columns_excluding_rowid` is called once per relevant +/// table inside merge_module_dbs (≈13 tables). Each call spawns a fresh +/// sqlite3 process; on a large module DB (rust: ~74MB) a single spawn + +/// open costs ~59ms, so 13 spawns cost ~770ms — the dominant cost of the +/// whole merge. This function collapses that into one spawn by UNION-ing +/// `pragma_table_info` table-valued function calls for every requested table. +/// +/// @param db_path Path of the module DB whose schema is read. +/// @param tables Table names whose columns are needed. +/// @return Map from table name to its comma-joined non-rowid column list, +/// preserving `PRAGMA table_info` column order (matches SELECT *). +pub(super) fn fetch_all_columns_excluding_rowid( + db_path: &str, + tables: &[&'static str], +) -> Result, String> { + if tables.is_empty() { + return Ok(std::collections::HashMap::new()); + } + // One query: for each table emit a pragma_table_info scan, tagged with + // the table name so we can group columns back to their table. Column + // `rowid` (the INTEGER PRIMARY KEY AUTOINCREMENT alias) is skipped here + // so it isn't copied on INSERT — same contract as fetch_columns_excluding_rowid. + let mut sql = String::new(); + for (i, t) in tables.iter().enumerate() { + if i > 0 { + sql.push_str(" UNION ALL "); + } + // cid/name are the only fields we need; type/notnull/dflt/pk ignored. + sql.push_str(&format!( + "SELECT '{t}' AS tbl, cid, name FROM pragma_table_info('{t}') WHERE name != 'rowid'", + t = t + )); + } + sql.push_str(" ORDER BY tbl, cid;"); + + let output = Command::new("sqlite3") + .arg(db_path) + .arg(&sql) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .map_err(|e| { + format!( + "spawn: {} [module=scheduler, method=fetch_all_columns_excluding_rowid]", + e + ) + })?; + if !output.status.success() { + let stderr = String::from_utf8_lossy(&output.stderr).to_string(); + return Err(format!( + "sqlite3 exit={}: {} [module=scheduler, method=fetch_all_columns_excluding_rowid]", + output.status.code().unwrap_or(-1), + stderr + )); + } + + // Output rows are `table_name|cid|column_name` (sqlite3 default '|' + // separator). Group columns per table in cid order. + let stdout = String::from_utf8_lossy(&output.stdout).to_string(); + let mut out: std::collections::HashMap<&'static str, String> = std::collections::HashMap::new(); + for line in stdout.lines() { + if line.is_empty() { + continue; + } + let mut fields = line.split('|'); + let tbl = fields.next().unwrap_or("").trim(); + fields.next(); // cid + let name = fields.next().unwrap_or("").trim(); + if tbl.is_empty() || name.is_empty() { + continue; + } + // Resolve the static &'static str table name from the requested list. + // Columns are joined with ", " — EXACTLY matching the format of + // fetch_columns_excluding_rowid, because remap_table_cols later + // splits on ", " to rebuild the inline SELECT column list. A bare + // comma here would collapse all columns into one split element and + // silently corrupt the id-remap INSERT. + if let Some(slot) = tables.iter().find(|t| **t == tbl) { + let entry = out.entry(slot).or_default(); + if !entry.is_empty() { + entry.push_str(", "); + } + entry.push_str(name); + } + } + Ok(out) +} + +/// Verify that every module DB exposes the same columns as the first one for +/// the tables the merge is about to copy. +/// +/// The merge script generates its INSERT column lists from module 0 and applies +/// them to every module (see merge_driver.rs step 2b). That is only sound while +/// the module DBs really do share a schema. A module DB from a different build, +/// or one preserved from an earlier run (`CODESCOPE_DB_PREFIX` incremental +/// mode), can have a column module 0 lacks or lack one module 0 has: +/// +/// * extra column in the module — `SELECT *` / the generated column list fills +/// it with its DEFAULT, so the module's value is dropped with no message; +/// * missing column in the module — the INSERT aborts mid-script, after earlier +/// modules were already committed, leaving a partially merged main.db whose +/// `merge.merged` the caller cannot distinguish from a clean run. +/// +/// Both are silent data loss, so they are checked BEFORE anything is written: +/// the caller gets a `merged: false` result naming the module, the table and +/// the differing columns, and main.db is left untouched. +/// +/// @param module_db_paths Module DBs in merge order (index 0 is the schema +/// source). +/// @param expected_cols Table -> ", "-joined column list, as produced by +/// `fetch_all_columns_excluding_rowid` for module 0. +/// @return Ok(()) when every module matches, Err(summary) otherwise. +pub(super) fn check_module_schema_consistency( + module_db_paths: &[String], + expected_cols: &std::collections::HashMap<&'static str, String>, +) -> Result<(), String> { + if module_db_paths.len() <= 1 || expected_cols.is_empty() { + return Ok(()); + } + // Sorted so the message is stable (HashMap iteration order is not). + let mut tables: Vec<&'static str> = expected_cols.keys().copied().collect(); + tables.sort_unstable(); + + let mut differences: Vec = Vec::new(); + // SQLite attaches at most 10 databases by default and the merge itself + // attaches one module at a time; batching keeps this check to a couple of + // extra processes for realistic module counts. + const ATTACH_BATCH: usize = 8; + for (chunk_index, chunk) in module_db_paths[1..].chunks(ATTACH_BATCH).enumerate() { + let mut sql = String::new(); + for (i, path) in chunk.iter().enumerate() { + // Module paths come from directory names; a quote in one would + // terminate the literal early (same reasoning as the merge's own + // ATTACH — see H3). + sql.push_str(&format!( + "ATTACH DATABASE '{}' AS c{};\n", + path.replace('\'', "''"), + i + )); + } + for (i, _) in chunk.iter().enumerate() { + for t in &tables { + // The two-argument form takes the schema name, so one process + // can read every attached module. A table the module lacks + // yields NULL and is reported as missing. + sql.push_str(&format!( + "SELECT '{}|{}|' || COALESCE((SELECT group_concat(name, ',') \ + FROM pragma_table_info('{}','c{}') WHERE name != 'rowid'), '');\n", + i, t, t, i + )); + } + } + + let output = match Command::new("sqlite3") + .arg(":memory:") + .arg(&sql) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + { + Ok(o) => o, + Err(e) => { + return Err(format!( + "schema check spawn failed: {} [module=scheduler, method=check_module_schema_consistency]", + e + )); + } + }; + if !output.status.success() { + return Err(format!( + "schema check sqlite3 exit={}: {} [module=scheduler, method=check_module_schema_consistency]", + output.status.code().unwrap_or(-1), + String::from_utf8_lossy(&output.stderr).trim() + )); + } + + for line in String::from_utf8_lossy(&output.stdout).lines() { + let mut fields = line.split('|'); + let local = fields.next().unwrap_or("").trim(); + let tbl = fields.next().unwrap_or("").trim(); + let cols = fields.next().unwrap_or("").trim(); + if tbl.is_empty() { + continue; + } + let Some(expected) = expected_cols.get(tbl) else { + continue; + }; + // fetch_all_columns_excluding_rowid joins with ", "; this query + // joins with ",". Normalise whitespace before comparing. + let normalise = |s: &str| s.replace(' ', ""); + if normalise(cols) == normalise(expected) { + continue; + } + let module_index: usize = + local.parse::().unwrap_or(0) + chunk_index * ATTACH_BATCH + 1; + if cols.is_empty() { + differences.push(format!( + "module #{} ({}) has no table '{}'", + module_index, module_db_paths[module_index], tbl + )); + } else { + differences.push(format!( + "module #{} ({}) has columns [{}] for '{}', module #0 has [{}]", + module_index, module_db_paths[module_index], cols, tbl, expected + )); + } + } + } + + if differences.is_empty() { + return Ok(()); + } + let shown: Vec = differences.iter().take(5).cloned().collect(); + let more = differences.len().saturating_sub(shown.len()); + Err(format!( + "module DBs do not share one schema — the merge would silently drop columns or leave a \ + partially merged DB: {}{} [module=scheduler, method=check_module_schema_consistency]", + shown.join("; "), + if more > 0 { + format!(" (+{} more)", more) + } else { + String::new() + } + )) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn make_db(path: &str, ddl: &str) { + let _ = std::fs::remove_file(path); + let out = Command::new("sqlite3").arg(path).arg(ddl).output().unwrap(); + assert!(out.status.success(), "sqlite3 failed for {}", path); + } + + #[test] + fn test_schema_consistency_catches_column_drift() { + // Regression (#21): the merge took its column list from module 0 and + // applied it to every module. A module missing a column aborted the + // script with earlier modules already committed; a module with an extra + // column had it silently default-filled. Both were invisible to the + // caller. + let dir = std::env::temp_dir(); + let pid = std::process::id(); + let a = dir.join(format!("codescope_schema_a_{}.db", pid)); + let b = dir.join(format!("codescope_schema_b_{}.db", pid)); + let (ap, bp) = (a.to_str().unwrap(), b.to_str().unwrap()); + + make_db( + ap, + "CREATE TABLE entity(id INTEGER PRIMARY KEY, name TEXT, lang TEXT);", + ); + let cols = fetch_all_columns_excluding_rowid(ap, &["entity"]).unwrap(); + assert_eq!(cols["entity"], "id, name, lang"); + + // Missing column in module 1. + make_db( + bp, + "CREATE TABLE entity(id INTEGER PRIMARY KEY, name TEXT);", + ); + let err = + check_module_schema_consistency(&[ap.to_string(), bp.to_string()], &cols).unwrap_err(); + assert!(err.contains("entity"), "{}", err); + assert!(err.contains("lang"), "{}", err); + + // Extra column in module 1. + make_db( + bp, + "CREATE TABLE entity(id INTEGER PRIMARY KEY, name TEXT, lang TEXT, extra TEXT);", + ); + let err = + check_module_schema_consistency(&[ap.to_string(), bp.to_string()], &cols).unwrap_err(); + assert!(err.contains("extra"), "{}", err); + + // Table missing from module 1. + make_db(bp, "CREATE TABLE other(id INTEGER PRIMARY KEY);"); + let err = + check_module_schema_consistency(&[ap.to_string(), bp.to_string()], &cols).unwrap_err(); + assert!(err.contains("has no table"), "{}", err); + + // Identical schemas merge without complaint. + make_db( + bp, + "CREATE TABLE entity(id INTEGER PRIMARY KEY, name TEXT, lang TEXT);", + ); + assert!(check_module_schema_consistency(&[ap.to_string(), bp.to_string()], &cols).is_ok()); + // A single module DB needs no comparison. + assert!(check_module_schema_consistency(&[ap.to_string()], &cols).is_ok()); + + let _ = std::fs::remove_file(ap); + let _ = std::fs::remove_file(bp); + } +} diff --git a/server/src/scheduler/mod.rs b/server/src/scheduler/mod.rs index 112b351..89df779 100644 --- a/server/src/scheduler/mod.rs +++ b/server/src/scheduler/mod.rs @@ -1,6 +1,6 @@ -//! Built-in parallel scheduler — replaces `codescope-parallel.sh`. +//! Built-in parallel scheduler — replaces the legacy `scripts/legacy/codescope-parallel.sh`. //! -//! Design principle (see `builtin-scheduler-design.md` §4.1): +//! Design principle: //! > The scheduler only manages CPU core allocation; it does NOT //! > participate in file discovery, parsing, or graph building. //! @@ -39,28 +39,52 @@ //! `FilterPolicy` for the authoritative skip rules. // dyn_config is wired into the dispatch path via DynSchedConfig. -// See index_parallel() -> DynSchedConfig::should_enable() and -// index_parallel_dynamic() -> DynSchedConfig::from_env() usage. +// See index_parallel() -> DynSchedConfig::should_enable(). mod chunk_plan; pub mod chunk_queue; +mod chunked; mod dyn_config; +mod mapped_file; mod merge; mod quarantine; mod shm; mod worker; -use dyn_config::{DynSchedConfig, sample_total_rss_mb}; use serde_json::{Value, json}; -use shm::SchedShm; -use std::collections::VecDeque; use std::path::{Path, PathBuf}; + +/// Join `name` onto the platform temp directory as a `String`. +/// +/// The scheduler hands these paths to subprocesses (as CLI arguments and env +/// vars) and to SQLite, both of which want a plain `str`. Using `temp_dir()` +/// rather than a literal `/tmp` matters on Windows, where `/tmp` does not +/// exist: the whole parallel path would fail at the first write. +fn temp_path(name: &str) -> String { + std::env::temp_dir() + .join(name) + .to_string_lossy() + .into_owned() +} + +/// Default DB-path prefix for an auto-named run. Per-module DBs are named +/// `_.db`, so the prefix is a path, not a bare name. +pub(crate) fn default_db_prefix(tag: &str, run_id: u128) -> String { + temp_path(&format!("codescope_{tag}_{run_id}")) +} + +/// Default shared-file path for a scheduler-created segment: temp directory +/// plus the PID, so concurrent runs never collide and a crashed run leaves at +/// most one stale file behind. +pub(crate) fn default_shm_path(tag: &str) -> String { + temp_path(&format!("codescope_{tag}_{}.shm", std::process::id())) +} use std::sync::atomic::{AtomicU32, Ordering}; use std::sync::{Arc, mpsc}; use std::time::{Duration, Instant}; use crate::discover; use merge::MergeResult; -use worker::run_module_worker; +use worker::{join_exclude_patterns, run_module_worker}; /// Default total worker cores when `--workers` is not specified. const DEFAULT_TOTAL_WORKERS: u32 = 8; @@ -103,6 +127,43 @@ pub(super) struct ModuleResult { pub error: Option, } +/// Whether a scheduler run counts as complete. +/// +/// `ok` used to mean "at least one worker produced nodes" (`success > 0`), so a +/// run in which a module crashed — or in which the module DBs failed to merge +/// into main.db — still reported `ok: true` while the caller's view of the +/// project was partial or empty. A run is complete only when no worker failed +/// AND the merge succeeded; `success` / `fail` / `merge` keep reporting the +/// partial picture so a caller that wants it can still act on it. +pub(super) fn run_complete(failed_workers: usize, merged: bool) -> bool { + failed_workers == 0 && merged +} + +/// Whether one module worker did its job. +/// +/// A worker that exited 0 without an error succeeded even when its files +/// yielded no symbols: a `benches/main.rs` that only declares a macro, or a +/// `lib.rs` that only re-exports, is indexed perfectly with zero nodes. +/// Counting that as a FAILURE made a complete run report `ok:false` — observed +/// on a real Rust project whose three modules all exited 0 with 221 files and +/// 6740 nodes merged, where `ok` was false because the one-file `benches` +/// module produced no nodes. Whether a module produced symbols is a property +/// of its files, not of the worker. +pub(super) fn worker_succeeded(r: &ModuleResult) -> bool { + r.exit_code == 0 && r.error.is_none() +} + +/// Whether the run indexed anything at all. +/// +/// This is the "an empty index is not a success" guard +/// (`test_empty_project_reports_incomplete`), and it belongs at the RUN level: +/// a project whose files all yield no symbols, or that had no files to index, +/// must not report success. An empty run keeps its own verdict from +/// [`run_complete`], which decides whether the merge produced a main.db. +pub(super) fn run_produced_index(total_nodes: u64, total_files_indexed: u64) -> bool { + total_nodes > 0 || total_files_indexed == 0 +} + /// Entry point: discover modules, dispatch workers, quarantine failures, /// return aggregated JSON summary. /// @@ -113,11 +174,16 @@ pub(super) struct ModuleResult { /// /// Output JSON schema: /// ```json -/// {"ok":true,"project_path":"...","total_workers":N,"parallel":N, -/// "duration_ms":N,"success":N,"fail":N,"total_nodes":N,"total_edges":N, -/// "total_files_indexed":N,"modules":[{...per-module...}]} +/// {"ok":true,"complete":true,"project_path":"...","total_workers":N, +/// "parallel":N,"duration_ms":N,"success":N,"fail":N,"total_nodes":N, +/// "total_edges":N,"total_files_indexed":N,"merge":{...}, +/// "modules":[{...per-module...}]} /// ``` /// +/// `ok` and `complete` are the same value: the run finished with no failed +/// worker and a successful merge. `success` / `fail` / `merge` report the +/// partial picture for callers that want it. +/// pub fn index_parallel(project_dir: &str, total_workers: u32, parallel: u32) -> String { let start = Instant::now(); @@ -151,8 +217,8 @@ pub fn index_parallel(project_dir: &str, total_workers: u32, parallel: u32) -> S let grammars_dir = std::env::var("GRAMMARS_DIR").unwrap_or_else(|_| "engine/grammars".to_string()); - // Generate a per-run DB prefix in /tmp so each invocation starts - // fresh and parallel runs never collide on DB files. + // Generate a per-run DB prefix in the temp directory so each invocation + // starts fresh and parallel runs never collide on DB files. let run_id = std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) .map(|d| d.as_nanos()) @@ -163,7 +229,7 @@ pub fn index_parallel(project_dir: &str, total_workers: u32, parallel: u32) -> S // Otherwise (auto run_id prefix) always start clean. let prefix_env = std::env::var("CODESCOPE_DB_PREFIX"); let keep_db = prefix_env.is_ok(); - let db_prefix = prefix_env.unwrap_or_else(|_| format!("/tmp/codescope_parallel_{}", run_id)); + let db_prefix = prefix_env.unwrap_or_else(|_| default_db_prefix("parallel", run_id)); eprintln!( "scheduler: project={} workers={} parallel={} db_prefix={} keep_db={}", @@ -192,8 +258,14 @@ pub fn index_parallel(project_dir: &str, total_workers: u32, parallel: u32) -> S let modules = match discover_val["modules"].as_array() { Some(a) if !a.is_empty() => a, _ => { + // Nothing was discovered to index. `ok: true` here made an empty + // index indistinguishable from a successful one and hid the + // discovery defect that produced it (#22: a flat project's own + // files were never counted as a module). A run that indexed no + // files did not complete, and now says so. return json!({ - "ok": true, + "ok": false, + "complete": false, "project_path": project_path, "duration_ms": start.elapsed().as_millis() as u64, "success": 0, @@ -239,7 +311,7 @@ pub fn index_parallel(project_dir: &str, total_workers: u32, parallel: u32) -> S // stable, current implementation of CPU-dynamic scheduling. It // replaces the old shm-core-pool `index_parallel_dynamic`, which // is retained only as dead-code reference (DS-1 deadlock risk). - return index_parallel_chunked(project_dir, total_workers, parallel); + return chunked::index_parallel_chunked(project_dir, total_workers, parallel); } // ── Phase 2: proportional worker allocation ──────────────── @@ -321,18 +393,44 @@ pub fn index_parallel(project_dir: &str, total_workers: u32, parallel: u32) -> S let alloc = *alloc; let handle = std::thread::spawn(move || { - let result = run_module_worker( - &exe_str, - &project_path, - &name, - files, - alloc, - &grammars_dir, - &db_prefix, - project_id, - None, // no quarantine initially - keep_db, - ); + // catch_unwind so a panicking worker cannot leak its slot in + // `active` (the dispatch loop's `while active >= parallel` would + // then wait forever) and cannot vanish from `results` (which would + // undercount `fail` and could report complete:true). Mirrors the + // dynamic path. + let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + run_module_worker( + &exe_str, + &project_path, + &name, + files, + alloc, + &grammars_dir, + &db_prefix, + project_id, + None, // no quarantine initially + keep_db, + ) + })); + let result = match result { + Ok(r) => r, + Err(_) => ModuleResult { + name: name.clone(), + exit_code: -5, + total_nodes: 0, + total_edges: 0, + files_indexed: 0, + candidate_files: files, + time_parse_ms: 0, + duration_secs: 0, + workers: alloc, + db_path: String::new(), + project_id, + error: Some( + "worker panicked [module=scheduler, method=index_parallel]".to_string(), + ), + }, + }; let _ = tx.send(result); active_clone.fetch_sub(1, Ordering::SeqCst); }); @@ -361,7 +459,9 @@ pub fn index_parallel(project_dir: &str, total_workers: u32, parallel: u32) -> S // data for that slot). let mut final_results: Vec = Vec::new(); for r in results { - if r.exit_code == 0 && (r.total_nodes > 0 || r.files_indexed == 0) { + // r.error is set for non-zero exit, missing JSON, AND engine + // ok:false — an engine failure must not pass as an empty module. + if r.exit_code == 0 && r.error.is_none() && (r.total_nodes > 0 || r.files_indexed == 0) { final_results.push(r); continue; } @@ -388,7 +488,7 @@ pub fn index_parallel(project_dir: &str, total_workers: u32, parallel: u32) -> S // it from the completion-order `idx` would collide with another // module's id (Phase 3 collects results in completion order, not // allocation order). - let excluded_env = quarantined.join(","); + let excluded_env = join_exclude_patterns(&quarantined); let retry = run_module_worker( &exe_str, &project_path, @@ -405,10 +505,7 @@ pub fn index_parallel(project_dir: &str, total_workers: u32, parallel: u32) -> S } // ── Phase 5: aggregate summary ──────────────────────────── - let success = final_results - .iter() - .filter(|r| r.exit_code == 0 && (r.total_nodes > 0 || r.files_indexed == 0)) - .count(); + let success = final_results.iter().filter(|r| worker_succeeded(r)).count(); let fail = final_results.len() - success; let total_nodes: u64 = final_results.iter().map(|r| r.total_nodes).sum(); let total_edges: u64 = final_results.iter().map(|r| r.total_edges).sum(); @@ -431,498 +528,9 @@ pub fn index_parallel(project_dir: &str, total_workers: u32, parallel: u32) -> S let module_db_paths: Vec = final_results .iter() - .filter(|r| r.exit_code == 0 && (r.total_nodes > 0 || r.files_indexed == 0)) - .map(|r| r.db_path.clone()) - .collect(); - - let merge_result = if module_db_paths.is_empty() { - MergeResult { - merged: false, - main_db_path: main_db.clone(), - tables_merged: 0, - rows_merged: 0, - duration_ms: 0, - error: Some("no successful module DBs to merge".to_string()), - } - } else { - merge::merge_module_dbs(&main_db, &module_db_paths) - }; - - // v0.2.5 (C2 fix): parallel workers deferred CSR construction because - // their packed BLOBs hold local entity ids that merge cannot remap. - // Rebuild each project's CSR from the globally-remapped relation table. - if merge_result.merged { - rebuild_csr_all_projects(&main_db, "index_parallel"); - } - - let modules_json: Vec = final_results - .iter() - .map(|r| { - json!({ - "name": r.name, - "exit_code": r.exit_code, - "files_indexed": r.files_indexed, - "candidate_files": r.candidate_files, - "total_nodes": r.total_nodes, - "total_edges": r.total_edges, - "time_parse_ms": r.time_parse_ms, - "duration_secs": r.duration_secs, - "workers": r.workers, - "db_path": r.db_path, - "error": r.error, - }) - }) - .collect(); - - json!({ - "ok": success > 0, - "project_path": project_path, - "db_prefix": db_prefix, - "main_db": merge_result.main_db_path, - "merge": { - "merged": merge_result.merged, - "tables_merged": merge_result.tables_merged, - "rows_merged": merge_result.rows_merged, - "duration_ms": merge_result.duration_ms, - "error": merge_result.error, - }, - "total_workers": total_workers, - "parallel": parallel, - "duration_ms": start.elapsed().as_millis() as u64, - "success": success, - "fail": fail, - "total_nodes": total_nodes, - "total_edges": total_edges, - "total_files_indexed": total_files_indexed, - "modules": modules_json - }) - .to_string() -} - -/// Dynamic-mode parallel indexer with worker queue + core reclaim. -/// -/// Phases: -/// 1. `discover_modules` — same as static (file walk, no parsing). -/// 2. Create `SchedShm` — shared memory for cross-process core -/// accounting. Falls back to static mode on creation failure. -/// 3. Sort modules by file count desc — largest first so big modules -/// start early and benefit from cores freed by small modules. -/// 4. Dispatch workers from a queue, claiming cores from the shm pool -/// before each spawn and releasing them on completion. This is the -/// core reclaim: small modules finish and free their cores for the -/// next pending module while large modules retain their proportional -/// allocation until done. -/// 5. Aggregate summary (same as static). -/// 6. Merge per-module DBs (same as static). -/// -/// Memory monitoring: child PIDs are sampled each poll tick via -/// `pgrep -P $$` + `ps -o rss=`; the total RSS is pushed to -/// `SchedShm::update_mem_usage`. If RSS exceeds `mem_limit_mb`, new -/// worker spawns are paused (existing workers keep running — we don't -/// kill them). -// Kept for reference; the chunk-level scheduler (index_parallel_chunked) -// replaces this path. Remove after migration is complete. -#[allow(dead_code)] -fn index_parallel_dynamic(project_dir: &str, total_workers: u32, parallel: u32) -> String { - let start = Instant::now(); - let total_workers = if total_workers == 0 { - DEFAULT_TOTAL_WORKERS - } else { - total_workers - }; - let parallel = if parallel == 0 { - DEFAULT_PARALLEL - } else { - parallel - }; - // Cap parallel at total_workers to avoid spawning more OS threads - // than cores available. The shm pool caps cores, not thread count, - // so without this cap a user could pass --parallel 100 and spawn - // 100 idle threads all waiting for 0-core claims. - let parallel = parallel.min(total_workers).max(1); - - let project_path_buf = match canonicalize_project_dir(project_dir) { - Ok(p) => p, - Err(e) => return error_json(&e, "scheduler", "index_parallel_dynamic"), - }; - let project_path = project_path_buf.to_string_lossy().to_string(); - - let exe_path = match resolve_self_exe() { - Ok(p) => p, - Err(e) => return error_json(&e, "scheduler", "index_parallel_dynamic"), - }; - let exe_str = exe_path.to_string_lossy().to_string(); - - let grammars_dir = - std::env::var("GRAMMARS_DIR").unwrap_or_else(|_| "engine/grammars".to_string()); - - let run_id = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map(|d| d.as_nanos()) - .unwrap_or(0); - let prefix_env = std::env::var("CODESCOPE_DB_PREFIX"); - let keep_db = prefix_env.is_ok(); - let db_prefix = prefix_env.unwrap_or_else(|_| format!("/tmp/codescope_parallel_{}", run_id)); - let _ = keep_db; // keep_db consumed in run_module_worker calls below - - eprintln!( - "scheduler: [dynamic] project={} workers={} parallel={} db_prefix={}", - project_path, total_workers, parallel, db_prefix - ); - - // ── Phase 1: discover modules ────────────────────────────── - let discover_json = discover::discover_modules(&project_path); - let discover_val: Value = match serde_json::from_str(&discover_json) { - Ok(v) => v, - Err(e) => { - return error_json( - &format!( - "discover parse failed: {} [module=scheduler, method=index_parallel_dynamic]", - e - ), - "scheduler", - "index_parallel_dynamic", - ); - } - }; - if discover_val["ok"] != true { - return discover_json; - } - - let modules = match discover_val["modules"].as_array() { - Some(a) if !a.is_empty() => a, - _ => { - return json!({ - "ok": true, - "project_path": project_path, - "duration_ms": start.elapsed().as_millis() as u64, - "success": 0, - "fail": 0, - "total_nodes": 0, - "total_edges": 0, - "total_files_indexed": 0, - "modules": [], - "note": "no source modules found" - }) - .to_string(); - } - }; - - let total_files_sum: u64 = modules - .iter() - .filter_map(|m| m["files"].as_u64()) - .sum::() - .max(1); - - // ── Phase 2: create SchedShm ─────────────────────────────── - // total_cores caps at total_workers so the dynamic scheduler - // never claims more than the user asked for. available_parallelism - // gives us the host CPU count, which we then min with total_workers. - let dyn_config = DynSchedConfig::from_env(); - let host_cores: u32 = std::thread::available_parallelism() - .map(|n| n.get() as u32) - .unwrap_or(total_workers.max(1)); - let total_cores = host_cores.min(total_workers.max(1)); - let mem_limit = dyn_config.mem_limit_mb; - let shm_path = dyn_config - .shm_path - .clone() - .unwrap_or_else(DynSchedConfig::default_shm_path); - - eprintln!( - "scheduler: [dynamic] shm_path={} total_cores={} mem_limit_mb={}", - shm_path, total_cores, mem_limit - ); - - let shm = match SchedShm::create(&shm_path, total_cores, mem_limit) { - Ok(s) => { - // Set the aggressive poll flag from config (1 = 50ms, 0 = 100ms). - s.set_aggressive(dyn_config.aggressive); - s - } - Err(e) => { - // Graceful fallback: disable dynamic sched and recurse into - // the static path. The env override is process-local and - // only affects this scheduler run. - eprintln!( - "scheduler: [dynamic] SchedShm create failed ({}), falling back to static mode [module=scheduler, method=index_parallel_dynamic]", - e - ); - // SAFETY: Rust 2024 marks set_var unsafe because it can race - // with concurrent getenv readers. This code path runs in the - // single-threaded scheduler init before any worker threads - // spawn, so there are no concurrent readers. - unsafe { - std::env::set_var("CODESCOPE_DYNAMIC_SCHED", "0"); - } - return index_parallel(project_dir, total_workers, parallel); - } - }; - - // RAII guard: remove the shm filesystem path when the scheduler - // exits, even on panic. Constructed IMMEDIATELY after create so - // nothing between create and guard can leak the shm file. - // SchedShm::drop also unlinks (via shm_unlink), so the second - // unlink is a no-op (ENOENT) — harmless redundancy. - struct ShmGuard { - path: String, - } - impl Drop for ShmGuard { - fn drop(&mut self) { - let _ = std::fs::remove_file(&self.path); - } - } - let _shm_guard = ShmGuard { - path: shm_path.clone(), - }; - - // Propagate shm path via env so worker subprocesses can attach. - // Command inherits env vars by default, so setting this once on - // the scheduler process is enough. - // SAFETY: Rust 2024 marks set_var unsafe because it can race with - // concurrent getenv readers. This runs in the single-threaded - // scheduler init before any worker threads spawn, so no races. - unsafe { - std::env::set_var("CODESCOPE_SCHED_SHM", &shm_path); - } - - // ── Phase 3: sort modules by file count desc ───────────── - // Largest first so big modules start early — they benefit most - // from core reclaim since small modules finish and free cores - // while the big module is still running. - let mut queue: VecDeque<(String, u64, u64)> = modules - .iter() - .enumerate() - .filter_map(|(i, m)| { - let name = m["name"].as_str()?.to_string(); - let files = m["files"].as_u64().unwrap_or(0); - // project_id is 1-indexed (see index_parallel for rationale). - Some((name, files, (i as u64) + 1)) + .filter(|r| { + r.exit_code == 0 && r.error.is_none() && (r.total_nodes > 0 || r.files_indexed == 0) }) - .collect(); - // VecDeque::from(Vec) preserves order; then sort in place. - // We convert to Vec, sort, then convert back since VecDeque - // has no built-in sort_by_key. - { - let mut v: Vec<_> = queue.drain(..).collect(); - v.sort_by_key(|b| std::cmp::Reverse(b.1)); - queue.extend(v); - } - - // ── Phase 4: dynamic worker dispatch with core reclaim ──── - let (tx, rx) = mpsc::channel::(); - let active = Arc::new(AtomicU32::new(0)); - let shm = Arc::new(shm); - let mut handles = Vec::new(); - let mut results: Vec = Vec::new(); - - loop { - // Drain completed workers (non-blocking). The worker thread - // already released its claimed cores via shm.release_cores - // before sending the result, so we just collect results here. - while let Ok(r) = rx.try_recv() { - results.push(r); - } - - // Sample total RSS of child processes. Pauses new spawns if - // over mem_limit (existing workers keep running — we don't - // kill them, just don't start new ones). - let child_pids = get_child_pids(); - let total_rss_mb = sample_total_rss_mb(&child_pids); - shm.update_mem_usage(total_rss_mb); - let mem_ok = total_rss_mb <= mem_limit; - if !mem_ok { - eprintln!( - "scheduler: [dynamic] memory limit exceeded ({} > {}), pausing new spawns [module=scheduler, method=index_parallel_dynamic]", - total_rss_mb, mem_limit - ); - } - - // Try to start new workers while we have capacity, pending - // tasks, available cores, and memory headroom. - while active.load(Ordering::SeqCst) < parallel && !queue.is_empty() && mem_ok { - if !shm.has_available() { - // No cores to claim right now — wait for an active - // worker to release some. Don't pop the task yet. - break; - } - - let (name, files, project_id) = - queue.pop_front().expect("queue non-empty in dispatch loop"); - - // Desired core count: proportional to file count, capped - // at total_workers. Matches the static allocation formula. - let want = if files == 0 { - 1 - } else { - let raw = (files as u128 * total_workers as u128).div_ceil(total_files_sum as u128); - let a = u32::try_from(raw as u64).unwrap_or(total_workers); - std::cmp::max(1, std::cmp::min(a, total_workers)) - }; - - // Claim cores atomically. If 0 returned, another worker - // beat us to it — put the task back at the front and wait. - let claimed = shm.claim_cores(want); - if claimed == 0 { - queue.push_front((name, files, project_id)); - break; - } - - active.fetch_add(1, Ordering::SeqCst); - - let active_clone = Arc::clone(&active); - let tx = tx.clone(); - let exe_str = exe_str.clone(); - let project_path = project_path.clone(); - let grammars_dir = grammars_dir.clone(); - let db_prefix = db_prefix.clone(); - let shm_clone = Arc::clone(&shm); - - let handle = std::thread::spawn(move || { - let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { - run_module_worker( - &exe_str, - &project_path, - &name, - files, - claimed, - &grammars_dir, - &db_prefix, - project_id, - None, - keep_db, - ) - })); - // Release the cores we claimed so the next pending - // worker can grab them. This is the core reclaim: a - // small module finishing frees its cores for the next - // big module still in the queue. - // catch_unwind guarantees this runs even if the worker - // panicked — without it, core leak + active underflow - // would cause a permanent hang (H-C). - shm_clone.release_cores(claimed); - let result = match result { - Ok(r) => r, - Err(_) => ModuleResult { - name: name.clone(), - exit_code: -5, - total_nodes: 0, - total_edges: 0, - files_indexed: 0, - candidate_files: files, - time_parse_ms: 0, - duration_secs: 0, - workers: claimed, - db_path: String::new(), - project_id, - error: Some( - "worker panicked [module=scheduler, method=index_parallel_dynamic]" - .to_string(), - ), - }, - }; - let _ = tx.send(result); - active_clone.fetch_sub(1, Ordering::SeqCst); - }); - handles.push(handle); - } - - // Exit when queue is drained AND no active workers remain. - if active.load(Ordering::SeqCst) == 0 && queue.is_empty() { - break; - } - - // Sleep longer when memory-paused: existing workers need time to - // finish and release RSS. 50ms polling burns CPU on pgrep+ps for - // no benefit — 1s is a reasonable backoff. - if !mem_ok { - std::thread::sleep(Duration::from_secs(1)); - } else { - std::thread::sleep(POLL_INTERVAL); - } - } - - drop(tx); - - // Drain any results that arrived between the last try_recv and - // the channel closure. - while let Ok(r) = rx.try_recv() { - results.push(r); - } - - for h in handles { - let _ = h.join(); - } - - // ── Phase 4b: quarantine failed modules ─────────────────── - // Retry failed modules with crashing files excluded. Tries to claim - // cores from the shm pool so the retry benefits from available cores - // (e.g., after small modules finished and freed their cores). - // Falls back to 1 worker if the pool is empty or shm is unavailable. - let mut final_results: Vec = Vec::new(); - for r in results { - if r.exit_code == 0 && (r.total_nodes > 0 || r.files_indexed == 0) { - final_results.push(r); - continue; - } - eprintln!( - "scheduler: [dynamic] module {} failed (exit={}, nodes={}) — starting quarantine", - r.name, r.exit_code, r.total_nodes - ); - let quarantined = quarantine::quarantine_module( - &exe_str, - &project_path, - &r.name, - &grammars_dir, - &db_prefix, - ); - if quarantined.is_empty() { - final_results.push(r); - continue; - } - let excluded_env = quarantined.join(","); - // Try to claim up to 4 cores, or fall back to 1. - let retry_workers = shm.claim_cores(4); - let retry_workers = if retry_workers == 0 { 1 } else { retry_workers }; - let retry = run_module_worker( - &exe_str, - &project_path, - &r.name, - r.files_indexed, - retry_workers, - &grammars_dir, - &db_prefix, - r.project_id, - Some(&excluded_env), - keep_db, - ); - // Release the claimed cores back to the pool. - shm.release_cores(retry_workers); - final_results.push(retry); - } - - // ── Phase 5: aggregate summary ──────────────────────────── - let success = final_results - .iter() - .filter(|r| r.exit_code == 0 && (r.total_nodes > 0 || r.files_indexed == 0)) - .count(); - let fail = final_results.len() - success; - let total_nodes: u64 = final_results.iter().map(|r| r.total_nodes).sum(); - let total_edges: u64 = final_results.iter().map(|r| r.total_edges).sum(); - let total_files_indexed: u64 = final_results.iter().map(|r| r.files_indexed).sum(); - - // ── Phase 6: merge per-module DBs into unified main DB ──── - let main_db = format!("{}_main.db", db_prefix); - // main.db is ALWAYS rebuilt from the module DBs below — keep_db only - // preserves the per-module DBs so workers can skip unchanged files. - // Keeping main.db too would double-count rows on INSERT OR IGNORE. - let _ = std::fs::remove_file(&main_db); - let _ = std::fs::remove_file(format!("{}-wal", main_db)); - let _ = std::fs::remove_file(format!("{}-shm", main_db)); - - let module_db_paths: Vec = final_results - .iter() - .filter(|r| r.exit_code == 0 && (r.total_nodes > 0 || r.files_indexed == 0)) .map(|r| r.db_path.clone()) .collect(); @@ -936,7 +544,7 @@ fn index_parallel_dynamic(project_dir: &str, total_workers: u32, parallel: u32) error: Some("no successful module DBs to merge".to_string()), } } else { - merge::merge_module_dbs(&main_db, &module_db_paths) + merge::merge_module_dbs(&main_db, &module_db_paths, &project_path) }; // v0.2.5 (C2 fix): parallel workers deferred CSR construction because @@ -965,8 +573,16 @@ fn index_parallel_dynamic(project_dir: &str, total_workers: u32, parallel: u32) }) .collect(); + // "ok" means the run completed AND produced a consistent index — not just + // "at least one worker finished". See run_complete() and + // run_produced_index(): the latter is the run-level "nothing indexed at + // all" guard, deliberately not a per-module condition (see + // worker_succeeded for why). + let complete = run_complete(fail, merge_result.merged) + && run_produced_index(total_nodes, total_files_indexed); json!({ - "ok": success > 0, + "ok": complete, + "complete": complete, "project_path": project_path, "db_prefix": db_prefix, "main_db": merge_result.main_db_path, @@ -979,7 +595,6 @@ fn index_parallel_dynamic(project_dir: &str, total_workers: u32, parallel: u32) }, "total_workers": total_workers, "parallel": parallel, - "sched_mode": "dynamic", "duration_ms": start.elapsed().as_millis() as u64, "success": success, "fail": fail, @@ -991,26 +606,6 @@ fn index_parallel_dynamic(project_dir: &str, total_workers: u32, parallel: u32) .to_string() } -/// Get child PIDs of the current process via `pgrep -P $$`. -/// Returns an empty vec if pgrep is unavailable (e.g., unsupported -/// platform) — memory monitoring just degrades to no-op in that case. -/// Kept for reference by the old index_parallel_dynamic path. -#[allow(dead_code)] -fn get_child_pids() -> Vec { - let parent_pid = std::process::id(); - let output = std::process::Command::new("pgrep") - .args(["-P", &parent_pid.to_string()]) - .stderr(std::process::Stdio::null()) - .output(); - match output { - Ok(out) if out.status.success() => String::from_utf8_lossy(&out.stdout) - .lines() - .filter_map(|l| l.trim().parse::().ok()) - .collect(), - _ => Vec::new(), - } -} - /// Canonicalise the project directory to an absolute path. /// Returns an error string suitable for inclusion in JSON output. fn canonicalize_project_dir(dir: &str) -> Result { @@ -1117,357 +712,6 @@ fn rebuild_csr_all_projects(main_db: &str, method: &str) { ); } -/// Chunk-level parallel indexer with work-stealing (CPU-dynamic scheduling). -/// -/// OPT-IN path: entered only when `CODESCOPE_CPU_DYNAMIC` / -/// `CODESCOPE_DYNAMIC_SCHED` is explicitly set. The DEFAULT (no flag) is the -/// static proportional allocator in `index_parallel` — see the project's -/// "static by default" scheduling principle (DYNAMIC_SCHED_REDESIGN.md). -/// -/// Stability: each worker writes its OWN DB (no concurrent WAL writers), -/// uses a unique 1-based project_id, and the final merge reuses the proven -/// per-unit-DB + merge_module_dbs machinery from the static path. -fn index_parallel_chunked(project_dir: &str, total_workers: u32, parallel: u32) -> String { - let start = Instant::now(); - let total_workers = if total_workers == 0 { - DEFAULT_TOTAL_WORKERS - } else { - total_workers - }; - let parallel = if parallel == 0 { - DEFAULT_PARALLEL - } else { - parallel - }; - // Cap parallel at total_workers (see Bug 9 fix). - let parallel = parallel.min(total_workers).max(1); - - let project_path_buf = match canonicalize_project_dir(project_dir) { - Ok(p) => p, - Err(e) => return error_json(&e, "scheduler", "index_parallel_chunked"), - }; - let project_path = project_path_buf.to_string_lossy().to_string(); - - let exe_path = match resolve_self_exe() { - Ok(p) => p, - Err(e) => return error_json(&e, "scheduler", "index_parallel_chunked"), - }; - let exe_str = exe_path.to_string_lossy().to_string(); - - let grammars_dir = - std::env::var("GRAMMARS_DIR").unwrap_or_else(|_| "engine/grammars".to_string()); - - let run_id = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map(|d| d.as_nanos()) - .unwrap_or(0); - let prefix_env = std::env::var("CODESCOPE_DB_PREFIX"); - let keep_db = prefix_env.is_ok(); - let db_prefix = prefix_env.unwrap_or_else(|_| format!("/tmp/codescope_chunked_{}", run_id)); - let _ = keep_db; // keep_db consumed in run_module_worker calls below - - // ── Phase 1: discover the GLOBAL file list ─────────────── - // One walk of the whole project yields every candidate source file. - // Unlike the static path (which plans per-module), the chunked path - // plans over the flat global list so chunks can span directory - // boundaries and balance by byte weight. - let discover_json = discover::discover_files(&project_path); - let discover_val: Value = match serde_json::from_str(&discover_json) { - Ok(v) => v, - Err(e) => { - return error_json( - &format!( - "discover_files parse failed: {} [module=scheduler, method=index_parallel_chunked]", - e - ), - "scheduler", - "index_parallel_chunked", - ); - } - }; - if discover_val["ok"] != true { - return discover_json; - } - let all_paths: Vec = discover_val["files"] - .as_array() - .map(|a| { - a.iter() - .filter_map(|v| v.as_str().map(|s| s.to_string())) - .collect() - }) - .unwrap_or_default(); - if all_paths.is_empty() { - return json!({ - "ok": true, - "project_path": project_path, - "duration_ms": start.elapsed().as_millis() as u64, - "success": 0, - "fail": 0, - "total_nodes": 0, - "total_edges": 0, - "total_files_indexed": 0, - "modules": [], - "note": "no source files found" - }) - .to_string(); - } - - // Build FileEntry list (path + size), sorted by path — plan_chunks - // REQUIRES the input sorted by path (see chunk_plan.rs Invariant). - let mut files: Vec = all_paths - .iter() - .map(|p| { - let size = std::fs::metadata(p).map(|m| m.len()).unwrap_or(0); - chunk_plan::FileEntry { - path: p.clone(), - size, - } - }) - .collect(); - files.sort_by(|a, b| a.path.cmp(&b.path)); - - // ── Phase 2: plan chunks ──────────────────────────────── - let chunks = chunk_plan::plan_chunks(&files, chunk_plan::TARGET_BYTES, chunk_plan::MAX_BYTES); - if chunks.len() > chunk_queue::MAX_CHUNKS { - eprintln!( - "scheduler: warning: plan_chunks produced {} chunks, truncating to MAX_CHUNKS={}; some files will not be indexed [module=scheduler, method=index_parallel_chunked]", - chunks.len(), - chunk_queue::MAX_CHUNKS - ); - } - let chunk_count = (chunks.len() as u32).min(chunk_queue::MAX_CHUNKS as u32); - if chunk_count == 0 { - return error_json( - "plan_chunks returned zero chunks [module=scheduler, method=index_parallel_chunked]", - "scheduler", - "index_parallel_chunked", - ); - } - - // Share the GLOBAL file list with workers via a temp JSON file. - // Workers slice it by each chunk's (file_start, file_count). - let files_json_path = format!("{}_chunk_files.json", db_prefix); - let files_json = serde_json::to_string(&all_paths).unwrap_or_default(); - if let Err(e) = std::fs::write(&files_json_path, &files_json) { - return error_json( - &format!( - "write chunk file list failed: {} [module=scheduler, method=index_parallel_chunked]", - e - ), - "scheduler", - "index_parallel_chunked", - ); - } - - // ── Phase 3: create + fill the chunk queue ────────────── - let shm_path = format!("/tmp/codescope_chunked_sched_{}.shm", std::process::id()); - let _ = std::fs::remove_file(&shm_path); - let queue = match chunk_queue::ChunkQueue::create(&shm_path, chunk_count) { - Ok(q) => q, - Err(e) => return error_json(&e, "scheduler", "index_parallel_chunked"), - }; - for (idx, ch) in chunks.iter().take(chunk_count as usize).enumerate() { - if let Err(e) = queue.write_chunk( - idx as u32, - ch.module_id, - ch.file_start as u32, - ch.file_count as u32, - ch.total_bytes, - ) { - return error_json( - &format!( - "write_chunk {} failed: {} [module=scheduler, method=index_parallel_chunked]", - idx, e - ), - "scheduler", - "index_parallel_chunked", - ); - } - } - - eprintln!( - "scheduler: [chunked] project={} files={} chunks={} workers={} parallel={}", - project_path, - all_paths.len(), - chunk_count, - total_workers, - parallel - ); - - // ── Phase 3: spawn workers, each with its OWN DB ────────── - // Every chunk-worker writes to a unique per-worker DB - // (`{db_prefix}_chunk_{worker_id}.db`) with a unique project_id - // (`worker_id + 1`), so there are NO concurrent WAL writers on a - // shared DB — the original shared_db design corrupted the index - // under concurrency. After all workers exit, Phase 4 merges the - // per-worker DBs into the unified main DB via the SAME - // merge_module_dbs machinery as the static path. - let mut handles = Vec::new(); - let mut results: Vec = Vec::new(); - let active = Arc::new(AtomicU32::new(0)); - let (tx, rx) = mpsc::channel::(); - - for worker_id in 0..total_workers { - // Wait for a free slot if at concurrency cap. - while active.load(Ordering::SeqCst) >= parallel { - std::thread::sleep(POLL_INTERVAL); - } - active.fetch_add(1, Ordering::SeqCst); - - // CPU set for this worker (static binding, Linux only; empty - // elsewhere so run_chunk_worker skips taskset wrapping). - #[cfg(target_os = "linux")] - let cpu_set = { - let host_cores = std::thread::available_parallelism() - .map(|n| n.get() as u32) - .unwrap_or(1) - .max(1); - let cores_per_worker = (host_cores / total_workers.max(1)).max(1); - let cpu_idx = (worker_id * cores_per_worker) % host_cores; - format!("{}", cpu_idx) - }; - #[cfg(not(target_os = "linux"))] - let cpu_set: String = String::new(); - - // Per-worker DB + unique project_id. Mirrors the static path's - // (idx + 1) project_id scheme so merge's id-remap and project - // disambiguation behave identically. - let worker_db = format!("{}_chunk_{}.db", db_prefix, worker_id); - let project_id: u64 = (worker_id as u64) + 1; - - let active_clone = Arc::clone(&active); - let tx = tx.clone(); - let exe_str = exe_str.clone(); - let grammars_dir = grammars_dir.clone(); - let shm_path = shm_path.clone(); - let worker_db = worker_db.clone(); - let files_json_path = files_json_path.clone(); - - let handle = std::thread::spawn(move || { - let result = worker::run_chunk_worker( - &exe_str, - &shm_path, - worker_id, - &cpu_set, - &worker_db, - &files_json_path, - project_id, - &grammars_dir, - ); - let _ = tx.send(result); - active_clone.fetch_sub(1, Ordering::SeqCst); - }); - handles.push(handle); - } - - drop(tx); - - // Collect results. - while let Ok(r) = rx.recv() { - results.push(r); - } - - // Wait for all threads to finish. - for h in handles { - let _ = h.join(); - } - - // ── Phase 4: merge per-worker DBs into the unified main DB ── - // Mirror index_parallel Phase 6: ATTACH each worker DB to a fresh - // main DB and INSERT OR IGNORE. Per-worker project_ids are unique - // (worker_id + 1) and merge_module_dbs remaps ids to avoid - // cross-worker collisions (same as the static path). - let main_db = format!("{}_main.db", db_prefix); - // main.db is ALWAYS rebuilt from the module DBs below — keep_db only - // preserves the per-module DBs so workers can skip unchanged files. - // Keeping main.db too would double-count rows on INSERT OR IGNORE. - let _ = std::fs::remove_file(&main_db); - let _ = std::fs::remove_file(format!("{}-wal", main_db)); - let _ = std::fs::remove_file(format!("{}-shm", main_db)); - - let worker_db_paths: Vec = results - .iter() - .filter(|r| r.exit_code == 0 && (r.total_nodes > 0 || r.files_indexed == 0)) - .map(|r| r.db_path.clone()) - .collect(); - - let merge_result = if worker_db_paths.is_empty() { - MergeResult { - merged: false, - main_db_path: main_db.clone(), - tables_merged: 0, - rows_merged: 0, - duration_ms: 0, - error: Some( - "no successful chunk-worker DBs to merge [module=scheduler, method=index_parallel_chunked]" - .to_string(), - ), - } - } else { - merge::merge_module_dbs(&main_db, &worker_db_paths) - }; - - // v0.2.5 (C2 fix): parallel chunk workers deferred CSR construction; - // rebuild each project's CSR from the globally-remapped relation table. - if merge_result.merged { - rebuild_csr_all_projects(&main_db, "index_parallel_chunked"); - } - - // ── Phase 5: aggregate summary ──────────────────────────── - let success = results - .iter() - .filter(|r| r.exit_code == 0 && (r.total_nodes > 0 || r.files_indexed == 0)) - .count(); - let fail = results.len() - success; - let total_nodes: u64 = results.iter().map(|r| r.total_nodes).sum(); - let total_edges: u64 = results.iter().map(|r| r.total_edges).sum(); - let total_files_indexed: u64 = results.iter().map(|r| r.files_indexed).sum(); - - let modules_json: Vec = results - .iter() - .map(|r| { - json!({ - "name": r.name, - "exit_code": r.exit_code, - "files_indexed": r.files_indexed, - "candidate_files": r.candidate_files, - "total_nodes": r.total_nodes, - "total_edges": r.total_edges, - "time_parse_ms": r.time_parse_ms, - "duration_secs": r.duration_secs, - "workers": r.workers, - "db_path": r.db_path, - "error": r.error, - }) - }) - .collect(); - - json!({ - "ok": success > 0, - "project_path": project_path, - "db_prefix": db_prefix, - "main_db": merge_result.main_db_path, - "merge": { - "merged": merge_result.merged, - "tables_merged": merge_result.tables_merged, - "rows_merged": merge_result.rows_merged, - "duration_ms": merge_result.duration_ms, - "error": merge_result.error, - }, - "total_workers": total_workers, - "parallel": parallel, - "sched_mode": "chunked", - "duration_ms": start.elapsed().as_millis() as u64, - "success": success, - "fail": fail, - "total_nodes": total_nodes, - "total_edges": total_edges, - "total_files_indexed": total_files_indexed, - "modules": modules_json - }) - .to_string() -} - #[cfg(test)] mod tests { use super::*; @@ -1482,6 +726,77 @@ mod tests { assert!(v["error"].as_str().unwrap().contains("boom")); } + #[test] + fn test_run_complete_requires_all_workers_and_merge() { + // Regression: `ok` was `success > 0`, so a crashed module — or a run + // whose module DBs never merged — still reported ok=true while the + // caller's view of the project was partial or empty. + assert!(run_complete(0, true)); + assert!(!run_complete(1, true), "a failed worker = partial index"); + assert!(!run_complete(0, false), "an unmerged run has no main.db"); + assert!(!run_complete(2, false)); + } + + #[test] + fn test_empty_project_reports_incomplete() { + // Regression (#23): a run that discovered no modules returned ok=true + // with no `complete` field, so an empty index looked like a successful + // one — and hid the discovery defect that produced it (#22). + let dir = std::env::temp_dir().join("codescope_test_empty_project"); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + let out = index_parallel(dir.to_str().unwrap(), 1, 1); + let v: serde_json::Value = serde_json::from_str(&out).unwrap(); + assert_eq!(v["ok"], false, "an empty index is not a success"); + assert_eq!(v["complete"], false); + assert_eq!(v["modules"].as_array().unwrap().len(), 0); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn test_module_without_symbols_is_a_success_not_a_failure() { + // Regression (found by indexing a real Rust project): a module that + // indexed its files but produced no symbols was counted as a FAILURE, + // so a complete run — all modules exit 0, merge ok — reported + // ok:false. Only a real failure (error or non-zero exit) may. + let mk = |exit: i32, nodes: u64, files: u64, err: Option<&str>| ModuleResult { + name: "m".into(), + exit_code: exit, + total_nodes: nodes, + total_edges: 0, + files_indexed: files, + candidate_files: files, + time_parse_ms: 0, + duration_secs: 0, + workers: 1, + db_path: String::new(), + project_id: 1, + error: err.map(|e| e.to_string()), + }; + assert!( + worker_succeeded(&mk(0, 0, 1, None)), + "exit 0 with no symbols is still a success" + ); + assert!( + !worker_succeeded(&mk(0, 500, 10, Some("boom"))), + "an error is a failure even at exit 0" + ); + assert!( + !worker_succeeded(&mk(101, 500, 10, None)), + "a non-zero exit is a failure" + ); + // The "empty index is not a success" guard stays, but at run level. + assert!( + !run_produced_index(0, 221), + "files indexed but no symbols = nothing to show for the run" + ); + assert!(run_produced_index(6740, 221), "a real index"); + assert!( + run_produced_index(0, 0), + "an empty run is judged by run_complete, not here" + ); + } + #[test] fn test_canonicalize_project_dir_rejects_missing_dir() { let r = canonicalize_project_dir("/nonexistent/does/not/exist/xyz"); diff --git a/server/src/scheduler/quarantine.rs b/server/src/scheduler/quarantine.rs index 7cab667..683ec0e 100644 --- a/server/src/scheduler/quarantine.rs +++ b/server/src/scheduler/quarantine.rs @@ -5,15 +5,15 @@ //! Once a crasher is found, the module is retried with that file //! excluded via `CODESCOPE_EXCLUDE_PATHS`. //! -//! Algorithm (mirrors `codescope-parallel.sh:find_crashing_file`): +//! Algorithm (mirrors the legacy `scripts/legacy/codescope-parallel.sh:find_crashing_file`): //! 1. Get candidate file list via `discover::discover_files()`. //! 2. Binary search: split list in half, run worker with `--file-list` //! on the left half. If it crashes, recurse left; else advance right. //! 3. When `left == right`, that file is the crasher. //! 4. Repeat up to `QUARANTINE_MAX_ITER` times to find multiple crashers. -use serde_json::Value; -use std::path::Path; +use serde_json::{Value, json}; +use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; use std::time::{Duration, Instant}; @@ -94,8 +94,22 @@ pub(super) fn quarantine_module( grammars_dir: &str, db_prefix: &str, ) -> Vec { - let module_dir = Path::new(project_dir).join(module_name); - let discover_json = discover::discover_files(module_dir.to_str().unwrap_or("")); + // The root module (see discover::ROOT_MODULE_NAME) is rooted at the project + // directory and owns only the files directly in it. Walking the whole tree + // here would hand quarantine the files the per-module workers own, and + // would make the exclusion globs relative to the wrong root. + let is_root_module = module_name == discover::ROOT_MODULE_NAME; + let module_dir = if is_root_module { + PathBuf::from(project_dir) + } else { + Path::new(project_dir).join(module_name) + }; + let discover_json = if is_root_module { + let files = discover::root_source_files(module_dir.to_str().unwrap_or("")); + json!({"ok": true, "total": files.len(), "files": files}).to_string() + } else { + discover::discover_files(module_dir.to_str().unwrap_or("")) + }; let discover_val: Value = match serde_json::from_str(&discover_json) { Ok(v) => v, Err(_) => return Vec::new(), @@ -125,17 +139,18 @@ pub(super) fn quarantine_module( // Filter out already-found crashers before each iteration so // the binary search can find additional crashers. + // + // Compare the module-relative PATH, not the basename: `crashers` holds + // module-relative paths (see make_relative_glob), and matching on the + // basename alone made `c/foo.cpp` count as "already quarantined" + // because `a/foo.cpp` crashed — so a healthy file was dropped from the + // retry without ever being tested. let active_files: Vec = files .iter() .filter_map(|f| { let s = f.as_str()?; - let basename = Path::new(s).file_name()?.to_string_lossy().to_string(); - if crashers.iter().any(|c| { - Path::new(c) - .file_name() - .map(|n| n == basename.as_str()) - .unwrap_or(false) - }) { + let rel = make_relative_glob(s, module_dir.to_str().unwrap_or("")); + if crashers.iter().any(|c| c == &rel) { None } else { Some(s.to_string()) @@ -306,7 +321,7 @@ fn run_subset_once( ]); cmd.env("GRAMMARS_DIR", grammars_dir); cmd.env("CODESCOPE_DB_PATH", db_path); - cmd.env("CODESCOPE_INDEX_MODE", "fast"); + super::worker::apply_worker_index_mode(&mut cmd); cmd.env("CODESCOPE_WORKERS", "1"); cmd.stdout(Stdio::null()).stderr(Stdio::null()); diff --git a/server/src/scheduler/shm.rs b/server/src/scheduler/shm.rs index b88dca4..4c7e1c4 100644 --- a/server/src/scheduler/shm.rs +++ b/server/src/scheduler/shm.rs @@ -1,37 +1,33 @@ -//! Shared-memory infrastructure for dynamic CPU scheduling. +//! Shared state for dynamic CPU scheduling. //! -//! `SchedState` is a `#[repr(C)]` struct stored in a POSIX shared-memory -//! file (`/tmp/codescope_sched_.shm`) mapped `MAP_SHARED` so the -//! scheduler process and worker subprocesses observe the same atomic -//! counters. Workers claim cores from a shared pool at startup and -//! release them on completion; the scheduler polls `generation` to -//! dispatch queued workers when cores become available. +//! `SchedState` is a `#[repr(C)]` struct stored in a shared file +//! (`/codescope_sched_.shm`) mapped writable and shared by +//! [`super::mapped_file::MappedFile`], so the scheduler process and worker +//! subprocesses observe the same atomic counters. Workers claim cores from a +//! shared pool at startup and release them on completion; the scheduler polls +//! `generation` to dispatch queued workers when cores become available. //! //! All mutable fields are `AtomicU32` and use `Ordering::Relaxed` — //! we only need cross-core visibility, not strict acquire/release //! ordering (the scheduler re-reads state in a poll loop anyway). //! -//! Owner (scheduler) creates the file via [`SchedShm::create`] and is -//! responsible for unlinking it on shutdown (handled in `Drop`). Workers -//! attach via [`SchedShm::open`] and only munmap on drop. +//! Owner (scheduler) creates the file via [`SchedShm::create`]; unmapping, +//! closing and removing it are `MappedFile`'s `Drop`. Workers attach via +//! [`SchedShm::open`] and only unmap. The mapping layer is platform-neutral, +//! so this module builds and works on Windows as well as POSIX. // Worker-side helpers (open/register_worker/mark_done/set_worker_cores/ // can_grab/poll_interval_ms/path) and the status constants they reference // are kept for the engine's monitor-thread implementation (see -// engine/src/engine_index_project.cpp) — the engine reads the shm directly -// via mmap, so these Rust helpers remain unused by the scheduler itself. +// engine/src/engine_index_project.cpp) — on POSIX the engine reads the same +// file directly via mmap, so these Rust helpers remain unused by the +// scheduler itself. (`engine_index_sched.cpp` compiles that reader out on +// Windows, where the engine instead runs its own in-process scheduler.) #![allow(dead_code)] -use std::ffi::CString; -use std::os::unix::io::RawFd; use std::sync::atomic::{AtomicU32, Ordering}; -use libc::{ - MAP_SHARED, O_CREAT, O_RDWR, PROT_READ, PROT_WRITE, S_IRUSR, S_IWUSR, c_void, close, ftruncate, - mmap, munmap, open, unlink, -}; -#[cfg(windows)] -const MAP_FAILED: *mut libc::c_void = !0 as *mut libc::c_void; +use super::mapped_file::MappedFile; /// Magic number stored in [`SchedState::magic`] — ASCII "SCHD" in /// little-endian. Workers check this on attach to verify the scheduler @@ -98,114 +94,54 @@ pub struct SchedState { pub worker_cores: [AtomicU32; 64], } -/// RAII wrapper around a `mmap`'d [`SchedState`]. +/// RAII wrapper around a mapped [`SchedState`]. /// /// The owner (scheduler) creates the file via [`SchedShm::create`] and -/// unlinks it on `Drop`. Workers attach via [`SchedShm::open`] and -/// only `munmap` on drop — they do not unlink. +/// [`MappedFile`]'s `Drop` removes it. Workers attach via [`SchedShm::open`] +/// and only unmap — they do not remove the file. pub struct SchedShm { - path: String, - fd: RawFd, + map: MappedFile, ptr: *mut SchedState, - is_owner: bool, } // Manual Debug impl — needed because `*mut SchedState` is a raw pointer // and does not auto-derive Debug. Tests use `Result::unwrap_err()` -// which requires `SchedShm: Debug`. We only emit the path + fd so we don't -// leak the mmap address in logs. +// which requires `SchedShm: Debug`. We only emit the path and ownership so we +// don't leak the mapping address in logs. impl std::fmt::Debug for SchedShm { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("SchedShm") - .field("path", &self.path) - .field("fd", &self.fd) - .field("is_owner", &self.is_owner) + .field("path", &self.map.path()) + .field("is_owner", &self.map.is_owner()) .finish_non_exhaustive() } } impl SchedShm { - /// Scheduler creates a new shared-memory file and initialises [`SchedState`]. + /// Scheduler creates a new shared file and initialises [`SchedState`]. /// - /// `path` — file path, typically `/tmp/codescope_sched_.shm`. + /// `path` — file path inside the temp directory, typically + /// `/codescope_sched_.shm` (see + /// [`super::dyn_config::DynamicConfig::default_shm_path`]). /// `total_cores` — total CPU budget workers can claim from. /// `mem_limit_mb` — RSS ceiling; [`SchedShm::can_grab`] returns false /// when `current_mem_mb` exceeds this. /// - /// Returns `Err` with a tagged message if `open`, `ftruncate`, or - /// `mmap` fails. On success the file is sized to `size_of::()` - /// and all fields are initialised before returning. + /// Returns `Err` with a tagged message if the file cannot be opened, sized + /// or mapped. On success the file is exactly `size_of::()` + /// bytes, all zero, and every field is initialised before returning. pub fn create(path: &str, total_cores: u32, mem_limit_mb: u32) -> Result { - let c_path = CString::new(path).map_err(|e| { - format!( - "path contains NUL byte: {} [module=scheduler, method=SchedShm::create]", - e - ) - })?; - - // SAFETY: c_path is a valid NUL-terminated CString; O_CREAT|O_RDWR - // creates the file if absent, mode 0600 restricts read/write to owner. - // Cast mode bits to c_int — on macOS S_IRUSR|S_IWUSR are u16, but - // libc::open's variadic mode arg is c_int (see open(2)). - let fd = unsafe { - open( - c_path.as_ptr(), - O_CREAT | O_RDWR, - (S_IRUSR | S_IWUSR) as libc::c_int, - ) - }; - if fd < 0 { - return Err(format!( - "open failed: {} [module=scheduler, method=SchedShm::create, path={}]", - std::io::Error::last_os_error(), - path, - )); - } - let size = std::mem::size_of::(); - // SAFETY: fd is a valid open descriptor. ftruncate sizes the file - // to exactly `size` bytes; cast to i64 is safe because the struct - // is ~584 bytes, well within off_t range. - if unsafe { ftruncate(fd, size as i64) } != 0 { - let err = std::io::Error::last_os_error(); - // SAFETY: fd is a valid open descriptor; close it to leak no fd. - unsafe { close(fd) }; - return Err(format!( - "ftruncate failed: {} [module=scheduler, method=SchedShm::create]", - err - )); - } - - // SAFETY: fd is valid and points to a file we just sized to `size`. - // MAP_SHARED makes writes visible to other processes that map the - // same file. PROT_READ|PROT_WRITE allows atomic counter updates. - let ptr = unsafe { - mmap( - std::ptr::null_mut(), - size, - PROT_READ | PROT_WRITE, - MAP_SHARED, - fd, - 0, - ) - }; - if ptr == libc::MAP_FAILED { - let err = std::io::Error::last_os_error(); - // SAFETY: fd is a valid open descriptor; close it to leak no fd. - unsafe { close(fd) }; - return Err(format!( - "mmap failed: {} [module=scheduler, method=SchedShm::create]", - err - )); - } - - // SAFETY: ptr is a valid, properly-aligned pointer to memory - // returned by mmap and sized to size_of::(). The - // file was just created/truncated, so the bytes are zero. No - // other process has opened this file yet (caller's responsibility - // to call create() before spawning workers), so exclusive mutable - // access here is safe. - let state = unsafe { &mut *(ptr as *mut SchedState) }; + // The mapping layer sizes the file to `size` and zero-fills it, so every + // field starts as 0 and only the non-zero defaults need writing. + let map = MappedFile::create(path, size, "SchedShm::create")?; + + // SAFETY: `map` is a valid, properly-aligned, all-zero view of + // `size_of::()` bytes, and no worker can have opened the + // file yet (callers create the segment before spawning workers), so + // this exclusive mutable borrow is sound. + let ptr = map.ptr() as *mut SchedState; + let state = unsafe { &mut *ptr }; state.magic = MAGIC; state.version = VERSION; state.total_cores.store(total_cores, Ordering::Relaxed); @@ -224,12 +160,7 @@ impl SchedShm { slot.store(0, Ordering::Relaxed); } - Ok(Self { - path: path.to_string(), - fd, - ptr: ptr as *mut SchedState, - is_owner: true, - }) + Ok(Self { map, ptr }) } /// Set the aggressive polling flag (1=aggressive 50ms, 0=normal 100ms). @@ -253,60 +184,21 @@ impl SchedShm { /// segment before the worker attached. Returns `Err` with a tagged /// message if `open`, `mmap`, or the magic check fails. pub fn open(path: &str) -> Result { - let c_path = CString::new(path).map_err(|e| { - format!( - "path contains NUL byte: {} [module=scheduler, method=SchedShm::open]", - e - ) - })?; - - // SAFETY: c_path is valid NUL-terminated; O_RDWR (no O_CREAT) — - // workers don't create the file, the scheduler must have done so. - let fd = unsafe { open(c_path.as_ptr(), O_RDWR) }; - if fd < 0 { - return Err(format!( - "open failed: {} [module=scheduler, method=SchedShm::open, path={}]", - std::io::Error::last_os_error(), - path, - )); - } - let size = std::mem::size_of::(); - // SAFETY: fd is valid; the file was created and sized by - // SchedShm::create, so the mapping fits the file exactly. - let ptr = unsafe { - mmap( - std::ptr::null_mut(), - size, - PROT_READ | PROT_WRITE, - MAP_SHARED, - fd, - 0, - ) - }; - if ptr == libc::MAP_FAILED { - let err = std::io::Error::last_os_error(); - // SAFETY: fd is a valid open descriptor; close it to leak no fd. - unsafe { close(fd) }; - return Err(format!( - "mmap failed: {} [module=scheduler, method=SchedShm::open]", - err - )); - } - - // SAFETY: ptr is valid SchedState-aligned memory returned by mmap. - // We only read `magic` here (a non-atomic u32); the scheduler - // writes it once in create() before any worker opens the file, - // so the read is race-free. - let state = unsafe { &*(ptr as *const SchedState) }; - // Cache the magic into a local BEFORE any munmap — reading through - // `state` after `munmap(ptr, size)` would be use-after-unmap. + // The owner created this file with the same size, so the mapping fits + // the segment exactly. + let map = MappedFile::open(path, size, "SchedShm::open")?; + + // SAFETY: `map` is a valid, properly-aligned view of `size_of::< + // SchedState>()` bytes. We only read `magic` here; the scheduler wrote + // it once in create() before any worker attached, so the read is + // race-free. + let ptr = map.ptr() as *mut SchedState; + let state = unsafe { &*ptr }; let observed_magic = state.magic; if observed_magic != MAGIC { - // SAFETY: ptr was returned by mmap with `size` bytes; unmap it. - unsafe { munmap(ptr, size) }; - // SAFETY: fd is a valid open descriptor; close it to leak no fd. - unsafe { close(fd) }; + // Dropping `map` unmaps and closes the segment; removing the file + // is the owner's job, so this error path leaks nothing. return Err(format!( "magic mismatch: 0x{:08x} (expected 0x{:08x}) \ [module=scheduler, method=SchedShm::open]", @@ -314,12 +206,7 @@ impl SchedShm { )); } - Ok(Self { - path: path.to_string(), - fd, - ptr: ptr as *mut SchedState, - is_owner: false, - }) + Ok(Self { map, ptr }) } /// Atomically claim up to `max_want` cores from the shared pool. @@ -491,52 +378,10 @@ impl SchedShm { /// The scheduler uses this to pass `CODESCOPE_SCHED_SHM=` to /// worker subprocesses so they can attach via [`SchedShm::open`]. pub fn path(&self) -> &str { - &self.path - } -} - -impl Drop for SchedShm { - fn drop(&mut self) { - let size = std::mem::size_of::(); - if !self.ptr.is_null() { - // SAFETY: self.ptr was returned by mmap with `size` bytes and - // has not been unmapped yet (Drop runs once per instance). - unsafe { - munmap(self.ptr as *mut c_void, size); - } - } - if self.fd >= 0 { - // SAFETY: self.fd is a valid open descriptor (or already closed, - // in which case close returns EBADF — harmless and ignored). - unsafe { - close(self.fd); - } - } - // Only the scheduler (owner) unlinks the file. Workers just - // munmap+close — the inode persists until the scheduler unlinks, - // and any in-flight mmap references stay valid (POSIX semantics). - if self.is_owner - && let Ok(c_path) = CString::new(self.path.clone()) - { - // SAFETY: c_path is a valid NUL-terminated CString. unlink - // removes the directory entry; existing mmap references - // remain valid until munmap (POSIX shared memory semantics). - unsafe { - unlink(c_path.as_ptr()); - } - } + self.map.path() } } -// SAFETY: SchedShm is Send because: -// - `ptr` points to mmap'd memory (MAP_SHARED) that is process-global; -// moving the Rust handle across threads does not affect the underlying -// memory or its visibility to other processes. -// - `fd` (RawFd = i32), `path` (String), and `is_owner` (bool) are all -// Send by default. -// Moving SchedShm to another thread grants no exclusive access to the -// shared state — the thread still observes the same atomic-updated -// SchedState that all processes/threads see. unsafe impl Send for SchedShm {} // SAFETY: SchedShm is Sync because: @@ -552,7 +397,7 @@ unsafe impl Send for SchedShm {} // read-only afterwards. `create()` completes before the // `SchedShm` is shared with worker threads (caller's responsibility), // so there are no concurrent reads during the writes. -// - `fd` and `is_owner` are only accessed in `Drop`, which takes +// - `map` (the `MappedFile`) is only touched by `Drop`, which takes // `&mut self` (exclusive access) — no concurrent access is possible. unsafe impl Sync for SchedShm {} @@ -562,10 +407,12 @@ mod tests { use std::sync::atomic::AtomicU64; /// Monotonic counter appended to test paths so parallel test - /// processes never collide on the same /tmp file. + /// processes never collide on the same temp file. static TEST_COUNTER: AtomicU64 = AtomicU64::new(0); - /// Build a unique /tmp path for one test invocation. + /// Build a unique path in the platform temp directory for one test + /// invocation. `temp_dir()` rather than a hardcoded `/tmp` because this + /// suite also runs on Windows hosts, where `/tmp` does not exist. fn unique_path() -> String { let pid = std::process::id(); let nanos = std::time::SystemTime::now() @@ -573,10 +420,10 @@ mod tests { .map(|d| d.as_nanos() as u64) .unwrap_or(0); let counter = TEST_COUNTER.fetch_add(1, Ordering::Relaxed); - format!( - "/tmp/codescope_sched_test_{}_{}_{}.shm", - pid, nanos, counter - ) + std::env::temp_dir() + .join(format!("codescope_sched_test_{pid}_{nanos}_{counter}.shm")) + .to_string_lossy() + .into_owned() } /// Read `available_cores` directly from the shared state. Used by diff --git a/server/src/scheduler/worker.rs b/server/src/scheduler/worker.rs index a174e68..963c31b 100644 --- a/server/src/scheduler/worker.rs +++ b/server/src/scheduler/worker.rs @@ -14,11 +14,31 @@ //! `CODESCOPE_EXCLUDE_PATHS`. use serde_json::Value; -use std::path::Path; +use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; use std::sync::mpsc; use std::time::{Duration, Instant}; +/// Parse mode every scheduler-spawned worker runs in, whatever the caller set. +/// +/// `index-parallel` parses in **fast** mode on purpose: the per-module workers +/// are followed by a merge and one graph/state pass over the unified DB, so a +/// worker that honoured `CODESCOPE_INDEX_MODE` would do per-module work whose +/// results the merge then discards. README §9 states this ("index-parallel +/// always parses in fast mode"); pinning it in one place keeps the four spawn +/// sites from drifting apart. +pub(super) const WORKER_INDEX_MODE: &str = "fast"; + +/// Force [`WORKER_INDEX_MODE`] on a worker command. +/// +/// A separate function so the contract is assertable: a test can inspect the +/// command's environment (`Command::get_envs`) instead of trusting four +/// scattered `cmd.env(...)` calls. `Command::env` writes last, so this also +/// overrides any `CODESCOPE_INDEX_MODE` the parent exported. +pub(super) fn apply_worker_index_mode(cmd: &mut Command) { + cmd.env("CODESCOPE_INDEX_MODE", WORKER_INDEX_MODE); +} + use super::{DEFAULT_WORKER_TIMEOUT_SECS, ModuleResult, POLL_INTERVAL}; /// Run one module worker subprocess. Returns a `ModuleResult` regardless @@ -33,6 +53,62 @@ use super::{DEFAULT_WORKER_TIMEOUT_SECS, ModuleResult, POLL_INTERVAL}; /// `quarantine_exclude` — when `Some(patterns)`, sets the /// `CODESCOPE_EXCLUDE_PATHS` env var so the worker's FilterPolicy /// skips the listed files. Patterns are comma-separated globs. +/// `dir/**` for every subdirectory of `dir`, comma-separated: the globs that +/// keep the root module's worker from descending into another module's files. +/// +/// `dir/**` rather than a bare `dir` because only the `/**` form makes +/// `FilterPolicy` skip a directory AND its contents (see +/// `filter_policy_detect.cpp`). Sorted so the exclude string is stable. +fn subdirectory_excludes(dir: &str) -> String { + let mut pats: Vec = Vec::new(); + if let Ok(entries) = std::fs::read_dir(dir) { + for entry in entries.flatten() { + if !entry.path().is_dir() { + continue; + } + if let Some(name) = entry.file_name().to_str() { + pats.push(format!("{}/**", name)); + } + } + } + pats.sort(); + join_exclude_patterns(&pats) +} + +/// Escape one `CODESCOPE_EXCLUDE_PATHS` pattern for the comma-separated list. +/// +/// `FilterPolicy::loadExcludeEnv` (engine/src/filter_policy_ignore.cpp) splits +/// the value on UNESCAPED commas, so a pattern containing a literal `,` — a +/// directory named `a,b` — would otherwise be torn into two invalid patterns +/// and the intended path would NOT be excluded. That matters most for the +/// quarantine list: the crashing file would be re-indexed by the retry worker +/// it was supposed to be skipped in. A literal backslash is escaped too, so +/// the C++ unescape is lossless. +/// +/// \param pattern A raw glob pattern (no list separators). +/// \return The pattern with `\` and `,` backslash-escaped. +fn escape_exclude_pattern(pattern: &str) -> String { + let mut out = String::with_capacity(pattern.len()); + for ch in pattern.chars() { + if ch == '\\' || ch == ',' { + out.push('\\'); + } + out.push(ch); + } + out +} + +/// Join patterns into a single `CODESCOPE_EXCLUDE_PATHS` value, escaping each +/// element first (see `escape_exclude_pattern`). The separating commas are +/// emitted unescaped, which is what the C++ side splits on. +pub(super) fn join_exclude_patterns(patterns: &[String]) -> String { + patterns + .iter() + .map(|p| escape_exclude_pattern(p)) + .collect::>() + .join(",") +} + #[allow(clippy::too_many_arguments)] pub(super) fn run_module_worker( exe: &str, @@ -46,8 +122,23 @@ pub(super) fn run_module_worker( quarantine_exclude: Option<&str>, keep_db: bool, ) -> ModuleResult { - let module_dir = Path::new(project_dir).join(module_name); - let module_db = format!("{}_{}.db", db_prefix, module_name); + // The root module (see discover::ROOT_MODULE_NAME) owns the files that sit + // directly in the target directory, so its worker is rooted AT the project + // directory and excludes every subdirectory: each one is either another + // module's root or a directory the discovery pass skipped. Excluding them + // is what keeps the split exact — the worker's own walk path is otherwise + // unchanged, so quarantine (which also works through + // CODESCOPE_EXCLUDE_PATHS) keeps working for this module too. + let is_root_module = module_name == crate::discover::ROOT_MODULE_NAME; + let module_dir = if is_root_module { + PathBuf::from(project_dir) + } else { + Path::new(project_dir).join(module_name) + }; + // File name for the module DB. "." would give the unreadable name + // "_..db". + let module_key = if is_root_module { "root" } else { module_name }; + let module_db = format!("{}_{}.db", db_prefix, module_key); // Normally start from a clean DB file — a stale DB would have // outdated graph_nodes from a previous (possibly crashed) run. @@ -60,7 +151,11 @@ pub(super) fn run_module_worker( let _ = std::fs::remove_file(format!("{}-shm", module_db)); } - let project_name = format!("parallel-{}", module_name); + let project_name = if is_root_module { + "parallel-root".to_string() + } else { + format!("parallel-{}", module_name) + }; let workers_str = workers.to_string(); // Pass the scheduler-assigned project_id to the worker. The worker // (main.rs) treats a non-zero value as a forced project_id and skips @@ -83,7 +178,14 @@ pub(super) fn run_module_worker( ]); cmd.env("GRAMMARS_DIR", grammars_dir); cmd.env("CODESCOPE_DB_PATH", &module_db); - cmd.env("CODESCOPE_INDEX_MODE", "fast"); + // The worker is rooted at the MODULE directory, so it would load the + // ignore files found there — usually none. The exclusions live at the + // project root (`.gitignore`, `.codescopeignore`), so pass it and let the + // engine load both: a module worker must apply the same rules as a + // whole-project index, or the parallel path indexes what the project + // declared out of scope. + cmd.env("CODESCOPE_PROJECT_ROOT", project_dir); + apply_worker_index_mode(&mut cmd); cmd.env("CODESCOPE_WORKERS", &workers_str); // Skip the ~280ms state-builder work in per-module workers; the // unified DB gets its async pass once after merge (see merge::merge_module_dbs). @@ -94,7 +196,19 @@ pub(super) fn run_module_worker( // the merged main.db where relation ids are already global, so CSR-based // graph queries don't return dangling neighbor ids. cmd.env("CODESCOPE_DEFER_CSR", "1"); - if let Some(exclude) = quarantine_exclude { + // The root module always excludes the subdirectories; a quarantine retry + // adds the files it proved to be crashers on top of that. + let root_exclude = if is_root_module { + subdirectory_excludes(project_dir) + } else { + String::new() + }; + let exclude = match (root_exclude.is_empty(), quarantine_exclude) { + (false, Some(q)) => Some(format!("{},{}", root_exclude, q)), + (false, None) => Some(root_exclude), + (true, q) => q.map(|s| s.to_string()), + }; + if let Some(exclude) = exclude.as_deref() { cmd.env("CODESCOPE_EXCLUDE_PATHS", exclude); } cmd.stdout(Stdio::piped()).stderr(Stdio::inherit()); @@ -219,15 +333,30 @@ pub(super) fn run_module_worker( None => (0, 0, 0, files_estimate, 0), }; - let error = if exit_code == 0 && parsed.is_some() { + // Engine-level failure (ok:false) is a module failure even when the + // process exits 0: failure envelopes omit total_nodes/files_indexed + // (both 0), which the success predicate would otherwise accept as an + // empty module. Only an explicit `"ok": true` counts as success — + // `ok != false` would treat an error envelope with no `ok` field + // (`{"error":"..."}`, serde_json Null) as a success. + let engine_ok = match &parsed { + Some(v) => v["ok"] == true, + None => false, + }; + let error = if exit_code == 0 && parsed.is_some() && engine_ok { None } else { // stderr is inherited (Stdio::inherit()), so the child's // stderr goes directly to the parent's stderr. The caller // can find the full error in the parent's stderr log. + let engine_msg = parsed + .as_ref() + .and_then(|v| v["error"].as_str()) + .map(|s| format!(" engine_error={}", s)) + .unwrap_or_default(); Some(format!( - "exit={} [module=scheduler, method=run_module_worker]", - exit_code + "exit={}{} [module=scheduler, method=run_module_worker]", + exit_code, engine_msg )) }; @@ -323,18 +452,24 @@ pub(super) fn make_relative_glob(abs_path: &str, module_dir: &str) -> String { let abs = Path::new(abs_path); let modp = Path::new(module_dir); let rel = abs.strip_prefix(modp).unwrap_or(abs); - let basename = rel - .file_name() - .map(|n| n.to_string_lossy().to_string()) - .unwrap_or_default(); - // Use just the basename so the glob matches the file anywhere - // under the module dir — this mirrors the script's behaviour - // (`find ... -path "*/$(basename pattern) -delete`). - if basename.is_empty() { - abs_path.to_string() - } else { - format!("*/{}", basename) + let rel_str = rel.to_string_lossy().replace('\\', "/"); + if rel_str.is_empty() { + return abs_path.to_string(); } + // Emit the path with its DIRECTORY, never the bare basename. + // + // The previous implementation returned `*/{basename}`, which matches any + // file with that name anywhere under the module root: quarantining a + // crashing `a/foo.cpp` also excluded the healthy `c/foo.cpp`, silently + // dropping it from the index while the run still reported success. + // + // One `**/` pattern covers every root the FilterPolicy may match against: + // `**` spans '/', so it matches both the module-relative form (`a/foo.cpp`, + // the retry worker is rooted at the module dir) and a project-rooted form + // (`module/a/foo.cpp`), at any nesting depth. Neither a bare `rel` nor + // `*/rel` would: the pattern is matched against the whole relative path and + // `*` does not cross '/' (see FilterPolicy::loadExcludeEnv). + format!("**/{}", rel_str) } /// Run one chunk worker subprocess. @@ -383,7 +518,7 @@ pub(super) fn run_chunk_worker( cmd.env("CODESCOPE_DB_PATH", worker_db); cmd.env("CODESCOPE_FILES_JSON", files_json_path); cmd.env("CODESCOPE_PROJECT_ID", &project_id_str); - cmd.env("CODESCOPE_INDEX_MODE", "fast"); + apply_worker_index_mode(&mut cmd); cmd.env("CODESCOPE_SKIP_ASYNC", "1"); // P3a (C2): chunk workers are parallel modules too — defer CSR so it is // rebuilt once on the merged DB from globally-remapped relation ids. @@ -414,7 +549,7 @@ pub(super) fn run_chunk_worker( taskset_cmd.env("CODESCOPE_DB_PATH", worker_db); taskset_cmd.env("CODESCOPE_FILES_JSON", files_json_path); taskset_cmd.env("CODESCOPE_PROJECT_ID", &project_id_str); - taskset_cmd.env("CODESCOPE_INDEX_MODE", "fast"); + apply_worker_index_mode(&mut taskset_cmd); taskset_cmd.env("CODESCOPE_SKIP_ASYNC", "1"); // P3a (C2): defer CSR on chunk workers too (see plain branch above). taskset_cmd.env("CODESCOPE_DEFER_CSR", "1"); @@ -447,28 +582,30 @@ pub(super) fn run_chunk_worker( } }; + // Drain stdout in a dedicated thread. A chunk worker can emit more than + // the OS pipe buffer (~64 KB); reading only after the child exits would + // block the child on write, so it never exits and the run looks like a + // timeout (false -4) with the data discarded. + let child_stdout = child.stdout.take(); + let reader = std::thread::spawn(move || -> Vec { + let mut buf = Vec::new(); + if let Some(mut s) = child_stdout { + use std::io::Read; + let _ = s.read_to_end(&mut buf); + } + buf + }); + // Wait for the child with a timeout. let timeout = Duration::from_secs(DEFAULT_WORKER_TIMEOUT_SECS); - let (status, stdout_bytes) = loop { + let status = loop { match child.try_wait() { - Ok(Some(status)) => { - // Read stdout. - let stdout = child - .stdout - .take() - .map(|mut s| { - let mut buf = Vec::new(); - use std::io::Read; - let _ = s.read_to_end(&mut buf); - buf - }) - .unwrap_or_default(); - break (status, stdout); - } + Ok(Some(status)) => break status, Ok(None) => { if t0.elapsed() > timeout { let _ = child.kill(); let _ = child.wait(); + let _ = reader.join(); return ModuleResult { name: format!("chunk-worker-{}", worker_id), exit_code: -4, @@ -490,6 +627,9 @@ pub(super) fn run_chunk_worker( std::thread::sleep(Duration::from_millis(50)); } Err(e) => { + let _ = child.kill(); + let _ = child.wait(); + let _ = reader.join(); return ModuleResult { name: format!("chunk-worker-{}", worker_id), exit_code: -3, @@ -511,6 +651,8 @@ pub(super) fn run_chunk_worker( } }; + let stdout_bytes = reader.join().unwrap_or_default(); + let duration = t0.elapsed().as_secs(); let exit_code = status.code().unwrap_or(-4); let stdout = String::from_utf8_lossy(&stdout_bytes).to_string(); @@ -527,12 +669,25 @@ pub(super) fn run_chunk_worker( None => (0, 0, 0, 0, 0), }; - let error = if exit_code == 0 && parsed.is_some() { + // Engine-level failure (ok:false) is a chunk-worker failure even at exit + // 0, exactly like run_module_worker. Without this check an ok:false + // envelope looks like an empty chunk: error stays None, the worker's DB is + // merged, and the run can report complete while its files were dropped. + let engine_ok = match &parsed { + Some(v) => v["ok"] == true, + None => false, + }; + let error = if exit_code == 0 && parsed.is_some() && engine_ok { None } else { + let engine_msg = parsed + .as_ref() + .and_then(|v| v["error"].as_str()) + .map(|s| format!(" engine_error={}", s)) + .unwrap_or_default(); Some(format!( - "exit={} [module=scheduler, method=run_chunk_worker]", - exit_code + "exit={}{} [module=scheduler, method=run_chunk_worker]", + exit_code, engine_msg )) }; @@ -576,10 +731,80 @@ mod tests { assert_eq!(v["discovery"]["candidate_files"], 100); } + /// Regression: `engine_ok` must be true only for an explicit `"ok":true`. + /// `ok != false` treated an error envelope with no `ok` field + /// (`{"error":"not initialized"}`, serde_json Null) as a success, so a + /// module whose engine refused to index was counted as an empty success. + #[test] + fn test_engine_ok_requires_explicit_true() { + let ok_true: Value = serde_json::from_str(r#"{"ok":true,"total_nodes":1}"#).unwrap(); + let ok_false: Value = serde_json::from_str(r#"{"ok":false,"error":"boom"}"#).unwrap(); + let no_ok: Value = serde_json::from_str(r#"{"error":"not initialized"}"#).unwrap(); + // Mirrors run_module_worker's predicate. + let engine_ok = |v: &Value| v["ok"] == true; + assert!(engine_ok(&ok_true)); + assert!(!engine_ok(&ok_false)); + assert!( + !engine_ok(&no_ok), + "an error envelope with no `ok` field must not count as success" + ); + } + + #[test] + fn test_subdirectory_excludes_use_recursive_globs() { + // A bare "sub" matches nothing in FilterPolicy, so only "sub/**" can + // stop the root module's worker from descending into another module. + assert_eq!(subdirectory_excludes("/nonexistent/dir/xyz"), ""); + let dir = std::env::temp_dir().join("codescope_test_subdir_excludes"); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(dir.join("a")).unwrap(); + std::fs::create_dir_all(dir.join("b")).unwrap(); + std::fs::write(dir.join("f.go"), "package main").unwrap(); + assert_eq!(subdirectory_excludes(dir.to_str().unwrap()), "a/**,b/**"); + let _ = std::fs::remove_dir_all(&dir); + } + + /// Regression (2026-09-27 review, D2-7): `CODESCOPE_EXCLUDE_PATHS` + /// is comma-separated and `FilterPolicy::loadExcludeEnv` splits on every + /// unescaped comma, so a pattern for a directory named "a,b" was torn into + /// `a\` + `b/**` and the path was never excluded. Escaping keeps it one + /// pattern; a literal backslash is escaped too so the C++ unescape is + /// lossless. #[test] - fn test_make_relative_glob_returns_basename_glob() { + fn test_join_exclude_patterns_escapes_separators() { + let pats = vec![ + "a,b/**".to_string(), + "plain/**".to_string(), + "back\\slash/**".to_string(), + ]; + assert_eq!( + join_exclude_patterns(&pats), + r"a\,b/**,plain/**,back\\slash/**" + ); + // Patterns without a comma or backslash are passed through unchanged. + assert_eq!(join_exclude_patterns(&["x/**".to_string()]), "x/**"); + // No patterns → empty value (the caller then leaves the env var unset). + assert_eq!(join_exclude_patterns(&[]), ""); + } + + #[test] + fn test_make_relative_glob_keeps_the_directory() { let g = make_relative_glob("/abs/path/engine/src/parser.cpp", "/abs/path/engine"); - assert_eq!(g, "*/parser.cpp"); + // `**/` spans separators, so one pattern covers the file at any depth + // and under either root. It must never degrade to the bare basename: + // `*/parser.cpp` also matched every other parser.cpp. + assert_eq!(g, "**/src/parser.cpp"); + } + + #[test] + fn test_make_relative_glob_distinguishes_same_named_files() { + // Regression: quarantining a crashing `a/foo.cpp` emitted `*/foo.cpp`, + // which also excluded the healthy `c/foo.cpp`. + let a = make_relative_glob("/m/a/foo.cpp", "/m"); + let c = make_relative_glob("/m/c/foo.cpp", "/m"); + assert_ne!(a, c); + assert_eq!(a, "**/a/foo.cpp"); + assert_eq!(c, "**/c/foo.cpp"); } #[test] @@ -589,4 +814,110 @@ mod tests { // to the absolute path. Both branches must produce a non-empty glob. assert!(!g.is_empty()); } + + /// Regression (2026-09-27 review, "配套测试缺口" #4): the README + /// documents that `index-parallel` parses in fast mode whatever the user + /// asked for, and nothing asserted it — the spawn sites could drift apart. + #[test] + fn test_spawned_workers_are_pinned_to_fast_mode() { + let mut cmd = Command::new("codescope"); + // Stand in for a user who exported the opposite mode. + cmd.env("CODESCOPE_INDEX_MODE", "normal"); + apply_worker_index_mode(&mut cmd); + + let mode: Vec<_> = cmd + .get_envs() + .filter(|(k, _)| *k == "CODESCOPE_INDEX_MODE") + .collect(); + assert_eq!( + mode.len(), + 1, + "the worker's parse mode must be stated exactly once" + ); + assert_eq!( + mode[0].1, + Some(std::ffi::OsStr::new(WORKER_INDEX_MODE)), + "index-parallel workers must run in fast mode; Command::env writes last, \ + so this also proves it overrides the caller's value" + ); + } +} + +/// Property-based coverage of the `CODESCOPE_EXCLUDE_PATHS` escaping contract +/// (REVIEW_0.2.7.md TEST-3 / code_rules §4). +/// +/// `join_exclude_patterns` is the producer; `FilterPolicy::loadExcludeEnv` +/// (engine/src/filter_policy_ignore.cpp) is the consumer and splits on every +/// unescaped comma. The two are written in different languages, so the only +/// thing that keeps them in sync is this round-trip property: a drift in +/// either escaping silently stops a quarantined path from being excluded. +#[cfg(test)] +mod proptests { + use super::*; + use proptest::prelude::*; + + /// Trim exactly what the C++ consumer trims around each pattern + /// (`find_first_not_of(" \t")`), dropping an all-whitespace entry. + fn push_trimmed(out: &mut Vec, pat: &mut String) { + let trimmed = pat.trim_matches(|c| c == ' ' || c == '\t'); + if !trimmed.is_empty() { + out.push(trimmed.to_string()); + } + pat.clear(); + } + + /// Reference re-implementation of the C++ splitter: split on unescaped + /// commas, `\,` and `\\` unescape to the bare character, any other + /// backslash is literal. + fn split_like_cpp(raw: &str) -> Vec { + let mut out = Vec::new(); + let mut pat = String::new(); + let chars: Vec = raw.chars().collect(); + let mut i = 0; + while i < chars.len() { + let ch = chars[i]; + if ch == '\\' && i + 1 < chars.len() && (chars[i + 1] == ',' || chars[i + 1] == '\\') { + pat.push(chars[i + 1]); + i += 2; + continue; + } + if ch == ',' { + push_trimmed(&mut out, &mut pat); + i += 1; + continue; + } + pat.push(ch); + i += 1; + } + push_trimmed(&mut out, &mut pat); + out + } + + proptest! { + #![proptest_config(ProptestConfig::with_cases(512))] + + #[test] + fn join_exclude_patterns_round_trips( + patterns in prop::collection::vec(any::(), 0..8) + ) { + // The producer never trims and never emits an empty element, so + // normalise the generated input the same way before comparing. + let pats: Vec = patterns + .iter() + .map(|p| p.trim_matches(|c| c == ' ' || c == '\t').to_string()) + .filter(|p| !p.is_empty()) + .collect(); + prop_assert_eq!(split_like_cpp(&join_exclude_patterns(&pats)), pats); + } + + /// A comma or backslash inside a pattern must survive the escape → + /// unescape round trip as itself. + #[test] + fn escape_exclude_pattern_is_lossless(pattern in any::()) { + let p = pattern.trim_matches(|c| c == ' ' || c == '\t'); + prop_assume!(!p.is_empty()); + let escaped = escape_exclude_pattern(p); + prop_assert_eq!(split_like_cpp(&escaped), vec![p.to_string()]); + } + } } diff --git a/server/src/tools/catalog.rs b/server/src/tools/catalog.rs new file mode 100644 index 0000000..6de6701 --- /dev/null +++ b/server/src/tools/catalog.rs @@ -0,0 +1,550 @@ +// catalog.rs — the MCP tool catalog. +// +// Split out of tools/mod.rs (see plan/rules/code_rules.md 1000-line rule). +// `all_tools()` is the full list of tool names, descriptions and JSON +// input schemas the server advertises in `tools/list`. It is pure data, +// so it moved as one block: the handler table stays in mod.rs, and a tool +// whose schema is missing from here (or vice versa) shows up immediately +// as a discover/execute mismatch. + +use crate::mcp::protocol::Tool; +use serde_json::json; + +// ─── Tool Definitions ─────────────────────────────────────────── + +pub fn all_tools() -> Vec { + vec![ + Tool { + name: "find_definition".into(), + description: "[DEPRECATED — use find_symbol] Find where a symbol is defined.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "symbol_name": {"type": "string", "description": "Name of the symbol to find"}, + "file_filter": {"type": "string", "description": "Optional: filter by file path substring"} + }, + "required": ["symbol_name"] + }), + }, + Tool { + name: "find_references".into(), + description: "Find all locations that reference a given symbol. References are recorded per call site (Calls and symbol_reference edges), so a symbol that is only USED AS A VALUE — a handler registered with a router (app.get(\"/x\", handler)), a callback passed by name, a function pointer — has no reference row; an empty result is therefore not proof that the symbol is unused, and get_routes / find_symbol / get_type_info can still locate it.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "symbol_name": {"type": "string"}, + "file_filter": {"type": "string"} + }, + "required": ["symbol_name"] + }), + }, + Tool { + name: "search_code".into(), + description: "[DEPRECATED — use search] Full-text search across code symbols, file paths, and comments.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "query": {"type": "string", "description": "Search query string"}, + "limit": {"type": "integer", "description": "Max results (default 20, max 100)"} + }, + "required": ["query"] + }), + }, + Tool { + name: "index_file".into(), + description: "Index a single source file: parse, build IR, and add to the code graph.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "file_path": {"type": "string", "description": "Absolute path to the source file"} + }, + "required": ["file_path"] + }), + }, + Tool { + name: "force_index_files".into(), + description: "Force-index specific files or directories, BYPASSING the default skip rules (test/, docs/, vendored/, node_modules/, .gitignore, etc.). Use this when the user asks to index a path that the default FilterPolicy would skip — e.g. 'go index xxx/yyy for me'. Files under the given paths are indexed regardless of the default skip list, so the user can pull in test fixtures, vendored deps, or generated code on demand. Nested compiler/package output trees are still skipped at any depth (node_modules/, target/, _deps/, __pycache__/, venvs, .git/, and build*/ dirs carrying build-system artifacts — CMake markers for any build*, Gradle leaves only for a dir named `build`) — pass such a directory itself as a path to index its contents anyway. Still respects: file size limit, language detectability, optional language whitelist.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "paths": { + "type": "array", + "items": {"type": "string"}, + "description": "Array of absolute file/dir paths to force-index. Directories are walked recursively, bypassing skip-dir rules." + }, + "path": { + "type": "string", + "description": "Convenience single-path form (merged into `paths` if both given)." + }, + "language_filter": { + "type": "string", + "description": "Optional comma-separated language whitelist (e.g. \"java,python\"). Empty = all detectable languages." + } + }, + "required": ["paths"] + }), + }, + Tool { + name: "get_graph_stats".into(), + description: "Get statistics about the current code graph (node count, edge count, file count).".into(), + input_schema: json!({ "type": "object", "properties": {} }), + }, + Tool { + name: "trace_flow".into(), + description: "Trace the execution flow from a function through its callees recursively. Returns a call chain showing how control flows through the codebase.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "function_name": {"type": "string", "description": "Starting function name"}, + "depth": {"type": "integer", "description": "How many levels to trace (default 3, max 10)"} + }, + "required": ["function_name"] + }), + }, + Tool { + name: "explain_symbol".into(), + description: "Get structured information about a symbol: definition location, callers, callees, and dependencies. Returns a comprehensive view of what the symbol does and how it relates to the rest of the codebase.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "symbol_name": {"type": "string", "description": "Name of the symbol to explain"} + }, + "required": ["symbol_name"] + }), + }, + Tool { + name: "verify_integrity".into(), + description: "Verify codebase integrity: checks that README-promised features actually exist in the code. Returns findings with evidence chains and confidence scores. The findings array is capped at max_findings (default 200); `truncated` is the completeness signal. `total` counts every verdict (including Supported ones, which are not listed as findings), so it is normally larger than the findings array.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "max_findings": { + "type": "integer", + "description": "Maximum number of findings to include in the response (default 200, max 2000). `truncated` is true when the array was cut short; `total` counts all verdicts, not just findings.", + "minimum": 1, + "maximum": 2000 + } + } + }), + }, + Tool { + name: "verify_claim".into(), + description: "Verify a single claim against the codebase. Dispatches the claim to the appropriate verifier (CapabilityVerifier, ContractVerifier, or ArchitectureVerifier) via the VerifierRegistry, persists the claim + evidence, and returns the verdict with confidence and evidence facts. The claim argument is a JSON OBJECT string, not free text: {\"type\":\"capability_exists|contract_holds|architecture_follows|function_implements\",\"subject\":\"\",\"predicate\":\"\"}. A missing or unrecognized type is rejected with error_code claim_type_unsupported rather than being guessed.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "claim": { + "type": "string", + "description": "The claim as a JSON object (also accepted as a JSON-encoded string). Fields: type (capability_exists|contract_holds|architecture_follows|function_implements), subject, predicate, object, scope, source_kind, source_ref" + } + }, + "required": ["claim"] + }), + }, + Tool { + name: "verify_summary".into(), + description: "Parse a natural-language summary (README excerpt, AI summary, PR description) into structured claims using the ClaimParser, then verify each claim. Returns aggregated verdicts and a trust_score. Useful for checking whether prose claims about the codebase are actually backed by code evidence.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "text": { + "type": "string", + "description": "Free-form text to parse + verify (e.g. 'CodeScope supports incremental indexing and is thread-safe')" + } + }, + "required": ["text"] + }), + }, + Tool { + name: "verify_review".into(), + description: "Verify a code review comment by parsing it into claims and dispatching each through the Claim -> Verifier -> Evidence pipeline. Each claim is stamped source_kind=\"code_review\" so the evidence table can be filtered by origin. Output shape is identical to verify_summary.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "text": { + "type": "string", + "description": "The review comment body (e.g. 'This function should be thread-safe and the README claims JWT support')" + } + }, + "required": ["text"] + }), + }, + Tool { + name: "verify_reality".into(), + description: "Verify a single AI statement about the current project reality. Returns a structured evidence report with an aggregate verdict (Supported / Contradicted / PartiallyVerified / Unknown), a confidence score, and the per-claim results array. Use this to check whether an AI's claim about project state is backed by code evidence.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "text": { + "type": "string", + "description": "A single natural-language statement (e.g. 'The login module supports JWT and Refresh tokens.')" + } + }, + "required": ["text"] + }), + }, + Tool { + name: "detect_drift".into(), + description: "Scan all declared capabilities and contracts for drift between documentation/code comments and the actual codebase. Detects MissingCapability (declared in README but no implementing entity with callers) and BrokenContract (declared but no enforcing code). Each drift is persisted as a finding row and returned in the JSON output.".into(), + input_schema: json!({ + "type": "object", + "properties": {} + }), + }, + Tool { + name: "detect_documentation_drift".into(), + description: "Scan README for language support claims (e.g. 'supports C++, Python, Go') and cross-reference with actual entities in the codebase. Reports DocumentationDrift (severity 1) for any claimed language with zero entities. Returns claimed_languages, found_languages, missing_languages, and drifts arrays. Enables end-to-end verification: AI says X, CodeScope checks the tables.".into(), + input_schema: json!({ + "type": "object", + "properties": {} + }), + }, + Tool { + name: "detect_capability_drift".into(), + description: "Scan declared capabilities and cross-reference with actual implementing entities in the codebase. Reports CapabilityDrift (severity 2) for any declared capability with no implementing entity that has callers. Returns total_capabilities count and drifts array. Part of end-to-end verification: AI says X, CodeScope checks the tables.".into(), + input_schema: json!({ + "type": "object", + "properties": {} + }), + }, + Tool { + name: "detect_architecture_drift".into(), + description: "Scan call edges for architecture layer violations (e.g. Repository calling Controller, Controller calling another Controller directly). Classifies entities into Controller/Service/Repository layers by naming convention and file path, then checks the relation table for reverse calls and same-layer bypasses. Reports ArchitectureDrift (severity 1) for each violating call edge.".into(), + input_schema: json!({ + "type": "object", + "properties": {} + }), + }, + Tool { + name: "explain_module".into(), + description: "Build a Knowledge Card for a named module/directory. Returns module info, entities, capabilities, contracts, findings, and an integrity_score. Falls back to deriving module info from the files table by path prefix when the modules table is empty.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "module_name": { + "type": "string", + "description": "Name of the module/directory (e.g. 'engine', 'server')" + } + }, + "required": ["module_name"] + }), + }, + Tool { + name: "enhance_project".into(), + description: "Run background enhancement for a project: full tree-sitter parse, call graph construction, FTS index build, and v0.3 semantic_fact extraction (Step 1.5). This is the prerequisite for build_evidence to produce non-empty findings — the semantic facts (sync/mutex/lock, memory/cstring/alloc, error/bare_except, pattern/todo, framework/gin, ffi/extern_call) are extracted here. Returns a JSON summary with files_processed, symbols_enhanced, call_edges, and timing breakdowns. v0.2.5: complexity metrics (cyclomatic/cognitive/nesting) and n-gram semantic vectors are restored and built during index; use engine_get_capabilities / engine_get_enhancement_status to see readiness.".into(), + input_schema: json!({ + "type": "object", + "properties": {} + }), + }, + Tool { + name: "build_evidence".into(), + description: "Build evidence findings for the project by applying the v0.3 rule set (sync/memory/error/pattern/framework/ffi) to the indexed semantic_fact rows. Each rule declares its fact needs and a combine mode (Collect / MissingMatch / MissingMatchPerFunction / Count); the engine returns a JSON array of Evidence objects with category, title, confidence, and items[]. Run after index/enhance so semantic facts are populated.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "category": { + "type": "string", + "enum": ["sync", "memory", "error", "pattern", "framework", "ffi"], + "description": "Optional: restrict the run to one rule category. Omit to run all categories." + } + } + }), + }, + Tool { + name: "build_project_state".into(), + description: "Build (or rebuild) and persist the project state snapshot. Runs the full v0.3 Evidence Pipeline (evidence aggregation + state queries) and UPSERTs the result into the project_state table. Returns the snapshot JSON describing what each inspector ran and what it found (category counts, confidence score, snapshot metadata). Use this after build_evidence to materialize a health snapshot.".into(), + input_schema: json!({ + "type": "object", + "properties": {} + }), + }, + Tool { + name: "get_project_state".into(), + description: "Get the previously persisted project state snapshot (without rebuilding). Returns the snapshot_json string for the project, or a JSON error object if no snapshot exists yet (run build_project_state first). Use this for fast reads of the latest health snapshot.".into(), + input_schema: json!({ + "type": "object", + "properties": {} + }), + }, + Tool { + name: "detect_changes".into(), + description: "Analyze the impact of code changes across the call graph. Given a list of modified files, returns directly modified functions, their callers (who calls them), and their callees (who they call).".into(), + input_schema: json!({ + "type": "object", + "properties": { + "modified_files": {"type": "string", "description": "JSON array of modified file paths"} + }, + "required": ["modified_files"] + }), + }, + Tool { + name: "find_symbol".into(), + description: "Find symbol(s) by exact name match. Returns id, kind, file path, line/column for each match.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "symbol_name": {"type": "string", "description": "Symbol name to search for"} + }, + "required": ["symbol_name"] + }), + }, + Tool { + name: "get_module_tree".into(), + description: "Get the hierarchical module (directory) tree for the current project. Returns modules with id, parent_id, name, path, and file_count.".into(), + input_schema: json!({ "type": "object", "properties": {} }), + }, + Tool { + name: "get_knowledge_graph".into(), + description: "Direct-query a knowledge-layer table (v0.2.1). Surfaces entity / relation / architecture_edge / module_edge / capability / document / module_summary so you can browse the knowledge graph directly instead of only benefiting indirectly via explain_module / detect_capability_drift / get_module_tree. Block-level: one call returns the whole result set.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "table": { + "type": "string", + "enum": ["entity", "relation", "architecture_edge", + "module_edge", "capability", "document", + "module_summary"], + "description": "Knowledge-layer table to query." + }, + "limit": { + "type": "integer", + "default": 100, + "description": "Max rows to return, clamped to [0, 1000]." + } + }, + "required": ["table"] + }), + }, + Tool { + name: "search".into(), + description: "Unified code search: FTS5 exact/prefix matching, complemented by n-gram semantic vector search (restored in v0.2.5) that adds lexically-similar name recall when FTS results run short. Supports prefix matching.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "query": {"type": "string", "description": "Search query string"}, + "limit": {"type": "integer", "description": "Max results (default 20, max 100)"} + }, + "required": ["query"] + }), + }, + Tool { + name: "find_callers".into(), + description: "Find all symbols that call the specified function. Auto-adapts to callgraph readiness: uses fast index first, falls back to full analysis.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "symbol_name": {"type": "string", "description": "Name of the symbol to find callers for"}, + "file_filter": {"type": "string", "description": "Optional: absolute file path. Restricts the callee to the given file, disambiguating homonyms (same name across files/classes, e.g. __init__, run, main). Empty = aggregate all files (legacy behavior, may produce noise on common names)."} + }, + "required": ["symbol_name"] + }), + }, + Tool { + name: "find_callees".into(), + description: "Find all symbols called by the specified function. Auto-adapts to callgraph readiness.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "symbol_name": {"type": "string", "description": "Name of the symbol to find callees for"}, + "file_filter": {"type": "string", "description": "Optional: absolute file path. Restricts the caller to the given file, disambiguating homonyms. See find_callers doc."} + }, + "required": ["symbol_name"] + }), + }, + Tool { + name: "find_callers_by_entity".into(), + description: "Find all symbols that call the specified entity (Step 7, plan §7.2). Targets a single entity by its id, so homonyms (same name across files/classes, e.g. __init__, run, main) are never aggregated. Use the candidates returned by find_callers/find_callees when a bare name is ambiguous, or locate the entity first via find_symbol, then pass its id here.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "entity_id": {"type": "integer", "description": "Entity id from the entity table (e.g. from the candidates list of an ambiguous bare-name query, or from find_symbol output)."} + }, + "required": ["entity_id"] + }), + }, + Tool { + name: "find_callees_by_entity".into(), + description: "Find all symbols called by the specified entity (Step 7, plan §7.2). Targets a single entity by its id, so homonyms are never aggregated.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "entity_id": {"type": "integer", "description": "Entity id from the entity table (e.g. from the candidates list of an ambiguous bare-name query, or from find_symbol output)."} + }, + "required": ["entity_id"] + }), + }, + Tool { + name: "get_verifier_registry_status".into(), + description: "Report verifier subsystem health (Step 9, plan §9.2): whether the verifier registry is armed, which public claim types are supported, and whether the canonical evidence backend (entity/relation) has data for the project. Pass project_id=0 to skip the evidence probe.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "project_id": {"type": "integer", "description": "Project id; 0 skips the evidence backend probe."} + }, + "required": [] + }), + }, + Tool { + name: "shortest_path".into(), + description: "Find the shortest call-graph path between two functions. Heuristic/approximate: the call graph is built from name-matched edges, so the BFS path may not reflect true runtime dispatch. Accepts either symbol names (from/to) or explicit graph node IDs (from_id/to_id).".into(), + input_schema: json!({ + "type": "object", + "properties": { + "from": {"type": "string", "description": "Source symbol name (resolved to a node ID via locate_by_name)"}, + "to": {"type": "string", "description": "Target symbol name (resolved to a node ID via locate_by_name)"}, + "from_id": {"type": "integer", "description": "Explicit source graph node ID (bypasses name resolution)"}, + "to_id": {"type": "integer", "description": "Explicit target graph node ID (bypasses name resolution)"} + }, + // The handler resolves BOTH endpoints and rejects a missing one + // (`resolve_endpoint` in h_shortest_path returns + // "either 'from' (symbol name) or 'from_id' (node id) is + // required"), so a client that reads only `properties` called + // with {} and got an error the schema could have prevented. + // `required` cannot say "the name OR the id", so each endpoint + // is stated as an anyOf and the two are combined with allOf. + "allOf": [ + {"anyOf": [{"required": ["from"]}, {"required": ["from_id"]}]}, + {"anyOf": [{"required": ["to"]}, {"required": ["to_id"]}]} + ] + }), + }, + Tool { + name: "connected_components".into(), + description: "Find connected components in the call graph via BFS over name-matched relation edges. Heuristic/approximate: call edges are inferred from name matches, so reported components are a heuristic grouping of the module structure.".into(), + input_schema: json!({ "type": "object", "properties": {} }), + }, + Tool { + name: "get_communities".into(), + description: "Detect communities (clusters) in the call graph via deterministic label propagation over CALLS edges. Returns {communities:[{id,label,member_count[,members]}], total_communities, returned_communities, inter_community_edges, truncated, approximation:\"heuristic\", note}. Summary-first: each community is a {id,label,member_count} summary unless include_members=true. Heuristic/approximate: edges are resolved by name matching, so indirect calls (virtual/pointer) may be missing.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "max_communities": {"type": "integer", "description": "Max communities to return (default 20, max 500)"}, + "include_members": {"type": "boolean", "description": "Include each community's member list (default false; keeps the payload small)"}, + "max_members": {"type": "integer", "description": "Max members per community when include_members is true (default 10, max 200)"} + } + }), + }, + Tool { + name: "get_entry_points".into(), + description: "Get likely entry points from the new schema (main/init/setup/run/handler). Returns symbol id, name, file path, and line.".into(), + input_schema: json!({ "type": "object", "properties": {} }), + }, + Tool { + name: "get_type_info".into(), + description: "Get type information for the project: type definitions (structs, enums, traits) and their reference counts. Optionally filter by type name. Returns JSON with types array.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "type_name": { + "description": "Optional: filter types by name (substring match)", + "type": "string" + } + } + }), + }, + Tool { + name: "get_routes".into(), + description: "Get HTTP routes registered in the project. Returns JSON with method, path, handler, file, and line for each route. Supports Gin, Echo, Chi, and net/http patterns.".into(), + input_schema: json!({ "type": "object", "properties": {} }), + }, + Tool { + name: "project_overview".into(), + description: "Get a comprehensive project overview: languages, modules/symbols, entry points, analysis progress, and ready features. Call this first after initialization.".into(), + input_schema: json!({ "type": "object", "properties": {} }), + }, + Tool { + name: "detect_ffi_boundaries".into(), + description: "Detect FFI (Foreign Function Interface) boundaries in the project. Returns language distribution, cross-language files, FFI-related symbols (extern, wasm, jni, cabi), one-sided external_symbols (called here but defined outside the index — e.g. a C prototype for a Rust export), and orphan symbols that may serve as FFI entry points.".into(), + input_schema: json!({ "type": "object", "properties": {} }), + }, + Tool { + name: "codescope_trace".into(), + description: "Explore a function's callers/callees recursively, or trace the shortest path between two functions. Use function_name+depth+direction for interactive exploration, or from+to for shortest path.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "function_name": {"type": "string", "description": "Starting function for interactive exploration"}, + "depth": {"type": "integer", "description": "How many levels to explore (default: 1, max: 10 — the server clamps to MAX_TRAVERSAL_DEPTH)"}, + "direction": {"type": "string", "description": "\"callers\", \"callees\", or \"both\" (default: \"both\")"}, + "from": {"type": "string", "description": "Source function for shortest path (legacy)"}, + "to": {"type": "string", "description": "Target function for shortest path (legacy)"} + }, + // Two modes, and the choice between them is what the schema has + // to state: interactive exploration needs a starting function + // (the handler answers `{"error":"empty symbol name"}` when it + // is missing), while the legacy shortest-path mode needs both + // ends. + "anyOf": [ + {"required": ["function_name"]}, + {"required": ["from", "to"]} + ] + }), + }, + Tool { + name: "count_tokens".into(), + description: "Estimate token count for a text string using DeepSeek's formula: ASCII*0.3 + non-ASCII*0.6.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "text": {"type": "string", "description": "Text to estimate tokens for"} + }, + "required": ["text"] + }), + }, + Tool { + name: "get_subgraph".into(), + description: "Fetch a local region of the code graph centered on a node: the center node plus its direct graph neighbors (1 hop). Returns {nodes:[...], total:N}. Optional node_types/edge_types are comma-separated integer id lists (e.g. \"0,1\") to filter by type. Use this to drill into a specific area instead of pulling the whole graph.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "node_id": {"type": "integer", "description": "Center graph node id (required)"}, + "radius": {"type": "integer", "description": "Hop radius (reserved, currently 1)"}, + "node_types": {"type": "string", "description": "Optional comma-separated node type ids, e.g. \"0,1\""}, + "edge_types": {"type": "string", "description": "Optional comma-separated edge type ids, e.g. \"1\""} + }, + "required": ["node_id"] + }), + }, + Tool { + name: "get_neighbors".into(), + description: "Fetch the direct neighbors (callers AND callees) of a graph node. Returns {neighbors:[{id,name,node_type,file_path,edge_type,direction}], total:N}. Optional edge_type filters to a single edge type id (-1 = all).".into(), + input_schema: json!({ + "type": "object", + "properties": { + "node_id": {"type": "integer", "description": "Graph node id (required)"}, + "edge_type": {"type": "integer", "description": "Edge type id filter (-1 = all, default -1)"}, + "radius": {"type": "integer", "description": "Hop radius (reserved, currently 1)"} + }, + "required": ["node_id"] + }), + }, + Tool { + name: "graph_query".into(), + description: "Query the code graph with a Cypher-like DSL. Syntax: MATCH (srcType[:srcName])-[edgeType]->(tgtType[:tgtName]). Node types: Function(0), Method(1), Class(2), Struct(3), Interface(4), Variable(5), Module(6), File(7). Edge types: References(0), Calls(1), Defines(2), Contains(3), Imports(4), Inherits(5). Empty type matches any. Multi-hop: edgeType*min..max, e.g. MATCH (Function)-[Calls*1..3]->(Function). An optional trailing clause bounds or projects the result: LIMIT is honoured (the response then also carries truncated:true), RETURN is accepted and currently has no effect, and any other trailing text is an error. Use LIMIT on large graphs: a broad pattern can exceed the MCP transport's 1 MiB message cap, which replaces the oversized response with an error. Returns {results:[{source, edge, target}], total:N}.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "dsl": {"type": "string", "description": "Graph query DSL string, e.g. \"MATCH (Function:main)-[Calls]->(Method)\""} + }, + "required": ["dsl"] + }), + }, + Tool { + name: "get_graph".into(), + description: "Retrieve the COMPLETE code graph in paginated pages (required for large projects). Returns {totals:{nodes,edges}, nodes:[...], edges:[...], has_more:{nodes,edges}}. Page nodes via node_offset/node_limit and edges via edge_offset/edge_limit. Optional node_types/edge_types filter by comma-separated integer ids. Iterate while has_more is true to reconstruct the entire graph.".into(), + input_schema: json!({ + "type": "object", + "properties": { + "node_offset": {"type": "integer", "description": "Node page start offset (default 0)"}, + "node_limit": {"type": "integer", "description": "Max nodes per page (default 5000, max 50000)"}, + "edge_offset": {"type": "integer", "description": "Edge page start offset (default 0)"}, + "edge_limit": {"type": "integer", "description": "Max edges per page (default 20000, max 200000)"}, + "node_types": {"type": "string", "description": "Optional comma-separated node type ids to include"}, + "edge_types": {"type": "string", "description": "Optional comma-separated edge type ids to include"} + } + }), + }, + ] +} diff --git a/server/src/tools/clamp.rs b/server/src/tools/clamp.rs new file mode 100644 index 0000000..129aafe --- /dev/null +++ b/server/src/tools/clamp.rs @@ -0,0 +1,294 @@ +//! Argument-clamping helpers for MCP tool inputs. +//! +//! Split out of tools/mod.rs (see plan/rules/code_rules.md 1000-line rule). +//! Every client-supplied numeric bound that feeds a recursive walk, a row +//! limit, or a typed filter is clamped here so a huge `i64` can never be +//! truncated by `as i32` into a negative or unbounded engine parameter. + +/// Upper bound for `limit` arguments accepted by query-style tools. +/// Values above this are clamped down to prevent unbounded result sets. +pub const MAX_QUERY_LIMIT: i64 = 100; +/// Default `limit` used when the client omits the argument. +pub const DEFAULT_QUERY_LIMIT: i64 = 20; + +/// Upper bound for recursive-traversal `depth` arguments. The C++ engine +/// expands `depth` levels of callers/callees recursively, so an unbounded +/// value would exhaust the stack and abort the long-running MCP process. +/// Matches the "max 10" documented in the tool schemas. +pub const MAX_TRAVERSAL_DEPTH: i64 = 10; +/// Default `depth` for `trace_flow` when the client omits it. +pub const DEFAULT_TRACE_FLOW_DEPTH: i64 = 3; +/// Default `depth` for `codescope_trace` when the client omits it. +pub const DEFAULT_CODESCOPE_TRACE_DEPTH: i64 = 1; +/// Upper bound for `radius` on neighbourhood / subgraph queries. +pub const MAX_NEIGHBOR_RADIUS: i64 = 3; +/// Upper bound for `get_knowledge_graph` limit (matches the C++ clamp). +pub const MAX_KNOWLEDGE_GRAPH_LIMIT: i64 = 1000; +/// Lowest / highest valid `relation.type` values, per +/// plan/rules/relation_contract.md (0 = References … 7 = HasType). +/// A value <= 0 means "no type filter"; values above the maximum are +/// clamped so a truncated i64 can never select an out-of-contract type. +pub const MIN_EDGE_TYPE_FILTER: i64 = -1; +pub const MAX_EDGE_TYPE_FILTER: i64 = 7; + +/// Default / upper bound for `get_communities` `max_communities`. The engine +/// emits one JSON object per community, so an unbounded value would produce a +/// payload proportional to the graph size. +pub const DEFAULT_MAX_COMMUNITIES: i64 = 20; +pub const MAX_MAX_COMMUNITIES: i64 = 500; +/// Default / upper bound for `get_communities` `max_members`. Only used when +/// `include_members` is true. +pub const DEFAULT_MAX_MEMBERS: i64 = 10; +pub const MAX_MAX_MEMBERS: i64 = 200; + +/// Default / upper bound for `verify_integrity` `max_findings`. The +/// findings array scales with project size (one entry per orphan module, +/// function, and drift rule hit), so an unbounded value overflows the MCP +/// write cap on a large index (T5 finding #15). Matches the engine clamp +/// in engine_verify_ffi.cpp. +pub const DEFAULT_MAX_FINDINGS: i64 = 200; +pub const MAX_FINDINGS_LIMIT: i64 = 2000; + +/// Clamp a client-supplied recursion depth into `[1, MAX_TRAVERSAL_DEPTH]`. +/// The engine recurses `depth` levels, so an unclamped value (or the +/// truncation of a huge i64 by `as i32`) could exhaust the stack and abort +/// the long-running MCP process. +pub fn clamp_depth(value: Option, default: i64) -> i32 { + value.unwrap_or(default).clamp(1, MAX_TRAVERSAL_DEPTH) as i32 +} + +/// Clamp a client-supplied neighbourhood radius into +/// `[1, MAX_NEIGHBOR_RADIUS]`. +pub fn clamp_radius(value: Option) -> i32 { + value.unwrap_or(1).clamp(1, MAX_NEIGHBOR_RADIUS) as i32 +} + +/// Clamp a client-supplied `relation.type` filter into the contract range. +pub fn clamp_edge_type(value: Option) -> i32 { + value + .unwrap_or(MIN_EDGE_TYPE_FILTER) + .clamp(MIN_EDGE_TYPE_FILTER, MAX_EDGE_TYPE_FILTER) as i32 +} + +/// Clamp a client-supplied `get_knowledge_graph` row limit into +/// `[0, MAX_KNOWLEDGE_GRAPH_LIMIT]`. +pub fn clamp_knowledge_limit(value: Option) -> i32 { + value + .unwrap_or(MAX_QUERY_LIMIT) + .clamp(0, MAX_KNOWLEDGE_GRAPH_LIMIT) as i32 +} + +/// Clamp a client-supplied `get_communities` community count into +/// `[1, MAX_MAX_COMMUNITIES]`. (The engine also treats a non-positive value +/// as "use the default", but clamping here keeps a huge i64 from being +/// truncated by `as i32` into a negative value.) +pub fn clamp_max_communities(value: Option) -> i32 { + value + .unwrap_or(DEFAULT_MAX_COMMUNITIES) + .clamp(1, MAX_MAX_COMMUNITIES) as i32 +} + +/// Clamp a client-supplied `get_communities` member count into +/// `[1, MAX_MAX_MEMBERS]`. +pub fn clamp_max_members(value: Option) -> i32 { + value + .unwrap_or(DEFAULT_MAX_MEMBERS) + .clamp(1, MAX_MAX_MEMBERS) as i32 +} + +/// Clamp a client-supplied `verify_integrity` `max_findings` into +/// `[1, MAX_FINDINGS_LIMIT]`. The engine caps the serialized findings +/// array at this value and reports `truncated` + the real `total`. +pub fn clamp_findings_limit(value: Option) -> i32 { + value + .unwrap_or(DEFAULT_MAX_FINDINGS) + .clamp(1, MAX_FINDINGS_LIMIT) as i32 +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_clamp_depth_bounds() { + // Missing argument → per-caller default. + assert_eq!(clamp_depth(None, DEFAULT_TRACE_FLOW_DEPTH), 3); + assert_eq!(clamp_depth(None, DEFAULT_CODESCOPE_TRACE_DEPTH), 1); + // Below the minimum collapses to 1 (a zero/negative depth is + // meaningless for a recursive walk). + assert_eq!(clamp_depth(Some(0), 3), 1); + assert_eq!(clamp_depth(Some(-5), 3), 1); + assert_eq!(clamp_depth(Some(1), 3), 1); + // Inside the range is preserved. + assert_eq!(clamp_depth(Some(5), 3), 5); + assert_eq!(clamp_depth(Some(MAX_TRAVERSAL_DEPTH), 3), 10); + // Above the maximum is clamped, not truncated. + assert_eq!(clamp_depth(Some(11), 3), 10); + assert_eq!(clamp_depth(Some(i64::MAX), 3), 10); + // 5_000_000_000 as i32 would be 705_032_704 — the old code would + // have recursed that many levels. + assert_eq!(clamp_depth(Some(5_000_000_000), 3), 10); + } + + #[test] + fn test_clamp_radius_bounds() { + assert_eq!(clamp_radius(None), 1); + assert_eq!(clamp_radius(Some(0)), 1); + assert_eq!(clamp_radius(Some(-3)), 1); + assert_eq!(clamp_radius(Some(1)), 1); + assert_eq!(clamp_radius(Some(MAX_NEIGHBOR_RADIUS)), 3); + assert_eq!(clamp_radius(Some(4)), 3); + assert_eq!(clamp_radius(Some(i64::MAX)), 3); + } + + #[test] + fn test_clamp_edge_type_bounds() { + // Absent / <= 0 means "no type filter" (the SQL only filters on + // edge_type > 0). + assert_eq!(clamp_edge_type(None), MIN_EDGE_TYPE_FILTER as i32); + assert_eq!(clamp_edge_type(Some(-1)), -1); + assert_eq!(clamp_edge_type(Some(-100)), -1); + assert_eq!(clamp_edge_type(Some(0)), 0); + // Contract range is preserved. + assert_eq!(clamp_edge_type(Some(1)), 1); + assert_eq!(clamp_edge_type(Some(MAX_EDGE_TYPE_FILTER)), 7); + // Above the contract maximum is clamped to HasType, never a + // truncated/out-of-contract type. + assert_eq!(clamp_edge_type(Some(8)), 7); + assert_eq!(clamp_edge_type(Some(i64::MAX)), 7); + } + + #[test] + fn test_clamp_knowledge_limit_bounds() { + assert_eq!(clamp_knowledge_limit(None), MAX_QUERY_LIMIT as i32); + // 0 is allowed (callers treat it as "no rows"). + assert_eq!(clamp_knowledge_limit(Some(0)), 0); + assert_eq!(clamp_knowledge_limit(Some(-10)), 0); + assert_eq!(clamp_knowledge_limit(Some(1)), 1); + assert_eq!( + clamp_knowledge_limit(Some(MAX_KNOWLEDGE_GRAPH_LIMIT)), + MAX_KNOWLEDGE_GRAPH_LIMIT as i32 + ); + // 5_000_000_000 as i32 is negative — the old code passed that + // straight through to the C++ row limit. + assert_eq!( + clamp_knowledge_limit(Some(5_000_000_000)), + MAX_KNOWLEDGE_GRAPH_LIMIT as i32 + ); + } + + #[test] + fn test_clamp_max_communities_bounds() { + assert_eq!(clamp_max_communities(None), DEFAULT_MAX_COMMUNITIES as i32); + assert_eq!(clamp_max_communities(Some(0)), 1); + assert_eq!(clamp_max_communities(Some(-5)), 1); + assert_eq!(clamp_max_communities(Some(100)), 100); + assert_eq!( + clamp_max_communities(Some(MAX_MAX_COMMUNITIES)), + MAX_MAX_COMMUNITIES as i32 + ); + assert_eq!( + clamp_max_communities(Some(9_999)), + MAX_MAX_COMMUNITIES as i32 + ); + // 5_000_000_000 as i32 is negative — must not reach the engine. + assert_eq!( + clamp_max_communities(Some(5_000_000_000)), + MAX_MAX_COMMUNITIES as i32 + ); + } + + #[test] + fn test_clamp_max_members_bounds() { + assert_eq!(clamp_max_members(None), DEFAULT_MAX_MEMBERS as i32); + assert_eq!(clamp_max_members(Some(0)), 1); + assert_eq!(clamp_max_members(Some(-1)), 1); + assert_eq!(clamp_max_members(Some(50)), 50); + assert_eq!(clamp_max_members(Some(9_999)), MAX_MAX_MEMBERS as i32); + assert_eq!( + clamp_max_members(Some(5_000_000_000)), + MAX_MAX_MEMBERS as i32 + ); + } + + #[test] + fn test_clamp_findings_limit_bounds() { + assert_eq!(clamp_findings_limit(None), DEFAULT_MAX_FINDINGS as i32); + assert_eq!(clamp_findings_limit(Some(0)), 1); + assert_eq!(clamp_findings_limit(Some(-1)), 1); + assert_eq!(clamp_findings_limit(Some(50)), 50); + assert_eq!(clamp_findings_limit(Some(9_999)), MAX_FINDINGS_LIMIT as i32); + assert_eq!( + clamp_findings_limit(Some(5_000_000_000)), + MAX_FINDINGS_LIMIT as i32 + ); + } +} + +/// Property-based coverage of the clamping contract (REVIEW_0.2.7.md TEST-3 / +/// code_rules §4: "Property-based testing where appropriate"). +/// +/// The literals above pin a few interesting values; these properties hold for +/// EVERY `i64`. The class of bug they guard is the one the module exists for: +/// a client value large enough that `value as i32` truncates to a negative or +/// out-of-contract number, which then reaches the engine unclamped. +#[cfg(test)] +mod proptests { + use super::*; + use proptest::prelude::*; + + /// (name, clamp fn, minimum, maximum, default-for-absent-argument) + type ClampCase = (&'static str, fn(Option) -> i32, i64, i64, i64); + + proptest! { + #![proptest_config(ProptestConfig::with_cases(512))] + + /// `depth` takes a per-caller default, so it is exercised separately. + #[test] + fn clamp_depth_is_exactly_i64_clamp(value in any::(), default in any::()) { + let got = clamp_depth(Some(value), default); + prop_assert_eq!(got, value.clamp(1, MAX_TRAVERSAL_DEPTH) as i32); + prop_assert!((1..=MAX_TRAVERSAL_DEPTH as i32).contains(&got)); + // An absent argument is clamped too, not passed through raw. + prop_assert_eq!(clamp_depth(None, default), default.clamp(1, MAX_TRAVERSAL_DEPTH) as i32); + } + + /// Inside the contract range the value must be preserved exactly. + #[test] + fn clamp_depth_is_identity_inside_the_range(value in 1i64..=MAX_TRAVERSAL_DEPTH) { + prop_assert_eq!(clamp_depth(Some(value), 1), value as i32); + } + + /// Every fixed-range clamp: the result equals `i64::clamp` (i.e. no + /// `as i32` truncation), stays inside the documented bounds, and the + /// absent-argument path clamps its default rather than trusting it. + #[test] + fn fixed_range_clamps_never_truncate(value in any::()) { + let cases: [ClampCase; 6] = [ + ("radius", clamp_radius, 1, MAX_NEIGHBOR_RADIUS, 1), + ("edge_type", clamp_edge_type, MIN_EDGE_TYPE_FILTER, MAX_EDGE_TYPE_FILTER, MIN_EDGE_TYPE_FILTER), + ("knowledge_limit", clamp_knowledge_limit, 0, MAX_KNOWLEDGE_GRAPH_LIMIT, MAX_QUERY_LIMIT), + ("max_communities", clamp_max_communities, 1, MAX_MAX_COMMUNITIES, DEFAULT_MAX_COMMUNITIES), + ("max_members", clamp_max_members, 1, MAX_MAX_MEMBERS, DEFAULT_MAX_MEMBERS), + ("findings_limit", clamp_findings_limit, 1, MAX_FINDINGS_LIMIT, DEFAULT_MAX_FINDINGS), + ]; + for (name, clamp_fn, lo, hi, default) in cases { + let got = clamp_fn(Some(value)); + prop_assert_eq!( + got, + value.clamp(lo, hi) as i32, + "{}: input {} did not clamp to i64::clamp", name, value + ); + prop_assert!( + (lo as i32..=hi as i32).contains(&got), + "{}: {} is outside [{}, {}]", name, got, lo, hi + ); + prop_assert_eq!( + clamp_fn(None), + default.clamp(lo, hi) as i32, + "{}: absent-argument default was not clamped", name + ); + } + } + } +} diff --git a/server/src/tools/discover.rs b/server/src/tools/discover.rs new file mode 100644 index 0000000..dbfcaaf --- /dev/null +++ b/server/src/tools/discover.rs @@ -0,0 +1,178 @@ +// discover.rs — quick project scan. +// +// Split out of tools/mod.rs (see plan/rules/code_rules.md 1000-line rule). +// Walks the directory tree and counts candidate source files per top-level +// directory, using the same filtering rules as the C++ engine's +// FilterPolicy. No parsing, no SQLite — just a fast filesystem walk, which +// is why it carries no dependency on the handler table. Source-file +// detection delegates to the engine (see is_source_file below). + +use serde_json::json; + +/// ── Discover: quick project scan ──────────────────────────────── +/// Walks the directory tree, counts candidate source files per top-level +/// directory (same filtering rules as the C++ engine's FilterPolicy). +/// No parsing, no SQLite — just a fast filesystem walk. +/// Output: JSON with per-directory file counts + total. +pub fn discover(dir_path: &str) -> String { + use std::collections::BTreeMap; + use std::path::Path; + + // Built-in skip dirs matching FilterPolicy's normal_skip_dirs_ + // Applied to EVERY path component (any depth). + fn is_skip_dir(name: &str) -> bool { + let lower = name.to_lowercase(); + matches!( + lower.as_str(), + ".git" + | ".svn" + | ".hg" + | ".bzr" + | "node_modules" + | ".venv" + | "venv" + | "env" + | ".env" + | "__pycache__" + | ".pytest_cache" + | ".mypy_cache" + | ".cache" + | ".codescope" + | "target" + | "build" + | "dist" + | ".next" + | ".turbo" + | ".direnv" + | "vendor" + | "third_party" + | "third-party" + | ".bundle" + | ".gem" + | "go_pkg" + | "pkg" + ) + } + + // Top-only skip dirs matching FilterPolicy's top_only_skip_dirs_ + // Source-bearing dirs that are rarely the focus of analysis + // (test/, docs/, bench/, examples/, ...). Matched ONLY against the + // first path component (top-level dirs), so Java packages like + // org/springframework/samples/petclinic are NOT falsely skipped. + fn is_top_only_skip_dir(name: &str) -> bool { + let lower = name.to_lowercase(); + matches!( + lower.as_str(), + "test" + | "tests" + | "docs" + | "doc" + | "documentation" + | "examples" + | "example" + | "samples" + | "sample" + | "scripts" + | "hack" + | "migrations" + | "seeds" + | "e2e" + | "integration" + | "locale" + | "locales" + | "i18n" + | "l10n" + | "assets" + | "static" + | "public" + | "media" + | "external" + | "vendored" + | "bench" + | "benchmarks" + ) + } + + fn is_skip_prefix(name: &str) -> bool { + name.starts_with('.') // hidden dirs/files + } + + // Countable source extensions, decided by the engine (FilterPolicy / + // engine_is_indexable_source). A hand-written list drifted from engine + // support before (.zig/.wasm/.mojo were counted but never parsed). + fn is_source_file(name: &str) -> bool { + crate::ffi::is_indexable_source(name) + } + + let root = Path::new(dir_path); + if !root.is_dir() { + return json!({ + "ok": false, + "error": format!("directory not found: {}", dir_path) + }) + .to_string(); + } + + let mut modules: BTreeMap = BTreeMap::new(); + let mut total_files: u64 = 0; + let mut total_dirs: u64 = 0; + let mut skipped_dirs: u64 = 0; + let mut skipped_files: u64 = 0; + + // Walk top-level entries first + if let Ok(entries) = std::fs::read_dir(root) { + for entry in entries.flatten() { + let name = entry.file_name(); + let name_str = name.to_string_lossy().to_string(); + let path = entry.path(); + + if path.is_dir() { + let base = name_str.clone(); + if is_skip_dir(&base) || is_top_only_skip_dir(&base) || is_skip_prefix(&base) { + skipped_dirs += 1; + continue; + } + total_dirs += 1; + + // Recursively count source files in this directory + let mut count: u64 = 0; + let walk = walkdir::WalkDir::new(&path).into_iter().filter_entry(|e| { + let fname = e.file_name().to_string_lossy(); + if e.depth() == 0 { + return true; + } + if e.file_type().is_dir() { + !is_skip_dir(&fname) && !is_skip_prefix(&fname) + } else { + true + } + }); + for entry in walk.flatten() { + if entry.file_type().is_file() { + let fname = entry.file_name().to_string_lossy(); + if is_source_file(&fname) { + count += 1; + } else { + skipped_files += 1; + } + } + } + if count > 0 { + modules.insert(base, count); + total_files += count; + } + } + } + } + + json!({ + "ok": true, + "project_path": dir_path, + "total_dirs": total_dirs, + "total_files": total_files, + "skipped_dirs": skipped_dirs, + "skipped_files": skipped_files, + "modules": modules + }) + .to_string() +} diff --git a/server/src/tools/docs_lint.rs b/server/src/tools/docs_lint.rs new file mode 100644 index 0000000..21304a6 --- /dev/null +++ b/server/src/tools/docs_lint.rs @@ -0,0 +1,146 @@ +// docs_lint.rs — documentation-liveness guards for the tool layer. +// +// Split out of tools/mod.rs to stay under the 1000-line rule (see +// plan/rules/code_rules.md); the tests it holds only need `std`, so they live +// here on their own. +// +// The guards exist because CodeScope's own README once promised things the code +// did not do. `CODESCOPE_VERBOSE` was documented in README §9 but never read by +// any production code, so setting it did nothing +// (2026-09-27 review, D3-3). These tests make that drift fail the build. + +#[cfg(test)] +mod tests { + /// Append the contents of every source file under `dir` to `out`. + fn collect_sources(dir: &std::path::Path, out: &mut String) { + let Ok(entries) = std::fs::read_dir(dir) else { + return; + }; + for entry in entries.flatten() { + let path = entry.path(); + if path.is_dir() { + collect_sources(&path, out); + continue; + } + let is_source = matches!( + path.extension().and_then(|e| e.to_str()), + Some("rs" | "cpp" | "h" | "hpp") + ); + if !is_source { + continue; + } + if let Ok(text) = std::fs::read_to_string(&path) { + out.push_str(&text); + out.push('\n'); + } + } + } + + /// True when `sources` contains a statement that READS `var` — a + /// `getenv("VAR")` / `env::var("VAR")` call, or a `getenv()` where + /// `` is the constant holding the variable name + /// (`kExcludePathsEnv = "CODESCOPE_EXCLUDE_PATHS"`). + /// + /// A set-only or comment-only mention deliberately does not count: that is + /// exactly the `CODESCOPE_VERBOSE` failure mode this guards against (the + /// variable was passed to workers but never read by production code). + fn has_read_site(sources: &str, var: &str) -> bool { + let quoted = format!("\"{var}\""); + for line in sources.lines() { + if !line.contains("ed) { + continue; + } + if line.contains("getenv") || line.contains("env::var") { + return true; + } + // `static constexpr const char *kFooEnv = "CODESCOPE_FOO";` + // The name is only indirect: look for a getenv() of that constant. + // The constant is the last identifier before the `=`. + let const_name = line + .split("ed) + .next() + .unwrap_or("") + .rsplit([' ', '*', '(', ',', '=', '\t', ';']) + .find(|token| { + !token.is_empty() && token.chars().all(|c| c.is_alphanumeric() || c == '_') + }); + if let Some(name) = const_name + && sources.contains(&format!("getenv({name})")) + { + return true; + } + } + false + } + + /// Regression (2026-09-27 review, D3-3): `CODESCOPE_VERBOSE` was + /// documented in README §9 but never read by any production code — setting + /// it did nothing. Every variable the §9 table promises must have a real + /// READER in `server/src` or `engine/src`, so a dead entry cannot creep + /// back in. A set-only or comment-only mention deliberately does not count. + #[test] + fn test_documented_env_vars_are_read_by_production_code() { + // Resolve every path from the compile-time crate directory so the test + // does not depend on the working directory it is launched from. + let crate_dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")); + let readme_path = crate_dir.join("../README.md"); + let readme = std::fs::read_to_string(&readme_path).unwrap_or_else(|e| { + panic!("cannot read {}: {e}", readme_path.display()); + }); + let section = readme + .split("## 9. Environment Variables") + .nth(1) + .expect("README must have a §9 Environment Variables section"); + let section = section.split("\n## ").next().unwrap_or(section); + + let mut vars: Vec = Vec::new(); + for line in section.lines() { + let Some(rest) = line.strip_prefix("| `") else { + continue; + }; + let Some(name) = rest.split('`').next() else { + continue; + }; + if name.starts_with("CODESCOPE_") { + vars.push(name.to_string()); + } + } + assert!( + vars.len() >= 9, + "expected the §9 table rows, parsed only {vars:?}" + ); + + let mut sources = String::new(); + collect_sources(&crate_dir.join("src"), &mut sources); + collect_sources(&crate_dir.join("../engine/src"), &mut sources); + + for var in &vars { + assert!( + has_read_site(&sources, var), + "README §9 documents `{var}` but no production code READS it \ + (a set-only or comment-only reference does not count)" + ); + } + } + + /// The liveness check must reject a variable that is only ever SET or + /// mentioned in a comment — the `CODESCOPE_VERBOSE` shape. + #[test] + fn test_env_var_liveness_rejects_set_only_variables() { + let sources = "\ + cmd.env(\"CODESCOPE_DEAD_VAR\", \"0\");\n\ + // CODESCOPE_DEAD_VAR is documented but never read\n\ + let _ = std::env::var(\"CODESCOPE_LIVE_VAR\").ok();\n\ + static constexpr const char *kIndirectEnv = \"CODESCOPE_INDIRECT\";\n\ + const char *p = getenv(kIndirectEnv);\n"; + assert!( + !has_read_site(sources, "CODESCOPE_DEAD_VAR"), + "a set-only variable must not count as read" + ); + assert!(has_read_site(sources, "CODESCOPE_LIVE_VAR")); + assert!( + has_read_site(sources, "CODESCOPE_INDIRECT"), + "an indirect getenv(const) read must count" + ); + } +} diff --git a/server/src/tools/indexing.rs b/server/src/tools/indexing.rs new file mode 100644 index 0000000..2867e59 --- /dev/null +++ b/server/src/tools/indexing.rs @@ -0,0 +1,846 @@ +// indexing.rs — indexing orchestration. +// +// Split out of tools/mod.rs (see plan/rules/code_rules.md 1000-line rule). +// Contains the worker subprocess runner (with timeout + orphan cleanup), +// the incremental file index entry point, the force-index file walker and +// its acceptability filter. These are kept together because they are one +// pipeline: `h_force_index_files` decides WHICH files to index and +// delegates the actual indexing to the same worker process that +// `h_index_file` uses for a single file. + +use crate::ffi; +use once_cell::sync::Lazy; +use serde_json::Value; +use std::process::{Command, Stdio}; +use std::sync::mpsc; +use std::time::Duration; + +// ─── Worker Supervisor (timeout + retry) ─────────────────────── +/// Default worker timeout in seconds when `CODESCOPE_WORKER_TIMEOUT` is unset. +const DEFAULT_WORKER_TIMEOUT_SECS: u64 = 300; +/// Interval between polls when waiting for a worker subprocess to finish. +const WORKER_POLL_INTERVAL: Duration = Duration::from_millis(100); + +static WORKER_TIMEOUT: Lazy = Lazy::new(|| { + Duration::from_secs( + std::env::var("CODESCOPE_WORKER_TIMEOUT") + .ok() + .and_then(|s| s.parse().ok()) + .unwrap_or(DEFAULT_WORKER_TIMEOUT_SECS), + ) +}); +const MAX_RETRIES: usize = 3; + +/// Number of attempts to re-initialise the C++ engine after a worker +/// run if the first `ffi::init` fails. The worker subprocess may hold +/// the SQLite WAL lock briefly or leave the DB in a busy state; a +/// short retry window recovers the engine instead of leaving the +/// server in an unusable "g_store == null" state where every +/// subsequent tool call returns "not initialised". See H4. +const ENGINE_INIT_MAX_ATTEMPTS: usize = 3; +/// Delay between engine re-init attempts. Chosen to be long enough +/// for the worker's WAL lock to release on a busy system but short +/// enough not to materially delay the index response. +const ENGINE_INIT_RETRY_DELAY: Duration = Duration::from_millis(500); + +/// Run a worker subprocess with timeout protection. +/// Returns `Ok(output)` on success, `Err(msg)` on timeout or failure. +/// On timeout the orphaned child is killed using a platform-appropriate +/// method: `kill -9` on Unix, `taskkill /F` on Windows. On any other +/// platform the process is logged as orphaned (the `Child` handle was moved +/// into the wait thread and is no longer accessible here). +fn run_worker( + exe: &str, + args: &[&str], + envs: &[(&str, &str)], +) -> Result { + let mut cmd = Command::new(exe); + cmd.args(args); + for (k, v) in envs { + cmd.env(k, v); + } + // Pipe stdout/stderr so the worker's output is captured instead of + // leaking into the MCP server's stdout transport stream. + cmd.stdout(Stdio::piped()).stderr(Stdio::piped()); + let child = cmd.spawn().map_err(|e| format!("spawn failed: {}", e))?; + let pid = child.id(); + + let (tx, rx) = mpsc::channel(); + + // Thread: wait for child completion + let tx_out = tx.clone(); + std::thread::spawn(move || { + let output = child.wait_with_output(); + let _ = tx_out.send(output); + }); + + // Poll for result with timeout + let start = std::time::Instant::now(); + loop { + if start.elapsed() > *WORKER_TIMEOUT { + // Kill orphaned child via a platform-appropriate method. The child + // handle was moved into the wait thread above, so we cannot call + // `child.kill()` here; instead we signal by PID. + #[cfg(unix)] + { + let _ = Command::new("kill").args(["-9", &pid.to_string()]).output(); + } + #[cfg(windows)] + { + let _ = Command::new("taskkill") + .args(["/F", "/PID", &pid.to_string()]) + .output(); + } + #[cfg(not(any(unix, windows)))] + { + eprintln!( + "warning: worker timeout on unsupported platform — process {} may be orphaned", + pid + ); + } + return Err(format!( + "worker timed out after {}s", + WORKER_TIMEOUT.as_secs() + )); + } + + match rx.try_recv() { + Ok(Ok(output)) => return Ok(output), + Ok(Err(e)) => return Err(format!("worker error: {}", e)), + Err(mpsc::TryRecvError::Empty) => { + std::thread::sleep(WORKER_POLL_INTERVAL); + } + Err(mpsc::TryRecvError::Disconnected) => { + return Err("worker channel disconnected".to_string()); + } + } + } +} + +/// Internal worker-subprocess indexer used by the MCP session auto-index +/// path. NOT registered as a public tool: index-parallel (with keep_db +/// incremental) fully replaces the serial index_project tool, so the +/// MCP tool list no longer exposes it (47→46 tools). The engine and the +/// worker subprocess entry point are shared with index-parallel and stay. +pub fn index_project_via_worker(project_id: u64, args: &Value) -> String { + let path = args["project_path"].as_str().unwrap_or(""); + + // Use worker subprocess for memory isolation + let self_exe = std::env::current_exe().ok(); + if let Some(exe) = self_exe { + let db_path = crate::tools::db_path() + .map(str::to_string) + .unwrap_or_else(|| ".codescope/codescope.db".to_string()); + let grammars_dir = std::env::var("GRAMMARS_DIR").unwrap_or_else(|_| "grammars".to_string()); + let lang = args["language_filter"].as_str().unwrap_or(""); + + // Derive a meaningful project name from the path's final component so the + // worker records a human-readable name instead of the placeholder + // "worker-project". + let project_name = std::path::Path::new(path) + .file_name() + .and_then(|n| n.to_str()) + .unwrap_or("unnamed") + .to_string(); + + for attempt in 1..=MAX_RETRIES { + // Shutdown engine before spawning worker to release SQLite lock + crate::ffi::shutdown(); + + let args_list = [ + "worker", + &db_path, + path, + lang, + &project_name, + &project_id.to_string(), + ]; + let envs = [ + ("GRAMMARS_DIR", &grammars_dir as &str), + ("CODESCOPE_DB_PATH", &db_path as &str), + ("CODESCOPE_VERBOSE", "0"), + ]; + + let result = run_worker(exe.to_str().unwrap_or("codescope"), &args_list, &envs); + + // Re-init engine after worker completes (regardless of success/failure). + // The worker subprocess may have left the SQLite WAL lock held briefly + // or the DB may be corrupted; checking the return value prevents the + // server from silently running with a null g_store for the rest of its + // lifetime (every subsequent tool call would return "not initialized"). + // + // H4: A single failed init must NOT leave the server in an unusable + // state. We retry with the original (pre-shutdown) db_path a few + // times with a short delay — this recovers the engine when the + // failure was transient (e.g. WAL lock not yet released). Only if + // every attempt fails do we propagate the error; the caller (and + // the operator) must restart the server in that case. + let mut last_init_code: i32 = 0; + let mut engine_recovered = false; + for init_attempt in 1..=ENGINE_INIT_MAX_ATTEMPTS { + let code = crate::ffi::init(&db_path); + if code == 0 { + engine_recovered = true; + break; + } + last_init_code = code; + if init_attempt < ENGINE_INIT_MAX_ATTEMPTS { + eprintln!( + "engine re-init attempt {}/{} failed (code={}); retrying in {:?} [module=mcp, tool=index_project, method=ffi::init]", + init_attempt, ENGINE_INIT_MAX_ATTEMPTS, code, ENGINE_INIT_RETRY_DELAY + ); + std::thread::sleep(ENGINE_INIT_RETRY_DELAY); + } + } + if !engine_recovered { + return format!( + "{{\"ok\":false,\"error\":\"engine re-initialization failed after index (code={}, attempts={}). The database may be locked or corrupted; the engine is now uninitialized — restart the server before issuing further tool calls. [module=mcp, tool=index_project, method=ffi::init]\"}}", + last_init_code, ENGINE_INIT_MAX_ATTEMPTS + ); + } + + match result { + Ok(out) => { + if out.status.success() { + let stdout = String::from_utf8_lossy(&out.stdout); + // Trigger background FTS build after a successful index. + // Uses spawn_fts_build to deduplicate concurrent builds + // and avoid a data race on the global C++ g_store. + if let Some(json_start) = stdout.find('{') + && let Some(json_end) = stdout[json_start..].rfind('}') + { + let candidate = &stdout[json_start..=json_start + json_end]; + // Validate the slice is well-formed JSON before returning + // it, so malformed worker output does not produce invalid + // JSON that would confuse the MCP client. + if serde_json::from_str::(candidate).is_ok() { + crate::ffi::spawn_fts_build(project_id); + return candidate.to_string(); + } + } + crate::ffi::spawn_fts_build(project_id); + return stdout.to_string(); + } + let stderr = String::from_utf8_lossy(&out.stderr); + let err_msg = stderr.lines().last().unwrap_or("unknown"); + if attempt < MAX_RETRIES { + eprintln!( + "worker attempt {} failed ({}), retrying...", + attempt, err_msg + ); + std::thread::sleep(Duration::from_secs(1)); + continue; + } + return format!( + "{{\"ok\":false,\"error\":\"worker failed after {} attempts: {}\"}}", + MAX_RETRIES, err_msg + ); + } + Err(msg) => { + if attempt < MAX_RETRIES { + eprintln!("worker attempt {}: {} — retrying...", attempt, msg); + std::thread::sleep(Duration::from_secs(1)); + continue; + } + return format!( + "{{\"ok\":false,\"error\":\"worker {} after {} attempts\"}}", + msg, MAX_RETRIES + ); + } + } + } + } + + // Fallback: in-process indexing with correct null handling for missing lang filter + // Fix: Use CString to ensure null-termination for FFI compatibility + let lang = args["language_filter"].as_str(); + let lang_cstring = lang.map(|s| std::ffi::CString::new(s).unwrap_or_default()); + let lang_ptr = lang_cstring + .as_ref() + .map_or(std::ptr::null(), |cs| cs.as_ptr() as *const _); + ffi::index_project(project_id, path, lang_ptr) +} + +pub(super) fn h_index_file(project_id: u64, args: &Value) -> String { + let path = args["file_path"].as_str().unwrap_or(""); + ffi::index_file(project_id, path) +} + +/// Force-index specific files or directories, bypassing FilterPolicy's +/// default skip rules (test/, docs/, vendored/, node_modules/, etc.). +/// +/// Use case: user says "go index xxx/yyy for me" — the AI calls this +/// tool with paths=[...]. Files under the given paths are indexed +/// regardless of the default skip list, so the user can pull in +/// test fixtures, vendored deps, or generated code on demand. Nested +/// compiler/package output trees are still skipped at any depth +/// (node_modules/, target/, _deps/, __pycache__/, venvs, .git/, and +/// build*/ dirs carrying build-system artifacts — CMake markers for any +/// build*, Gradle leaves only for a dir named `build`) — name such a +/// directory itself in paths to index its contents anyway. +/// +/// Args: +/// paths: array of absolute file/dir paths to force-index. +/// language_filter: optional comma-separated language whitelist +/// (e.g. "java,python"). When empty, all detectable languages +/// are indexed. +/// +/// Returns the engine_index_files JSON result +/// (files_indexed/nodes/edges/errors). +pub(super) fn h_force_index_files(project_id: u64, args: &Value) -> String { + // Collect paths: accept either `paths: [...]` or legacy + // `path: "..."` for single-path convenience. + let mut paths: Vec = Vec::new(); + if let Some(arr) = args["paths"].as_array() { + for v in arr { + if let Some(s) = v.as_str() + && !s.is_empty() + { + paths.push(s.to_string()); + } + } + } + if let Some(s) = args["path"].as_str() + && !s.is_empty() + { + paths.push(s.to_string()); + } + if paths.is_empty() { + return "{\"ok\":false,\"error\":\"paths is required (array of file/dir paths)\"}" + .to_string(); + } + + let lang_filter = args["language_filter"].as_str().unwrap_or(""); + + // Expand directories into individual file paths, bypassing + // FilterPolicy's shouldSkipEntry. We DO still respect: + // - file size limit (CODESCOPE_MAX_FILE_SIZE, default 5MB) + // - language detectability (detectLanguage must return non-null) + // - optional language_filter whitelist + // We DO NOT respect: + // - normal_skip_dirs_ / top_only_skip_dirs_ (test/, docs/, ...) + // - skip_suffixes_ for source extensions + // - .gitignore / .codescopeignore + // This is the "user override" path. + let max_size: u64 = std::env::var("CODESCOPE_MAX_FILE_SIZE") + .ok() + .and_then(|s| s.parse().ok()) + .unwrap_or(5 * 1024 * 1024); + + let mut lang_whitelist: Option> = None; + if !lang_filter.is_empty() { + let mut set = std::collections::HashSet::new(); + for part in lang_filter.split(',') { + let p = part.trim().to_lowercase(); + if !p.is_empty() { + set.insert(p); + } + } + lang_whitelist = Some(set); + } + + let mut all_files: Vec = Vec::new(); + let mut skipped_files = 0u64; + let mut skipped_dirs = 0u64; + + for p in &paths { + let path = std::path::Path::new(p); + if !path.exists() { + skipped_files += 1; + continue; + } + if path.is_file() { + // Single file — index directly if detectable. + if let Some(fp) = filter_acceptable_file(path, max_size, lang_whitelist.as_ref()) { + all_files.push(fp); + } else { + skipped_files += 1; + } + continue; + } + // Directory — walk it, bypassing skip-dir rules but still + // respecting file-level detectability + size. Start at depth 0. + walk_force_index( + path, + max_size, + lang_whitelist.as_ref(), + &mut all_files, + &mut skipped_files, + &mut skipped_dirs, + 0, + ); + } + + if all_files.is_empty() { + // Not a silent success: the caller asked to index and nothing was + // accepted — say so instead of reporting ok with zero files. + return format!( + "{{\"ok\":false,\"error\":\"no indexable source files accepted under the requested paths [module=mcp, method=h_force_index_files]\",\"files_indexed\":0,\"nodes\":0,\"edges\":0,\"errors\":0,\"skipped_files\":{},\"skipped_dirs\":{}}}", + skipped_files, skipped_dirs + ); + } + + // Build JSON file list and call engine_index_files. + let json_list = serde_json::Value::Array( + all_files + .iter() + .map(|s| serde_json::Value::String(s.clone())) + .collect(), + ) + .to_string(); + + // bypass_fail_fast = true: this tool promises to index the requested paths + // "regardless of the default skip rules", so a file that failed before is + // re-attempted instead of being dropped silently (the scheduler-driven + // worker callers pass false and keep the fail-fast skip). + let result = ffi::index_files(project_id, &json_list, true); + + // Annotate result with skip stats for transparency. + if let Ok(mut v) = serde_json::from_str::(&result) { + if let Some(obj) = v.as_object_mut() { + obj.insert("skipped_files".into(), skipped_files.into()); + obj.insert("skipped_dirs".into(), skipped_dirs.into()); + obj.insert("paths_requested".into(), paths.len().into()); + // The engine reports `ok: true` for a run that accepted files but + // indexed none — every parse failed (grammar unavailable for that + // language, unreadable file) — because "the call itself worked". + // The CLI's force-index already treats that as a failed run (it + // exits non-zero on `files_indexed == 0`); over MCP the envelope + // was the only signal, so a client saw `isError` absent and + // concluded the index was built. Report it here instead: the run + // produced no graph, so the tool call did not do what was asked. + if obj.get("ok") == Some(&serde_json::Value::Bool(true)) + && obj.get("files_indexed").and_then(|n| n.as_u64()) == Some(0) + { + obj.insert("ok".into(), serde_json::Value::Bool(false)); + obj.insert( + "error".into(), + serde_json::Value::String(format!( + "no files were indexed: {} candidate file(s) were accepted but none \ + produced symbols (unsupported language, parse failure, or empty files) \ + [module=mcp, method=h_force_index_files]", + obj.get("discovery") + .and_then(|d| d.get("candidate_files")) + .and_then(|n| n.as_u64()) + .unwrap_or(0), + )), + ); + } + } + return v.to_string(); + } + result +} + +/// Maximum recursion depth for `walk_force_index`. A deeply nested directory +/// tree (e.g. a chain of node_modules) or a crafted path would otherwise +/// consume the call stack until it overflows, panicking and crashing the +/// MCP server (local DoS, see H-B). 256 levels is far deeper than any +/// legitimate project tree; once reached we stop descending and log a +/// warning so the walk always terminates. +const MAX_WALK_DEPTH: u32 = 256; + +/// Whether force-index should accept this file name/extension. +/// +/// Delegates to the engine (`engine_is_indexable_source`) so the walk cannot +/// accept a language the indexer cannot parse, and so extensionless shebang +/// scripts the engine accepts are not dropped by a stale local extension list. +fn is_source_extension(name: &str) -> bool { + crate::ffi::is_indexable_source(name) +} + +/// Alias-folded language whitelist match. `c`/`cpp`/`c++` and the other +/// label pairs below form one family because the two layers have +/// historically used different labels for the same language (short forms +/// like `js`/`py` vs the engine's `javascript`/`python`) — exact-string +/// matching dropped whole languages whenever the labels differed. +fn lang_whitelisted(wl: &std::collections::HashSet, lang: &str) -> bool { + let alias_hit = |fam: &[&str]| fam.iter().any(|a| wl.contains(*a)); + match lang { + "c" | "cpp" | "c++" => alias_hit(&["c", "cpp", "c++"]), + "js" | "javascript" | "node" => alias_hit(&["js", "javascript", "node"]), + "ts" | "typescript" => alias_hit(&["ts", "typescript"]), + "py" | "python" => alias_hit(&["py", "python"]), + "go" | "golang" => alias_hit(&["go", "golang"]), + "kt" | "kotlin" => alias_hit(&["kt", "kts", "kotlin"]), + "rb" | "ruby" => alias_hit(&["rb", "ruby"]), + "rs" | "rust" => alias_hit(&["rs", "rust"]), + other => wl.contains(other), + } +} + +/// Check a single file path against force-index rules: +/// - must exist and be a regular file +/// - must be within max_size +/// - extension must be a recognised source extension +/// - must pass the optional language whitelist +/// +/// Returns the absolute path string if acceptable, None otherwise. +fn filter_acceptable_file( + path: &std::path::Path, + max_size: u64, + lang_whitelist: Option<&std::collections::HashSet>, +) -> Option { + let meta = match std::fs::metadata(path) { + Ok(m) => m, + Err(_) => return None, + }; + if !meta.is_file() { + return None; + } + if meta.len() > max_size { + return None; + } + + // Extension / name check via the engine (authoritative source list). + let file_name = path.file_name().and_then(|n| n.to_str())?; + if !is_source_extension(file_name) { + return None; + } + + // Language whitelist (maps extension -> language label) + if let Some(wl) = lang_whitelist { + let ext = path + .extension() + .and_then(|e| e.to_str()) + .map(|e| e.to_lowercase()) + .unwrap_or_default(); + let lang = match format!(".{}", ext).as_str() { + ".py" => "python", + ".cpp" | ".cc" | ".cxx" | ".h" | ".hpp" | ".hxx" | ".hh" => "cpp", + ".c" => "c", + ".rs" => "rust", + ".swift" => "swift", + ".js" | ".mjs" | ".cjs" => "javascript", + ".ts" => "typescript", + ".tsx" => "tsx", + ".go" => "go", + ".java" => "java", + ".kt" | ".kts" => "kotlin", + ".rb" => "ruby", + ".scala" => "scala", + _ => "", + }; + // Alias-fold before matching so short filter forms (`c`, `js`, + // `py`, …) accept the canonical label the map above emits. + if !lang.is_empty() && !lang_whitelisted(wl, lang) { + return None; + } + } + + // Return absolute path + let canon = std::fs::canonicalize(path).unwrap_or_else(|_| path.to_path_buf()); + Some(canon.to_string_lossy().to_string()) +} + +/// Build/package output classifier for the force-index walk. +/// +/// Unambiguous output/cache names are skipped outright. `build`/`build-*` +/// names are ambiguous — `build-tools/`, `build-scripts/`, `build-support/` +/// hold real source — so they are skipped only when the directory carries +/// build-system artifacts: CMake markers count for any `build*`, while the +/// Gradle leaves (classes/libs/generated/tmp) count only for the exact +/// name `build`. +/// +/// The list is the force-index subset of the engine's `FilterPolicy` +/// skip-dirs (`engine/src/filter_policy.cpp`, the source of truth for the +/// default path). It used to be a much shorter hand-written list, which is +/// how `dist/` slipped through: forcing an index of a Vite project walked the +/// minified bundle and produced a graph of ~700 hashed names (`JF`, `k`, `n0`) +/// next to 224 real TSX symbols, with community labels to match. Only the +/// genuinely ambiguous names are left to `FilterPolicy`: `obj`, `Debug`, +/// `Release`, `env`, `temp`/`tmp` and the source-level carve-outs +/// (`test/`, `docs/`, `plan/`, CI configs) that force-index exists to bypass. +fn is_build_output_dir(path: &std::path::Path) -> bool { + let lower = match path.file_name().and_then(|n| n.to_str()) { + Some(n) => n.to_lowercase(), + None => return false, + }; + if matches!( + lower.as_str(), + // JS/TS bundler output — generated by name, never hand-written. + "dist" + | ".dist" + | "out" + | ".out" + | ".next" + | ".nuxt" + | ".svelte-kit" + | ".angular" + | ".turbo" + | ".parcel-cache" + | ".docusaurus" + | ".expo" + | ".nx" + // Package caches + | "node_modules" + | "bower_components" + | "jspm_packages" + | ".npm" + | ".yarn" + | ".pnpm-store" + | ".pnp" + | ".deno" + | ".bun" + | ".nyc_output" + // Compiler / build trees + | "target" + | "_deps" + | ".build" + | "_build" + | "cmakefiles" + | "cmake-build-debug" + | "cmake-build-release" + | "bazel-bin" + | "bazel-out" + | "bazel-testlogs" + | ".buck-out" + | ".gradle" + | ".mvn" + | "_cpack_packages" + | "pods" + | "carthage" + | "deriveddata" + // Python environments + | "__pycache__" + | ".venv" + | "venv" + | "site-packages" + | ".virtualenv" + | ".virtualenvs" + | ".tox" + | ".mypy_cache" + | ".pytest_cache" + // Coverage, IaC state, VCS and this tool's own data dir + | "coverage" + | "htmlcov" + | ".terraform" + | ".terragrunt-cache" + | ".serverless" + | ".vercel" + | ".netlify" + | ".git" + | ".codescope" + ) { + return true; + } + if lower == "build" || lower.starts_with("build-") { + // CMake leaves CMakeCache.txt / CMakeFiles / compile_commands.json + // behind; Gradle leaves classes / libs / generated / tmp — but the + // Gradle build dir is always exactly `build`, while `build-*` is the + // CMake build-release style. Restrict the common Gradle leaf names to + // the exact name so source trees like build-tools/libs/ (real + // source) are not swallowed by a directory name heuristic. + if path.join("CMakeCache.txt").exists() + || path.join("CMakeFiles").is_dir() + || path.join("compile_commands.json").exists() + { + return true; + } + if lower == "build" { + return path.join("classes").is_dir() + || path.join("libs").is_dir() + || path.join("generated").is_dir() + || path.join("tmp").is_dir(); + } + return false; + } + false +} + +/// Recursively walk `root` for force-index. Bypasses all skip-dir +/// rules (test/, docs/, vendored/, node_modules/, etc.) — this is +/// the whole point of the force-index tool. Still respects: +/// - file size limit +/// - extension detectability +/// - optional language whitelist +/// +/// `depth` is the current recursion depth (call with 0 at the top). +/// Recursion stops once `depth >= MAX_WALK_DEPTH` to guarantee the walk +/// always terminates and cannot exhaust the stack (H-B). When the limit +/// is hit, a warning is logged and descent into that subtree is skipped. +fn walk_force_index( + root: &std::path::Path, + max_size: u64, + lang_whitelist: Option<&std::collections::HashSet>, + out_files: &mut Vec, + skipped_files: &mut u64, + skipped_dirs: &mut u64, + depth: u32, +) { + let entries = match std::fs::read_dir(root) { + Ok(e) => e, + Err(_) => { + *skipped_dirs += 1; + return; + } + }; + for entry in entries.flatten() { + let path = entry.path(); + // Use symlink_metadata (NOT following symlinks) to detect the + // entry's own type. Path::is_dir() follows symlinks, so a symlink + // loop (a -> b -> a) would make is_dir() always return true and + // the recursion would never terminate → stack overflow panic + // crashing the MCP server (local DoS). symlink_metadata gives us + // the link's own metadata without dereferencing, so we only recurse + // into real directories. + let is_real_dir = std::fs::symlink_metadata(&path) + .map(|m| m.is_dir()) + .unwrap_or(false); + if is_real_dir { + // Hard-skip compiler/package output trees even in force-index + // mode: they are generated artifacts, not user source, and + // indexing them explodes homonym counts (CMake probe `main`s, + // vendored tree-sitter scanners). The source-level carve-outs + // (test/, docs/, vendored/) stay bypassed. + if is_build_output_dir(&path) { + *skipped_dirs += 1; + continue; + } + // Guard against unbounded recursion: a pathologically deep + // directory tree would otherwise overflow the stack (H-B). + if depth >= MAX_WALK_DEPTH { + eprintln!( + "warning: walk_force_index reached max depth {} at {:?}; skipping subtree to avoid stack overflow [module=mcp, tool=force_index_files, method=walk_force_index]", + MAX_WALK_DEPTH, path + ); + *skipped_dirs += 1; + continue; + } + // Recurse unconditionally — bypass skip-dir rules. + walk_force_index( + &path, + max_size, + lang_whitelist, + out_files, + skipped_files, + skipped_dirs, + depth + 1, + ); + continue; + } + if path.is_file() { + match filter_acceptable_file(&path, max_size, lang_whitelist) { + Some(fp) => out_files.push(fp), + None => *skipped_files += 1, + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn wl(vals: &[&str]) -> std::collections::HashSet { + vals.iter().map(|s| s.to_string()).collect() + } + + #[test] + fn lang_whitelist_alias_folds_both_ways() { + // Short filter forms must accept the canonical label and the + // reverse — callers write "c"/"js"/"py" while the map emits + // "c"/"javascript"/"python". + assert!(lang_whitelisted(&wl(&["c"]), "c")); + assert!(lang_whitelisted(&wl(&["c"]), "cpp")); + assert!(lang_whitelisted(&wl(&["cpp"]), "c")); + assert!(lang_whitelisted(&wl(&["c++"]), "cpp")); + // Non-alias languages must still require an exact family hit. + assert!(lang_whitelisted(&wl(&["python"]), "python")); + assert!(!lang_whitelisted(&wl(&["python"]), "rust")); + assert!(!lang_whitelisted(&wl(&["c"]), "rust")); + // Empty language label (shebang-script path) is rejected only when + // non-empty check upstream decides; here empty label falls to + // `other` and must not match an unrelated filter. + assert!(!lang_whitelisted(&wl(&["c"]), "")); + } + + #[test] + fn build_output_dir_skips_artifacts_but_not_source_siblings() { + let tmp = std::env::temp_dir().join(format!("codescope_skip_test_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&tmp); + std::fs::create_dir_all(tmp.join("build-tools")).unwrap(); + std::fs::create_dir_all(tmp.join("build")).unwrap(); + std::fs::write(tmp.join("build").join("CMakeCache.txt"), "x").unwrap(); + std::fs::create_dir_all(tmp.join("build-scripts")).unwrap(); + std::fs::create_dir_all(tmp.join("target")).unwrap(); + std::fs::create_dir_all(tmp.join("node_modules")).unwrap(); + // Gradle-style output tree: no CMake markers, but generated/. The + // Gradle build dir is always exactly `build`, so this must skip. + std::fs::create_dir_all(tmp.join("gradle").join("build").join("generated")).unwrap(); + // build-tools/ with a Gradle-looking leaf name is still source: + // the leaf markers apply only to the exact name `build`. + std::fs::create_dir_all(tmp.join("build-tools").join("libs")).unwrap(); + std::fs::create_dir_all(tmp.join("build-gradle").join("generated")).unwrap(); + // build-release/ with CMake artifacts (the CMake build-release style) + // must still skip — this is the only skip route left for build-*. + std::fs::create_dir_all(tmp.join("build-release")).unwrap(); + std::fs::write(tmp.join("build-release").join("CMakeCache.txt"), "x").unwrap(); + // build-* with no artifacts at all must stay indexable. + std::fs::create_dir_all(tmp.join("build-plain")).unwrap(); + std::fs::create_dir_all(tmp.join("gradle-classes").join("build").join("classes")).unwrap(); + std::fs::create_dir_all(tmp.join("gradle-libs").join("build").join("libs")).unwrap(); + std::fs::create_dir_all(tmp.join("gradle-tmp").join("build").join("tmp")).unwrap(); + // A dir named exactly `build` holding source with none of the + // markers must not be swallowed by the name heuristic. + std::fs::create_dir_all(tmp.join("src-build").join("build")).unwrap(); + std::fs::write( + tmp.join("src-build").join("build").join("tool.c"), + "int t(){return 0;}\n", + ) + .unwrap(); + + // build/ with CMake artifacts is a build tree → skip. + assert!(is_build_output_dir(&tmp.join("build"))); + // Gradle build/ (exact name) with generated/ (no CMake artifacts) + // → skip too. + assert!(is_build_output_dir(&tmp.join("gradle").join("build"))); + // Each Gradle leaf name alone is enough for the exact name `build`. + assert!(is_build_output_dir( + &tmp.join("gradle-classes").join("build") + )); + assert!(is_build_output_dir(&tmp.join("gradle-libs").join("build"))); + assert!(is_build_output_dir(&tmp.join("gradle-tmp").join("build"))); + // build-* with CMake artifacts (build-release style) → skip; this is + // the only skip route for build-* after the leaf-marker narrowing. + assert!(is_build_output_dir(&tmp.join("build-release"))); + // Unambiguous output/cache names → skip regardless of content. + assert!(is_build_output_dir(&tmp.join("target"))); + assert!(is_build_output_dir(&tmp.join("node_modules"))); + // Front-end bundler output. `dist/` used to be indexed by force-index + // (the name was missing from this list while the engine's FilterPolicy + // had it), which filled the graph with the minified bundle's hashed + // names instead of the project's own sources. + for name in [ + "dist", + "out", + ".next", + ".nuxt", + ".svelte-kit", + ".turbo", + ".parcel-cache", + "coverage", + ".venv", + "site-packages", + ".terraform", + ] { + assert!( + is_build_output_dir(&tmp.join(name)), + "{name} must be skipped as build output" + ); + } + // Source trees that merely start with "build-" stay indexable — + // even when they contain Gradle-looking leaf names (build-tools/libs, + // build-gradle/generated): leaf markers never apply to build-*. + assert!(!is_build_output_dir(&tmp.join("build-tools"))); + assert!(!is_build_output_dir(&tmp.join("build-scripts"))); + assert!(!is_build_output_dir(&tmp.join("build-gradle"))); + // build-* with no artifacts at all (CMake or otherwise) stays + // indexable — name alone is never enough for `build*`. + assert!(!is_build_output_dir(&tmp.join("build-plain"))); + // A dir named exactly `build` with no markers stays indexable too. + assert!(!is_build_output_dir(&tmp.join("src-build").join("build"))); + + let _ = std::fs::remove_dir_all(&tmp); + } +} diff --git a/server/src/tools/mod.rs b/server/src/tools/mod.rs index f96dbdf..d86fd87 100644 --- a/server/src/tools/mod.rs +++ b/server/src/tools/mod.rs @@ -3,222 +3,37 @@ use serde_json::{Value, json}; use crate::ffi; use once_cell::sync::Lazy; use std::collections::HashMap; -use std::process::{Command, Stdio}; -use std::sync::mpsc; -use std::time::Duration; - -/// ── Discover: quick project scan ──────────────────────────────── -/// Walks the directory tree, counts candidate source files per top-level -/// directory (same filtering rules as the C++ engine's FilterPolicy). -/// No parsing, no SQLite — just a fast filesystem walk. -/// Output: JSON with per-directory file counts + total. -pub fn discover(dir_path: &str) -> String { - use std::collections::BTreeMap; - use std::path::Path; - - // Built-in skip dirs matching FilterPolicy's normal_skip_dirs_ - // Applied to EVERY path component (any depth). - fn is_skip_dir(name: &str) -> bool { - let lower = name.to_lowercase(); - matches!( - lower.as_str(), - ".git" - | ".svn" - | ".hg" - | ".bzr" - | "node_modules" - | ".venv" - | "venv" - | "env" - | ".env" - | "__pycache__" - | ".pytest_cache" - | ".mypy_cache" - | ".cache" - | ".codescope" - | "target" - | "build" - | "dist" - | ".next" - | ".turbo" - | ".direnv" - | "vendor" - | "third_party" - | "third-party" - | ".bundle" - | ".gem" - | "go_pkg" - | "pkg" - ) - } - - // Top-only skip dirs matching FilterPolicy's top_only_skip_dirs_ - // Source-bearing dirs that are rarely the focus of analysis - // (test/, docs/, bench/, examples/, ...). Matched ONLY against the - // first path component (top-level dirs), so Java packages like - // org/springframework/samples/petclinic are NOT falsely skipped. - fn is_top_only_skip_dir(name: &str) -> bool { - let lower = name.to_lowercase(); - matches!( - lower.as_str(), - "test" - | "tests" - | "docs" - | "doc" - | "documentation" - | "examples" - | "example" - | "samples" - | "sample" - | "scripts" - | "hack" - | "migrations" - | "seeds" - | "e2e" - | "integration" - | "locale" - | "locales" - | "i18n" - | "l10n" - | "assets" - | "static" - | "public" - | "media" - | "external" - | "vendored" - | "bench" - | "benchmarks" - ) - } +use std::sync::OnceLock; - fn is_skip_prefix(name: &str) -> bool { - name.starts_with('.') // hidden dirs/files - } - - // Countable source extensions (matching FilterPolicy::isSourceFile) - fn is_source_file(name: &str) -> bool { - let dot = name.rfind('.'); - if dot.is_none() { - return false; - } - let ext = &name[dot.unwrap()..].to_lowercase(); - matches!( - ext.as_str(), - ".c" | ".h" - | ".cpp" - | ".hpp" - | ".cc" - | ".cxx" - | ".hh" - | ".hxx" - | ".rs" - | ".go" - | ".py" - | ".java" - | ".kt" - | ".kts" - | ".js" - | ".jsx" - | ".ts" - | ".tsx" - | ".swift" - | ".rb" - | ".php" - | ".cs" - | ".fs" - | ".scala" - | ".clj" - | ".cljs" - | ".ex" - | ".exs" - | ".erl" - | ".hrl" - | ".vue" - | ".svelte" - | ".mjs" - | ".cjs" - | ".mts" - | ".cts" - | ".d.ts" - | ".wasm" - | ".zig" - | ".mojo" - ) - } +// ─── Resolved runtime configuration ───────────────────────────── +// +// The database path is resolved once from `--rootPath` (or the process +// environment) before the MCP server spawns any thread, then stored here. +// This replaces the previous `unsafe { env::set_var(...) }`: mutating +// `environ` is unsafe in Rust 2024 because it races with concurrent +// `getenv`, and the server later spawns worker threads. Tool handlers that +// need the path (e.g. the worker-subprocess indexer) read this immutable +// value instead. +static DB_PATH: OnceLock = OnceLock::new(); - let root = Path::new(dir_path); - if !root.is_dir() { - return json!({ - "ok": false, - "error": format!("directory not found: {}", dir_path) - }) - .to_string(); - } - - let mut modules: BTreeMap = BTreeMap::new(); - let mut total_files: u64 = 0; - let mut total_dirs: u64 = 0; - let mut skipped_dirs: u64 = 0; - let mut skipped_files: u64 = 0; - - // Walk top-level entries first - if let Ok(entries) = std::fs::read_dir(root) { - for entry in entries.flatten() { - let name = entry.file_name(); - let name_str = name.to_string_lossy().to_string(); - let path = entry.path(); - - if path.is_dir() { - let base = name_str.clone(); - if is_skip_dir(&base) || is_top_only_skip_dir(&base) || is_skip_prefix(&base) { - skipped_dirs += 1; - continue; - } - total_dirs += 1; - - // Recursively count source files in this directory - let mut count: u64 = 0; - let walk = walkdir::WalkDir::new(&path).into_iter().filter_entry(|e| { - let fname = e.file_name().to_string_lossy(); - if e.depth() == 0 { - return true; - } - if e.file_type().is_dir() { - !is_skip_dir(&fname) && !is_skip_prefix(&fname) - } else { - true - } - }); - for entry in walk.flatten() { - if entry.file_type().is_file() { - let fname = entry.file_name().to_string_lossy(); - if is_source_file(&fname) { - count += 1; - } else { - skipped_files += 1; - } - } - } - if count > 0 { - modules.insert(base, count); - total_files += count; - } - } - } - } +/// Record the resolved database path. First call wins; later calls are +/// ignored so handlers and worker subprocesses cannot desync. Intended to +/// run once at startup, before any thread is spawned. +pub fn set_db_path(path: String) { + let _ = DB_PATH.set(path); +} - json!({ - "ok": true, - "project_path": dir_path, - "total_dirs": total_dirs, - "total_files": total_files, - "skipped_dirs": skipped_dirs, - "skipped_files": skipped_files, - "modules": modules - }) - .to_string() +/// The resolved database path, or `None` before `set_db_path` runs. +pub fn db_path() -> Option<&'static str> { + DB_PATH.get().map(String::as_str) } +// The quick project scan lives in tools/discover.rs — a filesystem +// walk with no parsing and no SQLite, so it has no reason to share +// this file (see plan/rules/code_rules.md 1000-line rule). +mod discover; +pub use discover::discover; + /// Handler signature for a single MCP tool. type ToolHandler = fn(u64, &Value) -> String; @@ -238,537 +53,24 @@ fn h_find_references(project_id: u64, args: &Value) -> String { // ─── Worker Supervisor (timeout + retry) ─────────────────────── -/// Default worker timeout in seconds when `CODESCOPE_WORKER_TIMEOUT` is unset. -const DEFAULT_WORKER_TIMEOUT_SECS: u64 = 300; -/// Interval between polls when waiting for a worker subprocess to finish. -const WORKER_POLL_INTERVAL: Duration = Duration::from_millis(100); - -static WORKER_TIMEOUT: Lazy = Lazy::new(|| { - Duration::from_secs( - std::env::var("CODESCOPE_WORKER_TIMEOUT") - .ok() - .and_then(|s| s.parse().ok()) - .unwrap_or(DEFAULT_WORKER_TIMEOUT_SECS), - ) -}); -const MAX_RETRIES: usize = 3; - -/// Number of attempts to re-initialise the C++ engine after a worker -/// run if the first `ffi::init` fails. The worker subprocess may hold -/// the SQLite WAL lock briefly or leave the DB in a busy state; a -/// short retry window recovers the engine instead of leaving the -/// server in an unusable "g_store == null" state where every -/// subsequent tool call returns "not initialised". See H4. -const ENGINE_INIT_MAX_ATTEMPTS: usize = 3; -/// Delay between engine re-init attempts. Chosen to be long enough -/// for the worker's WAL lock to release on a busy system but short -/// enough not to materially delay the index response. -const ENGINE_INIT_RETRY_DELAY: Duration = Duration::from_millis(500); - -/// Upper bound for `limit` arguments accepted by query-style tools. -/// Values above this are clamped down to prevent unbounded result sets. -const MAX_QUERY_LIMIT: i64 = 100; -/// Default `limit` used when the client omits the argument. -const DEFAULT_QUERY_LIMIT: i64 = 20; - -/// Run a worker subprocess with timeout protection. -/// Returns `Ok(output)` on success, `Err(msg)` on timeout or failure. -/// On timeout the orphaned child is killed using a platform-appropriate -/// method: `kill -9` on Unix, `taskkill /F` on Windows. On any other -/// platform the process is logged as orphaned (the `Child` handle was moved -/// into the wait thread and is no longer accessible here). -fn run_worker( - exe: &str, - args: &[&str], - envs: &[(&str, &str)], -) -> Result { - let mut cmd = Command::new(exe); - cmd.args(args); - for (k, v) in envs { - cmd.env(k, v); - } - // Pipe stdout/stderr so the worker's output is captured instead of - // leaking into the MCP server's stdout transport stream. - cmd.stdout(Stdio::piped()).stderr(Stdio::piped()); - let child = cmd.spawn().map_err(|e| format!("spawn failed: {}", e))?; - let pid = child.id(); - - let (tx, rx) = mpsc::channel(); - - // Thread: wait for child completion - let tx_out = tx.clone(); - std::thread::spawn(move || { - let output = child.wait_with_output(); - let _ = tx_out.send(output); - }); - - // Poll for result with timeout - let start = std::time::Instant::now(); - loop { - if start.elapsed() > *WORKER_TIMEOUT { - // Kill orphaned child via a platform-appropriate method. The child - // handle was moved into the wait thread above, so we cannot call - // `child.kill()` here; instead we signal by PID. - #[cfg(unix)] - { - let _ = Command::new("kill").args(["-9", &pid.to_string()]).output(); - } - #[cfg(windows)] - { - let _ = Command::new("taskkill") - .args(["/F", "/PID", &pid.to_string()]) - .output(); - } - #[cfg(not(any(unix, windows)))] - { - eprintln!( - "warning: worker timeout on unsupported platform — process {} may be orphaned", - pid - ); - } - return Err(format!( - "worker timed out after {}s", - WORKER_TIMEOUT.as_secs() - )); - } - - match rx.try_recv() { - Ok(Ok(output)) => return Ok(output), - Ok(Err(e)) => return Err(format!("worker error: {}", e)), - Err(mpsc::TryRecvError::Empty) => { - std::thread::sleep(WORKER_POLL_INTERVAL); - } - Err(mpsc::TryRecvError::Disconnected) => { - return Err("worker channel disconnected".to_string()); - } - } - } -} - -/// Internal worker-subprocess indexer used by the MCP session auto-index -/// path. NOT registered as a public tool: index-parallel (with keep_db -/// incremental) fully replaces the serial index_project tool, so the -/// MCP tool list no longer exposes it (47→46 tools). The engine and the -/// worker subprocess entry point are shared with index-parallel and stay. -pub fn index_project_via_worker(project_id: u64, args: &Value) -> String { - let path = args["project_path"].as_str().unwrap_or(""); - - // Use worker subprocess for memory isolation - let self_exe = std::env::current_exe().ok(); - if let Some(exe) = self_exe { - let db_path = std::env::var("CODESCOPE_DB_PATH") - .unwrap_or_else(|_| ".codescope/codescope.db".to_string()); - let grammars_dir = std::env::var("GRAMMARS_DIR").unwrap_or_else(|_| "grammars".to_string()); - let lang = args["language_filter"].as_str().unwrap_or(""); - - // Derive a meaningful project name from the path's final component so the - // worker records a human-readable name instead of the placeholder - // "worker-project". - let project_name = std::path::Path::new(path) - .file_name() - .and_then(|n| n.to_str()) - .unwrap_or("unnamed") - .to_string(); - - for attempt in 1..=MAX_RETRIES { - // Shutdown engine before spawning worker to release SQLite lock - crate::ffi::shutdown(); - - let args_list = [ - "worker", - &db_path, - path, - lang, - &project_name, - &project_id.to_string(), - ]; - let envs = [ - ("GRAMMARS_DIR", &grammars_dir as &str), - ("CODESCOPE_DB_PATH", &db_path as &str), - ("CODESCOPE_VERBOSE", "0"), - ]; - - let result = run_worker(exe.to_str().unwrap_or("codescope"), &args_list, &envs); - - // Re-init engine after worker completes (regardless of success/failure). - // The worker subprocess may have left the SQLite WAL lock held briefly - // or the DB may be corrupted; checking the return value prevents the - // server from silently running with a null g_store for the rest of its - // lifetime (every subsequent tool call would return "not initialized"). - // - // H4: A single failed init must NOT leave the server in an unusable - // state. We retry with the original (pre-shutdown) db_path a few - // times with a short delay — this recovers the engine when the - // failure was transient (e.g. WAL lock not yet released). Only if - // every attempt fails do we propagate the error; the caller (and - // the operator) must restart the server in that case. - let mut last_init_code: i32 = 0; - let mut engine_recovered = false; - for init_attempt in 1..=ENGINE_INIT_MAX_ATTEMPTS { - let code = crate::ffi::init(&db_path); - if code == 0 { - engine_recovered = true; - break; - } - last_init_code = code; - if init_attempt < ENGINE_INIT_MAX_ATTEMPTS { - eprintln!( - "engine re-init attempt {}/{} failed (code={}); retrying in {:?} [module=mcp, tool=index_project, method=ffi::init]", - init_attempt, ENGINE_INIT_MAX_ATTEMPTS, code, ENGINE_INIT_RETRY_DELAY - ); - std::thread::sleep(ENGINE_INIT_RETRY_DELAY); - } - } - if !engine_recovered { - return format!( - "{{\"ok\":false,\"error\":\"engine re-initialization failed after index (code={}, attempts={}). The database may be locked or corrupted; the engine is now uninitialized — restart the server before issuing further tool calls. [module=mcp, tool=index_project, method=ffi::init]\"}}", - last_init_code, ENGINE_INIT_MAX_ATTEMPTS - ); - } - - match result { - Ok(out) => { - if out.status.success() { - let stdout = String::from_utf8_lossy(&out.stdout); - // Trigger background FTS build after a successful index. - // Uses spawn_fts_build to deduplicate concurrent builds - // and avoid a data race on the global C++ g_store. - if let Some(json_start) = stdout.find('{') - && let Some(json_end) = stdout[json_start..].rfind('}') - { - let candidate = &stdout[json_start..=json_start + json_end]; - // Validate the slice is well-formed JSON before returning - // it, so malformed worker output does not produce invalid - // JSON that would confuse the MCP client. - if serde_json::from_str::(candidate).is_ok() { - crate::ffi::spawn_fts_build(project_id); - return candidate.to_string(); - } - } - crate::ffi::spawn_fts_build(project_id); - return stdout.to_string(); - } - let stderr = String::from_utf8_lossy(&out.stderr); - let err_msg = stderr.lines().last().unwrap_or("unknown"); - if attempt < MAX_RETRIES { - eprintln!( - "worker attempt {} failed ({}), retrying...", - attempt, err_msg - ); - std::thread::sleep(Duration::from_secs(1)); - continue; - } - return format!( - "{{\"ok\":false,\"error\":\"worker failed after {} attempts: {}\"}}", - MAX_RETRIES, err_msg - ); - } - Err(msg) => { - if attempt < MAX_RETRIES { - eprintln!("worker attempt {}: {} — retrying...", attempt, msg); - std::thread::sleep(Duration::from_secs(1)); - continue; - } - return format!( - "{{\"ok\":false,\"error\":\"worker {} after {} attempts\"}}", - msg, MAX_RETRIES - ); - } - } - } - } - - // Fallback: in-process indexing with correct null handling for missing lang filter - // Fix: Use CString to ensure null-termination for FFI compatibility - let lang = args["language_filter"].as_str(); - let lang_cstring = lang.map(|s| std::ffi::CString::new(s).unwrap_or_default()); - let lang_ptr = lang_cstring - .as_ref() - .map_or(std::ptr::null(), |cs| cs.as_ptr() as *const _); - ffi::index_project(project_id, path, lang_ptr) -} - -fn h_index_file(project_id: u64, args: &Value) -> String { - let path = args["file_path"].as_str().unwrap_or(""); - ffi::index_file(project_id, path) -} - -/// Force-index specific files or directories, bypassing FilterPolicy's -/// default skip rules (test/, docs/, vendored/, node_modules/, etc.). -/// -/// Use case: user says "go index xxx/yyy for me" — the AI calls this -/// tool with paths=[...]. Files under the given paths are indexed -/// regardless of the default skip list, so the user can pull in -/// test fixtures, vendored deps, or generated code on demand. -/// -/// Args: -/// paths: array of absolute file/dir paths to force-index. -/// language_filter: optional comma-separated language whitelist -/// (e.g. "java,python"). When empty, all detectable languages -/// are indexed. -/// -/// Returns the engine_index_files JSON result -/// (files_indexed/nodes/edges/errors). -fn h_force_index_files(project_id: u64, args: &Value) -> String { - // Collect paths: accept either `paths: [...]` or legacy - // `path: "..."` for single-path convenience. - let mut paths: Vec = Vec::new(); - if let Some(arr) = args["paths"].as_array() { - for v in arr { - if let Some(s) = v.as_str() - && !s.is_empty() - { - paths.push(s.to_string()); - } - } - } - if let Some(s) = args["path"].as_str() - && !s.is_empty() - { - paths.push(s.to_string()); - } - if paths.is_empty() { - return "{\"ok\":false,\"error\":\"paths is required (array of file/dir paths)\"}" - .to_string(); - } - - let lang_filter = args["language_filter"].as_str().unwrap_or(""); - - // Expand directories into individual file paths, bypassing - // FilterPolicy's shouldSkipEntry. We DO still respect: - // - file size limit (CODESCOPE_MAX_FILE_SIZE, default 5MB) - // - language detectability (detectLanguage must return non-null) - // - optional language_filter whitelist - // We DO NOT respect: - // - normal_skip_dirs_ / top_only_skip_dirs_ (test/, docs/, ...) - // - skip_suffixes_ for source extensions - // - .gitignore / .codescopeignore - // This is the "user override" path. - let max_size: u64 = std::env::var("CODESCOPE_MAX_FILE_SIZE") - .ok() - .and_then(|s| s.parse().ok()) - .unwrap_or(5 * 1024 * 1024); - - let mut lang_whitelist: Option> = None; - if !lang_filter.is_empty() { - let mut set = std::collections::HashSet::new(); - for part in lang_filter.split(',') { - let p = part.trim().to_lowercase(); - if !p.is_empty() { - set.insert(p); - } - } - lang_whitelist = Some(set); - } - - let mut all_files: Vec = Vec::new(); - let mut skipped_files = 0u64; - let mut skipped_dirs = 0u64; - - for p in &paths { - let path = std::path::Path::new(p); - if !path.exists() { - skipped_files += 1; - continue; - } - if path.is_file() { - // Single file — index directly if detectable. - if let Some(fp) = filter_acceptable_file(path, max_size, lang_whitelist.as_ref()) { - all_files.push(fp); - } else { - skipped_files += 1; - } - continue; - } - // Directory — walk it, bypassing skip-dir rules but still - // respecting file-level detectability + size. Start at depth 0. - walk_force_index( - path, - max_size, - lang_whitelist.as_ref(), - &mut all_files, - &mut skipped_files, - &mut skipped_dirs, - 0, - ); - } - - if all_files.is_empty() { - return format!( - "{{\"ok\":true,\"files_indexed\":0,\"nodes\":0,\"edges\":0,\"errors\":0,\"skipped_files\":{},\"skipped_dirs\":{}}}", - skipped_files, skipped_dirs - ); - } - - // Build JSON file list and call engine_index_files. - let json_list = serde_json::Value::Array( - all_files - .iter() - .map(|s| serde_json::Value::String(s.clone())) - .collect(), - ) - .to_string(); - - let result = ffi::index_files(project_id, &json_list); - - // Annotate result with skip stats for transparency. - if let Ok(mut v) = serde_json::from_str::(&result) { - if let Some(obj) = v.as_object_mut() { - obj.insert("skipped_files".into(), skipped_files.into()); - obj.insert("skipped_dirs".into(), skipped_dirs.into()); - obj.insert("paths_requested".into(), paths.len().into()); - } - return v.to_string(); - } - result -} - -/// Maximum recursion depth for `walk_force_index`. A deeply nested directory -/// tree (e.g. a chain of node_modules) or a crafted path would otherwise -/// consume the call stack until it overflows, panicking and crashing the -/// MCP server (local DoS, see H-B). 256 levels is far deeper than any -/// legitimate project tree; once reached we stop descending and log a -/// warning so the walk always terminates. -const MAX_WALK_DEPTH: u32 = 256; - -/// Source extensions recognised by the C++ FilterPolicy::detectLanguage. -/// Mirrored here so the force-index walk can decide which files to -/// accept without crossing the FFI boundary for every entry. -const SOURCE_EXTENSIONS: &[&str] = &[ - ".py", ".cpp", ".cc", ".cxx", ".c", ".h", ".hpp", ".hxx", ".hh", ".rs", ".swift", ".js", - ".mjs", ".cjs", ".ts", ".tsx", ".go", ".java", ".kt", ".kts", ".rb", ".scala", -]; - -/// Check a single file path against force-index rules: -/// - must exist and be a regular file -/// - must be within max_size -/// - extension must be a recognised source extension -/// - must pass the optional language whitelist -/// -/// Returns the absolute path string if acceptable, None otherwise. -fn filter_acceptable_file( - path: &std::path::Path, - max_size: u64, - lang_whitelist: Option<&std::collections::HashSet>, -) -> Option { - let meta = match std::fs::metadata(path) { - Ok(m) => m, - Err(_) => return None, - }; - if !meta.is_file() { - return None; - } - if meta.len() > max_size { - return None; - } - - // Extension check (case-insensitive) - let ext = match path.extension().and_then(|e| e.to_str()) { - Some(e) => format!(".{}", e.to_lowercase()), - None => return None, - }; - if !SOURCE_EXTENSIONS.iter().any(|&s| s == ext) { - return None; - } - - // Language whitelist (maps extension -> language label) - if let Some(wl) = lang_whitelist { - let lang = match ext.as_str() { - ".py" => "python", - ".cpp" | ".cc" | ".cxx" | ".h" | ".hpp" | ".hxx" | ".hh" => "cpp", - ".c" => "c", - ".rs" => "rust", - ".swift" => "swift", - ".js" | ".mjs" | ".cjs" => "javascript", - ".ts" => "typescript", - ".tsx" => "tsx", - ".go" => "go", - ".java" => "java", - ".kt" | ".kts" => "kotlin", - ".rb" => "ruby", - ".scala" => "scala", - _ => "", - }; - if !lang.is_empty() && !wl.contains(lang) { - return None; - } - } - - // Return absolute path - let canon = std::fs::canonicalize(path).unwrap_or_else(|_| path.to_path_buf()); - Some(canon.to_string_lossy().to_string()) -} - -/// Recursively walk `root` for force-index. Bypasses all skip-dir -/// rules (test/, docs/, vendored/, node_modules/, etc.) — this is -/// the whole point of the force-index tool. Still respects: -/// - file size limit -/// - extension detectability -/// - optional language whitelist -/// -/// `depth` is the current recursion depth (call with 0 at the top). -/// Recursion stops once `depth >= MAX_WALK_DEPTH` to guarantee the walk -/// always terminates and cannot exhaust the stack (H-B). When the limit -/// is hit, a warning is logged and descent into that subtree is skipped. -fn walk_force_index( - root: &std::path::Path, - max_size: u64, - lang_whitelist: Option<&std::collections::HashSet>, - out_files: &mut Vec, - skipped_files: &mut u64, - skipped_dirs: &mut u64, - depth: u32, -) { - let entries = match std::fs::read_dir(root) { - Ok(e) => e, - Err(_) => { - *skipped_dirs += 1; - return; - } - }; - for entry in entries.flatten() { - let path = entry.path(); - // Use symlink_metadata (NOT following symlinks) to detect the - // entry's own type. Path::is_dir() follows symlinks, so a symlink - // loop (a -> b -> a) would make is_dir() always return true and - // the recursion would never terminate → stack overflow panic - // crashing the MCP server (local DoS). symlink_metadata gives us - // the link's own metadata without dereferencing, so we only recurse - // into real directories. See CODE_REVIEW_FINDINGS_2026-07-19.md C5. - let is_real_dir = std::fs::symlink_metadata(&path) - .map(|m| m.is_dir()) - .unwrap_or(false); - if is_real_dir { - // Guard against unbounded recursion: a pathologically deep - // directory tree would otherwise overflow the stack (H-B). - if depth >= MAX_WALK_DEPTH { - eprintln!( - "warning: walk_force_index reached max depth {} at {:?}; skipping subtree to avoid stack overflow [module=mcp, tool=force_index_files, method=walk_force_index]", - MAX_WALK_DEPTH, path - ); - *skipped_dirs += 1; - continue; - } - // Recurse unconditionally — bypass skip-dir rules. - walk_force_index( - &path, - max_size, - lang_whitelist, - out_files, - skipped_files, - skipped_dirs, - depth + 1, - ); - continue; - } - if path.is_file() { - match filter_acceptable_file(&path, max_size, lang_whitelist) { - Some(fp) => out_files.push(fp), - None => *skipped_files += 1, - } - } - } -} +// The worker timeout/retry constants above moved to tools/indexing.rs +// alongside the code that uses them (1000-line rule). +// +// Argument clamps (depth/radius/edge-type/limit bounds) live in +// tools/clamp.rs with their unit tests (1000-line rule). +mod clamp; +use clamp::{ + DEFAULT_CODESCOPE_TRACE_DEPTH, DEFAULT_QUERY_LIMIT, DEFAULT_TRACE_FLOW_DEPTH, MAX_QUERY_LIMIT, + clamp_depth, clamp_edge_type, clamp_findings_limit, clamp_knowledge_limit, + clamp_max_communities, clamp_max_members, clamp_radius, +}; + +// The indexing/worker machinery (subprocess orchestration, force-index +// file walking, acceptability filtering) lives in tools/indexing.rs — see +// plan/rules/code_rules.md (1000-line rule). +mod indexing; +pub use indexing::index_project_via_worker; +use indexing::{h_force_index_files, h_index_file}; fn h_get_graph_stats(project_id: u64, _args: &Value) -> String { ffi::get_graph_stats(project_id) @@ -784,12 +86,19 @@ fn h_search_code(project_id: u64, args: &Value) -> String { } fn h_detect_changes(project_id: u64, args: &Value) -> String { - let files = args["modified_files"].as_str().unwrap_or("[]"); - ffi::detect_changes(project_id, files) + // The engine's parameter is `modified_files_json` and it wants a JSON + // ARRAY of paths, but the schema advertises a plain string — so a caller + // passing what the schema describes (a bare path) sent invalid JSON and + // got `modified: []` back, i.e. "nothing changed", with no error. A real + // array is serialized now; a string is taken verbatim, so a caller that + // pre-encoded the array keeps working. + let files = json_or_string_arg(args, "modified_files", "[]"); + ffi::detect_changes(project_id, &files) } -fn h_verify_integrity(project_id: u64, _args: &Value) -> String { - ffi::verify_integrity(project_id) +fn h_verify_integrity(project_id: u64, args: &Value) -> String { + let max_findings = clamp_findings_limit(args["max_findings"].as_i64()); + ffi::verify_integrity(project_id, max_findings) } fn h_explain_symbol(project_id: u64, args: &Value) -> String { @@ -803,12 +112,68 @@ fn h_explain_symbol(project_id: u64, args: &Value) -> String { /// The claim JSON carries type/subject/predicate/object/scope fields /// consumed by the C++ VerifierRegistry. fn h_verify_claim(project_id: u64, args: &Value) -> String { - let claim_json = args["claim"].as_str().unwrap_or(""); + let claim_json = claim_json_from(args); if claim_json.is_empty() { return json!({"error": "claim field is required [module=mcp, tool=verify_claim]"}) .to_string(); } - ffi::verify_claim(project_id, claim_json) + ffi::verify_claim(project_id, &claim_json) +} + +/// A tool argument whose schema says "a JSON payload" but which the engine +/// reads as a string. +/// +/// Two tools had that shape mismatch — `verify_claim`'s `claim` (documented as +/// a JSON object, only a pre-serialized string accepted) and `detect_changes`' +/// `modified_files` (documented as a string, only a JSON array accepted, which +/// made a bare path silently mean "no changes") — so both accept the structured +/// form now and take a string verbatim, keeping existing callers working. +fn json_or_string_arg(args: &Value, key: &str, fallback: &str) -> String { + match &args[key] { + Value::String(s) => s.clone(), + Value::Array(_) | Value::Object(_) => args[key].to_string(), + _ => fallback.to_string(), + } +} + +/// The claim JSON carried by a tool call (see json_or_string_arg). +fn claim_json_from(args: &Value) -> String { + json_or_string_arg(args, "claim", "") +} + +/// Normalise a type-id filter argument (`node_types` / `edge_types`). +/// +/// The schemas advertise these as a "comma-separated" string, but the natural +/// client spelling — and how the neighbouring `get_subgraph` / `get_neighbors` +/// arguments read — is an array of ids. `.as_str()` returned `None` for an +/// array, which silently DROPPED the filter and answered with the whole graph: +/// a caller asking for node type 1 got type 0 back, with nothing in the +/// response saying the filter had been ignored. Arrays are joined now, and any +/// other shape is rejected loudly rather than ignored. +fn type_filter_arg(args: &Value, key: &str, tool: &str) -> Result, String> { + match &args[key] { + Value::Null => Ok(None), + Value::String(s) => Ok(Some(s.clone())), + Value::Array(items) => { + let mut ids = Vec::with_capacity(items.len()); + for item in items { + match item.as_i64() { + Some(n) if n >= 0 => ids.push(n.to_string()), + _ => { + return Err(format!( + "{key}: expected non-negative integer ids, got {item} \ + [module=mcp, tool={tool}]" + )); + } + } + } + Ok(Some(ids.join(","))) + } + other => Err(format!( + "{key}: expected a comma-separated string or an array of ids, got {other} \ + [module=mcp, tool={tool}]" + )), + } } /// Parse a natural-language summary into claims and verify each one. @@ -903,20 +268,6 @@ fn h_build_evidence(project_id: u64, args: &Value) -> String { ffi::build_evidence(project_id, category) } -/// Verify a natural-language claim against the project's indexed -/// evidence. Thin wrapper over the structured verify_claim path: -/// IntentParser → Claim mapping (capability/contract) → -/// verify_one_claim. Unrecognized intents return -/// error_code="intent_unrecognized". -fn h_verify_statement(project_id: u64, args: &Value) -> String { - let claim = args["claim"].as_str().unwrap_or(""); - if claim.is_empty() { - return json!({"error": "claim field is required [module=mcp, tool=verify_statement]"}) - .to_string(); - } - ffi::verify_statement(project_id, claim) -} - /// Build (or rebuild) and persist the project state snapshot. /// Returns the snapshot JSON describing what each inspector ran and /// what it found. The snapshot is also stored in the project_state @@ -933,7 +284,7 @@ fn h_get_project_state(project_id: u64, _args: &Value) -> String { fn h_trace_flow(project_id: u64, args: &Value) -> String { let name = args["function_name"].as_str().unwrap_or(""); - let depth = args["depth"].as_i64().unwrap_or(3) as i32; + let depth = clamp_depth(args["depth"].as_i64(), DEFAULT_TRACE_FLOW_DEPTH); ffi::explore_function(project_id, name, depth, "callees") } @@ -958,7 +309,7 @@ fn h_get_knowledge_graph(project_id: u64, args: &Value) -> String { return json!({"error": "table field is required [module=mcp, tool=get_knowledge_graph]"}) .to_string(); } - let limit = args["limit"].as_i64().unwrap_or(100) as i32; + let limit = clamp_knowledge_limit(args["limit"].as_i64()); ffi::get_knowledge_graph(project_id, table, limit) } @@ -1016,8 +367,24 @@ fn h_find_callees_by_entity(project_id: u64, args: &Value) -> String { // the verifier subsystem is armed, which public claim types are // supported, and whether the canonical evidence backend has data. fn h_verifier_registry_status(project_id: u64, args: &Value) -> String { - let _ = args; - ffi::get_verifier_registry_status(project_id) + // The tool's schema documents a `project_id` argument ("0 skips the + // evidence backend probe"), but this handler discarded the argument bag + // and always used the server's own project, so the documented argument + // did nothing. It is honoured now — see verifier_registry_project_id. + ffi::get_verifier_registry_status(verifier_registry_project_id(project_id, args)) +} + +/// Which project the verifier-registry probe reports on: the client's explicit +/// `project_id` argument when it sent one, otherwise the server's project. +/// +/// The engine documents `project_id == 0` as "skip the evidence backend probe" +/// (registry state is process-global, the entity/relation counts are not), so +/// honouring the argument is the point of the parameter rather than a way to +/// read another project's data. +fn verifier_registry_project_id(server_project_id: u64, args: &Value) -> u64 { + args.get("project_id") + .and_then(|v| v.as_u64()) + .unwrap_or(server_project_id) } // ── Graph path + component tools ─────────────────────────────── @@ -1091,6 +458,18 @@ fn h_connected_components(project_id: u64, _args: &Value) -> String { ffi::find_connected_components(project_id) } +/// Community detection over the CALLS graph (deterministic label +/// propagation). Summary-first by design: unless `include_members` is true, +/// each community is returned as {id, label, member_count}. The previous +/// implementation returned every member of every community, which produced +/// ~100K tokens for a single query. +fn h_get_communities(project_id: u64, args: &Value) -> String { + let max_communities = clamp_max_communities(args["max_communities"].as_i64()); + let max_members = clamp_max_members(args["max_members"].as_i64()); + let include_members = args["include_members"].as_bool().unwrap_or(false); + ffi::get_communities(project_id, max_members, max_communities, include_members) +} + /// Fetch a local region of the code graph centered on a node. fn h_get_subgraph(project_id: u64, args: &Value) -> String { let node_id = match args["node_id"].as_i64() { @@ -1100,10 +479,22 @@ fn h_get_subgraph(project_id: u64, args: &Value) -> String { .to_string(); } }; - let radius = args["radius"].as_i64().unwrap_or(1).clamp(1, 3) as i32; - let node_types = args["node_types"].as_str(); - let edge_types = args["edge_types"].as_str(); - ffi::get_subgraph(project_id, node_id, radius, node_types, edge_types) + let radius = clamp_radius(args["radius"].as_i64()); + let node_types = match type_filter_arg(args, "node_types", "get_subgraph") { + Ok(v) => v, + Err(e) => return json!({ "error": e }).to_string(), + }; + let edge_types = match type_filter_arg(args, "edge_types", "get_subgraph") { + Ok(v) => v, + Err(e) => return json!({ "error": e }).to_string(), + }; + ffi::get_subgraph( + project_id, + node_id, + radius, + node_types.as_deref(), + edge_types.as_deref(), + ) } /// Fetch the direct neighbors (callers + callees) of a graph node. @@ -1115,8 +506,8 @@ fn h_get_neighbors(project_id: u64, args: &Value) -> String { .to_string(); } }; - let edge_type = args["edge_type"].as_i64().unwrap_or(-1) as i32; - let radius = args["radius"].as_i64().unwrap_or(1) as i32; + let edge_type = clamp_edge_type(args["edge_type"].as_i64()); + let radius = clamp_radius(args["radius"].as_i64()); ffi::get_neighbors(project_id, node_id, edge_type, radius) } @@ -1139,20 +530,26 @@ fn h_get_graph(project_id: u64, args: &Value) -> String { .as_i64() .unwrap_or(20000) .clamp(1, 200000) as i32; - let node_types = args["node_types"].as_str(); - let edge_types = args["edge_types"].as_str(); + let node_types = match type_filter_arg(args, "node_types", "get_graph") { + Ok(v) => v, + Err(e) => return json!({ "error": e }).to_string(), + }; + let edge_types = match type_filter_arg(args, "edge_types", "get_graph") { + Ok(v) => v, + Err(e) => return json!({ "error": e }).to_string(), + }; // Validate type filters: only digits, commas, and spaces allowed let valid_filter = |s: &str| { s.chars() .all(|c| c.is_ascii_digit() || c == ',' || c == ' ') }; - if let Some(nt) = node_types + if let Some(nt) = node_types.as_deref() && !valid_filter(nt) { return json!({"error": "node_types: digits and commas only [module=mcp, tool=get_graph]"}) .to_string(); } - if let Some(et) = edge_types + if let Some(et) = edge_types.as_deref() && !valid_filter(et) { return json!({"error": "edge_types: digits and commas only [module=mcp, tool=get_graph]"}) @@ -1164,8 +561,8 @@ fn h_get_graph(project_id: u64, args: &Value) -> String { node_limit, edge_offset, edge_limit, - node_types, - edge_types, + node_types.as_deref(), + edge_types.as_deref(), ) } @@ -1196,7 +593,7 @@ fn h_codescope_trace(project_id: u64, args: &Value) -> String { if let Some(name) = args["function_name"].as_str() && !name.is_empty() { - let depth = args["depth"].as_i64().unwrap_or(1) as i32; + let depth = clamp_depth(args["depth"].as_i64(), DEFAULT_CODESCOPE_TRACE_DEPTH); let direction = args["direction"].as_str().unwrap_or("both"); return ffi::explore_function(project_id, name, depth, direction); } @@ -1259,7 +656,6 @@ static TOOL_HANDLERS: Lazy> = Lazy::new(|| { // v0.3 Evidence Pipeline m.insert("enhance_project", h_enhance_project as ToolHandler); m.insert("build_evidence", h_build_evidence as ToolHandler); - m.insert("verify_statement", h_verify_statement as ToolHandler); m.insert("build_project_state", h_build_project_state as ToolHandler); m.insert("get_project_state", h_get_project_state as ToolHandler); // Verify + Drift Layer (v0.4) @@ -1307,6 +703,7 @@ static TOOL_HANDLERS: Lazy> = Lazy::new(|| { "connected_components", h_connected_components as ToolHandler, ); + m.insert("get_communities", h_get_communities as ToolHandler); m.insert("get_entry_points", h_get_entry_points as ToolHandler); m.insert("get_type_info", h_get_type_info as ToolHandler); m.insert("get_routes", h_get_routes as ToolHandler); @@ -1326,524 +723,30 @@ static TOOL_HANDLERS: Lazy> = Lazy::new(|| { m }); -// ─── Tool Definitions ─────────────────────────────────────────── - -pub fn all_tools() -> Vec { - use super::mcp::protocol::Tool; - vec![ - Tool { - name: "find_definition".into(), - description: "[DEPRECATED — use find_symbol] Find where a symbol is defined.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "symbol_name": {"type": "string", "description": "Name of the symbol to find"}, - "file_filter": {"type": "string", "description": "Optional: filter by file path substring"} - }, - "required": ["symbol_name"] - }), - }, - Tool { - name: "find_references".into(), - description: "Find all locations that reference a given symbol.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "symbol_name": {"type": "string"}, - "file_filter": {"type": "string"} - }, - "required": ["symbol_name"] - }), - }, - Tool { - name: "search_code".into(), - description: "[DEPRECATED — use search] Full-text search across code symbols, file paths, and comments.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "query": {"type": "string", "description": "Search query string"}, - "limit": {"type": "integer", "description": "Max results (default 20, max 100)"} - }, - "required": ["query"] - }), - }, - Tool { - name: "index_file".into(), - description: "Index a single source file: parse, build IR, and add to the code graph.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "file_path": {"type": "string", "description": "Absolute path to the source file"} - }, - "required": ["file_path"] - }), - }, - Tool { - name: "force_index_files".into(), - description: "Force-index specific files or directories, BYPASSING the default skip rules (test/, docs/, vendored/, node_modules/, .gitignore, etc.). Use this when the user asks to index a path that the default FilterPolicy would skip — e.g. 'go index xxx/yyy for me'. Files under the given paths are indexed regardless of the default skip list, so the user can pull in test fixtures, vendored deps, or generated code on demand. Still respects: file size limit, language detectability, optional language whitelist.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "paths": { - "type": "array", - "items": {"type": "string"}, - "description": "Array of absolute file/dir paths to force-index. Directories are walked recursively, bypassing skip-dir rules." - }, - "path": { - "type": "string", - "description": "Convenience single-path form (merged into `paths` if both given)." - }, - "language_filter": { - "type": "string", - "description": "Optional comma-separated language whitelist (e.g. \"java,python\"). Empty = all detectable languages." - } - }, - "required": ["paths"] - }), - }, - Tool { - name: "get_graph_stats".into(), - description: "Get statistics about the current code graph (node count, edge count, file count).".into(), - input_schema: json!({ "type": "object", "properties": {} }), - }, - Tool { - name: "trace_flow".into(), - description: "Trace the execution flow from a function through its callees recursively. Returns a call chain showing how control flows through the codebase.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "function_name": {"type": "string", "description": "Starting function name"}, - "depth": {"type": "integer", "description": "How many levels to trace (default 3, max 10)"} - }, - "required": ["function_name"] - }), - }, - Tool { - name: "explain_symbol".into(), - description: "Get structured information about a symbol: definition location, callers, callees, and dependencies. Returns a comprehensive view of what the symbol does and how it relates to the rest of the codebase.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "symbol_name": {"type": "string", "description": "Name of the symbol to explain"} - }, - "required": ["symbol_name"] - }), - }, - Tool { - name: "verify_integrity".into(), - description: "Verify codebase integrity: checks that README-promised features actually exist in the code. Returns findings with evidence chains and confidence scores.".into(), - input_schema: json!({ "type": "object", "properties": {} }), - }, - Tool { - name: "verify_claim".into(), - description: "Verify a single claim against the codebase. Dispatches the claim to the appropriate verifier (CapabilityVerifier, ContractVerifier, or ArchitectureVerifier) via the VerifierRegistry, persists the claim + evidence, and returns the verdict with confidence and evidence facts.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "claim": { - "type": "string", - "description": "JSON object describing the claim. Fields: type (capability_exists|contract_holds|architecture_follows|function_implements), subject, predicate, object, scope, source_kind, source_ref" - } - }, - "required": ["claim"] - }), - }, - Tool { - name: "verify_summary".into(), - description: "Parse a natural-language summary (README excerpt, AI summary, PR description) into structured claims using the ClaimParser, then verify each claim. Returns aggregated verdicts and a trust_score. Useful for checking whether prose claims about the codebase are actually backed by code evidence.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "text": { - "type": "string", - "description": "Free-form text to parse + verify (e.g. 'CodeScope supports incremental indexing and is thread-safe')" - } - }, - "required": ["text"] - }), - }, - Tool { - name: "verify_review".into(), - description: "Verify a code review comment by parsing it into claims and dispatching each through the Claim -> Verifier -> Evidence pipeline. Each claim is stamped source_kind=\"code_review\" so the evidence table can be filtered by origin. Output shape is identical to verify_summary.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "text": { - "type": "string", - "description": "The review comment body (e.g. 'This function should be thread-safe and the README claims JWT support')" - } - }, - "required": ["text"] - }), - }, - Tool { - name: "verify_reality".into(), - description: "Verify a single AI statement about the current project reality. Returns a structured evidence report with an aggregate verdict (Supported / Contradicted / PartiallyVerified / Unknown), a confidence score, and the per-claim results array. Use this to check whether an AI's claim about project state is backed by code evidence.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "text": { - "type": "string", - "description": "A single natural-language statement (e.g. 'The login module supports JWT and Refresh tokens.')" - } - }, - "required": ["text"] - }), - }, - Tool { - name: "detect_drift".into(), - description: "Scan all declared capabilities and contracts for drift between documentation/code comments and the actual codebase. Detects MissingCapability (declared in README but no implementing entity with callers) and BrokenContract (declared but no enforcing code). Each drift is persisted as a finding row and returned in the JSON output.".into(), - input_schema: json!({ - "type": "object", - "properties": {} - }), - }, - Tool { - name: "detect_documentation_drift".into(), - description: "Scan README for language support claims (e.g. 'supports C++, Python, Go') and cross-reference with actual entities in the codebase. Reports DocumentationDrift (severity 1) for any claimed language with zero entities. Returns claimed_languages, found_languages, missing_languages, and drifts arrays. Enables end-to-end verification: AI says X, CodeScope checks the tables.".into(), - input_schema: json!({ - "type": "object", - "properties": {} - }), - }, - Tool { - name: "detect_capability_drift".into(), - description: "Scan declared capabilities and cross-reference with actual implementing entities in the codebase. Reports CapabilityDrift (severity 2) for any declared capability with no implementing entity that has callers. Returns total_capabilities count and drifts array. Part of end-to-end verification: AI says X, CodeScope checks the tables.".into(), - input_schema: json!({ - "type": "object", - "properties": {} - }), - }, - Tool { - name: "detect_architecture_drift".into(), - description: "Scan call edges for architecture layer violations (e.g. Repository calling Controller, Controller calling another Controller directly). Classifies entities into Controller/Service/Repository layers by naming convention and file path, then checks the relation table for reverse calls and same-layer bypasses. Reports ArchitectureDrift (severity 1) for each violating call edge.".into(), - input_schema: json!({ - "type": "object", - "properties": {} - }), - }, - Tool { - name: "explain_module".into(), - description: "Build a Knowledge Card for a named module/directory. Returns module info, entities, capabilities, contracts, findings, and an integrity_score. Falls back to deriving module info from the files table by path prefix when the modules table is empty.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "module_name": { - "type": "string", - "description": "Name of the module/directory (e.g. 'engine', 'server')" - } - }, - "required": ["module_name"] - }), - }, - Tool { - name: "enhance_project".into(), - description: "Run background enhancement for a project: full tree-sitter parse, call graph construction, FTS index build, and v0.3 semantic_fact extraction (Step 1.5). This is the prerequisite for build_evidence to produce non-empty findings — the semantic facts (sync/mutex/lock, memory/cstring/alloc, error/bare_except, pattern/todo, framework/gin, ffi/extern_call) are extracted here. Returns a JSON summary with files_processed, symbols_enhanced, call_edges, and timing breakdowns. v0.2.5: complexity metrics (cyclomatic/cognitive/nesting) and n-gram semantic vectors are restored and built during index; use engine_get_capabilities / engine_get_enhancement_status to see readiness.".into(), - input_schema: json!({ - "type": "object", - "properties": {} - }), - }, - Tool { - name: "build_evidence".into(), - description: "Build evidence findings for the project by applying the v0.3 rule set (sync/memory/error/pattern/framework/ffi) to the indexed semantic_fact rows. Each rule declares its fact needs and a combine mode (Collect / MissingMatch / MissingMatchPerFunction / Count); the engine returns a JSON array of Evidence objects with category, title, confidence, and items[]. Run after index/enhance so semantic facts are populated.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "category": { - "type": "string", - "enum": ["sync", "memory", "error", "pattern", "framework", "ffi"], - "description": "Optional: restrict the run to one rule category. Omit to run all categories." - } - } - }), - }, - Tool { - name: "verify_statement".into(), - description: "Verify a natural-language claim against the project's indexed evidence. Thin wrapper over the structured verify_claim path: the claim is parsed into an Intent (IntentParser), mapped to a structured Claim (capability_question -> capability_exists, safety/pattern_question -> contract_holds), and dispatched through the same verify_one_claim core as verify_claim. Returns JSON with verdict (Supported|Contradicted|PartiallyVerified|Unknown), confidence, and verifier-specific detail. Unrecognized intents return error_code=intent_unrecognized. Use this for yes/no questions about code behavior (e.g. 'does this project safely handle CString?'); for function-implements or architecture checks use verify_claim with the explicit type.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "claim": { - "type": "string", - "description": "Natural-language claim (e.g. 'does this project safely handle CString?')" - } - }, - "required": ["claim"] - }), - }, - Tool { - name: "build_project_state".into(), - description: "Build (or rebuild) and persist the project state snapshot. Runs the full v0.3 Evidence Pipeline (evidence aggregation + state queries) and UPSERTs the result into the project_state table. Returns the snapshot JSON describing what each inspector ran and what it found (category counts, confidence score, snapshot metadata). Use this after build_evidence to materialize a health snapshot.".into(), - input_schema: json!({ - "type": "object", - "properties": {} - }), - }, - Tool { - name: "get_project_state".into(), - description: "Get the previously persisted project state snapshot (without rebuilding). Returns the snapshot_json string for the project, or a JSON error object if no snapshot exists yet (run build_project_state first). Use this for fast reads of the latest health snapshot.".into(), - input_schema: json!({ - "type": "object", - "properties": {} - }), - }, - Tool { - name: "detect_changes".into(), - description: "Analyze the impact of code changes across the call graph. Given a list of modified files, returns directly modified functions, their callers (who calls them), and their callees (who they call).".into(), - input_schema: json!({ - "type": "object", - "properties": { - "modified_files": {"type": "string", "description": "JSON array of modified file paths"} - }, - "required": ["modified_files"] - }), - }, - Tool { - name: "find_symbol".into(), - description: "Find symbol(s) by exact name match. Returns id, kind, file path, line/column for each match.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "symbol_name": {"type": "string", "description": "Symbol name to search for"} - }, - "required": ["symbol_name"] - }), - }, - Tool { - name: "get_module_tree".into(), - description: "Get the hierarchical module (directory) tree for the current project. Returns modules with id, parent_id, name, path, and file_count.".into(), - input_schema: json!({ "type": "object", "properties": {} }), - }, - Tool { - name: "get_knowledge_graph".into(), - description: "Direct-query a knowledge-layer table (v0.2.1). Surfaces entity / relation / architecture_edge / module_edge / capability / document / module_summary so you can browse the knowledge graph directly instead of only benefiting indirectly via explain_module / detect_capability_drift / get_module_tree. Block-level: one call returns the whole result set.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "table": { - "type": "string", - "enum": ["entity", "relation", "architecture_edge", - "module_edge", "capability", "document", - "module_summary"], - "description": "Knowledge-layer table to query." - }, - "limit": { - "type": "integer", - "default": 100, - "description": "Max rows to return, clamped to [0, 1000]." - } - }, - "required": ["table"] - }), - }, - Tool { - name: "search".into(), - description: "Unified code search: FTS5 exact/prefix matching, complemented by n-gram semantic vector search (restored in v0.2.5) that adds lexically-similar name recall when FTS results run short. Supports prefix matching.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "query": {"type": "string", "description": "Search query string"}, - "limit": {"type": "integer", "description": "Max results (default 20, max 100)"} - }, - "required": ["query"] - }), - }, - Tool { - name: "find_callers".into(), - description: "Find all symbols that call the specified function. Auto-adapts to callgraph readiness: uses fast index first, falls back to full analysis.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "symbol_name": {"type": "string", "description": "Name of the symbol to find callers for"}, - "file_filter": {"type": "string", "description": "Optional: absolute file path. Restricts the callee to the given file, disambiguating homonyms (same name across files/classes, e.g. __init__, run, main). Empty = aggregate all files (legacy behavior, may produce noise on common names)."} - }, - "required": ["symbol_name"] - }), - }, - Tool { - name: "find_callees".into(), - description: "Find all symbols called by the specified function. Auto-adapts to callgraph readiness.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "symbol_name": {"type": "string", "description": "Name of the symbol to find callees for"}, - "file_filter": {"type": "string", "description": "Optional: absolute file path. Restricts the caller to the given file, disambiguating homonyms. See find_callers doc."} - }, - "required": ["symbol_name"] - }), - }, - Tool { - name: "find_callers_by_entity".into(), - description: "Find all symbols that call the specified entity (Step 7, plan §7.2). Targets a single entity by its id, so homonyms (same name across files/classes, e.g. __init__, run, main) are never aggregated. Use the candidates returned by find_callers/find_callees when a bare name is ambiguous, or locate the entity first via find_symbol, then pass its id here.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "entity_id": {"type": "integer", "description": "Entity id from the entity table (e.g. from the candidates list of an ambiguous bare-name query, or from find_symbol output)."} - }, - "required": ["entity_id"] - }), - }, - Tool { - name: "find_callees_by_entity".into(), - description: "Find all symbols called by the specified entity (Step 7, plan §7.2). Targets a single entity by its id, so homonyms are never aggregated.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "entity_id": {"type": "integer", "description": "Entity id from the entity table (e.g. from the candidates list of an ambiguous bare-name query, or from find_symbol output)."} - }, - "required": ["entity_id"] - }), - }, - Tool { - name: "get_verifier_registry_status".into(), - description: "Report verifier subsystem health (Step 9, plan §9.2): whether the verifier registry is armed, which public claim types are supported, and whether the canonical evidence backend (entity/relation) has data for the project. Pass project_id=0 to skip the evidence probe.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "project_id": {"type": "integer", "description": "Project id; 0 skips the evidence backend probe."} - }, - "required": [] - }), - }, - Tool { - name: "shortest_path".into(), - description: "Find the shortest call-graph path between two functions. Heuristic/approximate: the call graph is built from name-matched edges, so the BFS path may not reflect true runtime dispatch. Accepts either symbol names (from/to) or explicit graph node IDs (from_id/to_id).".into(), - input_schema: json!({ - "type": "object", - "properties": { - "from": {"type": "string", "description": "Source symbol name (resolved to a node ID via locate_by_name)"}, - "to": {"type": "string", "description": "Target symbol name (resolved to a node ID via locate_by_name)"}, - "from_id": {"type": "integer", "description": "Explicit source graph node ID (bypasses name resolution)"}, - "to_id": {"type": "integer", "description": "Explicit target graph node ID (bypasses name resolution)"} - } - }), - }, - Tool { - name: "connected_components".into(), - description: "Find connected components in the call graph via BFS over name-matched relation edges. Heuristic/approximate: call edges are inferred from name matches, so reported components are a heuristic grouping of the module structure.".into(), - input_schema: json!({ "type": "object", "properties": {} }), - }, - Tool { - name: "get_entry_points".into(), - description: "Get likely entry points from the new schema (main/init/setup/run/handler). Returns symbol id, name, file path, and line.".into(), - input_schema: json!({ "type": "object", "properties": {} }), - }, - Tool { - name: "get_type_info".into(), - description: "Get type information for the project: type definitions (structs, enums, traits) and their reference counts. Optionally filter by type name. Returns JSON with types array.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "type_name": { - "description": "Optional: filter types by name (substring match)", - "type": "string" - } - } - }), - }, - Tool { - name: "get_routes".into(), - description: "Get HTTP routes registered in the project. Returns JSON with method, path, handler, file, and line for each route. Supports Gin, Echo, Chi, and net/http patterns.".into(), - input_schema: json!({ "type": "object", "properties": {} }), - }, - Tool { - name: "project_overview".into(), - description: "Get a comprehensive project overview: languages, modules/symbols, entry points, analysis progress, and ready features. Call this first after initialization.".into(), - input_schema: json!({ "type": "object", "properties": {} }), - }, - Tool { - name: "detect_ffi_boundaries".into(), - description: "Detect FFI (Foreign Function Interface) boundaries in the project. Returns language distribution, cross-language files, FFI-related symbols (extern, wasm, jni, cabi), and orphan symbols that may serve as FFI entry points.".into(), - input_schema: json!({ "type": "object", "properties": {} }), - }, - Tool { - name: "codescope_trace".into(), - description: "Explore a function's callers/callees recursively, or trace the shortest path between two functions. Use function_name+depth+direction for interactive exploration, or from+to for shortest path.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "function_name": {"type": "string", "description": "Starting function for interactive exploration"}, - "depth": {"type": "integer", "description": "How many levels to explore (default: 1, max: 5)"}, - "direction": {"type": "string", "description": "\"callers\", \"callees\", or \"both\" (default: \"both\")"}, - "from": {"type": "string", "description": "Source function for shortest path (legacy)"}, - "to": {"type": "string", "description": "Target function for shortest path (legacy)"} - } - }), - }, - Tool { - name: "count_tokens".into(), - description: "Estimate token count for a text string using DeepSeek's formula: ASCII*0.3 + non-ASCII*0.6.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "text": {"type": "string", "description": "Text to estimate tokens for"} - }, - "required": ["text"] - }), - }, - Tool { - name: "get_subgraph".into(), - description: "Fetch a local region of the code graph centered on a node: the center node plus its direct graph neighbors (1 hop). Returns {nodes:[...], total:N}. Optional node_types/edge_types are comma-separated integer id lists (e.g. \"0,1\") to filter by type. Use this to drill into a specific area instead of pulling the whole graph.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "node_id": {"type": "integer", "description": "Center graph node id (required)"}, - "radius": {"type": "integer", "description": "Hop radius (reserved, currently 1)"}, - "node_types": {"type": "string", "description": "Optional comma-separated node type ids, e.g. \"0,1\""}, - "edge_types": {"type": "string", "description": "Optional comma-separated edge type ids, e.g. \"1\""} - }, - "required": ["node_id"] - }), - }, - Tool { - name: "get_neighbors".into(), - description: "Fetch the direct neighbors (callers AND callees) of a graph node. Returns {neighbors:[{id,name,node_type,file_path,edge_type,direction}], total:N}. Optional edge_type filters to a single edge type id (-1 = all).".into(), - input_schema: json!({ - "type": "object", - "properties": { - "node_id": {"type": "integer", "description": "Graph node id (required)"}, - "edge_type": {"type": "integer", "description": "Edge type id filter (-1 = all, default -1)"}, - "radius": {"type": "integer", "description": "Hop radius (reserved, currently 1)"} - }, - "required": ["node_id"] - }), - }, - Tool { - name: "graph_query".into(), - description: "Query the code graph with a Cypher-like DSL. Syntax: MATCH (srcType[:srcName])-[edgeType]->(tgtType[:tgtName]). Node types: Function(0), Method(1), Class(2), Struct(3), Interface(4), Variable(5), Module(6), File(7). Edge types: References(0), Calls(1), Defines(2), Contains(3), Imports(4), Inherits(5). Empty type matches any. Multi-hop: edgeType*min..max, e.g. MATCH (Function)-[Calls*1..3]->(Function). Returns {results:[{source, edge, target}], total:N}.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "dsl": {"type": "string", "description": "Graph query DSL string, e.g. \"MATCH (Function:main)-[Calls]->(Method)\""} - }, - "required": ["dsl"] - }), - }, - Tool { - name: "get_graph".into(), - description: "Retrieve the COMPLETE code graph in paginated pages (required for large projects). Returns {totals:{nodes,edges}, nodes:[...], edges:[...], has_more:{nodes,edges}}. Page nodes via node_offset/node_limit and edges via edge_offset/edge_limit. Optional node_types/edge_types filter by comma-separated integer ids. Iterate while has_more is true to reconstruct the entire graph.".into(), - input_schema: json!({ - "type": "object", - "properties": { - "node_offset": {"type": "integer", "description": "Node page start offset (default 0)"}, - "node_limit": {"type": "integer", "description": "Max nodes per page (default 5000, max 50000)"}, - "edge_offset": {"type": "integer", "description": "Edge page start offset (default 0)"}, - "edge_limit": {"type": "integer", "description": "Max edges per page (default 20000, max 200000)"}, - "node_types": {"type": "string", "description": "Optional comma-separated node type ids to include"}, - "edge_types": {"type": "string", "description": "Optional comma-separated edge type ids to include"} - } - }), - }, - ] -} +// The tool catalog (input schemas + descriptions for every tool) lives in +// tools/catalog.rs — see plan/rules/code_rules.md (1000-line rule). +mod catalog; +pub use catalog::all_tools; + +// Documentation-liveness guards for README §9 (own file: 1000-line rule, see +// tools/clamp.rs for the same pattern). +#[cfg(test)] +mod docs_lint; // ─── Execute ──────────────────────────────────────────────────── pub fn execute(project_id: u64, tool_name: &str, args: &Value) -> String { let handler = match TOOL_HANDLERS.get(tool_name) { Some(h) => h, - None => return json!({"error": "Unknown tool"}).to_string(), + None => { + return json!({ + "error": format!( + "Unknown tool: {} [module=tools, method=execute]", + tool_name + ) + }) + .to_string(); + } }; handler(project_id, args) @@ -1885,6 +788,41 @@ mod tests { } } + /// The endpoint-taking tools resolve their arguments at runtime and reject + /// a call without them, but the catalog advertised no requirement at all, + /// so a client that read `tools/list` and called with `{}` got + /// "either 'from' (symbol name) or 'from_id' (node id) is required" — an + /// error the schema could have prevented. The rule is "the NAME or the ID", + /// which `required` cannot express, so it is stated as allOf/anyOf and + /// pinned here: dropping it would silently send clients back to guessing. + #[test] + fn test_endpoint_tools_advertise_their_endpoints() { + for name in ["shortest_path", "codescope_trace"] { + let tool = all_tools() + .into_iter() + .find(|t| t.name == name) + .unwrap_or_else(|| panic!("{name} missing from the catalog")); + let schema = &tool.input_schema; + assert!( + schema.get("anyOf").is_some() || schema.get("allOf").is_some(), + "{name} must declare that it needs an endpoint: {schema}" + ); + } + // shortest_path needs BOTH ends; codescope_trace needs either the + // exploration name or the two shortest-path ends. + let sp = all_tools() + .into_iter() + .find(|t| t.name == "shortest_path") + .unwrap(); + let any_of = sp.input_schema["allOf"].as_array().unwrap(); + assert_eq!( + any_of.len(), + 2, + "one anyOf per endpoint: {}", + sp.input_schema + ); + } + #[test] fn test_unknown_tool_returns_error() { let result = execute(0, "nonexistent_tool_xyz", &json!({})); @@ -1952,4 +890,102 @@ mod tests { ); } } + + /// Regression (2026-09-27 review, D1-1): README §5 used to claim + /// MCP clients "call `index_project` by name" while the tool was in + /// neither the catalog nor TOOL_HANDLERS, so such a call is an + /// Unknown-tool error. The documented contract is now "session-only + /// auto-index, not callable by name" — lock it in. + #[test] + fn test_index_project_is_not_callable_by_name() { + let tools = all_tools(); + assert!( + !tools.iter().any(|t| t.name == "index_project"), + "index_project must not be advertised in tools/list" + ); + assert!( + !TOOL_HANDLERS.contains_key("index_project"), + "index_project must not have a by-name handler" + ); + let result = execute(0, "index_project", &json!({"project_path": "/tmp"})); + assert!( + result.contains("Unknown tool"), + "tools/call index_project must report Unknown tool, got: {result}" + ); + } + + // ── Argument clamping ───────────────────────────────────────── + // + // These guards exist because the engine recurses `depth` levels and + // builds result sets of `limit` rows: an unclamped value used to reach + // the C++ side, where `as i32` silently truncated huge i64s into + // Clamp unit tests (depth/radius/edge-type/limit bounds) live in + // tools/clamp.rs next to the helpers they cover (1000-line rule). + + #[test] + fn test_json_or_string_arg_accepts_structured_and_pre_encoded() { + // The schema documents `claim` as a JSON object, but a pre-serialized + // string used to be the only shape accepted, so every caller had to + // double-encode it. Both work now; the string form is taken verbatim + // so existing callers keep working. + assert_eq!( + claim_json_from(&json!({"claim": {"type": "capability_exists"}})), + "{\"type\":\"capability_exists\"}" + ); + assert_eq!( + claim_json_from(&json!({"claim": "{\"type\":\"capability_exists\"}"})), + "{\"type\":\"capability_exists\"}" + ); + // A missing or wrongly-typed claim stays empty, so the handler reports + // the missing-argument error instead of sending garbage to the engine. + assert!(claim_json_from(&json!({})).is_empty()); + assert!(claim_json_from(&json!({"claim": 7})).is_empty()); + assert!(claim_json_from(&json!({"claim": null})).is_empty()); + + // detect_changes takes a JSON ARRAY of paths; a bare path (what its + // schema used to describe) is not valid JSON and meant "no changes". + assert_eq!( + json_or_string_arg( + &json!({"modified_files": ["a/b.rs"]}), + "modified_files", + "[]" + ), + "[\"a/b.rs\"]" + ); + assert_eq!( + json_or_string_arg( + &json!({"modified_files": "[\"a/b.rs\"]"}), + "modified_files", + "[]" + ), + "[\"a/b.rs\"]" + ); + assert_eq!(json_or_string_arg(&json!({}), "modified_files", "[]"), "[]"); + } + + #[test] + fn test_verifier_registry_project_id_honours_the_argument() { + // The schema documents `project_id` ("0 skips the evidence backend + // probe") and the handler used to throw the argument bag away, so the + // documented argument did nothing. Regression: the explicit argument + // wins, an absent one falls back to the server's project, and 0 is + // passed through as the documented "skip the probe" value rather than + // being treated as "absent". + let args = json!({"project_id": 7}); + assert_eq!(verifier_registry_project_id(3, &args), 7); + assert_eq!(verifier_registry_project_id(3, &json!({})), 3); + assert_eq!( + verifier_registry_project_id(3, &json!({"project_id": 0})), + 0 + ); + // A non-integer or negative value is not a usable id: fall back. + assert_eq!( + verifier_registry_project_id(3, &json!({"project_id": "7"})), + 3 + ); + assert_eq!( + verifier_registry_project_id(3, &json!({"project_id": -1})), + 3 + ); + } } diff --git a/server/tests/test_graph_ffi.rs b/server/tests/test_graph_ffi.rs index 27d48ce..1371fcb 100644 --- a/server/tests/test_graph_ffi.rs +++ b/server/tests/test_graph_ffi.rs @@ -9,7 +9,7 @@ // Test strategy mirrors test_knowledge_ffi.rs: initialize the engine + // create a project, then verify the JSON envelope shape on an empty DB. // The null/uninitialized-store case is covered by test_graph_ffi_null.rs -// (a separate binary so it cannot race with the engine_init/shutdown +// (a separate binary so it cannot race with the create/destroy // calls here — cargo runs #[test] functions in parallel threads sharing // the same global g_store). // @@ -25,13 +25,68 @@ use std::path::PathBuf; // Mirrors the declarations in server/src/ffi/mod.rs but kept local so this // integration test compiles as a standalone binary. +/// Opaque engine handle — the C ABI's `engine_t` (engine/include/engine.h). +#[repr(C)] +struct CodescopeEngine { + _private: [u8; 0], +} +type EngineHandle = *mut CodescopeEngine; + +/// The engine instance this test binary drives. TD-1 knife 3 made the ABI +/// handle-based, so the suite holds the handle here instead of relying on +/// process-global engine state (which the server owns in ffi/mod.rs). +static ENGINE: std::sync::atomic::AtomicPtr = + std::sync::atomic::AtomicPtr::new(std::ptr::null_mut()); + +/// The handle every stateful call below passes. +fn engine_handle() -> EngineHandle { + ENGINE.load(std::sync::atomic::Ordering::Acquire) +} + +/// Create the engine instance on `db_path`; returns 0 on success (the old +/// `engine_init` contract) and releases any instance a previous test left. +fn engine_open(db_path: *const c_char) -> i32 { + let handle = unsafe { engine_create(db_path) }; + let previous = ENGINE.swap(handle, std::sync::atomic::Ordering::AcqRel); + if !previous.is_null() { + unsafe { engine_destroy(previous) }; + } + if handle.is_null() { -1 } else { 0 } +} + +/// Release the engine instance. Safe when no instance is live. +fn engine_close() { + let handle = ENGINE.swap(std::ptr::null_mut(), std::sync::atomic::Ordering::AcqRel); + if !handle.is_null() { + unsafe { engine_destroy(handle) }; + } +} + unsafe extern "C" { - fn engine_init(db_path: *const c_char) -> i32; - fn engine_shutdown(); - fn engine_create_project(root_path: *const c_char, name: *const c_char) -> u64; - fn engine_find_shortest_path(project_id: u64, source_id: u64, target_id: u64) -> *mut c_char; - fn engine_locate_by_name(project_id: u64, name: *const c_char) -> *mut c_char; - fn engine_find_connected_components(project_id: u64) -> *mut c_char; + fn engine_create(db_path: *const c_char) -> EngineHandle; + fn engine_destroy(handle: EngineHandle); + fn engine_create_project( + handle: EngineHandle, + root_path: *const c_char, + name: *const c_char, + ) -> u64; + fn engine_index_file( + handle: EngineHandle, + project_id: u64, + file_path: *const c_char, + ) -> *mut c_char; + fn engine_find_shortest_path( + handle: EngineHandle, + project_id: u64, + source_id: u64, + target_id: u64, + ) -> *mut c_char; + fn engine_locate_by_name( + handle: EngineHandle, + project_id: u64, + name: *const c_char, + ) -> *mut c_char; + fn engine_find_connected_components(handle: EngineHandle, project_id: u64) -> *mut c_char; fn engine_free_string(ptr: *mut c_char); } @@ -57,7 +112,7 @@ fn take_string(ptr: *mut c_char) -> String { static COUNTER: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); /// The engine is a process-wide singleton (g_store). Rust runs tests in -/// parallel threads by default, so concurrent engine_init/engine_shutdown +/// parallel threads by default, so concurrent create/destroy /// from different tests races the singleton and aborts (SIGABRT). Serialize /// engine access with a global mutex: each test takes the guard as its /// first statement and drops it (RAII) when the test ends — equivalent to @@ -78,7 +133,7 @@ fn temp_db_path() -> PathBuf { } /// Test fixture: initialize the engine + create a project, returning the -/// project_id. The caller MUST call engine_shutdown() at the end. +/// project_id. The caller MUST call teardown_engine() at the end. fn setup_engine() -> u64 { let db_path = temp_db_path(); let _ = std::fs::remove_file(&db_path); @@ -86,12 +141,12 @@ fn setup_engine() -> u64 { let _ = std::fs::remove_file(format!("{}-shm", db_path.display())); let db_c = cstr(db_path.to_str().unwrap_or("/tmp/codescope_test.db")); - let rc = unsafe { engine_init(db_c.as_ptr()) }; - assert_eq!(rc, 0, "engine_init should return 0 on success"); + let rc = engine_open(db_c.as_ptr()); + assert_eq!(rc, 0, "engine_create should succeed"); let root_c = cstr("/tmp/test-project"); let name_c = cstr("test-graph-ffi"); - let pid = unsafe { engine_create_project(root_c.as_ptr(), name_c.as_ptr()) }; + let pid = unsafe { engine_create_project(engine_handle(), root_c.as_ptr(), name_c.as_ptr()) }; assert!( pid > 0, "engine_create_project should return a positive project_id" @@ -100,7 +155,7 @@ fn setup_engine() -> u64 { } fn teardown_engine() { - unsafe { engine_shutdown() }; + engine_close(); } // ── Tests: initialized engine, empty DB ───────────────────────── @@ -109,7 +164,7 @@ fn teardown_engine() { fn test_find_connected_components_empty_db_returns_envelope() { let _engine_guard = lock_engine(); let pid = setup_engine(); - let result = take_string(unsafe { engine_find_connected_components(pid) }); + let result = take_string(unsafe { engine_find_connected_components(engine_handle(), pid) }); teardown_engine(); let json: serde_json::Value = @@ -149,7 +204,7 @@ fn test_find_connected_components_zero_project_id_does_not_crash() { let _pid = setup_engine(); // project_id 0 does not exist; the inspector must not crash and must // still return the documented envelope. - let result = take_string(unsafe { engine_find_connected_components(0) }); + let result = take_string(unsafe { engine_find_connected_components(engine_handle(), 0) }); teardown_engine(); let json: serde_json::Value = @@ -174,7 +229,7 @@ fn test_find_shortest_path_zero_ids_returns_json() { let pid = setup_engine(); // source_id=target_id=0 cannot exist; the query engine must still // return valid JSON (empty path or error), not crash. - let result = take_string(unsafe { engine_find_shortest_path(pid, 0, 0) }); + let result = take_string(unsafe { engine_find_shortest_path(engine_handle(), pid, 0, 0) }); teardown_engine(); let json: serde_json::Value = @@ -191,7 +246,8 @@ fn test_locate_by_name_empty_db_returns_locations_array() { let _engine_guard = lock_engine(); let pid = setup_engine(); let name_c = cstr("nonexistent_symbol"); - let result = take_string(unsafe { engine_locate_by_name(pid, name_c.as_ptr()) }); + let result = + take_string(unsafe { engine_locate_by_name(engine_handle(), pid, name_c.as_ptr()) }); teardown_engine(); let json: serde_json::Value = @@ -209,3 +265,54 @@ fn test_locate_by_name_empty_db_returns_locations_array() { result ); } + +// ── Test: real index -> query (end-to-end) ────────────────────── +// +// The envelope-only tests above cannot catch a query path that returns a +// well-formed but wrong result. This one indexes a real Python file through +// the FFI boundary, then asserts the symbol is actually discoverable — the +// "index small project → query → verify" scenario code_rules §4 requires of +// integration tests. +#[test] +fn test_index_python_file_then_locate_symbol_end_to_end() { + let _engine_guard = lock_engine(); + let pid = setup_engine(); + + let src_path = std::env::temp_dir().join(format!( + "codescope_e2e_{}_{}.py", + std::process::id(), + COUNTER.fetch_add(1, std::sync::atomic::Ordering::SeqCst) + )); + std::fs::write(&src_path, "def codescope_e2e_probe(x):\n return x + 1\n") + .expect("write temp source file"); + + let path_c = cstr(src_path.to_str().unwrap()); + let index_result = + take_string(unsafe { engine_index_file(engine_handle(), pid, path_c.as_ptr()) }); + let index_json: serde_json::Value = + serde_json::from_str(&index_result).expect("index_file should return valid JSON"); + assert_eq!( + index_json["ok"], true, + "index_file should report ok:true, got: {}", + index_result + ); + + let name_c = cstr("codescope_e2e_probe"); + let locate_result = + take_string(unsafe { engine_locate_by_name(engine_handle(), pid, name_c.as_ptr()) }); + teardown_engine(); + let _ = std::fs::remove_file(&src_path); + + let json: serde_json::Value = + serde_json::from_str(&locate_result).expect("locate_by_name should return valid JSON"); + assert!( + json["total"].as_i64().unwrap_or(0) >= 1, + "indexed symbol must be discoverable, got: {}", + locate_result + ); + assert!( + locate_result.contains("codescope_e2e_probe"), + "location must name the indexed symbol, got: {}", + locate_result + ); +} diff --git a/server/tests/test_graph_ffi_null.rs b/server/tests/test_graph_ffi_null.rs index b602630..d30bb9e 100644 --- a/server/tests/test_graph_ffi_null.rs +++ b/server/tests/test_graph_ffi_null.rs @@ -1,25 +1,49 @@ // Null-store FFI tests for the graph path + connected-components functions. // // This file is a SEPARATE test binary from test_graph_ffi.rs on purpose. -// cargo runs each #[test] function in parallel threads that share the same -// process and therefore the same global C++ `g_store` singleton. If a -// null-store test ran in the same process as a test that calls -// engine_init(), the init could set g_store before the null-store test -// runs, turning a "null store → error JSON" assertion into a flaky failure. +// cargo runs each #[test] function in parallel threads, and the suite used to +// share one process-global engine: a test that called the old engine_init() +// could install state before a null-store assertion ran, turning its +// "null store → error JSON" expectation into a flaky failure. // -// By keeping every null-store assertion in a binary that NEVER calls -// engine_init(), g_store is guaranteed to be null for the whole process -// lifetime, so the error-JSON contract is deterministic. +// TD-1 knife 3 removed that class of flakiness from the design: the instance +// is passed per call, so this binary simply never creates one and every call +// below passes a null handle. The separation is kept so a stray instance +// cannot appear behind the assertions, and so the null contract is stated in +// one obvious place. // // The happy-path (initialized engine) tests live in test_graph_ffi.rs. use std::ffi::{CStr, CString}; use std::os::raw::c_char; +/// Opaque engine handle — the C ABI's `engine_t` (engine/include/engine.h). +#[repr(C)] +struct CodescopeEngine { + _private: [u8; 0], +} +type EngineHandle = *mut CodescopeEngine; + +/// This binary never creates an instance: every call below passes a null +/// handle, which is what an engine that was never `engine_create`d looks like +/// (TD-1 knife 3). Calling a tool before `init` takes the same path. +fn engine_handle() -> EngineHandle { + std::ptr::null_mut() +} + unsafe extern "C" { - fn engine_find_shortest_path(project_id: u64, source_id: u64, target_id: u64) -> *mut c_char; - fn engine_locate_by_name(project_id: u64, name: *const c_char) -> *mut c_char; - fn engine_find_connected_components(project_id: u64) -> *mut c_char; + fn engine_find_shortest_path( + handle: EngineHandle, + project_id: u64, + source_id: u64, + target_id: u64, + ) -> *mut c_char; + fn engine_locate_by_name( + handle: EngineHandle, + project_id: u64, + name: *const c_char, + ) -> *mut c_char; + fn engine_find_connected_components(handle: EngineHandle, project_id: u64) -> *mut c_char; fn engine_free_string(ptr: *mut c_char); } @@ -39,8 +63,8 @@ fn take_string(ptr: *mut c_char) -> String { #[test] fn test_find_connected_components_null_store_returns_error_json() { - // Do NOT call engine_init — g_store is null for this whole process. - let result = take_string(unsafe { engine_find_connected_components(1) }); + // Do NOT create an instance — every call here passes a null handle. + let result = take_string(unsafe { engine_find_connected_components(engine_handle(), 1) }); let json: serde_json::Value = serde_json::from_str(&result).expect("null store should still return valid JSON"); @@ -85,9 +109,9 @@ fn test_find_connected_components_null_store_returns_error_json() { #[test] fn test_find_shortest_path_null_store_returns_error_json() { - // Before engine_init the query engine is null; the C++ side returns - // {"path":[],"error":"not initialized"}. - let result = take_string(unsafe { engine_find_shortest_path(1, 100, 200) }); + // With a null handle the query engine is unreachable; the C++ side + // returns {"path":[],"error":"not initialized"}. + let result = take_string(unsafe { engine_find_shortest_path(engine_handle(), 1, 100, 200) }); let json: serde_json::Value = serde_json::from_str(&result).expect("null store should still return valid JSON"); @@ -107,7 +131,7 @@ fn test_find_shortest_path_null_store_returns_error_json() { #[test] fn test_locate_by_name_null_store_returns_error_json() { let name_c = cstr("does_not_matter"); - let result = take_string(unsafe { engine_locate_by_name(1, name_c.as_ptr()) }); + let result = take_string(unsafe { engine_locate_by_name(engine_handle(), 1, name_c.as_ptr()) }); let json: serde_json::Value = serde_json::from_str(&result).expect("null store should still return valid JSON"); diff --git a/server/tests/test_index_e2e.rs b/server/tests/test_index_e2e.rs new file mode 100644 index 0000000..058fea2 --- /dev/null +++ b/server/tests/test_index_e2e.rs @@ -0,0 +1,272 @@ +// End-to-end integration test: index a real fixture project through the FFI +// boundary, then assert SEMANTIC query results — not just the JSON envelope. +// +// Why this exists (REVIEW_0.2.7.md TEST-1): the other +// integration files (test_graph_ffi.rs, test_knowledge_ffi.rs) are mostly +// envelope checks on an empty database — they would still pass if the indexer +// wrote nothing and every query returned an empty array. code_rules.md §4 +// ("Integration tests must use real dependencies") requires the +// "index small project -> query -> verify" scenario, so this file: +// +// 1. initialises the engine on a fresh temp DB and creates a project whose +// root is the checked-in accuracy fixture (engine/tests/accuracy/fixtures/python), +// 2. indexes that directory via engine_index_project, +// 3. asserts engine_get_graph_stats reports real symbols, +// 4. asserts engine_find_definition resolves a symbol defined in a.py, +// 5. asserts engine_search_code finds a fixture file. +// +// The fixture is shared with the C++ accuracy gate, so the test does not add +// new fixture files. +// +// Like the sibling integration tests, Rust runs #[test] functions in parallel +// threads while the engine still keeps some state process-wide — the verifier +// registry, the async knowledge builder and the store's index-progress and +// parse-failure buffers, TD-1's documented residuals — so every test serialises +// engine access through ENGINE_LOCK. + +use std::ffi::{CStr, CString}; +use std::os::raw::c_char; +use std::path::{Path, PathBuf}; + +// ── Direct extern "C" bindings ────────────────────────────────── +// Mirrors the declarations in server/src/ffi/decls.rs but kept local so this +// integration test compiles as a standalone binary. + +/// Opaque engine handle — the C ABI's `engine_t` (engine/include/engine.h). +#[repr(C)] +struct CodescopeEngine { + _private: [u8; 0], +} +type EngineHandle = *mut CodescopeEngine; + +/// The engine instance this test binary drives. TD-1 knife 3 made the ABI +/// handle-based, so the suite holds the handle here instead of relying on +/// process-global engine state (which the server owns in ffi/mod.rs). +static ENGINE: std::sync::atomic::AtomicPtr = + std::sync::atomic::AtomicPtr::new(std::ptr::null_mut()); + +/// The handle every stateful call below passes. +fn engine_handle() -> EngineHandle { + ENGINE.load(std::sync::atomic::Ordering::Acquire) +} + +/// Create the engine instance on `db_path`; returns 0 on success (the old +/// `engine_init` contract) and releases any instance a previous test left. +fn engine_open(db_path: *const c_char) -> i32 { + let handle = unsafe { engine_create(db_path) }; + let previous = ENGINE.swap(handle, std::sync::atomic::Ordering::AcqRel); + if !previous.is_null() { + unsafe { engine_destroy(previous) }; + } + if handle.is_null() { -1 } else { 0 } +} + +/// Release the engine instance. Safe when no instance is live. +fn engine_close() { + let handle = ENGINE.swap(std::ptr::null_mut(), std::sync::atomic::Ordering::AcqRel); + if !handle.is_null() { + unsafe { engine_destroy(handle) }; + } +} + +unsafe extern "C" { + fn engine_create(db_path: *const c_char) -> EngineHandle; + fn engine_destroy(handle: EngineHandle); + fn engine_create_project( + handle: EngineHandle, + root_path: *const c_char, + name: *const c_char, + ) -> u64; + fn engine_index_project( + handle: EngineHandle, + project_id: u64, + dir_path: *const c_char, + language_filter: *const c_char, + ) -> *mut c_char; + fn engine_get_graph_stats(handle: EngineHandle, project_id: u64) -> *mut c_char; + fn engine_find_definition( + handle: EngineHandle, + project_id: u64, + symbol_name: *const c_char, + file_filter: *const c_char, + ) -> *mut c_char; + fn engine_build_fts(handle: EngineHandle, project_id: u64) -> *mut c_char; + fn engine_search_code( + handle: EngineHandle, + project_id: u64, + query: *const c_char, + limit: i32, + ) -> *mut c_char; + fn engine_free_string(ptr: *mut c_char); +} + +// ── Helpers ────────────────────────────────────────────────────── + +fn cstr(s: &str) -> CString { + let sanitized: String = s.replace('\0', "\u{FFFD}"); + CString::new(sanitized).unwrap_or_else(|_| CString::new("").unwrap()) +} + +/// Take ownership of a heap-allocated C string and free it via the engine's +/// allocator. Returns an owned String. +fn take_string(ptr: *mut c_char) -> String { + if ptr.is_null() { + return String::new(); + } + let s = unsafe { CStr::from_ptr(ptr).to_string_lossy().into_owned() }; + unsafe { engine_free_string(ptr) }; + s +} + +/// Parse the engine's JSON reply, failing the test with the raw payload when +/// it is not valid JSON (that alone is a contract violation). +fn parse(context: &str, raw: &str) -> serde_json::Value { + serde_json::from_str(raw) + .unwrap_or_else(|e| panic!("{context} returned invalid JSON ({e}): {raw}")) +} + +/// Serialize engine access: the engine is a process-wide singleton and cargo +/// runs #[test] functions in parallel threads. +static ENGINE_LOCK: std::sync::OnceLock> = std::sync::OnceLock::new(); +fn lock_engine() -> std::sync::MutexGuard<'static, ()> { + ENGINE_LOCK + .get_or_init(|| std::sync::Mutex::new(())) + .lock() + .unwrap_or_else(|e| e.into_inner()) +} + +static COUNTER: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + +fn temp_db_path() -> PathBuf { + let pid = std::process::id(); + let n = COUNTER.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + std::env::temp_dir().join(format!("codescope_test_index_e2e_{pid}_{n}.db")) +} + +/// Stage the shared Python accuracy fixture into a fresh temp directory and +/// return its path. +/// +/// The fixture SOURCE lives at `engine/tests/accuracy/fixtures/python`, but the +/// indexer deliberately filters test-corpus paths out of the graph +/// (`buildGraph`'s `%/tests/%` exclusion in `store_graph.cpp`: "AI only needs +/// production code"). Indexing the fixture in place would therefore yield +/// `semantic_records > 0` but `entity == 0`, which tests the filter rather than +/// the pipeline. Copying the two fixture files (not adding new ones) into a +/// temp dir keeps the shared fixture as the single source of truth while +/// exercising a real production-path index. +fn stage_python_fixture() -> PathBuf { + let src = + Path::new(env!("CARGO_MANIFEST_DIR")).join("../engine/tests/accuracy/fixtures/python"); + let dst = std::env::temp_dir().join(format!( + "codescope_test_index_e2e_fixture_{}_{}", + std::process::id(), + COUNTER.fetch_add(1, std::sync::atomic::Ordering::SeqCst) + )); + let _ = std::fs::remove_dir_all(&dst); + std::fs::create_dir_all(&dst).expect("create staged fixture dir"); + for name in ["a.py", "b.py"] { + std::fs::copy(src.join(name), dst.join(name)) + .unwrap_or_else(|e| panic!("copy fixture {name} from {src:?}: {e}")); + } + dst +} + +/// Test fixture: initialise the engine + create a project rooted at the +/// fixture dir. The caller MUST call engine_close() at the end. +fn setup_engine(root: &Path) -> u64 { + let db_path = temp_db_path(); + let _ = std::fs::remove_file(&db_path); + let _ = std::fs::remove_file(format!("{}-wal", db_path.display())); + let _ = std::fs::remove_file(format!("{}-shm", db_path.display())); + + let db_c = cstr( + db_path + .to_str() + .unwrap_or("/tmp/codescope_test_index_e2e.db"), + ); + let rc = engine_open(db_c.as_ptr()); + assert_eq!(rc, 0, "engine_create should succeed"); + + let root_c = cstr(root.to_str().expect("fixture path is valid UTF-8")); + let name_c = cstr("test-index-e2e"); + let pid = unsafe { engine_create_project(engine_handle(), root_c.as_ptr(), name_c.as_ptr()) }; + assert!(pid > 0, "engine_create_project should return a positive id"); + pid +} + +// ── Test: index fixture -> query -> verify ────────────────────── + +#[test] +fn test_index_fixture_project_queries_are_semantic() { + let _engine_guard = lock_engine(); + let fixture = stage_python_fixture(); + let pid = setup_engine(&fixture); + + // Step 1: index the fixture directory. + let dir_c = cstr(fixture.to_str().unwrap()); + let indexed = take_string(unsafe { + engine_index_project(engine_handle(), pid, dir_c.as_ptr(), std::ptr::null()) + }); + let index_json = parse("engine_index_project", &indexed); + assert_eq!( + index_json["ok"], true, + "indexing the fixture must report ok:true, got: {indexed}" + ); + assert!( + index_json["files_indexed"].as_u64().unwrap_or(0) >= 2, + "both fixture files (a.py, b.py) must be indexed, got: {indexed}" + ); + + // Step 2: the graph must hold real symbols. This is the assertion the + // envelope-only tests were missing: a broken writer reports ok:true with + // zero nodes. + let stats = take_string(unsafe { engine_get_graph_stats(engine_handle(), pid) }); + let stats_json = parse("engine_get_graph_stats", &stats); + assert!( + stats_json["total_nodes"].as_u64().unwrap_or(0) > 0, + "an indexed fixture must produce nodes, got: {stats}" + ); + assert!( + stats_json["files_with_symbols"].as_u64().unwrap_or(0) > 0, + "at least one file must yield symbols, got: {stats}" + ); + + // Step 3: find_definition must resolve a symbol that only exists in the + // fixture source (defined in b.py, called from a.py). + let name_c = cstr("bravo"); + let found = take_string(unsafe { + engine_find_definition(engine_handle(), pid, name_c.as_ptr(), std::ptr::null()) + }); + let found_json = parse("engine_find_definition", &found); + assert!( + found_json["total"].as_u64().unwrap_or(0) >= 1, + "find_definition('bravo') must hit the indexed symbol, got: {found}" + ); + assert!( + found.contains("bravo"), + "the definition payload must name the symbol, got: {found}" + ); + + // Step 4: search_code must return the fixture file holding the query term. + // FTS is built during indexing, but rebuild it explicitly so the test does + // not depend on index-time ordering. + let fts = take_string(unsafe { engine_build_fts(engine_handle(), pid) }); + let fts_json = parse("engine_build_fts", &fts); + assert_eq!(fts_json["ok"], true, "build_fts must succeed, got: {fts}"); + + let query_c = cstr("alpha"); + let hits = + take_string(unsafe { engine_search_code(engine_handle(), pid, query_c.as_ptr(), 10) }); + let hits_json = parse("engine_search_code", &hits); + assert!( + hits_json["total"].as_u64().unwrap_or(0) >= 1, + "search_code('alpha') must find the fixture symbol, got: {hits}" + ); + assert!( + hits.contains("a.py"), + "the search hit must point at the fixture file, got: {hits}" + ); + + engine_close(); + let _ = std::fs::remove_dir_all(&fixture); +} diff --git a/server/tests/test_knowledge_ffi.rs b/server/tests/test_knowledge_ffi.rs index df937b7..02b3b51 100644 --- a/server/tests/test_knowledge_ffi.rs +++ b/server/tests/test_knowledge_ffi.rs @@ -8,10 +8,10 @@ // // Test strategy: // 1. Create a fresh SQLite DB in a temp directory. -// 2. Initialize the engine (engine_init). +// 2. Create the engine instance (engine_create). // 3. Create a project (engine_create_project). // 4. Call each FFI function and assert on the JSON output shape. -// 5. Shut down the engine (engine_shutdown) to release the DB lock. +// 5. Destroy the instance (engine_destroy) to release the DB lock. // // The tests are robust to empty databases: they verify JSON structure and // field presence, not specific verdicts that would depend on indexed code. @@ -24,14 +24,71 @@ use std::path::PathBuf; // These mirror the declarations in server/src/ffi/mod.rs but are kept // local so this integration test compiles as a standalone binary. +/// Opaque engine handle — the C ABI's `engine_t` (engine/include/engine.h). +#[repr(C)] +struct CodescopeEngine { + _private: [u8; 0], +} +type EngineHandle = *mut CodescopeEngine; + +/// The engine instance this test binary drives. TD-1 knife 3 made the ABI +/// handle-based, so the suite holds the handle here instead of relying on +/// process-global engine state (which the server owns in ffi/mod.rs). +static ENGINE: std::sync::atomic::AtomicPtr = + std::sync::atomic::AtomicPtr::new(std::ptr::null_mut()); + +/// The handle every stateful call below passes. +fn engine_handle() -> EngineHandle { + ENGINE.load(std::sync::atomic::Ordering::Acquire) +} + +/// Create the engine instance on `db_path`; returns 0 on success (the old +/// `engine_init` contract) and releases any instance a previous test left. +fn engine_open(db_path: *const c_char) -> i32 { + let handle = unsafe { engine_create(db_path) }; + let previous = ENGINE.swap(handle, std::sync::atomic::Ordering::AcqRel); + if !previous.is_null() { + unsafe { engine_destroy(previous) }; + } + if handle.is_null() { -1 } else { 0 } +} + +/// Release the engine instance. Safe when no instance is live. +fn engine_close() { + let handle = ENGINE.swap(std::ptr::null_mut(), std::sync::atomic::Ordering::AcqRel); + if !handle.is_null() { + unsafe { engine_destroy(handle) }; + } +} + unsafe extern "C" { - fn engine_init(db_path: *const c_char) -> i32; - fn engine_shutdown(); - fn engine_create_project(root_path: *const c_char, name: *const c_char) -> u64; - fn engine_verify_integrity(project_id: u64) -> *mut c_char; - fn engine_verify_claim(project_id: u64, claim_json: *const c_char) -> *mut c_char; - fn engine_verify_summary(project_id: u64, text: *const c_char) -> *mut c_char; - fn engine_explain_module(project_id: u64, module_name: *const c_char) -> *mut c_char; + fn engine_create(db_path: *const c_char) -> EngineHandle; + fn engine_destroy(handle: EngineHandle); + fn engine_create_project( + handle: EngineHandle, + root_path: *const c_char, + name: *const c_char, + ) -> u64; + fn engine_verify_integrity( + handle: EngineHandle, + project_id: u64, + max_findings: i32, + ) -> *mut c_char; + fn engine_verify_claim( + handle: EngineHandle, + project_id: u64, + claim_json: *const c_char, + ) -> *mut c_char; + fn engine_verify_summary( + handle: EngineHandle, + project_id: u64, + text: *const c_char, + ) -> *mut c_char; + fn engine_explain_module( + handle: EngineHandle, + project_id: u64, + module_name: *const c_char, + ) -> *mut c_char; fn engine_free_string(ptr: *mut c_char); } @@ -59,7 +116,7 @@ fn take_string(ptr: *mut c_char) -> String { static COUNTER: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); /// The engine is a process-wide singleton (g_store). Rust runs tests in -/// parallel threads by default, so concurrent engine_init/engine_shutdown +/// parallel threads by default, so concurrent create/destroy /// from different tests races the singleton and aborts (SIGABRT). Serialize /// engine access with a global mutex held from setup_engine until /// teardown_engine — this matches `--test-threads=1` semantics without @@ -80,7 +137,7 @@ fn temp_db_path() -> PathBuf { } /// Test fixture: initialize the engine + create a project, returning the -/// project_id. The caller is responsible for calling engine_shutdown() +/// project_id. The caller is responsible for calling engine_close() /// at the end of the test. fn setup_engine() -> u64 { let db_path = temp_db_path(); @@ -90,12 +147,12 @@ fn setup_engine() -> u64 { let _ = std::fs::remove_file(format!("{}-shm", db_path.display())); let db_c = cstr(db_path.to_str().unwrap_or("/tmp/codescope_test.db")); - let rc = unsafe { engine_init(db_c.as_ptr()) }; - assert_eq!(rc, 0, "engine_init should return 0 on success"); + let rc = engine_open(db_c.as_ptr()); + assert_eq!(rc, 0, "engine_create should succeed"); let root_c = cstr("/tmp/test-project"); let name_c = cstr("test-knowledge"); - let pid = unsafe { engine_create_project(root_c.as_ptr(), name_c.as_ptr()) }; + let pid = unsafe { engine_create_project(engine_handle(), root_c.as_ptr(), name_c.as_ptr()) }; assert!( pid > 0, "engine_create_project should return a positive project_id" @@ -106,7 +163,7 @@ fn setup_engine() -> u64 { /// Teardown: shut down the engine to release the SQLite EXCLUSIVE lock /// so the next test can create its own DB. fn teardown_engine() { - unsafe { engine_shutdown() }; + engine_close(); } // ── Tests ──────────────────────────────────────────────────────── @@ -121,7 +178,8 @@ fn test_verify_summary_parses_claims() { // extract these claims and the verifiers should return a verdict. let text = "CodeScope supports incremental indexing and is thread-safe"; let text_c = cstr(text); - let result = take_string(unsafe { engine_verify_summary(pid, text_c.as_ptr()) }); + let result = + take_string(unsafe { engine_verify_summary(engine_handle(), pid, text_c.as_ptr()) }); teardown_engine(); @@ -162,7 +220,8 @@ fn test_verify_summary_aggregates_trust_score() { let pid = setup_engine(); let text = "This library is memory-safe, zero-copy, and lock-free."; let text_c = cstr(text); - let result = take_string(unsafe { engine_verify_summary(pid, text_c.as_ptr()) }); + let result = + take_string(unsafe { engine_verify_summary(engine_handle(), pid, text_c.as_ptr()) }); teardown_engine(); @@ -198,7 +257,8 @@ fn test_verify_summary_empty_text() { let _engine_guard = lock_engine(); let pid = setup_engine(); let text_c = cstr(""); - let result = take_string(unsafe { engine_verify_summary(pid, text_c.as_ptr()) }); + let result = + take_string(unsafe { engine_verify_summary(engine_handle(), pid, text_c.as_ptr()) }); teardown_engine(); @@ -218,7 +278,8 @@ fn test_verify_claim_single_capability() { let pid = setup_engine(); let claim_json = r#"{"type":"capability_exists","subject":"incremental indexing","predicate":"implemented_by","object":"engine","scope":"repository","source_kind":"manual","source_ref":"test-1"}"#; let claim_c = cstr(claim_json); - let result = take_string(unsafe { engine_verify_claim(pid, claim_c.as_ptr()) }); + let result = + take_string(unsafe { engine_verify_claim(engine_handle(), pid, claim_c.as_ptr()) }); teardown_engine(); @@ -262,7 +323,8 @@ fn test_verify_claim_empty_json() { let _engine_guard = lock_engine(); let pid = setup_engine(); let claim_c = cstr(""); - let result = take_string(unsafe { engine_verify_claim(pid, claim_c.as_ptr()) }); + let result = + take_string(unsafe { engine_verify_claim(engine_handle(), pid, claim_c.as_ptr()) }); teardown_engine(); @@ -283,7 +345,8 @@ fn test_verify_claim_missing_subject() { // Valid JSON but missing the required "subject" field. let claim_json = r#"{"type":"capability_exists"}"#; let claim_c = cstr(claim_json); - let result = take_string(unsafe { engine_verify_claim(pid, claim_c.as_ptr()) }); + let result = + take_string(unsafe { engine_verify_claim(engine_handle(), pid, claim_c.as_ptr()) }); teardown_engine(); @@ -301,7 +364,8 @@ fn test_explain_module_returns_knowledge_card() { let _engine_guard = lock_engine(); let pid = setup_engine(); let module_c = cstr("engine"); - let result = take_string(unsafe { engine_explain_module(pid, module_c.as_ptr()) }); + let result = + take_string(unsafe { engine_explain_module(engine_handle(), pid, module_c.as_ptr()) }); teardown_engine(); @@ -349,7 +413,8 @@ fn test_explain_module_empty_name() { let _engine_guard = lock_engine(); let pid = setup_engine(); let module_c = cstr(""); - let result = take_string(unsafe { engine_explain_module(pid, module_c.as_ptr()) }); + let result = + take_string(unsafe { engine_explain_module(engine_handle(), pid, module_c.as_ptr()) }); teardown_engine(); @@ -370,7 +435,7 @@ fn test_explain_module_empty_name() { fn test_verify_integrity_returns_findings() { let _engine_guard = lock_engine(); let pid = setup_engine(); - let result = take_string(unsafe { engine_verify_integrity(pid) }); + let result = take_string(unsafe { engine_verify_integrity(engine_handle(), pid, 200) }); teardown_engine(); @@ -399,4 +464,83 @@ fn test_verify_integrity_returns_findings() { "verify_integrity output should contain a numeric 'total', got: {}", result ); + + // New in the max_findings contract: `truncated` is the completeness + // signal and `limit` echoes the applied cap. `truncated` must be false + // here — the default cap (200) is above the fixture's finding count. + assert!( + json.get("truncated") + .map(|v| v.is_boolean()) + .unwrap_or(false), + "verify_integrity output should contain a boolean 'truncated', got: {}", + result + ); + assert!( + json.get("limit").map(|v| v.is_number()).unwrap_or(false), + "verify_integrity output should contain a numeric 'limit', got: {}", + result + ); + assert_eq!( + json["limit"].as_i64(), + Some(200), + "limit should echo the requested cap" + ); + assert_eq!( + json["truncated"].as_bool(), + Some(false), + "a complete findings array must not report truncated, got: {}", + result + ); + let findings_len = json["findings"].as_array().map(|a| a.len()).unwrap_or(0); + assert!( + findings_len <= 200, + "findings array must respect the cap ({} > 200)", + findings_len + ); +} + +#[test] +fn test_verify_integrity_truncates_at_max_findings() { + let _engine_guard = lock_engine(); + let pid = setup_engine(); + let result = take_string(unsafe { engine_verify_integrity(engine_handle(), pid, 1) }); + + teardown_engine(); + + let json: serde_json::Value = + serde_json::from_str(&result).expect("verify_integrity should return valid JSON"); + + // With max_findings=1 the array must be capped at one element and the + // truncation flag must fire. `total` still reports the full verdict + // count (including Supported, which are never listed as findings). + assert_eq!( + json["limit"].as_i64(), + Some(1), + "limit should echo the requested cap, got: {}", + result + ); + let findings_len = json["findings"].as_array().map(|a| a.len()).unwrap_or(0); + assert!( + findings_len <= 1, + "max_findings=1 must cap the findings array, got {} entries", + findings_len + ); + let total = json["total"].as_i64().unwrap_or(0); + if total > 1 { + assert_eq!( + json["truncated"].as_bool(), + Some(true), + "a capped array must report truncated when total={} > 1 finding was emitted, got: {}", + total, + result + ); + } + // total counts verdicts (Supported included), so it must be at least + // the number of listed findings. + assert!( + total as usize >= findings_len, + "total ({}) must be >= findings length ({})", + total, + findings_len + ); } diff --git a/skills/skills.md b/skills/skills.md index 0f3896d..5fa37dc 100644 --- a/skills/skills.md +++ b/skills/skills.md @@ -28,7 +28,7 @@ CodeScope parses source code into a unified AST IR, builds a call graph + refere | `relation` | Inter-entity relations (containment / definition) | | `document` | README / Architecture.md / comment documents | -> **`module_summary.role` is a multi-signal fusion classifier (v0.2.1).** It auto-populates via CASE in `engine/src/model/state_builder.cpp`, fusing call-graph counts with two signals the call graph cannot give: `pub_count` (`entity.visibility=1`, pub/public/export per language Visitor — distinguishes external interface layers from internal) and `entry_reachable` (`graph_nodes.is_entry_point`). 8 roles by priority: `test` → `api` → `entry` → `core` → `utility` → `business` → `dead` → `infra` (true fallback). Thresholds are `constexpr` in `state_builder.h` — retune against `bun` if `infra > 30%`. See `docs/dev_plans/role_classifier_plan.md` (includes the v0.2.1 threshold retune log). Treat `role` as a fusion-informed hint, not a verdict. +> **`module_summary.role` is a multi-signal fusion classifier (v0.2.1).** It auto-populates via CASE in `engine/src/model/state_builder.cpp`, fusing call-graph counts with two signals the call graph cannot give: `pub_count` (`entity.visibility=1`, pub/public/export per language Visitor — distinguishes external interface layers from internal) and `entry_reachable` (`graph_nodes.is_entry_point`). 8 roles by priority: `test` → `api` → `entry` → `core` → `utility` → `business` → `dead` → `infra` (true fallback). Thresholds are `constexpr` in `state_builder.h` — retune against `bun` if `infra > 30%`. Treat `role` as a fusion-informed hint, not a verdict. --- @@ -67,7 +67,7 @@ CodeScope parses source code into a unified AST IR, builds a call graph + refere | Tool | Status | Workaround | |------|--------|------------| | `get_hotspots` | ❌ no MCP tool | (hotspots.sh is stale) | -| `get_communities` | ⚠️ C++ has it, MCP doesn't | `connected_components` as lightweight alternative | +| `get_communities` | ✅ Available (label propagation; summary-first) | `connected_components` for a coarser grouping | | `locate_code` | ❌ not implemented | `explain_symbol` | > `graph_query` IS implemented (TOOL_HANDLERS in server/src/tools/mod.rs) — earlier docs listed it as "not wired"; that is stale. Use it for custom graph-pattern queries. @@ -79,22 +79,22 @@ CodeScope parses source code into a unified AST IR, builds a call graph + refere | Variable | Default | Description | |----------|---------|-------------| | `CODESCOPE_DB_PATH` | `.codescope/codescope.db` | SQLite database path | -| `CODESCOPE_INDEX_MODE` | `normal` | Index mode: `fast` / `normal` / `strict`（NORMAL 默认；见下方 Index modes） | -| `CODESCOPE_WORKERS` | `min(hw,8)` | 解析 worker 线程数（`kDefaultParseWorkers=8`，并行索引路径为模块数×动态分配） | -| `CODESCOPE_SKIP_ASYNC` | (unset) | 设为 1 跳过异步 model/state/FTS 阶段（仅并行调度路径设置） | -| `CODESCOPE_PROFILE_RESOLVER` | (unset) | 启用 resolver 分阶段计时（输出 `[module=resolver, method=run]` 明细） | +| `CODESCOPE_INDEX_MODE` | `normal` | Index mode: `fast` / `normal` / `deep` (`normal` is the default) plus the discovery-only `strict`; see Index modes below | +| `CODESCOPE_WORKERS` | `min(hw,8)` | parse-worker threads (`kDefaultParseWorkers=8`; the parallel index path scales with module count) | +| `CODESCOPE_SKIP_ASYNC` | (unset) | set to 1 to skip the async model/state/FTS phases (set only on the parallel scheduling path) | +| `CODESCOPE_PROFILE_RESOLVER` | (unset) | enable per-phase resolver timing (prints `[module=resolver, method=run]` detail) | | `CODESCOPE_VERBOSE` | `1` | Set 0 to disable batch logs | | `CODESCOPE_MAX_FILE_SIZE` | 5MB | Max indexed file size | ## Index modes -| Mode | 枚举值 | 额外剪枝 | FTS | 用途 | +| Mode | Enum value | Extra pruning | FTS | Purpose | |------|--------|----------|-----|------| -| `fast` | `FAST` | ✅ 额外跳过 logs/.output/测试报告等 11 类目录 + 4 类缓存文件（`fast_extra_skip_dirs_`/`fast_extra_filenames_`，见 filter_policy.cpp） | ❌ 跳过 | 最快，数据≈全量（对源码干净项目几乎无差异） | -| `normal` | `NORMAL` | 仅基础 skip 表 | ✅ | 默认 | -| `strict` | `STRICT` | 基础 skip + detectLanguage 白名单 gate（仅索引源码文件） | ✅ | 最严格，数据最精简 | +| `fast` | `FAST` | ✅ also skips 11 directory classes (logs/.output/test reports) + 4 cache-file classes (`fast_extra_skip_dirs_`/`fast_extra_filenames_`, see filter_policy.cpp) | ❌ skipped | fastest; data ≈ full (almost no difference on a clean source tree) | +| `normal` | `NORMAL` | the base skip table only | ✅ | default | +| `strict` | `STRICT` | base skips + the detectLanguage whitelist gate (source files only) | ✅ | smallest dataset; affects discovery only | -> **已知问题（2026-08-11，已修复）**：fast 模式此前与 normal 几乎无差异——`fast_extra_skip_dirs_` 为空集（预留未实现），唯一差异是跳过 FTS。已补全剪枝集合并修复 `setMode()` 未重建 active_skip_dirs_ 的 bug。详见 `docs/optimization/perf-full-index-2026-08-11.md` §9/§10。 +> **Known issue (2026-08-11, fixed)**: fast mode used to differ from normal only by skipping FTS — `fast_extra_skip_dirs_` was an empty set (reserved, never implemented). The pruning sets are complete now and the `setMode()` bug (it did not rebuild `active_skip_dirs_`) is fixed. See `docs/optimization/perf-full-index-2026-08-11.zh.md` §9/§10 (Chinese). ## Supported languages diff --git a/tests/token_savings_report.md b/tests/token_savings_report.md deleted file mode 100644 index 46cde10..0000000 --- a/tests/token_savings_report.md +++ /dev/null @@ -1,20 +0,0 @@ -# Token Savings Report - -> CodeScope vs. reading raw source files — measured across real analysis scenarios. -> Hardware: Apple M3 Max, 36 GB RAM. OS: macOS. - -## Measured Savings - -| Scenario | CodeScope (tokens) | Raw Source (tokens) | Savings | -|----------|-------------------|---------------------|---------| -| Find function definition | ~21 | ~2,265 | **99.1%** | -| Trace callers | ~18 | ~2,000 | **99.1%** | -| Architecture overview | ~32 | ~1,875 | **98.3%** | -| Function analysis | ~43 | ~4,733 | **99.1%** | -| Symbol search | ~23 | ~958 | **97.6%** | -| USB driver subsystem | ~250 | ~24,000 | **99.0%** | -| Linux kernel scheduler | ~180 | ~15,000 | **98.8%** | -| COW process trace | ~85 | ~8,500 | **99.0%** | -| **Average** | **~81** | **~7,416** | **98.9%** | - -Chinese version: `docs/zh/token_savings_report.md`