From 2e1c41a368e99c1861fd231240300be10dffd5b4 Mon Sep 17 00:00:00 2001 From: odieyang <520yangziqi@gmail.com> Date: Sun, 2 Aug 2026 17:55:26 -0700 Subject: [PATCH 1/2] docs(interview): audit which ledger figures are unused and which changes have a before and after MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A read-only pass over the brief's §4 ledger and the git history, answering two questions: which quotable figures have not reached the resume material, and which changes can be told as "I made a change, here is the outcome". Ledger usage, 77 rows: 46 already used, 23 unused but safe, 8 unused and unsafe. Worth knowing that rows 2 and 3 -- the edge counts, the figure an outside reader would most readily take for a headline -- appear in the resume material only inside the forbidden-phrasings table, as an example of what not to write. That is deliberate. The unused figures are re-tiered by whether they can carry a bullet on their own (A), need a qualifying sentence (B, interview ammunition only), or are scale figures (C). Tier C is completed from the whole repository rather than only from the ledger, since scale is what a non-technical reader reads first and the resume material used it least. Improvement chains are only counted when all four elements are present: a sourced before-state, a change with a file:line or commit, a sourced after-state, and causality recorded in code or in a commit rather than inferred afterwards. Each is classified, and the classification is the point: - Engineering, 4. Unifying the symbol-resolution rule (verified citations 48-54 -> 62-67, redaction markers 17-21 -> 9-13, symbol-form citations surviving for the first time). Hydrating graph evidence (L3 call-graph contribution -0.0067 -> +0.0226). Filtering test code out of retrieval (47% of retrieved candidates -> 0%). And call resolution, where the commit measured the downstream effect itself and recorded it as zero -- that one cannot be told as an outcome. - Measurement correction, 5. Symmetric scoring, the GRAPH_TOOLS set, the B3.5 arm, dropping groundedness from the composite, and scoring an uncited answer zero. - Parameter change, 4. One of them is unusable: the comment behind the rerank passage cap quotes latency figures that were observed once and never benchmarked, and this project quotes no latency anywhere. Engineering chains are few and that is the project's shape rather than a gap in the search. The dense iteration is almost entirely about how things are measured. Measurement corrections are not moved into the engineering column to improve the ratio -- calling one an optimisation invites "how much did it improve", and the honest answer is that the metric did not move, the earlier measurement was wrong. Seven chains are listed separately as "suspected, evidence incomplete" so they stay out of the resume material. --- docs/interview/DCODE_DATA_INVENTORY.md | 439 +++++++++++++++++++++++++ 1 file changed, 439 insertions(+) create mode 100644 docs/interview/DCODE_DATA_INVENTORY.md diff --git a/docs/interview/DCODE_DATA_INVENTORY.md b/docs/interview/DCODE_DATA_INVENTORY.md new file mode 100644 index 0000000..1b6d0bf --- /dev/null +++ b/docs/interview/DCODE_DATA_INVENTORY.md @@ -0,0 +1,439 @@ +# Dcode 数据盘点:还有哪些数字没用、哪些能构成「改进 → 成效」 + +> **这是一次只读盘点。**目的有两个:一是把 `DCODE_TECH_BRIEF.md` §4 台账逐条过一遍,找出还没进简历素材、但可以安全使用的数字;二是去 git 历史和产物里找**有前后对比**的改动,判断哪些能写成完整的「我做了改进 → 产生了成效」。 +> +> 没有改代码、没有改产物、没有重跑评测。本次新查证的数字列在文末,按 A/B/C/D 标来源等级。 + +--- + +## Step 1 — 台账使用率盘点 + +台账共 **77 行**(§4.1–§4.7)。逐条对照 `DCODE_RESUME_ASSETS_CN.md` 判定。 + +### 1.1 分区汇总 + +| 分区 | 行数 | 已用 | 未用可安全使用 | 未用且不该用 | +|---|---:|---:|---:|---:| +| §4.1 语料与索引规模 | 13 | 5 | 6 | 2 | +| §4.2 检索与模型参数 | 13 | 8 | 5 | 0 | +| §4.3 Agent 编排参数 | 10 | 6 | 4 | 0 | +| §4.4 评测协议与题集 | 9 | 6 | 2 | 1 | +| §4.5 评测结果 | 22 | 16 | 6 | 0 | +| §4.6 工程规模与质量 | 7 | 7 | 0 | 0 | +| §4.7 B0 | 3 | 0 | 0 | 3 | +| **合计** | **77** | **46** | **23** | **8** | + +### 1.2 逐条判定 + +**已用(46 条)** —— 只列出在哪一节用的,不重复数值。 + +| 台账条目 | 用在 asset 哪一节 | +|---|---| +| 473 · 303 · references/imports/inherits | 「六条禁止说法」表(**只作为反例出现,不是正面主张**) | +| 2615 · 316 / 12.1% · 1076 | 面试口径 第 8 条 | +| K=60 · 2:1 · 50 · 16 · 256 · 取 5 · k1=1.2/b=0.75 | Agent 架构 d「检索栈」 | +| embedding 重试 12 | Agent 架构 e「重试策略」表 | +| 步数上限 14 · 工具数 11 · 证据预算 10 | Agent 架构 c / d / e + bullet 3 | +| groundedness 阈值 0.95 | 「它在 AI 链路里的位置」表 | +| history 6/2000/4000 · 合成 700/0.1 · 四种 TTL | Agent 架构 d + 后端基础设施 | +| 33 = 5/16/12 · manual/graph_reverse 16/17 | bullet 4 + 面试口径 第 5 条 | +| 阈值 0.05 · 重复 3 次 · 协议名 v2 · GRAPH_TOOLS 六个 | Agent 架构 g + bullet 4 | +| unsupported · 四个 margin · 0.0061 · 0.034 · 三次重复值 | 面试口径 第 1–2 条 | +| +0.0218/+0.0226 · +0.1466 | 面试口径 第 3 条 + 「如果只读五句话」 | +| 1.000 · answers_without_citations 0 · 4/3 倍 | 给 mentor 的说明 第 3 条 | +| +0.0283 vs +0.0183 | 面试口径 第 5 条 | +| B1 三次逐位相同 | 面试口径 第 2 条 | +| 291 · 73/16 · 10,952/7,678 · 4,746/1,445 | 给 mentor 的说明 第 2 条 | +| 7 workspace · 6+2 服务 · 4+1 表 | bullet 1 + 后端基础设施 | + +**未用但可安全使用(23 条)** + +| # | 数值 | 含义 | 等级 | +|---|---|---|---| +| 1 | **726** | chunk 总数 | A(C) | +| 5 | **724** | symbol 总数 | C+D | +| 9 | **37** | 参与索引的 `.py` 文件数 | C | +| 10 | **62** | 33 题去重后的 gold chunk 数 | A | +| 11 | **9 / 20** | `@overload` 重复符号数 / 被丢弃记录数 | C+D | +| 14 | **768** | embedding 维度 | A | +| 15 | **1024** | 代码默认维度 | B | +| 23 | **8** | BM25 语料 LRU 容量 | B | +| 24 | **20s / 3** | reranker 超时 / 重试 | B | +| 26 | **1024** | embedding `max_seq_length` | B | +| 30 | **1200** | 单条证据字符上限 | B | +| 31 | **64** | 证据水合批量上限 | B | +| 32 | **6** | 基类 BFS 深度上限 | B | +| 40 | **2.12/1.38 vs 4.29/2.94** | 两类题的平均 gold / 跨文件数 | C | +| 43 | **5** | 检索截断 k | A | +| 57 | **`unsupported`(四项口径)** | 旧口径同样不通过 | A | +| 58 | **12/12/12 与 14/12/16** | 旧 origin 集合下的 graph 命中 | A | +| 59 | **0/0/0 与 4/3/5** | 新 origin 集合下同一量 | B+C | +| 60 | **440/248/152/111/36** | B4 三次合计的被引用证据来源计数 | A | +| 61 | **9/16 vs 10/17** | 引用了 graph 来源证据的题目比例 | A | +| 63 | **9/33** | B4 recall 天花板 <1.0 的题数 | C | +| 64–65 | **q-009: 6→5** · **0.667/1.000/0.6508** | `[:k]` 截断的真实代价 + 手算复现 | A / A+C | +| 66 | **0** | 运行中观测到的 api/agent 错误数 | A | + +**未用且不该用(8 条)** + +| # | 数值 | 不该用的理由 | +|---|---|---| +| 12 | 2 / 44 / 453 | D 级,且描述的是题集属性不是系统能力。读者无法判断 44 行的均值算好还是坏 | +| 13 | `414f0513…` | 是标识符不是数字,简历上无信息量 | +| 39 | `db813124…` | 同上。它的价值在于证明题集冻结,那件事用文字说比贴哈希有效 | +| 75 | B0 file 级 0.3081/0.2955/0.2841 | §4.7 的限定语明说「H1 那次运行里没有任何 arm 有 file 级指标,所以这三个数在仓库里没有可比对象」。写上去必然引出「跟谁比」,而答案是没得比 | +| 76 | B0 groundedness 0.6364 / 无引用 12 | 同上,且 §4.9 第 2 条禁止把 groundedness 笼统当成绩 | +| 77 | B0 题数 33 | 与 #37 重复,且会把读者引向不可复现的 B0 | +| — | B0 chunk 级 0.0 三项 | §4.9 **第 1 条**:结构性 0,不是成绩 | +| — | 任何 v1 协议下的 margin | §4.9 **第 3 条**:被自己废弃的协议,只能说明改动方向 | + +> **一个必须点明的现状**:台账里排第 2、3 的 `473` 和 `303` 在 asset 里**只出现在「禁止说法」表**里,作为反例。也就是说,这个项目最容易被外行读成亮点的数字("我建了一个几百条边的调用图"),在简历素材里是被主动封掉的。这是对的,但要知道它是有意为之。 + +**三个计数:已用 46 · 未用可安全使用 23 · 未用且不该用 8。** + +--- + +## Step 2 — 按「能不能进 bullet」重新分档 + +### A 档 · 可直接进 bullet(不需要限定语,读者不需要背景知识) + +| 数值 | 含义 | 等级 | 为什么不需要限定 | +|---|---|---|---| +| **726** | 一个仓库索引出的代码块数 | A(C) | 量级自明。它说明系统处理的是真实仓库不是玩具 | +| **724** | 符号总数 | C+D | 同上 | +| **37** | 参与索引的 Python 文件数 | C | 同上 | +| **62** | 33 题的去重 gold chunk 数 | A | 说明题集有明确的标准答案,不是主观打分 | +| **768** | embedding 维度 | A | 是模型属性,说出来无歧义 | +| **5** | 检索截断 k | A | 与 33 题、3 次重复同属评测配置,读者能拼出规模 | +| **0** | 运行中观测到的 api/agent 错误数 | A | 495 次调用零错误,是稳定性证据 | + +### B 档 · 只能口头说(**面试弹药,不进简历**) + +两类:需要跟一句限定语才安全的,以及纯配置参数。 + +| 数值 | 含义 | 为什么不进简历 | +|---|---|---| +| **473 / 303** | 边总数 / 调用边 | 必须带「四种边之和,调用边只有 303」,简历放不下这句限定 | +| **2615 → 316 / 12.1%** | 调用点解析率 | 必须带分母定义。裸写「12% 解析率」像能力不足,实际是声明放弃类型推断 | +| **1024(代码默认维度)** | 与实跑的 768 不一致 | 必须解释为什么两个数不一样 | +| **K=60 · 2:1** | RRF 常数与权重 | 纯配置。读者无法判断 60 好不好 | +| **50 → 16 → 5** | 候选池 → rerank → 返回 | 纯配置,但**口头讲很有效**,能说清检索是三段漏斗 | +| **1.2 / 0.75** | BM25 k1 / b | 纯配置。价值在「参数写进产物所以基线可复现」,那句话已在 bullet 5 | +| **14** | 步数上限 | 已在 bullet 3,但**它的故事在口头**(从 8 提上来的原因) | +| **10** | 证据上下文预算 | 需要「对所有 arm 相同,约束的是最强臂」才有意义 | +| **6** | 基类 BFS 深度上限 | 需要「是 MRO 的近似不是 MRO」 | +| **1200 · 64 · 8 · 20s/3 · 1024(seq)** | 各类上限与超时 | 纯配置,量级读者无法评价 | +| **9 / 20** | `@overload` 重复符号 | 是缺陷不是成绩。口头讲很加分(我自己审计出来的) | +| **2.12/1.38 vs 4.29/2.94** | 两类题的 gold 规模差 | 需要 taxonomy×source 混淆的整段背景 | +| **12/12/12 与 14/12/16** | 旧 origin 集合的命中 | 必须带「这是被废弃的旧集合」 | +| **0/0/0 与 4/3/5** | 新集合同一量 | 必须带「这是重算值,不是已提交字段」 | +| **440/248/152/111/36** | 被引用证据来源计数 | 需要解释四个图工具为零 | +| **9/16 vs 10/17** | 图来源证据的题目比例 | 需要 graph_reverse 偏置整段背景 | +| **9/33 · q-009 · 0.667/1.000/0.6508** | recall 天花板与手算例 | 需要 v2 计分口径整段背景。**口头是最好的弹药之一** | +| **`unsupported`(四项口径)** | 旧口径同样不通过 | 需要「两次协议改动」整段背景 | + +### C 档 · 规模类(**当前 asset 用得最少,对非技术读者最有效**) + +台账已有的 + 本次新查证的,全部列出。 + +| 数值 | 含义 | 出处 | 等级 | 已用? | +|---|---|---|---|---| +| **7** | uv workspace 包 | `pyproject.toml:8-16` | B | 已用 | +| **6 + 2** | 常驻容器服务 + profile sidecar | `docker-compose.yml` | B | 已用 | +| **4 + 1** | 运行时表 + 溯源表 | `db/models.py` | B | 已用 | +| **11** | agent 工具数 | `tools/__init__.py:22-39` | B | 已用 | +| **5** | LangGraph 节点数 | `graph.py:774-778` | B | 已用 | +| **291 / 73** | pytest / vitest 用例数 | 实跑 | C | 已用 | +| **10,952 + 7,678 / 4,746 + 1,445** | 后端 / 前端 源码+测试行数 | `wc -l` | C | 已用 | +| **726 / 724 / 37** | chunk / symbol / 文件数 | `provenance.json` · DB · 重算 | A(C) / C+D / C | **未用** | +| **33 = 5/16/12** | 题数与分层 | `provenance.json` | A+B | 已用 | +| **3** | 重复次数 | `run_config.json` | A | 已用 | +| **5 级** | baseline 阶梯层数 | `baselines/__init__.py:40-47` | B | 已用 | +| **7** | SSE 事件类型闭集 | `events.py:13-21` | B | 已用(role prep 附录) | +| **6 / 4 / 5** | 索引外部状态 / PipelineStage / stage 函数 | `schemas.py:18-27`·`pipeline.py:51-56` | B | 已用 | +| **495** | 一轮完整评测的 agent 调用次数 | 5 arms × 3 repeats × 33 题,由 `run_config.json` 推出 | **A(派生)** | **未用** | +| **9** | 内部检索/图查询路由数 | `routes/internal.py` 的 `@router.` 计数 | **B** | **未用** | +| **5** | 公开 API 路由数 | `routes/{repos,query,inspector}.py` 的 `@router.` 计数 | **B** | **未用** | +| **5** | alembic migration 文件数 | `infra/migrations/versions/*.py` | **B** | **未用** | +| **32 / 16** | pytest / vitest 测试文件数 | 文件计数 | **C** | **未用** | +| **4** | 前端路由数 | `App.tsx` 的 ` **C 档里最值得上简历的三个**:**364 个自动化测试**、**495 次 agent 调用/轮评测**、**726 个代码块**。它们量级自明,不需要限定语,而且对不读代码的人最有说服力。 + +--- + +## Step 3 — 「改进 → 成效」链条 + +只有同时满足四条才算完整:①改动前状态有出处 ②改动有 `file:line` 或 commit ③改动后状态有出处 ④因果关系在代码或提交记录里有依据,不是事后推断。 + +### 甲类 · 工程改进(修了缺陷或提升能力,前后可比) + +#### 甲-1 · 符号解析规则统一 —— 守卫和工具此前用两套规则 + +| 要素 | 内容 | +|---|---| +| **前** | 守卫 `_verify_symbol` 用 exact match,工具 `routes/internal.py` 用 exact + dotted suffix。同一请求里工具说「在这儿」,守卫说「不存在,删掉」。**九次单臂运行中,symbol 形式的引用活下来的次数是 0 · 0 · 0**(`results/b4-citation-fix-experiment.md` 结论表,原始臂与 A 臂共六次运行全为 0) | +| **改** | commit `029b9de` "fix(shared): one rule for what a name refers to, applied by the guardrait too",规则收进 `packages/shared/src/dcode_shared/symbols.py:38-53`,SQL 侧 `:56-64` 只做超集缩小 | +| **后** | 同一实验的 B 臂三次运行:**symbol 形式引用 2 · 2 · 2**(首次存活);**已验证引用 65 · 62 · 67**(原始臂 54 · 48 · 51);**抹除标记 12 · 13 · 9**(原始臂 17 · 19 · 21) | +| **因果** | 实验设计写在同一文件:一个 session、一份索引、一对模型 sidecar,**只变 agent 镜像,且逐臂在容器内 grep 验证**。机制另有独立论证:索引里 724 个符号中 719 个带 `src.`/`tests.` 前缀,短名行根本不存在 | + +**可以说出口的版本**:「我发现引用验证的守卫和检索工具用了两套符号解析规则,同一个请求里工具说找到了、守卫说不存在。统一成一套之后,**已验证引用从 48–54 条涨到 62–67 条,抹除标记从 17–21 降到 9–13**,而且 symbol 形式的引用第一次活了下来——在此之前九次运行里一条都没有。」 + +**必须一起说的话**:这九次是**单臂运行,不携带 H1 判定**(`results/README.md` 明标 "not a verdict")。而且实验报告自己写了「**计数是证据,均值是弱的**」——groundedness 均值的比值是 1.26 和 1.93,在 n=3 下约等于 p≈0.25 和 p≈0.1,报告明说不作为统计显著性。**只引用计数,不引用均值。** + +#### 甲-2 · 图证据补源码 + 统一重排 —— 修掉一个测量偏差 + +| 要素 | 内容 | +|---|---| +| **前** | `results/eval-h1-uniform-v2-2026-07-31/h1_report.json` → `diagnostics.per_taxonomy.L3.graph_margin_B4_vs_B3.5` = **−0.0067**。也就是在架构级问题上,**关掉调用图的臂反而赢了有调用图的臂** | +| **改** | commit `2379c39` "feat(agent): let graph evidence carry code and compete on relevance"。三处:按 `chunk_id` 批量补源码(`graph.py:498-529` → `GET /internal/get_chunks`,服务端 `internal.py:121-163`);同一个 cross-encoder 统一打分(`graph.py:437-490`);上下文预算对所有臂相同(`graph.py:432`) | +| **后** | 下一次运行 `results/eval-h1-ranked-evidence-2026-07-31/` 同一字段 = **+0.0245**。当前记录运行 = **+0.0226** | +| **因果** | commit 正文明写诊断与范围:「This is the diagnosed cause, in three parts, **all system changes -- no metric, threshold or question was touched**」。诊断本身是机制性的:图结果只有 `file:line`,检索结果带源码,打分模型比的是哪边有东西可读 | + +**可以说出口的版本**:「有一次评测里,关掉调用图的对照臂在架构级问题上**反而赢了**(−0.0067)。查下来是测量偏差:图查询结果只有文件名行号,检索结果带源码,重排模型比的是哪边有东西可读。我给图结果补上源码再统一打分,**同一个指标从 −0.0067 变成 +0.0226**。」 + +**必须一起说的话**:这两个数都是**诊断项,代码层面排除在主判定之外**;而且它们来自两次不同的单次运行,**没有跑重复**,所以不能当成一个有误差棒的效应量。 + +#### 甲-3 · 检索结果过滤测试代码 + +| 要素 | 内容 | +|---|---| +| **前** | commit `4cb8660` 正文:「Measured on the 2026-07-31 suite: **47% of retrieved candidates and between 26% and 48% of cited evidence came from tests/**, in every arm.」同一事实也写进 `packages/shared/src/dcode_shared/testpaths.py:10-12` | +| **改** | commit `4cb8660`。判定规则 `testpaths.py:25-33`,应用点 `apps/api/src/dcode_api/routes/internal.py:302-311`(**排序之后**过滤,候选池与融合算术不动) | +| **后** | 我从当前记录运行重算:五个臂的检索候选里 tests/ 占比 **0/165 = 0%**,被引用证据里 **0%**(B2 0/132、B3 0/134、B3.5 0/166、B4 0/183) | +| **因果** | 过滤应用在共享检索路径上,所有臂同时生效;规则是二值的,没有可调权重 | + +**可以说出口的版本**:「有个产品缺陷跟假设无关:问『Requests 怎么把 cookie 变成请求头』,系统会拿 `test_requests.py` 回答。**当时 47% 的检索候选来自 tests/**。我加了一条二值规则,在排序之后过滤测试文件,问题里提到测试就放行。**现在是 0%。**」 + +**必须一起说的话**:commit 自己披露了利益冲突——「this was found while looking for the 0.0036 by which L3 missed the bar, and **no ground-truth file lives under tests/, so the change moves toward the ground truth**」。这条改动同时也机械地帮了指标,作者主动记下了这一点。讲的时候要自己先说。 + +#### 甲-4 · 继承与局部构造的调用解析 —— **能力提升可测,下游效果作者自测为零** + +| 要素 | 内容 | +|---|---| +| **前** | `self.method()` 定义在基类上时解析不到;`p = PreparedRequest()` 后 `p.prepare(...)` 解析不到。commit `d613caa` 测得当时的调用边数 **267** | +| **改** | commit `d613caa`。继承先于调用解析(`stages/graph.py:81-91`),基类广度优先查找深度上限 6(`:363-402`);局部构造类型绑定只认字面量(`:405-445`) | +| **后** | 同 commit 测得 **279**(+12)。我在 `414f051` 上用当前代码离线重算:库里 **303**,当前代码 **316**(+13),其余三种边逐项相同 | +| **因果** | commit 正文点名了解锁的三条边:`prepare_body → _encode_files`、`Session.send → resolve_redirects`、`Session.prepare_request → PreparedRequest.prepare` | + +⚠️ **这一条不能写成成效。**commit 自己测了下游影响并写下结论:「**this unlocks ZERO additional end-to-end L3 ground-truth chains**」——因为那些链条大多终结在异常类或构造函数上,它们是被 raise 或 instantiate 的,不是被 call 的,任何 `calls` 边都到不了。另外「**Not active in any recorded run yet**」,需要重索引才生效。 + +⚠️ **还有一处数字不一致要知道**:commit 记的是 267 → 279,我重算的是 303 → 316。两者都是 +12/+13,但**测的是不同快照**(库里现存的两个 `psf/requests` 索引,`f361ead` 与 `414f051`,calls 边都是 303,都不等于 267 或 279)。所以 commit 里那对数字**在当前库里核对不到**,只能引用我重算的 303 → 316,并标 D/C 级。 + +### 乙类 · 测量修正(改的是评测口径或实验设计,不是系统能力) + +> **这一档不许包装成性能优化或效果提升。**它的正确叙述是「我发现我的测量方法错了,修正后结论变了」。 + +#### 乙-1 · 计分协议 v1 → v2,四个 agent 臂统一规则 + +- **前**:v1 对 B4 用「答案里已验证的引用」,对 B2/B3 用「检索 top-k」,两套规则。 +- **改**:commit `b39e756` "feat(eval): one official scoring rule for every decision arm";实现 `apps/eval/src/dcode_eval/run.py:83` + `baselines/__init__.py:31`;理由写在 `run.py:26-38`。 +- **后**:我用同一份产物两种口径各算一遍——v1 下 L2 margin 是 **+0.0822**(过线),v2 下是 **+0.0439**(不过线)。 +- **因果**:`run.py:26-38` 原文:「A decision that moves depending on which arm is measured how is not a decision.」 +- **口径**:「我换过计分口径,**换完我自己的结论从通过变成没通过**。」 + +#### 乙-2 · GRAPH_TOOLS 集合修正 —— 度量测错了对象 + +- **前**:统计「图找到了多少 gold」的工具集合包含 `get_file_outline`,而**没有图的对照臂保留这个工具**。旧集合下 B3.5 记 **12/12/12**、B4 记 **14/12/16**。 +- **改**:commit `b17621b`;集合收进 `packages/shared/src/dcode_shared/graph_tools.py:58-67`,两个消费者共用(`state.py:57`、`run.py:506`)。 +- **后**:新集合下 B3.5 **0/0/0**、B4 **4/3/5**。 +- **因果**:`graph_tools.py:17-38` 写了完整论证——B3.5 是关掉图的 B4,它的正确值只能是 0;旧集合给它打 12 分,说明测的不是图。commit `40d5712` 另有一次实跑复核。 +- **口径**:「我发现统计口径把一个无图臂也保留的工具算成了图的功劳。**一个分不清有图和没图的度量,不是偏高,是测错了对象。**」 + +#### 乙-3 · 引入 B3.5 对照实验臂 + +- **前**:只有 `B4 − B3`,它在调用图或多步取证任一变动时都会动,说不出是哪个。 +- **改**:commit `ae72d12` "feat(eval): route B2 through the agent and add the B3.5 arm";mode 表 `state.py:32-44`;排除在判据外 `run.py:599-601`、`:605-616`,测试钉住 `test_run.py:686`。 +- **后**:差值可拆——L2 图 +0.0218 / 循环 +0.0221;L3 图 +0.0226 / 循环 +0.1466。 +- **因果**:`run.py:605-616` 原文:「promoting an arm into the decision rule would be changing the pass criteria after the fact」。 +- **口径**:「我加了一个我不需要的臂。**它把我自己项目的卖点降级了,而我没有用它去改判决。**」 + +#### 乙-4 · 去掉 composite 里的 groundedness 项 + +- **前**:四项 composite,其中 groundedness 在所有臂上恒为 1.000。 +- **改**:commit `38f8036`;`run.py:666` 的 `_COMPOSITE_TERMS`,理由长注释在 `run.py:646-665`。 +- **后**:三项 composite。被删项恒等,去掉等价于每个 margin 乘 4/3,即门槛从 0.050 降到 0.0375。四项口径同时发布在 `h1_report.json` 的 `four_term`,**两种口径都是 unsupported**。 +- **因果**:`run.py:648-665` 原文:「It is a lowered bar, honestly labelled; calling it anything else would be false.」 +- **口径**:「我删了一项指标,**那确实是降门槛**,我不换说法。三条辩护都能核对:事先声明、两种口径同时发布、两种都不通过。」 + +#### 乙-5 · 零引用答案计分 1.0 → 0.0 + +- **前**:`verify` 在答案没有任何引用时返回 **1.0**。一个学会闭嘴的 agent 能刷满分。 +- **改**:commit `5dae9f7`;`apps/agent/src/dcode_agent/groundedness.py:272`,推理写在 `:257-272`。 +- **后**:返回 **0.0**,并用独立计数 `answers_without_citations` 区分「没引用」与「引用全错」(`run.py:458`)。 +- **因果**:`results/b4-citation-fix-experiment.md` 的「Correction, 2026-07-30」一节做了**精确重算而非重跑**,证明旧约定曾让该文件自己的两条结论失效:post-fix 均值 0.8896 → **0.8479**,效应符号反转;方差「4× 降低」→ **没有降低**。 +- **口径**:「我改了一个计分约定,**结果是我自己此前那个实验的两条结论都不成立了**。改完的数字对我更不利。」 + +### 丙类 · 参数调整(代码注释记录了理由) + +| 参数 | 前 → 后 | 理由出处 | 能不能引用 | +|---|---|---|---| +| `max_steps` | **8 → 14** | `apps/agent/src/dcode_agent/settings.py:9-13`:B4 的扩展把 8 跑满(1 search + 3 read_file + 3 find_references 只剩 1 步给 get_file_outline,而无图臂在同样预算下跑到 2 个),walk 终止在上限而非规划器判断 | ✅ 可引用,已在 bullet 3 | +| RRF 权重 | **1:1 → 2:1(dense:sparse)** | `packages/shared/src/dcode_shared/settings.py:50-52`:等权时观察到 B2 纯稠密的 nDCG 反超 B3 混合 | ✅ 可引用,两个方向各有测试钉住 | +| `max_seq_length` | **8192(Jina 默认)→ 1024** | `apps/embedding/src/dcode_embedding/main.py:41-44`:8192 的注意力开销**曾在运行中 OOM 杀死 sidecar** | ✅ 可引用,但是事故记录不是优化成果 | +| rerank passage 截断 | **全长 → 256** | `apps/api/src/dcode_api/routes/internal.py:53-56` | ⚠️ **不可引用** —— 该注释含「10 条 rerank 超 40s / 短 passage ~1s」的延迟数字,**这不是压测结果,是一次未记录的观察**。项目全局禁止延迟数字,这条注释是唯一的例外来源,必须封掉 | + +--- + +## Step 4 — 诚实结论 + +**甲类 4 条(其中 1 条下游效果作者自测为零)· 乙类 5 条 · 丙类 4 条(其中 1 条不可引用)。** + +**甲类少,而且这是这个项目的性质决定的,不是我漏找了。** + +理由有三条,都可查: + +1. **这个项目的迭代重心本来就在乙类。**257 个 commit 里,2026-07-30 到 07-31 这两天的密集迭代几乎全是评测口径与实验设计——`b39e756`(统一计分)、`ae72d12`(加对照臂)、`38f8036`(改 composite)、`b17621b`(修 origin 集合)、`5dae9f7`(改计分约定)、`57c9542`(加 file 级计分)。这些改的都是**怎么测**,不是**系统能多做什么**。 + +2. **甲类的三条都不是「让指标涨」,而是「修掉一个让指标失真或让产品难用的缺陷」。**符号规则统一是修「工具和守卫互相打架」;图证据补源码是修「测量偏差」;测试代码过滤是修「问实现却回答测试文件」。它们的成效是**计数**(引用条数、抹除标记数、tests/ 占比),不是效果百分比。 + +3. **唯一一条纯能力提升(甲-4,调用解析),作者自己测出下游效果为零,并且记了下来。**这比它有效果更值得讲。 + +**我没有为了让表格好看把乙类挪进甲类。**乙类里的每一条如果包装成「优化」,第一层追问就会问「优化前后指标涨了多少」,而正确答案是「指标没涨,是我发现之前的测法错了」——那时候再解释就晚了。 + +--- + +## Step 5 — 可加进 asset 的增补建议 + +### 5.1 现有 bullet 的增补版本(长度不增加) + +#### bullet 1 + +> **现**:…含 7 个 uv workspace 包、6 个常驻容器服务、2 个按需启动的模型 sidecar。四个应用包之间零 Python import… +> +> **改**:…含 7 个 uv workspace 包、6 个常驻服务、2 个模型 sidecar、**364 个自动化测试**。四个应用包之间零 Python import… + +**替换了什么**:把「常驻容器服务」的「容器」和「按需启动的」两个修饰词去掉,腾出的位置放 **364 个自动化测试**(C 档派生,291 pytest + 73 vitest 实跑之和)。 +**为什么**:原句三个数字都是「有几个进程」,同质、信息重复。测试数是另一个维度,且对不读代码的人最直观。字数基本持平。 + +#### bullet 4 + +> **现**:…33 题冻结题集,**分层标注为人工标注,代码中无判定器**。跑三次重复,以分离答案生成的随机性。… +> +> **改**:…33 题冻结题集共 **62 个标准答案代码块**,**分层标注为人工标注,代码中无判定器**。跑三次重复共 **495 次 agent 调用**,以分离答案生成的随机性。… + +**替换了什么**:插入两个 A 档数字——**62**(gold chunk 数,`repeat-1/B4/per_question.jsonl` 聚合)和 **495**(5 arms × 3 repeats × 33 题,由 `run_config.json` 派生)。 +**为什么**:原句的「33 题」和「三次重复」单独看都很小,读者容易觉得这是个玩具评测。加上 62 个标准答案和 495 次调用,规模感立刻不同,而且两个数都不需要限定语。 + +#### bullet 5 + +> **现**:…两路经**加权 RRF 融合(K=60,dense:sparse=2:1)**,再送 cross-encoder 重排。**证据排序特征刻意不含来源与图距离**… +> +> **改**:…两路经**加权 RRF 融合**后送 cross-encoder 重排,并在排序后过滤测试代码,**把检索候选里 tests/ 的占比从 47% 降到 0**。**证据排序特征刻意不含来源与图距离**… + +**替换了什么**:拿掉 `K=60,dense:sparse=2:1`(B 档纯配置,读者判断不了 60 好不好),换成**甲-3 的前后对比 47% → 0%**。 +**为什么**:这是全文唯一一个「缺陷 → 修复 → 可核对的前后数字」,而且量级自明。K=60 那对参数移到面试弹药里讲更有用。 + +### 5.2 一条候选新 bullet(依赖甲-1) + +> **排查并统一**引用验证的符号解析规则:此前引用验证守卫用精确匹配、检索工具用后缀匹配,同一请求内工具报「已找到」而守卫报「不存在并删除」。收敛为共享单一规则后(`packages/shared/src/dcode_shared/symbols.py:38-53`),在受控的三臂九次对照运行中,**已验证引用由 48–54 条升至 62–67 条,抹除标记由 17–21 降至 9–13**,符号形式引用首次得以保留。 + +**依赖链条**:甲-1。 +**必须带的限定**:这九次是单臂运行,**不携带 H1 判定**;实验报告自己写明「计数是证据,均值是弱的」,所以**只引用计数,不引用 groundedness 均值**。 +**建议**:这条比现有任何一条更像「工程师排查并修复了一个具体缺陷」。如果 mentor 觉得六条太多,可以用它替换 bullet 6(索引流水线那条),因为 bullet 6 讲的是标准的后端工程,可替代性更高。 + +### 5.3 面试弹药清单(**不进简历**) + +**B 档全部 —— 每条一句口头说法** + +| 数值 | 口头说法 | +|---|---| +| 473 / 303 | 「边有四百多条,但那是四种边之和,调用边只有三百出头。」 | +| 2615 → 316 | 「解析器遍历到两千六百多个调用点,解析出三百多个,十几个百分点。这是无类型推断的静态分析的合理量级。」 | +| 1024 vs 768 | 「代码默认 1024,实跑 768。维度在 migration 执行那一刻烧进列定义,改维度得重建库。」 | +| K=60 / 2:1 | 「用名次融合不用分数相加,因为 BM25 分数无界、余弦有界,名次是唯一不用假设分布就能比的量。权重非等权是因为等权时纯稠密会反超混合。」 | +| 50 → 16 → 5 | 「检索是三段漏斗:两路各取 50 个候选,融合后重排前 16 条,最后返回 5 条。」 | +| 1.2 / 0.75 | 「BM25 参数写进运行产物,所以那条基线可复现。这也是我不用 Postgres 全文检索的原因——`ts_rank` 不是 BM25。」 | +| 14 | 「从 8 提上来的。提它是因为最强的臂总是终止在预算上,不在规划器的判断上——那种情况下测的是预算不是策略。」 | +| 10 | 「对所有臂相同,约束的是最强的那个:图证据必须挤掉检索证据才能进上下文。」 | +| 6(BFS 深度) | 「是 MRO 的近似不是 MRO,忽略 C3 线性化,菱形继承可能选错分支。」 | +| 1200 / 64 / 8 / 20s·3 / 1024 | 「各处上限。reranker 只重试 3 次而且不重试超时,因为重试慢的 CPU 推理只会让单线程服务更慢。」 | +| 9 / 20(`@overload`) | 「符号表按名字去重保留第一条,而 `@overload` 存根排在真实现前面。9 个符号、20 个 chunk 指向的是三行省略号。这是我自己审计出来的。」 | +| 2.12 vs 4.29 | 「人工题平均 2 个标准答案,反推题 4 个多。所以那两类题的难度本来就不一样。」 | +| 12/12/12 与 14/12/16 | 「这是被废弃的旧集合的值。它给没有图的臂打 12 分,给有图的臂打 14 分。」 | +| 0/0/0 与 4/3/5 | 「修好之后的值。注意这是我从已提交字节重算的,不是产物里的字段。」 | +| 440/248/152/111/36 | 「被引用证据的来源计数。六个图工具里有四个一次都没出现——它们赢不过检索结果。」 | +| 9/16 vs 10/17 | 「反推题并没有更多地触发调用图,两类题几乎一样。」 | +| 9/33 · q-009 | 「有九道题 recall 天花板本来就不到 1。q-009 最典型:B4 产出六条已验证证据,第六条被截断切掉,而那条正是标准答案。」 | +| 四项口径 unsupported | 「两种口径都不通过,所以那次改动没有改变结论。」 | + +**乙类全部 —— 每条一句口头说法** + +| 乙类 | 口头说法 | +|---|---| +| 乙-1 计分协议 v1→v2 | 「我换过计分口径,让四个臂用同一条规则。**换完我自己的结论从通过变成没通过。**」 | +| 乙-2 GRAPH_TOOLS 修正 | 「统计口径把一个无图臂也保留的工具算成了图的功劳。一个分不清有图和没图的度量,不是偏高,是测错了对象。」 | +| 乙-3 引入 B3.5 | 「我加了一个我不需要的臂。它把我自己项目的卖点降级了,而我没有用它去改判决。」 | +| 乙-4 去掉 groundedness 项 | 「我删了一项指标,那确实是降门槛。我不换说法——三条辩护都能核对,而且两种口径都不通过。」 | +| 乙-5 零引用计 0 分 | 「我改了一个计分约定,结果我自己此前那个实验的两条结论都不成立了。改完的数字对我更不利。」 | + +> **这五条合起来是一个完整的能力叙事**:我有能力发现自己的测量方法错了,并且在修正会让自己难看的时候照样修。**这比任何一条性能提升都更难伪造,也更难被追问击穿。** + +--- + +## 文末汇总 + +### 本次新找到的数字及其来源等级 + +| 数值 | 含义 | 出处 | 等级 | +|---|---|---|---| +| **9** | 内部检索/图查询路由数 | `apps/api/src/dcode_api/routes/internal.py` 的 `@router.` 计数 | **B** | +| **5** | 公开 API 路由数 | `routes/repos.py` + `routes/query.py` + `routes/inspector.py` | **B** | +| **5** | alembic migration 文件数 | `infra/migrations/versions/*.py` | **B** | +| **32 / 16** | pytest / vitest 测试文件数 | 文件计数 | **C** | +| **4** | 前端路由数 | `apps/frontend/src/App.tsx` 的 ` Date: Sun, 2 Aug 2026 17:55:44 -0700 Subject: [PATCH 2/2] docs(interview): add bilingual resume material with a per-figure usage guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The same facts as the technical brief, reorganised for a mentor who will edit a resume and who reads once. Chinese and English are complete independently rather than one being a translation of the other. Figures come only from the brief's §4 ledger and the data inventory. Where the ledger attaches a qualifier, the text either carries it or uses a phrasing that does not need one. Lead order is deliberate: citation verification, then how the agent is kept in check, then the evaluation methodology. The call graph does not open, because my own control arm measured it down and opening with it means opening on the weakest link. The non-ML boundary is stated outright -- no training, no loss, no RL, every model an off-the-shelf call, and the static call-graph resolution labelled as program analysis. Two Mermaid diagrams, drawn only from components that exist. `chunks.tsv` and `index_runs` are dashed and marked NOT WIRED so nobody reads them as part of the data flow. The section that makes this usable is "what each number proves", in three groups. Group one is safe on a resume, with a column saying what each figure means to a reader rather than restating the figure. Group two lists the qualifier each number needs and the specific follow-up question that lands if it is missing. Group three stays verbal, with the reason it stays off the resume: needs full context, is a diagnostic excluded from the verdict, is configuration a reader cannot judge, or is not an outcome. Bullets carry only ledger-backed figures or mechanistic results. There are no throughput, latency, token-cost or accuracy-lift numbers, and the notes to my mentor explain per category why adding any would mean inventing them. Six phrasings that collapse under the first follow-up are listed as must-not-reintroduce, since an editing pass would otherwise restore them. Two disclosures are written where they cannot be missed. Filtering test code also mechanically moves the metric toward the ground truth, and the commit that made the change says so; I have to say it first rather than have it read out of the commit. And the citation counts come from single-baseline runs carrying no verdict, whose own report says the counts are the evidence and the means are weak, so the means are never quoted. Seven bullets is too many for one resume, so the file ends with a by-channel selection instead of a deletion: four for technical applications, three for HR screens, with a reason for each. --- docs/interview/DCODE_RESUME_ASSETS_CN.md | 645 +++++++++++++++++++++++ docs/interview/DCODE_RESUME_ASSETS_EN.md | 641 ++++++++++++++++++++++ 2 files changed, 1286 insertions(+) create mode 100644 docs/interview/DCODE_RESUME_ASSETS_CN.md create mode 100644 docs/interview/DCODE_RESUME_ASSETS_EN.md diff --git a/docs/interview/DCODE_RESUME_ASSETS_CN.md b/docs/interview/DCODE_RESUME_ASSETS_CN.md new file mode 100644 index 0000000..12b52cd --- /dev/null +++ b/docs/interview/DCODE_RESUME_ASSETS_CN.md @@ -0,0 +1,645 @@ +# Dcode —— 一个能查证自己答案的代码理解系统 + +## 如果只读五句话 + +1. 这个系统给需要快速读懂一个陌生 Python 仓库的开发者用。 +2. 你提交一个 Git 地址,它把代码切块存进数据库,之后你能用自然语言提问。 +3. 我搭了整条链路:索引流水线、混合检索、回答问题的 agent,以及前端界面。 +4. 我另外做了一套离线评测:判定标准提前写死,另加一组对照实验分清哪个部件起了作用。 +5. 结果是我的主假设没通过,而且我最初主打的那个功能,作用比预期小。 + +--- + +## 关于这份文档 + +这是 `docs/interview/` 下的第三份材料,写给帮我改简历的 mentor 看。 + +| 文件 | 用途 | +|---|---| +| `DCODE_TECH_BRIEF.md` | 全量技术底稿。§4 是可引用数字台账,每个数字标了来源等级 A/B/C/D,并写明引用它时必须一起说的那句话。§4.9 列出不能引用的数字。附录 A 是 25 条文档漂移清单 | +| `DCODE_AGENT_ROLE_PREP.md` | 面试稿,按招聘方考察的能力维度组织 | +| **本文** | 简历素材 | + +正文里的数字全部来自底稿 §4 台账。台账给某个数字附了限定语的,本文要么带上那句限定语,要么换一个不需要限定的说法。本文新增的数字放在**附录 A**,同样标来源等级。底稿和 README 说法冲突时以代码为准。 + +### 项目类型:后端基础设施 + LLM 应用,两边都占,以 agent 侧为主 + +判断依据是两套东西都真的写了,没有硬凑: + +| 后端基础设施 | LLM / agent | +|---|---| +| RabbitMQ 消费者,驱动一个严格单调的六态索引状态机 | LangGraph 五节点 agent 状态机 | +| PostgreSQL + pgvector 四张运行时表,带 HNSW 和反向边索引 | 11 个工具,参数用 Pydantic schema 定义,经注册表调度 | +| Redis 四个键空间,各有各的 TTL | 引用 ID 由服务端下发,未通过验证的引用会被删掉 | +| SSE 网关:截断历史、算缓存键、命中时逐字节重放 | 五级 baseline 离线评测,判定标准预注册,另有对照实验臂 | +| 提交幂等、SSRF 白名单、内部密钥、Alembic 迁移 | 混合检索:BM25 + pgvector + 加权 RRF + cross-encoder 重排 | + +主线放在 agent 侧。后端那一半写出来,是为了让 agent 的每一步都能复现、能缓存、能查证。 + +这个项目没有训练、没有 loss、没有 RL、没有微调,模型全是现成调用。静态调用图解析有算法含量,但它属于程序分析(static program analysis),跟机器学习不是一回事。全文都会把这条边界标清楚。 + +--- + +## 项目描述 + +Dcode 帮开发者读懂一个陌生的 Python 仓库。 + +你提交一个 Git 地址。worker 把仓库克隆下来,按函数、方法、类的边界切块。之后算好向量,抽出符号表和调用关系。 + +你提问的时候,一个 agent 会调用工具去查证,再给出答案。 + +**答案里的每一条引用,系统在发出去之前都会去数据库查一遍。**查不到的引用会从答案正文里删掉,换成一个占位符。groundedness(引用可信度:答案里的引用有几成能在索引里查到)这个分数按**删除之前**的比例算,所以删除不能把分数变高。 + +前端也跟着这条规则走。如果一次问答的流断了、从来没收到过终答,界面会把它标成第四种状态。它的草稿降级显示,引用不给点击。因为「没检查过」和「检查了没通过」是两回事。 + +围绕这套机制,我做了一套能证伪自己的离线评测。它有五级 baseline 阶梯,判定标准提前写死。另有一组对照实验,专门用来拆解增益来源。 + +--- + +## 架构图 + +两张图。第一张画一次提问在系统里怎么走,第二张画有哪些服务、数据存在哪。 + +### 1. 查询链路的 agent 控制流 + +```mermaid +flowchart TD + U["User question + client-side history"] --> GW["API Gateway
POST /api/v1/query"] + GW --> HB["Bound history
6 turns / 2000 chars total / 4000 per turn"] + HB --> CK["Build cache key
query : repo_id : sha256 of history + query"] + CK --> HIT{"Redis hit?"} + HIT -->|"yes"| RP["Replay the buffered SSE stream byte for byte"] + HIT -->|"no"| PX["Proxy to agent, internal key header"] + + PX --> CTX["contextualize
rewrite follow-up into a standalone query"] + CTX --> PLAN["plan
rule-based tool selection"] + PLAN -->|"tool selected"| TC["tool_call
validate args, cache lookup, execute"] + TC -->|"no error"| PLAN + PLAN -->|"no tool left / step cap 14 / error"| SYN["synthesize
stream LLM answer over ranked evidence"] + TC -->|"error, degrade"| SYN + SYN --> GC["groundedness_check"] + + GC --> VER["Verify every citation against the live index"] + VER --> RED["Redact unverified references from the answer text"] + RED --> SCORE["Score = verified ratio BEFORE redaction"] + + SCORE --> EV["SSE out: thought / tool_call / tool_result / partial_answer,
then citation events, then exactly one final_answer"] + EV --> FE["SPA thread"] + RP --> FE + + PX -.->|"agent unreachable"| ERR["error event only, stream terminates honestly"] + ERR --> FE +``` + +### 2. 服务与数据层 + +```mermaid +graph TD + SPA["SPA - React 18 + Vite, served by nginx"] + API["API Gateway - FastAPI"] + AG["Agent - FastAPI + LangGraph"] + W["Index Worker - RabbitMQ consumer, prefetch 1"] + EMB["Embedding sidecar - Jina v2-base-code on CPU"] + RR["Reranker sidecar - BGE reranker v2-m3 on CPU"] + + PG[("PostgreSQL + pgvector
repos / chunks / symbols / edges")] + RD[("Redis
embed: / tool: / query: / job:")] + MQ["RabbitMQ - queue dcode.index_jobs"] + VOL["Shared volume repo_workdirs"] + + SPA -->|"HTTP + SSE"| API + API -->|"AMQP publish, persistent"| MQ + MQ -->|"consume"| W + API -->|"HTTP SSE proxy"| AG + AG -->|"HTTP internal retrieval + graph API"| API + + W -->|"POST /embed"| EMB + API -->|"POST /rerank"| RR + AG -->|"POST /rerank"| RR + + W --> PG + API --> PG + AG --> PG + W --> RD + API --> RD + AG --> RD + W -->|"git clone writes"| VOL + AG -->|"read_file / grep / list_directory"| VOL + + W --> P1["clone"] + P1 --> P2["parse"] + P2 --> P3["chunk"] + P3 --> P4["embed"] + P4 --> P5["graph"] + + TSV["chunks.tsv + GIN index"] -.->|"NOT WIRED - no writer anywhere in the codebase"| PG + IR["index_runs table + append-only trigger"] -.->|"NOT WIRED - no writer anywhere in the codebase"| PG +``` + +第二张图里有两条虚线。`chunks.tsv` 和它的 GIN 索引建在库里,但全仓库没有任何代码往里写(稀疏检索走的是应用层手写的 BM25)。`index_runs` 表连不可变触发器都写好了,同样没有写入方。我把它们单独画成虚线,免得读图的人以为它们在主数据流上。 + +--- + +## 它在 AI 链路里的位置 + +这一节说清楚:系统吃什么进去、调了哪些模型、吐什么出来。 + +| 环节 | 内容 | +|---|---| +| **输入** | 离线索引侧吃一个 Git 地址。在线查询侧吃一句自然语言问题,加上客户端带过来的历史轮次 | +| **调用的模型** | `jinaai/jina-embeddings-v2-base-code` 算向量,自托管 HTTP sidecar 跑在 CPU 上。`BAAI/bge-reranker-v2-m3` 做重排,也是自托管 sidecar。`gpt-4o-mini` 写答案,走 OpenAI 流式接口。judge 位置留了抽象类,但**没有接任何模型** | +| **调用的工具** | 11 个:语义检索、找定义、找引用、找调用邻居、找调用路径、模块依赖正向和反向、文件结构、读文件、ripgrep、列目录 | +| **产出** | 一条 SSE 流:先推理轨迹,再流式答案,然后是**只含已验证引用**的引用事件,最后一条终答带 groundedness 分数。前端据此渲染可点击的引用 chip 和源码 inspector | +| **不产出** | 不训练、不微调、不产生模型权重、不产生偏好数据 | + +放到更大的图景里看,这套东西是给 agent 用的评测和查证基建。它不产出策略,它产出判断策略好坏的能力。任何团队想用 RL 或偏好优化改进一个 agent,都得先回答两个问题。奖励信号从哪来?涨的那部分怎么确认是真的?这个项目做的就是这两件事的工程侧。 + +--- + +## Agent 架构 + +### a. 编排 + +agent 是一个五节点的 LangGraph 状态机,工具选择由一张手写规则表决定。 + +节点定义在 `apps/agent/src/dcode_agent/graph.py:774-778`: + +``` +START → contextualize → plan ⇄ tool_call → synthesize → groundedness_check → END +``` + +`plan → tool_call → plan` 是 ReAct 循环。`plan` 和 `tool_call` 都能有条件地跳到 `synthesize`(`graph.py:782-791`)。请求完全无状态,没装 LangGraph 的 checkpointer,历史由客户端每次带上(`graph.py:770-771`)。 + +**规划器是一张手写规则表,不走 LLM**(`graph.py:798-812`)。这个选择要分三层讲清楚。 + +**第一层:为什么这么选。** 项目的核心测量是一组对照实验。B3.5 这个臂是 B4 关掉调用图、其余全部不变。这个差值要有意义,前提是两个臂之间除了工具集合以外没有别的差异。而答案生成本身已经够随机了:三次相同配置的运行,某一层的 margin 摆动幅度超过了判定门槛本身(见「面试口径」第 2 条)。规划器要是也用 LLM,差值里就会混进「模型这次碰巧选了哪个工具」,因果归因就站不住。规则表把两个臂的差异锁在一张映射表里(`state.py:32-44`),而且这张表能被单元测试穷举。 + +**第二层:代价是一个真实的产品缺陷。** 路由靠关键词表(`graph.py:1087-1112` 和 `:1312-1358`)。一个不含关键词的架构问题,会在第一次检索之后直接停下。而且外部看不出两种情况的区别。规划器判断没有下一步,和规划器没看懂问题,在 SSE 流里长得一模一样。 + +**第三层:要发货我会怎么改。** 主路径换成 LLM function calling。11 个工具的 JSON Schema 已经能通过 `ToolRegistry.manifest()` 导出(`tools/base.py:58-67`),接口层改动很小。规则表退到两个位置:模型给的参数校验不过时兜一次,以及守住几条安全边界。步数上限、arm 级别的工具禁用、同参数不重复调用这三条不交给模型决定。 + +换完之后要补三件事,对照实验才能继续做。一是把规划调用的 temperature 单独配置。二是跑一次 A/A 实验,只换规划器的随机种子,量出规划方差。三是把工具调用序列记进产物。 + +### b. LLM 对接 + +系统只用 OpenAI 的接口写答案,并且强制模型只能引用服务端给它的 ID。 + +| 项 | 内容 | 位置 | +|---|---|---| +| Provider | OpenAI Chat Completions,`openai` 2.49.0,`base_url` 可覆盖以指向兼容端点 | `apps/agent/src/dcode_agent/llm.py:129-154` | +| 模型 | `gpt-4o-mini` 写答案。配成 `stub` 时退回规则模板 | `llm.py:211-236` | +| Streaming | `stream=True`,每个 delta 转成一条 `partial_answer` 事件 | `llm.py:156-180`;`graph.py:266-274` | +| 引用协议 | 系统 prompt 里五条强制规则:只能引用服务端目录里的 `[C#]` ID;ID 放在反引号外面;不许拿文件名、行号、符号当替代引用;每个具体主张附一个 ID;证据不够就直说 | `llm.py:43-53` | +| 调用图规则 | 另外三条:已解析的静态边和源码里可见的调用表达式必须分开说,未解析的不许说成已解析 | `llm.py:54-60` | +| 语言 | 答案语言按**原始问句**判定,追问改写不影响 | `llm.py:75-86`;`graph.py:269` | +| 追问改写 | 单独一次调用,`temperature=0`,`max_tokens=128` | `llm.py:182-208` | + +### c. 工具层 + +11 个工具,参数都用 Pydantic schema 定义,执行前先校验。 + +注册表在 `apps/agent/src/dcode_agent/tools/__init__.py:22-39`。 + +| 类别 | 工具 | 出口 | +|---|---|---| +| 检索 | `search_code` | HTTP 打网关的内部检索口 | +| 符号和图 | `find_definition` · `find_references` · `find_call_path` · `get_call_neighbors` · `get_dependencies` · `get_dependents` | HTTP 打网关的内部图查询口 | +| 文件结构 | `get_file_outline` | HTTP | +| 本地 | `read_file` · `grep` · `list_directory` | 共享 volume 或 ripgrep 子进程 | + +**统一抽象。** 每个工具是一个 `Tool[ArgsT, ResultT]`,暴露 `name`、`description`、`ArgsSchema`、`execute`、`cache_key` 五样东西(`tools/base.py:25-38`)。工具从不直接连数据库(`base.py:10-12`),只走 HTTP 或本地文件系统。 + +**参数校验放在执行之前。** `tool_call_node` 先跑 `tool.ArgsSchema(**args)`(`graph.py:113`)。校验失败走降级,不抛给用户(`:114-117`)。`read_file` 还有一个 `model_validator`,拒绝起始行小于 1 或起始行大于结束行的区间(`tools/read_file.py:17-24`)。 + +**路径穿越拦两层。** 第一层在字符串上判:绝对路径直接拒(`tools/common.py:55`),`..` 逃出仓库根目录也直接拒(`:63`)。第二层在解析之后用 `is_relative_to(root)` 再兜一次(`:74-75`)。三个碰文件系统的工具各有一个穿越测试(`test_tools_execute.py:216`、`:237`、`:261`)。 + +**缓存键的写法推出一条不显然的约束。** 工具缓存键是 `tool:{tool}:{repo_id}:{sha256(canonical_json(args))[:16]}`(`packages/shared/src/dcode_shared/cache.py:19-21`)。既然键由参数算出来,那么任何影响工具行为的东西都必须放进参数。`search_code` 的检索模式因此是一个工具参数,不是环境变量。模式要是挂在 arm 级别的环境里,两个只差检索模式的实验臂会撞同一条缓存条目。评测结果就被静默污染了。理由写在 `tools/search_code.py:8-11`。 + +### d. Context 与 Memory + +送进 prompt 的证据先补齐源码,再由同一个模型统一排序。 + +**先补源码。** 检索结果自带源码。图查询结果只有 `file:line`,没有代码。两者直接一起送进 prompt,打分模型会偏向有源码的那一边。它比的是哪边有东西可读,跟相关性无关。这是测量偏差。 + +所以代码先按 `chunk_id` 批量取回图结果的源码(`graph.py:498-529`,服务端 `GET /internal/get_chunks`)。补齐之后,再让同一个 cross-encoder(把问题和代码拼成一段文本一起送进模型打分,比分别编码更准但更慢)给所有候选打分(`graph.py:437-490`)。 + +**排序特征里刻意不放证据来源和图距离**(`graph.py:451-454`)。要测调用图有没有用,就不能在打分函数里假定它有用。一个在评测集上调出来的「偏好结构证据」先验,就是一个为了让假设通过而挑的超参数。代价我认:六个图工具里有四个在整轮评测中一次都没产出过被引用的证据。 + +**上下文预算是 10 条,所有实验臂一样**(`graph.py:432`)。这个数约束的是最强的那个臂:图证据必须挤掉检索证据才能进上下文,不能白送。最弱的臂最多只有 5 个检索块,给它更大预算没意义。给最强的臂更大预算,会让上下文长度变成混淆变量。 + +**超预算的证据清掉正文但保留 ID**(`graph.py:486-488`)。它仍然能被引用、能被验证,只是不占上下文。这样结构性叙述不会引用到一个不存在的 ID。 + +**检索栈**(`apps/api/src/dcode_api/routes/internal.py:241-299`): + +``` +sparse: 应用层手写 Okapi BM25,k1=1.2,b=0.75 + 参数写进运行产物,所以这条基线可复现 +dense : pgvector HNSW 余弦 + ↓ 两路各取 50 个候选 +fuse : 加权 RRF,K=60,dense:sparse = 2:1 + ↓ +rerank: cross-encoder 重排前 16 条 + ↓ +take : 取 5 条,并在排序之后过滤测试文件(候选池和融合算术不受影响) +``` + +术语解释一下。**BM25** 是经典的关键词打分算法,按词频和文档长度算相关度。**pgvector** 是 PostgreSQL 的向量扩展,能直接在库里做向量近邻查询。**RRF**(Reciprocal Rank Fusion)把两个排序结果按名次融合,不比较分数。 + +融合用 RRF,不用「分数归一化后相加」。BM25 分数无界,还依赖语料的 idf 和文档长度分布。余弦相似度有界,分布也很集中。名次是这两个排序器之间唯一不需要假设分布就能比较的量。权重不取等权有实验依据,而且两个方向各有一个测试钉住(`test_internal_routes.py:444` 和 `:465`)。 + +**多轮历史由客户端带上,网关截断一次**:最多 6 轮,总共 2000 字符,单轮 4000 字符(`apps/api/src/dcode_api/settings.py:15-17`)。截断只做一次,让规划器、转发给 agent 的 body、缓存键看到同一份。只有收到过终答的轮次才会被回填成上下文(`useThread.ts:43-55`)。没经过删除的草稿,永远不会作为历史再被引用。 + +**服务端没有长期记忆**,这是刻意的。一旦有服务端会话状态,缓存键就不再决定输出,评测的可复现性也就没了。 + +### e. 可靠性与控制 + +agent 最多跑 14 步,失败也算一步。工具出错时照样给答案,基础设施出错时直接报错。 + +**步数上限 14,失败也计步**(`apps/agent/src/dcode_agent/settings.py:14`,递增点在 `graph.py:168` 和 `:189`)。这个数从 8 提上来的。提它的原因是我发现最强的实验臂总是终止在预算上,而不是终止在规划器的判断上。那种情况下测的是预算,不是策略。失败也计步,循环长度的上界才是硬的。 + +**除步数外还有七条提前退出分支。**它们是:规划器返回空、`state.error` 非空(两处)、步数触顶(两处)、待执行工具为空、工具调用出边遇错。其中一条在当前图拓扑下走不到,底稿漂移清单里已经标成防御性分支。 + +**失败分两层处理,分界在 `_record_tool_failure` 这个函数里**(`graph.py:175-209`): + +| 层 | 处理 | 用户看到 | +|---|---|---| +| 工具层:参数校验失败、执行失败、缓存解码失败 | 降级,转去生成答案 | 照样拿到答案,开头写明哪个工具出错了 | +| 合成层:LLM 流式失败 | 降级,退回规则模板生成 | 照样拿到答案 | +| 排序层:reranker 失败、证据取源码失败 | 降级,退回 observation 顺序或该条留空 | 无感知 | +| 基础设施层:registry 缺失、验证时数据库出错 | 终止,发 `error` 事件 | 明确的失败,不给答案 | + +**重试策略每一层理由都不同:** + +| 层 | 重试次数 | 理由 | +|---|---|---| +| 工具层 / 内部 HTTP | 零 | 工具失败多半是参数或索引问题,重试不会变好。降级到「用已有证据回答」更快,也更诚实 | +| embedding | 12 次,指数退避封顶 30s。4xx 立即抛出 | 索引是离线的,可以等。模型 sidecar 冷启动要几分钟 | +| reranker | 3 次。`ReadTimeout` 刻意不在重试白名单里 | 查询是在线的,等不了。重试一个慢的 CPU 推理,只会往单线程模型服务上堆重复工作 | +| 网关 → agent | 显式不重试 | 流式响应不幂等。重试会把已经流出去的那半个答案再发一遍 | + +### f. 幻觉控制与 attribution + +模型只能引用服务端给它的 ID,查不到的引用会被删掉,而分数按删除之前算。 + +**第一,引用 ID 由服务端下发。** 模型只能引用本次请求发给它的 `[C#]` ID。不在目录里的 ID 直接判失败(`groundedness.py:184-194`)。 + +**第二,普通反引号代码不算引用**(`groundedness.py:150-151`)。`self.client.retrieve` 这种写法是代码格式化,不是证据主张。上一版把所有点分内联代码都当引用,结果模型每写一个方法名都在「引用」。 + +**第三,先验证再删除,分数按删除之前算。** 未通过验证的引用从正文里删掉(`groundedness.py:407-426`),只有通过验证的才变成引用事件(`graph.py:727-738`)。但分数取删除**之前**的比例(`groundedness.py:403`)。这一步是整套机制的关键:分数要是按删除之后算,那每个答案都是满分,删除就成了洗白。 + +**第四,一条引用都没有的答案计 0 分,不计满分**(`groundedness.py:272`)。另外用一个独立计数把「没引用」和「引用全错」分开。不这么做的话,一个学会闭嘴的 agent 就能刷满分。我也没有选「从分母里剔除这类答案」,因为「在不确定的地方就不引用」同样能抬高均值。 + +$$ +g(a)=\begin{cases}\dfrac{1}{n}\displaystyle\sum_{i=1}^{n} v(r_i), & n > 0\\[2ex] 0, & n=0 \quad \text{(约定,非推导)}\end{cases} +$$ + +**还有一个容易漏但不能错的细节。** 符号验证用 SQL `LIKE '%.name'` 缩小候选范围。Python 标识符里下划线极常见,而下划线在 `LIKE` 里是单字符通配符。不转义的话,`api._get` 会连 `api.aget` 一起匹配上。这是守卫里的静默假阳性,是这套系统唯一不能出的错误(`packages/shared/src/dcode_shared/symbols.py:56-64`)。同一个文件还记着一件事:符号解析规则曾经有两份,工具一份、守卫一份。结果同一个请求里,工具说「在这儿」,守卫说「不存在,删掉」。现在收成一份了。 + +### g. 评估与可观测 + +五级 baseline,判定标准提前写死,另有一个对照实验臂用来拆解增益来源。 + +四个 agent 臂共用同一个生成模型、同一份 prompt、同一套引用协议、同一个守卫,差异只来自一张映射表(`apps/agent/src/dcode_agent/state.py:32-44`)。这是对照实验能成立的前提。 + +| arm | 检索 | 答案路径 | 进不进判据 | +|---|---|---|---| +| B0 | 外部 GitHub code search | 模板 | 不进(不可复现) | +| B1 | 应用层 BM25 | 模板 | 不进(检索参照) | +| B2 | 只用 dense | 共享 agent 生成 + 同一守卫 | 进 | +| B3 | hybrid | 同上,不做工具扩展 | 进 | +| **B3.5** | hybrid | 同上,保留读文件和文件结构,去掉图工具 | **不进(对照实验臂)** | +| B4 | hybrid | 同上,全部工具 | 进 | + +**判定标准是预注册的合取**:阈值 0.05,两个推理层乘两个对手等于四个比较,四个全过才算过(`apps/eval/src/dcode_eval/run.py:545-560`)。预注册的意思是,判定标准在跑实验之前就写进代码并提交,跑完不许改。用合取而不是取平均,因为取平均允许一层的大胜掩盖另一层的失败。 + +打分用的 composite 是三个检索指标取平均得到的合成分。三个指标是:recall(该找到的找回了几成)、MRR(第一个正确结果排第几名,越靠前分越高)、nDCG(正确结果排得越靠前分越高,比 MRR 多考虑后面的命中)。 + +**B3.5 这个对照实验臂把系统增益拆成两项。**消融实验(ablation)的做法是关掉一个部件、其他全部不变,看指标掉多少: + +$$ +\underbrace{B4 - B3}_{\text{整个 agent 系统}} = \underbrace{(B4 - B3.5)}_{\text{调用图本身}} + \underbrace{(B3.5 - B3)}_{\text{多步取证}} +$$ + +**我给自己加了一条约束**:B3.5 在代码层面被排除在判据之外(`run.py:599-601`、`:605-616`,由 `apps/eval/tests/test_run.py:686` 钉住)。把一个新臂加进判据,就是事后改判定规则。所以这个项目里最有价值的发现只能作为诊断项出现,动不了主判定。 + +**可观测的东西**:SSE(Server-Sent Events,服务端单向推送,适合流式输出)推理轨迹分 `thought`、`tool_call`、`tool_result` 三类事件,前端可以展开逐步看每次工具调用的参数和结果摘要。groundedness 药丸只在收到终答后显示分数,流式期间是中性色。索引侧有结构化阶段日志和 Redis 实时快照。跳过的文件和失败原因都会回传前端。 + +有一条必须写明:**judge 没接,所以答案质量完全没有测量。**`Judge` 抽象类和四轴 rubric、pairwise 接口都写好了(`apps/eval/src/dcode_eval/metrics/judge.py:27-36`),但后面没有接任何真实模型,产物里 `pairwise_win_rate` 全是 `null`。这套评测能测「有没有找到正确的代码、引用是不是真的存在」,测不了「解释得对不对、清不清楚」。 + +### h. 工程化 + +SPA 只跟网关说话,四个应用包之间一行 Python import 都没有。 + +**单一 API 缝。** SPA 从不直连 agent 或数据库。agent 反过来经 HTTP 打网关的内部检索口。整个仓库是 7 个 uv workspace 包(`pyproject.toml:8-16`)。四个应用包之间零 Python import,只经 HTTP、AMQP、SQL、Redis 通信。 + +评测 harness 是最干净的一个。它只依赖 `httpx` 和共享 schema 包(`apps/eval/pyproject.toml:6-9`)。它在结构上没办法 import 被测系统,只能像外部客户端一样打 HTTP 口。这保证了评测不会因为共享进程内状态而作弊。 + +**部署与检查链**: + +| 项 | 内容 | +|---|---| +| 容器化 | `docker compose` 起 6 个常驻服务,加 2 个挂 profile 按需启动的模型 sidecar。每个服务带健康检查 | +| 本地检查 | `make check` = lint + typecheck + test(`Makefile:80`)。`make lint` 还多跑一步评测产物一致性校验(`Makefile:70`)。UI 和文档里显示的评测数字要是和 `results/` 对不上,lint 直接失败 | +| CI | `.github/workflows/ci.yml` 两个 job。Python 那个跑 ruff、mypy --strict、pytest、eval smoke。Frontend 那个跑 eslint、tsc、vitest、build | +| 类型 | Python 全量 `mypy --strict`。前端 TS strict,另有一个编译期测试钉住前后端 schema 镜像 | + +这里有一个区分要说准:**评测产物一致性校验只在 `make lint` 里跑,CI 不跑它。**`ci.yml` 的 Python job 只有 ruff、mypy、pytest、eval smoke 四步。 + +--- + +## 后端基础设施 + +这一节讲索引流水线、消息队列、数据库、缓存这些不涉及模型的部分。 + +### 索引:六态严格单调状态机加双写 + +外部能看到的状态有六个:`queued → cloning → parsing → embedding → graphing → ready`,任意一点都能转 `failed`(`packages/shared/src/dcode_shared/schemas.py:18-27`)。实现上是 4 个 `PipelineStage` 条目驱动 5 个 stage 函数:`clone`、`parse`、`chunk`、`embed`、`graph`(`pipeline.py:51-56`)。`parsing` 这一个状态下跑 parse 和 chunk 两个函数。 + +单调性由数据结构保证。阶段顺序是一个不可变元组,没有跳转表,没有条件分支。 + +**双写。** Postgres 的 `repos` 行是持久真相。Redis 的 `job:{repo_id}` 是给前端轮询用的细粒度快照,里面有每个阶段的状态、进度、warning 列表。Redis 写失败会被吞掉,不影响索引本身。 + +**有一处原子性很关键。** 删除旧 chunk 和递增 `index_revision` 在同一个事务里完成(`stages/embed.py:173-179`)。因为稀疏检索的语料缓存以 `(repo_id, index_revision)` 为键,两者必须一起变。 + +### 消息队列与幂等 + +`POST /repos` 先落库并 commit,再发布持久化 AMQP 消息(`apps/api/src/dcode_api/deps.py:44-58`)。顺序不能反,反了 worker 会读不到行。 + +**发布失败有补偿。** 行已经持久了,这时把它标成 `failed` 并回 503,不留一个永远 queued 的僵尸行。 + +worker 侧设了 `prefetch_count=1`,队列是 durable 的,处理完才 ack(`apps/worker/src/dcode_worker/main.py:19-30`)。`handle_job` 保证不向 RabbitMQ 抛异常,抛了消息会被无限重投。 + +**URL 幂等。** 有 6 种状态算可复用,`failed` 刻意不在其中(`routes/repos.py:97-104`)。上一次失败正是用户想重试的时候。URL 变体归一化做得很窄:只处理尾斜杠、`.git` 后缀、scheme 和 host 的大小写。路径大小写保留(`:107-123`),因为把两个真正不同的仓库合并,比留一个重复行更糟。 + +### 数据模型与索引 + +四张运行时表:`repos`、`chunks`、`symbols`、`edges`(`packages/shared/src/dcode_shared/db/models.py`)。全部按 `repo_id` 隔离。四个 Postgres 原生 ENUM 和 Pydantic 的 StrEnum 一一对应,由测试钉住。 + +索引建在 alembic migration 里,不建在 ORM 里。因为 pgvector 的 operator class 没法用 SQLAlchemy 可移植地表达。具体有:`chunks.embedding` 上的 HNSW `vector_cosine_ops`、`(repo_id, file_path)`、symbols 上的 `(repo_id, qualified_name)` 唯一索引、edges 上的正向和反向双索引 `(repo_id, source_id, edge_type)` 与 `(repo_id, target_id, edge_type)`。反向那个专为「谁引用了它」这类查询建的。 + +**向量维度是在 migration 执行那一刻绑定的。**`Vector(embedding_dim)` 同时出现在 ORM 和 migration 里,而 migration 那份会把 `alembic upgrade` 执行时的环境值固化进 DDL。所以改维度必须重建库。这是个容易踩的运维陷阱,已经写进文档。 + +### Redis 四个键空间 + +| 键 | TTL | 设计要点 | +|---|---|---| +| `embed:{model}:{sha256(text)}` | 永久 | 按内容寻址。同一段代码出现在不同仓库里,只算一次向量 | +| `tool:{tool}:{repo_id}:{args_hash}` | 24h | 含参数哈希。所以影响工具行为的东西都得进参数 | +| `query:{repo_id}:{sha256(history + query)}` | 1h | 含对话历史。不含的话,上下文依赖的追问会和单轮同问题撞键 | +| `job:{repo_id}` | 完成后 7 天 | 进行中不设 TTL,完成才设 | + +两个细节。用 `\x1f`(unit separator)分隔历史和问题,因为这个字符不可能出现在 JSON 里,边界没有歧义。历史为空时,digest 和旧版单轮键逐字节相同。所以加多轮之后,历史缓存不会被孤立。有测试钉住这一点。 + +### SSE 网关 + +网关一边往前端转发,一边把整条流缓冲下来。**只有不含 `error` 事件的流才写回缓存**(`routes/query.py:72-73`)。缓存一个失败的回答一小时,比不缓存更糟。命中缓存时,整条流逐字节重放。 + +**agent 不可达时**,网关自己发一个明确标了 `(skeleton)` 的 `thought` 事件,再发一个 `error` 事件(`:96-109`)。这样 SSE 协议是诚实终止的,不是静默挂断。 + +### 安全边界 + +- **SSRF 白名单。** 仓库 URL 只接受 http(s)、ssh、git 和 scp-like 四种形式。`localhost`、私有地址、环回、链路本地、组播、保留、未指定的 IP 全部拒掉(`routes/repos.py:187-224`)。 +- **内部密钥。** 所有 `/internal/*` 路由都校验 `X-Dcode-Internal-Key` 头(网关 `routes/internal.py:39-49`,agent `main.py:176-181`)。前端拿不到这一层。 +- **路径穿越。** 见「工具层」。 +- **配置硬化测试。** 有测试断言 `.env.example` 和生产 compose 不得内置已知弱凭据,必须显式要求 secret 环境变量(`packages/shared/tests/test_config_hardening.py`)。 + +--- + +## 面试口径 + +这一节写完整结论,包括没通过的那部分。它和简历 bullet 信息一致,只是密度不同,理由见「给 mentor 的说明」第 4 条。 + +**1. 主假设的判定是 `unsupported`。** 判据是预注册的合取,四个比较过了三个。架构级对两个对手都过(+0.247 和 +0.169)。跨文件级对稠密 RAG 过(+0.136),对 hybrid+rerank 差 0.0061 没过。我不撤回也不软化这个结论,它是我自己写的执行规则算出来的。 + +**2. 跨文件那条测不出来。** 三次相同配置的运行跑出 +0.038、+0.006、+0.088。极差 0.083 比 0.05 的判定门槛本身还宽。0.0061 这个差额对应的重复间标准差是 0.034(总体口径,样本口径是 0.042),差额只有噪声的五分之一多一点。所以正确的补救是加题或换语料,不是多跑重复。在这个题量下解决这个量级的差异,需要约 100 次重复。健全性检查是:纯检索基线三次重复的指标逐位相同,说明方差来自答案生成,不来自检索。 + +**3. 对照实验的结论要带层级限定。** 调用图的孤立贡献是跨文件级 +0.0218、架构级 +0.0226。多步取证的贡献是跨文件级 +0.0221、架构级 +0.1466。所以「多步取证远大于调用图」只在架构级成立,跨文件级两者相当,比值 1.01×。这两个数都是诊断项,代码层面排除在判决之外。 + +**4. 两次评测协议改动,方向相反。** 计分协议从 v1 改到 v2,让四个 agent 臂用同一条规则。改完我自己的结论从 supported 变成 unsupported。同一批改动里我还把 groundedness 从 composite 拿掉了。那确实是降门槛。被删项在所有臂上恒为 1.000,去掉它等价于把门槛从 0.050 降到 0.0375。我不换说法。 + +能给的辩护只有三条,都能核对。事先声明并提交。两种口径同时发布在产物里。两种口径都返回 unsupported。 + +**5. 题集有一处混淆是我后来才发现的。** 分层标注和题目来源在这份数据里共线。单文件层全是人工题,反推题全在跨文件和架构级两层,而且 gold 集合大一倍。所以「在架构级领先」和「在反推题上领先」分不开。我测过混淆的方向,它不利于自己。调用图在反推题上的孤立贡献是 +0.0183,在人工题上是 +0.0283。反推题上更小。但这不等于排除了偏置。偏置的形式是「题集里缺什么」,用题集自身的指标测不出来。这条已经补进运行产物的 limitations。 + +**6. 记录的那次运行跑在比当前代码旧的调用图上。** 用当前 worker 代码离线重跑同一语料,会得到 316 条调用边,而库里是 303 条,多 4.3%。其余三种边逐项相同,所以差异被精确定位到调用解析这一个函数上。改进已经提交,但需要重索引才生效,而中途重索引会让语料在几次对比之间移动。 + +**7. judge 没接,答案质量完全没有测量。** 见「评估与可观测」。 + +**8. 静态调用图的精度是量化过的。** 解析器实际访问到 2615 个 `ast.Call` 点,解析成内部符号的是 316 个。分母是 worker 真正遍历到的调用点,不含模块级语句和嵌套类的方法。这个量级对「无类型推断的纯静态 AST 分析」是合理的。最大的一类未解析是 1076 个 `obj.X()`,接收者的类型未知。解决它需要类型推断,而类型推断是我明确声明放弃的能力。漏一条边是安全的,建错一条边才危险。所以设计上宁可不建边。未解析的调用也没有被丢弃,而是显式暴露给模型,prompt 里有规则禁止把它说成已解析。 + +--- + +## 简历可用 bullet 草稿 + +效果一栏只用两种写法:台账里的 A/B 级数字(带限定语),或者机制性结果。没有吞吐、延迟、成本、准确率提升数字,原因见下一节。 + +1. **设计并实现**一套端到端代码理解系统(FastAPI + LangGraph + PostgreSQL/pgvector + RabbitMQ + Redis + React 18),含 7 个 uv workspace 包、6 个常驻服务、2 个模型 sidecar、**364 个自动化测试**。四个应用包之间零 Python import,只经 HTTP / AMQP / SQL / Redis 通信。评测 harness 因此无法 import 被测系统。 + +2. **构建**引用验证机制:引用 ID 由服务端下发,模型只能引用本次请求发给它的 ID。**未通过索引验证的引用会被从答案正文中移除**。groundedness 分数按移除之前的比例计算,**所以移除不能提高分数**。一条引用都没有的答案计 0 分而非满分,堵掉「少说少错」的刷分路径。 + +3. **实现**有界 agent 编排:LangGraph 5 节点状态机,11 个工具用 Pydantic schema 定义参数。路径穿越拦两层,每个碰文件系统的工具各有一个穿越测试。**步数上限 14 且失败也计步**。这个数由 8 提高而来。最强实验臂持续终止在预算上,说明预算在替规划器做决定。另有 7 条提前退出分支。失败分两层处理,工具层降级、基础设施层终止,重试次数逐层不同。 + +4. **设计**可证伪的离线评测:**五级 baseline 阶梯加预注册合取判据**,四个比较全过才算过。33 题冻结题集共 **62 个标准答案代码块**,**分层标注为人工标注,代码中无判定器**。三次重复共 **495 次 agent 调用**,以分离生成的随机性。另自建 **B3.5 对照实验臂**,把增益拆成调用图与多步取证两项。**该臂在代码层面被排除在判据之外**,防止事后改规则。 + +5. **搭建**混合检索链路:应用层实现 Okapi BM25(参数写进产物,基线可复现),配 pgvector HNSW 余弦。两路经**加权 RRF 融合**后 cross-encoder 重排,排序后过滤测试代码,**候选里测试文件占比从 47% 降到 0**。**排序不含来源与图距离**,避免在打分里预设结论。 + +6. **建立**异步索引流水线:**六态严格单调状态机**,4 个 PipelineStage 驱动 5 个 stage 函数。Postgres 持久真相加 Redis 实时快照双写,chunk 替换与语料版本号在**同一事务**内递增。`POST /repos` 在 URL 上幂等复用,而**失败态刻意不复用**以保留重试语义。`make lint` 阶段校验评测产物一致性,**UI 与文档中的评测数字若与产物漂移则直接失败**。 + +7. **排查并统一**引用验证的符号解析规则:此前守卫用精确匹配、检索工具用后缀匹配,同一请求内工具报「已找到」而守卫报「不存在,删除」。收敛为一条共享规则后,在三臂九次受控对照运行中,**已验证引用由 48–54 条升至 62–67 条,抹除标记由 17–21 降至 9–13**,符号形式引用由每次运行 0 条变为 2 条。 + +### 七条太多了,按渠道取舍 + +我不删,你自己选。两种渠道看的东西不一样。 + +**技术岗投递,留 4 条:2 · 4 · 7 · 3。** + +| 留 | 理由 | +|---|---| +| 2 引用验证 | 技术面试最想看的是「你怎么让一个会编的模型给出可核对的输出」 | +| 4 评测设计 | 它证明我能定一个自己可能过不了的标准 | +| 7 符号规则 | 唯一一条完整的「排查缺陷 → 前后计数」,最像日常工作 | +| 3 agent 编排 | 步数、失败分层、重试策略,是这个岗位的基本功 | + +拿掉 1 / 5 / 6,因为它们讲的是标准的后端与检索工程,可替代性最高。 + +**HR 池与网申,留 3 条:1 · 2 · 5。** + +| 留 | 理由 | +|---|---| +| 1 系统规模 | 非技术读者先看体量:几个包、几个服务、多少测试 | +| 2 引用验证 | 一句话能讲清的卖点,不需要背景知识 | +| 5 检索链路 | BM25、向量、重排是岗位关键词,筛简历的系统和人都在找它们 | + +拿掉 3 / 4 / 6 / 7,因为它们要读者理解评测设计或排障过程,在 30 秒里讲不完。 + +--- + +## 数据清单:每个数字能证明什么 + +这一节把所有能用的数字集中在一起,并写清每个数字对读者意味着什么。分成三组:第一组可以直接写进简历,第二组要跟一句限定语才能用,第三组只在面试里口头说。 + +### 第一组 · 可以直接写进简历 + +这一组不需要背景知识。读者扫一眼就知道量级。 + +| 数字 | 它是什么 | 它能证明什么 | 出处 | 已用在 bullet | +|---|---|---|---|---| +| **726** | 一个仓库索引出的代码块数 | 处理的是真实仓库,不是玩具样例 | 底稿 §4.1(A) | 未用 | +| **724 / 37** | 符号总数 / 参与索引的 Python 文件数 | 索引不止切块,还建了符号表 | 底稿 §4.1(C+D / C) | 未用 | +| **62** | 33 道题去重后的标准答案代码块数 | 对错有客观依据,不靠我判断 | 底稿 §4.1(A) | 已用 | +| **364** | 自动化测试用例总数 | 有人在持续维护,不是一次性 demo | 盘点 C 档(C,291+73) | 已用 | +| **24,821** | 代码总行数 | 规模一个人能读完,也够复杂 | 盘点 C 档(C 派生) | 未用 | +| **257** | git commit 总数 | 改动是渐进的,有完整历史 | 盘点 C 档(C) | 未用 | +| **495** | 一轮完整评测里 agent 被调用的次数 | 这是一套评测系统,不是跑几个例子 | 盘点 C 档(A 派生) | 已用 | +| **33 = 5 / 16 / 12** | 题数与三个难度层 | 题目按难度分层,能看出哪层弱 | 底稿 §4.4(A+B) | 已用 | +| **3** | 同一配置重复跑的次数 | 我知道单次运行会骗人 | 底稿 §4.4(A) | 已用 | +| **5** | baseline(对照基线)阶梯的层数 | 有参照系,不是自己跟自己比 | 盘点 C 档(B) | 已用 | +| **5** | 检索截断 k | 打分只看前五条,标准固定 | 底稿 §4.4(A) | 未用 | +| **7** | uv workspace 包数 | 边界是切开的,不是一个大文件 | 底稿 §4.6(B) | 已用 | +| **6 + 2** | 常驻服务数 + 模型 sidecar 数 | 跑起来要编排,不是单进程脚本 | 底稿 §4.6(B) | 已用 | +| **11** | agent 工具数 | agent 真的会调工具去查证 | 底稿 §4.3(B) | 已用 | +| **5** | LangGraph 节点数 | 流程是显式的,能画出来也能测 | 盘点 C 档(B) | 已用 | +| **768** | embedding 向量维度 | 用的是真实模型,不是占位 | 底稿 §4.2(A) | 未用 | +| **0** | 整轮评测观测到的服务错误数 | 跑完一整轮没有故障 | 底稿 §4.5(A) | 未用 | +| **47% → 0%** | 检索候选里测试文件的占比,改动前后 | 我修过一个用户能感觉到的缺陷 | 盘点 甲-3(B 前 / C 后) | 已用 | +| **48–54 → 62–67** | 已验证引用条数,符号规则统一前后 | 修完之后能引用的证据变多了 | 盘点 甲-1(A) | 已用 | +| **17–21 → 9–13** | 抹除标记数,同一改动前后 | 用户看到的残缺引用变少了 | 盘点 甲-1(A) | 已用 | +| **0 → 2** | 每次运行存活的符号引用数,同一改动前后 | 一类引用此前全军覆没,现在活了 | 盘点 甲-1(A) | 已用 | + +### 第二组 · 要跟一句限定语才能用 + +这一组的数字单独说出去会被追问,跟上那句话才安全。 + +| 数字 | 必须同时说的那句话 | 不说会怎样 | 出处 | +|---|---|---|---| +| **473** | 这是四种边之和,调用边只有 303 | 对方问「imports 占多少」,当场见底 | 底稿 §4.1 | +| **303** | 这是库里的值,当前代码重索引会得 316 | 对方拉代码自己重算,两个数对不上 | 底稿 §4.1 | +| **2615 → 316,12.1%** | 分母是 worker 真正遍历到的调用点 | 裸写 12% 像能力不足,实际是声明放弃类型推断 | 底稿 §4.1 | +| **groundedness 1.000**(引用可信度) | B1 的 1.0 是代码里的字面量,agent 臂的才是实测;还要配上「无引用答案数为 0」 | 对方问「怎么可能满分」,答不上就像在刷分 | 底稿 §4.5 | +| **candidate_recall 0.560** | 三个 hybrid 臂完全相同,它测的是检索侧 | 当成 B4 的成绩说,对方一比就发现三臂一样 | 底稿 §4.9 第 15 条 | +| **768 / 1024** | 实跑 768,代码和 compose 的默认都是 1024 | 对方读 `settings.py` 看到 1024,以为我记错了 | 底稿 §4.2 | +| **`unsupported`** | 四个比较过了三个 | 只说没通过,听起来像全盘失败 | 底稿 §4.5 | +| **0.0061** | 对应的重复间标准差是 0.034 | 说「只差 0.006」会被追问方差,然后暴露极差 0.083 | 底稿 §4.5 | +| **+0.038 / +0.006 / +0.088** | 极差 0.083 比 0.05 的门槛还宽 | 单挑第三次说「过了」,那是挑有利重复 | 底稿 §4.5 | +| **+0.0218 / +0.0226 / +0.0221 / +0.1466** | 这四个是诊断项,代码层面排除在判定之外 | 当主结论说,对方会问为什么不进判据 | 底稿 §4.5 | +| **4/3** | 等价于把门槛从 0.050 降到 0.0375 | 不说就是隐瞒降门槛,被读到 `run.py` 注释更难看 | 底稿 §4.5 | +| **12/12/12 与 14/12/16** | 这是被废弃的旧集合的值 | 拿旧值当成绩,对方问新值就崩 | 底稿 §4.5 | +| **0/0/0 与 4/3/5** | 这是重算值,不是已提交字段 | 对方在产物里搜不到这两组数 | 底稿 §4.5 | +| **440/248/152/111/36** | 六个图工具里有四个一次都没出现 | 只报总数像图工具很有用,实际四个是零 | 底稿 §4.5 | +| **9/16 vs 10/17** | 两类题几乎一样,反推题没有更多触发调用图 | 单报一个比例,读者不知道跟谁比 | 底稿 §4.5 | +| **BM25 的 1.2 / 0.75**(关键词打分算法的两个参数) | BM25 是应用层手写的,不走 `tsv` 和 GIN 索引 | 对方以为我用了 Postgres 全文检索 | 底稿 §4.2 | +| **291** | 只做了收集,没跑执行 | 说「291 个测试全绿」,被要求现场跑就露馅 | 底稿 §4.6 | +| **73** | `CLAUDE.md` 里写的 71 已经过时 | 两份文档数字打架 | 底稿 §4.6 | +| **10,952** | 排除了 `node_modules` 下 16 个共 1743 行的 `.py` | 对方自己 `wc -l` 会得到 12,695 | 底稿 §4.6 | +| **6 + 2** | `make up` 不会启动那两个 sidecar | 现场演示时每个查询都死在 embedding | 底稿 §4.6 | +| **4 + 1** | `index_runs` 表建好了,代码里没人写 | 对方翻 migration 发现那张表是空的 | 底稿 §4.6 | +| **9/33 · q-009** | v2 口径下 recall(该找到的找回了几成)被答案引用条数封顶 | 对方问「为什么 recall 这么低」 | 底稿 §4.5 | +| **B0 的 file 级三个数** | H1 那次运行里没有任何 arm 有 file 级指标 | 对方问「跟谁比」,而答案是没得比 | 底稿 §4.7 | + +### 第三组 · 只在面试里口头说 + +| 内容 | 为什么不进简历 | 口头怎么讲 | +|---|---|---| +| 计分协议 v1 → v2,四个 agent 臂统一规则 | 需要整段评测背景 | 「我换过计分口径,让四个臂用同一条规则。换完我自己的结论从通过变成没通过。」 | +| `GRAPH_TOOLS` 集合修正 | 需要整段背景 | 「统计口径把一个无图臂也保留的工具算成了图的功劳。一个分不清有图和没图的度量,测的不是图。」 | +| 引入 B3.5 对照臂(消融:关掉一个部件、其他不变,看指标掉多少) | 是诊断项,代码层面不进判定 | 「我加了一个我不需要的臂。它把我自己项目的卖点降级了,而我没有用它去改判决。」 | +| 从 composite(三个检索指标的平均分)里去掉 groundedness 项 | 它本身不是成效 | 「我删了一项指标,那确实是降门槛。三条辩护都能核对,而且两种口径都不通过。」 | +| 零引用答案的计分从 1.0 改成 0.0 | 它本身不是成效 | 「我改了一个计分约定,结果我自己此前那个实验的两条结论都不成立了。改完的数字对我更不利。」 | +| 图证据补源码,L3 调用图贡献 **−0.0067 → +0.0226** | 是诊断项,不进判定 | 「有一次评测里,关掉调用图的对照臂在架构级问题上反而赢了。查下来是测量偏差:图结果只有文件名行号,检索结果带源码。补上源码再统一打分,同一个指标从负变正。」 | +| 调用解析改进,调用边 **303 → 316** | 它本身不是成效 | 「我修了继承和局部构造两种调用形态。边数多了十几条,但我自己测了下游影响,解锁的架构级标准答案链条是零,而且要重索引才生效。」 | +| RRF 常数 **K=60**、权重 **dense:sparse = 2:1** | 是配置,读者判断不了 60 好不好 | 「用 RRF(按名次融合两个排序结果)而不是分数相加,因为 BM25 分数无界、余弦有界。权重非等权是因为等权时纯稠密会反超混合。」 | +| 候选漏斗 **50 → 16 → 5** | 是配置 | 「检索是三段漏斗:两路各取 50 个候选,融合后送 cross-encoder(把问题和代码拼一起打分)重排前 16 条,最后返回 5 条。」 | +| BM25 的 **k1=1.2 / b=0.75** | 是配置 | 「参数写进运行产物,所以这条基线可复现。这也是我不用 `ts_rank` 的原因,它不是 BM25。」 | +| 步数上限 **14** | 是配置 | 「从 8 提上来的。提它是因为最强的臂总是终止在预算上,不在规划器的判断上。」 | +| 证据上下文预算 **10** | 是配置 | 「对所有臂相同,约束的是最强的那个。图证据必须挤掉检索证据才能进上下文。」 | +| 基类 BFS 深度上限 **6** | 是配置 | 「是 MRO 的近似,不是 MRO。它忽略 C3 线性化,菱形继承可能选错分支。」 | +| 向量存储用 **pgvector** 的 HNSW 余弦索引 | 是技术选型,不是成效 | 「向量和调用图放在同一个 Postgres 里,用 pgvector(PostgreSQL 的向量扩展)做近邻查询,省掉一个独立向量库。」 | + +### 两条必须由我先说的提醒 + +**关于 47% → 0% 这条。** 提交记录自己披露了一个利益冲突。这个改动是在查「L3 差 0.0036」的原因时发现的,而且没有任何标准答案文件在 `tests/` 目录下。所以过滤测试代码这件事,同时也机械地帮了指标。这句话必须由我在面试里主动先说。被对方从 commit 里读到再问出来,性质就变了。 + +**关于符号规则统一的三组计数。** 那九次是单臂运行,不携带主假设判定,`results/README.md` 明确标了「not a verdict」。实验报告自己写了「计数是证据,均值是弱的」。所以我只引用三组计数,不引用 groundedness 均值。均值的比值在 n=3 下约等于 p≈0.25 和 p≈0.1,报告明说不作为统计显著性。 + +--- + +## 给 mentor 的说明 + +### 1. 这个项目故意没有吞吐 / 延迟 / token 成本 / 准确率提升数字 + +请不要建议我补这类数字。补了就是编的,而这个项目的全部卖点就是不编。逐条说明: + +| 缺什么 | 为什么没有 | +|---|---| +| **吞吐 / QPS / 延迟** | **从未做过性能压测。**全部跑在单机上,推理 sidecar 用 CPU。产物里没有任何计时字段,代码里也没有任何 latency 埋点。任何数字都只能是现编的 | +| **token 成本** | **从未做过成本核算。**代码中未找到任何 token 计数逻辑,预算维度只有「步数」一个 | +| **准确率提升 / 效果提升 %** | 评测是单仓库 33 题的离线评测,且主假设判定为 `unsupported`。写「提升 X%」要么是挑有利子集,要么是拿诊断项当结论,两者都会在追问第二层崩掉 | +| **答案质量指标** | **judge 未接**,`pairwise_win_rate` 在产物里全是 `null`。这套评测能说「引用是不是真的存在」,说不了「答得好不好」 | + +### 2. 可以替代「量化效果」的三类证据 + +| 类型 | 例子 | +|---|---| +| **机制性保证** | 未通过索引验证的引用会被从答案中移除;分数按移除之前算,所以移除不能洗白;零引用计 0 分;被打断的会话是独立状态,引用一律不绑定 | +| **方法论强度** | 判定标准预注册;五级 baseline 且四个 agent 臂共用同一模型、prompt、协议、守卫;自建对照实验臂做因果拆解,并把它排除在判据之外;跑三次重复加方差分析;协议改动方向对自己不利仍然照发结论 | +| **工程规模** | 7 个 workspace 包、6+2 个容器服务、4+1 张表、11 个工具、5 节点状态机、291 个 pytest 用例、73 个 vitest 用例、Python 源码 10,952 行加测试 7,678 行、前端 4,746 行加测试 1,445 行。测试行数接近源码三分之一,而且多数钉的是引用验证这类规则,不是渲染 | + +### 3. 六条禁止说法,润色时请不要改回去 + +这六条每一条都会在技术面试的第一或第二层追问里崩掉: + +| # | 不能写 | 正确说法 | +|---|---|---| +| 1 | 「调用图 473 条边」 | 473 是**四种边之和**(calls / references / imports / inherits),**调用边只有 303** | +| 2 | 把 groundedness 1.0 笼统写成成绩 | 纯检索基线的 1.0 是代码里的**字面量**,agent 臂的才是实测。而且只有配上「无引用答案数为 0」才有意义 | +| 3 | 「多步取证是调用图的六倍」 | **只在架构级成立**(+0.1466 对 +0.0226)。**跨文件级两者相当**(+0.0221 对 +0.0218) | +| 4 | 把 recall 写成检索能力 | v2 口径下打分对象是答案里已验证的引用。三个 hybrid 臂的检索侧 recall **完全相同,都是 0.560** | +| 5 | 「支持 Postgres 全文检索」 | `chunks.tsv` 与 GIN 索引存在但**全仓无写入点**,稀疏检索是**应用层手写 Okapi BM25** | +| 6 | 「多语言 / 跨语言代码理解」 | 解析器是 stdlib `ast`,**只扫 `*.py`** | + +### 4. H1 判定为什么在两处写法不同 + +简历 bullet 卖的是「可证伪的评测设计」。面试口径写完整结论,包括 `unsupported` 和跨文件层测不出来。 + +理由是两者用途不同。简历要在 30 秒内让人决定要不要聊,它是筛选材料。面试口径要在追问三层之后还站得住,它是承压材料。两者信息一致,密度不同。bullet 说的是「我建了一套能证伪自己的评测」,这是真的。面试口径补上「而它确实证伪了我」,这也是真的,而且是加分项。把「我的假设失败了」直接写成 bullet,等于把承压材料放进筛选位。我还没机会解释方法论,就已经被筛掉了。 + +--- + +## ⚠️ 需我本人确认 / 补充的点 + +| # | 待确认 | 说明 | +|---|---|---| +| 1 | **团队分工与我的职责边界** | 仓库 git 历史里有 PR merge 记录,说明可能不止一个人。简历必须写清哪些部分我主导、哪些是协作。**这一项不确认不能投** | +| 2 | **时间跨度** | 起止月份。git 首个 commit 日期可以参考,但不等于投入时间 | +| 3 | **是否公开仓库、是否放链接** | 要放的话注意仓库里有 `.env.example`、评测产物和内部文档。建议先过一遍有没有不宜公开的内容 | +| 4 | **是否放线上地址** | 目前是本地 docker compose 部署。除了一份生产 compose overlay,没有看到线上实例。没有线上地址的话,简历不要暗示有 | +| 5 | **D 级数字要不要上简历** | 台账里标 **D** 的数字(303 条调用边、316 这个重算值、第二个语料的索引规模等)依赖运行中的数据库,仓库里核对不到。我倾向:简历正文不用 D 级数字,面试口径可以用,但要主动说明来源等级 | +| 6 | **英文版的项目名与一句话定位** | `DCODE_RESUME_ASSETS_EN.md` 里的措辞要你确认,看是否和简历抬头、LinkedIn 一致 | + +--- + +## 附录 A:本文档新增数字的台账 + +来源等级沿用底稿。**A** 是已提交产物字段,**B** 是已提交源码常量。**C** 是本轮重算。**D** 依赖运行中的库或本机 `.env`,仓库里核对不到。 + +| 数值 | 含义 | 证据位置 | 等级 | **必须同时说的那句话** | +|---|---|---|---|---| +| **5** | LangGraph 节点数 | `apps/agent/src/dcode_agent/graph.py:774-778` | **B** | 「`plan ⇄ tool_call` 是 ReAct 循环,两者都能有条件跳到 `synthesize`」 | +| **7** | SSE 事件类型闭集大小 | `packages/shared/src/dcode_shared/events.py:13-21` | **B** | 「是闭集,用 `Literal` 类型钉住,载荷都是 Pydantic 模型」 | +| **4** | 共用同一映射表的 agent mode 数 | `apps/agent/src/dcode_agent/state.py:32-44` | **B** | 「四个 agent 臂共用同一模型、prompt、协议、守卫,差异只来自这张表,这是对照实验能成立的前提」 | +| **6 / 4 / 5** | 索引的外部可见状态数 / `PipelineStage` 条目数 / stage 函数数 | `packages/shared/src/dcode_shared/schemas.py:18-27`;`apps/worker/src/dcode_worker/pipeline.py:51-56` | **B** | 「6 是 `queued→cloning→parsing→embedding→graphing→ready` 这条正常路径,另有 `failed`。`parsing` 一个状态下跑 parse 和 chunk 两个函数」 | +| **2 jobs** | CI 的 job 数与步骤 | `.github/workflows/ci.yml:10,31` | **B** | 「**CI 不跑评测产物一致性校验**,那一步只在 `make lint` 里(`Makefile:70`)」 | +| **180s** | git clone 超时 | `apps/worker/src/dcode_worker/stages/clone.py:11` | **B** | 「超时后杀掉子进程并转 `failed`,不会挂死 worker」 | +| **1MB / 20k** | 单文件跳过阈值 / 单 chunk 内容截断 | `apps/worker/src/dcode_worker/settings.py:12-13` | **B** | 「两者都可以设 0 关闭。跳过的文件会以 warning 形式回传前端,不是静默丢弃」 | + +## 附录 B:从底稿沿用但改写过的表述 + +| # | 底稿表述 | 本文改法 | 理由 | +|---|---|---|---| +| 1 | 「六态状态机」 | 「6 个外部可见状态 / 4 个 `PipelineStage` / 5 个 stage 函数」 | 底稿的「六态」对状态枚举是准确的,但读者容易理解成「6 个 stage 函数」。实际是 4 个条目驱动 5 个函数,`parsing` 跑两个 | +| 2 | 「`make lint` 会校验评测产物一致性」 | 补一句「CI 不跑这一步」 | 读 `ci.yml` 发现 Python job 只有 ruff、mypy、pytest、eval smoke 四步,没有 `sync_eval_artifacts.py --check`。不区分会让人以为 CI 强制了这条 | +| 3 | 「多步取证是调用图的六倍」 | 全文凡出现必带**架构级**限定,紧跟「跨文件级两者相当,比值 1.01×」 | 该结论只在一层成立。简历 bullet 里完全不用这个比值,只在面试口径列两层四个数 | +| 4 | 调用图作为项目卖点靠前出现 | 顺序改成先讲引用验证,再讲怎么控住 agent,最后讲评测。调用图放在「评估」和「面试口径」第 8 条 | 它是被自己的对照实验测小的那一环,放开头等于用最弱的环节开局 | +| 5 | 底稿把 473 与 303 分列在台账两行 | 正文只在面试口径出现 303,并标注它是库里的值。简历 bullet 不用边数 | 边数无论怎么写都需要一句限定,不适合放进要在 30 秒内被读懂的 bullet | +| 6 | 底稿用 `B1/B2/B3/B3.5/B4` 代号叙述 | 给 mentor 的段落改用「纯检索基线 / 稠密 RAG / hybrid / 无图对照臂 / 全系统」 | mentor 不读底稿,代号会造成阅读障碍。技术段落保留代号,方便和底稿对照 | +| 7 | 底稿称 groundedness 1.000 为「实测」 | 本文写明「纯检索基线的 1.0 是字面量,agent 臂的才是实测,而且要配 `answers_without_citations = 0` 才有意义」 | 这是六条禁止说法之一,必须在简历素材里写死,免得润色时被改成成绩 | diff --git a/docs/interview/DCODE_RESUME_ASSETS_EN.md b/docs/interview/DCODE_RESUME_ASSETS_EN.md new file mode 100644 index 0000000..677195b --- /dev/null +++ b/docs/interview/DCODE_RESUME_ASSETS_EN.md @@ -0,0 +1,641 @@ +# Dcode — a code-understanding system that checks its own answers + +## If you only read five sentences + +1. This system is for developers who need to get up to speed on someone else's Python repository. +2. You give it a Git address, it splits the code into chunks and stores them, and then you can ask questions in plain language. +3. I built the whole path: the indexing pipeline, the retrieval, the agent that answers questions, and the frontend. +4. I also built an offline evaluation: the pass criteria were written down before any run, and I added a controlled experiment to separate which part actually helped. +5. The result was that my main hypothesis did not pass, and the feature I originally built the project around helped less than I expected. + +--- + +## About this document + +This is the third file under `docs/interview/`. It's written for the mentor who will edit my resume. + +| File | Purpose | +|---|---| +| `DCODE_TECH_BRIEF.md` | Full technical brief. §4 is a ledger of quotable numbers, each tagged with a source level A/B/C/D and the sentence that has to be said alongside it. §4.9 lists numbers that must not be quoted. Appendix A has 25 documentation-drift entries | +| `DCODE_AGENT_ROLE_PREP.md` | Interview script, organised around what a hiring side looks for | +| **This file** | Resume material | + +Every number in the body comes from the brief's §4 ledger. Where the ledger attaches a qualifier to a number, this file either carries that qualifier or uses a phrasing that doesn't need one. Numbers new to this file are in **Appendix A**, tagged the same way. Where the brief and the README disagree, the code wins. + +### Project type: backend infrastructure plus an LLM application, led by the agent side + +Both halves are really there. Neither is padding: + +| Backend infrastructure | LLM / agent | +|---|---| +| A RabbitMQ consumer driving a strictly monotonic six-state indexing machine | A five-node LangGraph agent state machine | +| PostgreSQL + pgvector, four runtime tables, HNSW and reverse-edge indexes | 11 tools with Pydantic schemas for arguments, dispatched through a registry | +| Four Redis key spaces, each with its own TTL | The server issues citation IDs, and citations that fail verification get deleted | +| SSE gateway: trim history, compute a cache key, replay byte for byte on a hit | Five-tier offline baseline ladder, pre-registered criteria, plus a control arm | +| Idempotent submission, SSRF allowlist, internal service key, Alembic migrations | Hybrid retrieval: BM25 + pgvector + weighted RRF + cross-encoder rerank | + +The agent side leads. The backend half exists so that every step the agent takes can be reproduced, cached and checked. + +There is no training here, no loss, no RL, no fine-tuning. Every model is an off-the-shelf call. The static call-graph resolution is non-trivial, but it belongs to static program analysis, not machine learning. I keep that line visible throughout. + +--- + +## Project description + +Dcode helps a developer read an unfamiliar Python repository. + +You submit a Git address. A worker clones the repository, splits it at function, method and class boundaries, computes embeddings, and extracts a symbol table plus call relationships. After that you ask a question, and an agent calls tools to gather evidence before writing an answer. + +**Before the system emits any citation, it looks that citation up in the database.** A citation it cannot find gets deleted from the answer text and replaced with a placeholder. The groundedness score (what share of an answer's citations can be found in the index) is computed on the ratio **before** deletion, so deleting cannot raise the score. + +The frontend follows the same rule. If a stream drops and no final answer ever arrives, the UI marks that turn as a fourth state. The draft is demoted and its citations are not clickable. "Never checked" and "checked and failed" are two different claims. + +Around all of this I built an offline evaluation that can falsify me. It has a five-tier baseline ladder and pass criteria fixed before any run. A separate control experiment isolates where the gain came from. + +--- + +## Architecture + +Two diagrams. The first shows how one question travels through the system. The second shows which services exist and where data lives. + +### 1. Agent control flow on the query path + +```mermaid +flowchart TD + U["User question + client-side history"] --> GW["API Gateway
POST /api/v1/query"] + GW --> HB["Bound history
6 turns / 2000 chars total / 4000 per turn"] + HB --> CK["Build cache key
query : repo_id : sha256 of history + query"] + CK --> HIT{"Redis hit?"} + HIT -->|"yes"| RP["Replay the buffered SSE stream byte for byte"] + HIT -->|"no"| PX["Proxy to agent, internal key header"] + + PX --> CTX["contextualize
rewrite follow-up into a standalone query"] + CTX --> PLAN["plan
rule-based tool selection"] + PLAN -->|"tool selected"| TC["tool_call
validate args, cache lookup, execute"] + TC -->|"no error"| PLAN + PLAN -->|"no tool left / step cap 14 / error"| SYN["synthesize
stream LLM answer over ranked evidence"] + TC -->|"error, degrade"| SYN + SYN --> GC["groundedness_check"] + + GC --> VER["Verify every citation against the live index"] + VER --> RED["Redact unverified references from the answer text"] + RED --> SCORE["Score = verified ratio BEFORE redaction"] + + SCORE --> EV["SSE out: thought / tool_call / tool_result / partial_answer,
then citation events, then exactly one final_answer"] + EV --> FE["SPA thread"] + RP --> FE + + PX -.->|"agent unreachable"| ERR["error event only, stream terminates honestly"] + ERR --> FE +``` + +### 2. Services and data layer + +```mermaid +graph TD + SPA["SPA - React 18 + Vite, served by nginx"] + API["API Gateway - FastAPI"] + AG["Agent - FastAPI + LangGraph"] + W["Index Worker - RabbitMQ consumer, prefetch 1"] + EMB["Embedding sidecar - Jina v2-base-code on CPU"] + RR["Reranker sidecar - BGE reranker v2-m3 on CPU"] + + PG[("PostgreSQL + pgvector
repos / chunks / symbols / edges")] + RD[("Redis
embed: / tool: / query: / job:")] + MQ["RabbitMQ - queue dcode.index_jobs"] + VOL["Shared volume repo_workdirs"] + + SPA -->|"HTTP + SSE"| API + API -->|"AMQP publish, persistent"| MQ + MQ -->|"consume"| W + API -->|"HTTP SSE proxy"| AG + AG -->|"HTTP internal retrieval + graph API"| API + + W -->|"POST /embed"| EMB + API -->|"POST /rerank"| RR + AG -->|"POST /rerank"| RR + + W --> PG + API --> PG + AG --> PG + W --> RD + API --> RD + AG --> RD + W -->|"git clone writes"| VOL + AG -->|"read_file / grep / list_directory"| VOL + + W --> P1["clone"] + P1 --> P2["parse"] + P2 --> P3["chunk"] + P3 --> P4["embed"] + P4 --> P5["graph"] + + TSV["chunks.tsv + GIN index"] -.->|"NOT WIRED - no writer anywhere in the codebase"| PG + IR["index_runs table + append-only trigger"] -.->|"NOT WIRED - no writer anywhere in the codebase"| PG +``` + +The second diagram has two dashed edges. `chunks.tsv` and its GIN index exist in the database, but no code anywhere in the repository writes to them (sparse retrieval runs on hand-written BM25 in the application layer). The `index_runs` table even has an append-only trigger, and it likewise has no writer. I drew them separately so nobody reads them as part of the main data flow. + +--- + +## Where it sits in the AI stack + +This section says what goes in, which models get called, and what comes out. + +| Stage | Detail | +|---|---| +| **Input** | The offline indexing side takes a Git address. The online query side takes one plain-language question plus the prior turns the client sends along | +| **Models called** | `jinaai/jina-embeddings-v2-base-code` computes embeddings, running as a self-hosted HTTP sidecar on CPU. `BAAI/bge-reranker-v2-m3` does reranking, also a self-hosted sidecar. `gpt-4o-mini` writes the answer over OpenAI's streaming API. There is an abstract class for a judge, but **no model is wired behind it** | +| **Tools called** | 11 of them: semantic search, find definition, find references, call neighbours, call paths, forward and reverse module dependencies, file outline, read file, ripgrep, list directory | +| **Output** | One SSE stream: the reasoning trace first, then the streamed answer, then citation events carrying **only verified references**, then one final answer with a groundedness score. The frontend renders clickable citation chips and a source inspector from this | +| **Not produced** | No training, no fine-tuning, no model weights, no preference data | + +Zoom out and this is evaluation and verification plumbing for agents. It does not produce a policy. It produces the ability to judge whether a policy is any good. Any team improving an agent with RL or preference optimisation has to answer two questions first. Where does the reward signal come from? How do you confirm the gain is real? This project is the engineering side of those two questions. + +--- + +## Agent architecture + +### a. Orchestration + +The agent is a five-node LangGraph state machine, and a hand-written rule table picks the tools. + +The nodes are defined at `apps/agent/src/dcode_agent/graph.py:774-778`: + +``` +START → contextualize → plan ⇄ tool_call → synthesize → groundedness_check → END +``` + +`plan → tool_call → plan` is the ReAct loop. Both `plan` and `tool_call` can branch conditionally into `synthesize` (`graph.py:782-791`). Requests are fully stateless. No LangGraph checkpointer is installed, and the client sends history on every call (`graph.py:770-771`). + +**The planner is a hand-written rule table, not an LLM** (`graph.py:798-812`). That choice needs three layers to hold up. + +**Layer one: why I chose it.** The central measurement in this project is a controlled experiment. Arm B3.5 is arm B4 with the call graph switched off and everything else unchanged. For that difference to mean anything, the two arms must differ only in their tool set. Answer generation is already stochastic enough on its own: across three identical runs, one level's margin swung by more than the pass threshold itself (see Interview positions, item 2). If the planner also ran on an LLM, the difference would additionally contain "which tool the model happened to pick this time", and the causal attribution would not hold. A rule table confines the gap between arms to one lookup table (`state.py:32-44`), and that table can be exhaustively unit-tested. + +**Layer two: the cost is a real product defect.** Routing runs off keyword tables (`graph.py:1087-1112` and `:1312-1358`). An architecture question containing none of those keywords stops after the first retrieval. Worse, an outside observer cannot tell two cases apart: the planner deciding there is no next step, and the planner failing to understand the question. In the SSE stream they look identical. + +**Layer three: what I would change to ship this.** Move the main path to LLM function calling. The JSON Schema for all 11 tools already exports through `ToolRegistry.manifest()` (`tools/base.py:58-67`), so the interface work is small. The rule table then retreats to two jobs: catch the case where the model's arguments fail validation, and hold a few safety limits. The step cap, the arm-level tool disabling, and the rule against repeating a call with identical arguments stay out of the model's hands. + +After that switch, three things have to be added back before the controlled experiment works again. Configure the planner's temperature separately from synthesis. Run an A/A experiment that varies only the planner's seed, to size the planner's own variance. And record the tool-call sequence into the artifacts. + +### b. LLM integration + +The system calls one OpenAI endpoint to write answers, and forces the model to cite only IDs the server handed it. + +| Item | Detail | Location | +|---|---|---| +| Provider | OpenAI Chat Completions, `openai` 2.49.0, with an overridable `base_url` for compatible endpoints | `apps/agent/src/dcode_agent/llm.py:129-154` | +| Model | `gpt-4o-mini` writes the answer. Set to `stub` and it falls back to a rule-based template | `llm.py:211-236` | +| Streaming | `stream=True`, and each delta becomes one `partial_answer` event | `llm.py:156-180`; `graph.py:266-274` | +| Citation protocol | Five mandatory rules in the system prompt. Cite only `[C#]` IDs from the server's catalog. Keep IDs outside backticks. Never substitute a filename, line number or symbol for an ID. Attach an ID to each concrete claim. Say so plainly when the evidence is not enough | `llm.py:43-53` | +| Call-graph rules | Three more. Resolved static edges and call expressions visible only in source must be described separately, and an unresolved target must never be called resolved | `llm.py:54-60` | +| Language | The answer's language follows the **original** question. Rewriting a follow-up does not change it | `llm.py:75-86`; `graph.py:269` | +| Follow-up rewriting | A separate call at `temperature=0` and `max_tokens=128` | `llm.py:182-208` | + +### c. Tool layer + +There are 11 tools. Every one declares its arguments as a Pydantic schema, and validation runs before execution. + +The registry lives at `apps/agent/src/dcode_agent/tools/__init__.py:22-39`. + +| Category | Tools | Exit | +|---|---|---| +| Retrieval | `search_code` | HTTP to the gateway's internal retrieval route | +| Symbols and graph | `find_definition` · `find_references` · `find_call_path` · `get_call_neighbors` · `get_dependencies` · `get_dependents` | HTTP to the gateway's internal graph routes | +| File structure | `get_file_outline` | HTTP | +| Local | `read_file` · `grep` · `list_directory` | Shared volume or a ripgrep subprocess | + +**One abstraction.** Each tool is a `Tool[ArgsT, ResultT]` exposing five things: `name`, `description`, `ArgsSchema`, `execute` and `cache_key` (`tools/base.py:25-38`). Tools never connect to the database (`base.py:10-12`). They go over HTTP or to the local filesystem. + +**Validation runs before execution.** `tool_call_node` calls `tool.ArgsSchema(**args)` first (`graph.py:113`). A validation failure degrades instead of reaching the user (`:114-117`). `read_file` adds a `model_validator` that rejects a start line below 1 or a start line past the end line (`tools/read_file.py:17-24`). + +**Path traversal is blocked at two layers.** The first layer works on the string: absolute paths are rejected (`tools/common.py:55`), and so is any `..` that escapes the repository root (`:63`). The second layer runs after resolution and checks `is_relative_to(root)` (`:74-75`). Each of the three tools that touches the filesystem has its own traversal test (`test_tools_execute.py:216`, `:237`, `:261`). + +**The cache-key format implies a constraint that is easy to miss.** A tool cache key is `tool:{tool}:{repo_id}:{sha256(canonical_json(args))[:16]}` (`packages/shared/src/dcode_shared/cache.py:19-21`). Since the key comes from the arguments, anything that changes a tool's behaviour has to live in the arguments. That is why `search_code` takes its retrieval mode as a tool argument instead of reading it from the environment. Put the mode in arm-level ambient state and two experimental arms that differ only in retrieval mode collide on one cache entry, which silently contaminates the evaluation. The reasoning sits at `tools/search_code.py:8-11`. + +### d. Context and memory + +Evidence gets its source code filled in first, then one model ranks all of it together. + +**Fill in the source first.** Retrieval results arrive with source code attached. Graph results arrive as bare `file:line`. Send both into the prompt as they are, and the scoring model is no longer comparing relevance. It is comparing which side has something to read. That is measurement bias. So the code batches the graph results by `chunk_id` and fetches their source (`graph.py:498-529`, server side `GET /internal/get_chunks`), and only then does one cross-encoder (a model that reads the question and the code together as one text and scores the pair, more accurate than encoding them separately but slower) score every candidate (`graph.py:437-490`). + +**Ranking features deliberately exclude evidence origin and graph distance** (`graph.py:451-454`). If you are testing whether the call graph helps, you cannot assume it helps inside the scoring function. A "prefer structural evidence" prior tuned on the evaluation set is a hyper-parameter picked to make the hypothesis pass. I take the cost: four of the six graph tools never produced a single cited piece of evidence across the whole evaluation. + +**The context budget is ten items, identical for every arm** (`graph.py:432`). That number constrains the strongest arm. Graph evidence has to displace retrieval evidence to earn a slot. The weakest arm never has more than five retrieved chunks, so a bigger budget would do nothing for it. A bigger budget for the strongest arm would turn context length into a confound. + +**Over-budget evidence loses its text but keeps its ID** (`graph.py:486-488`). It stays citable and verifiable, it just stops occupying context. That way a structural explanation never references an ID that does not exist. + +**The retrieval stack** (`apps/api/src/dcode_api/routes/internal.py:241-299`): + +``` +sparse: hand-written Okapi BM25, k1=1.2, b=0.75 + the parameters are written into the run artifact, so this baseline reproduces +dense : pgvector HNSW cosine + ↓ 50 candidates from each side +fuse : weighted RRF, K=60, dense:sparse = 2:1 + ↓ +rerank: cross-encoder over the top 16 + ↓ +take : keep 5, and filter test files after ranking (candidate pool and fusion math untouched) +``` + +Three terms, briefly. **BM25** is the classic keyword scoring algorithm, weighing term frequency against document length. **pgvector** is PostgreSQL's vector extension, which runs nearest-neighbour queries inside the database. **RRF** (Reciprocal Rank Fusion) merges two ranked lists by position rather than by score. + +I fuse with RRF instead of normalising scores and adding them. BM25 scores are unbounded and depend on the corpus's IDF and length distribution, while cosine similarity is bounded and tightly clustered. Rank is the only quantity you can compare across those two rankers without assuming a distribution. The non-equal weighting has an empirical reason, and a test pins it from each direction (`test_internal_routes.py:444` and `:465`). + +**The client sends multi-turn history, and the gateway trims it once**: at most 6 turns, 2000 characters total, 4000 per turn (`apps/api/src/dcode_api/settings.py:15-17`). Trimming happens in one place, so the planner, the body proxied to the agent, and the cache key all see the same turns. Only turns that received a final answer are fed back as context (`useThread.ts:43-55`), so a draft that never went through redaction can never re-enter as history. + +**There is no server-side long-term memory**, and that is on purpose. With server-side session state the cache key would stop determining the output, and the evaluation would stop being reproducible. + +### e. Reliability and control + +The agent runs at most 14 steps, and a failed step counts. When a tool fails it still answers. When infrastructure fails it returns an error instead. + +**Step cap 14, failures included** (`apps/agent/src/dcode_agent/settings.py:14`, incremented at `graph.py:168` and `:189`). I raised it from 8 after noticing that the strongest arm kept terminating at the budget rather than at the planner's decision. In that situation you are measuring your budget, not your policy. Counting failed steps is what makes the loop's upper bound real. + +**Seven early-exit branches besides the cap**: the planner returns nothing, `state.error` is set (two places), the cap is reached (two places), no tool is pending, and an error on the tool-call edge. One of them is unreachable under the current graph topology, and the brief's drift list already flags it as defensive. + +**Failures split into two layers, and the split lives in one function, `_record_tool_failure`** (`graph.py:175-209`): + +| Layer | Behaviour | What the user sees | +|---|---|---| +| Tool: bad arguments, execution failure, cache decode failure | Degrade, go write the answer | Still gets an answer, with a note naming the tool that failed | +| Synthesis: LLM stream failure | Degrade, fall back to the rule-based template | Still gets an answer | +| Ranking: reranker failure, source-fetch failure | Degrade, fall back to observation order or leave that item blank | Nothing visible | +| Infrastructure: missing registry, database error during verification | Terminate, emit an `error` event | An explicit failure, and no answer | + +**Retry counts differ per layer, each for its own reason:** + +| Layer | Retries | Reason | +|---|---|---| +| Tool / internal HTTP | None | Tool failures are usually argument or index problems, and retrying does not fix those. Degrading to "answer from what we already have" is faster and more honest | +| Embedding | 12, exponential backoff capped at 30s. 4xx raises immediately | Indexing is offline and can wait. A model sidecar's cold start takes minutes | +| Reranker | 3, and `ReadTimeout` is deliberately left off the retry list | Querying is online and cannot wait. Retrying slow CPU inference just piles duplicate work onto a single-threaded model server | +| Gateway to agent | None, explicitly | A streaming response is not idempotent. A retry re-sends the half-answer already on the wire | + +### f. Hallucination control and attribution + +The model may only cite IDs the server gave it, citations it cannot verify get deleted, and the score is computed before deletion. + +**First, the server issues the citation IDs.** The model may cite only the `[C#]` IDs sent for this request. An ID absent from the catalog fails outright (`groundedness.py:184-194`). + +**Second, ordinary backticked code is not a citation** (`groundedness.py:150-151`). Writing `self.client.retrieve` is formatting, not an evidence claim. An earlier version treated every dotted inline token as a citation, and the model ended up "citing" something every time it named a method. + +**Third, verify then delete, and score before deleting.** References that fail verification are removed from the text (`groundedness.py:407-426`), and only verified ones become citation events (`graph.py:727-738`). The score, though, uses the ratio **before** deletion (`groundedness.py:403`). This step carries the whole mechanism. Score after deletion and every answer is perfect, because deletion becomes laundering. + +**Fourth, an answer with no citations scores 0 rather than full marks** (`groundedness.py:272`), and a separate count distinguishes "cited nothing" from "every citation failed". Without that, an agent that learns to say nothing scores full marks. I also rejected the option of dropping such answers from the denominator, since "don't cite where you're unsure" would raise the average the same way. + +$$ +g(a)=\begin{cases}\dfrac{1}{n}\displaystyle\sum_{i=1}^{n} v(r_i), & n > 0\\[2ex] 0, & n=0 \quad \text{(a convention, not a derivation)}\end{cases} +$$ + +**One detail is easy to miss and cannot be got wrong.** Symbol verification narrows candidates with SQL `LIKE '%.name'`. Underscores are everywhere in Python identifiers, and an underscore is a single-character wildcard in `LIKE`. Leave it unescaped and `api._get` also matches `api.aget`. That is a silent false positive inside a guard, and it is the one error this system cannot afford (`packages/shared/src/dcode_shared/symbols.py:56-64`). The same file records a related fix. Symbol resolution used to exist in two copies, one in the tools and one in the guard. Inside a single request the tool would say "here it is" while the guard said "that does not exist, strip it". There is one copy now. + +### g. Evaluation and observability + +Five baselines, pass criteria fixed before any run, and one control arm that separates where the gain came from. + +Four agent arms share the same generation model, the same prompt, the same citation protocol and the same guard. Their only difference comes from one lookup table (`apps/agent/src/dcode_agent/state.py:32-44`). That is the precondition for the controlled experiment to mean anything. + +| Arm | Retrieval | Answer path | In the criterion | +|---|---|---|---| +| B0 | External GitHub code search | Template | No (not reproducible) | +| B1 | Application-layer BM25 | Template | No (retrieval reference) | +| B2 | Dense only | Shared agent generation, same guard | Yes | +| B3 | Hybrid | Same, no tool expansion | Yes | +| **B3.5** | Hybrid | Same, keeps file reading and outlines, drops graph tools | **No (control arm)** | +| B4 | Hybrid | Same, all tools | Yes | + +**The pass criterion is a pre-registered conjunction**: a 0.05 threshold, two reasoning levels times two rivals, and all four comparisons must clear (`apps/eval/src/dcode_eval/run.py:545-560`). Pre-registered means the criteria were written into the code and committed before the run, and could not be changed afterwards. I used a conjunction rather than an average, because averaging lets a big win at one level hide a failure at the other. + +Scoring uses a composite, which is the mean of three retrieval metrics. Those three are recall (what share of the answers you should have found came back), MRR (how high the first correct result ranked), and nDCG (rewards correct results ranking higher, and unlike MRR it also counts the later hits). + +**Arm B3.5 splits the system's gain into two parts.** An ablation means switching off one component, leaving everything else unchanged, and seeing how far the metric falls: + +$$ +\underbrace{B4 - B3}_{\text{the whole agent system}} = \underbrace{(B4 - B3.5)}_{\text{the call graph itself}} + \underbrace{(B3.5 - B3)}_{\text{multi-step evidence gathering}} +$$ + +**I put a constraint on myself here.** B3.5 is excluded from the criterion in code (`run.py:599-601`, `:605-616`, pinned by `apps/eval/tests/test_run.py:686`). Adding a new arm to the criterion would be changing the pass rules after the fact. So the most valuable finding in this project can only appear as a diagnostic. It cannot move the verdict. + +**What is observable.** The SSE (Server-Sent Events, a one-way push from server to browser, well suited to streaming output) trace carries three event types: `thought`, `tool_call` and `tool_result`. The frontend expands them to show each call's arguments and a summary of its result. A groundedness pill shows a score only once a final answer arrives, and stays neutral while streaming. The indexing side emits structured per-stage logs and a live Redis snapshot. Skipped files and failure reasons both reach the frontend. + +One thing has to be stated plainly: **the judge is not wired, so answer quality is completely unmeasured.** The `Judge` abstract class exists with its four-axis rubric and pairwise interface (`apps/eval/src/dcode_eval/metrics/judge.py:27-36`), but no real model sits behind it, and `pairwise_win_rate` is `null` throughout the artifacts. This evaluation can tell you whether the right code was found and whether a citation genuinely exists. It cannot tell you whether the explanation is correct or clear. + +### h. Engineering + +The SPA talks only to the gateway, and the four application packages import zero Python from each other. + +**A single API seam.** The SPA never connects directly to the agent or the database. The agent, in turn, calls the gateway's internal retrieval routes over HTTP. The repository is 7 uv workspace packages (`pyproject.toml:8-16`), and the four application packages share no Python imports. They communicate over HTTP, AMQP, SQL and Redis only. + +The evaluation harness is the cleanest case. It depends on `httpx` and the shared schema package and nothing else (`apps/eval/pyproject.toml:6-9`), so it cannot import the system it measures. It can only reach that system the way an external client would. That is what stops the evaluation from cheating through shared in-process state. + +**Deployment and checks:** + +| Item | Detail | +|---|---| +| Containers | `docker compose` brings up 6 long-running services plus 2 model sidecars behind opt-in profiles. Every service has a health check | +| Local checks | `make check` = lint + typecheck + test (`Makefile:80`). `make lint` runs one extra step that verifies evaluation-artifact consistency (`Makefile:70`). If a figure shown in the UI or the docs drifts from `results/`, lint fails | +| CI | `.github/workflows/ci.yml`, two jobs. The Python one runs ruff, mypy --strict, pytest and eval smoke. The frontend one runs eslint, tsc, vitest and build | +| Types | `mypy --strict` across all Python packages. TS strict on the frontend, plus a compile-time test that pins the hand-mirrored frontend/backend schema | + +One distinction to state accurately: **the evaluation-artifact consistency check runs only in `make lint`, and CI does not run it.** The Python job in `ci.yml` has four steps: ruff, mypy, pytest, eval smoke. + +--- + +## Backend infrastructure + +This section covers the parts with no model in them: the indexing pipeline, the queue, the database, the caches. + +### Indexing: a strictly monotonic six-state machine with dual writes + +Six states are visible from outside: `queued → cloning → parsing → embedding → graphing → ready`, and `failed` is reachable from any of them (`packages/shared/src/dcode_shared/schemas.py:18-27`). The implementation is 4 `PipelineStage` entries driving 5 stage functions: `clone`, `parse`, `chunk`, `embed`, `graph` (`pipeline.py:51-56`). The `parsing` state runs two of those functions. + +Monotonicity comes from the data structure. The stage order is an immutable tuple with no jump table and no conditional branches. + +**Dual writes.** The Postgres `repos` row is the durable truth. The Redis `job:{repo_id}` key is the fine-grained snapshot the frontend polls, holding per-stage state, progress and a warning list. A failed Redis write is swallowed and never affects indexing. + +**One piece of atomicity matters.** Deleting the old chunks and incrementing `index_revision` happen inside the same transaction (`stages/embed.py:173-179`), because the sparse-retrieval corpus cache is keyed on `(repo_id, index_revision)` and the two have to move together. + +### Queue and idempotency + +`POST /repos` persists and commits the row first, then publishes a persistent AMQP message (`apps/api/src/dcode_api/deps.py:44-58`). The order cannot be swapped, or the worker reads no row. + +**A publish failure has a compensating path.** The row is already durable at that point, so it gets marked `failed` and the caller receives a 503. No permanently-queued zombie row is left behind. + +On the worker side: `prefetch_count=1`, a durable queue, and acknowledgement only after the handler returns (`apps/worker/src/dcode_worker/main.py:19-30`). `handle_job` is guaranteed never to raise back to RabbitMQ, since a raise means the message gets redelivered forever. + +**URL idempotency.** Six statuses count as reusable, and `failed` is deliberately excluded (`routes/repos.py:97-104`). A previous failure is exactly when a user wants a retry. URL normalisation stays narrow: trailing slash, `.git` suffix, and the case of the scheme and host. Path case is preserved (`:107-123`), because merging two genuinely different repositories is worse than keeping one duplicate row. + +### Data model and indexes + +Four runtime tables: `repos`, `chunks`, `symbols`, `edges` (`packages/shared/src/dcode_shared/db/models.py`). All are scoped by `repo_id`. Four native Postgres ENUMs map one-for-one onto Pydantic `StrEnum`s, pinned by tests. + +Indexes live in the Alembic migration rather than the ORM, because pgvector's operator class has no portable expression through SQLAlchemy. They are: HNSW `vector_cosine_ops` on `chunks.embedding`, `(repo_id, file_path)`, a unique `(repo_id, qualified_name)` on symbols, and both forward and reverse edge indexes, `(repo_id, source_id, edge_type)` and `(repo_id, target_id, edge_type)`. The reverse one exists for "who references this" queries. + +**The vector dimension is bound when the migration runs.** `Vector(embedding_dim)` appears in both the ORM and the migration, and the migration's copy freezes whatever the environment held at `alembic upgrade` time into the DDL. Changing the dimension therefore means rebuilding the database. That is an easy operational trap, and it is documented as one. + +### Four Redis key spaces + +| Key | TTL | Design point | +|---|---|---| +| `embed:{model}:{sha256(text)}` | None | Content-addressed. The same code appearing in two repositories gets embedded once | +| `tool:{tool}:{repo_id}:{args_hash}` | 24h | Includes an argument hash, so anything affecting tool behaviour has to live in the arguments | +| `query:{repo_id}:{sha256(history + query)}` | 1h | Includes conversation history. Without it, a context-dependent follow-up would collide with the same string asked single-turn | +| `job:{repo_id}` | 7 days after completion | No TTL while running, set only on completion | + +Two details. `\x1f` (unit separator) delimits history from question, because that byte cannot appear inside the JSON, so the boundary is unambiguous. With empty history the digest is byte-identical to the old single-turn key, so adding multi-turn support did not orphan existing cached answers, and a test pins that. + +### SSE gateway + +The gateway forwards to the browser and buffers the whole stream at the same time. **It writes back to the cache only when the stream contains no `error` event** (`routes/query.py:72-73`). Caching a failed answer for an hour is worse than not caching. On a cache hit the whole stream replays byte for byte. + +**When the agent is unreachable**, the gateway emits one `thought` event explicitly labelled `(skeleton)` followed by an `error` event (`:96-109`). The SSE protocol then terminates honestly instead of the connection just dropping. + +### Security boundaries + +- **SSRF allowlist.** Repository URLs must be http(s), ssh, git or scp-like. `localhost`, private addresses, loopback, link-local, multicast, reserved and unspecified IPs are all rejected (`routes/repos.py:187-224`). +- **Internal service key.** Every `/internal/*` route checks an `X-Dcode-Internal-Key` header (gateway `routes/internal.py:39-49`, agent `main.py:176-181`). The frontend never reaches that layer. +- **Path traversal.** See the tool layer. +- **Configuration hardening tests.** A test asserts that `.env.example` and the production compose file carry no known weak credentials and require secrets explicitly (`packages/shared/tests/test_config_hardening.py`). + +--- + +## Interview positions + +This section states the full conclusion, including the part that did not pass. It carries the same information as the resume bullets at a different density, and the reason for that split is in "Notes for my mentor", item 4. + +**1. The main hypothesis came back `unsupported`.** The criterion was a pre-registered conjunction, and three of four comparisons cleared. Architecture-level questions cleared against both rivals (+0.247 and +0.169). Cross-file questions cleared against dense RAG (+0.136) and missed hybrid-plus-rerank by 0.0061. I do not retract or soften that. My own executable rule produced it. + +**2. The cross-file result is not measurable at this scale.** Three identical runs produced +0.038, +0.006 and +0.088. The spread of 0.083 is wider than the 0.050 threshold itself. That 0.0061 shortfall sits against a between-repeat standard deviation of 0.034 (population figure; 0.042 as a sample statistic), so the shortfall is about a fifth of the noise. The fix is more questions or a second corpus, not more repeats. Resolving a difference this small at this suite size would take on the order of 100 runs. As a sanity check, the pure-retrieval baseline produced byte-identical metrics across all three repeats, which places the variance in answer generation and not in retrieval. + +**3. The controlled experiment's result needs a per-level qualifier.** The call graph on its own contributes +0.0218 at cross-file level and +0.0226 at architecture level. Multi-step evidence gathering contributes +0.0221 at cross-file level and +0.1466 at architecture level. So "multi-step evidence gathering far exceeds the call graph" holds only at architecture level. At cross-file level the two are comparable, a ratio of 1.01. Both figures are diagnostics and both are excluded from the verdict in code. + +**4. Two protocol changes ran in opposite directions.** Moving the scoring protocol from v1 to v2 put all four agent arms on one rule, and that change turned my own verdict from supported into unsupported. In the same batch I dropped groundedness from the composite. That genuinely lowered the bar. The removed term is 1.000 on every arm, so dropping it is arithmetically identical to moving the threshold from 0.050 to 0.0375. I do not dress that up. + +Three defences are available, and all three are checkable. It was declared and committed before the run it judged. Both readings are published side by side in the artifact. Both readings return unsupported. + +**5. The question set has a confound I only found later.** Taxonomy label and question provenance are collinear in this data. Every single-file question is hand-written, and every reverse-constructed question sits at the two higher levels with roughly twice the ground truth. So "leads at architecture level" and "leads on the reverse-constructed batch" cannot be separated. I measured which way it runs, and it does not favour me. The call graph's isolated contribution is +0.0183 on reverse-constructed questions against +0.0283 on hand-written ones. It is smaller on the reverse-constructed batch. That still does not rule the bias out. The bias takes the form of what is missing from the set, and the set's own metrics cannot detect that. It is now recorded in the run's limitations. + +**6. The recorded run used an older call graph than the current code produces.** Re-running today's worker offline over the same corpus yields 316 call edges, against 303 in the database, a difference of 4.3%. The other three edge types match exactly, which places the difference in call resolution alone. The improvement is committed but needs a re-index to take effect, and re-indexing mid-comparison would move the corpus underneath the earlier runs. + +**7. The judge is not wired, so answer quality is unmeasured.** See "Evaluation and observability". + +**8. The static call graph's precision is quantified.** The resolver visits 2615 `ast.Call` sites and resolves 316 of them. The denominator is the call sites the worker actually walks, excluding module-level statements and methods of nested classes. That rate is reasonable for pure static AST analysis with no type inference. The largest unresolved group is 1076 `obj.X()` calls whose receiver type is unknown, and resolving those needs type inference, which this analysis explicitly declines. Missing an edge is safe; asserting a wrong one is not, so the design refuses to emit an edge it cannot ground. Unresolved calls are not discarded either. They are surfaced to the model explicitly, and a prompt rule forbids presenting them as resolved. + +--- + +## Draft resume bullets + +The impact clause uses two forms only: an A/B-level figure from the ledger with its qualifier, or a mechanistic result. There are no throughput, latency, cost or accuracy-lift figures, and the next section explains why. + +1. **Designed and built** an end-to-end code-understanding system (FastAPI + LangGraph + PostgreSQL/pgvector + RabbitMQ + Redis + React 18) spanning 7 uv workspace packages, 6 long-running services, 2 model sidecars and 364 automated tests. The four application packages share zero Python imports and communicate only over HTTP, AMQP, SQL and Redis, so the evaluation harness cannot import the system it measures. + +2. **Built** a citation-verification mechanism in which the server issues the evidence IDs and the model may cite only IDs sent for that request. **Any citation that fails index verification is removed from the answer text**, and the groundedness score uses the ratio before removal, **so removal cannot raise the score**. An answer citing nothing scores 0 rather than full marks, which closes the "say less, score more" exploit. + +3. **Implemented** bounded agent orchestration: a 5-node LangGraph state machine and 11 tools with Pydantic argument schemas, path traversal blocked at two layers with a traversal test per filesystem tool. **A step cap of 14 in which failures also consume steps**, raised from 8 after the strongest experimental arm kept terminating at the budget rather than at the planner's decision. Seven early-exit branches, plus a layered failure policy where tool-layer failures degrade and infrastructure failures terminate, with retry counts differing per layer. + +4. **Designed** a falsifiable offline evaluation: **a five-tier baseline ladder with a pre-registered conjunctive criterion** where all four comparisons must clear. A frozen 33-question suite with **62 ground-truth chunks**, **hand-labelled by level with no classifier in the code**. Three repeats totalling **495 agent invocations**, separating the stochasticity of generation. Plus a purpose-built **B3.5 control arm** splitting the gain into a call-graph term and a multi-step-evidence term. **That arm is excluded from the criterion in code**, so the rules cannot be changed afterwards. + +5. **Built** a hybrid retrieval path: hand-implemented Okapi BM25 (parameters recorded in the run artifact, so the baseline reproduces) alongside pgvector HNSW cosine, fused by **weighted RRF**, reranked by a cross-encoder, then filtering test files after ranking, **cutting them in the candidate set from 47% to 0**. **Ranking excludes provenance and graph distance**, so scoring cannot presuppose the conclusion under test. + +6. **Built** an asynchronous indexing pipeline: **a strictly monotonic six-state machine** in which 4 pipeline stages drive 5 stage functions. It dual-writes a durable Postgres row and a live Redis snapshot, and increments the corpus revision in the same transaction as the chunk replacement. `POST /repos` reuses an existing repository idempotently by URL while **deliberately not reusing failed rows**, which preserves retry semantics. `make lint` verifies evaluation-artifact consistency and **fails the build if any figure shown in the UI or docs drifts from the recorded run**. + +7. **Traced and unified** the symbol-resolution rule behind citation verification: the guard matched exactly while the retrieval tools matched on suffix, so inside one request a tool reported a symbol found and the guard stripped it as nonexistent. After collapsing both onto one shared rule, a controlled three-arm nine-run comparison showed **verified citations rising from 48–54 to 62–67 and redaction markers falling from 17–21 to 9–13**, with symbol-form citations surviving for the first time, 0 to 2 per run. + +### Seven is too many — pick by channel + +I am not deleting any. You choose. The two channels look for different things. + +**Technical applications, keep 4: 2 · 4 · 7 · 3.** + +| Keep | Why | +|---|---| +| 2 citation verification | A technical interviewer wants to know how I make a model that invents things produce checkable output | +| 4 evaluation design | It shows I can set a standard I might fail | +| 7 symbol rule | The one complete "found a defect, here are the before and after counts" story, closest to daily work | +| 3 agent orchestration | Step caps, layered failure, retry policy — the basics for this role | + +Drop 1 / 5 / 6. They describe standard backend and retrieval engineering, which is the most replaceable. + +**HR screens and online applications, keep 3: 1 · 2 · 5.** + +| Keep | Why | +|---|---| +| 1 system scale | A non-technical reader starts with size: how many packages, services, tests | +| 2 citation verification | A selling point that fits in one sentence and needs no background | +| 5 retrieval path | BM25, vectors and reranking are the role keywords both people and filters search for | + +Drop 3 / 4 / 6 / 7. They ask the reader to follow an evaluation design or a debugging trail, which does not fit in thirty seconds. + +--- + +## Data inventory: what each number proves + +This section gathers every usable number in one place and says what each one means to a reader. There are three groups: the first can go straight onto a resume, the second needs a qualifying sentence, and the third stays verbal in an interview. + +### Group 1 · Safe to put on a resume + +These need no background. A reader takes in the magnitude at a glance. + +| Number | What it is | What it proves | Source | Already in a bullet | +|---|---|---|---|---| +| **726** | Code chunks indexed from one repository | A real repository, not a toy sample | Brief §4.1 (A) | No | +| **724 / 37** | Symbols indexed / Python files indexed | Indexing produces a symbol table as well as chunks | Brief §4.1 (C+D / C) | No | +| **62** | Distinct ground-truth chunks behind the 33 questions | Correctness has an objective anchor, not my judgement | Brief §4.1 (A) | Yes | +| **364** | Automated test cases | Someone maintains this, not a one-off demo | Inventory tier C (C, 291+73) | Yes | +| **24,821** | Total lines of code | Large enough to be real, small enough to read | Inventory tier C (derived) | No | +| **257** | Git commits | Incremental work with a full history | Inventory tier C (C) | No | +| **495** | Agent invocations in one full evaluation round | An evaluation system, not a handful of examples | Inventory tier C (A, derived) | Yes | +| **33 = 5 / 16 / 12** | Questions across three difficulty levels | Questions are tiered, so a weak level shows up | Brief §4.4 (A+B) | Yes | +| **3** | Repeats of the same configuration | I know a single run can mislead | Brief §4.4 (A) | Yes | +| **5** | Tiers in the baseline (comparison arm) ladder | There is a reference frame, not self-comparison | Inventory tier C (B) | Yes | +| **5** | Retrieval cutoff k | Scoring looks at the top five only, and that is fixed | Brief §4.4 (A) | No | +| **7** | uv workspace packages | Boundaries are cut, not one large file | Brief §4.6 (B) | Yes | +| **6 + 2** | Long-running services + model sidecars | It needs orchestration, not a single script | Brief §4.6 (B) | Yes | +| **11** | Agent tools | The agent genuinely calls tools to check things | Brief §4.3 (B) | Yes | +| **5** | LangGraph nodes | The flow is explicit, drawable and testable | Inventory tier C (B) | Yes | +| **768** | Embedding vector dimension | A real embedding model, not a placeholder | Brief §4.2 (A) | No | +| **0** | Service errors observed across a full round | A full evaluation round ran without a failure | Brief §4.5 (A) | No | +| **47% → 0%** | Share of retrieved candidates from test files, before and after | I fixed a defect users could feel | Inventory, engineering chain 3 (B before / C after) | Yes | +| **48–54 → 62–67** | Verified citations, before and after unifying the symbol rule | After the fix, more evidence could be cited | Inventory, engineering chain 1 (A) | Yes | +| **17–21 → 9–13** | Redaction markers, same change | Users see fewer stripped-out references | Inventory, engineering chain 1 (A) | Yes | +| **0 → 2** | Surviving symbol citations per run, same change | A whole class of citation had been dying, and now survives | Inventory, engineering chain 1 (A) | Yes | + +### Group 2 · Needs a qualifying sentence + +Say any of these alone and the follow-up question lands immediately. The qualifier is what makes it safe. + +| Number | The sentence that must go with it | What happens without it | Source | +|---|---|---|---| +| **473** | This is the sum of four edge types; call edges number 303 | Asked "how many are imports", I have nothing | Brief §4.1 | +| **303** | This is the value in the database; today's code re-indexed gives 316 | They pull the code, recompute, and the numbers disagree | Brief §4.1 | +| **2615 → 316, 12.1%** | The denominator is the call sites the worker actually walks | Bare "12%" reads as weakness; it is a declined capability | Brief §4.1 | +| **groundedness 1.000** (share of citations found in the index) | B1's 1.0 is a literal in the code; only the agent arms' is measured, and it needs "answers citing nothing: 0" beside it | Asked "how is that a perfect score", silence looks like gaming | Brief §4.5 | +| **candidate_recall 0.560** | All three hybrid arms are identical here; this measures the retrieval side | Quote it as B4's result and one comparison exposes that all three match | Brief §4.9 item 15 | +| **768 / 1024** | 768 is what runs; the code and compose both default to 1024 | They read `settings.py`, see 1024, and assume I misremembered | Brief §4.2 | +| **`unsupported`** | Three of four comparisons cleared | Stated alone it sounds like total failure | Brief §4.5 | +| **0.0061** | The between-repeat standard deviation is 0.034 | "Only 0.006 off" invites a variance question that exposes the 0.083 spread | Brief §4.5 | +| **+0.038 / +0.006 / +0.088** | The spread of 0.083 is wider than the 0.050 bar | Picking the third run and saying "it passed" is cherry-picking | Brief §4.5 | +| **+0.0218 / +0.0226 / +0.0221 / +0.1466** | All four are diagnostics, excluded from the verdict in code | Presented as the main result, they ask why it is not in the criterion | Brief §4.5 | +| **4/3** | Equivalent to moving the bar from 0.050 to 0.0375 | Omitting it hides a lowered bar, and the `run.py` comment says so anyway | Brief §4.5 | +| **12/12/12 and 14/12/16** | These come from the superseded origin set | Quoted as a result, the next question about the new set breaks it | Brief §4.5 | +| **0/0/0 and 4/3/5** | These are recomputed, not a committed artifact field | They search the artifacts and find neither number | Brief §4.5 | +| **440/248/152/111/36** | Four of the six graph tools never appear at all | The total alone suggests the graph tools earn their keep; four score zero | Brief §4.5 | +| **9/16 vs 10/17** | The two question types are nearly identical here | One ratio on its own leaves the reader without a comparator | Brief §4.5 | +| **BM25 k1=1.2 / b=0.75** (keyword scoring parameters) | BM25 is hand-written in the application layer; it does not use `tsv` or the GIN index | They conclude I used Postgres full-text search | Brief §4.2 | +| **291** | Collection only; I did not run them | "291 tests, all green" collapses when someone asks me to run them | Brief §4.6 | +| **73** | The 71 in `CLAUDE.md` is out of date | Two of my own documents disagree | Brief §4.6 | +| **10,952** | Excludes 16 `.py` files totalling 1743 lines under `node_modules` | They run `wc -l` themselves and get 12,695 | Brief §4.6 | +| **6 + 2** | `make up` does not start those two sidecars | In a live demo every query dies at embedding | Brief §4.6 | +| **4 + 1** | The `index_runs` table exists with no writer in the code | They open the migration and find the table empty | Brief §4.6 | +| **9/33 · q-009** | Under the v2 protocol, recall (share of what should be found that came back) is capped by how much the answer cites | They ask why recall is so low | Brief §4.5 | +| **B0's three file-level figures** | No arm in the recorded H1 run has file-level metrics | They ask "compared to what", and there is no comparator | Brief §4.7 | + +### Group 3 · Verbal only, in an interview + +| Item | Why it stays off the resume | How to say it | +|---|---|---| +| Scoring protocol v1 → v2, one rule for all four agent arms | Needs the whole evaluation story | "I changed the scoring protocol so all four arms used one rule. That change turned my own verdict from supported into unsupported." | +| Correcting the `GRAPH_TOOLS` set | Needs full context | "The counter credited the graph for a tool the no-graph arm still keeps. A measure that cannot tell a graph arm from a no-graph arm is not measuring the graph." | +| Adding the B3.5 control arm (ablation: switch off one component, hold everything else, see how far the metric falls) | It is a diagnostic and excluded from the verdict in code | "I added an arm I did not need. It demoted the headline feature of my own project, and I did not use it to move the verdict." | +| Dropping groundedness from the composite (the mean of three retrieval metrics) | It is not an improvement | "I removed a metric term, and that genuinely lowered the bar. Three defences are checkable, and both readings return unsupported." | +| Scoring an uncited answer 1.0 → 0.0 | It is not an improvement | "I changed a scoring convention, and two conclusions from my own earlier experiment stopped holding. The corrected numbers run against me." | +| Hydrating graph evidence: L3 call-graph contribution **−0.0067 → +0.0226** | It is a diagnostic and excluded from the verdict | "In one run the no-graph control arm actually beat the full arm on architecture questions. The cause was measurement bias: graph results carried only file and line, retrieval results carried source. After I filled in the source and ranked everything together, the same figure went from negative to positive." | +| Call resolution: call edges **303 → 316** | It is not an outcome | "I resolved two call shapes the analysis had missed. That adds a dozen edges, but I measured the downstream effect myself and it unlocks zero additional architecture-level ground-truth chains, and it needs a re-index to take effect." | +| RRF constant **K=60**, weights **dense:sparse = 2:1** | Configuration; a reader cannot judge whether 60 is good | "I fuse with RRF, which merges two ranked lists by position, instead of adding normalised scores. BM25 scores are unbounded and cosine is not. The weighting is unequal because at 1:1 the dense-only arm out-scored hybrid." | +| Candidate funnel **50 → 16 → 5** | Configuration | "Retrieval is a three-stage funnel: 50 candidates from each side, the top 16 reranked by a cross-encoder that reads question and code together, and 5 returned." | +| BM25 **k1=1.2 / b=0.75** | Configuration | "The parameters go into the run artifact, so that baseline reproduces. It is also why I did not use `ts_rank`, which is not BM25." | +| Step cap **14** | Configuration | "Raised from 8, because the strongest arm kept terminating at the budget rather than at the planner's decision." | +| Context budget **10** | Configuration | "Identical for every arm, which constrains the strongest one. Graph evidence has to displace retrieval evidence to earn a slot." | +| Base-class BFS depth **6** | Configuration | "It approximates the MRO without being the MRO. It ignores C3 linearisation, so a diamond can resolve to the wrong sibling." | +| Vector storage on **pgvector** with an HNSW cosine index | A technology choice, not an outcome | "Vectors and the call graph live in one Postgres, with pgvector (its vector extension) doing the nearest-neighbour query. That removes a separate vector store." | + +### Two things I have to say first, before anyone asks + +**About the 47% → 0% figure.** The commit message discloses a conflict of interest. I found this while investigating why L3 missed the bar by 0.0036, and no ground-truth file lives under `tests/`, so filtering test code also mechanically helps the metric. I have to say that sentence myself, in the interview. It changes character entirely if the other side reads it out of the commit first. + +**About the three counts from unifying the symbol rule.** Those nine runs are single-baseline runs and carry no H1 verdict; `results/README.md` labels them "not a verdict". The experiment report states that the counts are the evidence and the means are the weak part. So I quote the three counts and never the groundedness means. On n=3 those mean ratios sit around p≈0.25 and p≈0.1, and the report says explicitly that it is not claiming statistical significance. + +--- + +## Notes for my mentor + +### 1. This project deliberately has no throughput, latency, token-cost or accuracy-lift numbers + +Please do not suggest adding any. Anything I added would be invented, and not inventing is what this project is for. Point by point: + +| Missing | Why | +|---|---| +| **Throughput / QPS / latency** | **No load testing was ever done.** Everything runs on one machine with CPU inference sidecars. The artifacts contain no timing fields and the code has no latency instrumentation. Any number would be made up | +| **Token cost** | **No cost accounting was ever done.** There is no token-counting logic anywhere in the code. The only budget dimension is step count | +| **Accuracy lift, "improved X%"** | The evaluation is offline, one repository, 33 questions, and the main hypothesis came back `unsupported`. Writing "improved by X%" would mean cherry-picking a favourable subset or quoting a diagnostic as a result. Both collapse at the second follow-up question | +| **Answer-quality metrics** | **The judge is not wired**, and `pairwise_win_rate` is `null` throughout. This evaluation can say whether a citation genuinely exists. It cannot say whether the answer is good | + +### 2. Three kinds of evidence that stand in for quantified impact + +| Type | Examples | +|---|---| +| **Mechanistic guarantees** | Citations that fail verification are removed from the answer. The score is computed before removal, so removal cannot launder it. Zero citations scores zero. An interrupted turn is a distinct state and its citations never bind | +| **Methodological strength** | Pass criteria were pre-registered. Five baselines, with four agent arms sharing one model, one prompt, one protocol and one guard. A self-built control arm does the causal split and is excluded from the criterion. Three repeats with variance analysis. A protocol change that ran against my own interest was published anyway | +| **Engineering scale** | 7 workspace packages, 6+2 containerised services, 4+1 tables, 11 tools, a 5-node state machine, 291 pytest cases, 73 vitest cases, 10,952 lines of Python plus 7,678 of tests, 4,746 lines of frontend plus 1,445 of tests. Test volume is close to a third of source, and most of it pins rules like citation verification rather than rendering | + +### 3. Six phrasings that must not come back during editing + +Each one collapses at the first or second follow-up in a technical interview: + +| # | Do not write | What is true | +|---|---|---| +| 1 | "a call graph with 473 edges" | 473 is the **sum of four edge types** (calls / references / imports / inherits). **Call edges number 303** | +| 2 | Groundedness 1.0 presented as a result | The retrieval-only baseline's 1.0 is a **literal in the code**. Only the agent arms' figure is measured, and it only means something alongside "answers citing nothing: 0" | +| 3 | "multi-step evidence gathering is six times the call graph" | **Only at architecture level** (+0.1466 against +0.0226). **At cross-file level the two are comparable** (+0.0221 against +0.0218) | +| 4 | Recall described as retrieval capability | Under the v2 protocol the scored set is the verified citations in the answer. The three hybrid arms have **identical retrieval-side recall, 0.560** | +| 5 | "supports Postgres full-text search" | `chunks.tsv` and its GIN index exist but **have no writer anywhere in the codebase**. Sparse retrieval is **hand-written Okapi BM25 in the application layer** | +| 6 | "multi-language / cross-language code understanding" | The parser is the stdlib `ast` module and scans **`*.py` only** | + +### 4. Why the H1 verdict reads differently in two places + +The bullets sell a falsifiable evaluation design. The interview section states the full conclusion, including `unsupported` and the cross-file level being unmeasurable. + +The two serve different purposes. A resume has thirty seconds to earn a conversation, so it is screening material. Interview positions have to survive three layers of follow-up, so they are load-bearing material. The information is the same and the density differs. The bullet says "I built an evaluation capable of falsifying my own hypothesis", which is true. The interview section adds "and it did falsify it", which is also true and works in my favour. Putting "my hypothesis failed" straight into a bullet moves load-bearing material into a screening slot, where it gets filtered before I can explain the methodology. + +--- + +## ⚠️ For me to confirm or supply + +| # | Open item | Note | +|---|---|---| +| 1 | **Team split and the boundary of my own work** | The git history contains PR merges, so this may not have been solo. The resume has to be clear about what I led and what I contributed to. **Not submittable until this is settled** | +| 2 | **Time span** | Start and end months. The first commit date is a reference point, not a measure of effort | +| 3 | **Public repository, and whether to link it** | If linking, note the repository holds `.env.example`, evaluation artifacts and internal documents. Worth a pass for anything not meant to be public | +| 4 | **Live URL** | Deployment today is local `docker compose`. Apart from a production compose overlay there is no sign of a hosted instance. With no live URL, the resume should not imply one | +| 5 | **Whether D-level figures belong on a resume** | Figures tagged **D** in the ledger (the 303 call edges, the recomputed 316, the second corpus's index size) depend on a running database and cannot be checked from the repository. My inclination: keep D-level figures out of the resume body, and allow them in interview with the source level stated | +| 6 | **Project name and one-liner** | Confirm the wording here matches whatever heads the resume and the LinkedIn entry | + +--- + +## Appendix A: ledger for figures new to this document + +Source levels follow the brief. **A** is a committed artifact field. **B** is a committed source constant. **C** is recomputed this pass. **D** depends on a running database or the local `.env`, and cannot be checked from the repository. + +| Value | Meaning | Evidence | Level | **The sentence that must go with it** | +|---|---|---|---|---| +| **5** | LangGraph node count | `apps/agent/src/dcode_agent/graph.py:774-778` | **B** | "`plan ⇄ tool_call` is the ReAct loop, and both nodes can branch conditionally into `synthesize`" | +| **7** | Size of the closed SSE event set | `packages/shared/src/dcode_shared/events.py:13-21` | **B** | "It is a closed set pinned by a `Literal` type, and every payload is a Pydantic model" | +| **4** | Agent modes sharing one lookup table | `apps/agent/src/dcode_agent/state.py:32-44` | **B** | "The four agent arms share one model, one prompt, one protocol and one guard. The difference comes only from this table, which is what makes the controlled experiment valid" | +| **6 / 4 / 5** | Indexing: externally visible states, `PipelineStage` entries, stage functions | `packages/shared/src/dcode_shared/schemas.py:18-27`; `apps/worker/src/dcode_worker/pipeline.py:51-56` | **B** | "Six is the normal path `queued→cloning→parsing→embedding→graphing→ready`, with `failed` alongside. The `parsing` state runs both parse and chunk" | +| **2 jobs** | CI jobs and their steps | `.github/workflows/ci.yml:10,31` | **B** | "**CI does not run the evaluation-artifact consistency check.** That step exists only in `make lint` (`Makefile:70`)" | +| **180s** | git clone timeout | `apps/worker/src/dcode_worker/stages/clone.py:11` | **B** | "On timeout the subprocess is killed and the job moves to `failed`. The worker does not hang" | +| **1MB / 20k** | Per-file skip threshold, per-chunk content cap | `apps/worker/src/dcode_worker/settings.py:12-13` | **B** | "Both can be disabled by setting 0. Skipped files reach the frontend as warnings rather than being dropped silently" | + +## Appendix B: phrasings carried over from the brief but rewritten here + +| # | In the brief | Here | Why | +|---|---|---|---| +| 1 | "six-state machine" | "six externally visible states, 4 `PipelineStage` entries, 5 stage functions" | "Six states" is accurate for the enum, but a reader takes it to mean six stage functions. It is 4 entries driving 5 functions, since `parsing` runs two | +| 2 | "`make lint` verifies evaluation-artifact consistency" | Adds "CI does not run this step" | `ci.yml`'s Python job has four steps: ruff, mypy, pytest, eval smoke. No `sync_eval_artifacts.py --check`. Without the distinction a reader assumes CI enforces it | +| 3 | "multi-step evidence gathering is six times the call graph" | Always qualified to **architecture level**, immediately followed by "at cross-file level the two are comparable, a ratio of 1.01" | The claim holds at one level only. The bullets use no ratio at all, and the interview section gives all four figures across both levels | +| 4 | The call graph appears early as the project's selling point | Order is now citation verification first, then how the agent is kept in check, then the evaluation. The call graph appears under "Evaluation" and in interview position 8 | My own controlled experiment measured it down, so opening with it means opening on the weakest link | +| 5 | 473 and 303 sit as two separate ledger rows | 303 appears only in the interview section, flagged as the value in the database. The bullets use no edge count | Any edge count needs a qualifier however it is written, which does not fit a bullet that has thirty seconds to land | +| 6 | Arms referred to as `B1/B2/B3/B3.5/B4` throughout | Mentor-facing passages say "retrieval-only baseline / dense RAG / hybrid / no-graph control arm / full system" | A mentor has not read the brief, and the codes get in the way. Technical sections keep them for cross-referencing | +| 7 | Groundedness 1.000 described as measured | Stated explicitly: the retrieval-only baseline's 1.0 is a literal, only the agent arms' is measured, and it needs `answers_without_citations = 0` alongside it to mean anything | This is one of the six forbidden phrasings, so the resume material has to nail it down before an editing pass turns it back into a result |