test(rag): full offline acceptance and failed real ask evidence

.github/workflows/tests.yml

@@ -88,3 +88,54 @@ jobs:
             tests/test_refusal_v2_ai_proxy_runner.py \
             tests/test_refusal_v3_ai_proxy.py \
             tests/test_refusal_v3_ai_proxy_runner.py
+
+      - name: Install declared offline evaluation test dependencies
+        run: pip install -r requirements.txt -r experiments/evals/requirements-eval.txt
+
+      - name: Run complete offline test directory
+        env:
+          SQL_ECHO: "false"
+          AUTH_DEMO_USERS_JSON: ""
+          JWT_SECRET: offline-ci-test-only
+          PYTEST_DISABLE_PLUGIN_AUTOLOAD: "1"
+          ANONYMIZED_TELEMETRY: "False"
+          DEEPEVAL_TELEMETRY_OPT_OUT: "YES"
+          DEEPEVAL_UPDATE_WARNING_OPT_OUT: "YES"
+          HF_HUB_OFFLINE: "1"
+          HF_DATASETS_OFFLINE: "1"
+          NUMBA_NUM_THREADS: "1"
+          RAG_HYBRID_SEARCH_ENABLED: "false"
+          RAG_HYBRID_ASK_ENABLED: "false"
+          OPENAI_API_KEY: offline-placeholder
+        run: |
+          python -B - <<'PY'
+          import gc, ipaddress, os, sys, tempfile
+          from pathlib import Path
+          import pytest
+          private_data = str((Path.cwd() / "data").resolve()).casefold() + os.sep
+          def guard(event, args):
+              if event == "open" and isinstance(args[0], (str, bytes, os.PathLike)):
+                  target = str(Path(os.fsdecode(args[0])).resolve()).casefold()
+                  if target.startswith(private_data):
+                      raise RuntimeError("private project data forbidden in offline tests")
+              if event in ("socket.connect", "socket.getaddrinfo"):
+                  host = args[1][0] if event == "socket.connect" else args[0]
+                  if host == "localhost":
+                      return
+                  try:
+                      if ipaddress.ip_address(host).is_loopback:
+                          return
+                  except ValueError:
+                      pass
+                  raise RuntimeError("external network forbidden in offline tests")
+          sys.addaudithook(guard)
+          with tempfile.TemporaryDirectory(prefix="offline_ci_") as temp:
+              os.environ["DATABASE_URL"] = "sqlite:///" + str(Path(temp) / "tests.db")
+              try:
+                  code = pytest.main(["-q", "--tb=short", "-p", "no:cacheprovider", "tests"])
+              finally:
+                  import database
+                  database.engine.dispose()
+                  gc.collect()
+          raise SystemExit(code)
+          PY

README.md

@@ -289,13 +289,34 @@ Hybrid 检索（同一租户/分类、已索引当前版本，最多 Top14）
 ```
 
 - 固定分类模型 `qwen3.5-plus-2026-04-20`，复用原 v5.3 提示词与严格解析器，不读取评测样本、标签或 DEV；分类最多 2048 输出 tokens，生成最多 1024，两者超时各 30 秒、无 SDK 自动重试。检索侧 embedding 仍沿用原客户端策略。
-- 问题和最多 14 条全文会额外发往分类服务；放行后再发往配置的生成服务。非空请求比搜索最多增加两次 chat 调用；没有新增全局费用硬限额。本阶段没有真实执行这些新增调用。
+- 问题和最多 14 条全文会额外发往分类服务；放行后再发往配置的生成服务。非空请求比搜索最多增加两次 chat 调用；没有新增全局费用硬限额。首次真实问答验收未通过，见下一节。
 - 分类格式错误、非法行引用、截断、超时、空生成或答案缺少/含未批准的 `[Source N]` 均为脱敏 HTTP 503 + `failed` 审计，不算正确拒答、不暗中放行或回退 Dense。显式选择 Dense 并移除实验策略可回退；关闭 ask 开关只禁用实验问答。
 - 发送分类/生成上下文前及返回前复查 SQL 可见性和内容，变化则终止；仍不是跨 SQL/Chroma 原子快照。返回 `top_distance=null`，sources 保留距离与重排分数的不同语义。
 - 响应与成功/正常拒答审计包含 `refusal_policy`、`evidence_gate`（需求、覆盖检查、原文证据、缺口与歧义）；失败审计仅记录模式、策略及异常类型，不泄露提供方错误正文。AgentOps 可按两个新增拒答状态过滤，汇总 `retrieval_status_counts` 显示全部状态。
 
 **校验只证明格式、引用出处和状态接线，不证明语义蕴含、逐句忠实或抗提示注入。** 不把有效引用或实验 gate 放行写成答案正确率，也不改写历史 NO_PROMOTION / FAIL。
 
+## 3.3 集成验收状态（2026-10-17）
+
+**当前不接受为已验收发布版。** 已补齐仓库声明的评测测试依赖，在隔离 Windows / Python 3.12.4 环境运行完整 `tests/`：622 项通过、零失败/跳过；不只原重点回归集。主 pytest 进程禁止外部 socket 与读取项目私有 `data/`，测试使用临时数据；CI 增加相同完整离线检查。离线成功不是模型质量验收。
+
+真实问答计划复用同一脚本，默认仍然零调用：
+
+```bash
+python -m scripts.smoke_hybrid_runtime --ask
+# 仅在单独授权后执行；异常即停，不自动续跑或重新执行：
+python -m scripts.smoke_hybrid_runtime --ask --execute --env-file .env
+```
+
+固定 7 篇虚构文档、8 类案例，预期标签只在本地检查。独立上限为 14 次北京 embedding、6 次新加坡重排、6 次新加坡分类、2 次北京生成，禁用重试和重定向；请求体最多 24 KiB、累计 96 KiB，保守预算 ¥2 而非账单硬限额。生成只获准用于两个预设可回答案例；若负例错误放行，发送守卫会阻断生成并判验收失败，不改写为正常拒答。旧搜索模式仍保留原 8/3 上限。
+
+本次获授权执行一次，在首个可回答案例停止：9 次 embedding、1 次重排、1 次分类均 HTTP 200；生成尝试约 30 秒未获得可用响应，接口 HTTP 503。已报告 usage 合计 2639 tokens，**生成用量未知，可能计费，不能当作零费用**。剩余 7 类真实案例未执行，没有重试；[完整失败记录](docs/reports/hybrid_acceptance_2026-10-17.json) 保留输出、调用计数及执行时脚本哈希。
+
+另有语义警示：问题明确指定版本 1.0，分类器却把 `version_and_time` 标为 `NOT_REQUIRED`。来源确实覆盖该版本，但这个审计判断不符合提示词约定，格式和引用通过不能证明语义判断可靠。
+生成请求没有显式关闭思考模式，而 [官方文档](https://www.alibabacloud.com/help/en/model-studio/deep-thinking) 说明 Qwen3.5 Plus 默认开启思考。这是应排查的配置风险，**不是本次超时唯一原因的证据**；没有为通过验收而修改超时、提示词、业务代码或重跑。
+
+Docker 引擎当前不可连接，真实 HTTP 服务、容器重启/持久化与回退验收均未完成；Ticket Agent 仍使用 Dense，不声明全链 Hybrid。下一步先诊断/修复生成请求配置与超时观测，再单独授权复测；当前保持默认开关关闭。
+
 ---
 
 # 4. RAG 评测与迭代：从 Dense 基线到当前状态

docs/reports/hybrid_acceptance_2026-10-17.json

@@ -0,0 +1,197 @@
+{
+  "date": "2026-10-17",
+  "status": "FAILED_NOT_RELEASE_ACCEPTED",
+  "runtime_base_commit": "73174a03eaa17db3ee334e3d3042f5315b60d30a",
+  "execution_count": 1,
+  "script": "scripts/smoke_hybrid_runtime.py",
+  "script_sha256_at_execution": "53a54ca40c2ab5c8d452ae9dce0c1c354c5c0b18ab7bd63e35721909d1647643",
+  "cases_module": "scripts/hybrid_ask_cases.py",
+  "cases_sha256_at_execution": "50e051ea61e162e2534ba7867b68a9b8cd7766e4d0ac983edd7c92905f76a082",
+  "scope": "Fixed 7 fictional documents / 8-case isolated ASGI acceptance; real provider HTTP; no running-server or public deployment",
+  "authorized_limits": {
+    "embedding": 14,
+    "rerank": 6,
+    "classifier": 6,
+    "generation": 2
+  },
+  "budget_cny": 2,
+  "retries": 0,
+  "offline_tests": {
+    "collected_and_passed": 622,
+    "failed": 0,
+    "skipped": 0,
+    "environment": "isolated Windows Python environment; requirements.txt plus experiments/evals/requirements-eval.txt; external Python socket and private project data guards",
+    "initial_collection_errors_resolved": 22
+  },
+  "observed": {
+    "attempts": 12,
+    "embedding": 9,
+    "rerank": 1,
+    "classifier": 1,
+    "generation": 1,
+    "known_reported_tokens": 2639,
+    "generation_usage": "UNKNOWN: no usable response; may have been billed",
+    "total_request_body_bytes": 12780
+  },
+  "failure": {
+    "case": "supported",
+    "expected_status": "ok",
+    "http_status": 503,
+    "provider_generation_response_status": null,
+    "generation_transport_duration_ms": 30059,
+    "cause": "No usable generation response within the 30-second timeout; exact timeout phase was not recorded"
+  },
+  "semantic_warning": {
+    "facet": "version_and_time",
+    "observed": {
+      "status": "NOT_REQUIRED",
+      "evidence": [],
+      "reason": "The question specifies a fixed version (1.0) and does not request current/latest status or time-sensitive information beyond that static version constraint."
+    },
+    "assessment": "The question explicitly constrains version 1.0. NOT_REQUIRED conflicts with the frozen policy for explicit version constraints, even though the available source covers version 1.0. A valid locator/schema is not semantic quality proof."
+  },
+  "diagnostic_hypothesis": {
+    "observation": "The generation request omits enable_thinking while using qwen3.5-plus with timeout=30 and max_tokens=1024",
+    "official_documentation": "https://www.alibabacloud.com/help/en/model-studio/deep-thinking",
+    "documented_default": "Qwen3.5 Plus thinking enabled by default",
+    "conclusion": "Configuration risk, not a proven unique cause of this timeout; no parameter changes or paid retries performed"
+  },
+  "cases": [
+    {
+      "case": "supported",
+      "expected_status": "ok",
+      "http_status": 503,
+      "actual_status": null,
+      "answer": null,
+      "gate": null,
+      "provider_attempts": 4
+    },
+    {
+      "case": "missing",
+      "status": "NOT_RUN_AFTER_FIRST_FAILURE"
+    },
+    {
+      "case": "version",
+      "status": "NOT_RUN_AFTER_FIRST_FAILURE"
+    },
+    {
+      "case": "conflict",
+      "status": "NOT_RUN_AFTER_FIRST_FAILURE"
+    },
+    {
+      "case": "chinese",
+      "status": "NOT_RUN_AFTER_FIRST_FAILURE"
+    },
+    {
+      "case": "unrelated",
+      "status": "NOT_RUN_AFTER_FIRST_FAILURE"
+    },
+    {
+      "case": "empty",
+      "status": "NOT_RUN_AFTER_FIRST_FAILURE"
+    },
+    {
+      "case": "deleted",
+      "status": "NOT_RUN_AFTER_FIRST_FAILURE"
+    }
+  ],
+  "provider_attempts": [
+    {
+      "kind": "embedding",
+      "bytes": 279,
+      "status": 200,
+      "total_tokens": 50,
+      "latency_ms": 273
+    },
+    {
+      "kind": "embedding",
+      "bytes": 238,
+      "status": 200,
+      "total_tokens": 32,
+      "latency_ms": 124
+    },
+    {
+      "kind": "embedding",
+      "bytes": 219,
+      "status": 200,
+      "total_tokens": 38,
+      "latency_ms": 181
+    },
+    {
+      "kind": "embedding",
+      "bytes": 233,
+      "status": 200,
+      "total_tokens": 34,
+      "latency_ms": 135
+    },
+    {
+      "kind": "embedding",
+      "bytes": 247,
+      "status": 200,
+      "total_tokens": 36,
+      "latency_ms": 130
+    },
+    {
+      "kind": "embedding",
+      "bytes": 229,
+      "status": 200,
+      "total_tokens": 40,
+      "latency_ms": 147
+    },
+    {
+      "kind": "embedding",
+      "bytes": 172,
+      "status": 200,
+      "total_tokens": 28,
+      "latency_ms": 168
+    },
+    {
+      "kind": "embedding",
+      "bytes": 152,
+      "status": 200,
+      "total_tokens": 24,
+      "latency_ms": 152
+    },
+    {
+      "kind": "embedding",
+      "bytes": 152,
+      "status": 200,
+      "total_tokens": 24,
+      "latency_ms": 187
+    },
+    {
+      "kind": "rerank",
+      "bytes": 446,
+      "status": 200,
+      "total_tokens": 82,
+      "latency_ms": 394
+    },
+    {
+      "kind": "classifier",
+      "bytes": 9477,
+      "status": 200,
+      "total_tokens": 2251,
+      "completion_text": "{\n\"requirements\": [\n{\n\"requirement\": \"Provide resolution steps for NovaVPN 1.0 error 809 specifically when the firewall already allows VPN traffic.\",\n\"status\": \"SUPPORTED\",\n\"evidence\": [\n{\n\"source_id\": \"Source 1\",\n\"segment_ids\": [\n1\n]\n}\n],\n\"reason\": \"Source 1 explicitly provides the complete recovery steps (close client, reopen, reconnect) for this exact product, version, error code, and condition.\"\n}\n],\n\"coverage_checks\": {\n\"requested_outcomes\": {\n\"status\": \"SUPPORTED\",\n\"evidence\": [\n{\n\"source_id\": \"Source 1\",\n\"segment_ids\": [\n1\n]\n}\n],\n\"reason\": \"The source provides direct actionable steps to resolve the specific error under the specified condition.\"\n},\n\"applicability\": {\n\"status\": \"SUPPORTED\",\n\"evidence\": [\n{\n\"source_id\": \"Source 1\",\n\"segment_ids\": [\n1\n]\n}\n],\n\"reason\": \"The source matches the product (NovaVPN), version (1.0), error code (809), and environmental condition (firewall allows VPN traffic) requested in the question.\"\n},\n\"version_and_time\": {\n\"status\": \"NOT_REQUIRED\",\n\"evidence\": [],\n\"reason\": \"The question specifies a fixed version (1.0) and does not request current/latest status or time-sensitive information beyond that static version constraint.\"\n}\n},\n\"unresolved_ambiguities\": []\n}",
+      "latency_ms": 6140
+    },
+    {
+      "kind": "generation",
+      "bytes": 936,
+      "status": null,
+      "total_tokens": null,
+      "latency_ms": 30059
+    }
+  ],
+  "pending": [
+    "Repair/diagnose generation configuration and timeout observability before separately authorized re-test",
+    "Remaining 7 real-provider cases",
+    "Independent semantic quality acceptance; v5.3 remains experimental",
+    "Real running-server HTTP, Docker restart/persistence and rollback acceptance; Docker engine unavailable",
+    "Ticket Agent remains Dense; no all-Hybrid ticket-chain claim"
+  ],
+  "non_actions": [
+    "No business documents, TechQA, ground-truth labels or secrets in model inputs",
+    "No ticket creation, main checkout changes, persistent Hybrid enablement or public deployment",
+    "No automatic rerun, no spent/unknown requests relabeled as normal refusal"
+  ]
+}