Repository navigation
test(rag): full offline acceptance and failed real ask evidence - #85
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Outcome: NOT release-accepted
The complete offline suite passes, but the separately authorized real-provider ask acceptance stopped on its first positive case with HTTP 503. This PR records the failure, not a successful rollout or a model-quality promotion. No business/runtime algorithm, prompt, timeout, default switch or ticket path is changed.
Offline coverage
tests/on Windows Python 3.12.4: latest 622 passed, zero failures/skips. Earlier pre-guard run: 621 passed with one third-party Numba cast warning.data/access; fixtures use temporary databases/data. HF offline mode and DeepEval telemetry opt-outs enabled.Bounded ask acceptance
Reuse the existing plan-only smoke entry point with
--ask; add a fixed seven-document/eight-case fixture module. Expected outcomes remain local and are not sent to models. Default search smoke retains its old 8/3 caps.Human approved at most 14 Beijing text-embedding-v4, 6 Singapore qwen3-rerank, 6 Singapore qwen3.5-plus-2026-04-20 classifiers and 2 Beijing qwen3.5-plus generations, planning budget CNY 2. No automatic retry/redirect; abort on the first unexpected result. Endpoint/model/text/prompt/body-size allowlists are checked before sending.
Observed once, then stopped
version_and_time=NOT_REQUIREDdespite an explicit version 1.0 constraint. The source covers 1.0, but this audit status contradicts the policy; a valid schema/locator is not semantic proof.Safe per-attempt metadata, classifier completion, case statuses and execution-time script hashes are retained in
docs/reports/hybrid_acceptance_2026-10-17.json.Still pending
Generation configuration/timeout diagnosis and an explicitly authorized re-test; remaining real cases; independent semantic acceptance. Docker engine is unavailable, so live-server HTTP/restart/persistence/rollback acceptance is not passed. Ticket Agent remains Dense; no all-Hybrid chain or public deployment claim.
Primary checkout changes, existing databases, .env and persistent switches remain untouched. No business documents, TechQA, ground-truth labels or secrets in model inputs; credentials only used for normal provider authentication. Merging this PR does not enable the experiment or resolve the observed failure.