Skip to content

test(rag): full offline acceptance and failed real ask evidence - #85

Merged
Air000000 merged 1 commit into
mainfrom
codex/hybrid-acceptance
Oct 17, 2026
Merged

Air000000 merged 1 commit into
mainfrom
codex/hybrid-acceptance

Conversation

@Air000000

Copy link
Copy Markdown
Owner

Outcome: NOT release-accepted

The complete offline suite passes, but the separately authorized real-provider ask acceptance stopped on its first positive case with HTTP 503. This PR records the failure, not a successful rollout or a model-quality promotion. No business/runtime algorithm, prompt, timeout, default switch or ticket path is changed.

Offline coverage

  • Installed both already-declared requirement files in the isolated virtual environment, resolving initial missing optional dependencies / 22 collection-error files.
  • Entire tests/ on Windows Python 3.12.4: latest 622 passed, zero failures/skips. Earlier pre-guard run: 621 passed with one third-party Numba cast warning.
  • Main pytest process denies external sockets and private project data/ access; fixtures use temporary databases/data. HF offline mode and DeepEval telemetry opt-outs enabled.
  • Extend CI beyond the old focused subset: install declared evaluation dependencies, then run the complete offline directory under the same safety guard.
  • Add acceptance transport tests for ground-truth/extra-field/foreign-body rejection and the six-call classifier ceiling. Ruff, pip dependency consistency, workflow YAML/inline-Python compilation and diff checks pass.

Bounded ask acceptance

Reuse the existing plan-only smoke entry point with --ask; add a fixed seven-document/eight-case fixture module. Expected outcomes remain local and are not sent to models. Default search smoke retains its old 8/3 caps.

Human approved at most 14 Beijing text-embedding-v4, 6 Singapore qwen3-rerank, 6 Singapore qwen3.5-plus-2026-04-20 classifiers and 2 Beijing qwen3.5-plus generations, planning budget CNY 2. No automatic retry/redirect; abort on the first unexpected result. Endpoint/model/text/prompt/body-size allowlists are checked before sending.

Observed once, then stopped

  • Actually attempted: 9 embedding + 1 rerank + 1 classifier + 1 generation = 12.
  • Embedding/rerank/classifier returned HTTP 200; known usage totals 2639 tokens.
  • Generation produced no usable response within the 30-second timeout; transport duration 30,059 ms, HTTP status and usage unknown. It may have been billed. API returned 503.
  • First positive case failed; the remaining seven real cases were NOT RUN. No paid rerun, no increased timeout, no post-hoc prompt changes.
  • A semantic warning is preserved too: the classifier marked version_and_time=NOT_REQUIRED despite an explicit version 1.0 constraint. The source covers 1.0, but this audit status contradicts the policy; a valid schema/locator is not semantic proof.
  • Generation omits enable_thinking, while official docs say Qwen3.5 Plus thinking is enabled by default. This is a configuration risk to investigate, not a proven unique cause of this timeout. Exact timeout phase was not recorded; do not invent one.

Safe per-attempt metadata, classifier completion, case statuses and execution-time script hashes are retained in docs/reports/hybrid_acceptance_2026-10-17.json.

Still pending

Generation configuration/timeout diagnosis and an explicitly authorized re-test; remaining real cases; independent semantic acceptance. Docker engine is unavailable, so live-server HTTP/restart/persistence/rollback acceptance is not passed. Ticket Agent remains Dense; no all-Hybrid chain or public deployment claim.

Primary checkout changes, existing databases, .env and persistent switches remain untouched. No business documents, TechQA, ground-truth labels or secrets in model inputs; credentials only used for normal provider authentication. Merging this PR does not enable the experiment or resolve the observed failure.

@Air000000
Air000000 merged commit 338d47f into main Oct 17, 2026
1 check passed
@Air000000
Air000000 deleted the codex/hybrid-acceptance branch October 17, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant