Skip to content

test(rag): bounded real-provider Hybrid runtime smoke - #83

Merged
Air000000 merged 1 commit into
mainfrom
codex/hybrid-runtime-smoke
Oct 16, 2026
Merged

Air000000 merged 1 commit into
mainfrom
codex/hybrid-runtime-smoke

Conversation

@Air000000

Copy link
Copy Markdown
Owner

Summary

  • Add a plan-only-by-default synthetic smoke script for the optional Hybrid search runtime. Real execution requires explicit human authorization; the CLI flag itself does not grant it.
  • Exercise actual ASGI routes and Demo JWT with isolated temporary SQLite, Chroma and upload storage. Reuse existing indexing/search services rather than changing business behavior or deploying a server.
  • Bound wire attempts to 8 Beijing text-embedding-v4 + 3 Singapore qwen3-rerank, no SDK/transport retries or redirects, fixed endpoint/model/text allowlists, 4 KiB per body / 16 KiB cumulative. Record only safe attempt metadata and provider usage.
  • Add offline full-flow, budget/egress and failed-send accounting checks to the existing Hybrid test file already in CI.

Authorized observed run — PASS

Executed once after the user explicitly approved exactly the four fixed synthetic documents/queries, maximum 8 embedding + 3 rerank calls, planning budget CNY 1.

  • Actual: 8 embedding + 3 rerank requests, all HTTP 200, no retries.
  • Provider usage: 126 embedding tokens + 162 rerank tokens = 288 total; 1,925 cumulative JSON body bytes. Usage is not a billing receipt; no actual billed amount is asserted.
  • Passed: upload/index, real JWT tenant scope, spoofed body tenant ignored, Dense default preserved, basic synthetic Hybrid VPN ranking, category filtering, deletion, empty corpus and typed retrieval audit.
  • Recorded safe per-attempt evidence and script SHA-256 in docs/reports/hybrid_runtime_smoke_2026-10-16.json.

Verification and boundaries

  • 340 offline tests passed; ruff check ., in-memory script compilation, result consistency checks and git diff --check passed.
  • No business documents, TechQA, labels or secrets in model inputs. Provider credentials were used only for normal API authentication.
  • No answer generation, refusal experiments, ticket creation, existing-service configuration changes or public deployment. Primary checkout changes untouched.
  • This is an isolated ASGI wiring smoke, not live-server HTTP/Docker/production-load acceptance, independent retrieval-quality proof or evidence that Hybrid outperforms Dense. Default Hybrid remains disabled; ask/refusal/ticket paths remain Dense.

@Air000000
Air000000 merged commit d34e36d into main Oct 16, 2026
1 check passed
@Air000000
Air000000 deleted the codex/hybrid-runtime-smoke branch October 16, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant