006 local ml tier - #2
Merged
Merged
Conversation
…es not survive Eight tasks of feasibility work behind the eval crate's opt-in `ml` feature, so no shipping crate resolves Candle to learn any of it. The classifiers work. Prompt Guard 2 86M loads in Candle's debertav2 with no ONNX path and separates by 0.9974 against ProtectAI's 0.6118 — at 444ms per inference, 1.08 GiB, and a gated licence. Candle on x86 is 3-9x slower than the Apple Silicon figures this feature was scoped against, which promotes selective inference from optimisation to requirement. The embedding half is where the measurement bites. Ranking the injected segment against its siblings reaches 55.6% top-1 (T006). Cutting prose finer to fix the diluted rows made it worse, 55.6% -> 51.9% (T007) — the bound is sibling homogeneity, not payload isolation. And asking the same score whether a document is hostile at all returns 3.1% against a 25% criterion (T008): every document has a most-unlike-its-siblings segment, and its oddness carries no information. M1's 55.6% and M2's 3.1% are the same score answering different questions. The ranker survives; the detector does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eant to corroborate it does not Phase 1. `crates/ml` is a workspace member, `finalize::with_ml` merges its findings, and the corroboration requirement that was supposed to gate them is gone. T008 measured the second signal at 3.1% TPR against a 25% kill criterion. Row 2 of the corroboration table — classifier plus outlier anomaly — was the only row that could produce a finding the structural tier had not already produced, so keeping the table without it leaves a tier that reaches none of the sixteen payloads the rules cannot phrase. That is SC-601, and it is the reason the feature exists. The choice was a tier that cannot meet its success criterion or a tier with one less layer of defence, and the requirement was dropped rather than propped up with a threshold nobody could defend. The old table is struck through in the contract, not deleted. What now controls false positives is the threshold and SC-602, and nothing else. SC-602 stops being a checkbox and becomes the gate deciding whether --ml may ever default on. The threshold is per-model and not portable: ProtectAI scores an ordinary imperative at 388 per-mille, Prompt Guard 2 scores the same text at 4. Severity is a bounded ramp, 40 at threshold to 75 at 1000, rather than the probability itself. The ceiling sits below the structural maximum of 90 on purpose — a rule an operator can read outranks a model nobody can. Probability and threshold are stored per-mille rather than as f32. Verdict derives Eq, and the contract's determinism argument already wanted the quantisation: a verdict recording 0.87421 would differ between two machines agreeing about every decision made from it. Three corrections to the contract, found by implementing it. with_ml takes five arguments, not three — bands because the score moves, bounds because ML excerpts must cross the same sanitisation boundary structural ones do. Reasons are reordered by offset and must be, or output depends on which tiers ran and SC-011 fails for every --ml scan. A truncated verdict is refused on rejudge's D9 argument, since recomputing from survivors lowers the score while claiming to add evidence. Inference returns Outcome rather than Option. The contract said an error returns 0.0 for a classification and a zero vector for an embedding; both are fail-opens. 0.0 reads as a confident benign, and a zero vector is a valid-looking input to cosine similarity that scores as maximally unlike everything — an inference error would manufacture the top outlier in the document. T012 and T013's acceptance bars are restated against Phase 0's measurements rather than asserted as written; T004 predicted its own would fail and it would have. What is asserted is separation and ordering. FR-612's chunking is tested rather than assumed: a payload past the 512-token window still scores 900+, which is the test that fails under mean pooling or silent truncation. ci/check-ml-isolation.sh exists before the Phase 2 edge does. ci/check-cli-dependencies.sh, which T010 named, never existed. Core still pins at 27 crates and still builds for wasm32. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add source policies, opt-in export permissions, CLI/schema attribution, and an exact-byte replay harness with recorded lab and authored-case experiments. Bind destination grants to explicit export phrases so an unrelated approved URL cannot clear an unapproved export. Preserve coverage gaps, truncation, policy attribution, and prior reports across verdict composition. Bound escaped judge requests and repair confusable spans after invalid UTF-8. Keep the local model experiments and export detector's remaining precision/recall limitations documented. Validation: 549 workspace tests passed, 2 existing fixture-quality failures, 11 ignored; 57 evaluation and 55 offline CLI tests passed. Workspace Clippy, formatting, and core Wasm build passed. All 60 authored experiment decisions were unchanged by the destination correction. No deployment or live judge run.
Pin all 71 fixture identities, labels, detection outcomes, and coverage statuses in a reviewed regression baseline. Keep the eight misses and one false positive visible, and run the unchanged positive target and enforced 200-case/1% benign target as separate manual/tag-triggered release-quality checks. Test and lint workspace defaults, the offline CLI, and Candle separately. Restore the offline HTTP/TLS dependency guard and enable the default CLI ML isolation guard. Add a manual/tag-triggered inference workflow that verifies pinned public model assets before running all eight real-weight tests offline. Missing assets fail explicit inference runs instead of silently passing. Make live judge agreement/discrimination tests opt-in so canary credentials do not trigger live evaluation. The credential gate now propagates suite failures. Document the distinct guarantees, known quality gaps, and local reproduction. Validation: sequential workspace and credential-leak suites passed; 55 offline CLI, 19 Candle logic, 8 real-weight, and 57 evaluation tests passed. All three Clippy configurations, formatting, actionlint, dependency/isolation guards, generated-corpus check, and offline evaluation gate passed. Mutation checks confirmed changed fixture outcomes, missing model assets, and failed test suites fail their respective gates. The two absolute fixture-quality checks still fail as documented. Workflows were validated locally; nothing was pushed or deployed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.