Update Cargo.lock with 40 changed files (#2679)#2736
Conversation
Automatic checkpoint to preserve work in progress. Tests and implementation saved before refactoring phase.
…d function The score → threshold → identity-dedup → store_fact_with_provenance gate was implemented verbatim in two seams — the IPC server's gated_fact_write and the in-process InProcessFactSink::commit_fact — despite the design explicitly requiring both to reach an identical disposition. Extract the shared core into fact_reliability::commit_gated_fact (returning FactGateDecision) so the two seams can never drift; each seam now only resolves its own grounding (store-existence vs batch-membership) and defers the decision. Also dedupe the reliability-gate block_rate computation via gate_block_rate, add a unit test for commit_gated_fact, and update the write-boundary-gate doc. Behavior-preserving: full test suite (incl. SEC-T1..T5 server-gate + distill dedup/quarantine suites) passes; clippy clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Four clarity-preserving optimizations on the paths touched by #2679, none of which change behavior (full test suite + clippy stay green): - score_fact_reliability: bucket word count with split_whitespace().take(3) so a large (up to 64 KiB) content is no longer fully scanned just to decide the 0 / 1-2 / >=3 bucket. - canonical_concept: fold ASCII lowercasing into the single canonicalization pass, dropping a second heap allocation per concept scored. - ledger_record_stored: bump the pass count in place via get_mut, allocating a key String only for the first accepted fact of a pass instead of every one. - InProcessFactSink: precompute the batch episode ids into a HashSet so per-fact grounding is an O(1) lookup instead of an O(batch) scan per fact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| .env("SIMARD_MEMORY_SOCKET", &self.memory_socket) | ||
| // Tag this pass so the daemon's write ledger can report accepted-fact | ||
| // counts back to this runner. | ||
| .env("SIMARD_DISTILL_PASS_ID", pass_id) |
There was a problem hiding this comment.
HIGH — pass_id never reaches the agent, so the real-path fact count is always 0.
invoke_recipe exports SIMARD_DISTILL_PASS_ID here and also passes -c pass_id=…, but nothing consumes either:
simard memory remember(run_remember_fact) reads the pass id only from the--pass-idflag — it never readsSIMARD_DISTILL_PASS_ID.distill-episodes.yamlnever declarespass_idincontext, never references{{pass_id}}, and never tells the agent to pass--pass-id.
So every agent-issued remember runs with an empty pass_id; the server no-ops the ledger for an empty id (ledger_record_stored early-returns), and run_agentic → drain_pass_ledger(pass_id) returns 0. Net effect on the production subprocess path: facts are stored correctly, but DistillCommit.facts == 0 always, so DistillReport.fact_count, the simard.distill.facts counter, the distill_success_rate fact_count, and reduction_pct (always 100%) are all wrong. This defeats acceptance criterion #9 ("fact count sourced from successful remember writes").
Every test passes --pass-id explicitly (see tests_gated_write_2679.rs, bin_simard_memory_remember_cli.rs), which is exactly why the suite is green and misses this.
Suggested fix (matches the already-exported env var): in run_remember_fact/run_remember_procedure, fall back to the env var when the flag is absent:
let pass_id = parsed.pass_id.clone()
.or_else(|| std::env::var("SIMARD_DISTILL_PASS_ID").ok())
.unwrap_or_default();Add a test that drives the real path (no explicit --pass-id, env set) and asserts a non-zero drained count.
| Some(id) => format!("distill:{id}"), | ||
| None => "distill".to_string(), | ||
| }; | ||
| let pass_id = parsed.pass_id.clone().unwrap_or_default(); |
There was a problem hiding this comment.
HIGH (same root cause). pass_id defaults to empty whenever --pass-id is absent, which is the production case (the distiller agent is never instructed to pass it). Combined with the dead SIMARD_DISTILL_PASS_ID env export in invoke_recipe, the write ledger is never populated on the real path and the drained fact count is always 0. Prefer reading SIMARD_DISTILL_PASS_ID here as a fallback so the count works without relying on the LLM remembering a flag. Same applies to run_remember_procedure (line ~470).
| prerequisites, | ||
| source_episode_ids, | ||
| pass_id: _, | ||
| } => match memory.store_procedure_with_provenance( |
There was a problem hiding this comment.
MEDIUM — procedures bypass the write-boundary gate entirely. StoreProcedureProvenance stores unconditionally: no grounding check against source_episode_ids, no reliability score, no dedup (same in InProcessFactSink::commit_procedure and the remember-procedure CLI). The recipe prompt promises "a fact whose id is not in the batch is quarantined by the server," but a procedure citing fabricated/nonexistent episode ids is persisted with no such guard. If suppressing ungrounded records is a goal of #2679, this is an open door. At minimum ground the procedure (require ≥1 source_episode_id to resolve via episode_exists) or document the intentional asymmetry.
| } | ||
| Ok(self | ||
| .lock()? | ||
| .get_episodes(usize::MAX, true) |
There was a problem hiding this comment.
MEDIUM (perf) — episode_exists materializes the entire episode set on every call. get_episodes(usize::MAX, true) loads all episodes (including compressed) into a Vec, then linear-scans. In gated_fact_write this runs per cited episode-id per fact (source_episode_ids.iter().any(episode_exists)). For a pass writing N facts against a store of E episodes that is O(N·E) with a full materialization each call — a real cost on the hot write path for a long-lived store. Consider a targeted existence lookup, or have the server resolve/caches the batch episode-id set once per pass_id.
| // backend that ignores the filter. | ||
| let new_content = content.trim(); | ||
| let existing = memory | ||
| .search_facts(concept, 5, confidence) |
There was a problem hiding this comment.
LOW — dedup recall is capped at 5. search_facts(concept, 5, confidence) only inspects the top 5 same-concept facts, so an equal-or-stronger true duplicate ranked beyond position 5 is not detected and a near-duplicate can be stored. Dedup is best-effort, but the magic 5 silently bounds it; a brief comment on why 5 suffices (or keying the search more tightly than the concept label) would help.
rysweet
left a comment
There was a problem hiding this comment.
Step 17b — Comprehensive Code Review (#2679: delete the brittle distillation JSON parse gate)
Verdict: request changes (posted as a comment — gh is authed as the PR author, so the formal Request changes state is unavailable). The core architecture is sound and well-documented, but there is one HIGH functional defect that the current tests structurally cannot catch, plus a few medium/low items.
What's good
- The central goal is achieved. The distillation result path no longer parses any agent-authored document:
interpret_recipe_exitdecides success from exit status alone,parse_fail/ParseFailureis genuinely unreachable and removed, and a noisy/trailing-comma stdout can no longer discard a batch. Sharedextracthelpers are left intact for out-of-scope callers. - Gate consolidation is clean.
fact_reliability::commit_gated_factis the single shared score→threshold→dedup→persist core; both seams (IPCStoreFactGated, in-processInProcessFactSink) defer to it, so they cannot drift. Good doc comments and unit coverage. - Solid socket hardening.
MAX_FRAMEcap enforced before allocation, 0700 parent dir / 0600 socket perms, fail-closed grounding, per-field length caps, and the server never trusts the clientconfidencehint. - Retry-safety preserved: on runner failure no episode is marked, so the batch stays fully retry-eligible.
HIGH — real-path fact count is always 0 (pass_id never reaches the agent)
invoke_recipe exports SIMARD_DISTILL_PASS_ID and passes -c pass_id=…, but nothing consumes either:
simard memory rememberreads the pass id only from--pass-id; it never reads the env var.distill-episodes.yamlnever declarespass_id, never uses{{pass_id}}, and never tells the agent to pass--pass-id.
So every agent remember runs with an empty pass_id → ledger_record_stored no-ops → drain_pass_ledger returns 0 → DistillCommit.facts == 0 on the production subprocess path. Facts are still stored, but DistillReport.fact_count, the simard.distill.facts counter, the distill_success_rate fact_count, and reduction_pct (pinned at 100%) are all wrong — defeating acceptance criterion #9. Every test passes --pass-id explicitly, which is precisely why the suite is green and misses this. Fix: fall back to SIMARD_DISTILL_PASS_ID in the CLI when the flag is absent, and add a real-path test (env set, no flag) asserting a non-zero drained count. (Inline on distillation.rs:767 and operator_cli/memory.rs:366.)
MEDIUM
- Procedures bypass the gate entirely — no grounding, no reliability score, no dedup in
StoreProcedureProvenance,commit_procedure, or theremember-procedureCLI. A procedure citing fabricated episode ids is persisted unconditionally, unlike facts. Ground it or document the asymmetry. (Inline onserver.rs:234.) episode_existsperf — materializes the full episode set (get_episodes(usize::MAX, true)) and linear-scans on every call; invoked per cited id per fact on the hot write path → O(N·E) per pass. Use a targeted existence lookup or cache the batch's id set per pass. (Inline onlibrary_adapter.rs:1183.)
LOW
- Dedup recall capped at 5 (
search_facts(concept, 5, confidence)): an equal-or-stronger duplicate beyond the top 5 slips through. Best-effort, but worth a note or tighter keying. (Inline onfact_reliability.rs:224.) pass_ledgeronly shrinks on drain — an undrainedpass_idleaks for the daemon's lifetime. Bounded by pass count; add a size/TTL cap once the HIGH item is fixed (until then the ledger stays empty anyway).- Exit-code overload:
run_remember_factreturns3for both "no daemon" and a gated-write transport error, but the help text documents3as "no daemon" only. Minor diagnosability nit.
Checklist
- Code quality/standards: ✅ good structure, strong docs, no new stray
println!/eprintln!in hot paths (CLI stdout/stderr is the intentional contract). - Test coverage:
⚠️ adequate for the gate/parse-removal, but has a blind spot — no test drives the real recipe path without an explicit--pass-id, which hides the HIGH defect. Procedure grounding is untested. - No TODOs/stubs/swallowed exceptions: ✅ (best-effort telemetry paths intentionally swallow and are documented).
- No unimplemented functions: ✅.
- Logic correctness: ❌ the
pass_id/ledger wiring is broken end-to-end on the production path (HIGH). - Edge-case handling: ✅ empty/oversized/ungrounded facts handled;
⚠️ ungrounded procedures are not.
Recommendation: fix the HIGH pass_id wiring (+ real-path test) before merge; address procedure grounding and episode_exists cost as fast-follows if not in this PR.
Step 17c — Security Review (#2679: delete the brittle distillation JSON-parse gate)Read-only security specialist traced every attacker-controlled input source (IPC socket bytes, CLI argv, env vars, untrusted episode/agent content) to each sink (allocation, deserialization, subprocess spawn, filesystem, socket bind). Verdict: ✅ PASS — no exploitable vulnerability introduced by this PRNo CRITICAL/HIGH/MEDIUM finding meets the reporting bar: no injection, no unsafe deserialization, no panic reachable from network/CLI input, no privilege-boundary crossing, no secret leak. Verified safe (checks that passed)
Defense-in-depth notes (NOT vulnerabilities — all within the same-user trust boundary)
Security checklist: ✅ security requirements met · ✅ no new vulnerabilities · ✅ sensitive-data handling clean · ✅ authZ = same-user socket boundary (adequate) · ✅ no injection (argv-only spawn, size-bounded/error-mapped deserialization). |
Step 17d — Philosophy Guardian Review (#2679: delete the brittle distillation JSON parse gate)Evaluated against Compliance checklist
❌ Zero-BS violation (blocking) — silent degradation of the metrics path
The runner exports Required to reach compliance: wire a real consumer (env-var fallback in the CLI or Non-blocking philosophy notes
VerdictNEEDS CHANGES. Strong simplification win (this is the right architectural direction), but it does not pass the Zero-BS / error-visibility gate: the metrics path silently degrades and dead env-var plumbing looks functional while doing nothing. Fix the pass-id propagation (loud failure + real-path test) before merge; the remaining notes are non-blocking. Posted as a comment: |
The distill runner exported SIMARD_DISTILL_PASS_ID and passed a dead -c pass_id, but nothing consumed either: the remember CLI only read the --pass-id flag and the recipe never told the agent to pass it. Every agent remember write therefore resolved an empty pass id, the daemon's per-pass ledger no-op'd it, and drain_pass_ledger returned 0 — so fact_count / simard.distill.facts / reduction_pct always reported 0 / 100% even though facts were stored (silent metrics degradation, acceptance criterion #9). Only tests, which passed --pass-id explicitly, exercised the ledger, hiding the production breakage. Fix: the remember CLI now falls back to the SIMARD_DISTILL_PASS_ID env var (inherited by every remember subprocess the agent spawns) when no --pass-id flag is given. This is robust to LLM formatting — no flag the agent must echo. Removed the dead -c pass_id plumbing, shared the env var name as a const, documented the channel in the recipe, and added a real-path regression test (no flag, env set => ledger counts the fact) plus a resolve_pass_id precedence test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Step 17e — Blocking issue addressed (commit e375b1a)The HIGH / blocking finding from the code review (16b) and philosophy review (16d) — the Root cause (confirmed)The runner exported Fix (robust, not LLM-dependent)
Re-review verdicts
Verification
The MEDIUM/LOW items (procedure write-path asymmetry, |
Implements the remaining non-blocking review items from Step 17 (the sole blocking pass-id propagation bug was already fixed in e375b1a): - MEDIUM: gate procedure writes for grounding symmetry with facts. A procedure that cites source episodes none of which resolve now has its fabricated provenance rejected fail-closed, instead of storing dangling PROCEDURE_DERIVES_FROM edges. A procedure citing no sources is unchanged. - MEDIUM(perf): add batch `any_episode_exists` so the write-boundary gate materializes the episode set once per fact instead of once per cited id (O(cited·episodes) -> O(episodes)). Default trait impl falls back to per-id probing; LibraryCognitiveMemory overrides it. - LOW: name the dedup prior-scan window (DEDUP_PRIOR_SCAN_LIMIT). - LOW: name the remember/remember-procedure exit codes and document the intentionally shared code 3 (failed daemon round-trip) vs 4 (completed round-trip, gate rejected). - Def-in-depth: document that ungated StoreFact is the trusted in-process/same-user direct-write path, distinct from the gated distillation boundary. Adds regression tests: ungrounded-procedure rejection and multi-id batch grounding. fmt + clippy -D warnings clean; targeted suites green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Step 18b — Review feedback implemented (commit
|
| # | Item (severity) | Resolution |
|---|---|---|
| 1 | pass_id not propagated — HIGH/blocking | ✅ already fixed (e375b1af, verified) |
| 2 | Procedures bypass the write-boundary gate — MEDIUM | Added a grounding guard: a procedure that cites source episodes none of which resolve is now rejected fail-closed (no dangling PROCEDURE_DERIVES_FROM edges); a procedure citing nothing is unchanged. Documented the fact/procedure symmetry (procedures carry no reliability score). |
| 3 | episode_exists full-materializes per call — MEDIUM(perf) |
Added batch any_episode_exists: the gate now materializes the episode set once per fact instead of once per cited id (O(cited·episodes) → O(episodes)). Default trait impl falls back to per-id probing; LibraryCognitiveMemory overrides for the hot path. |
| 4 | Dedup recall capped at magic 5 — LOW |
Extracted DEDUP_PRIOR_SCAN_LIMIT with a justification comment. |
| 5 | Exit code 3 overloaded/unnamed — LOW |
Named EXIT_STORED/EXIT_USAGE/EXIT_NO_DAEMON/EXIT_QUARANTINED; documented that 3 is intentionally shared for any failed daemon round-trip, distinct from 4 (completed round-trip, gate rejected). Keeps the documented 0/2/3/4 contract stable. |
| 6 | Un-gated StoreFact still dispatchable — def-in-depth |
store_fact is a broadly-used general API (manual writes, imports, tests), so removing it would break legitimate callers. Documented it as the trusted in-process/same-user direct-write path, distinct from the gated distillation boundary. |
Security review (17c) passed with no required changes.
Tests: added regression coverage — ungrounded-procedure rejection and
multi-id batch grounding. cargo fmt, cargo clippy --all-targets -D warnings,
and the pre-push release test subset (cognitive_memory|bootstrap|memory_ipc|memory_consolidation)
are all green.
…actiondeserialization
📊 Coverage Summary
Coverage data from CI run. Test files matching |
Summary
Concise workflow-generated PR for Cargo.lock.
Issue
Closes #2679
Changed files
Diff stat
Behavior
Implemented behavior through these branch commits:
8e18de7 feat: Delete the brittle JSON extraction/deserialization gate from Simard's di
ea0a652 perf(distill): trim redundant work on the per-fact write-boundary gate
6207b8b refactor(distill): consolidate the write-boundary gate into one shared function
9735567 wip: checkpoint after implementation (steps 7-8)
Validation
ℹ NODE_OPTIONS=--max-old-space-size=32768 (saved preference). To change: /home/azureuser/.amplihack/config
I'll start by understanding the current state of the repository and the changes made.
● Get branch name, status, and recent commits (shell)
│ cd /home/azureuser/src/Simard/worktrees/feat/issue-2679-delete-the-brittle-json-extractiondeseri
│ alization && git branch --show-current && echo "---STATUS---" && git status && echo "---LOG---"
│ && git --no-pager log --oneline -15
└ 24 lines…
Let me look at the changes made in the 3 commits ahead of origin.
● Show file-level diff stats against origin (shell)
Risk
No high-risk subsystem pattern detected from changed paths.
Checklist
This PR was created as a draft for review before merging.
Step 16b: Outside-In Testing Results
Tested the PR branch from the user/consumer boundary as an external user would drive it.
Detected toolchain
Cargo.tomlat repo root, cratesimardv0.28.0).simardbinary, specifically the newsimard memory remember/simard memory remember-procedurewrite tools the distiller agent drives during a semantic-handoff pass (SIMARD_MEMORY_SOCKET).amplihack-memorypinned via git reve005a596…(memory-engine logic lives inamplihack-memory-lib, per the G2 constraint — no new fork depth insrc/memory_consolidation).Chosen strategy
Per the
qa-teamskill's repo-type detection, a Rust CLI repo usescargo testas the outside-in harness (no gadugi framework required). Ran one simple happy-path scenario at the true process boundary (spawned binary) plus several edge/integration scenarios, including the headline #2679 regression (noisy/trailing-comma agent stdout must no longer be able to fail the pipeline because it is no longer parsed).Scenarios
cargo test --test bin_simard_memory_remember_cli remember_cli_stores_grounded_fact_end_to_endstored concept=bug-pattern; fact present viasearch_facts{"facts":[…],}on stdout is SUCCESS (stdout is never parsed)cargo test --lib sec_t6_noisy_trailing_comma_stdout_is_not_parsed_and_does_not_failinterpret_recipe_exitreturnsOkcargo test --lib sec_t6_empty_stdout_on_exit_zero_is_success_not_parse_failureinterpret_recipe_exit(exit 0, "")→Okparse_failclass)cargo test --lib nonzero_exit_is_a_surfaced_terminal_failureRpcErrornamingdistill+exitedcargo test --test bin_simard_memory_remember_cli remember_cli_quarantines_ungrounded_factquarantined,search_factsemptycargo test --test bin_simard_memory_remember_cli remember_cli_no_daemon_exits_3no reachable memory daemon--content→ usage error (exit 2), never reaches daemoncargo test --test bin_simard_memory_remember_cli remember_cli_missing_required_flag_exits_2missing required --contentcargo test --test bin_simard_memory_remember_cli remember_procedure_cli_stores_end_to_endstored name=fix-and-verifyBroader affected-module sweep (single command):
Process-boundary suite:
Constraint verification
extract_json_payload/balanced_objects/serde_json::from_strgate remains in the distillation result path — the trailing-comma/noisy-stdout failure mode is now structurally impossible (there is no parse to fail). Remainingextract_json_payloadcallers are only insrc/ooda_brain/recipe_brain.rs(verdict paths), filed as follow-ons.parse_failfailure class is no longer reachable; the only result-path failure is a non-zero recipe exit.DistillRecipeRunner::runstub keeps working (defaultrun_agenticcommits its facts through the in-process gated sink).println!/eprintln!in prod hot paths — the CLI's stdout/stderr are its intentional, test-asserted user contract.Bridgeidentifier introduced —bridgeis a pre-existingSimardErrorfield (used acrossrpc.rs,gym_client.rs, etc.); only new usages were added, tracked by follow-on Pre-existing 'bridge' identifier leaks into the new 'simard memory remember' error output (#2679 follow-on) #2735.bridgeidentifier leak in remember error output).Fix count
0 — all 8 outside-in scenarios (and the full 54-test affected-module sweep) passed on the first run; no diagnose/fix/commit iterations were required.