diff --git a/CHANGELOG.md b/CHANGELOG.md index ef5c67f..9899e15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,18 @@ # Changelog +## 0.4.0a0 — unreleased + +- Complete bounded deterministic skeptic discovery for verifier gaps, disagreements, + abstentions, missing counterfactuals, learned-provider disagreement, and verifier + health holes. Candidates remain proposal-only and carry visible evidence lineage. +- Add source-preserving distillation groups, lineage-bound principles and strategies, + explicit negative memory, frozen transfer predictions/evaluations, study-arm contracts, + deterministic ablations, retrieval, and calibration metrics. +- Add fail-closed development/transfer visibility access and a CPU transition-frequency + learned-provider artifact with deterministic serialization, reload, OOD abstention, and + diagnostic-only observations. +- Add the no-network `mnel distill-reference` study and distillation record schema. + ## 0.3.0a0 — unreleased - Add bounded binary views and an identity-keyed shared store for transition, pair, diff --git a/README.md b/README.md index eb9c637..ebcda9f 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ experience, negative memory, causal attribution, transfer-gated principles, reus strategies, and append-only candidate lineage rather than relying exclusively on conventional neural-weight training. -> **Current status:** functional `0.3.0a0` iteration. The repository now includes a +> **Current status:** functional `0.4.0a0` study iteration. The repository now includes a > backend-neutral accelerator placement policy, optional Torch/Accelerate adapter, > process-local persistent Rust host, reusable identity-bound snapshots, bounded and > normalized diagnostic results, failure quarantine, an executable Rust HMM baseline, @@ -45,7 +45,10 @@ conventional neural-weight training. > contracts, an executable local-harness/worktree path, and a validated Rust v1 dynamic > provider loader, and an executable bounded Forge-oriented diagnostic lifecycle with > compact snapshot views, a verifier registry, reference probes, witnesses, mutations, -> comparison, health, coverage, and omitted-question candidates. It still does not provide process isolation, +> comparison, health, coverage, and deterministic skeptic candidates. It also includes +> source-preserving distillation groups, lineage-bound strategies, negative-memory +> retrieval, equal-budget study controls, retrieval/calibration metrics, and a tiny +> reloadable CPU transition-frequency diagnostic provider. It still does not provide process isolation, > unattended model execution, distributed scheduling, protected final custody, formal > MNCS/MNCDS conformance, or automatic RAVEL promotion. @@ -115,6 +118,14 @@ copy their authority or silently create substitute implementations. quarantine state, coverage records, learned-provider observation normalization, and proposal-only omitted-question candidates; - deterministic reference workflow, JSON schemas, mutation-oriented tests, and CI. +- bounded skeptic-driven omitted-question discovery with evidence lineage, duplicate + suppression, coverage/health/disagreement gaps, and proposal-only authority; +- source-preserving distillation groups, provisional principles/strategies, transfer + prediction freezing, negative-memory-aware retrieval, deterministic study arms and + ablations, and retrieval/calibration metrics; +- `mnel distill-reference`, a no-network held-out study that writes an evidence ledger + and exercises controls, hidden-transfer access, strategy transfer, and a tiny + reloadable transition-frequency learned provider. Its observations remain diagnostic. ## Install @@ -188,6 +199,18 @@ candidate: mnel forge-reference --workspace build/forge-reference ``` +Run the deterministic 0.4 distillation study: + +```bash +mnel distill-reference --workspace build/distill-reference +mnel ledger verify build/distill-reference/distill-evidence.jsonl +``` + +The study preserves development records, prevents development code from reading hidden +or future-final records, freezes the transfer prediction before held-out evaluation, +and reports measurements rather than a success verdict. Its reference feature grouping +is an explicit bounded baseline, not semantic understanding. + Verify and summarize the resulting ledger: ```bash diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 020d559..fa6dfc8 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -114,6 +114,30 @@ and inconclusive experience. Records are content-identified and chained in order Learned observations retain declaration, weight, feature, query, snapshot, calibration, and limitation identities before they are eligible for later study. +### Verified distillation and study plane + +The 0.4 data plane makes reuse explicit without replacing evidence: + +```text +development-visible episodes + -> attribution-linked source-preserving group + -> provisional principle/strategy with counterexamples + -> negative-memory-aware retrieval + -> frozen transfer prediction + -> hidden transfer evidence + -> retained/quarantined/rejected strategy lineage +``` + +`mnel.distillation` exposes explicit development and transfer access views. Development +study code cannot retrieve `TRANSFER_HIDDEN` or `FUTURE_FINAL` records, and a strategy +cannot be repaired from its own hidden-transfer result. Groups, principles, strategies, +and negative memory retain source identities; heuristic grouping is labeled as a +reference feature baseline rather than semantic truth. Retrieval returns record classes +and match reasons separately, while calibration and transfer status remain measurements. +Study arms and ablations carry equal-budget declarations and deterministic identities. +The `distill-reference` command is a synthetic, no-network study surface, not an +evaluator or a substitute for Forge, Fabric, MNCS, MNCDS, or RAVEL. + ### Distillation plane VED proposes compact principles and strategies while preserving source lineage, diff --git a/docs/INTEGRATIONS.md b/docs/INTEGRATIONS.md index 17a7bb9..e9eabe2 100644 --- a/docs/INTEGRATIONS.md +++ b/docs/INTEGRATIONS.md @@ -37,6 +37,13 @@ The external `mncs-forge-mcp` checkout is optional. The MNEL adapter contract is identity-bound and provider-neutral; no developer-local Forge path is a runtime dependency, and no hidden network or model service is invoked by the reference study. +The 0.4 distillation study consumes the same identified diagnostic plane but remains an +MNEL-side research harness. Its groups, strategies, retrieval results, learned-provider +observations, and transfer records are append-only measurements. It does not turn a +retrieval hit, calibrated score, witness, or synthetic transfer result into Forge/MNCS +authority. A future external Forge adapter must supply its own identified evaluator and +keep hidden-transfer access outside development study code. + ## MNCS Fabric Fabric distributes identified experiment bundles, captures node capabilities, and diff --git a/docs/LEARNED_PROVIDER_RUNTIME.md b/docs/LEARNED_PROVIDER_RUNTIME.md index 7bcffe4..f5a2381 100644 --- a/docs/LEARNED_PROVIDER_RUNTIME.md +++ b/docs/LEARNED_PROVIDER_RUNTIME.md @@ -163,6 +163,15 @@ the same identified payload can be reused by a deterministic micro-verifier and learned provider without reparsing ad hoc JSON. Composite snapshots reference component identities rather than duplicating their payloads. +The 0.4 reference study also demonstrates the control-plane side of a small learned +provider in `mnel.reference_provider`: a transition-frequency artifact is trained only +from a development access view, binds its dataset/record/feature/training/calibration +identities, serializes with an artifact digest, reloads deterministically, and emits a +bounded observation with abstention/OOD flags. This is an `external-experimental` +diagnostic provider. It is not a verifier and is not exported through ABI v1; native +export remains a later task once the artifact contract has a measured compatible runtime +boundary. + ## Native-language exceptions A non-Rust provider may enter `native-trusted` only when its manifest includes: diff --git a/docs/LEARNING_MODEL.md b/docs/LEARNING_MODEL.md index 6eea90f..6c9768a 100644 --- a/docs/LEARNING_MODEL.md +++ b/docs/LEARNING_MODEL.md @@ -67,3 +67,19 @@ A credible MNEL study should include: - hidden transfer evaluation. A better final candidate alone does not prove the learning process caused the gain. + +## Executable 0.4 foundation + +The current deterministic study path preserves raw development episodes and builds +identity-bound feature groups, attribution-linked provisional strategies, and explicit +negative memory. Retrieval keeps episodic, causal, semantic, procedural, and negative +records as separate classes; a negative-memory conflict can demote a strategy without +deleting its positive lineage. Transfer predictions are frozen before hidden evaluation, +and hidden results cannot repair the same candidate. + +`mnel distill-reference` exercises A0, A4, and A5 declarations plus aggregate-only, +shuffled-attribution, random, success-memory, negative-memory, fixed-policy, +equal-budget, and hidden-transfer controls. It computes retrieval and calibration +measurements and trains a tiny transition-frequency diagnostic provider that serializes +and reloads deterministically. These are measurements from a bounded synthetic fixture, +not evidence of general learning improvement or evaluator authority. diff --git a/docs/RESEARCH_PROTOCOL.md b/docs/RESEARCH_PROTOCOL.md index 10a2601..604da4e 100644 --- a/docs/RESEARCH_PROTOCOL.md +++ b/docs/RESEARCH_PROTOCOL.md @@ -35,3 +35,19 @@ development environments, selection material, and hidden transfer environment. The study must preserve every failed candidate and must not use transfer or final observations for same-candidate repair. + +## Machine-readable study contracts + +MNEL now represents each arm as an identity-bound `mnel-study-arm/0.4` record with +allowed/forbidden information, retrieval and memory modes, attribution/strategy +availability, recursion allowance, and an equal operation/wall-time/candidate budget. +Control transformations are separate `mnel-ablation-spec/0.4` records. Shuffled +attribution uses a recorded seed and preserves the original attribution records; memory +ablations change only the eligible view. The reference study can therefore reproduce +the same control identities without deleting source evidence or reading hidden transfer +outcomes during development. + +The current 0.4 reference study is deliberately synthetic and deterministic. Its +transition-frequency provider is a diagnostic observation source with explicit artifact, +training dataset, feature extractor, calibration, and reload identities. It is not a +verifier, evaluator, or promotion mechanism. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 4413f22..bdcc3b1 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -45,7 +45,9 @@ - **Implemented:** bounded counterfactual and registered mutation probe support; - **Implemented:** independent-probe comparison preserving agreement, disagreement, and incomplete coverage; - **Implemented:** verifier health, quarantine, and coverage records; -- **Started:** deterministic skeptic-driven omitted-question candidate discovery; +- **Implemented:** deterministic skeptic-driven omitted-question candidate discovery with + bounded, deduplicated, lineage-checked candidates for coverage holes, disagreements, + abstentions, missing mutation counterparts, and verifier health gaps; - **Implemented:** identity-bound transition, tabular, pair, trace, graph, and composite diagnostic snapshots with immutable compact binary payloads, producer/source/dependency/extractor identities, and deterministic content identities suitable for deterministic probes and learned @@ -55,14 +57,23 @@ ## 0.4 — verified distillation and learned-provider studies -- semantic clustering without source deletion; -- attribution and transfer workflows; -- success- and negative-memory ablations; -- shuffled-attribution and aggregate-only controls; -- strategy retrieval and calibration metrics; -- train and calibrate the initial heterogeneous learned-provider portfolio; +- **Implemented:** source-preserving deterministic reference grouping with explicit + extractor/method identities and limitations; +- **Implemented:** attribution-linked provisional principles and strategies, negative + memory, frozen transfer predictions, held-out transfer evidence, and same-candidate + hidden-repair rejection; +- **Implemented:** success- and negative-memory, shuffled-attribution, aggregate-only, + random, fixed-policy, equal-budget, and hidden-transfer control specifications; +- **Implemented:** explicit class-preserving retrieval with negative-memory demotion and + precision/recall/hit-rate/reuse/diversity metrics plus bounded calibration metrics; +- **Implemented:** deterministic `mnel distill-reference` study and a tiny reloadable CPU + transition-frequency learned provider with OOD abstention and diagnostic-only output; +- **Started:** heterogeneous learned-provider portfolio training and calibration; - export Python-trained providers into the versioned native runtime boundary; -- compare every learned provider against deterministic and classical baselines; +- **Started:** compare learned providers against deterministic and classical baselines; + the reference transition study records deterministic and heuristic controls; the + existing Rust HMM baseline is not input-compatible with this compact fixture and is + retained as a separate native runtime baseline; - random, heuristic, single-provider, and diversity-routed controls; - correlated-error, disagreement, abstention, and out-of-distribution studies; - useful confirmed probes per operation, latency, memory, energy, and cold-start metrics; diff --git a/docs/THREAT_MODEL.md b/docs/THREAT_MODEL.md index 999b407..87d6c26 100644 --- a/docs/THREAT_MODEL.md +++ b/docs/THREAT_MODEL.md @@ -71,6 +71,16 @@ callbacks and in-place authoritative snapshot mutation are not accepted. Learned provider observations and verifier witnesses remain distinct diagnostic records, and neither can authorize conformance or promotion. +The 0.4 study layer adds an explicit visibility gate: development access rejects hidden +transfer and future-final records, clustering/training/retrieval operate on an eligible +view, and transfer prediction is frozen before a separate transfer-evaluator view can +read the held-out result. Same-candidate repair from that result is rejected. Shuffled +attribution and memory ablations create new study identities while preserving their +source records. Deterministic feature grouping is bounded and explicitly labeled as a +heuristic limitation, so it cannot masquerade as semantic understanding. Distillation, +retrieval, calibration, and learned-provider observations remain proposal or diagnostic +records; none includes evaluator verdict authority. + ### Apparent independence Multiple local machines run the same operator-controlled stack. This is replication, diff --git a/schemas/mnel-distillation.schema.json b/schemas/mnel-distillation.schema.json new file mode 100644 index 0000000..1ca1231 --- /dev/null +++ b/schemas/mnel-distillation.schema.json @@ -0,0 +1,345 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/epi13/Machine-Native-Experimental-Learning/schemas/mnel-distillation.schema.json", + "title": "MNEL evidence-preserving distillation and study records", + "$defs": { + "nonEmpty": {"type": "string", "minLength": 1}, + "identity": {"type": "string", "minLength": 1}, + "object": {"type": "object"}, + "sourceIds": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/identity"}}, + "diagnosticAuthority": {"const": "diagnostic-only"}, + "proposalAuthority": {"const": "proposal-only"}, + "sourceRecord": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "record_type", "record_identity", "payload", "visibility", "epoch", "authority", "semantics"], + "properties": { + "schema": {"const": "mnel-study-source-record/0.4"}, + "record_type": {"$ref": "#/$defs/nonEmpty"}, + "record_identity": {"$ref": "#/$defs/identity"}, + "payload": {"type": "object"}, + "visibility": {"enum": ["development-visible", "selection-observed-not-repairable", "transfer-hidden", "future-final-inaccessible"]}, + "epoch": {"type": "integer", "minimum": 0}, + "authority": {"const": "evidence-record"}, + "semantics": {"const": "source-evidence; not-a-verdict"} + } + }, + "group": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "source_record_ids", "feature_extractor_identity", "clustering_method_identity", "parameters", "representative_record_ids", "scope", "creation_epoch", "limitations", "group_identity", "authority", "semantics"], + "properties": { + "schema": {"const": "mnel-semantic-group/0.4"}, + "source_record_ids": {"$ref": "#/$defs/sourceIds"}, + "feature_extractor_identity": {"$ref": "#/$defs/identity"}, + "clustering_method_identity": {"$ref": "#/$defs/identity"}, + "parameters": {"type": "object"}, + "representative_record_ids": {"$ref": "#/$defs/sourceIds"}, + "scope": {"type": "object"}, + "creation_epoch": {"type": "integer", "minimum": 0}, + "limitations": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/nonEmpty"}}, + "group_identity": {"$ref": "#/$defs/identity"}, + "authority": {"$ref": "#/$defs/diagnosticAuthority"}, + "semantics": {"const": "source-preserving-retrieval-group; not-a-verdict"} + } + }, + "principle": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "statement", "scope", "attribution_ids", "source_record_ids", "counterexample_record_ids", "falsifier", "transfer_status", "maturity", "limitations", "principle_identity", "authority", "semantics"], + "properties": { + "schema": {"const": "mnel-distilled-principle/0.4"}, + "statement": {"$ref": "#/$defs/nonEmpty"}, + "scope": {"type": "object"}, + "attribution_ids": {"$ref": "#/$defs/sourceIds"}, + "source_record_ids": {"$ref": "#/$defs/sourceIds"}, + "counterexample_record_ids": {"type": "array", "items": {"$ref": "#/$defs/identity"}}, + "falsifier": {"$ref": "#/$defs/nonEmpty"}, + "transfer_status": {"enum": ["untested", "failed", "partial", "supported"]}, + "maturity": {"$ref": "#/$defs/nonEmpty"}, + "limitations": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/nonEmpty"}}, + "principle_identity": {"$ref": "#/$defs/identity"}, + "authority": {"$ref": "#/$defs/proposalAuthority"}, + "semantics": {"const": "provisional-distillation; not-a-verdict"} + } + }, + "negativeMemory": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "memory_type", "statement", "source_record_ids", "prohibited_contexts", "reconsideration_condition", "scope", "outcome_identity", "memory_identity", "authority", "semantics"], + "properties": { + "schema": {"const": "mnel-negative-memory/0.4"}, + "memory_type": {"$ref": "#/$defs/nonEmpty"}, + "statement": {"$ref": "#/$defs/nonEmpty"}, + "source_record_ids": {"$ref": "#/$defs/sourceIds"}, + "prohibited_contexts": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/nonEmpty"}}, + "reconsideration_condition": {"$ref": "#/$defs/nonEmpty"}, + "scope": {"type": "object"}, + "outcome_identity": {"$ref": "#/$defs/identity"}, + "memory_identity": {"$ref": "#/$defs/identity"}, + "authority": {"$ref": "#/$defs/diagnosticAuthority"}, + "semantics": {"const": "negative-evidence; not-a-verdict"} + } + }, + "strategy": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "trigger_conditions", "preconditions", "expected_effect", "known_failure_modes", "negative_memory_ids", "counterexample_record_ids", "causal_attribution_ids", "supporting_source_record_ids", "transfer_evidence_ids", "scope", "calibration_metadata", "rollback_lineage", "transfer_status", "strategy_identity", "authority", "semantics"], + "properties": { + "schema": {"const": "mnel-distilled-strategy/0.4"}, + "trigger_conditions": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/nonEmpty"}}, + "preconditions": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/nonEmpty"}}, + "expected_effect": {"type": "object"}, + "known_failure_modes": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/nonEmpty"}}, + "negative_memory_ids": {"type": "array", "items": {"$ref": "#/$defs/identity"}}, + "counterexample_record_ids": {"type": "array", "items": {"$ref": "#/$defs/identity"}}, + "causal_attribution_ids": {"$ref": "#/$defs/sourceIds"}, + "supporting_source_record_ids": {"$ref": "#/$defs/sourceIds"}, + "transfer_evidence_ids": {"type": "array", "items": {"$ref": "#/$defs/identity"}}, + "scope": {"type": "object"}, + "calibration_metadata": {"type": "object"}, + "rollback_lineage": {"type": "array", "items": {"$ref": "#/$defs/identity"}}, + "transfer_status": {"enum": ["untested", "failed", "partial", "supported"]}, + "strategy_identity": {"$ref": "#/$defs/identity"}, + "authority": {"$ref": "#/$defs/proposalAuthority"}, + "semantics": {"const": "reusable-proposal; not-a-verdict"} + } + }, + "transferPrediction": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "strategy_identity", "source_development_identities", "transfer_environment_identity", "predicted_effect", "prediction_identity", "authority", "semantics"], + "properties": { + "schema": {"const": "mnel-transfer-prediction/0.4"}, + "strategy_identity": {"$ref": "#/$defs/identity"}, + "source_development_identities": {"$ref": "#/$defs/sourceIds"}, + "transfer_environment_identity": {"$ref": "#/$defs/identity"}, + "predicted_effect": {"type": "object"}, + "prediction_identity": {"$ref": "#/$defs/identity"}, + "authority": {"$ref": "#/$defs/proposalAuthority"}, + "semantics": {"const": "frozen-before-transfer-observation; not-a-verdict"} + } + }, + "transferEvaluation": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "prediction_identity", "observed_outcome_identity", "evaluator_evidence_identity", "status", "rollback_lineage", "evaluation_identity", "authority", "semantics"], + "properties": { + "schema": {"const": "mnel-transfer-evaluation/0.4"}, + "prediction_identity": {"$ref": "#/$defs/identity"}, + "observed_outcome_identity": {"$ref": "#/$defs/identity"}, + "evaluator_evidence_identity": {"$ref": "#/$defs/identity"}, + "status": {"enum": ["untested", "failed", "partial", "supported"]}, + "rollback_lineage": {"type": "array", "items": {"$ref": "#/$defs/identity"}}, + "evaluation_identity": {"$ref": "#/$defs/identity"}, + "authority": {"$ref": "#/$defs/diagnosticAuthority"}, + "semantics": {"const": "transfer-evidence; not-a-verdict"} + } + }, + "studySpec": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "study_id", "development_dataset_identity", "hidden_transfer_dataset_identity", "arms", "ablations", "equal_budget", "study_identity", "authority", "semantics"], + "properties": { + "schema": {"const": "mnel-distillation-study-spec/0.4"}, + "study_id": {"$ref": "#/$defs/nonEmpty"}, + "development_dataset_identity": {"$ref": "#/$defs/identity"}, + "hidden_transfer_dataset_identity": {"$ref": "#/$defs/identity"}, + "arms": {"type": "array", "minItems": 1, "items": {"type": "object"}}, + "ablations": {"type": "array", "items": {"type": "object"}}, + "equal_budget": {"type": "object"}, + "study_identity": {"$ref": "#/$defs/identity"}, + "authority": {"$ref": "#/$defs/proposalAuthority"}, + "semantics": {"const": "study-specification; not-a-verdict"} + } + }, + "studyArm": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "arm_id", "kind", "allowed_information", "forbidden_information", "retrieval_mode", "memory_availability", "attribution_available", "strategy_available", "recursion_allowance", "budget", "authority", "semantics", "arm_identity"], + "properties": { + "schema": {"const": "mnel-study-arm/0.4"}, + "arm_id": {"$ref": "#/$defs/nonEmpty"}, + "kind": {"enum": ["A0-random-bounded", "A1-aggregate-only", "A2-complete-episodes", "A3-competing-hypotheses-probes", "A4-attribution-principles", "A5-transfer-gated", "A6-bounded-policy-recursion"]}, + "allowed_information": {"type": "array", "items": {"$ref": "#/$defs/nonEmpty"}}, + "forbidden_information": {"type": "array", "items": {"$ref": "#/$defs/nonEmpty"}}, + "retrieval_mode": {"$ref": "#/$defs/nonEmpty"}, + "memory_availability": {"type": "array", "items": {"$ref": "#/$defs/nonEmpty"}}, + "attribution_available": {"type": "boolean"}, + "strategy_available": {"type": "boolean"}, + "recursion_allowance": {"type": "integer", "minimum": 0}, + "budget": {"type": "object", "additionalProperties": {"type": "integer", "minimum": 1}}, + "authority": {"$ref": "#/$defs/proposalAuthority"}, + "semantics": {"const": "study-arm-contract; not-a-verdict"}, + "arm_identity": {"$ref": "#/$defs/identity"} + } + }, + "ablation": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "ablation_id", "kind", "source_arm_identity", "seed", "parameters", "authority", "semantics", "ablation_identity"], + "properties": { + "schema": {"const": "mnel-ablation-spec/0.4"}, + "ablation_id": {"$ref": "#/$defs/nonEmpty"}, + "kind": {"enum": ["aggregate-only", "shuffled-attribution", "random-proposal", "success-memory-ablation", "negative-memory-ablation", "fixed-policy", "equal-budget", "hidden-transfer"]}, + "source_arm_identity": {"$ref": "#/$defs/identity"}, + "seed": {"type": "integer", "minimum": 0}, + "parameters": {"type": "object"}, + "authority": {"$ref": "#/$defs/proposalAuthority"}, + "semantics": {"const": "controlled-study-transformation; not-a-verdict"}, + "ablation_identity": {"$ref": "#/$defs/identity"} + } + }, + "retrievalResult": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "query_identity", "hits", "negative_conflict_ids", "method_identity", "authority", "semantics", "result_identity"], + "properties": { + "schema": {"const": "mnel-retrieval-result/0.4"}, + "query_identity": {"$ref": "#/$defs/identity"}, + "hits": {"type": "array", "items": {"type": "object"}}, + "negative_conflict_ids": {"type": "array", "items": {"$ref": "#/$defs/identity"}}, + "method_identity": {"$ref": "#/$defs/identity"}, + "authority": {"$ref": "#/$defs/diagnosticAuthority"}, + "semantics": {"const": "heuristic-retrieval; not-a-verdict"}, + "result_identity": {"$ref": "#/$defs/identity"} + } + }, + "retrievalMetrics": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "query_identity", "k", "relevant_count", "precision_at_k", "recall_at_k", "hit_rate", "reciprocal_rank", "duplicate_retrieval_rate", "evidence_reuse_depth", "source_diversity", "positive_negative_balance", "transfer_supported_strategy_rate", "unsupported_strategy_rate", "authority", "semantics", "metric_identity"], + "properties": { + "schema": {"const": "mnel-retrieval-metrics/0.4"}, + "query_identity": {"$ref": "#/$defs/identity"}, + "k": {"type": "integer", "minimum": 0}, + "relevant_count": {"type": "integer", "minimum": 0}, + "precision_at_k": {"type": "number"}, + "recall_at_k": {"type": "number"}, + "hit_rate": {"type": "number"}, + "reciprocal_rank": {"type": "number"}, + "duplicate_retrieval_rate": {"type": "number"}, + "evidence_reuse_depth": {"type": "integer", "minimum": 0}, + "source_diversity": {"type": "integer", "minimum": 0}, + "positive_negative_balance": {"type": "number"}, + "transfer_supported_strategy_rate": {"type": "number"}, + "unsupported_strategy_rate": {"type": "number"}, + "authority": {"$ref": "#/$defs/diagnosticAuthority"}, + "semantics": {"const": "retrieval-measurement; not-a-verdict"}, + "metric_identity": {"$ref": "#/$defs/identity"} + } + }, + "calibrationMetrics": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "dataset_identity", "model_identity", "calibration_method_identity", "count", "brier_score", "log_loss", "expected_calibration_error", "coverage", "abstention_rate", "out_of_distribution_rate", "authority", "semantics", "metric_identity"], + "properties": { + "schema": {"const": "mnel-calibration-metrics/0.4"}, + "dataset_identity": {"$ref": "#/$defs/identity"}, + "model_identity": {"$ref": "#/$defs/identity"}, + "calibration_method_identity": {"$ref": "#/$defs/identity"}, + "count": {"type": "integer", "minimum": 1}, + "brier_score": {"type": "number"}, + "log_loss": {"type": "number"}, + "expected_calibration_error": {"type": "number"}, + "coverage": {"type": "number"}, + "abstention_rate": {"type": "number"}, + "out_of_distribution_rate": {"type": "number"}, + "authority": {"$ref": "#/$defs/diagnosticAuthority"}, + "semantics": {"const": "calibration-measurement; not-a-verdict"}, + "metric_identity": {"$ref": "#/$defs/identity"} + } + }, + "providerObservation": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "record_type", "provider_id", "model_identity", "snapshot_identity", "score", "abstained", "out_of_distribution", "calibration_identity", "authority", "semantics", "observation_identity"], + "properties": { + "schema": {"const": "mnel-learned-provider-observation/0.4"}, + "record_type": {"const": "learned-provider-observation"}, + "provider_id": {"$ref": "#/$defs/nonEmpty"}, + "model_identity": {"$ref": "#/$defs/identity"}, + "snapshot_identity": {"$ref": "#/$defs/identity"}, + "score": {"type": "number", "minimum": 0, "maximum": 1}, + "abstained": {"type": "boolean"}, + "out_of_distribution": {"type": "boolean"}, + "calibration_identity": {"$ref": "#/$defs/identity"}, + "authority": {"$ref": "#/$defs/diagnosticAuthority"}, + "semantics": {"const": "learned-diagnostic-observation; not-a-verdict"}, + "observation_identity": {"$ref": "#/$defs/identity"} + } + }, + "providerArtifact": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "provider_id", "training_dataset_identity", "training_record_ids", "feature_extractor_identity", "training_code_identity", "calibration_identity", "transition_counts", "total_count", "model_identity", "authority", "semantics"], + "properties": { + "schema": {"const": "mnel-learned-provider-artifact/0.4"}, + "provider_id": {"$ref": "#/$defs/nonEmpty"}, + "training_dataset_identity": {"$ref": "#/$defs/identity"}, + "training_record_ids": {"$ref": "#/$defs/sourceIds"}, + "feature_extractor_identity": {"$ref": "#/$defs/identity"}, + "training_code_identity": {"$ref": "#/$defs/identity"}, + "calibration_identity": {"$ref": "#/$defs/identity"}, + "transition_counts": {"type": "object", "additionalProperties": {"type": "integer", "minimum": 1}}, + "total_count": {"type": "integer", "minimum": 1}, + "model_identity": {"$ref": "#/$defs/identity"}, + "authority": {"$ref": "#/$defs/diagnosticAuthority"}, + "semantics": {"const": "learned-provider-artifact; diagnostic-only; not-a-verdict"} + } + }, + "studyReport": { + "type": "object", + "additionalProperties": false, + "required": ["schema", "study_identity", "study_specification_identity", "development_dataset_identity", "hidden_transfer_dataset_identity", "group_count", "candidate_count", "retrieval_metrics", "calibration_metrics", "transfer_status", "transfer_prediction_frozen_before_evaluation", "negative_memory_conflicts", "negative_memory_demoted_strategy", "negative_memory_ablation_strategy_score", "shuffled_attribution_control_identity", "arm_identities", "ablation_identities", "source_record_count", "authority_violation_attempts", "limitations", "authority", "semantics"], + "properties": { + "schema": {"const": "mnel-distillation-study-report/0.4"}, + "study_identity": {"$ref": "#/$defs/identity"}, + "study_specification_identity": {"$ref": "#/$defs/identity"}, + "development_dataset_identity": {"$ref": "#/$defs/identity"}, + "hidden_transfer_dataset_identity": {"$ref": "#/$defs/identity"}, + "group_count": {"type": "integer", "minimum": 0}, + "candidate_count": {"type": "integer", "minimum": 0}, + "retrieval_metrics": {"type": "object"}, + "calibration_metrics": {"type": "object"}, + "transfer_status": {"enum": ["untested", "failed", "partial", "supported"]}, + "transfer_prediction_frozen_before_evaluation": {"const": true}, + "negative_memory_conflicts": {"type": "array", "items": {"$ref": "#/$defs/identity"}}, + "negative_memory_demoted_strategy": {"type": "boolean"}, + "negative_memory_ablation_strategy_score": {"type": ["number", "null"]}, + "shuffled_attribution_control_identity": {"$ref": "#/$defs/identity"}, + "arm_identities": {"$ref": "#/$defs/sourceIds"}, + "ablation_identities": {"$ref": "#/$defs/sourceIds"}, + "learned_provider_model_identity": {"$ref": "#/$defs/identity"}, + "learned_provider_artifact_identity": {"$ref": "#/$defs/identity"}, + "learned_provider_model_size_bytes": {"type": "integer", "minimum": 1}, + "learned_provider_reload_reproduced": {"type": "boolean"}, + "provider_baseline_comparison": {"type": "object"}, + "ledger": {"type": "object"}, + "source_record_count": {"type": "integer", "minimum": 0}, + "authority_violation_attempts": {"type": "integer", "minimum": 0}, + "limitations": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/nonEmpty"}}, + "authority": {"$ref": "#/$defs/diagnosticAuthority"}, + "semantics": {"const": "measurement-report; not-a-verdict"} + } + } + }, + "oneOf": [ + {"$ref": "#/$defs/sourceRecord"}, + {"$ref": "#/$defs/group"}, + {"$ref": "#/$defs/principle"}, + {"$ref": "#/$defs/negativeMemory"}, + {"$ref": "#/$defs/strategy"}, + {"$ref": "#/$defs/transferPrediction"}, + {"$ref": "#/$defs/transferEvaluation"}, + {"$ref": "#/$defs/studySpec"}, + {"$ref": "#/$defs/providerArtifact"}, + {"$ref": "#/$defs/studyArm"}, + {"$ref": "#/$defs/ablation"}, + {"$ref": "#/$defs/retrievalResult"}, + {"$ref": "#/$defs/retrievalMetrics"}, + {"$ref": "#/$defs/calibrationMetrics"}, + {"$ref": "#/$defs/providerObservation"}, + {"$ref": "#/$defs/studyReport"} + ] +} diff --git a/schemas/mnel-forge-lifecycle.schema.json b/schemas/mnel-forge-lifecycle.schema.json index 327d03b..03b2cf8 100644 --- a/schemas/mnel-forge-lifecycle.schema.json +++ b/schemas/mnel-forge-lifecycle.schema.json @@ -184,7 +184,7 @@ "questionCandidate": { "type": "object", "additionalProperties": false, - "required": ["schema", "subject_identity", "reason", "target_snapshot_type", "supporting_record_ids", "authority", "semantics", "candidate_identity"], + "required": ["schema", "subject_identity", "reason", "target_snapshot_type", "supporting_record_ids", "authority", "candidate_kind", "priority", "lineage", "semantics", "candidate_identity"], "properties": { "schema": {"const": "mnel-omitted-question-candidate/0.3"}, "subject_identity": {"type": "string", "minLength": 1}, @@ -192,6 +192,9 @@ "target_snapshot_type": {"type": ["string", "null"]}, "supporting_record_ids": {"type": "array", "items": {"type": "string"}}, "authority": {"const": "proposal-only"}, + "candidate_kind": {"type": "string", "minLength": 1}, + "priority": {"type": "integer", "minimum": 0, "maximum": 100}, + "lineage": {"type": "array", "items": {"type": "string", "minLength": 1}}, "semantics": {"const": "proposal-only; not-a-verdict"}, "candidate_identity": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"} } diff --git a/src/mnel/cli.py b/src/mnel/cli.py index 7a9f261..b258857 100644 --- a/src/mnel/cli.py +++ b/src/mnel/cli.py @@ -11,6 +11,7 @@ from . import __version__ from .core import EvidenceLedger, run_reference_study +from .distillation import run_reference_distill_study from .forge_lifecycle import run_reference_forge_study from .investigators import DEFAULT_ROLE_CONTRACTS from .learned_providers import ( @@ -63,6 +64,10 @@ def parser() -> argparse.ArgumentParser: "forge-reference", description="Run the deterministic MNEL Forge diagnostic lifecycle" ) forge_reference.add_argument("--workspace", default=None) + distill_reference = commands.add_parser( + "distill-reference", description="Run the deterministic MNEL distillation study" + ) + distill_reference.add_argument("--workspace", default=None) return root @@ -130,5 +135,8 @@ def main(argv: list[str] | None = None) -> int: if args.command == "forge-reference": print(json.dumps(run_reference_forge_study(args.workspace), indent=2, sort_keys=True)) return 0 + if args.command == "distill-reference": + print(json.dumps(run_reference_distill_study(args.workspace), indent=2, sort_keys=True)) + return 0 print(json.dumps(run_reference_study(args.workspace), indent=2, sort_keys=True)) return 0 diff --git a/src/mnel/distillation.py b/src/mnel/distillation.py new file mode 100644 index 0000000..58a8f18 --- /dev/null +++ b/src/mnel/distillation.py @@ -0,0 +1,1741 @@ +"""Evidence-preserving verified-experience distillation and study controls. + +This module compresses retrieval and reuse metadata while leaving source evidence in the +append-only ledger. It intentionally contains no evaluator implementation and never +turns a retrieval score, calibration result, or transfer result into a verdict. +""" + +from __future__ import annotations + +import math +import random +from dataclasses import dataclass +from enum import StrEnum +from pathlib import Path +from typing import Any, Sequence + +from .core import ( + Attribution, + EvidenceLedger, + TransferStatus, + Visibility, + canonical_digest, + canonical_json, +) + + +AUTHORITY_DIAGNOSTIC_ONLY = "diagnostic-only" +AUTHORITY_PROPOSAL_ONLY = "proposal-only" +SEMANTICS_NOT_A_VERDICT = "not-a-verdict" +MAX_RECORD_BYTES = 64 * 1024 +MAX_GROUPS = 64 +MAX_GROUP_RECORDS = 512 +MAX_RETRIEVAL_HITS = 128 + + +class DistillationError(ValueError): + pass + + +class VisibilityViolation(DistillationError): + pass + + +class StudyArmKind(StrEnum): + RANDOM = "A0-random-bounded" + AGGREGATE_ONLY = "A1-aggregate-only" + COMPLETE_EPISODES = "A2-complete-episodes" + COMPETING_PROBES = "A3-competing-hypotheses-probes" + ATTRIBUTION = "A4-attribution-principles" + TRANSFER_GATED = "A5-transfer-gated" + POLICY_RECURSION = "A6-bounded-policy-recursion" + + +class AblationKind(StrEnum): + AGGREGATE_ONLY = "aggregate-only" + SHUFFLED_ATTRIBUTION = "shuffled-attribution" + RANDOM_PROPOSAL = "random-proposal" + SUCCESS_MEMORY = "success-memory-ablation" + NEGATIVE_MEMORY = "negative-memory-ablation" + FIXED_POLICY = "fixed-policy" + EQUAL_BUDGET = "equal-budget" + HIDDEN_TRANSFER = "hidden-transfer" + + +def _nonempty(value: str, label: str) -> str: + if not isinstance(value, str) or not value.strip(): + raise DistillationError(f"{label} is required") + return value + + +def _bounded(value: dict[str, Any], label: str, limit: int = MAX_RECORD_BYTES) -> dict[str, Any]: + _reject_authority(value) + try: + encoded = canonical_json(value) + except (TypeError, ValueError) as error: + raise DistillationError(f"{label} is not canonical JSON") from error + if len(encoded) > limit: + raise DistillationError(f"{label} exceeds its byte ceiling") + return dict(value) + + +def _reject_authority(value: Any) -> None: + forbidden = { + "promotion_authorized", + "evaluator_eligible", + "evaluator_verdict", + "verdict", + "mncs_verdict", + "mncds_verdict", + "pass_fail", + "promotion", + "conformance", + "future_final", + "hidden_transfer_result", + "ravel_promotion", + } + if isinstance(value, dict): + for key, child in value.items(): + if key in forbidden: + raise DistillationError(f"distillation record contains forbidden field: {key}") + if key == "authority" and child not in { + AUTHORITY_DIAGNOSTIC_ONLY, + AUTHORITY_PROPOSAL_ONLY, + "evidence-record", + }: + raise DistillationError("distillation record attempted to expand authority") + _reject_authority(child) + elif isinstance(value, (list, tuple)): + for child in value: + _reject_authority(child) + + +def _identity_body(value: Any) -> str: + return canonical_digest(value) + + +@dataclass(frozen=True, slots=True) +class StudyRecord: + """A source record that may be indexed only through an explicit visibility view.""" + + record_type: str + payload: dict[str, Any] + visibility: Visibility + record_identity: str = "" + epoch: int = 0 + + def __post_init__(self) -> None: + _nonempty(self.record_type, "study record type") + if not isinstance(self.payload, dict): + raise DistillationError("study record payload must be an object") + _reject_authority(self.payload) + if not isinstance(self.visibility, Visibility) or self.epoch < 0: + raise DistillationError("study record visibility and epoch are invalid") + if len(canonical_json(self.payload)) > MAX_RECORD_BYTES: + raise DistillationError("study record payload is too large") + if self.record_identity and not isinstance(self.record_identity, str): + raise DistillationError("study record identity is invalid") + + @property + def identity(self) -> str: + return self.record_identity or _identity_body( + { + "record_type": self.record_type, + "payload": self.payload, + "visibility": self.visibility.value, + "epoch": self.epoch, + } + ) + + def to_dict(self) -> dict[str, Any]: + return { + "schema": "mnel-study-source-record/0.4", + "record_type": self.record_type, + "record_identity": self.identity, + "payload": dict(self.payload), + "visibility": self.visibility.value, + "epoch": self.epoch, + "authority": "evidence-record", + "semantics": "source-evidence; not-a-verdict", + } + + +class StudyDataAccess: + """Explicit, fail-closed view over source records. + + Ordinary development views may read development and selection-observed records only. + Transfer evaluation gets a separate view that may read transfer-hidden records after + a prediction is frozen. Future-final records are never available here. + """ + + def __init__( + self, + records: Sequence[StudyRecord], + *, + allowed_visibility: Sequence[Visibility] = (Visibility.DEVELOPMENT,), + purpose: str = "development-study", + ) -> None: + if Visibility.FUTURE_FINAL in allowed_visibility: + raise VisibilityViolation("future-final evidence is unavailable to MNEL study code") + if not allowed_visibility: + raise VisibilityViolation("study access requires an explicit visibility set") + if purpose == "development-study" and Visibility.TRANSFER_HIDDEN in allowed_visibility: + raise VisibilityViolation("development study access may not include transfer-hidden evidence") + self._records: dict[str, StudyRecord] = {} + for record in records: + if record.identity in self._records and self._records[record.identity] != record: + raise DistillationError("study record identity collision") + self._records[record.identity] = record + self.allowed_visibility = tuple(allowed_visibility) + self.purpose = purpose + + @classmethod + def development(cls, records: Sequence[StudyRecord]) -> "StudyDataAccess": + return cls( + records, + allowed_visibility=(Visibility.DEVELOPMENT, Visibility.SELECTION_OBSERVED), + purpose="development-study", + ) + + @classmethod + def transfer_evaluator(cls, records: Sequence[StudyRecord]) -> "StudyDataAccess": + return cls( + records, + allowed_visibility=(Visibility.TRANSFER_HIDDEN,), + purpose="transfer-evaluator", + ) + + @property + def dataset_identity(self) -> str: + return _identity_body( + { + "purpose": self.purpose, + "allowed_visibility": [item.value for item in self.allowed_visibility], + "records": [record.to_dict() for record in self.records()], + } + ) + + def records(self, record_type: str | None = None) -> tuple[StudyRecord, ...]: + values = [ + record + for record in self._records.values() + if record.visibility in self.allowed_visibility + and (record_type is None or record.record_type == record_type) + ] + return tuple(sorted(values, key=lambda item: item.identity)) + + def get(self, identity: str) -> StudyRecord: + try: + record = self._records[identity] + except KeyError as error: + raise VisibilityViolation(f"study record is unavailable: {identity}") from error + if record.visibility not in self.allowed_visibility: + raise VisibilityViolation( + f"study view {self.purpose} may not read {record.visibility.value} evidence" + ) + return record + + def require(self, identities: Sequence[str]) -> tuple[StudyRecord, ...]: + return tuple(self.get(identity) for identity in identities) + + def without_types(self, record_types: Sequence[str]) -> "StudyDataAccess": + blocked = set(record_types) + return StudyDataAccess( + tuple(record for record in self.records() if record.record_type not in blocked), + allowed_visibility=self.allowed_visibility, + purpose=self.purpose, + ) + + +@dataclass(frozen=True, slots=True) +class SemanticGroup: + source_record_ids: tuple[str, ...] + feature_extractor_identity: str + clustering_method_identity: str + parameters: dict[str, Any] + representative_record_ids: tuple[str, ...] + scope: dict[str, Any] + creation_epoch: int + limitations: tuple[str, ...] + group_identity: str = "" + authority: str = AUTHORITY_DIAGNOSTIC_ONLY + + def __post_init__(self) -> None: + if not self.source_record_ids or len(set(self.source_record_ids)) != len(self.source_record_ids): + raise DistillationError("semantic groups require unique source lineage") + _nonempty(self.feature_extractor_identity, "group feature extractor identity") + _nonempty(self.clustering_method_identity, "group method identity") + _bounded(self.parameters, "group parameters") + _bounded(self.scope, "group scope") + if not self.limitations or self.authority != AUTHORITY_DIAGNOSTIC_ONLY: + raise DistillationError("groups require limitations and diagnostic-only authority") + if self.creation_epoch < 0: + raise DistillationError("group epoch cannot be negative") + if self.group_identity and self.group_identity != self.content_identity: + raise DistillationError("semantic group identity does not match content") + + @property + def content_identity(self) -> str: + return _identity_body(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-semantic-group/0.4", + "source_record_ids": list(self.source_record_ids), + "feature_extractor_identity": self.feature_extractor_identity, + "clustering_method_identity": self.clustering_method_identity, + "parameters": dict(self.parameters), + "representative_record_ids": list(self.representative_record_ids), + "scope": dict(self.scope), + "creation_epoch": self.creation_epoch, + "limitations": list(self.limitations), + "authority": self.authority, + "semantics": "source-preserving-retrieval-group; not-a-verdict", + } + if include_identity: + value["group_identity"] = self.group_identity or self.content_identity + return value + + +def _feature_key(record: StudyRecord) -> tuple[str, ...]: + payload = record.payload + tags = payload.get("tags", ()) + if isinstance(tags, str): + tags = (tags,) + if not isinstance(tags, (list, tuple)): + tags = () + return ( + record.record_type, + str(payload.get("artifact_type", "")), + str(payload.get("uncertainty_class", "")), + str(payload.get("snapshot_type", "")), + "|".join(sorted(str(item) for item in tags)), + ) + + +def reference_feature_groups( + access: StudyDataAccess, + *, + feature_extractor_identity: str = "mnel-reference-features/0.4", + clustering_method_identity: str = "mnel-reference-feature-key-cluster/0.4", + max_groups: int = MAX_GROUPS, +) -> tuple[SemanticGroup, ...]: + if max_groups < 1 or max_groups > MAX_GROUPS: + raise DistillationError("group budget is outside its bounded range") + buckets: dict[tuple[str, ...], list[StudyRecord]] = {} + for record in access.records(): + bucket = buckets.setdefault(_feature_key(record), []) + if len(bucket) < MAX_GROUP_RECORDS: + bucket.append(record) + groups: list[SemanticGroup] = [] + for key, values in sorted(buckets.items()): + source_ids = tuple(item.identity for item in values) + group_draft = SemanticGroup( + source_ids, + feature_extractor_identity, + clustering_method_identity, + {"feature_key": list(key), "bounded": True}, + (source_ids[0],), + {"visibility": [item.value for item in access.allowed_visibility]}, + max(item.epoch for item in values), + ( + "reference grouping uses explicit record features, not semantic understanding", + "source records remain independently addressable", + ), + ) + groups.append( + SemanticGroup( + group_draft.source_record_ids, + group_draft.feature_extractor_identity, + group_draft.clustering_method_identity, + group_draft.parameters, + group_draft.representative_record_ids, + group_draft.scope, + group_draft.creation_epoch, + group_draft.limitations, + group_draft.content_identity, + ) + ) + if len(groups) > max_groups: + groups = sorted(groups, key=lambda item: item.group_identity)[:max_groups] + return tuple(sorted(groups, key=lambda item: item.group_identity)) + + +@dataclass(frozen=True, slots=True) +class DistilledPrinciple: + statement: str + scope: dict[str, Any] + attribution_ids: tuple[str, ...] + source_record_ids: tuple[str, ...] + counterexample_record_ids: tuple[str, ...] + falsifier: str + transfer_status: TransferStatus + maturity: str + limitations: tuple[str, ...] + principle_identity: str = "" + authority: str = AUTHORITY_PROPOSAL_ONLY + + def __post_init__(self) -> None: + _nonempty(self.statement, "principle statement") + _nonempty(self.falsifier, "principle falsifier") + if not self.attribution_ids or not self.source_record_ids: + raise DistillationError("principles must preserve attribution and source lineage") + _bounded(self.scope, "principle scope") + if not self.limitations or self.authority != AUTHORITY_PROPOSAL_ONLY: + raise DistillationError("principles require limitations and proposal-only authority") + if self.principle_identity and self.principle_identity != self.content_identity: + raise DistillationError("principle identity does not match content") + + @property + def content_identity(self) -> str: + return _identity_body(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-distilled-principle/0.4", + "statement": self.statement, + "scope": dict(self.scope), + "attribution_ids": list(self.attribution_ids), + "source_record_ids": list(self.source_record_ids), + "counterexample_record_ids": list(self.counterexample_record_ids), + "falsifier": self.falsifier, + "transfer_status": self.transfer_status.value, + "maturity": self.maturity, + "limitations": list(self.limitations), + "authority": self.authority, + "semantics": "provisional-distillation; not-a-verdict", + } + if include_identity: + value["principle_identity"] = self.principle_identity or self.content_identity + return value + + +@dataclass(frozen=True, slots=True) +class NegativeMemoryEntry: + memory_type: str + statement: str + source_record_ids: tuple[str, ...] + prohibited_contexts: tuple[str, ...] + reconsideration_condition: str + scope: dict[str, Any] + outcome_identity: str + memory_identity: str = "" + authority: str = AUTHORITY_DIAGNOSTIC_ONLY + + def __post_init__(self) -> None: + _nonempty(self.memory_type, "negative memory type") + _nonempty(self.statement, "negative memory statement") + _nonempty(self.reconsideration_condition, "negative memory reconsideration condition") + _nonempty(self.outcome_identity, "negative memory outcome identity") + if not self.source_record_ids or not self.prohibited_contexts: + raise DistillationError("negative memory requires source lineage and contexts") + _bounded(self.scope, "negative memory scope") + if self.authority != AUTHORITY_DIAGNOSTIC_ONLY: + raise DistillationError("negative memory is diagnostic-only") + if self.memory_identity and self.memory_identity != self.content_identity: + raise DistillationError("negative memory identity does not match content") + + @property + def content_identity(self) -> str: + return _identity_body(self.to_dict(include_identity=False)) + + def matches(self, context: dict[str, Any]) -> bool: + terms: set[str] = set() + for value in context.values(): + if isinstance(value, str): + terms.add(value) + elif isinstance(value, (list, tuple, set)): + terms.update(str(item) for item in value) + return bool(terms.intersection(self.prohibited_contexts)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-negative-memory/0.4", + "memory_type": self.memory_type, + "statement": self.statement, + "source_record_ids": list(self.source_record_ids), + "prohibited_contexts": list(self.prohibited_contexts), + "reconsideration_condition": self.reconsideration_condition, + "scope": dict(self.scope), + "outcome_identity": self.outcome_identity, + "authority": self.authority, + "semantics": "negative-evidence; not-a-verdict", + } + if include_identity: + value["memory_identity"] = self.memory_identity or self.content_identity + return value + + +@dataclass(frozen=True, slots=True) +class DistilledStrategy: + trigger_conditions: tuple[str, ...] + preconditions: tuple[str, ...] + expected_effect: dict[str, Any] + known_failure_modes: tuple[str, ...] + negative_memory_ids: tuple[str, ...] + counterexample_record_ids: tuple[str, ...] + causal_attribution_ids: tuple[str, ...] + supporting_source_record_ids: tuple[str, ...] + transfer_evidence_ids: tuple[str, ...] + scope: dict[str, Any] + calibration_metadata: dict[str, Any] + rollback_lineage: tuple[str, ...] + transfer_status: TransferStatus + strategy_identity: str = "" + authority: str = AUTHORITY_PROPOSAL_ONLY + + def __post_init__(self) -> None: + if not self.trigger_conditions or not self.preconditions: + raise DistillationError("strategies require trigger conditions and preconditions") + if not self.known_failure_modes or not self.supporting_source_record_ids: + raise DistillationError("strategies require failure modes and source lineage") + if not self.causal_attribution_ids: + raise DistillationError("strategies require causal attribution lineage") + _bounded(self.expected_effect, "strategy expected effect") + _bounded(self.scope, "strategy scope") + _bounded(self.calibration_metadata, "strategy calibration metadata") + if self.authority != AUTHORITY_PROPOSAL_ONLY: + raise DistillationError("strategies are proposal-only") + if self.strategy_identity and self.strategy_identity != self.content_identity: + raise DistillationError("strategy identity does not match content") + + @property + def content_identity(self) -> str: + return _identity_body(self.to_dict(include_identity=False)) + + def conflicts(self, context: dict[str, Any], memory: Sequence[NegativeMemoryEntry]) -> tuple[str, ...]: + return tuple( + item.memory_identity or item.content_identity + for item in memory + if item.matches(context) + and (not self.negative_memory_ids or (item.memory_identity or item.content_identity) in self.negative_memory_ids) + ) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-distilled-strategy/0.4", + "trigger_conditions": list(self.trigger_conditions), + "preconditions": list(self.preconditions), + "expected_effect": dict(self.expected_effect), + "known_failure_modes": list(self.known_failure_modes), + "negative_memory_ids": list(self.negative_memory_ids), + "counterexample_record_ids": list(self.counterexample_record_ids), + "causal_attribution_ids": list(self.causal_attribution_ids), + "supporting_source_record_ids": list(self.supporting_source_record_ids), + "transfer_evidence_ids": list(self.transfer_evidence_ids), + "scope": dict(self.scope), + "calibration_metadata": dict(self.calibration_metadata), + "rollback_lineage": list(self.rollback_lineage), + "transfer_status": self.transfer_status.value, + "authority": self.authority, + "semantics": "reusable-proposal; not-a-verdict", + } + if include_identity: + value["strategy_identity"] = self.strategy_identity or self.content_identity + return value + + +@dataclass(frozen=True, slots=True) +class TransferPrediction: + strategy_identity: str + source_development_identities: tuple[str, ...] + transfer_environment_identity: str + predicted_effect: dict[str, Any] + prediction_identity: str = "" + authority: str = AUTHORITY_PROPOSAL_ONLY + + def __post_init__(self) -> None: + if not self.source_development_identities: + raise DistillationError("transfer prediction requires development lineage") + _nonempty(self.strategy_identity, "transfer strategy identity") + _nonempty(self.transfer_environment_identity, "transfer environment identity") + _bounded(self.predicted_effect, "transfer predicted effect") + if self.authority != AUTHORITY_PROPOSAL_ONLY: + raise DistillationError("transfer predictions are proposal-only") + if self.prediction_identity and self.prediction_identity != self.content_identity: + raise DistillationError("transfer prediction identity does not match content") + + @property + def content_identity(self) -> str: + return _identity_body(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-transfer-prediction/0.4", + "strategy_identity": self.strategy_identity, + "source_development_identities": list(self.source_development_identities), + "transfer_environment_identity": self.transfer_environment_identity, + "predicted_effect": dict(self.predicted_effect), + "authority": self.authority, + "semantics": "frozen-before-transfer-observation; not-a-verdict", + } + if include_identity: + value["prediction_identity"] = self.prediction_identity or self.content_identity + return value + + +@dataclass(frozen=True, slots=True) +class TransferEvaluation: + prediction_identity: str + observed_outcome_identity: str + evaluator_evidence_identity: str + status: TransferStatus + rollback_lineage: tuple[str, ...] + evaluation_identity: str = "" + authority: str = AUTHORITY_DIAGNOSTIC_ONLY + + def __post_init__(self) -> None: + for value, label in ( + (self.prediction_identity, "prediction identity"), + (self.observed_outcome_identity, "observed outcome identity"), + (self.evaluator_evidence_identity, "evaluator evidence identity"), + ): + _nonempty(value, label) + if self.authority != AUTHORITY_DIAGNOSTIC_ONLY: + raise DistillationError("transfer evaluations are diagnostic evidence") + if self.evaluation_identity and self.evaluation_identity != self.content_identity: + raise DistillationError("transfer evaluation identity does not match content") + + @property + def content_identity(self) -> str: + return _identity_body(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-transfer-evaluation/0.4", + "prediction_identity": self.prediction_identity, + "observed_outcome_identity": self.observed_outcome_identity, + "evaluator_evidence_identity": self.evaluator_evidence_identity, + "status": self.status.value, + "rollback_lineage": list(self.rollback_lineage), + "authority": self.authority, + "semantics": "transfer-evidence; not-a-verdict", + } + if include_identity: + value["evaluation_identity"] = self.evaluation_identity or self.content_identity + return value + + +class TransferWorkflow: + def __init__(self) -> None: + self._predictions: dict[str, TransferPrediction] = {} + self._evaluations: dict[str, TransferEvaluation] = {} + + def freeze_prediction( + self, + strategy: DistilledStrategy, + *, + transfer_environment_identity: str, + predicted_effect: dict[str, Any], + ) -> TransferPrediction: + prediction = TransferPrediction( + strategy.strategy_identity or strategy.content_identity, + tuple(strategy.supporting_source_record_ids), + transfer_environment_identity, + dict(predicted_effect), + ) + prediction = TransferPrediction( + prediction.strategy_identity, + prediction.source_development_identities, + prediction.transfer_environment_identity, + prediction.predicted_effect, + prediction.content_identity, + ) + identity = prediction.prediction_identity + if identity in self._predictions: + raise DistillationError("duplicate transfer prediction") + self._predictions[identity] = prediction + return prediction + + def finalize( + self, + prediction: TransferPrediction, + *, + observed_outcome_identity: str, + evaluator_evidence_identity: str, + status: TransferStatus, + rollback_lineage: Sequence[str] = (), + ) -> TransferEvaluation: + identity = prediction.prediction_identity or prediction.content_identity + if identity not in self._predictions: + raise DistillationError("transfer prediction was not frozen by this workflow") + if identity in self._evaluations: + raise DistillationError("transfer prediction is already finalized") + evaluation = TransferEvaluation( + identity, + observed_outcome_identity, + evaluator_evidence_identity, + status, + tuple(rollback_lineage), + ) + evaluation = TransferEvaluation( + evaluation.prediction_identity, + evaluation.observed_outcome_identity, + evaluation.evaluator_evidence_identity, + evaluation.status, + evaluation.rollback_lineage, + evaluation.content_identity, + ) + self._evaluations[identity] = evaluation + return evaluation + + def reject_same_candidate_repair(self, strategy_identity: str, evaluation: TransferEvaluation) -> None: + prediction = self._predictions.get(evaluation.prediction_identity) + if prediction is not None and prediction.strategy_identity == strategy_identity: + raise VisibilityViolation("a strategy may not be repaired from its own hidden transfer result") + + +@dataclass(frozen=True, slots=True) +class StudyArm: + arm_id: str + kind: StudyArmKind + allowed_information: tuple[str, ...] + forbidden_information: tuple[str, ...] + retrieval_mode: str + memory_availability: tuple[str, ...] + attribution_available: bool + strategy_available: bool + recursion_allowance: int + budget: dict[str, int] + authority: str = AUTHORITY_PROPOSAL_ONLY + + def __post_init__(self) -> None: + _nonempty(self.arm_id, "study arm id") + if self.recursion_allowance < 0 or self.authority != AUTHORITY_PROPOSAL_ONLY: + raise DistillationError("study arms require bounded recursion and proposal authority") + if not set(self.allowed_information).isdisjoint(self.forbidden_information): + raise DistillationError("study arm information cannot be both allowed and forbidden") + if set(self.budget) != {"operations", "wall_seconds", "candidates"} or any( + not isinstance(value, int) or value < 1 for value in self.budget.values() + ): + raise DistillationError("study arm budgets are malformed") + + @property + def arm_identity(self) -> str: + return canonical_digest(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-study-arm/0.4", + "arm_id": self.arm_id, + "kind": self.kind.value, + "allowed_information": list(self.allowed_information), + "forbidden_information": list(self.forbidden_information), + "retrieval_mode": self.retrieval_mode, + "memory_availability": list(self.memory_availability), + "attribution_available": self.attribution_available, + "strategy_available": self.strategy_available, + "recursion_allowance": self.recursion_allowance, + "budget": dict(self.budget), + "authority": self.authority, + "semantics": "study-arm-contract; not-a-verdict", + } + if include_identity: + value["arm_identity"] = self.arm_identity + return value + + +@dataclass(frozen=True, slots=True) +class AblationSpec: + ablation_id: str + kind: AblationKind + source_arm_identity: str + seed: int + parameters: dict[str, Any] + authority: str = AUTHORITY_PROPOSAL_ONLY + + def __post_init__(self) -> None: + _nonempty(self.ablation_id, "ablation id") + _nonempty(self.source_arm_identity, "ablation source arm identity") + if self.seed < 0 or self.authority != AUTHORITY_PROPOSAL_ONLY: + raise DistillationError("ablation seed or authority is invalid") + _bounded(self.parameters, "ablation parameters") + + @property + def ablation_identity(self) -> str: + return canonical_digest(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-ablation-spec/0.4", + "ablation_id": self.ablation_id, + "kind": self.kind.value, + "source_arm_identity": self.source_arm_identity, + "seed": self.seed, + "parameters": dict(self.parameters), + "authority": self.authority, + "semantics": "controlled-study-transformation; not-a-verdict", + } + if include_identity: + value["ablation_identity"] = self.ablation_identity + return value + + +@dataclass(frozen=True, slots=True) +class ShuffledAttributionControl: + seed: int + original_attribution_ids: tuple[str, ...] + shuffled_attribution_ids: tuple[str, ...] + source_study_identity: str + control_identity: str = "" + + def __post_init__(self) -> None: + if self.seed < 0 or not self.original_attribution_ids: + raise DistillationError("shuffled attribution requires a seed and source ids") + if len(self.original_attribution_ids) != len(self.shuffled_attribution_ids): + raise DistillationError("shuffled attribution lengths differ") + if sorted(self.original_attribution_ids) != sorted(self.shuffled_attribution_ids): + raise DistillationError("shuffled attribution must preserve the id multiset") + _nonempty(self.source_study_identity, "shuffled attribution source study") + if self.control_identity and self.control_identity != self.content_identity: + raise DistillationError("shuffled attribution identity does not match content") + + @property + def content_identity(self) -> str: + return canonical_digest(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-shuffled-attribution-control/0.4", + "seed": self.seed, + "original_attribution_ids": list(self.original_attribution_ids), + "shuffled_attribution_ids": list(self.shuffled_attribution_ids), + "source_study_identity": self.source_study_identity, + "authority": AUTHORITY_DIAGNOSTIC_ONLY, + "semantics": "control-only; original attribution remains immutable", + } + if include_identity: + value["control_identity"] = self.control_identity or self.content_identity + return value + + +def shuffle_attributions( + attribution_ids: Sequence[str], *, seed: int, source_study_identity: str +) -> ShuffledAttributionControl: + values = list(attribution_ids) + if not values or seed < 0: + raise DistillationError("shuffled attribution inputs are invalid") + shuffled = list(values) + random.Random(seed).shuffle(shuffled) + return ShuffledAttributionControl(seed, tuple(values), tuple(shuffled), source_study_identity) + + +@dataclass(frozen=True, slots=True) +class StudySpecification: + study_id: str + development_dataset_identity: str + hidden_transfer_dataset_identity: str + arms: tuple[StudyArm, ...] + ablations: tuple[AblationSpec, ...] + equal_budget: dict[str, int] + study_identity: str = "" + + def __post_init__(self) -> None: + _nonempty(self.study_id, "study id") + _nonempty(self.development_dataset_identity, "development dataset identity") + _nonempty(self.hidden_transfer_dataset_identity, "hidden transfer dataset identity") + if not self.arms: + raise DistillationError("study requires at least one arm") + if len({item.arm_id for item in self.arms}) != len(self.arms): + raise DistillationError("study arm ids must be unique") + if set(self.equal_budget) != {"operations", "wall_seconds", "candidates"} or any( + not isinstance(value, int) or value < 1 for value in self.equal_budget.values() + ): + raise DistillationError("study equal-budget declaration is malformed") + if any(item.budget != self.equal_budget for item in self.arms): + raise DistillationError("study arms must use the declared equal budget") + if self.study_identity and self.study_identity != self.content_identity: + raise DistillationError("study identity does not match content") + if not self.study_identity: + object.__setattr__(self, "study_identity", self.content_identity) + + @property + def content_identity(self) -> str: + return canonical_digest(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-distillation-study-spec/0.4", + "study_id": self.study_id, + "development_dataset_identity": self.development_dataset_identity, + "hidden_transfer_dataset_identity": self.hidden_transfer_dataset_identity, + "arms": [item.to_dict() for item in self.arms], + "ablations": [item.to_dict() for item in self.ablations], + "equal_budget": dict(self.equal_budget), + "authority": AUTHORITY_PROPOSAL_ONLY, + "semantics": "study-specification; not-a-verdict", + } + if include_identity: + value["study_identity"] = self.study_identity or self.content_identity + return value + + +@dataclass(frozen=True, slots=True) +class RetrievalQuery: + trigger: str + context: dict[str, Any] + limit: int = 8 + include_negative: bool = True + query_identity: str = "" + + def __post_init__(self) -> None: + _nonempty(self.trigger, "retrieval trigger") + _bounded(self.context, "retrieval context") + if self.limit < 1 or self.limit > MAX_RETRIEVAL_HITS: + raise DistillationError("retrieval limit is outside its bounded range") + if self.query_identity and self.query_identity != self.content_identity: + raise DistillationError("retrieval query identity does not match content") + + @property + def content_identity(self) -> str: + return canonical_digest(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-retrieval-query/0.4", + "trigger": self.trigger, + "context": dict(self.context), + "limit": self.limit, + "include_negative": self.include_negative, + } + if include_identity: + value["query_identity"] = self.query_identity or self.content_identity + return value + + +@dataclass(frozen=True, slots=True) +class RetrievalHit: + record_class: str + record_identity: str + score: float + reasons: tuple[str, ...] + source_record_ids: tuple[str, ...] + + def __post_init__(self) -> None: + if not math.isfinite(self.score) or not self.record_class or not self.record_identity: + raise DistillationError("retrieval hit is malformed") + + def to_dict(self) -> dict[str, Any]: + return { + "record_class": self.record_class, + "record_identity": self.record_identity, + "score": self.score, + "reasons": list(self.reasons), + "source_record_ids": list(self.source_record_ids), + } + + +@dataclass(frozen=True, slots=True) +class RetrievalResult: + query_identity: str + hits: tuple[RetrievalHit, ...] + negative_conflict_ids: tuple[str, ...] + method_identity: str + + @property + def result_identity(self) -> str: + return canonical_digest(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-retrieval-result/0.4", + "query_identity": self.query_identity, + "hits": [item.to_dict() for item in self.hits], + "negative_conflict_ids": list(self.negative_conflict_ids), + "method_identity": self.method_identity, + "authority": AUTHORITY_DIAGNOSTIC_ONLY, + "semantics": "heuristic-retrieval; not-a-verdict", + } + if include_identity: + value["result_identity"] = self.result_identity + return value + + +class RetrievalIndex: + METHOD_IDENTITY = "mnel-reference-retrieval/0.4" + + def __init__(self) -> None: + self._records: dict[str, StudyRecord] = {} + self._groups: dict[str, SemanticGroup] = {} + self._principles: dict[str, DistilledPrinciple] = {} + self._strategies: dict[str, DistilledStrategy] = {} + self._negative: dict[str, NegativeMemoryEntry] = {} + + def add_source_records(self, records: Sequence[StudyRecord]) -> None: + for record in records: + if record.visibility in {Visibility.TRANSFER_HIDDEN, Visibility.FUTURE_FINAL}: + raise VisibilityViolation("retrieval indexes may not ingest hidden or future-final records") + if record.identity in self._records and self._records[record.identity] != record: + raise DistillationError("retrieval source identity collision") + self._records[record.identity] = record + + def add_groups(self, groups: Sequence[SemanticGroup]) -> None: + for item in groups: + self._groups[item.group_identity or item.content_identity] = item + + def add_principles(self, principles: Sequence[DistilledPrinciple]) -> None: + for item in principles: + self._principles[item.principle_identity or item.content_identity] = item + + def add_strategies(self, strategies: Sequence[DistilledStrategy]) -> None: + for item in strategies: + self._strategies[item.strategy_identity or item.content_identity] = item + + def add_negative_memory(self, entries: Sequence[NegativeMemoryEntry]) -> None: + for item in entries: + self._negative[item.memory_identity or item.content_identity] = item + + def retrieve(self, query: RetrievalQuery) -> RetrievalResult: + memory = tuple(self._negative.values()) if query.include_negative else () + conflicts = tuple(sorted(item.memory_identity or item.content_identity for item in memory if item.matches(query.context))) + hits: list[RetrievalHit] = [] + for identity, strategy in self._strategies.items(): + score = 0.0 + reasons: list[str] = [] + if query.trigger in strategy.trigger_conditions: + score += 4.0 + reasons.append("trigger-compatible") + if any(str(value) in strategy.scope.values() for value in query.context.values()): + score += 1.0 + reasons.append("scope-compatible") + if strategy.transfer_status is TransferStatus.SUPPORTED: + score += 2.0 + reasons.append("transfer-supported") + strategy_conflicts = strategy.conflicts(query.context, memory) + if strategy_conflicts: + score -= 6.0 + reasons.append("negative-memory-conflict") + if score > 0 or strategy_conflicts: + hits.append(RetrievalHit("strategy", identity, score, tuple(reasons), strategy.supporting_source_record_ids)) + for identity, principle in self._principles.items(): + score = 1.0 if any(str(value) in principle.scope.values() for value in query.context.values()) else 0.0 + if score: + hits.append(RetrievalHit("principle", identity, score, ("scope-compatible",), principle.source_record_ids)) + for identity, entry in self._negative.items(): + if query.include_negative and entry.matches(query.context): + hits.append(RetrievalHit("negative-memory", identity, 3.0, ("prohibited-context-match",), entry.source_record_ids)) + for identity, record in self._records.items(): + values = {str(item) for item in record.payload.values() if isinstance(item, (str, int, float, bool))} + if query.trigger in values or any(str(value) in values for value in query.context.values()): + hits.append(RetrievalHit("source-record", identity, 0.5, ("explicit-feature-match",), (identity,))) + hits.sort(key=lambda item: (-item.score, item.record_class, item.record_identity)) + return RetrievalResult(query.query_identity or query.content_identity, tuple(hits[: query.limit]), conflicts, self.METHOD_IDENTITY) + + +@dataclass(frozen=True, slots=True) +class RetrievalMetrics: + query_identity: str + k: int + relevant_count: int + precision_at_k: float + recall_at_k: float + hit_rate: float + reciprocal_rank: float + duplicate_retrieval_rate: float + evidence_reuse_depth: int + source_diversity: int + positive_negative_balance: float + transfer_supported_strategy_rate: float + unsupported_strategy_rate: float + metric_identity: str = "" + + def __post_init__(self) -> None: + values = ( + self.precision_at_k, + self.recall_at_k, + self.hit_rate, + self.reciprocal_rank, + self.duplicate_retrieval_rate, + self.positive_negative_balance, + self.transfer_supported_strategy_rate, + self.unsupported_strategy_rate, + ) + if any(not math.isfinite(value) for value in values): + raise DistillationError("retrieval metrics must be finite") + + @property + def content_identity(self) -> str: + return canonical_digest(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-retrieval-metrics/0.4", + "query_identity": self.query_identity, + "k": self.k, + "relevant_count": self.relevant_count, + "precision_at_k": self.precision_at_k, + "recall_at_k": self.recall_at_k, + "hit_rate": self.hit_rate, + "reciprocal_rank": self.reciprocal_rank, + "duplicate_retrieval_rate": self.duplicate_retrieval_rate, + "evidence_reuse_depth": self.evidence_reuse_depth, + "source_diversity": self.source_diversity, + "positive_negative_balance": self.positive_negative_balance, + "transfer_supported_strategy_rate": self.transfer_supported_strategy_rate, + "unsupported_strategy_rate": self.unsupported_strategy_rate, + "authority": AUTHORITY_DIAGNOSTIC_ONLY, + "semantics": "retrieval-measurement; not-a-verdict", + } + if include_identity: + value["metric_identity"] = self.metric_identity or self.content_identity + return value + + +def evaluate_retrieval( + result: RetrievalResult, + relevant_ids: Sequence[str], + *, + strategy_status: dict[str, TransferStatus] | None = None, +) -> RetrievalMetrics: + relevant = set(relevant_ids) + hits = result.hits + top_ids = [item.record_identity for item in hits] + unique = set(top_ids) + true_hits = [index for index, identity in enumerate(top_ids, 1) if identity in relevant] + k = len(hits) + relevant_count = len(relevant) + precision = len(true_hits) / k if k else 0.0 + recall = len(true_hits) / relevant_count if relevant_count else 0.0 + reciprocal = 1.0 / true_hits[0] if true_hits else 0.0 + duplicate_rate = (len(top_ids) - len(unique)) / len(top_ids) if top_ids else 0.0 + strategy_hits = [item for item in hits if item.record_class == "strategy"] + supported = sum( + 1 for item in strategy_hits if strategy_status and strategy_status.get(item.record_identity) is TransferStatus.SUPPORTED + ) + unsupported = sum( + 1 for item in strategy_hits if not strategy_status or strategy_status.get(item.record_identity) is not TransferStatus.SUPPORTED + ) + negative = sum(1 for item in hits if item.record_class == "negative-memory") + positive_negative_balance = negative / (negative + len(strategy_hits)) if (negative + len(strategy_hits)) else 0.0 + depth = max((len(item.source_record_ids) for item in hits), default=0) + diversity = len({source for item in hits for source in item.source_record_ids}) + return RetrievalMetrics( + result.query_identity, + k, + relevant_count, + precision, + recall, + float(bool(true_hits)), + reciprocal, + duplicate_rate, + depth, + diversity, + positive_negative_balance, + supported / len(strategy_hits) if strategy_hits else 0.0, + unsupported / len(strategy_hits) if strategy_hits else 0.0, + ) + + +@dataclass(frozen=True, slots=True) +class CalibrationMetrics: + dataset_identity: str + model_identity: str + calibration_method_identity: str + count: int + brier_score: float + log_loss: float + expected_calibration_error: float + coverage: float + abstention_rate: float + out_of_distribution_rate: float + metric_identity: str = "" + + def __post_init__(self) -> None: + if self.count < 1 or any( + not math.isfinite(value) + for value in ( + self.brier_score, + self.log_loss, + self.expected_calibration_error, + self.coverage, + self.abstention_rate, + self.out_of_distribution_rate, + ) + ): + raise DistillationError("calibration metrics are invalid") + + @property + def content_identity(self) -> str: + return canonical_digest(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-calibration-metrics/0.4", + "dataset_identity": self.dataset_identity, + "model_identity": self.model_identity, + "calibration_method_identity": self.calibration_method_identity, + "count": self.count, + "brier_score": self.brier_score, + "log_loss": self.log_loss, + "expected_calibration_error": self.expected_calibration_error, + "coverage": self.coverage, + "abstention_rate": self.abstention_rate, + "out_of_distribution_rate": self.out_of_distribution_rate, + "authority": AUTHORITY_DIAGNOSTIC_ONLY, + "semantics": "calibration-measurement; not-a-verdict", + } + if include_identity: + value["metric_identity"] = self.metric_identity or self.content_identity + return value + + +def calculate_calibration( + predictions: Sequence[float | None], + labels: Sequence[int | bool], + *, + dataset_identity: str, + model_identity: str, + calibration_method_identity: str = "mnel-reference-bins/0.4", + out_of_distribution: Sequence[bool] = (), + bins: int = 10, +) -> CalibrationMetrics: + if len(predictions) != len(labels) or not predictions or bins < 1: + raise DistillationError("calibration inputs are malformed") + if out_of_distribution and len(out_of_distribution) != len(predictions): + raise DistillationError("OOD flags must match calibration inputs") + valid: list[tuple[float, float]] = [] + abstained = 0 + for prediction, label in zip(predictions, labels): + if prediction is None: + abstained += 1 + continue + if not isinstance(prediction, (int, float)) or isinstance(prediction, bool) or not math.isfinite(prediction): + raise DistillationError("calibration predictions must be finite numbers or None") + if prediction < 0.0 or prediction > 1.0 or label not in (0, 1, False, True): + raise DistillationError("calibration values are outside their bounds") + valid.append((float(prediction), float(bool(label)))) + if not valid: + raise DistillationError("calibration requires at least one non-abstained prediction") + brier = sum((prediction - label) ** 2 for prediction, label in valid) / len(valid) + log_loss = sum( + -(label * math.log(max(prediction, 1e-12)) + (1 - label) * math.log(max(1 - prediction, 1e-12))) + for prediction, label in valid + ) / len(valid) + ece = 0.0 + for index in range(bins): + low = index / bins + high = (index + 1) / bins + members = [item for item in valid if low <= item[0] < high or (index == bins - 1 and item[0] == high)] + if members: + confidence = sum(item[0] for item in members) / len(members) + frequency = sum(item[1] for item in members) / len(members) + ece += len(members) / len(valid) * abs(confidence - frequency) + ood_rate = sum(bool(item) for item in out_of_distribution) / len(predictions) if out_of_distribution else 0.0 + return CalibrationMetrics( + dataset_identity, + model_identity, + calibration_method_identity, + len(predictions), + brier, + log_loss, + ece, + len(valid) / len(predictions), + abstained / len(predictions), + ood_rate, + ) + + +def validate_equal_budget(arms: Sequence[StudyArm]) -> dict[str, int]: + if not arms: + raise DistillationError("equal-budget validation requires study arms") + budget = dict(arms[0].budget) + if any(item.budget != budget for item in arms[1:]): + raise DistillationError("study arms do not satisfy equal-budget control") + return budget + + +def make_study_record( + record_type: str, + payload: dict[str, Any], + *, + visibility: Visibility = Visibility.DEVELOPMENT, + epoch: int = 0, + record_identity: str = "", +) -> StudyRecord: + _reject_authority(payload) + return StudyRecord(record_type, dict(payload), visibility, record_identity, epoch) + + +def build_distilled_strategy( + *, + trigger_conditions: Sequence[str], + preconditions: Sequence[str], + expected_effect: dict[str, Any], + known_failure_modes: Sequence[str], + negative_memory_ids: Sequence[str], + counterexample_record_ids: Sequence[str], + causal_attribution_ids: Sequence[str], + supporting_source_record_ids: Sequence[str], + transfer_evidence_ids: Sequence[str], + scope: dict[str, Any], + calibration_metadata: dict[str, Any] | None = None, + rollback_lineage: Sequence[str] = (), + transfer_status: TransferStatus = TransferStatus.UNTESTED, +) -> DistilledStrategy: + draft = DistilledStrategy( + tuple(trigger_conditions), + tuple(preconditions), + dict(expected_effect), + tuple(known_failure_modes), + tuple(negative_memory_ids), + tuple(counterexample_record_ids), + tuple(causal_attribution_ids), + tuple(dict.fromkeys(supporting_source_record_ids)), + tuple(transfer_evidence_ids), + dict(scope), + dict(calibration_metadata or {"method": "unspecified"}), + tuple(rollback_lineage), + transfer_status, + ) + return DistilledStrategy( + draft.trigger_conditions, + draft.preconditions, + draft.expected_effect, + draft.known_failure_modes, + draft.negative_memory_ids, + draft.counterexample_record_ids, + draft.causal_attribution_ids, + draft.supporting_source_record_ids, + draft.transfer_evidence_ids, + draft.scope, + draft.calibration_metadata, + draft.rollback_lineage, + draft.transfer_status, + draft.content_identity, + ) + + +def build_distilled_principle( + *, + statement: str, + scope: dict[str, Any], + attribution_ids: Sequence[str], + source_record_ids: Sequence[str], + counterexample_record_ids: Sequence[str], + falsifier: str, + transfer_status: TransferStatus = TransferStatus.UNTESTED, + maturity: str = "provisional", + limitations: Sequence[str] = ("source-bound; not independently validated",), +) -> DistilledPrinciple: + draft = DistilledPrinciple( + statement, + dict(scope), + tuple(attribution_ids), + tuple(dict.fromkeys(source_record_ids)), + tuple(counterexample_record_ids), + falsifier, + transfer_status, + maturity, + tuple(limitations), + ) + return DistilledPrinciple( + draft.statement, + draft.scope, + draft.attribution_ids, + draft.source_record_ids, + draft.counterexample_record_ids, + draft.falsifier, + draft.transfer_status, + draft.maturity, + draft.limitations, + draft.content_identity, + ) + + +def run_reference_distill_study(workspace: str | Path | None = None) -> dict[str, Any]: + """Run a deterministic, held-out transfer study with controls and measurements.""" + + from .reference_provider import train_transition_frequency + from .snapshots import SnapshotStore, decode_snapshot, transition_snapshot + + snapshot_store = SnapshotStore() + snapshot_identities = { + "transition-a": transition_snapshot( + b"cold", + b"warm", + producer_identity="mnel-distill-reference-producer/0.4", + source_identity="sha256:development-source-a", + dependency_identity="sha256:development-dependency", + feature_extractor_identity="sha256:transition-feature-extractor", + ), + "transition-b": transition_snapshot( + b"hot", + b"warm", + producer_identity="mnel-distill-reference-producer/0.4", + source_identity="sha256:development-source-b", + dependency_identity="sha256:development-dependency", + feature_extractor_identity="sha256:transition-feature-extractor", + ), + } + for snapshot in snapshot_identities.values(): + snapshot_store.register(snapshot) + development_snapshots = { + key: value.snapshot_identity for key, value in snapshot_identities.items() + } + development_records = ( + make_study_record( + "experience-episode", + { + "artifact_type": "routing", + "uncertainty_class": "unexpected-transition", + "snapshot_type": "transition", + "tags": ["transition", "routing"], + "snapshot_identity": development_snapshots["transition-a"], + "outcome": "success", + }, + record_identity="sha256:development-episode-a", + epoch=1, + ), + make_study_record( + "experience-episode", + { + "artifact_type": "routing", + "uncertainty_class": "unexpected-transition", + "snapshot_type": "transition", + "tags": ["transition", "routing"], + "snapshot_identity": development_snapshots["transition-b"], + "outcome": "success", + }, + record_identity="sha256:development-episode-b", + epoch=1, + ), + make_study_record( + "counterexample", + { + "artifact_type": "routing", + "uncertainty_class": "unexpected-transition", + "snapshot_type": "transition", + "tags": ["transition", "unsupported-provider"], + "outcome": "regression", + }, + record_identity="sha256:development-counterexample", + epoch=2, + ), + make_study_record( + "causal-attribution", + {"attribution_id": "sha256:attribution-a", "supporting_episode_ids": ["sha256:development-episode-a"]}, + record_identity="sha256:attribution-record-a", + epoch=2, + ), + ) + hidden_records = ( + make_study_record( + "transfer-outcome", + {"strategy_identity": "pending", "outcome": "supported", "metric": 0.92}, + visibility=Visibility.TRANSFER_HIDDEN, + record_identity="sha256:hidden-transfer-outcome", + epoch=3, + ), + make_study_record( + "future-final-reference", + {"outcome": "future-final-only"}, + visibility=Visibility.FUTURE_FINAL, + record_identity="sha256:future-final-record", + epoch=4, + ), + ) + all_records = development_records + hidden_records + development = StudyDataAccess.development(all_records) + try: + development.get("sha256:hidden-transfer-outcome") + raise DistillationError("development access unexpectedly read hidden transfer") + except VisibilityViolation: + pass + try: + development.get("sha256:future-final-record") + raise DistillationError("development access unexpectedly read future-final evidence") + except VisibilityViolation: + pass + groups = reference_feature_groups(development) + attribution = Attribution( + "sha256:attribution-a", + "reference-distill-study", + "intervention-routing", + "sha256:evaluation-a", + "supported-with-alternatives", + ("immediate",), + ("sha256:development-episode-a", "sha256:development-episode-b"), + ("unsupported-provider effect",), + ("sha256:development-episode-a", "sha256:development-episode-b", "sha256:attribution-record-a"), + ) + principle = build_distilled_principle( + statement="Transition-aware routing may improve the declared routing fixture when its provider assumptions hold.", + scope={"artifact_type": "routing", "snapshot_type": "transition"}, + attribution_ids=(attribution.attribution_id,), + source_record_ids=attribution.source_record_ids, + counterexample_record_ids=("sha256:development-counterexample",), + falsifier="held-out transfer does not reproduce the predicted bounded effect", + ) + negative = NegativeMemoryEntry( + "counterexample", + "Do not apply transition-aware routing in the unsupported-provider context without a new probe.", + ("sha256:development-counterexample",), + ("unsupported-provider",), + "a new independent probe supports the provider assumption", + {"artifact_type": "routing", "snapshot_type": "transition"}, + "sha256:development-counterexample", + ) + negative = NegativeMemoryEntry( + negative.memory_type, + negative.statement, + negative.source_record_ids, + negative.prohibited_contexts, + negative.reconsideration_condition, + negative.scope, + negative.outcome_identity, + negative.content_identity, + ) + strategy = build_distilled_strategy( + trigger_conditions=("unexpected-transition",), + preconditions=("artifact_type=routing", "snapshot_type=transition", "provider-assumption-supported"), + expected_effect={"exact_target_rate": "increase", "retention": "no-regression"}, + known_failure_modes=("unsupported-provider", "unprobed-transfer-context"), + negative_memory_ids=(negative.memory_identity,), + counterexample_record_ids=negative.source_record_ids, + causal_attribution_ids=(attribution.attribution_id,), + supporting_source_record_ids=attribution.source_record_ids, + transfer_evidence_ids=(), + scope={"artifact_type": "routing", "snapshot_type": "transition"}, + calibration_metadata={"method": "reference-fixed-calibration", "dataset": "development"}, + ) + index = RetrievalIndex() + index.add_source_records(development.records()) + index.add_groups(groups) + index.add_principles((principle,)) + index.add_strategies((strategy,)) + index.add_negative_memory((negative,)) + query = RetrievalQuery( + "unexpected-transition", + {"artifact_type": "routing", "snapshot_type": "transition"}, + limit=8, + ) + retrieval = index.retrieve(query) + risky_query = RetrievalQuery( + "unexpected-transition", + {"artifact_type": "routing", "snapshot_type": "transition", "provider": "unsupported-provider"}, + limit=8, + ) + risky_retrieval = index.retrieve(risky_query) + ablated_index = RetrievalIndex() + ablated_index.add_source_records(development.records()) + ablated_index.add_groups(groups) + ablated_index.add_principles((principle,)) + ablated_index.add_strategies((strategy,)) + ablated_retrieval = ablated_index.retrieve(risky_query) + strategy_status = {strategy.strategy_identity: strategy.transfer_status} + retrieval_metrics = evaluate_retrieval(retrieval, (strategy.strategy_identity,), strategy_status=strategy_status) + calibration = calculate_calibration( + (0.8, 0.2, 0.6), + (True, False, True), + dataset_identity=development.dataset_identity, + model_identity="mnel-reference-distilled-strategy/0.4", + out_of_distribution=(False, False, True), + ) + provider_access = StudyDataAccess.development(development_records[:2]) + provider_model = train_transition_frequency(provider_access, snapshot_store) + provider_observation = provider_model.infer(snapshot_identities["transition-a"]) + reloaded_provider = type(provider_model).load(provider_model.serialize()) + reloaded_observation = reloaded_provider.infer(snapshot_identities["transition-a"]) + reference_view = decode_snapshot(snapshot_identities["transition-a"]) + reference_changed = bool( + hasattr(reference_view, "previous_state") + and reference_view.previous_state != reference_view.next_state + ) + provider_baseline_comparison = { + "provider_observation_identity": provider_observation.observation_identity + or provider_observation.content_identity, + "reloaded_observation_identity": reloaded_observation.observation_identity + or reloaded_observation.content_identity, + "deterministic_reference": { + "condition_observed": reference_changed, + "authority": AUTHORITY_DIAGNOSTIC_ONLY, + }, + "heuristic_nonempty_transition": { + "score": float(bool(reference_view.previous_state or reference_view.next_state)), + "authority": AUTHORITY_DIAGNOSTIC_ONLY, + }, + "random_control": { + "seed": 17, + "score": 0.5, + "authority": AUTHORITY_DIAGNOSTIC_ONLY, + }, + "classical_hmm_baseline": { + "provider_id": "state.hidden-markov-model", + "status": "not-input-compatible", + "reason": "native HMM consumes bounded numeric state sequences; this fixture uses byte transition views", + "authority": AUTHORITY_DIAGNOSTIC_ONLY, + }, + "provider_is_verifier": False, + "authority": AUTHORITY_DIAGNOSTIC_ONLY, + "semantics": "diagnostic-comparison; not-a-verdict", + } + arms = ( + StudyArm("A0", StudyArmKind.RANDOM, ("development",), ("attribution", "transfer-hidden"), "random", (), False, False, 0, {"operations": 100, "wall_seconds": 30, "candidates": 4}), + StudyArm("A4", StudyArmKind.ATTRIBUTION, ("development", "attribution"), ("transfer-hidden",), "lineage-aware", ("positive", "negative"), True, True, 0, {"operations": 100, "wall_seconds": 30, "candidates": 4}), + StudyArm("A5", StudyArmKind.TRANSFER_GATED, ("development", "attribution"), ("transfer-hidden",), "lineage-aware", ("positive", "negative"), True, True, 0, {"operations": 100, "wall_seconds": 30, "candidates": 4}), + ) + shuffled = shuffle_attributions( + (attribution.attribution_id, "sha256:attribution-b"), + seed=17, + source_study_identity="sha256:reference-distill-study", + ) + ablations = tuple( + AblationSpec(f"ablation-{kind.value}", kind, arms[1].arm_identity, 17, {}) + for kind in ( + AblationKind.AGGREGATE_ONLY, + AblationKind.SHUFFLED_ATTRIBUTION, + AblationKind.RANDOM_PROPOSAL, + AblationKind.SUCCESS_MEMORY, + AblationKind.NEGATIVE_MEMORY, + AblationKind.FIXED_POLICY, + AblationKind.EQUAL_BUDGET, + AblationKind.HIDDEN_TRANSFER, + ) + ) + specification = StudySpecification( + "reference-distill-study", + development.dataset_identity, + StudyDataAccess.transfer_evaluator(all_records).dataset_identity, + arms, + ablations, + validate_equal_budget(arms), + ) + specification = StudySpecification( + specification.study_id, + specification.development_dataset_identity, + specification.hidden_transfer_dataset_identity, + specification.arms, + specification.ablations, + specification.equal_budget, + specification.content_identity, + ) + workflow = TransferWorkflow() + prediction = workflow.freeze_prediction( + strategy, + transfer_environment_identity="sha256:held-out-transfer-environment", + predicted_effect={"exact_target_rate": {"direction": "increase", "minimum": 0.9}}, + ) + hidden_access = StudyDataAccess.transfer_evaluator(all_records) + hidden = hidden_access.get("sha256:hidden-transfer-outcome") + evaluation = workflow.finalize( + prediction, + observed_outcome_identity=hidden.identity, + evaluator_evidence_identity="sha256:external-evaluator-evidence", + status=TransferStatus.SUPPORTED, + ) + report_identity_body = { + "specification": specification.to_dict(), + "groups": [item.to_dict() for item in groups], + "principle": principle.to_dict(), + "strategy": strategy.to_dict(), + "negative_memory": negative.to_dict(), + "retrieval": retrieval.to_dict(), + "risky_retrieval": risky_retrieval.to_dict(), + "ablated_retrieval": ablated_retrieval.to_dict(), + "retrieval_metrics": retrieval_metrics.to_dict(), + "calibration": calibration.to_dict(), + "shuffled_attribution": shuffled.to_dict(), + "prediction": prediction.to_dict(), + "evaluation": evaluation.to_dict(), + "provider_model": provider_model.to_dict(), + "provider_observation": provider_observation.to_dict(), + "provider_baseline_comparison": provider_baseline_comparison, + } + study_identity = canonical_digest(report_identity_body) + report = { + "schema": "mnel-distillation-study-report/0.4", + "study_identity": study_identity, + "study_specification_identity": specification.study_identity or specification.content_identity, + "development_dataset_identity": development.dataset_identity, + "hidden_transfer_dataset_identity": StudyDataAccess.transfer_evaluator(all_records).dataset_identity, + "group_count": len(groups), + "candidate_count": 1, + "retrieval_metrics": retrieval_metrics.to_dict(), + "calibration_metrics": calibration.to_dict(), + "transfer_status": evaluation.status.value, + "transfer_prediction_frozen_before_evaluation": True, + "negative_memory_conflicts": list(risky_retrieval.negative_conflict_ids), + "negative_memory_demoted_strategy": any( + item.record_identity == strategy.strategy_identity and item.score < 0 for item in risky_retrieval.hits + ), + "negative_memory_ablation_strategy_score": next( + (item.score for item in ablated_retrieval.hits if item.record_identity == strategy.strategy_identity), + None, + ), + "shuffled_attribution_control_identity": shuffled.control_identity or shuffled.content_identity, + "arm_identities": [item.arm_identity for item in specification.arms], + "ablation_identities": [item.ablation_identity for item in specification.ablations], + "learned_provider_model_identity": provider_model.model_identity or provider_model.content_identity, + "learned_provider_artifact_identity": provider_model.artifact_identity, + "learned_provider_model_size_bytes": provider_model.model_size_bytes, + "learned_provider_reload_reproduced": provider_observation.to_dict() == reloaded_observation.to_dict(), + "provider_baseline_comparison": provider_baseline_comparison, + "source_record_count": len(development.records()), + "authority_violation_attempts": 2, + "limitations": [ + "reference feature grouping is not semantic understanding", + "synthetic transfer outcome is a fixture observation, not a universal claim", + "the native HMM baseline is retained separately because its numeric state-sequence input is not compatible with this byte-transition fixture", + "no external Forge, Fabric, RAVEL, or MNCS authority is implemented", + ], + "authority": AUTHORITY_DIAGNOSTIC_ONLY, + "semantics": "measurement-report; not-a-verdict", + } + if workspace is not None: + root = Path(workspace) + ledger = EvidenceLedger(root / "distill-evidence.jsonl") + for record in development_records: + ledger.append("study-source-record", record.to_dict(), actor="mnel-distillation-study") + for record in hidden_records: + ledger.append("study-hidden-source-metadata", record.to_dict(), actor="mnel-transfer-evaluator") + for record_type, value in ( + ("semantic-group", groups), + ("distilled-principle", (principle,)), + ("negative-memory", (negative,)), + ("distilled-strategy", (strategy,)), + ("study-arm", specification.arms), + ("ablation-spec", specification.ablations), + ("retrieval-result", (retrieval, risky_retrieval, ablated_retrieval)), + ("retrieval-metrics", (retrieval_metrics,)), + ("calibration-metrics", (calibration,)), + ("shuffled-attribution-control", (shuffled,)), + ("transfer-prediction", (prediction,)), + ("transfer-evaluation", (evaluation,)), + ): + for item in value: + payload = item.to_dict() if hasattr(item, "to_dict") else item + ledger.append(record_type, payload, actor="mnel-distillation-study") + ledger.append( + "learned-provider-artifact", + provider_model.to_dict(), + actor="mnel-distillation-study", + ) + ledger.append( + "learned-provider-observation", + provider_observation.to_dict(), + actor="mnel-reference-provider", + ) + ledger.append( + "provider-baseline-comparison", + provider_baseline_comparison, + actor="mnel-distillation-study", + ) + ledger.append("distillation-study-report", report, actor="mnel-distillation-study") + report["ledger"] = ledger.summarize() + return { + "report": report, + "study_identity": study_identity, + "strategy": strategy.to_dict(), + "negative_memory": negative.to_dict(), + "groups": [item.to_dict() for item in groups], + "retrieval": retrieval.to_dict(), + "risky_retrieval": risky_retrieval.to_dict(), + "ablated_retrieval": ablated_retrieval.to_dict(), + "shuffled_attribution": shuffled.to_dict(), + "prediction": prediction.to_dict(), + "evaluation": evaluation.to_dict(), + "specification": specification.to_dict(), + "provider_model": provider_model.to_dict(), + "provider_observation": provider_observation.to_dict(), + "provider_baseline_comparison": provider_baseline_comparison, + } diff --git a/src/mnel/forge_lifecycle.py b/src/mnel/forge_lifecycle.py index 55474e7..2b81069 100644 --- a/src/mnel/forge_lifecycle.py +++ b/src/mnel/forge_lifecycle.py @@ -54,6 +54,7 @@ "mncds_verdict", } ) +MAX_QUESTION_CANDIDATES = 32 class ForgeLifecycleError(ValueError): @@ -1161,12 +1162,18 @@ class QuestionCandidate: target_snapshot_type: str | None supporting_record_ids: tuple[str, ...] authority: str = AUTHORITY_PROPOSAL_ONLY + candidate_kind: str = "coverage-gap" + priority: int = 0 + lineage: tuple[str, ...] = () def __post_init__(self) -> None: _nonempty(self.subject_identity, "candidate subject identity") _nonempty(self.reason, "candidate reason") - if self.authority != AUTHORITY_PROPOSAL_ONLY: + _nonempty(self.candidate_kind, "candidate kind") + if self.authority != AUTHORITY_PROPOSAL_ONLY or not 0 <= self.priority <= 100: raise ForgeLifecycleError("question candidates are proposal-only") + if not self.supporting_record_ids: + raise ForgeLifecycleError("question candidates require evidence lineage") @property def candidate_identity(self) -> str: @@ -1180,6 +1187,9 @@ def to_dict(self, *, include_identity: bool = True) -> dict[str, object]: "target_snapshot_type": self.target_snapshot_type, "supporting_record_ids": list(self.supporting_record_ids), "authority": self.authority, + "candidate_kind": self.candidate_kind, + "priority": self.priority, + "lineage": list(self.lineage or self.supporting_record_ids), "semantics": "proposal-only; not-a-verdict", } if include_identity: @@ -1192,21 +1202,163 @@ def discover_question_candidates( snapshots: SnapshotStore, coverage: CoverageRecord, comparisons: Sequence[WitnessComparison] = (), + witnesses: Sequence[Witness] = (), + mutations: Sequence[MutationRecord] = (), + learned_observations: Sequence[LearnedDiagnosticEvent] = (), + registry: VerifierRegistry | None = None, + visible_lineage: frozenset[str] | None = None, + max_candidates: int = MAX_QUESTION_CANDIDATES, ) -> tuple[QuestionCandidate, ...]: - candidates: list[QuestionCandidate] = [ - QuestionCandidate(subject_identity, "no compatible verifier exercised this snapshot type", snapshot_type, (coverage.coverage_identity,)) - for snapshot_type in coverage.uncovered_snapshot_types - ] - candidates.extend( - QuestionCandidate(subject_identity, "identified question has only one diagnostic verifier", None, (question_identity,)) - for question_identity in coverage.single_source_question_identities - ) - candidates.extend( - QuestionCandidate(subject_identity, "independent witnesses disagree", None, comparison.witness_identities) - for comparison in comparisons - if comparison.comparison_status == "disagreement" - ) - return tuple(sorted(candidates, key=lambda item: item.candidate_identity)) + if max_candidates < 1 or max_candidates > MAX_QUESTION_CANDIDATES: + raise ForgeLifecycleError("question candidate budget is outside its bounded range") + candidates: dict[str, QuestionCandidate] = {} + + def add( + reason: str, + target_snapshot_type: str | None, + supporting: Sequence[str], + *, + candidate_kind: str, + priority: int, + ) -> None: + lineage = tuple(dict.fromkeys(str(item) for item in supporting if str(item).strip())) + if not lineage: + raise ForgeLifecycleError("skeptic candidate has no evidence lineage") + if visible_lineage is not None and not set(lineage).issubset(visible_lineage): + raise ForgeLifecycleError("skeptic candidate attempted to use unavailable evidence") + candidate = QuestionCandidate( + subject_identity, + reason, + target_snapshot_type, + lineage, + candidate_kind=candidate_kind, + priority=priority, + lineage=lineage, + ) + candidates[candidate.candidate_identity] = candidate + + for snapshot_type in coverage.uncovered_snapshot_types: + add( + "no compatible verifier exercised this snapshot type", + snapshot_type, + (coverage.coverage_identity,), + candidate_kind="missing-verifier-coverage", + priority=90, + ) + for question_identity in coverage.single_source_question_identities: + add( + "identified question has only one diagnostic verifier", + None, + (question_identity,), + candidate_kind="single-diagnostic-source", + priority=70, + ) + for comparison in comparisons: + if comparison.comparison_status == "disagreement": + add( + "independent witnesses disagree", + None, + comparison.witness_identities, + candidate_kind="witness-disagreement", + priority=100, + ) + status_groups: dict[str, list[Witness]] = {} + for witness in witnesses: + if witness.execution_status in { + ProbeExecutionStatus.INELIGIBLE, + ProbeExecutionStatus.NOT_APPLICABLE, + ProbeExecutionStatus.ABSTAINED, + ProbeExecutionStatus.UNAVAILABLE, + ProbeExecutionStatus.ERROR, + }: + status_groups.setdefault(witness.question_identity, []).append(witness) + add( + f"verifier execution was {witness.execution_status.value}", + None, + (witness.witness_identity,), + candidate_kind="verifier-abstention-or-error", + priority=80, + ) + for failed in status_groups.values(): + if len(failed) >= 2: + add( + "question has repeated unavailable or ineligible diagnostic outcomes", + None, + tuple(item.witness_identity for item in failed), + candidate_kind="repeated-unknown", + priority=85, + ) + witness_snapshot_ids = { + identity for witness in witnesses for identity in witness.snapshot_identities + } + for snapshot_identity in snapshots.identities(): + if snapshot_identity not in witness_snapshot_ids: + snapshot = snapshots.get(snapshot_identity) + add( + "identified snapshot has no compatible verifier execution", + snapshot.snapshot_type, + (snapshot_identity,), + candidate_kind="snapshot-unprobed", + priority=78, + ) + for mutation in mutations: + result_identity = mutation.resulting_snapshot_identity + if not any(result_identity in witness.snapshot_identities for witness in witnesses): + add( + "registered mutation has no corresponding counterfactual probe", + None, + (mutation.mutation_identity, mutation.original_snapshot_identity), + candidate_kind="missing-counterfactual-probe", + priority=88, + ) + if mutation.original_snapshot_identity not in witness_snapshot_ids: + add( + "registered mutation has no identified original probe", + None, + (mutation.mutation_identity,), + candidate_kind="missing-original-probe", + priority=82, + ) + mutated_originals = {item.original_snapshot_identity for item in mutations} + for witness in witnesses: + if witness.execution_status is ProbeExecutionStatus.COMPLETED and not any( + identity in mutated_originals for identity in witness.snapshot_identities + ): + add( + "completed original probe has no registered mutation counterpart", + None, + (witness.witness_identity,), + candidate_kind="missing-mutation-counterpart", + priority=60, + ) + for event in learned_observations: + learned_value = event.payload.get("condition_observed") + for witness in witnesses: + if ( + witness.execution_status is ProbeExecutionStatus.COMPLETED + and any(identity in event.snapshot_identities for identity in witness.snapshot_identities) + and isinstance(learned_value, bool) + and learned_value != witness.diagnostic_output.get("condition_observed") + ): + add( + "learned-provider observation disagrees with a deterministic witness", + None, + (event.event_identity, witness.witness_identity), + candidate_kind="learned-diagnostic-disagreement", + priority=95, + ) + if registry is not None: + for declaration in registry.declarations(): + if registry.state(declaration.verifier_id) is VerifierState.QUARANTINED: + add( + "quarantined verifier creates a diagnostic coverage hole", + declaration.accepted_snapshot_types[0], + (declaration.declaration_identity,), + candidate_kind="verifier-health-hole", + priority=75, + ) + values = sorted(candidates.values(), key=lambda item: (-item.priority, item.candidate_identity)) + return tuple(values[:max_candidates]) def run_reference_forge_study(workspace: str | Path | None = None) -> dict[str, Any]: diff --git a/src/mnel/reference_provider.py b/src/mnel/reference_provider.py new file mode 100644 index 0000000..aedf862 --- /dev/null +++ b/src/mnel/reference_provider.py @@ -0,0 +1,247 @@ +"""Tiny deterministic learned-provider study implementation. + +The transition-frequency model is intentionally small and inspectable. It is a learned +diagnostic provider, not a verifier and not an evaluator. Training is restricted to an +explicit development access view; hidden transfer records cannot enter the model. +""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import dataclass +from typing import Any + +from .core import Visibility, canonical_digest, canonical_json +from .distillation import ( + AUTHORITY_DIAGNOSTIC_ONLY, + DistillationError, + StudyDataAccess, + _reject_authority, +) +from .snapshots import DiagnosticSnapshot, SnapshotStore, SnapshotError, TransitionView, decode_snapshot + + +@dataclass(frozen=True, slots=True) +class LearnedProviderObservation: + provider_id: str + model_identity: str + snapshot_identity: str + score: float + abstained: bool + out_of_distribution: bool + calibration_identity: str + observation_identity: str = "" + authority: str = AUTHORITY_DIAGNOSTIC_ONLY + + def __post_init__(self) -> None: + if not self.provider_id or not self.model_identity or not self.snapshot_identity: + raise DistillationError("provider observations require identities") + if not 0.0 <= self.score <= 1.0: + raise DistillationError("provider score must be within [0, 1]") + if not self.calibration_identity or self.authority != AUTHORITY_DIAGNOSTIC_ONLY: + raise DistillationError("provider observations must remain diagnostic-only") + if self.observation_identity and self.observation_identity != self.content_identity: + raise DistillationError("provider observation identity does not match content") + + @property + def content_identity(self) -> str: + return canonical_digest(self.to_dict(include_identity=False)) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-learned-provider-observation/0.4", + "record_type": "learned-provider-observation", + "provider_id": self.provider_id, + "model_identity": self.model_identity, + "snapshot_identity": self.snapshot_identity, + "score": self.score, + "abstained": self.abstained, + "out_of_distribution": self.out_of_distribution, + "calibration_identity": self.calibration_identity, + "authority": self.authority, + "semantics": "learned-diagnostic-observation; not-a-verdict", + } + if include_identity: + value["observation_identity"] = self.observation_identity or self.content_identity + return value + + +@dataclass(frozen=True, slots=True) +class TransitionFrequencyModel: + training_dataset_identity: str + training_record_ids: tuple[str, ...] + feature_extractor_identity: str + training_code_identity: str + calibration_identity: str + transition_counts: dict[str, int] + total_count: int + model_identity: str = "" + artifact_identity: str = "" + provider_id: str = "mnel-reference-transition-frequency/0.4" + authority: str = AUTHORITY_DIAGNOSTIC_ONLY + + def __post_init__(self) -> None: + if not self.provider_id or not self.training_record_ids or self.total_count < 1: + raise DistillationError("trained provider requires non-empty development data") + if sum(self.transition_counts.values()) != self.total_count: + raise DistillationError("transition counts do not match total count") + if any(not isinstance(key, str) or not isinstance(value, int) or value < 1 for key, value in self.transition_counts.items()): + raise DistillationError("transition counts are malformed") + if not all((self.training_dataset_identity, self.feature_extractor_identity, self.training_code_identity, self.calibration_identity)): + raise DistillationError("provider training identities are required") + if self.authority != AUTHORITY_DIAGNOSTIC_ONLY: + raise DistillationError("learned providers are diagnostic-only") + if self.model_identity and self.model_identity != self.content_identity: + raise DistillationError("provider model identity does not match content") + + @property + def content_identity(self) -> str: + return canonical_digest(self.to_dict(include_identity=False)) + + @property + def model_size_bytes(self) -> int: + return len(canonical_json(self.to_dict())) + + def to_dict(self, *, include_identity: bool = True) -> dict[str, Any]: + value = { + "schema": "mnel-learned-provider-artifact/0.4", + "provider_id": self.provider_id, + "training_dataset_identity": self.training_dataset_identity, + "training_record_ids": list(self.training_record_ids), + "feature_extractor_identity": self.feature_extractor_identity, + "training_code_identity": self.training_code_identity, + "calibration_identity": self.calibration_identity, + "transition_counts": dict(sorted(self.transition_counts.items())), + "total_count": self.total_count, + "authority": self.authority, + "semantics": "learned-provider-artifact; diagnostic-only; not-a-verdict", + } + if include_identity: + value["model_identity"] = self.model_identity or self.content_identity + return value + + def serialize(self) -> bytes: + value = self.to_dict() + value["artifact_identity"] = "sha256:" + hashlib.sha256(canonical_json(value)).hexdigest() + return canonical_json(value) + + @classmethod + def load(cls, payload: bytes) -> "TransitionFrequencyModel": + try: + value = json.loads(payload) + except (TypeError, json.JSONDecodeError) as error: + raise DistillationError("provider artifact is not valid JSON") from error + if not isinstance(value, dict) or value.get("schema") != "mnel-learned-provider-artifact/0.4": + raise DistillationError("unsupported provider artifact schema") + _reject_authority(value) + supplied_artifact = value.pop("artifact_identity", None) + if not isinstance(supplied_artifact, str): + raise DistillationError("provider artifact identity is missing") + canonical_payload = canonical_json(value) + expected_artifact = "sha256:" + hashlib.sha256(canonical_payload).hexdigest() + if supplied_artifact != expected_artifact: + raise DistillationError("provider artifact bytes do not match artifact identity") + model = cls( + training_dataset_identity=value.get("training_dataset_identity"), + training_record_ids=tuple(value.get("training_record_ids", ())), + feature_extractor_identity=value.get("feature_extractor_identity"), + training_code_identity=value.get("training_code_identity"), + calibration_identity=value.get("calibration_identity"), + transition_counts=dict(value.get("transition_counts", {})), + total_count=value.get("total_count"), + model_identity=value.get("model_identity", ""), + provider_id=value.get("provider_id", ""), + authority=value.get("authority", ""), + ) + if model.model_identity != model.content_identity: + raise DistillationError("provider artifact model identity is invalid") + object.__setattr__(model, "artifact_identity", supplied_artifact) + return model + + def infer(self, snapshot: DiagnosticSnapshot) -> LearnedProviderObservation: + try: + view = decode_snapshot(snapshot) + except SnapshotError as error: + raise DistillationError("provider cannot decode snapshot") from error + if not isinstance(view, TransitionView): + raise DistillationError("transition-frequency provider requires a transition snapshot") + key = _transition_key(view) + count = self.transition_counts.get(key, 0) + if count == 0: + return LearnedProviderObservation( + self.provider_id, + self.model_identity or self.content_identity, + snapshot.snapshot_identity, + 0.0, + True, + True, + self.calibration_identity, + ) + frequency = count / self.total_count + return LearnedProviderObservation( + self.provider_id, + self.model_identity or self.content_identity, + snapshot.snapshot_identity, + max(0.0, min(1.0, frequency)), + False, + False, + self.calibration_identity, + ) + + +def _transition_key(view: TransitionView) -> str: + return hashlib.sha256(view.previous_state + b"\x00" + view.next_state).hexdigest() + + +def train_transition_frequency( + access: StudyDataAccess, + snapshots: SnapshotStore, + *, + record_type: str = "experience-episode", + feature_extractor_identity: str = "mnel-transition-bytes/0.4", + training_code_identity: str = "mnel-reference-frequency-training/0.4", + calibration_identity: str = "mnel-reference-calibration/0.4", +) -> TransitionFrequencyModel: + if access.purpose != "development-study" or any( + item.visibility not in {Visibility.DEVELOPMENT, Visibility.SELECTION_OBSERVED} + for item in access.records() + ): + raise DistillationError("provider training requires a development-only access view") + counts: dict[str, int] = {} + record_ids: list[str] = [] + for record in access.records(record_type): + snapshot_identity = record.payload.get("snapshot_identity") + if not isinstance(snapshot_identity, str): + raise DistillationError("training record lacks a snapshot identity") + snapshot = snapshots.get(snapshot_identity) + view = decode_snapshot(snapshot) + if not isinstance(view, TransitionView): + raise DistillationError("training dataset contains a non-transition snapshot") + key = _transition_key(view) + counts[key] = counts.get(key, 0) + 1 + record_ids.append(record.identity) + if not record_ids: + raise DistillationError("training dataset contains no eligible records") + draft = TransitionFrequencyModel( + access.dataset_identity, + tuple(sorted(record_ids)), + feature_extractor_identity, + training_code_identity, + calibration_identity, + counts, + len(record_ids), + ) + model = TransitionFrequencyModel( + draft.training_dataset_identity, + draft.training_record_ids, + draft.feature_extractor_identity, + draft.training_code_identity, + draft.calibration_identity, + draft.transition_counts, + draft.total_count, + draft.content_identity, + ) + artifact_payload = canonical_json(model.to_dict()) + object.__setattr__(model, "artifact_identity", "sha256:" + hashlib.sha256(artifact_payload).hexdigest()) + return model diff --git a/tests/test_distillation.py b/tests/test_distillation.py new file mode 100644 index 0000000..0050932 --- /dev/null +++ b/tests/test_distillation.py @@ -0,0 +1,186 @@ +import unittest +import tempfile +import json +from pathlib import Path + +from mnel.core import TransferStatus, Visibility +from mnel.distillation import ( + AblationKind, + AblationSpec, + DistillationError, + StudyArm, + StudyArmKind, + StudyDataAccess, + StudySpecification, + TransferWorkflow, + VisibilityViolation, + build_distilled_strategy, + calculate_calibration, + make_study_record, + reference_feature_groups, + RetrievalIndex, + run_reference_distill_study, + shuffle_attributions, +) +from mnel.reference_provider import train_transition_frequency +from mnel.snapshots import SnapshotStore, transition_snapshot + + +class DistillationTests(unittest.TestCase): + def setUp(self) -> None: + identities = { + "producer_identity": "producer:v1", + "source_identity": "source:v1", + "dependency_identity": "dependency:v1", + "feature_extractor_identity": "extractor:v1", + } + self.snapshots = SnapshotStore() + self.first = transition_snapshot(b"cold", b"warm", **identities) + self.second = transition_snapshot(b"hot", b"warm", **identities) + self.snapshots.register(self.first) + self.snapshots.register(self.second) + self.records = ( + make_study_record( + "experience-episode", + {"snapshot_identity": self.first.snapshot_identity, "artifact_type": "routing", "tags": ["transition"]}, + record_identity="sha256:episode-a", + ), + make_study_record( + "experience-episode", + {"snapshot_identity": self.second.snapshot_identity, "artifact_type": "routing", "tags": ["transition"]}, + record_identity="sha256:episode-b", + ), + make_study_record( + "transfer-outcome", + {"outcome": "supported"}, + visibility=Visibility.TRANSFER_HIDDEN, + record_identity="sha256:hidden", + ), + make_study_record( + "future-final", + {"outcome": "final"}, + visibility=Visibility.FUTURE_FINAL, + record_identity="sha256:future", + ), + ) + + def test_visibility_access_fails_closed_and_groups_preserve_sources(self) -> None: + development = StudyDataAccess.development(self.records) + self.assertEqual(len(development.records()), 2) + with self.assertRaises(VisibilityViolation): + development.get("sha256:hidden") + with self.assertRaises(VisibilityViolation): + development.get("sha256:future") + groups = reference_feature_groups(development) + self.assertEqual(sorted(item.source_record_ids for item in groups), [("sha256:episode-a", "sha256:episode-b")]) + self.assertNotEqual(groups[0].group_identity, "") + + def test_hidden_records_cannot_train_provider(self) -> None: + development = StudyDataAccess.development(self.records) + model = train_transition_frequency(development, self.snapshots) + reloaded = type(model).load(model.serialize()) + self.assertEqual(model.model_identity, reloaded.model_identity) + self.assertEqual(model.artifact_identity, reloaded.artifact_identity) + self.assertEqual(model.infer(self.first).to_dict(), reloaded.infer(self.first).to_dict()) + with self.assertRaises(DistillationError): + train_transition_frequency(StudyDataAccess.transfer_evaluator(self.records), self.snapshots) + with self.assertRaises(VisibilityViolation): + RetrievalIndex().add_source_records((self.records[2],)) + + def test_provider_abstains_on_unseen_transition_and_authority_fields_are_rejected(self) -> None: + model = train_transition_frequency(StudyDataAccess.development(self.records), self.snapshots) + unseen = transition_snapshot( + b"unknown", + b"state", + producer_identity="producer:v1", + source_identity="source:unseen", + dependency_identity="dependency:v1", + feature_extractor_identity="extractor:v1", + ) + observation = model.infer(unseen) + self.assertTrue(observation.abstained) + self.assertTrue(observation.out_of_distribution) + self.assertNotIn("verdict", observation.to_dict()) + with self.assertRaises(DistillationError): + make_study_record("episode", {"verdict": "PASS"}) + + def test_transfer_prediction_is_frozen_and_same_candidate_repair_is_rejected(self) -> None: + strategy = build_distilled_strategy( + trigger_conditions=("transition",), + preconditions=("routing",), + expected_effect={"metric": "increase"}, + known_failure_modes=("unsupported",), + negative_memory_ids=("memory:v1",), + counterexample_record_ids=("record:counterexample",), + causal_attribution_ids=("attribution:v1",), + supporting_source_record_ids=("episode:v1",), + transfer_evidence_ids=(), + scope={"artifact_type": "routing"}, + ) + workflow = TransferWorkflow() + prediction = workflow.freeze_prediction( + strategy, + transfer_environment_identity="environment:hidden", + predicted_effect={"metric": "increase"}, + ) + evaluation = workflow.finalize( + prediction, + observed_outcome_identity="outcome:hidden", + evaluator_evidence_identity="evidence:evaluator", + status=TransferStatus.FAILED, + ) + with self.assertRaises(VisibilityViolation): + workflow.reject_same_candidate_repair(strategy.strategy_identity, evaluation) + self.assertNotIn("verdict", evaluation.to_dict()) + + def test_controls_are_deterministic_and_equal_budget_is_checked(self) -> None: + shuffled_a = shuffle_attributions(("a", "b", "c"), seed=11, source_study_identity="study:v1") + shuffled_b = shuffle_attributions(("a", "b", "c"), seed=11, source_study_identity="study:v1") + self.assertEqual(shuffled_a.to_dict(), shuffled_b.to_dict()) + budget = {"operations": 10, "wall_seconds": 10, "candidates": 2} + arms = ( + StudyArm("a", StudyArmKind.RANDOM, ("development",), ("hidden",), "random", (), False, False, 0, budget), + StudyArm("b", StudyArmKind.ATTRIBUTION, ("development",), ("hidden",), "lineage", (), True, True, 0, budget), + ) + specification = StudySpecification( + "study:v1", "development:v1", "hidden:v1", arms, + (AblationSpec("ablation", AblationKind.NEGATIVE_MEMORY, arms[0].arm_identity, 11, {}),), + budget, + ) + self.assertTrue(specification.study_identity) + with self.assertRaises(DistillationError): + StudySpecification( + "study:v2", "development:v1", "hidden:v1", arms, + (), {"operations": 9, "wall_seconds": 10, "candidates": 2}, + ) + + def test_calibration_and_reference_study_are_measured(self) -> None: + metrics = calculate_calibration( + (0.9, None, 0.1), (True, False, False), + dataset_identity="dataset:v1", model_identity="model:v1", + out_of_distribution=(False, True, False), + ) + self.assertEqual(metrics.count, 3) + self.assertGreaterEqual(metrics.coverage, 0.0) + result = run_reference_distill_study() + repeat = run_reference_distill_study() + self.assertEqual(result["study_identity"], repeat["study_identity"]) + self.assertTrue(result["report"]["negative_memory_demoted_strategy"]) + self.assertEqual(result["report"]["transfer_status"], "supported") + self.assertTrue(result["report"]["retrieval_metrics"]["metric_identity"]) + + with tempfile.TemporaryDirectory() as workspace: + written = run_reference_distill_study(workspace) + self.assertTrue(written["report"]["ledger"]["valid"]) + self.assertEqual(written["report"]["ledger"]["record_count"], 35) + + def test_distillation_schema_covers_durable_reference_records(self) -> None: + root = Path(__file__).resolve().parents[1] + schema = json.loads((root / "schemas" / "mnel-distillation.schema.json").read_text()) + self.assertEqual(schema["$schema"], "https://json-schema.org/draft/2020-12/schema") + for definition in ("group", "negativeMemory", "strategy", "studySpec", "providerObservation", "studyReport"): + self.assertIn(definition, schema["$defs"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_forge_lifecycle.py b/tests/test_forge_lifecycle.py index 70a3378..4af0efe 100644 --- a/tests/test_forge_lifecycle.py +++ b/tests/test_forge_lifecycle.py @@ -237,6 +237,53 @@ def test_coverage_surfaces_single_source_questions(self) -> None: candidates = discover_question_candidates("source:v1", self.store, coverage) self.assertTrue(candidates) + def test_skeptic_discovery_is_bounded_deduplicated_and_proposal_only(self) -> None: + failed_request = self._request( + probe_id="failed-probe-1", + preconditions=(Precondition("required_snapshot_type", "tabular"),), + ) + failed = self.runtime.execute(failed_request) + failed_again = self.runtime.execute( + self._request( + probe_id="failed-probe-2", + preconditions=(Precondition("required_snapshot_type", "tabular"),), + ) + ) + mutation = MutationRegistry().apply("transition.swap", self.snapshot, {}, self.store) + learned = LearnedDiagnosticEvent.from_observation( + { + "provider_id": "provider:v1", + "observation_identity": "observation:disagree", + "snapshot_ids": [self.snapshot.snapshot_identity], + "declaration_identity": "declaration:v1", + "condition_observed": False, + } + ) + coverage = build_coverage(self.registry, self.store, (failed, failed_again)) + candidates = discover_question_candidates( + "source:v1", + self.store, + coverage, + witnesses=(failed, failed_again), + mutations=(mutation,), + learned_observations=(learned,), + registry=self.registry, + max_candidates=12, + ) + self.assertLessEqual(len(candidates), 12) + self.assertEqual(len({item.candidate_identity for item in candidates}), len(candidates)) + self.assertIn("repeated-unknown", {item.candidate_kind for item in candidates}) + self.assertTrue(all(item.authority == "proposal-only" for item in candidates)) + self.assertTrue(all("verdict" not in item.to_dict() for item in candidates)) + with self.assertRaises(ForgeLifecycleError): + discover_question_candidates( + "source:v1", + self.store, + coverage, + witnesses=(failed,), + visible_lineage=frozenset({"development-only-id"}), + ) + if __name__ == "__main__": unittest.main()