diff --git a/CHANGELOG.md b/CHANGELOG.md index 20b7289a..ea721ce5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -115,7 +115,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang - `prompt_source` identity gate: instruction and prompt boilerplate is not unique latent content and is not erased by a stopword list; `identity_recovery_rate` reports exact kind matches, with a contract test comparing correct recovery with an all-unique collapse on a mixed known-truth fixture (ADR 0004/0012). - `location_membership` identity gate: geographic and market assignments are time-varying memberships, not permanent entity identity and not language channels; recovered location kinds match known truth at a higher computed rate than collapsing every assignment to entity identity (ADR 0003). - `membership_target` identity gate: language, episode, template, department, and opportunity-pool memberships cannot collapse into the entity/project pair stored by migration `0006`; comparison-contract tests record recovered target kinds against an entity-collapse baseline (ADR 0003). - +- `event_core` first-story detection gate: first-story versus follow-up labels stay distinct from promoted instances, false-alarm and miss rates are computed from known truth, and calibrated detection scores recover the binary first-story target with lower RMSE than an always-first detector. - `tepp_api` naruon live loopback HTTP/1.1 listener: `serve_one` installs a read/write deadline, requires a loopback `Host`, refuses `Transfer-Encoding` and NIM/proxy credential headers, parses `knowledge_cutoff` as RFC 3339 and refuses a future cutoff, keys analysis-run idempotency by tenant plus key, and proves both analysis-run and export POSTs over a real `TcpStream`. Not a production TLS/`$PORT` service (ADR 0011). - `tepp_api` adaptive orchestration router (ADR 0010): versioned `direct`/`verify`/`committee`/`conductor`/`abstain` selection from CPU `f64` risk, ambiguity, evidence, and token-budget inputs; recorded stages, recursion, decomposition, access lists, and role-specific reasoning effort; fail-closed document-controlled policy/access/credentials; LLM plans remain proposals under deterministic statistical authority; comparable-budget ablation requires a direct baseline; credential-free contextual-orchestrator binding. Live NIM HTTP remains accepted-target. - `tepp_api` purpose-bound provider-payload minimization: time-bounded `PurposeGrant` evaluation, fail-closed expired/not-yet-valid/inverted/cross-tenant/impossible-calendar denial, semantic UTC calendar validation, refusal to copy identity mappings into model-provider payloads or ordinary logs, preservation of opaque analytical identifiers and membership roles (no blanket PII mask), a separately authorized scientific re-identification path, and an internally bound FIPS 180-4 SHA-256 audit digest appended through `ReidentificationAuditSink` before disclosure. diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index 7bf758d8..f7733223 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -38,6 +38,7 @@ TEPP's approved PRD v0.4 and implementation plan are the primary product baselin | Actions fleet research doctoring | [`docs/research/actions-workflow-fleet.md`](docs/research/actions-workflow-fleet.md) | | Mention-confidence Brier doctoring | [`docs/research/mention-confidence-brier.md`](docs/research/mention-confidence-brier.md) | | Event-intelligence status-gate doctoring | [`docs/research/event-intelligence-status-gates.md`](docs/research/event-intelligence-status-gates.md) | +| First-story detection FAR/miss doctoring | [`docs/research/first-story-detection-calibration.md`](docs/research/first-story-detection-calibration.md) | | VRAM budget / GPU fallback doctoring | [`docs/research/vram-budget-types.md`](docs/research/vram-budget-types.md) | | Causal-identification gate doctoring | [`docs/research/causal-identification-gate.md`](docs/research/causal-identification-gate.md) | | TDT story-segmentation `WindowDiff`/`Pk` doctoring | [`docs/research/tdt-story-segmentation.md`](docs/research/tdt-story-segmentation.md) | diff --git a/crates/event_core/src/error.rs b/crates/event_core/src/error.rs index d862bf73..1ab853f8 100644 --- a/crates/event_core/src/error.rs +++ b/crates/event_core/src/error.rs @@ -24,6 +24,10 @@ pub enum EventError { DetectionIsNotTransition, /// A CHRONOS prediction was treated as an observed or promoted fact. PredictionIsNotFact, + /// A first-story detection was treated as an event instance. + FirstStoryIsNotEventInstance, + /// An unknown first-story label name was supplied. + UnknownFirstStoryLabel, /// A TDT track assignment was treated as an event instance. EventTrackIsNotEventInstance, /// A TDT track assignment was treated as a state transition. @@ -60,6 +64,8 @@ impl fmt::Display for EventError { Self::UnknownEventRole => "unknown event role", Self::DetectionIsNotTransition => "detection is not a state transition", Self::PredictionIsNotFact => "prediction is not an observed fact", + Self::FirstStoryIsNotEventInstance => "first-story detection is not an event instance", + Self::UnknownFirstStoryLabel => "unknown first-story label", Self::EventTrackIsNotEventInstance => "event track is not an event instance", Self::EventTrackIsNotStateTransition => "event track is not a state transition", Self::UnknownEventTrackLabel => "unknown event track label", @@ -121,6 +127,12 @@ mod tests { "prediction is not an observed fact", ), ( + EventError::FirstStoryIsNotEventInstance, + "first-story detection is not an event instance", + ), + ( + EventError::UnknownFirstStoryLabel, + "unknown first-story label", EventError::EventTrackIsNotEventInstance, "event track is not an event instance", ), diff --git a/crates/event_core/src/first_story.rs b/crates/event_core/src/first_story.rs new file mode 100644 index 00000000..0e36a42f --- /dev/null +++ b/crates/event_core/src/first_story.rs @@ -0,0 +1,175 @@ +//! First-story detection scores stay distinct from promoted instances. + +use crate::{EventConfidence, EventError, EventInstanceId, EventMentionId}; + +/// TDT first-story versus follow-up label. +/// +/// A first-story decision is detection evidence. It is never a promoted event +/// instance and cannot create a forward state transition by itself. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum FirstStoryLabel { + /// The mention is scored as the onset of a new story. + FirstStory, + /// The mention is scored as a continuation of an earlier story. + FollowUp, +} + +impl FirstStoryLabel { + /// Return the stable wire label name. + #[must_use] + pub const fn wire_name(self) -> &'static str { + match self { + Self::FirstStory => "first_story", + Self::FollowUp => "follow_up", + } + } + + /// Parse a stable wire first-story label. + /// + /// # Errors + /// + /// Returns [`EventError::UnknownFirstStoryLabel`] for unrecognized names. + pub fn from_wire_name(name: &str) -> Result { + match name { + "first_story" => Ok(Self::FirstStory), + "follow_up" => Ok(Self::FollowUp), + _ => Err(EventError::UnknownFirstStoryLabel), + } + } + + /// Return whether this label is a first-story detection. + #[must_use] + pub const fn is_first_story(self) -> bool { + matches!(self, Self::FirstStory) + } + + /// Return the binary probability target used for RMSE. + /// + /// First-story truth is `1.0`; follow-up truth is `0.0`. + #[must_use] + pub const fn as_probability_target(self) -> f64 { + match self { + Self::FirstStory => 1.0, + Self::FollowUp => 0.0, + } + } +} + +/// Threshold a first-story probability into a detection label. +/// +/// The threshold is inclusive: `probability >= threshold` is a first story. +#[must_use] +pub fn decide_first_story( + probability: EventConfidence, + threshold: EventConfidence, +) -> FirstStoryLabel { + if probability.value() >= threshold.value() { + FirstStoryLabel::FirstStory + } else { + FirstStoryLabel::FollowUp + } +} + +/// Explicit refusal to treat a first-story detection as an event instance. +/// +/// # Errors +/// +/// Always returns [`EventError::FirstStoryIsNotEventInstance`]. +pub fn refuse_first_story_as_instance( + _mention_id: EventMentionId, +) -> Result { + Err(EventError::FirstStoryIsNotEventInstance) +} + +/// False-alarm rate: follow-ups labeled first story, over follow-up truth. +/// +/// # Errors +/// +/// Returns [`EventError::InvalidWirePayload`] when lengths differ, either +/// slice is empty, or the truth stream contains no follow-up. +pub fn first_story_false_alarm_rate( + truth: &[FirstStoryLabel], + decided: &[FirstStoryLabel], +) -> Result { + rate_over_class( + truth, + decided, + FirstStoryLabel::FollowUp, + FirstStoryLabel::FirstStory, + ) +} + +/// Miss rate: first stories labeled follow-up, over first-story truth. +/// +/// # Errors +/// +/// Returns [`EventError::InvalidWirePayload`] when lengths differ, either +/// slice is empty, or the truth stream contains no first story. +pub fn first_story_miss_rate( + truth: &[FirstStoryLabel], + decided: &[FirstStoryLabel], +) -> Result { + rate_over_class( + truth, + decided, + FirstStoryLabel::FirstStory, + FirstStoryLabel::FollowUp, + ) +} + +fn rate_over_class( + truth: &[FirstStoryLabel], + decided: &[FirstStoryLabel], + class: FirstStoryLabel, + error_label: FirstStoryLabel, +) -> Result { + if truth.is_empty() || truth.len() != decided.len() { + return Err(EventError::InvalidWirePayload); + } + let mut class_count = 0_u32; + let mut error_count = 0_u32; + for (truth_label, decided_label) in truth.iter().zip(decided) { + if *truth_label == class { + class_count += 1; + if *decided_label == error_label { + error_count += 1; + } + } + } + if class_count == 0 { + return Err(EventError::InvalidWirePayload); + } + Ok(f64::from(error_count) / f64::from(class_count)) +} + +#[cfg(test)] +mod tests { + use super::{ + FirstStoryLabel, decide_first_story, first_story_false_alarm_rate, first_story_miss_rate, + refuse_first_story_as_instance, + }; + use crate::{EventConfidence, EventError, EventMentionId}; + + #[test] + fn first_story_helpers_cover_local_branches() { + let mention = EventMentionId::new(); + assert_eq!( + refuse_first_story_as_instance(mention), + Err(EventError::FirstStoryIsNotEventInstance) + ); + let high = EventConfidence::new(0.8).expect("high"); + let low = EventConfidence::new(0.2).expect("low"); + assert_eq!(decide_first_story(high, low), FirstStoryLabel::FirstStory); + assert_eq!(decide_first_story(low, high), FirstStoryLabel::FollowUp); + let mixed_truth = [FirstStoryLabel::FirstStory, FirstStoryLabel::FollowUp]; + let mixed_decided = [FirstStoryLabel::FollowUp, FirstStoryLabel::FirstStory]; + assert!( + (first_story_false_alarm_rate(&mixed_truth, &mixed_decided).expect("far") - 1.0).abs() + < f64::EPSILON + ); + assert!( + (first_story_miss_rate(&mixed_truth, &mixed_decided).expect("miss") - 1.0).abs() + < f64::EPSILON + ); + } +} diff --git a/crates/event_core/src/lib.rs b/crates/event_core/src/lib.rs index 9a190cb1..4b64b908 100644 --- a/crates/event_core/src/lib.rs +++ b/crates/event_core/src/lib.rs @@ -4,6 +4,9 @@ //! //! TEPP separates **fallible event mentions** grounded in evidence from //! **versioned event instances** used for temporal state, multilevel membership, +//! and scientific estimation. Mentions and first-story detections never +//! silently become instances, and TDT detections and CHRONOS predictions +//! remain measurement or hypothesis artifacts until independently promoted. //! and scientific estimation. Mentions never silently become instances. TDT //! detections and CHRONOS predictions remain measurement or hypothesis //! artifacts until independently promoted. Track assignments, story @@ -13,6 +16,7 @@ mod confidence; mod error; +mod first_story; mod identifier; mod instance; mod intelligence; @@ -30,6 +34,16 @@ pub use confidence::EventConfidence; pub use confidence::mention_brier_score; /// Fail-closed event-ontology errors. pub use error::EventError; +/// First-story versus follow-up detection label. +pub use first_story::FirstStoryLabel; +/// Threshold a first-story probability into a detection label. +pub use first_story::decide_first_story; +/// False-alarm rate for first-story detections. +pub use first_story::first_story_false_alarm_rate; +/// Miss rate for first-story detections. +pub use first_story::first_story_miss_rate; +/// Explicit refusal to treat a first-story detection as an instance. +pub use first_story::refuse_first_story_as_instance; /// Opaque event-instance identifier. pub use identifier::EventInstanceId; /// Opaque event-mention identifier. diff --git a/crates/event_core/tests/first_story_contract.rs b/crates/event_core/tests/first_story_contract.rs new file mode 100644 index 00000000..4a189c6e --- /dev/null +++ b/crates/event_core/tests/first_story_contract.rs @@ -0,0 +1,166 @@ +//! First-story detections are not instances; FAR/miss are computed from truth. + +use event_core::{ + EventConfidence, EventError, EventMentionId, FirstStoryLabel, decide_first_story, + first_story_false_alarm_rate, first_story_miss_rate, refuse_first_story_as_instance, +}; + +fn computed_rmse(truth: &[f64], recovered: &[f64]) -> f64 { + assert_eq!(truth.len(), recovered.len()); + let n = f64::from(u32::try_from(truth.len()).expect("tiny fixture")); + let sse: f64 = truth + .iter() + .zip(recovered) + .map(|(truth_value, recovered_value)| { + let residual = truth_value - recovered_value; + residual * residual + }) + .sum(); + (sse / n).sqrt() +} + +fn decide_all(scores: &[f64], threshold: f64) -> Vec { + let cut = EventConfidence::new(threshold).expect("threshold"); + scores + .iter() + .map(|score| decide_first_story(EventConfidence::new(*score).expect("score"), cut)) + .collect() +} + +#[test] +fn first_story_detection_cannot_be_cast_to_an_instance() { + assert_eq!( + refuse_first_story_as_instance(EventMentionId::new()), + Err(EventError::FirstStoryIsNotEventInstance) + ); +} + +#[test] +fn false_alarm_and_miss_rates_are_computed_from_known_truth() { + let truth = [ + FirstStoryLabel::FirstStory, + FirstStoryLabel::FollowUp, + FirstStoryLabel::FollowUp, + FirstStoryLabel::FirstStory, + FirstStoryLabel::FollowUp, + FirstStoryLabel::FollowUp, + ]; + let calibrated = decide_all(&[0.90, 0.10, 0.15, 0.85, 0.20, 0.05], 0.50); + let always_first = decide_all(&[1.0, 1.0, 1.0, 1.0, 1.0, 1.0], 0.50); + + let calibrated_far = first_story_false_alarm_rate(&truth, &calibrated).expect("far"); + let naive_far = first_story_false_alarm_rate(&truth, &always_first).expect("naive far"); + let calibrated_miss = first_story_miss_rate(&truth, &calibrated).expect("miss"); + let naive_miss = first_story_miss_rate(&truth, &always_first).expect("naive miss"); + + assert!( + calibrated_far < naive_far, + "computed FAR {calibrated_far} must be below always-first FAR {naive_far}" + ); + assert!(calibrated_miss <= naive_miss); + assert!( + calibrated_far.abs() < 1e-15 && calibrated_miss.abs() < 1e-15, + "calibrated stream must recover FAR 0 and miss 0; far={calibrated_far} miss={calibrated_miss}" + ); +} + +#[test] +fn mixed_detection_errors_recover_half_far_and_half_miss() { + let truth = [ + FirstStoryLabel::FirstStory, + FirstStoryLabel::FollowUp, + FirstStoryLabel::FollowUp, + FirstStoryLabel::FirstStory, + ]; + let decided = [ + FirstStoryLabel::FirstStory, + FirstStoryLabel::FirstStory, + FirstStoryLabel::FollowUp, + FirstStoryLabel::FollowUp, + ]; + let far = first_story_false_alarm_rate(&truth, &decided).expect("far"); + let miss = first_story_miss_rate(&truth, &decided).expect("miss"); + let recovered = [far, miss]; + let truth_rates = [0.5_f64, 0.5]; + let rmse = computed_rmse(&truth_rates, &recovered); + assert!( + rmse < 1e-15, + "known-truth FAR/miss RMSE {rmse} (far={far} miss={miss})" + ); +} + +#[test] +fn calibrated_first_story_scores_have_lower_rmse_than_always_first() { + let truth_labels = [ + FirstStoryLabel::FirstStory, + FirstStoryLabel::FollowUp, + FirstStoryLabel::FollowUp, + FirstStoryLabel::FirstStory, + FirstStoryLabel::FollowUp, + FirstStoryLabel::FollowUp, + ]; + let truth: Vec = truth_labels + .iter() + .copied() + .map(FirstStoryLabel::as_probability_target) + .collect(); + let calibrated = [0.90_f64, 0.10, 0.15, 0.85, 0.20, 0.05]; + let always_first = [1.0_f64, 1.0, 1.0, 1.0, 1.0, 1.0]; + let calibrated_rmse = computed_rmse(&truth, &calibrated); + let naive_rmse = computed_rmse(&truth, &always_first); + assert!( + calibrated_rmse < naive_rmse, + "computed calibrated RMSE {calibrated_rmse} must be below always-first RMSE {naive_rmse}" + ); +} + +#[test] +fn rate_helpers_fail_closed_on_empty_mismatch_and_missing_class() { + let first = [FirstStoryLabel::FirstStory]; + let follow = [FirstStoryLabel::FollowUp]; + assert_eq!( + first_story_false_alarm_rate(&[], &[]), + Err(EventError::InvalidWirePayload) + ); + assert_eq!( + first_story_miss_rate(&first, &[]), + Err(EventError::InvalidWirePayload) + ); + assert_eq!( + first_story_false_alarm_rate(&first, &first), + Err(EventError::InvalidWirePayload) + ); + assert_eq!( + first_story_miss_rate(&follow, &follow), + Err(EventError::InvalidWirePayload) + ); +} + +#[test] +fn labels_round_trip_and_threshold_is_inclusive() { + assert_eq!(FirstStoryLabel::FirstStory.wire_name(), "first_story"); + assert_eq!(FirstStoryLabel::FollowUp.wire_name(), "follow_up"); + assert_eq!( + FirstStoryLabel::from_wire_name("first_story").expect("parse"), + FirstStoryLabel::FirstStory + ); + assert_eq!( + FirstStoryLabel::from_wire_name("follow_up").expect("parse"), + FirstStoryLabel::FollowUp + ); + assert_eq!( + FirstStoryLabel::from_wire_name("maybe_new"), + Err(EventError::UnknownFirstStoryLabel) + ); + assert!(FirstStoryLabel::FirstStory.is_first_story()); + assert!(!FirstStoryLabel::FollowUp.is_first_story()); + assert!((FirstStoryLabel::FirstStory.as_probability_target() - 1.0).abs() < f64::EPSILON); + assert!((FirstStoryLabel::FollowUp.as_probability_target() - 0.0).abs() < f64::EPSILON); + + let half = EventConfidence::new(0.5).expect("half"); + assert_eq!(decide_first_story(half, half), FirstStoryLabel::FirstStory); + assert_eq!( + decide_first_story(EventConfidence::new(0.49).expect("below"), half), + FirstStoryLabel::FollowUp + ); +} diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index 4d3df9ce..fdc71b6b 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -61,6 +61,9 @@ The full APA 7th standards/literature register remains `docs/research/standards- | candidate K statistical/Pareto gates + blinded LLM review | ADR 0012; research | future `model_selection` | accepted-target | | compositional topic correlation / stable clustering | ADR 0005/0012; research | future `network_analysis` | accepted-target | | posterior ESEM / longitudinal invariance / DSEM | ADR 0005 | `psychometric_fit` ESEM loading and DSEM lag gates on the active PR; `psychometric_core` input gates remain #49; invariance/multilevel remain accepted-target | active-PR | +| CPU bounded multithreading + GPU/VRAM streaming/parity | ADR 0001/0006 | future `compute_backend` | accepted-target | +| TDT detection/tracking vs CHRONOS schema/prediction/temporal consistency | ADR 0016; PRD/research | `event_core` first-story FAR/miss and mention-confidence Brier scoring on the active PR; remaining TDT/CHRONOS stack and any future `event_intelligence` crate remain accepted-target | active-PR | +| evidence-bounded LLM interpretation | ADR 0010/0012; PRD | `tepp_api` router implemented-main plus future `interpretation_gateway` | partial | | CPU bounded multithreading + GPU/VRAM streaming/parity | ADR 0001/0006 | `compute_backend` CPU `f64` reference, bounded planning, and VRAM-budget refusal are active; full GPU streaming and CPU/GPU parity remain future | partial | | TDT detection/tracking vs CHRONOS schema/prediction/temporal consistency | ADR 0016; PRD/research | `prediction_contradiction` bounded Allen promotion gate on the active PR (`refuse_promotion` requires coverage; `refuse_contradiction_or_adjacency` is not promotion authority; remaining TDT/CHRONOS tasks stay accepted-target) | active-PR | | TDT detection/tracking vs CHRONOS schema/prediction/temporal consistency | ADR 0016; PRD/research | `event_core` TDT tracking pair precision/recall and identity-switch rate on the active PR; remaining TDT/CHRONOS stack and any future `event_intelligence` crate remain accepted-target | active-PR | diff --git a/docs/adr/0016-tdt-chronos-event-intelligence-boundary.md b/docs/adr/0016-tdt-chronos-event-intelligence-boundary.md index d5438ee6..f9f9590b 100644 --- a/docs/adr/0016-tdt-chronos-event-intelligence-boundary.md +++ b/docs/adr/0016-tdt-chronos-event-intelligence-boundary.md @@ -1,5 +1,8 @@ # ADR 0016 — TDT, CHRONOS, and Event Ontology intelligence boundary +**Decision status:** Accepted +**Implementation maturity:** active-PR — evidence-layer admission, first-story false-alarm/miss rates, and detection-versus-instance refusal live in existing `event_core`, alongside the bounded predicted-vs-observed Allen promotion gate whose coverage authorization precedes any unmatched predicted mass; full TDT tracking/calibration and CHRONOS schema extraction/prediction layers remain accepted-target +**Date:** 2026-08-12 **Decision status:** Accepted **Implementation maturity:** active-PR — bounded predicted-vs-observed Allen promotion gate, including coverage before unmatched predicted mass may be authorized for promotion; TDT detection/tracking, CHRONOS schema extraction, prediction calibration, and path-consistency laws remain accepted-target diff --git a/docs/adr/README.md b/docs/adr/README.md index b1170d65..c780dcb2 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -66,8 +66,7 @@ Read [`ADR_POLICY.md`](ADR_POLICY.md) first. **Decision status and implementatio | [0013](0013-bitemporal-persistence-reproducibility-and-split-authority.md) | Bitemporal persistence, reproducibility manifests, and relation-aware split authority | Accepted | partial | Owns PostgreSQL adapter semantics, immutable run/split manifests, leakage-safe partitions, and recovery identity; optional `live-sqlx` `PgPool`, live PG CI, tenant RLS, and `0006` membership implemented-main; `0007` retention/deletion/legal-hold on the active PR; remaining physical ERD/backup accepted-target. | | [0014](0014-scientific-claim-promotion-and-release-evidence.md) | Scientific claim promotion and release evidence authority | Accepted | partial | Separates design, implementation, scientific/product claim, and release authority; repository SBOM/provenance generator implemented; checkpoint-versus-estimator refusal is `checkpoint_authority` on the active PR; full release bundle remaining. | | [0015](0015-autonomous-development-review-and-merge-authority.md) | Autonomous development, review, and merge authority separation | Accepted | active-PR | Separates model proposal, deterministic verification, publication, independent review, and merge/release authority. | -| [0016](0016-tdt-chronos-event-intelligence-boundary.md) | TDT, CHRONOS, and Event Ontology intelligence boundary | Accepted | active-PR | Bounded predicted-vs-observed Allen promotion gate: `refuse_promotion` requires observed coverage; remaining TDT/CHRONOS tasks stay accepted-target. | -| [0016](0016-tdt-chronos-event-intelligence-boundary.md) | TDT, CHRONOS, and Event Ontology intelligence boundary | Accepted | accepted-target | Separates observed evidence, detection/tracking, prediction/schema inference, temporal consistency, and promoted transition authority. | +| [0016](0016-tdt-chronos-event-intelligence-boundary.md) | TDT, CHRONOS, and Event Ontology intelligence boundary | Accepted | active-PR | First-story FAR/miss in existing `event_core`; remaining TDT/CHRONOS stack remains accepted-target. | | [0020](0020-span-grounded-semantic-units.md) | Span-grounded semantic units; language tags are not identity | Accepted | active-PR | First ADR 0004 production slice. Does not claim concept alignment, invariance, or a topic estimator. | | [0017](0017-hourly-contextual-orchestrator-gateway.md) | Hourly contextual-orchestrator gateway and all-provider model discovery | Accepted | active-PR | Keeps proposal-model execution behind a pinned loopback gateway while preserving independent verifier, publisher, reviewer, and merge authority. | | [0018](0018-consumer-scoped-analysis-run-ingress.md) | Consumer-scoped modular analysis-run ingress | Accepted | active-PR | Narrows ADR 0011 for the closed consumer registry, credential-free exchange, and consumer-qualified idempotency namespace; production TLS remains separate. | diff --git a/docs/research/first-story-detection-calibration.md b/docs/research/first-story-detection-calibration.md new file mode 100644 index 00000000..9e03f798 --- /dev/null +++ b/docs/research/first-story-detection-calibration.md @@ -0,0 +1,31 @@ +# First-story detection calibration + +## Scope + +This note doctors the `event_core` gate that keeps TDT first-story detection distinct from event-instance promotion: + +1. a first-story versus follow-up label is detection evidence, not a promoted instance; +2. false-alarm and miss rates are computed from known-truth labels; +3. calibrated first-story probabilities recover the binary onset target with lower RMSE than an always-first detector. + +No database migration is allocated. A later TDT tracker may consume these scores as measurement evidence only. + +## Authoritative sources + +Allan, J., Carbonell, J., Doddington, G., Yamron, J., & Yang, Y. (1998). Topic detection and tracking pilot study: Final report. In *Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop* (pp. 194–218). + +Allan, J. (Ed.). (2002). *Topic detection and tracking: Event-based information organization*. Kluwer Academic Publishers. + +Fiscus, J. G., & Doddington, G. R. (2002). Topic detection and tracking evaluation overview. In J. Allan (Ed.), *Topic detection and tracking: Event-based information organization* (pp. 17–31). Kluwer Academic Publishers. + +## Application + +Allan et al. (1998) and Allan (2002) define first-story detection as a *new-event onset* task whose official evaluation reports false-alarm and miss rates, not instance identity. Fiscus and Doddington (2002) keep those detection scores in the measurement layer. TEPP therefore refuses to cast a first-story label as an event instance and requires computed FAR, miss, and RMSE against known truth (Allan et al., 1998; Allan, 2002; Fiscus & Doddington, 2002). + +## Verification + +- `refuse_first_story_as_instance` always returns `FirstStoryIsNotEventInstance`; +- `decide_first_story` uses an inclusive probability threshold; +- `first_story_false_alarm_rate` and `first_story_miss_rate` fail closed on empty, mismatched, or single-class streams; +- computed RMSE of known first-story targets is lower under calibrated probabilities than under an always-first detector; +- a mixed stream with one false alarm and one miss recovers FAR `0.5` and miss `0.5` with residual RMSE below `1e-15`. diff --git a/docs/validation/temporal-event-foundation.md b/docs/validation/temporal-event-foundation.md index f3432e39..d70a6e45 100644 --- a/docs/validation/temporal-event-foundation.md +++ b/docs/validation/temporal-event-foundation.md @@ -20,6 +20,7 @@ This report tracks exact-head scientific and engineering evidence required befor | Knowledge-cutoff identity | `cutoff_clock` | active-PR | this PR | recovered cutoff flags vs availability-time stand-in | ADR 0002 | | Allen path-consistency | `temporal_core` | implemented-main | — | unit + budget tests | Task 4 / PR #9 | | Event mention/instance | `event_core` | partial | — | unit + fail-closed promotion | Task 5 / PR #13 | +| First-story FAR/miss | `event_core` | active-PR | this PR | computed FAR/miss + RMSE vs always-first | ADR 0016; `docs/research/first-story-detection-calibration.md` | | Multiple membership | `membership_core` | partial | nested ICC + non-nested refusal | unit + ESS + nested ICC recovery | Task 7 / PR #12 + #25 + this increment | | TDT tracking stability | `event_core` | active-PR | this PR | pair P/R + switch rate + RMSE vs always-one-track | ADR 0016; `docs/research/event-tracking-calibration.md` | | CHRONOS schema-slot accuracy | `event_core` | active-PR | this PR | computed slot P/R + RMSE vs always-fill | ADR 0016; `docs/research/chronos-schema-slot-calibration.md` | diff --git a/registered_agents.json b/registered_agents.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/registered_agents.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/task_agent_mapping.json b/task_agent_mapping.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/task_agent_mapping.json @@ -0,0 +1 @@ +{} \ No newline at end of file