-
Notifications
You must be signed in to change notification settings - Fork 0
feat(event): score mention confidence with a known-truth Brier rule #64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
b056ac9
d2d552d
22a5955
c11adce
40e6926
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| //! Mention confidence recovers known Brier scores against binary truth. | ||
|
|
||
| use event_core::{EventConfidence, EventError, mention_brier_score}; | ||
|
|
||
| #[test] | ||
| fn perfectly_calibrated_forecasts_recover_zero_brier() { | ||
| let forecasts = [ | ||
| EventConfidence::new(0.0).expect("0"), | ||
| EventConfidence::new(1.0).expect("1"), | ||
| EventConfidence::new(0.0).expect("0"), | ||
| EventConfidence::new(1.0).expect("1"), | ||
| ]; | ||
| let outcomes = [false, true, false, true]; | ||
| let score = mention_brier_score(&forecasts, &outcomes).expect("brier"); | ||
| assert!(score.abs() < 1e-15, "perfect Brier {score}"); | ||
| } | ||
|
|
||
| #[test] | ||
| fn constant_half_recovers_quarter_and_mismatches_fail_closed() { | ||
| let forecasts = [ | ||
| EventConfidence::new(0.5).expect("half"), | ||
| EventConfidence::new(0.5).expect("half"), | ||
| ]; | ||
| let outcomes = [false, true]; | ||
| let score = mention_brier_score(&forecasts, &outcomes).expect("half"); | ||
| let residual = score - 0.25; | ||
| let rmse = (residual * residual).sqrt(); | ||
| assert!(rmse < 1e-15, "Brier RMSE {rmse}"); | ||
| assert_eq!( | ||
| mention_brier_score(&forecasts, &[true]), | ||
| Err(EventError::InvalidWirePayload) | ||
| ); | ||
| assert_eq!( | ||
| mention_brier_score(&[], &[]), | ||
| Err(EventError::InvalidWirePayload) | ||
| ); | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # Mention-confidence Brier score | ||
|
|
||
| ## Scope | ||
|
|
||
| This note doctors the `event_core` calibration contract for fallible event mentions: | ||
|
|
||
| 1. mention confidence is a probability on `[0, 1]`; | ||
| 2. `mention_brier_score` is the mean squared error against binary truth; | ||
| 3. empty or length-mismatched streams fail closed. | ||
|
|
||
| TDT/CHRONOS promotion remains on the event-intelligence active PR. No database migration is allocated. | ||
|
|
||
| ## Authoritative sources | ||
|
|
||
| Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. *Monthly Weather Review, 78*(1), 1–3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2 | ||
|
|
||
| Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. *Journal of the American Statistical Association, 102*(477), 359–378. https://doi.org/10.1198/016214506000001437 | ||
|
|
||
| ## Application | ||
|
|
||
| Brier (1950) defines the mean squared error of a probability forecast. Gneiting and Raftery (2007) treat the Brier score as a strictly proper scoring rule, so a mention that is certain when true and impossible when false is uniquely optimal. TEPP therefore scores mention confidence against known binary outcomes rather than treating a high score as an event instance (Brier, 1950; Gneiting & Raftery, 2007). | ||
|
Comment on lines
+13
to
+21
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📝 Info: APA 7 reference placed in separate doctoring note, matching repo pattern CONTRIBUTING.md states APA 7 references are recorded in Was this helpful? React with 👍 or 👎 to provide feedback. |
||
|
|
||
| ## Verification | ||
|
|
||
| - forecasts `(0,1,0,1)` against outcomes `(false,true,false,true)` recover Brier `0`; | ||
| - constant `0.5` against mixed outcomes recovers `0.25` with computed residual RMSE; | ||
| - empty and mismatched streams return `InvalidWirePayload`. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📝 Info: Brier score computation is correct and fail-closed
mention_brier_score(confidence.rs) computes the mean of squared residuals against binary truth, matching the standard Brier score. It fails closed on empty or length-mismatched inputs viaEventError::InvalidWirePayload, andEventConfidencealready guarantees finite values in [0,1], so no NaN/overflow risk exists. The logic verified against the documented test vectors (perfect 0, constant-0.5 = 0.25).Was this helpful? React with 👍 or 👎 to provide feedback.