Evidence Grading Engine

Evidence Grading Engine

Author: Ciprian Ștefan Pleșca

This page documents src/openlongevity/evidence.py, the module responsible for assigning an evidence level (A–G) and a navigation score to EvidenceRecord instances. The engine is explicitly framed by the project as a transparent heuristic for navigating a literature set, not a validated measure of scientific truth or clinical effect.

The A–G hierarchy

flowchart TD
    A["Level A — Systematic review\nbase score 0.95"]
    B["Level B — Randomized controlled trial\nbase score 0.85"]
    C["Level C — Clinical study\nbase score 0.70"]
    D["Level D — Observational study\nbase score 0.50"]
    E["Level E — Animal study\nbase score 0.35"]
    F["Level F — In-vitro study\nbase score 0.20"]
    G["Level G — Computational / retracted\nbase score 0.10"]

    A --> B --> C --> D --> E --> F --> G

    style A fill:#2e7d32,color:#fff
    style B fill:#558b2f,color:#fff
    style C fill:#9e9d24,color:#000
    style D fill:#f9a825,color:#000
    style E fill:#ef6c00,color:#fff
    style F fill:#d84315,color:#fff
    style G fill:#6d4c41,color:#fff

Loading

EvidenceEngine.grade() maps StudyType directly to EvidenceLevel through the _LEVEL_BY_TYPE table, with one override: any record whose retraction_status is RETRACTED is graded G regardless of its original study type. This is a deliberate navigational choice, not a claim that a retracted systematic review is methodologically equivalent to a computational model — the README is explicit that "clients must retain original study design alongside publication status" precisely because the G-grade collapses two different kinds of "low confidence" into one bucket for navigation purposes.

The navigation score formula

EvidenceEngine.score() computes a bounded value in [0.0, 1.0] through a sequence of multiplicative adjustments:

flowchart LR
    S0["base_score(study_type) × confidence"] --> S1{retraction_status\n== RETRACTED?}
    S1 -- yes --> Z["score = 0.0"]
    S1 -- no --> S2{replication_status}
    S2 -- "replicated / independent" --> M1["× 1.15"]
    S2 -- "unreplicated / unknown" --> M2["× 0.85"]
    S2 -- other --> M3["× 1.0"]
    M1 --> S3
    M2 --> S3
    M3 --> S3
    S3{sample_size is not None?} -- yes --> M4["× min(1.15, 0.85 + n/(n+200))"]
    S3 -- no --> S4
    M4 --> S4
    S4{publication_date present\nand parseable?} -- yes --> M5["× max(0.75, 1.0 − age_years×0.01)"]
    S4 -- no --> S5
    M5 --> S5
    S5["clamp to [0.0, 1.0], round to 4 decimals"] --> OUT["navigation_score"]

Loading

Term-by-term interpretation

• Base score × confidence. The study-type base score (Level A = 0.95 down to Level G = 0.10) is scaled by the record's self-reported confidence (already constrained to [0, 1] by EvidenceRecord.__post_init__). This means a systematic review reported with low confidence can score below a well-supported animal study — the formula does not let study type alone dominate.
• Replication multiplier. Replicated or independently confirmed findings receive a 15% boost; unreplicated or unknown-replication findings receive a 15% penalty. Any other string value for replication_status (the field is a free-form str, not an enum) leaves the score unchanged, which is a permissive default worth noting for callers passing nonstandard values.
• Sample-size multiplier. min(1.15, 0.85 + n/(n+200)) is a saturating curve: as n → ∞, the multiplier approaches its ceiling of 1.15; at n = 0 it evaluates to 0.85. This rewards larger samples without letting sample size alone dominate the score, and it is only applied when sample_size is not None — records that do not report a sample size are neither rewarded nor penalized on this axis.
• Recency decay. max(0.75, 1.0 − age_years × 0.01) reduces the score by roughly 1 percentage point per year since publication, floored at a 25% maximum reduction. A ValueError from an unparseable publication_date is caught and silently skips this adjustment — the record is neither penalized nor rewarded for having a badly formatted date.
• Retraction short-circuits everything. If retraction_status is RetractionStatus.RETRACTED, the method returns 0.0 immediately, regardless of any other multiplier already computed.

Worked example

Consider an RCT (base_score = 0.85) with confidence = 0.8, replication_status = "replicated", sample_size = 300, published exactly 2 years ago, not retracted:

1. 0.85 × 0.8 = 0.68
2. Replicated → 0.68 × 1.15 = 0.782
3. Sample size: min(1.15, 0.85 + 300/500) = min(1.15, 1.45) = 1.15 → 0.782 × 1.15 = 0.8993
4. Recency: max(0.75, 1.0 − 2×0.01) = 0.98 → 0.8993 × 0.98 ≈ 0.8813
5. Clamp and round → 0.8813

This score is a navigation aid, not a probability of the finding being true — the docstring on score() states this directly: "Return a transparent navigation score, not a validated effect estimate."

Summarization and contradiction detection

EvidenceEngine.summarize() aggregates a collection of records into:

• records — total count including retracted records
• active_records — count excluding retracted records
• evidence_distribution — a sorted count of grades among active records
• mean_confidence and mean_navigation_score — computed only over active records
• a fixed disclaimer string, always present in the output

EvidenceEngine.contradictions() groups records by their tag string (" ".join(record.tags).casefold()) and flags any group containing both FindingDirection.POSITIVE and FindingDirection.NEGATIVE records as a Contradiction. This is intentionally coarse — it is a surfacing mechanism for a human reviewer to inspect, not an automated resolution of conflicting evidence, and its accuracy is entirely dependent on consistent, well-chosen tags at the point where records are created.

sequenceDiagram
    autonumber
    participant Caller
    participant Engine as EvidenceEngine
    Caller->>Engine: contradictions(records)
    Engine->>Engine: group records by casefolded tag string
    loop each tag group
        Engine->>Engine: collect POSITIVE identifiers
        Engine->>Engine: collect NEGATIVE identifiers
        alt both non-empty
            Engine-->>Caller: Contradiction(topic, positive_ids, negative_ids, explanation)
        end
    end

Loading

Next

See Research-Gap-Detection for how the same EvidenceRecord collection is used to surface missing categories of evidence, and Scientific-Methodology-and-Limitations for how these outputs should — and should not — be interpreted.