Skip to content

[v4] Implement predatory journal flagging #7

Closed
#44
Closed
[v4] Implement predatory journal flagging#7
#44
Labels
heuristicForensic heuristic developmentprompt-engineeringPrompt design and optimizationv4Planned for v4

Description

@cpitzi
cpitzi
opened on Mar 23, 2026

Summary

Identify references citing journals classified as predatory or questionable by established assessment methodologies.

See roadmap/v4-features.md (Priority 3) for full specification.

Open Questions

  • Data source: Cabells Predatory Reports is the gold standard but is commercial. Is API access feasible? Cost?
  • Alternative sources: Beall's List (archived, not maintained), community lists, DOAJ membership as positive signal
  • Classification language: "Predatory" is contentious. Should the auditor use "potentially predatory," "not indexed in DOAJ," or another framing?

Implementation Approach

  1. Check journal against known predatory journal databases/lists via web search
  2. Check DOAJ membership as a positive legitimacy signal
  3. Check Scopus/PubMed indexing as additional legitimacy signals
  4. Flag journals that appear on predatory lists or lack any recognized indexing

Acceptance Criteria

  • Identifies journals on known predatory lists
  • Distinguishes "confirmed predatory" from "potentially predatory"
  • Does not flag journals removed from predatory lists after improvement
  • Handles journal name variations (predatory journals frequently rename)
  • Classified as Elevated risk (not High — the cited paper may still be legitimate research)

Dependencies

  • Requires resolution of the data source question before implementation

Activity

  1. cpitzi
    added
    v4Planned for v4
    prompt-engineeringPrompt design and optimization
    heuristicForensic heuristic development
    on Mar 23, 2026

  2. cpitzi commented on Jun 19, 2026

    @cpitzi
    cpitzi
    on Jun 19, 2026
    OwnerAuthor

    Data-source decision (2026-06-19): Hybrid — indexing whitelists + free community blacklists

    Resolving the open data-source and classification-language questions.

    Architecture constraint: the auditor is a single prompt using live web search — no code, no API layer, no database. So Cabells Predatory Reports is out of scope for now (paywalled, and would require the API-integration pipeline #11, which is deferred). All signals must be freely reachable via web search at run time.

    Approach

    Primary — positive signal (whitelists): for each cited journal, check indexing in DOAJ (api.doaj.org), PubMed/MEDLINE, the Scopus source list, and the Web of Science Master Journal List. Any reputable indexing is a legitimacy signal that clears the journal.

    Secondary — corroboration (free blacklists): cross-reference freely-available community lists (archived Beall's List, Stop Predatory Journals). Corroborating evidence only — never the sole basis for a flag.

    Flagging: a journal indexed in none of the recognized databases while claiming to be peer-reviewed is flagged Elevated — never High in isolation (the cited paper may be legitimate research in a weak venue); escalate to High only when combined with another heuristic trigger.

    Classification language (resolves the contentious-language question): report factually — e.g., "not indexed in DOAJ, Scopus, PubMed, or Web of Science; also listed on [community source]." Avoid the verdict word "predatory"; at most "potentially predatory / unverified venue." This sidesteps the defamation and methodological risk of labeling a named journal predatory.

    Must NOT flag: grey literature (government/organizational reports have no journal indexing — exempt, per the existing grey-lit rule); established subscription journals indexed in Scopus/WoS; journals that left blacklists after improving. Must handle journal renaming.

    Implementation shape

    Ships as a new Heuristic 10 in a new prompts/v6-auditor.md (v5 is frozen by its committed #40 baseline), plus docs/heuristics.md + README + CLAUDE.md updates, a dedicated test set (using clearly-fabricated or well-documented-defunct venues, framed as "unindexed" rather than naming a real journal "predatory"), and a v6 baseline-validation follow-up mirroring #40.

    This unblocks #7.

  3. cpitzi
    added a commit that references this issue on Jun 19, 2026

    fix(test-sets): correct broken clean-control DOIs in predatory-venues…

    0e30015
  4. cpitzi
    closed this as completedin #44on Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    heuristicForensic heuristic developmentprompt-engineeringPrompt design and optimizationv4Planned for v4

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions