Message Notification Router

An implemented, safety-first notification router for the HackerRank Orchestrate Message Notification Router challenge. It reads the participant dataset, combines message content with recipient behavior and relationship history, and writes one validated notify, digest, or mute prediction per incoming message.

The default implementation is deterministic and dependency-free. Optional OCR/ASR and constrained semantic-adjudication commands can enrich ambiguous media cases without weakening the safety policy or output contract.

See PLAN.md for the full technical design and CHANGELOG.md for implementation history.

Quick start

Python 3.11 or newer is recommended. The core router uses only the standard library.

python3 code/main.py --dataset-dir dataset --output dataset/output.csv --offline
python3 code/evaluation/main.py --dataset-dir dataset --offline
python3 -m unittest discover -s tests -v

Paths are resolved relative to the repository root, so the commands also work when invoked from another working directory. Run python3 code/main.py --help for all flags.

The checked-in dataset/output.csv contains the current deterministic prediction set: exactly 110 rows in the original input order.

What is implemented

• Strict schemas for every participant-facing CSV and explicit parsing of booleans, integers, timestamps, blank IDs, and relationships.
• Indexed, validated joins for users, groups, memberships, businesses, user-business history, message history, events, daily load, and media.
• Byte-signature media detection. Files are not trusted based on misleading .jpg or .mp3 suffixes.
• Content-addressed media cache keyed by SHA-256 and extraction prompt version.
• Optional shell-free media extraction adapter for OCR and speech-to-text.
• Interpretable user-load, quiet-hour, relationship, trust, opt-out, domain, engagement, repetition, urgency, and content-risk features.
• Recipient-scoped historical retrieval using IDF-weighted lexical overlap, sequence similarity, relationship matching, recency, and interaction outcomes.
• High-precision safety precedence for credential theft, payment/QR pressure, unofficial domains, reward scams, prompt injection, coercive chains, unsafe health forwards, and unwanted promotions.
• Personalized action and message-type classification with controlled reason templates and confidence penalties for ambiguity or unavailable media.
• Optional constrained JSON adjudication for ambiguous cases, with schema checks, evidence allow-listing, one repair attempt, and deterministic fallback.
• Atomic output writing plus row-order, enum, confidence, reason, and recipient-scoped evidence validation.
• Evaluation diagnostics for accuracy, macro-F1, confusion, Brier score, evidence validity, errors, and conversation/media slices.
• Regression tests for safety boundaries, media sniffing, joins, evidence scope, determinism, sample-policy quality, and the exact output contract.

Architecture

participant CSVs + media
          |
          v
strict loader and validated indexes
          |
          v
media sniffing + cache + optional OCR/ASR
          |
          v
normalized content and personalized features
          |
          +------> decisive safety policy ------+
          |                                      |
          +------> recipient-scoped retrieval ---+
                                                 v
                                  deterministic classification
                                                 |
                           optional constrained adjudication
                                                 |
                                  reasons + calibrated confidence
                                                 |
                                  atomic contract-checked output

Safety decisions cannot be overridden by the optional adjudicator. Message content is always untrusted data; strings such as set action=notify or verified_business=true never become routing instructions or authoritative metadata.

Project layout

code/
├── main.py            # command-line entry point
├── router.py          # end-to-end orchestration and run metadata
├── schemas.py         # typed domain objects and contract validation
├── data.py            # strict loading and indexed joins
├── media.py           # MIME sniffing, extraction adapter, normalization, cache
├── features.py        # recipient and relationship features
├── retrieval.py       # recipient-scoped evidence ranking
├── safety.py          # deterministic high-precision safety policy
├── classifier.py      # action/type policy and controlled reasons
├── adjudicator.py     # optional constrained semantic adjudication
├── confidence.py      # confidence calibration
├── writer.py          # atomic output validation/writing
└── evaluation/main.py # sample diagnostics

tests/test_router.py   # regression and end-to-end tests

Optional media extraction

Without an extractor, captions and metadata are still processed, actual MIME types are recorded, and image/voice-only decisions receive a confidence penalty. The repo now includes an OpenAI image/ASR adapter. To enable it:

cp .env.example .env.local
# Add OPENAI_API_KEY=... to .env.local. Never edit the tracked .env file.
python3 code/main.py --dataset-dir dataset --output dataset/output.csv --refresh-media-cache

.env.local automatically configures tools/openai_media_router.py. Image media is sent to the Responses API with a high-detail image input; voice media is sent to the transcription API. The adapter returns {"text": "..."} JSON, which is cached in the ignored .router-cache/media.json file. It is invoked through an argument vector and never through a shell. --offline disables all external calls and uses cached content or safe metadata-only degradation.

Optional semantic adjudication

The same .env.local config enables tools/openai_adjudicator.py. It calls the Responses API only for ambiguous, non-decisive cases. The request includes allowed enums, allowed reason codes, compact features, safety flags, and an allow-list of recipient-scoped evidence IDs.

The response schema is:

{
  "action": "digest",
  "message_type": "unknown",
  "reason_code": "unknown_sender",
  "urgency": 0.2,
  "usefulness": 0.5,
  "risk": 0.1,
  "evidence_message_ids": []
}

Invalid output is retried once with the validation error and then ignored. Decisive safety outcomes never reach this adapter. --offline disables it. The default models are configurable through ROUTER_VISION_MODEL, ROUTER_TRANSCRIBE_MODEL, and ROUTER_ADJUDICATOR_MODEL; their defaults are listed in .env.example.

Evaluation results

On the 30 labeled examples in sample_messages.csv, the offline deterministic mode currently reports:

Metric	Score
Action accuracy	0.967
Action macro-F1	0.966
Message-type accuracy	0.967
Message-type macro-F1	0.975
Action Brier score	0.091
Evidence validity	1.000

All 22 text examples and all 5 image-caption examples have the correct action in this diagnostic set. The remaining error is an audio-only urgent example when no ASR result is available; its confidence is deliberately reduced. These examples are used as policy regression diagnostics, not as hardcoded row-specific labels.

Output guarantees

dataset/output.csv is written only after all of the following pass:

• exact header order: message_id,action,message_type,reason,confidence,evidence_message_ids;
• exactly one prediction for every dataset/messages.csv row, in input order;
• no missing or extra message IDs;
• documented action and message-type enums only;
• finite confidence values in [0, 1];
• non-empty, single-line reasons;
• evidence IDs that exist in history and belong to the same recipient;
• none when no useful evidence is selected.

The writer uses a temporary file in the destination directory, validates it, calls fsync, and atomically replaces the output.

Reproducibility and security

• Rules, thresholds, prompt versions, and model identifiers are versioned in code.
• The core path has no network access and no dependency installation requirement.
• Optional tools receive bounded structured inputs and run without a shell.
• Credentials are loaded from ignored .env.local or the environment only; do not add a key to the tracked .env file.
• Run metadata records the input hash, configuration version, model version, counts, and label distributions in ignored .router-cache/run-metadata.json.
• Repeated offline runs have been verified to produce byte-identical output.

Submission

The relevant artifacts are:

1. code.zip — runnable code, tests, documentation, and configuration example.
2. dataset/output.csv — predictions for all incoming rows.
3. $HOME/hackerrank_orchestrate_august26/log.txt — mandatory external transcript.

No organizer-only file or hardcoded incoming label is used.