An AI-assisted multi-scenario decision engine that combines structured LLM analysis with deterministic scoring, ranking, and evaluation.
ScenarioRank helps a person compare options under different business conditions. A user provides a decision, candidates, and a scenario. The system asks an LLM for strictly structured interpretation, then deterministic code calculates fit, risk, projected outcome, and the final ranking. The model explains the result; it does not get to arbitrarily choose the winner.
For example, a hiring team can compare leadership candidates for a post-merger integration, choose whether to optimise for fit, risk-adjusted performance, or outcome, and review the trade-offs behind the recommendation.
- Hybrid architecture: probabilistic language interpretation is separated from deterministic decision logic.
- OpenAI Responses API with structured outputs, runtime schemas, retries, and provider-neutral boundaries.
- Scenario-aware ranking, optional top-four pairing analysis, JSON and SSE responses, and token/cost metadata.
- Shared Zod contracts protect browser/server transport; provider output schemas remain internal.
- A local-first evaluation harness runs the real production pipeline with fake providers, a versioned synthetic benchmark, deterministic graders, and intentionally gated live runs.
flowchart LR
UI[React + TypeScript] --> API[Node + Express API]
API --> PIPE[Decision pipeline]
PIPE --> AI[OpenAI provider]
AI --> PIPE
PIPE --> SCORE[Deterministic scoring]
SCORE --> CONTRACT[Zod contracts]
CONTRACT --> RESULT[Ranked, explainable result]
RESULT --> UI
The evaluation path is intentionally separate from production traffic:
Versioned benchmark → production pipeline → fake or live provider → deterministic graders → evaluation report
| Decision setup | Streaming progress |
|---|---|
 |
 |
decision-benchmark-v1 v1.1.0 contains 16 fully synthetic cases (21 scenario executions), 11 deterministic graders, an eight-dimension human-review rubric, and seven fake-provider profiles. The committed fixture baseline is clean_pass: 16 clean cases, with zero known-defect observations, affected executions, unexpected failures, or unexpected defect resolutions.
The harness is offline by default. It validates orchestration and deterministic invariants, not LLM intelligence or real-world hiring validity. Live runs require an explicit case, --live, a priced model, and a conservative dollar budget. In the final portfolio validation, a real gpt-5-mini single-scenario run passed all applicable graders (3 provider attempts, 9,208 tokens, $0.014352 estimated cost). A pairing run returned a valid complete result but selected a different synthetic expected pair; no prompt, score, or benchmark data was tuned to force a pass.
The current suite has 676 tests: 105 frontend, 241 server, and 330 evaluation tests. It covers browser transport/SSE parsing, accessible UI rendering, HTTP contracts, structured OpenAI adapter behaviour, pipeline integrity, deterministic scoring, benchmark validation, live-budget refusals, and repository safeguards.
- Frontend: React 18, TypeScript, Vite, React Router
- Backend: Node.js, Express, ESM
- AI: OpenAI Responses API with structured outputs
- Validation: Zod
- Testing: Vitest
- Evaluation: custom versioned, local-first harness
ScenarioRank received Best Implementation in a BMW-related competition. The original award snapshot remains preserved as the bmw-award-original tag and archive/bmw-award-original branch. V2 is a later architectural redesign focused on testability, correctness, and explainability.
npm ci
cp .env.example .env.local
# Set OPENAI_API_KEY in .env.local; never commit it.
node server.mjs
# In another terminal:
npm run devThe frontend defaults to http://localhost:5173 and the backend to http://localhost:3001. See .env.example for safe configuration shape and the repository map for project navigation.
Key decisions are recorded in the ADRs, including the provider boundary, Node/Express retention, local-first evaluation harness, and signed risk-adjusted score. risk_adjusted_score is a signed -100…100 penalty-adjusted net score: higher is better and a negative value means modelled penalties exceed weighted fit.
This is a research and decision-support prototype, not an autonomous hiring system. Scores are heuristic rather than empirically calibrated; the benchmark is synthetic; live evaluation is deliberately budget-gated; the application has no authentication, persistence, rate limiting, or audit trail. SR-P3A-002 remains separately open, and Phase 3B has not started. See known limitations.