diff --git a/CHANGELOG.md b/CHANGELOG.md index f4e0f08..0ca7628 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,22 @@ # Changelog +## Unreleased + +- Add `evalarc view [DIR]`: a read-only, loopback-only local viewer that lists + every report under a folder with its verdict, following the `inspect view` / + `promptfoo view` pattern with the standard library only. It rejects foreign + `Host` headers, path traversal and directory listings. `--write-index` writes + the same index as a static page. +- Report pages: light and dark themes from the system setting, sticky table + headers, tabular numbers, sortable columns (WAI-ARIA APG pattern) and a row + filter on long tables. The enhancement script is a packaged asset inlined and + allowed by CSP hash; pages remain complete without JavaScript. +- `scripts/check_reports_browser.cjs` now uses axe-core (WCAG 2.2 A/AA, light and + dark) instead of a custom contrast check, and runs end-to-end reviewer tasks: + verdict on the first screen, skip link, keyboard sorting, filtering, 320 px + reflow, no-JavaScript, print, and browsing from the viewer index. + `@axe-core/playwright` 4.13.0 is a pinned dev dependency. + ## 0.17.3 — 2026-10-01 - fix: fail the diff gate when the baseline run did not finish. evalarc diff and the GitHub Action now fail the gate (exit 1) when the baseline run did not finish. Before this change, an Inspect baseline log with a status other than success could pass, because checks missing from it were classified as 'added', which never blocks. diff.json now records "baseline_incomplete": true. The key is omitted for a finished baseline, so those reports are unchanged apart from created_at and version. Default stdout prints 'The baseline run is incomplete (status error); N check(s) appear only in the current run and were not compared. Rerun the baseline to completion; the gate fails.' summary.md always includes the line below the table: 'The baseline run did not finish (status `error`); the gate fails because checks missing from it cannot be compared. N check(s) appear only in the current run. Rerun the baseline to completion.' Its heading says 'the baseline run is incomplete' only when there are no blocking changes and the current run finished; otherwise the existing blocking or both-incomplete heading appears. index.html shows 'Gate failed' with the reason 'the baseline run is incomplete'. When no blocking rows exist, it tells you to rerun the baseline. evalarc verify recomputes the same result. docs/ci-gate.md and the README CI-gate paragraph document the rule. diff --git a/README.md b/README.md index 9b3ba1f..082b96e 100644 --- a/README.md +++ b/README.md @@ -92,6 +92,7 @@ to produce your first HTML report without cloning the source. | A cheaper model, lower thinking effort or shorter prompt | Require no lost checks and recorded cost or tokens at most a set ratio of the baseline | [Cost at equal quality](docs/ci-gate.md#cost-at-equal-quality) | | Saved results you plan to tune against | Find saturation, never-passing and flaky checks, inconsistent grading, truncation, self-grading, pipeline errors and noise above the change you need | [Eval health](docs/eval-health.md) | | A grader you are about to trust, or a change judged by preference | Blind spot checks with false accepts/rejects; blind randomized A/B against the baseline with position-bias checks | [Blind judging](docs/judging.md) | +| A folder of reports from many runs | Open one local index with every report's verdict; sort, filter and drill in | [Browse reports](docs/viewer.md) | | A new evaluation for your application | Scaffold cases, a programmatic grader, a runner and a hillclimb config; review the inputs before any run | [Evaluation project](docs/eval-init.md) | | A prompt or skill you want to hillclimb with your own model-calling commands | Run the loop: tuning failures only, edits limited to allowed files, pasted-case and leakage rollback, keep/rollback by held-out results | [Hillclimb run](docs/hillclimb-run.md) | | Results of each step of a hillclimbing loop | Replay keep/rollback rules on tuning and held-out cases, triage stalls, and decide whether to merge | [Hillclimb review](docs/hillclimb-review.md) | diff --git a/README.zh-CN.md b/README.zh-CN.md index b130c61..8e9cb23 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -77,6 +77,7 @@ PyPI 与 GitHub Releases 提供相同的 wheel 和源码包，也支持 | 更便宜的模型、更低思考强度或更短的提示词 | 要求不丢失检查项，且记录的成本或 token 不超过基线的设定倍数 | [保质降本](docs/ci-gate.zh-CN.md#保质降本) | | 准备据以调优的已保存结果 | 发现饱和、始终失败或波动的检查、评分不一致、截断、自评、流水线错误，以及大于目标效应的噪声 | [评测健康检查](docs/eval-health.zh-CN.md) | | 准备采信的评分器，或按偏好评判的改动 | 盲抽查并统计误判通过/失败；与基线做随机 A/B 盲评并检查位置偏差 | [盲评](docs/judging.zh-CN.md) | +| 多次运行产生的一批报告 | 用一个本地索引查看每份报告的结论，可排序、筛选并逐个打开 | [浏览报告](docs/viewer.zh-CN.md) | | 为自己的应用新建评测 | 生成用例、程序化评分器、评测脚本与调优配置；运行前审查输入 | [创建评测项目](docs/eval-init.zh-CN.md) | | 想用自己的模型调用命令做爬山调优的提示词或技能 | 运行循环：只给调优失败、只许改允许的文件、粘贴用例与泄露自动回滚、按留出结果保留或回滚 | [运行爬山调优](docs/hillclimb-run.zh-CN.md) | | 爬山调优每一步的结果 | 按调优与留出用例重放保留/回滚规则，分类停滞原因，判断是否合并 | [爬山调优复核](docs/hillclimb-review.zh-CN.md) | diff --git a/SECURITY.md b/SECURITY.md index c75fd15..dc0f7e1 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -46,6 +46,10 @@ model or network calls. The hillclimb `allow` list compares the workspace (inclu on any other change; it is a guard, not a sandbox, and cannot see or undo changes outside the workspace. +`evalarc view` serves report folders read-only on `127.0.0.1` only, rejects other +`Host` headers, path traversal and directory listings, and has no API. Report pages +allow exactly one inline script by CSP hash and no network access. + Local mode requires `--trust-local`. It executes with the host user's filesystem and network privileges. Clearing the child environment does not provide a sandbox. Use local mode only for code you trust, such as the bundled audit diff --git a/docs/viewer.md b/docs/viewer.md new file mode 100644 index 0000000..41e6535 --- /dev/null +++ b/docs/viewer.md @@ -0,0 +1,50 @@ +# Browse reports + +Every EvalArc command that takes `--output` writes a self-contained HTML report. +`evalarc view` lists all of them under a folder with their verdicts and opens it +in your browser. [中文](viewer.zh-CN.md). + +```bash +evalarc view runs # http://127.0.0.1:7576/, opens a browser +evalarc view runs --port 8000 --no-browser +evalarc view runs --write-index site/index.html # static index for CI artifacts or Pages +``` + +The index shows each report folder (newest first) with its verdict (✓ pass, +✗ fail, ! check), kind and summary, read from the verdict banner the report +already contains. It sorts and filters like every report table. + +## Why a local viewer, not a desktop app + +Evaluation tools converge on the same pattern: a read-only local web server plus +the browser you already have (`inspect view`, `promptfoo view`, `mlflow ui`), +with an IDE extension where tighter integration is needed (Inspect's VS Code +extension), and a static bundle for sharing (`inspect view bundle`, +`promptfoo eval -o report.html`). A desktop shell (Electron, Tauri) would add a +100 MB+ runtime or a Rust toolchain for no capability the browser lacks, and +would break EvalArc's no-runtime-dependency install. `evalarc view` uses the +Python standard library only. + +## Security + +The viewer binds to `127.0.0.1` only and serves files inside the given folder: +it rejects requests whose `Host` is not `127.0.0.1:PORT` or `localhost:PORT` +(DNS-rebinding protection), path traversal, symlinks leaving the folder and +directory listings. It is read-only and has no API. To view reports on a remote +machine, forward the port over SSH instead of exposing it: + +```bash +ssh -L 7576:127.0.0.1:7576 user@remote # then run evalarc view there +``` + +## Report pages + +Reports open with the decision (verdict, reason, next step) and work in light +and dark mode, on phones, at 400% zoom, in print and without JavaScript. With +JavaScript, table headers become sort buttons (WAI-ARIA APG sortable-table +pattern) and tables with eight or more rows get a labelled filter. The script is +inlined and allowed by a SHA-256 hash in each page's Content-Security-Policy; no +other script, network request or remote asset is permitted. +`scripts/check_reports_browser.cjs` checks every report kind and the viewer with +axe-core (WCAG 2.2 A/AA) and end-to-end reviewer tasks in CI. Automated checks +do not replace testing with a screen reader. diff --git a/docs/viewer.zh-CN.md b/docs/viewer.zh-CN.md new file mode 100644 index 0000000..d6e537e --- /dev/null +++ b/docs/viewer.zh-CN.md @@ -0,0 +1,27 @@ +# 浏览报告 + +所有带 `--output` 的 EvalArc 命令都会写出自包含的 HTML 报告。`evalarc view` 列出某目录下的 +全部报告及其结论，并在浏览器中打开。[English](viewer.md)。 + +```bash +evalarc view runs # http://127.0.0.1:7576/，自动打开浏览器 +evalarc view runs --port 8000 --no-browser +evalarc view runs --write-index site/index.html # 为 CI 产物或 Pages 生成静态索引 +``` + +索引按时间倒序列出每个报告目录的结论（✓ 通过、✗ 失败、! 需检查）、类型与摘要，可排序和筛选。 + +**为什么是本地查看器而非桌面客户端：** 主流评测工具都采用只读本地 Web 服务加用户已有浏览器的 +方式（`inspect view`、`promptfoo view`、`mlflow ui`），需要深度集成时提供 IDE 插件，分享时 +生成静态包。Electron/Tauri 桌面壳会带来 100 MB 以上运行时或 Rust 工具链，却不提供浏览器没有 +的能力，还会破坏 EvalArc 零运行时依赖的安装方式。`evalarc view` 只用 Python 标准库。 + +**安全：** 只绑定 `127.0.0.1`，只提供该目录内的文件；拒绝非 `127.0.0.1:端口`/`localhost:端口` +的 Host（防 DNS 重绑定）、路径穿越、指向目录外的符号链接和目录列表；只读、无 API。远程机器请用 +`ssh -L 7576:127.0.0.1:7576 user@remote` 转发端口，不要直接暴露。 + +**报告页面：** 先给结论（结果、原因、下一步），支持深浅色、手机、400% 缩放、打印和禁用 JavaScript。 +启用 JavaScript 时，表头变为排序按钮（WAI-ARIA APG 可排序表格模式），8 行以上的表格带筛选框。 +脚本内联并由页面 CSP 中的 SHA-256 哈希放行，不允许其他脚本、网络请求或远程资源。 +`scripts/check_reports_browser.cjs` 在 CI 中用 axe-core（WCAG 2.2 A/AA）和端到端任务检查全部报告 +类型与查看器；自动检查不能替代屏幕阅读器实测。 diff --git a/package-lock.json b/package-lock.json index 975cc1b..a9a4794 100644 --- a/package-lock.json +++ b/package-lock.json @@ -8,9 +8,33 @@ "name": "evalarc-evidence-site", "version": "0.17.3", "devDependencies": { + "@axe-core/playwright": "4.13.0", "playwright": "1.63.0" } }, + "node_modules/@axe-core/playwright": { + "version": "4.13.0", + "resolved": "https://registry.npmjs.org/@axe-core/playwright/-/playwright-4.13.0.tgz", + "integrity": "sha512-6YLx+kxXu5GJceG4ozFg+33a2EMTdjYwWGloJ3sb9Kta5pp+ZNS53uxGVog5JetIY8s++P5UrtX+cri+u0VAVg==", + "dev": true, + "license": "MPL-2.0", + "dependencies": { + "axe-core": "~4.13.0" + }, + "peerDependencies": { + "playwright-core": ">= 1.0.0" + } + }, + "node_modules/axe-core": { + "version": "4.13.0", + "resolved": "https://registry.npmjs.org/axe-core/-/axe-core-4.13.0.tgz", + "integrity": "sha512-UzGt8zg7Ny8djbYMhxl2zuEevVa7r2gJjYY5Lwr1xM7+XU2nd6CkIWFTVcCIbAP63vSz71NaVyyuSk9lHKcy0A==", + "dev": true, + "license": "MPL-2.0", + "engines": { + "node": ">=4" + } + }, "node_modules/playwright": { "version": "1.63.0", "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.63.0.tgz", diff --git a/package.json b/package.json index 726c200..56081d2 100644 --- a/package.json +++ b/package.json @@ -7,6 +7,7 @@ "test:site": "node scripts/check_site.cjs" }, "devDependencies": { + "@axe-core/playwright": "4.13.0", "playwright": "1.63.0" } } diff --git a/scripts/check_reports_browser.cjs b/scripts/check_reports_browser.cjs index 8db448b..7dfc210 100644 --- a/scripts/check_reports_browser.cjs +++ b/scripts/check_reports_browser.cjs @@ -1,47 +1,53 @@ -// Render every offline report EvalArc writes and check layout and accessibility basics: -// a verdict banner first, landmarks, scoped headers, keyboard-reachable tables, -// no horizontal page overflow at phone width, no console errors, no network requests. +// End-to-end usability and accessibility checks for EvalArc's offline reports and the +// `evalarc view` local viewer. Accessibility rules come from axe-core (WCAG 2.2 A/AA) +// in both light and dark color schemes; the task scenarios cover what a reviewer does: +// read the decision on the first screen, move by keyboard, sort and filter tables, read +// on a phone or at 400% zoom (320 px reflow), read without JavaScript, print, and +// browse every report from one index. const assert = require("node:assert/strict"); -const { execFileSync } = require("node:child_process"); +const { execFileSync, spawn } = require("node:child_process"); const fs = require("node:fs"); const os = require("node:os"); const path = require("node:path"); const { chromium } = require("playwright"); +const { default: AxeBuilder } = require("@axe-core/playwright"); const root = path.resolve(__dirname, ".."); const evalarc = process.env.EVALARC || "evalarc"; +const python = process.env.PYTHON || "python3"; const out = fs.mkdtempSync(path.join(os.tmpdir(), "evalarc-reports-")); +const runs = path.join(out, "runs"); const inspect = path.join(root, "examples/results-diff/inspect"); const climb = path.join(root, "examples/hillclimb-review"); const run = (...args) => { try { execFileSync(evalarc, args, { stdio: "pipe" }); } catch (error) { if (error.status === 2) throw error; } }; run("diff", `${inspect}/baseline.json`, `${inspect}/current.json`, "--held-out", `${inspect}/split.json`, - "--harness", `${inspect}/harness`, "--output", `${out}/diff`); + "--harness", `${inspect}/harness`, "--output", `${runs}/diff`); run("eval-health", `${inspect}/baseline.json`, `${inspect}/current.json`, "--cases", `${inspect}/cases.json`, - "--min-effect", "0.05", "--output", `${out}/health`); + "--min-effect", "0.05", "--output", `${runs}/health`); const steps = fs.readdirSync(`${climb}/results`).sort().map(name => `${climb}/results/${name}`); -run("hillclimb-review", ...steps, "--held-out", `${climb}/split.json`, "--objective", "cost", "--output", `${out}/climb`); +run("hillclimb-review", ...steps, "--held-out", `${climb}/split.json`, "--objective", "cost", "--output", `${runs}/climb`); run("judge-packet", "grader", `${inspect}/current.json`, "--sample", "6", "--output", `${out}/spot`); const key = JSON.parse(fs.readFileSync(`${out}/spot/key.json`, "utf8")); const template = JSON.parse(fs.readFileSync(`${out}/spot/share/verdicts.template.json`, "utf8")); for (const [id, hidden] of Object.entries(key.items)) template.verdicts[id] = hidden.recorded_passed ? "pass" : "fail"; fs.writeFileSync(`${out}/verdicts.json`, JSON.stringify(template)); -run("judge-score", `${out}/spot`, `${out}/verdicts.json`, "--output", `${out}/score`); +run("judge-score", `${out}/spot`, `${out}/verdicts.json`, "--output", `${runs}/score`); run("eval-init", `${out}/proj`); -run("review-inputs", `${out}/proj/cases.jsonl`, "--output", `${out}/review`); -// Earlier report kinds, rendered from committed evidence without running candidates. -const render = (kind, source, target) => execFileSync("python3", ["-c", ` +run("review-inputs", `${out}/proj/cases.jsonl`, "--output", `${runs}/review`); +const render = (kind, source, target) => execFileSync(python, ["-c", ` import json, sys from pathlib import Path from evalarc import report data = json.loads(Path(sys.argv[2]).read_text()) getattr(report, "render_" + sys.argv[1])(data, Path(sys.argv[3]))`, kind, source, target]); -render("audit", `${root}/examples/support-audit/audit.json`, `${out}/audit/index.html`); -render("evaluation", `${root}/examples/evaluation/evaluation.json`, `${out}/evaluation/index.html`); -render("comparison", `${root}/examples/comparison/comparison.json`, `${out}/comparison/index.html`); -render("repetition", `${root}/examples/repetition/repetition.json`, `${out}/repetition/index.html`); -render("suite", `${root}/examples/suite/suite.json`, `${out}/suite/index.html`); +for (const [kind, source] of [["audit", "support-audit/audit.json"], ["evaluation", "evaluation/evaluation.json"], + ["comparison", "comparison/comparison.json"], ["repetition", "repetition/repetition.json"], ["suite", "suite/suite.json"]]) { + fs.mkdirSync(`${runs}/${kind}`, { recursive: true }); + fs.copyFileSync(`${root}/examples/${source}`, `${runs}/${kind}/${path.basename(source)}`); + render(kind, `${root}/examples/${source}`, `${runs}/${kind}/index.html`); +} const reports = { diff: /Gate failed/, health: /warning\(s\) before tuning|Ready to tune/, climb: /Merge/, @@ -49,65 +55,139 @@ const reports = { audit: /declared faults detected/, evaluation: /Resolved|Not resolved|Unassessed/, comparison: /regressed/, repetition: /attempt\(s\) resolved/, suite: /job gate\(s\) accepted/, }; +const WCAG = ["wcag2a", "wcag2aa", "wcag21a", "wcag21aa", "wcag22aa"]; + +async function axe(page, label) { + const result = await new AxeBuilder({ page }).withTags(WCAG).analyze(); + const violations = result.violations.map(v => `${v.id} (${v.nodes.length}): ${v.nodes[0].target}`); + assert.deepEqual(violations, [], `${label}: axe violations`); +} -// WCAG 2.2 AA contrast for every visible text node against its effective background. -function contrastProblems() { - const parse = color => (color.match(/[\d.]+/g) || []).map(Number); - const lum = ([r, g, b]) => { - const f = c => { c /= 255; return c <= 0.03928 ? c / 12.92 : ((c + 0.055) / 1.055) ** 2.4; }; - return 0.2126 * f(r) + 0.7152 * f(g) + 0.0722 * f(b); - }; - const background = element => { - for (let node = element; node; node = node.parentElement) { - const value = parse(getComputedStyle(node).backgroundColor); - if (value.length >= 3 && (value.length < 4 || value[3] > 0)) return value; - } - return parse(getComputedStyle(document.documentElement).backgroundColor); - }; +async function open(browser, url, options = {}) { + const context = await browser.newContext(options); + const page = await context.newPage(); const problems = []; - const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT); - for (let node = walker.nextNode(); node; node = walker.nextNode()) { - const element = node.parentElement; - if (!node.textContent.trim() || !element || !element.getClientRects().length) continue; - const style = getComputedStyle(element); - if (style.visibility === "hidden" || element.closest(".skip")) continue; - const [a, b] = [lum(parse(style.color)), lum(background(element))]; - const ratio = (Math.max(a, b) + 0.05) / (Math.min(a, b) + 0.05); - const size = parseFloat(style.fontSize); - const large = size >= 24 || (size >= 18.66 && Number(style.fontWeight) >= 700); - if (ratio < (large ? 3 : 4.5)) problems.push(`${ratio.toFixed(2)} "${node.textContent.trim().slice(0, 30)}"`); + page.on("console", message => message.type() === "error" && problems.push(message.text())); + page.on("pageerror", error => problems.push(String(error))); + page.on("request", request => { + const url = request.url(); + if (!url.startsWith("file:") && !url.startsWith("http://127.0.0.1:")) problems.push(url); + }); + await page.goto(url); + return { context, page, problems }; +} + +async function checkReport(browser, name, verdict) { + const url = `file://${runs}/${name}/index.html`; + for (const [scheme, viewport] of [["dark", { width: 1280, height: 800 }], ["light", { width: 390, height: 844 }]]) { + const { context, page, problems } = await open(browser, url, { colorScheme: scheme, viewport }); + // Task 1: the decision is the first content and fits on the first screen. + const banner = page.locator("main .verdict").first(); + assert.match(await banner.innerText(), verdict, `${name}: unexpected verdict`); + const first = await page.locator("main > *:not(.eyebrow):not(h1)").first().getAttribute("class"); + assert.match(first || "", /verdict/, `${name}: the verdict is not the first content`); + const box = await banner.boundingBox(); + assert(box.y + box.height <= viewport.height, `${name} ${scheme}: verdict below the first screen`); + // Task 2: keyboard users can skip straight to the content. + await page.keyboard.press("Tab"); + assert.equal(await page.evaluate(() => document.activeElement.className), "skip", `${name}: skip link`); + await axe(page, `${name} ${scheme}`); + assert.deepEqual(problems, [], `${name}: console errors or network requests`); + await context.close(); + } + // Task 5: 320 px wide (400% zoom) reflows without horizontal page scrolling. + const narrow = await open(browser, url, { viewport: { width: 320, height: 640 } }); + assert(await narrow.page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1), + `${name}: page overflows at 320px`); + await narrow.context.close(); + // Task 6: without JavaScript every row is still there and nothing is broken. + const plain = await open(browser, url, { javaScriptEnabled: false }); + assert.equal(await plain.page.locator("th button, .tools").count(), 0, `${name}: no-JS leftovers`); + assert.equal(await plain.page.locator("tbody tr[hidden]").count(), 0, `${name}: hidden rows without JS`); + await plain.context.close(); +} + +async function checkSortAndFilter(browser) { + // Task 3: sort the hillclimb steps by cost ratio with the keyboard. + const { context, page, problems } = await open(browser, `file://${runs}/climb/index.html`); + const header = page.getByRole("columnheader", { name: /Cost ratio/ }); + await header.getByRole("button").focus(); + await page.keyboard.press("Enter"); + assert.equal(await header.getAttribute("aria-sort"), "ascending"); + const table = page.locator("table", { has: header }); + const column = await header.evaluate(th => th.cellIndex); + const values = async () => (await table.locator(`tbody tr td:nth-child(${column + 1})`).allInnerTexts()) + .map(Number).filter(Number.isFinite); + const ascending = await values(); + assert.deepEqual(ascending, [...ascending].sort((a, b) => a - b), "ascending sort"); + await page.keyboard.press("Enter"); + assert.equal(await header.getAttribute("aria-sort"), "descending"); + const descending = await values(); + assert.deepEqual(descending, [...descending].sort((a, b) => b - a), "descending sort"); + assert.equal(await page.locator("[aria-sort]").count(), 1, "aria-sort on one header only"); + await axe(page, "climb after sorting"); + await context.close(); + // Task 4: filter the diff's changed checks to the regressions and back. + const diff = await open(browser, `file://${runs}/diff/index.html`); + const filter = diff.page.getByLabel("Filter rows").first(); + await filter.fill("regressed"); + const count = diff.page.locator(".count").first(); + assert.match(await count.innerText(), /^2 of \d+ rows$/); + const visible = await diff.page.locator("table").first().locator("tbody tr:not([hidden])").allInnerTexts(); + assert(visible.every(text => /regressed/.test(text)), "filtered rows match"); + await filter.fill(""); + assert.match(await count.innerText(), /^(\d+) of \1 rows$/); + await axe(diff.page, "diff with filter"); + // Task 7: printing keeps the verdict and drops interactive controls. + await diff.page.emulateMedia({ media: "print" }); + assert(await diff.page.locator(".verdict").first().isVisible()); + assert(await diff.page.locator(".tools").first().isHidden()); + assert.deepEqual([...problems, ...diff.problems], []); + await diff.context.close(); +} + +async function checkViewer(browser) { + // Task 8: browse every report from one local index and open one. + const server = spawn(evalarc, ["view", runs, "--port", "0", "--no-browser"], { stdio: ["ignore", "pipe", "pipe"] }); + const url = await new Promise((resolve, reject) => { + let text = ""; + server.stdout.on("data", chunk => { + text += chunk; + const match = text.match(/http:\/\/127\.0\.0\.1:\d+\//); + if (match) resolve(match[0]); + }); + server.on("exit", code => reject(new Error(`evalarc view exited ${code}`))); + setTimeout(() => reject(new Error("evalarc view did not start")), 15000); + }); + try { + const { context, page, problems } = await open(browser, url); + const rows = page.locator("tbody tr"); + assert.equal(await rows.count(), Object.keys(reports).length, "every report listed"); + assert.match(await page.locator("main").innerText(), /with a failing verdict/); + await axe(page, "viewer index"); + await page.getByRole("link", { name: "climb", exact: true }).click(); + await page.getByText(/Merge/).first().waitFor(); + assert.match(page.url(), /\/climb\/index\.html$/); + await page.goBack(); + await page.getByLabel("Filter rows").fill("Hillclimb"); + assert.equal(await page.locator("tbody tr:not([hidden])").count(), 1); + assert.deepEqual(problems, []); + await context.close(); + } finally { + server.kill(); } - return [...new Set(problems)].slice(0, 5); } (async () => { const browser = await chromium.launch(); - for (const [name, verdict] of Object.entries(reports)) { - for (const width of [1280, 390]) { - const page = await browser.newPage({ viewport: { width, height: 900 } }); - const problems = []; - page.on("console", message => message.type() === "error" && problems.push(message.text())); - page.on("request", request => !request.url().startsWith("file:") && problems.push(request.url())); - await page.goto(`file://${out}/${name}/index.html`); - assert.deepEqual(problems, [], `${name}: console errors or network requests`); - const banner = page.locator("main .verdict").first(); - assert(await banner.isVisible(), `${name}: verdict banner missing`); - assert.match(await banner.innerText(), verdict, `${name}: unexpected verdict`); - const first = await page.locator("main > *:not(.eyebrow):not(h1)").first().getAttribute("class"); - assert.match(first || "", /verdict/, `${name}: the verdict is not the first content`); - assert.equal(await page.locator("th:not([scope])").count(), 0, `${name}: unscoped header`); - assert.equal(await page.locator(".scroll:not([tabindex='0'])").count(), 0, `${name}: table not focusable`); - assert(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1), - `${name}: page overflows at ${width}px`); - assert.equal(await page.locator("html").getAttribute("lang"), "en"); - assert.deepEqual(await page.evaluate(contrastProblems), [], `${name}: contrast below WCAG AA`); - assert.equal(await page.locator("main").count(), 1, `${name}: main landmark`); - await page.keyboard.press("Tab"); - assert.equal(await page.evaluate(() => document.activeElement.className), "skip", `${name}: skip link`); - await page.close(); - } + try { + for (const [name, verdict] of Object.entries(reports)) await checkReport(browser, name, verdict); + await checkSortAndFilter(browser); + await checkViewer(browser); + } finally { + await browser.close(); } - await browser.close(); fs.rmSync(out, { recursive: true, force: true }); - console.log(`Checked ${Object.keys(reports).length} report pages at two widths.`); + console.log(`Checked ${Object.keys(reports).length} report kinds (light, dark, 390/320 px, no-JS, print), ` + + "keyboard sorting and filtering, and the local viewer with axe-core WCAG 2.2 AA."); })().catch(error => { console.error(error); process.exit(1); }); diff --git a/src/evalarc/assets/report_enhance.js b/src/evalarc/assets/report_enhance.js new file mode 100644 index 0000000..3d1b119 --- /dev/null +++ b/src/evalarc/assets/report_enhance.js @@ -0,0 +1,78 @@ +// Progressive enhancement for EvalArc offline reports. Inlined into every page and +// allowed by a CSP hash; without JavaScript every table still renders in full. +// Sortable columns follow the WAI-ARIA APG sortable table pattern (a button inside +// each header, aria-sort on the sorted column only). Tables with 8+ rows get a +// labelled row filter with a live count. Tables marked data-static are left alone. +(() => { + const number = text => { + const match = text.replace(/,/g, "").match(/[-+]?\d*\.?\d+/); + return match ? parseFloat(match[0]) : NaN; + }; + let filters = 0; + document.querySelectorAll("main table").forEach(table => { + const body = table.tBodies[0]; + const head = table.tHead; + if (!body || !head || body.rows.length < 3 || table.dataset.static !== undefined) return; + const rows = [...body.rows]; + const headers = [...head.rows[0].cells]; + headers.forEach((th, column) => { + const label = th.textContent.trim(); + if (!label) return; + const button = document.createElement("button"); + button.type = "button"; + const icon = document.createElement("span"); + icon.setAttribute("aria-hidden", "true"); + icon.textContent = "↕"; + button.append(document.createTextNode(label + " "), icon); + th.textContent = ""; + th.append(button); + button.addEventListener("click", () => { + const direction = th.getAttribute("aria-sort") === "ascending" ? "descending" : "ascending"; + headers.forEach(other => { + other.removeAttribute("aria-sort"); + const mark = other.querySelector("button span"); + if (mark) mark.textContent = "↕"; + }); + th.setAttribute("aria-sort", direction); + icon.textContent = direction === "ascending" ? "▲" : "▼"; + const key = row => (row.cells[column] ? row.cells[column].textContent.trim() : ""); + const sorted = [...body.rows].sort((x, y) => { + const a = key(x), b = key(y), na = number(a), nb = number(b); + const order = !isNaN(na) && !isNaN(nb) ? na - nb + : a.localeCompare(b, undefined, { numeric: true }); + return direction === "ascending" ? order : -order; + }); + sorted.forEach(row => body.append(row)); + }); + }); + if (rows.length < 8) return; + filters += 1; + const id = "filter-" + filters; + const tools = document.createElement("div"); + tools.className = "tools"; + const label = document.createElement("label"); + label.htmlFor = id; + label.textContent = "Filter rows"; + const input = document.createElement("input"); + input.type = "search"; + input.id = id; + input.autocomplete = "off"; + const count = document.createElement("span"); + count.className = "count"; + count.setAttribute("aria-live", "polite"); + const update = () => { + const query = input.value.trim().toLowerCase(); + let shown = 0; + rows.forEach(row => { + const hit = !query || row.textContent.toLowerCase().includes(query); + row.hidden = !hit; + shown += hit; + }); + count.textContent = `${shown} of ${rows.length} rows`; + }; + input.addEventListener("input", update); + tools.append(label, input, count); + (table.closest(".scroll") || table).before(tools); + update(); + }); +})(); diff --git a/src/evalarc/cli.py b/src/evalarc/cli.py index acbb929..5e19c81 100644 --- a/src/evalarc/cli.py +++ b/src/evalarc/cli.py @@ -373,6 +373,21 @@ def parser() -> argparse.ArgumentParser: climb.add_argument("--output", type=Path) climb.add_argument("--markdown", type=Path) climb.add_argument("--json", action="store_true") + view = commands.add_parser( + "view", + help="browse every report under a folder in your browser (local only)", + description=( + "Serve report folders under DIRECTORY on 127.0.0.1 with an index of their " + "verdicts, like inspect view or promptfoo view. Read-only; no network access " + "beyond loopback. --write-index writes the same index as a static page." + ), + ) + view.add_argument("directory", type=Path, nargs="?", default=Path("runs")) + view.add_argument("--port", type=int, default=7576) + view.add_argument("--no-browser", action="store_true", help="do not open a browser") + view.add_argument( + "--write-index", type=Path, metavar="FILE", help="write a static index page and exit" + ) project = commands.add_parser( "eval-init", help="create a runnable evaluation project: cases, app stub, grader, runner, loop config", @@ -532,6 +547,10 @@ def main(argv: list[str] | None = None) -> int: return code if args.command == "diff": return _results_diff(args) + if args.command == "view": + from evalarc.viewer import command as view + + return view(args) if args.command == "eval-init": from evalarc.eval_project import init_command diff --git a/src/evalarc/report.py b/src/evalarc/report.py index b185db7..dc1908d 100644 --- a/src/evalarc/report.py +++ b/src/evalarc/report.py @@ -1,63 +1,103 @@ -"""Offline HTML audit reports with no scripts or remote assets.""" +"""Offline HTML reports: self-contained, no remote assets, one CSP-pinned inline script.""" from __future__ import annotations +import base64 +import hashlib import html import json import re +from importlib.resources import files from pathlib import Path STYLE = """ -:root{color-scheme:dark;font:16px/1.6 system-ui,sans-serif;background:#101618;color:#e0ece7} -body{max-width:1080px;margin:60px auto;padding:0 24px} -.eyebrow{color:#86dbaf;letter-spacing:.18em;font-size:12px;font-weight:700} -h1{font-size:clamp(32px,6vw,64px);letter-spacing:-.045em;line-height:1.1;margin:16px 0} -p{color:#acbeb5;max-width:760px}.cards{display:flex;gap:20px;flex-wrap:wrap;margin:36px 0} -.card{border:1px solid #33463d;border-radius:12px;padding:18px 26px;flex:1;min-width:150px} -.number{font-size:38px;color:#96e8b9;font-weight:650}.label{color:#b7c6bf} -.scroll{overflow:auto}table{border-collapse:collapse;width:100%;font-size:14px} -td,th{text-align:left;padding:14px;border-bottom:1px solid #33463d}th{color:#86dbaf} -code{font-family:ui-monospace,monospace}h2{margin-top:42px} +:root{color-scheme:light dark;--bg:#101618;--surface:#152019;--surface-2:#17221c; +--text:#e0ece7;--muted:#acbeb5;--border:#33463d;--accent:#96e8b9;--accent-2:#86dbaf; +--bad:#edb68d;--warn:#e6d48a;--pass-bg:#152019;--fail-bg:#211b16;--warn-bg:#1f1d14; +--radius:12px;font:16px/1.6 system-ui,-apple-system,"Segoe UI",sans-serif; +background:var(--bg);color:var(--text)} +@media(prefers-color-scheme:light){:root{--bg:#ffffff;--surface:#f3f7f5;--surface-2:#eef3f0; +--text:#142019;--muted:#46564e;--border:#c9d6cf;--accent:#0b6b43;--accent-2:#0b6b43; +--bad:#a3420c;--warn:#7a5d00;--pass-bg:#eef8f2;--fail-bg:#fff4ec;--warn-bg:#fffbe6}} +*,*::before,*::after{box-sizing:border-box} +body{max-width:1120px;margin:40px auto;padding:0 24px} +.eyebrow{color:var(--accent-2);letter-spacing:.14em;font-size:12px;font-weight:700} +h1{font-size:clamp(28px,4vw,44px);letter-spacing:-.03em;line-height:1.15;margin:10px 0 18px} +h2{margin-top:40px;font-size:22px}h3{font-size:17px} +p{color:var(--muted);max-width:76ch}.cards{display:flex;gap:16px;flex-wrap:wrap;margin:24px 0} +.card{border:1px solid var(--border);border-radius:var(--radius);padding:16px 22px;flex:1; +min-width:150px;background:var(--surface)} +.number{font-size:32px;color:var(--accent);font-weight:650;font-variant-numeric:tabular-nums} +.label{color:var(--muted)} +.scroll{overflow:auto;max-height:70vh;border:1px solid var(--border);border-radius:var(--radius)} +table{border-collapse:collapse;width:100%;font-size:14px;font-variant-numeric:tabular-nums} +td,th{text-align:left;padding:10px 14px;border-bottom:1px solid var(--border);vertical-align:top} +th{color:var(--accent-2);background:var(--surface);position:sticky;top:0;z-index:1} +tbody tr:hover{background:var(--surface-2)}tbody tr:last-child td{border-bottom:0} +code{font-family:ui-monospace,SFMono-Regular,Menlo,monospace;font-size:.92em} .dimension{display:flex;gap:16px;align-items:center;max-width:620px;margin:10px 0} -.dimension span:first-child{width:150px}meter{flex:1;accent-color:#86dbaf} -footer{margin-top:48px;border-top:1px solid #33463d;padding-top:20px;font-size:13px} +.dimension span:first-child{width:150px}meter{flex:1;accent-color:var(--accent)} +footer{margin-top:48px;border-top:1px solid var(--border);padding-top:20px;font-size:13px; +color:var(--muted)} .metadata{overflow-wrap:anywhere;font-size:13px} -pre{overflow:auto;padding:18px;background:#17221c;font-size:12px;max-height:480px} -summary{cursor:pointer;color:#96e8b9}details{margin:16px 0} +pre{overflow:auto;padding:16px;background:var(--surface-2);font-size:12px;max-height:480px; +border-radius:8px;white-space:pre-wrap} +summary{cursor:pointer;color:var(--accent)}details{margin:12px 0} summary{min-height:44px;padding:10px 0} -a:focus-visible,summary:focus-visible{outline:3px solid #96e8b9;outline-offset:4px} -.audit-control{border:1px solid #33463d;border-radius:12px;padding:12px 18px} +a:focus-visible,summary:focus-visible,button:focus-visible,input:focus-visible, +.scroll:focus-visible{outline:3px solid var(--accent);outline-offset:3px} +.audit-control{border:1px solid var(--border);border-radius:var(--radius);padding:12px 18px} .audit-control summary{overflow-wrap:anywhere}.audit-control p{overflow-wrap:anywhere} -.audit-coverage{border-left:3px solid #edb68d;padding:12px 20px;background:#17221c} -@media(max-width:640px){body{margin:28px auto;padding:0 16px}.cards{gap:12px} -.card{min-width:0;flex-basis:100%}.dimension{flex-wrap:wrap}.audit-control{padding:10px 14px}} -a{color:#96e8b9}.failed,.agent_error,.environment_error{color:#edb68d} -.passed{color:#96e8b9}td{overflow-wrap:anywhere} -.scroll table{min-width:680px}td:first-child{min-width:170px;overflow-wrap:normal} -.change-cards{display:none}.change-card{border:1px solid #33463d;border-radius:12px;padding:16px} +.audit-coverage{border-left:3px solid var(--bad);padding:12px 20px;background:var(--surface-2)} +a{color:var(--accent)}.failed,.agent_error,.environment_error{color:var(--bad)} +.passed{color:var(--accent)}td{overflow-wrap:anywhere} +.scroll table{min-width:680px}td:first-child{min-width:150px;overflow-wrap:normal} +.change-cards{display:none} +.change-card{border:1px solid var(--border);border-radius:var(--radius);padding:16px} .change-card h3{font-size:16px;margin:0;overflow-wrap:anywhere}.change-card p{margin:10px 0 0} -@media(max-width:640px){.change-table{display:none}.change-cards{display:grid;gap:16px}} -.verdict{border:1px solid #33463d;border-left:6px solid #96e8b9;border-radius:12px; -padding:18px 24px;margin:28px 0;background:#152019} -.verdict.fail{border-left-color:#edb68d;background:#211b16}.verdict.warn{border-left-color:#e6d48a} -.verdict h2{margin:0;font-size:22px}.verdict p{margin:8px 0 0;color:#d2ded8} -.verdict .next{color:#e0ece7}.tone{white-space:nowrap} -.tone.ok{color:#96e8b9}.tone.bad{color:#edb68d}.tone.warn{color:#e6d48a}.tone.info{color:#acbeb5} -.findings{list-style:none;padding:0;display:grid;gap:12px;max-width:900px} -.findings li{border:1px solid #33463d;border-radius:10px;padding:12px 16px} -.findings code{color:#cfe3d9}.skip{position:absolute;left:-200vw} -.skip:focus{left:24px;top:12px;background:#101618;padding:8px} -.scroll:focus-visible{outline:3px solid #96e8b9;outline-offset:2px} +.verdict{border:1px solid var(--border);border-left:6px solid var(--accent); +border-radius:var(--radius);padding:16px 22px;margin:20px 0 8px;background:var(--pass-bg)} +.verdict.fail{border-left-color:var(--bad);background:var(--fail-bg)} +.verdict.warn{border-left-color:var(--warn);background:var(--warn-bg)} +.verdict h2{margin:0;font-size:21px}.verdict p{margin:6px 0 0;color:var(--text)} +.tone{white-space:nowrap}.tone.ok{color:var(--accent)}.tone.bad{color:var(--bad)} +.tone.warn{color:var(--warn)}.tone.info{color:var(--muted)} +.findings{list-style:none;padding:0;display:grid;gap:10px;max-width:900px} +.findings li{border:1px solid var(--border);border-radius:10px;padding:12px 16px} +.skip{position:absolute;left:-200vw} +.skip:focus{left:24px;top:12px;background:var(--bg);padding:8px;z-index:5} td code{overflow-wrap:anywhere} -@media(max-width:640px){.scroll table{min-width:560px}td,th{padding:10px} -main .cards .card{flex-basis:calc(50% - 6px);padding:12px 14px}main .number{font-size:26px}} -@media print{:root{color-scheme:light;background:#fff;color:#111}body{margin:0;max-width:none} -p,.label,.verdict p,.verdict .next,.tone.info{color:#222}th,a,.number,.eyebrow,.tone.ok{color:#064} -.tone.bad,.tone.warn,.failed{color:#8a3b00} -.verdict,.card,.findings li{background:#fff;break-inside:avoid} -pre{background:#f4f4f4;max-height:none}.scroll{overflow:visible}.skip{display:none}} +.tools{display:flex;gap:12px;align-items:center;flex-wrap:wrap;margin:8px 0} +.tools input{font:inherit;padding:8px 12px;border:1px solid var(--border);border-radius:8px; +background:var(--bg);color:var(--text);min-height:44px;min-width:min(320px,100%)} +th button{all:unset;cursor:pointer;display:inline-flex;gap:6px;align-items:center; +min-height:32px;color:inherit;font-weight:700} +th button:focus-visible{outline:3px solid var(--accent);outline-offset:2px;border-radius:4px} +th button span{opacity:.6}[aria-sort] button span{opacity:1} +.count{font-size:13px;color:var(--muted)} +@media(max-width:640px){body{margin:24px auto;padding:0 16px}.cards{gap:12px} +.card{min-width:0;flex-basis:100%}.dimension{flex-wrap:wrap}.audit-control{padding:10px 14px} +.change-table{display:none}.change-cards{display:grid;gap:16px} +.scroll table{min-width:560px}td,th{padding:8px 10px} +main .cards .card{flex-basis:calc(50% - 6px);padding:12px 14px}main .number{font-size:24px}} +@media print{:root{--bg:#fff;--surface:#fff;--surface-2:#f4f4f4;--text:#111;--muted:#222; +--border:#bbb;--accent:#064;--accent-2:#064;--bad:#8a3b00;--warn:#6a5000;--pass-bg:#fff; +--fail-bg:#fff;--warn-bg:#fff}body{margin:0;max-width:none} +.verdict,.card,.findings li,tr{break-inside:avoid}pre{max-height:none} +.scroll{overflow:visible;max-height:none;border:0}th{position:static} +.skip,.tools{display:none}} +@media(prefers-reduced-motion:reduce){*{scroll-behavior:auto!important}} """ +# Progressive enhancement (sortable columns, row filter), inlined from a packaged asset +# so each report stays one self-contained file; the CSP allows exactly this script. +ENHANCE = files("evalarc").joinpath("assets", "report_enhance.js").read_text(encoding="utf-8") +SCRIPT_HASH = base64.b64encode(hashlib.sha256(ENHANCE.encode()).digest()).decode() +CSP = ( + "default-src 'none'; style-src 'unsafe-inline'; " + f"script-src 'sha256-{SCRIPT_HASH}'; base-uri 'none'; form-action 'none'" +) + TONES = {"ok": "✓", "bad": "✗", "warn": "!", "info": "·"} @@ -147,11 +187,12 @@ def render_audit(data: dict, destination: Path) -> None: f"{len(fragile)} fault(s) depend on a single detecting case." if fragile else "", "Add cases that detect the single-case faults independently." if fragile else "", ) - document = """ + document = ( + """ - + + EvalArc · Grader audit " '' f'
EVALARC / {_esc(kind.upper())}
' - f"

{_esc(title)}

{body}
" + f"

{_esc(title)}

{body}" + f'' ) destination.parent.mkdir(parents=True, exist_ok=True) destination.write_text(document, encoding="utf-8") diff --git a/src/evalarc/viewer.py b/src/evalarc/viewer.py new file mode 100644 index 0000000..fa9c259 --- /dev/null +++ b/src/evalarc/viewer.py @@ -0,0 +1,229 @@ +"""Browse every EvalArc report under a folder in the browser: ``evalarc view``. + +The mainstream pattern for evaluation tools (``inspect view``, ``promptfoo view``, +``mlflow ui``) is a read-only local web server plus the user's browser, not a desktop +application. This viewer follows it with the standard library only: it binds to +127.0.0.1, serves files under one root, rejects other Host headers, and lists each +report folder with the verdict its page already states. ``--write-index`` writes the +same list as a static page for CI artifacts or GitHub Pages. +""" + +from __future__ import annotations + +import html +import http.server +import os +import re +import sys +import threading +import webbrowser +from pathlib import Path +from urllib.parse import quote, unquote, urlsplit + +from evalarc.report import _esc, _page, _table + +# Report JSON next to index.html identifies the report kind. +KINDS = { + "diff.json": "Results diff", + "health.json": "Eval health", + "hillclimb.json": "Hillclimb review", + "score.json": "Judge score", + "review.json": "Input review", + "evaluation.json": "Evaluation", + "comparison.json": "Comparison", + "repetition.json": "Repeatability", + "suite.json": "Suite", + "audit.json": "Grader audit", + "decisions.json": "Decision coverage", +} +MAX_REPORTS = 5000 +MAX_DEPTH = 6 +VERDICT = re.compile( + r'
]*>

(.*?)

(?:

(.*?)

)?', re.S +) +DEFAULT_PORT = 7576 + + +def discover(root: Path) -> list[dict]: + """Report folders under root: index.html plus a known report JSON.""" + root = root.resolve() + found = [] + for directory, subdirs, names in os.walk(root): + path = Path(directory) + depth = len(path.relative_to(root).parts) + subdirs[:] = sorted( + d for d in subdirs if not d.startswith(".") and d not in ("node_modules", "share") + ) + if depth >= MAX_DEPTH: + subdirs[:] = [] + if "index.html" not in names: + continue + kind = next((KINDS[name] for name in KINDS if name in names), None) + if kind is None: + continue + page = path / "index.html" + text = page.read_text(encoding="utf-8", errors="replace")[:200_000] + match = VERDICT.search(text) + state, title, detail = match.groups() if match else ("unknown", "", "") + found.append( + { + "path": path.relative_to(root).as_posix() or ".", + "kind": kind, + "state": state, + "title": _plain(title), + "detail": _plain(detail or ""), + "modified": page.stat().st_mtime, + } + ) + if len(found) >= MAX_REPORTS: + break + return sorted(found, key=lambda row: row["modified"], reverse=True) + + +def index_html(reports: list[dict], root_label: str, link_prefix: str = "") -> str: + """The index page: one row per report, newest first, with its verdict.""" + import tempfile + + tone = {"pass": ("ok", "✓ pass"), "fail": ("bad", "✗ fail"), "warn": ("warn", "! check")} + rows = [] + for row in reports: + cls, label = tone.get(row["state"], ("info", "· no verdict")) + href = quote(f"{link_prefix}{row['path']}/index.html".replace("./index.html", "index.html")) + rows.append( + [ + f'{_esc(label)}', + f'{_esc(row["path"])}', + _esc(row["kind"]), + _esc(row["title"]) + + (f"
{_esc(row['detail'])}" if row["detail"] else ""), + ] + ) + failing = sum(row["state"] == "fail" for row in reports) + summary = ( + f"

{len(reports)} report(s) under {_esc(root_label)}, newest first. " + f"{failing} with a failing verdict.

" + ) + body = summary + ( + _table("Reports", ["Verdict", "Report", "Kind", "Summary"], rows) + if rows + else "

No EvalArc reports found. Run a command with --output first.

" + ) + with tempfile.TemporaryDirectory() as temporary: + target = Path(temporary) / "index.html" + _page("Reports", "Every report in one place.", body, target) + return target.read_text(encoding="utf-8") + + +class _Handler(http.server.SimpleHTTPRequestHandler): + root: Path + allowed_hosts: set[str] + label: str + + def __init__(self, *args, **kwargs): + super().__init__(*args, directory=str(self.root), **kwargs) + + def log_message(self, *_): # quiet by default; the URL is printed once + pass + + def end_headers(self): + self.send_header("X-Content-Type-Options", "nosniff") + self.send_header("Referrer-Policy", "no-referrer") + self.send_header("Cache-Control", "no-store") + self.send_header("X-Frame-Options", "DENY") + super().end_headers() + + def _allowed(self) -> bool: + if self.headers.get("Host", "") not in self.allowed_hosts: + self.send_error(421, "Unrecognized Host") + return False + path = Path(self.translate_path(urlsplit(self.path).path)).resolve() + if not path.is_relative_to(self.root): + self.send_error(404) + return False + return True + + def do_GET(self): + if not self._allowed(): + return + route = unquote(urlsplit(self.path).path) + if route in ("/", "/index.html"): + page = index_html(discover(self.root), self.label, "/").encode("utf-8") + self.send_response(200) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(page))) + self.end_headers() + self.wfile.write(page) + return + if ( + Path(self.translate_path(route)).is_dir() + and not (Path(self.translate_path(route)) / "index.html").is_file() + ): + self.send_error(404) # no directory listings + return + super().do_GET() + + def do_HEAD(self): + if self._allowed(): + super().do_HEAD() + + def list_directory(self, path): # never list folders + self.send_error(404) + return None + + +def serve(root: Path, port: int, open_browser: bool) -> http.server.ThreadingHTTPServer: + root = root.resolve() + handler = type( + "Handler", + (_Handler,), + { + "root": root, + "label": str(root), + "allowed_hosts": {f"127.0.0.1:{port}", f"localhost:{port}"}, + }, + ) + server = http.server.ThreadingHTTPServer(("127.0.0.1", port), handler) + if port == 0: # tests: learn the chosen port and allow it + actual = server.server_address[1] + handler.allowed_hosts = {f"127.0.0.1:{actual}", f"localhost:{actual}"} + if open_browser: + url = f"http://127.0.0.1:{server.server_address[1]}/" + threading.Timer(0.3, lambda: webbrowser.open(url)).start() + return server + + +def command(args) -> int: + root = args.directory + if not root.is_dir(): + print(f"evalarc: {root} is not a directory", file=sys.stderr) + return 2 + if args.write_index: + target = args.write_index + prefix = os.path.relpath(root.resolve(), target.resolve().parent).replace(os.sep, "/") + prefix = "" if prefix == "." else prefix + "/" + reports = discover(root) + target.write_text(index_html(reports, str(root), prefix), encoding="utf-8") + print(f"Wrote {target} listing {len(reports)} report(s)") + return 0 + try: + server = serve(root, args.port, not args.no_browser) + except OSError as error: + print(f"evalarc: cannot listen on 127.0.0.1:{args.port}: {error}", file=sys.stderr) + return 2 + port = server.server_address[1] + print( + f"EvalArc reports at http://127.0.0.1:{port}/ (local only; Ctrl+C to stop)\n" + f"For a remote machine: ssh -L {port}:127.0.0.1:{port} HOST", + flush=True, + ) + try: + server.serve_forever() + except KeyboardInterrupt: + pass + finally: + server.server_close() + return 0 + + +def _plain(fragment: str) -> str: + return html.unescape(re.sub(r"<[^>]+>", "", fragment)).strip() diff --git a/tests/test_viewer.py b/tests/test_viewer.py new file mode 100644 index 0000000..95b19ba --- /dev/null +++ b/tests/test_viewer.py @@ -0,0 +1,106 @@ +import http.client +import json +import threading +from pathlib import Path + +import pytest + +from evalarc.cli import main +from evalarc.report import CSP, ENHANCE, SCRIPT_HASH +from evalarc.viewer import discover, serve + +ROOT = Path(__file__).resolve().parents[1] +INSPECT = ROOT / "examples/results-diff/inspect" + + +@pytest.fixture +def runs(tmp_path): + runs = tmp_path / "runs" + main( + [ + "diff", + str(INSPECT / "baseline.json"), + str(INSPECT / "current.json"), + "--output", + str(runs / "diff"), + ] + ) + main( + [ + "eval-health", + str(INSPECT / "baseline.json"), + str(INSPECT / "current.json"), + "--output", + str(runs / "nested/health"), + ] + ) + (runs / "notes").mkdir() + (runs / "notes/index.html").write_text("

not a report

") + return runs + + +@pytest.fixture +def server(runs): + instance = serve(runs, 0, open_browser=False) + thread = threading.Thread(target=instance.serve_forever, daemon=True) + thread.start() + yield instance.server_address[1] + instance.shutdown() + instance.server_close() + + +def get(port, path, host=None): + connection = http.client.HTTPConnection("127.0.0.1", port, timeout=10) + connection.putrequest("GET", path, skip_host=True) + connection.putheader("Host", host or f"127.0.0.1:{port}") + connection.endheaders() + response = connection.getresponse() + return response.status, response.read().decode("utf-8", "replace"), dict(response.headers) + + +def test_discover_lists_reports_with_their_verdict(runs): + reports = {row["path"]: row for row in discover(runs)} + assert set(reports) == {"diff", "nested/health"} + assert reports["diff"]["state"] == "fail" and "Gate failed" in reports["diff"]["title"] + assert reports["nested/health"]["kind"] == "Eval health" + + +def test_index_and_reports_are_served_locally(server): + status, page, headers = get(server, "/") + assert status == 200 and "2 report(s)" in page and 'href="/diff/index.html"' in page + assert headers["X-Content-Type-Options"] == "nosniff" + assert headers["X-Frame-Options"] == "DENY" + status, page, _ = get(server, "/diff/index.html") + assert status == 200 and "Gate failed" in page + assert get(server, "/diff/diff.json")[0] == 200 + + +@pytest.mark.parametrize( + "path,host,status", + [ + ("/", "evil.example", 421), # DNS rebinding: only loopback Host names are served + ("/../../../etc/passwd", None, 404), + ("/diff/", None, 200), # folder with index.html + ("/nested/", None, 404), # no directory listings + ("/missing.html", None, 404), + ], +) +def test_server_boundaries(server, path, host, status): + assert get(server, path, host)[0] == status + + +def test_write_index_uses_relative_links(runs, tmp_path, capsys): + target = tmp_path / "site/index.html" + target.parent.mkdir() + assert main(["view", str(runs), "--write-index", str(target)]) == 0 + page = target.read_text() + assert 'href="../runs/diff/index.html"' in page + assert main(["view", str(tmp_path / "missing")]) == 2 + + +def test_reports_allow_only_the_packaged_script(runs): + page = (runs / "diff/index.html").read_text() + assert f"'sha256-{SCRIPT_HASH}'" in CSP and CSP in page + assert page.count('