Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/workflows/jev-scripts.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
name: jev-scripts
# Runs the advisory Jev layer's own unit tests (node:test, zero deps), including the
# boundary guard that keeps Jev out of the deterministic gate. NOT a gate itself - it
# tests the proposer-side scripts. Only fires when those scripts change.
on:
pull_request:
paths: ['scripts/jev/**']
push:
branches: [main]
paths: ['scripts/jev/**']
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: '22'
- run: node --test scripts/jev/*.test.mjs
1 change: 1 addition & 0 deletions docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ fails the verb instead of going unchecked.
- `npm-audit-ratchet.mjs` — ratcheted `npm audit` for the ts `audit` verb: fails on any critical not in `.audit-allowlist.json` and on stale entries, so accepted CVE debt can only shrink (npm audit has no native per-advisory ignore). Degrades to plain `npm audit --audit-level=critical` with no allowlist. Reads the report from stdin.
- `foundry-init.sh` — one-shot repo scaffold.
- `setup-labels.sh` — create the GitHub labels the workflows + ticket state machine need (agent:ready/working/blocked, align, ruleset-change, autofix). Idempotent; run by `foundry-init`.
- `jev/` — the **advisory** Jev layer: `client.mjs` (provider port lifted from jev-triage), `route.mjs` (pure routing/triage core), and `review.mjs` (runnable review pre-filter: `node scripts/jev/review.mjs [baseRef]` prints per-file `{review, lens, reason}` routing JSON). Near-free System-One decisions on the **proposer** side only: which diff hunks warrant deep review and on which lens. Opt-in via `JEV_API_KEY` (+ `JEV_PROVIDER`/`JEV_MODEL`/`TYPESAFE_AI_BASE_URL`); absent key ⇒ every caller fails open to current behavior (review all). **Never in the deterministic gate** — `boundary.test.mjs` reds the build if a gate workflow or mise verb references it; the `jev-scripts.yml` workflow runs the suite (`node --test scripts/jev/*.test.mjs`) on change.

## Agent half — `cmaintz-skills`

Expand Down
62 changes: 62 additions & 0 deletions scripts/jev/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# scripts/jev - Foundry's advisory Jev layer

Near-free [Jev](https://typesafe.ai) (TypeSafe System One) decisions for the
**proposer** side of Foundry: which diff hunks deserve deep review and on which
lens, and whether a turn looks destructive. This is where Jev earns its place -
upstream of the expensive LLM, never upstream of the deterministic oracle.

## The one rule

Jev is ~68% accurate, so it **never** sits in the authoritative gate
(`lint` / `typecheck` / `test` / `audit`). It only routes and triages: a wrong
call costs a missed shortcut, never a false pass. `boundary.test.mjs` fails the
build if any CI workflow or mise gate verb ever references this directory.

## Pieces

- `client.mjs` - the `JevProvider` port (`TypeSafeProvider`, `CloudflareProvider`,
`postJson` with 429/529 backoff) plus `providerFromEnv`, which returns `null`
when no key is set so every caller fails open to current behavior.
- `route.mjs` - pure, no-network core: `routeReview(answers, cfg)`,
`triageToolcall(answer, cfg)`, and the question templates. Fail-open is the
invariant: a missing or low-confidence answer always widens review, never narrows it.

## Usage

```js
import { providerFromEnv } from './scripts/jev/client.mjs';
import { reviewQuestions, routeReview } from './scripts/jev/route.mjs';

const jev = providerFromEnv();
if (!jev) return; // no key: skip Jev, review everything as before

const { answers } = await jev.evaluate({
state: { file, hunk, ticket },
questions: reviewQuestions(),
});
const { review, lens, reason } = routeReview(answers);
// review only the above-threshold hunks, on `lens`; log the rest (still get baseline review).
```

Or run the ready-made review pre-filter over a diff and consume its routing JSON:

```
node scripts/jev/review.mjs [baseRef] # baseRef defaults to origin/main
JEV_REVIEW_MIN_FILES=8 node scripts/jev/review.mjs # skip Jev on small diffs
```

It prints, per changed file, `{ review, lens, reason }`. No key (or a diff at/under the
min-files gate) routes every file to review - Jev only ever narrows spend, never the net.

## Config

Read from the environment (keys never in CI):
`JEV_API_KEY`, `JEV_PROVIDER` (`typesafe` | `cloudflare`), `JEV_MODEL`
(defaults to `jev-latest`), `CLOUDFLARE_ACCOUNT_ID`, and `TYPESAFE_AI_BASE_URL`
(point the direct call at a self-host, proxy, or mock).

## Tests

```
node --test scripts/jev/*.test.mjs
```
50 changes: 50 additions & 0 deletions scripts/jev/boundary.test.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
import assert from 'node:assert/strict';
import { existsSync, readdirSync, readFileSync } from 'node:fs';
import { join } from 'node:path';
import { test } from 'node:test';

// The invariant (spec foundry-jev-integration, deterministic oracle / probabilistic
// proposer): Jev is advisory-only and must NEVER be referenced by a GATE-defining file -
// a gate workflow or a mise verb. A reference there would put a ~68%-accurate model in
// the authoritative pass/fail path. This test reds the build if that happens.
//
// It scans the gate files explicitly (not every workflow) so a separate test-runner
// workflow that runs these very tests is allowed - that is not the gate. Run from the repo root.
const BANNED = 'scripts/jev';

const GATE_WORKFLOWS = [
'gate.yml',
'security.yml',
'tier0.yml',
'_ts.yml',
'_java.yml',
'_dotnet.yml',
'_php.yml',
'_guards.yml',
'_semgrep.yml',
].map((name) => join('.github/workflows', name));

function miseFiles() {
try {
return readdirSync('mise')
.filter((name) => name.endsWith('.toml'))
.map((name) => join('mise', name));
} catch {
return [];
}
}

function assertNoneReference(files, why) {
for (const file of files) {
if (!existsSync(file)) continue;
assert.ok(!readFileSync(file, 'utf8').includes(BANNED), `${file} references ${BANNED} - ${why}`);
}
}

test('no gate workflow references the jev scripts', () => {
assertNoneReference(GATE_WORKFLOWS, 'Jev must stay out of the CI gate');
});

test('no mise gate verb references the jev scripts', () => {
assertNoneReference(miseFiles(), 'Jev must stay off the deterministic gate');
});
81 changes: 81 additions & 0 deletions scripts/jev/client.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
// The Jev provider port for Foundry's advisory layer. Lifted from jev-triage's
// adapters, verified against https://docs.typesafe.ai/api (2026-09).
//
// ADVISORY ONLY. Nothing here may be imported by a gate verb (lint/typecheck/test/
// audit) or by gate.yml / tier0.yml - ruleset_guard.py fails the build if it is.
// Jev routes where to spend expensive LLM effort; it never decides pass/fail.
//
// Zero dependencies; uses global fetch (Node 18+).

const RETRYABLE = new Set([429, 529]);

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

/** POST JSON with the docs' recommended exponential backoff on 429/529. */
export async function postJson(url, headers, body, maxAttempts = 4) {
for (let attempt = 1; ; attempt++) {
const res = await fetch(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json', ...headers },
body: JSON.stringify(body),
});
if (res.ok) return res.json();
if (RETRYABLE.has(res.status) && attempt < maxAttempts) {
await sleep(250 * 2 ** (attempt - 1));
continue;
}
throw new Error(`Jev request failed: ${res.status} ${await res.text()}`);
}
}

/** TypeSafe first-party adapter: POST {baseUrl}/systemone, Bearer auth. */
export class TypeSafeProvider {
constructor(apiKey, model = 'jev-latest', baseUrl = 'https://api.typesafe.ai/v1') {
this.apiKey = apiKey;
this.model = model;
this.baseUrl = baseUrl;
}

async evaluate({ state, questions }) {
return postJson(
`${this.baseUrl}/systemone`,
{ Authorization: `Bearer ${this.apiKey}` },
{ model: this.model, state, questions },
);
}
}

/** Cloudflare Workers AI adapter: model in the body, no `result` envelope. */
export class CloudflareProvider {
constructor(accountId, apiToken, model = 'typesafe/jev') {
this.accountId = accountId;
this.apiToken = apiToken;
this.model = model;
}

async evaluate({ state, questions }) {
const url = `https://api.cloudflare.com/client/v4/accounts/${this.accountId}/ai/run`;
return postJson(
url,
{ Authorization: `Bearer ${this.apiToken}` },
{ model: this.model, input: { state, questions } },
);
}
}

/**
* Build a provider from the environment, or return null when no key is set so
* every caller fails open to current behavior (Jev is strictly opt-in).
*
* Reads: JEV_PROVIDER (typesafe|cloudflare), JEV_MODEL, JEV_API_KEY,
* CLOUDFLARE_ACCOUNT_ID, and TYPESAFE_AI_BASE_URL (self-host / proxy / mock).
*/
export function providerFromEnv(env = process.env) {
if ((env.JEV_PROVIDER ?? 'typesafe') === 'cloudflare') {
if (!env.CLOUDFLARE_ACCOUNT_ID || !env.JEV_API_KEY) return null;
return new CloudflareProvider(env.CLOUDFLARE_ACCOUNT_ID, env.JEV_API_KEY, env.JEV_MODEL || undefined);
}
if (!env.JEV_API_KEY) return null;
const baseUrl = env.TYPESAFE_AI_BASE_URL || 'https://api.typesafe.ai/v1';
return new TypeSafeProvider(env.JEV_API_KEY, env.JEV_MODEL || 'jev-latest', baseUrl);
}
83 changes: 83 additions & 0 deletions scripts/jev/client.test.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { CloudflareProvider, TypeSafeProvider, postJson, providerFromEnv } from './client.mjs';

const realFetch = globalThis.fetch;

function mockFetch(responses) {
const calls = [];
const queue = [...responses];
globalThis.fetch = async (url, init) => {
calls.push({ url, init });
const r = queue.shift();
return {
ok: r.status >= 200 && r.status < 300,
status: r.status,
json: async () => r.body,
text: async () => JSON.stringify(r.body ?? ''),
};
};
return calls;
}

async function withMock(responses, fn) {
const calls = mockFetch(responses);
try {
return await fn(calls);
} finally {
globalThis.fetch = realFetch;
}
}

test('postJson posts JSON and parses the response', async () => {
await withMock([{ status: 200, body: { hello: 'world' } }], async (calls) => {
const out = await postJson('https://x/y', { Authorization: 'Bearer k' }, { a: 1 });
assert.deepEqual(out, { hello: 'world' });
assert.equal(calls[0].init.method, 'POST');
assert.equal(calls[0].init.headers['Content-Type'], 'application/json');
assert.equal(calls[0].init.headers.Authorization, 'Bearer k');
assert.deepEqual(JSON.parse(calls[0].init.body), { a: 1 });
});
});

test('postJson retries on 429 then succeeds', async () => {
await withMock([{ status: 429 }, { status: 200, body: { ok: true } }], async (calls) => {
const out = await postJson('https://x/y', {}, {});
assert.deepEqual(out, { ok: true });
assert.equal(calls.length, 2);
});
});

test('postJson throws on a non-retryable status', async () => {
await withMock([{ status: 500, body: 'boom' }], async () => {
await assert.rejects(() => postJson('https://x/y', {}, {}, 1), /500/);
});
});

test('TypeSafeProvider posts to /systemone with model and auth', async () => {
await withMock([{ status: 200, body: { answers: {} } }], async (calls) => {
await new TypeSafeProvider('key', 'jev-latest').evaluate({ state: { q: 1 }, questions: { a: {} } });
assert.match(calls[0].url, /\/v1\/systemone$/);
const body = JSON.parse(calls[0].init.body);
assert.equal(body.model, 'jev-latest');
assert.deepEqual(body.state, { q: 1 });
assert.equal(calls[0].init.headers.Authorization, 'Bearer key');
});
});

test('providerFromEnv returns null without a key (fail-open)', () => {
assert.equal(providerFromEnv({}), null);
});

test('providerFromEnv builds a TypeSafe provider and honors the base-url override', () => {
const p = providerFromEnv({ JEV_API_KEY: 'k', TYPESAFE_AI_BASE_URL: 'http://localhost:9/v1' });
assert.ok(p instanceof TypeSafeProvider);
assert.equal(p.baseUrl, 'http://localhost:9/v1');
assert.equal(p.model, 'jev-latest');
});

test('providerFromEnv builds a Cloudflare provider, or null when the account is missing', () => {
const p = providerFromEnv({ JEV_PROVIDER: 'cloudflare', JEV_API_KEY: 'k', CLOUDFLARE_ACCOUNT_ID: 'acc' });
assert.ok(p instanceof CloudflareProvider);
assert.equal(providerFromEnv({ JEV_PROVIDER: 'cloudflare', JEV_API_KEY: 'k' }), null);
});
71 changes: 71 additions & 0 deletions scripts/jev/review.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
#!/usr/bin/env node
// v0.1 review pre-filter (runnable). Asks Jev, per changed file, whether it warrants
// deep review and on which lens, then prints routing JSON the review skill consumes.
//
// Advisory and fail-open: no key, or a diff at/under the min-files gate, routes every
// file to review (current behavior) - Jev only ever narrows spend, never the safety net.
//
// node scripts/jev/review.mjs [baseRef] # baseRef defaults to origin/main
// JEV_REVIEW_MIN_FILES=8 node scripts/jev/review.mjs # skip Jev on small diffs

import { execSync } from 'node:child_process';
import { providerFromEnv } from './client.mjs';
import { reviewQuestions, routeReview } from './route.mjs';

/** Route each changed file: ask Jev per file, or review-all when Jev is unavailable. */
export async function routeFiles(provider, files, minFiles = 0) {
if (!provider || files.length <= minFiles) {
const reason = provider ? `at or below the ${minFiles}-file gate` : 'no JEV_API_KEY (review all)';
return files.map((f) => ({ file: f.file, review: true, lens: null, reason }));
}
const routed = [];
for (const { file, patch } of files) {
const { answers } = await provider.evaluate({ state: { file, diff: patch }, questions: reviewQuestions() });
routed.push({ file, ...routeReview(answers) });
}
return routed;
}

/** Split `git diff` text into one patch per file (skips deletions to /dev/null). */
export function parseDiff(text) {
const out = [];
let current = null;
for (const line of text.split('\n')) {
if (line.startsWith('diff --git')) {
if (current) out.push({ file: current.file, patch: current.lines.join('\n') });
current = { file: fileFromHeader(line), lines: [line] };
} else if (current) {
if (line.startsWith('+++ ')) current.file = plusPath(line) ?? current.file;
current.lines.push(line);
}
}
if (current) out.push({ file: current.file, patch: current.lines.join('\n') });
return out.filter((d) => d.file !== '/dev/null');
}

function fileFromHeader(line) {
const match = /^diff --git a\/.+ b\/(.+)$/.exec(line);
return match?.[1] ?? 'unknown';
}

function plusPath(line) {
const path = line.slice(4).trim();
if (path === '/dev/null') return '/dev/null';
return path.startsWith('b/') ? path.slice(2) : path;
}

async function main() {
const baseRef = process.argv[2] ?? 'origin/main';
const diff = execSync(`git diff ${baseRef}`, { encoding: 'utf8', maxBuffer: 32 * 1024 * 1024 });
const minFiles = Number(process.env.JEV_REVIEW_MIN_FILES ?? '0');
const routed = await routeFiles(providerFromEnv(), parseDiff(diff), minFiles);
process.stdout.write(`${JSON.stringify(routed, null, 2)}\n`);
}

// Run only as a CLI, not when imported by tests.
if (import.meta.url === `file://${process.argv[1]}` || process.argv[1]?.endsWith('review.mjs')) {
main().catch((err) => {
process.stderr.write(`jev-review: ${err?.message ?? err}\n`);
process.exit(0); // advisory: never break the caller
});
}
Loading
Loading