Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ make lint # custom architectural lint rules (CE001+)
make verify # All of the above + coverage check (CI equivalent)
```

When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE027–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name.)
When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's three onboarding surfaces honest: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, and every `github.com/marketplace/actions/<slug>` link plus the shields badge label must match `action.yml`'s `name:`.)

Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise.

Expand Down
26 changes: 17 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills

[![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
[![GitHub Marketplace](https://img.shields.io/badge/marketplace-coder__eval-2ea44f.svg)](https://github.com/marketplace/actions/coder_eval)
[![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com)
[![Docs](https://img.shields.io/badge/docs-coder--eval.com%2Fdocs-1f6feb.svg)](https://coder-eval.com/docs)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
[![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
Expand Down Expand Up @@ -97,12 +97,18 @@ the full setup.

## Use as a GitHub Action

A composite action at the repo root runs `coder-eval` as a CI gate — it installs
the pinned CLI, runs your tasks, writes a JUnit XML report, appends `run.md` to
the job summary, and fails the step on any task/gate failure:
A composite action — on the Marketplace as
[**coder_eval**](https://github.com/marketplace/actions/coder_eval) — runs
`coder-eval` as a CI gate. It installs the pinned CLI, runs your tasks, writes a
JUnit XML report, appends `run.md` to the job summary, and fails the step on any
task/gate failure:

```yaml
- uses: UiPath/coder_eval@v0 # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly
- uses: actions/setup-node@v4 # the claude-code agent needs the Claude CLI…
with: { node-version: '20' }
- run: npm install -g @anthropic-ai/claude-code

- uses: UiPath/coder_eval@v0 # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly)
with:
tasks: tests/tasks/**/*.yaml
model: claude-sonnet-5
Expand Down Expand Up @@ -154,9 +160,11 @@ code: the step fails if *either* coder-eval exits non-zero *or* any task's
alone.

> **Agent runtime is the caller's responsibility.** The action is agent-agnostic —
> it installs `coder-eval` but no coding-agent runtime. Tasks using the default
> `claude-code` agent need the `claude` CLI on `PATH` (`actions/setup-node` +
> `npm install -g @anthropic-ai/claude-code`) in the job before the action runs.
> it installs `coder-eval` but no coding-agent runtime, which is why the example
> above starts with `actions/setup-node` +
> `npm install -g @anthropic-ai/claude-code`: the default `claude-code` agent
> needs the `claude` CLI on `PATH` before the action runs. Swap those steps for
> your own agent's runtime as needed.

> **Security.** Evaluated tasks execute agent-generated code. Do **not** run this
> action under `pull_request_target` with secrets exposed to untrusted fork PRs —
Expand Down Expand Up @@ -188,7 +196,7 @@ alone.
| [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |
| [Dialog Mode](docs/DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user |
| [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver, with custom images |
| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor |
| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor |
| [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI |
| [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json |
| [How It Compares](docs/comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts |
Expand Down
36 changes: 23 additions & 13 deletions docs/CI_GATE.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,46 @@
---
description: >-
Run Coder Eval as a CI gate — the packaged composite GitHub Action, JUnit XML
output for test-report ingestion, and an optional per-task score floor.
Run Coder Eval as a CI gate — the coder_eval GitHub Action from the Actions
Marketplace, JUnit XML output for test-report ingestion, and an optional
per-task score floor.
---

# CI Gate: GitHub Action & JUnit reports

Coder Eval ships a **packaged CI gate**: a composite GitHub Action that installs
the CLI, runs your tasks, emits a JUnit XML report, appends the run summary to the
job summary, and fails the build on any task/gate failure. This page is the
reference for the Action and the JUnit output. For a step-by-step walkthrough
Coder Eval ships a **packaged CI gate**: a composite GitHub Action — on the
Actions Marketplace as
[**coder_eval**](https://github.com/marketplace/actions/coder_eval) — that
installs the CLI, runs your tasks, emits a JUnit XML report, appends the run
summary to the job summary, and fails the build on any task/gate failure. This
page is the reference for the Action and the JUnit output. For a walkthrough
(including a hand-rolled workflow), see
[Tutorial 02 — Running Coder Eval in CI](tutorials/02-ci-pipeline.md).

## The GitHub Action

A composite action lives at the repo root (`action.yml`), so you can reference it
directly:
The action is published on the GitHub Actions Marketplace as
[**coder_eval**](https://github.com/marketplace/actions/coder_eval). It is a
composite action living at the repo root (`action.yml`), so you reference it by
repo path — there is no Marketplace install step:

```yaml
- uses: UiPath/coder_eval@v0 # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly
- uses: actions/setup-node@v4 # the claude-code agent needs the Claude CLI…
with: { node-version: '20' }
- run: npm install -g @anthropic-ai/claude-code

- uses: UiPath/coder_eval@v0 # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly)
with:
tasks: tests/tasks/**/*.yaml
model: claude-sonnet-5
env: |
ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
```

The action is **agent-agnostic** — it installs `coder-eval` but *not* any
coding-agent runtime. Tasks using the default `claude-code` agent need the
`claude` CLI on `PATH` (Node + `@anthropic-ai/claude-code`), provided by your job
*before* this step runs.
The first two steps are there because the action is **agent-agnostic** — it
installs `coder-eval` but *not* any coding-agent runtime. Tasks using the default
`claude-code` agent need the `claude` CLI on `PATH` (Node +
`@anthropic-ai/claude-code`), provided by your job *before* the action runs; swap
those steps for your own agent's runtime as needed.

### Inputs

Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ New here? Start with **[Tutorial 01 — Your First Evaluation](tutorials/01-firs
| [Bring Your Own Dataset](DATASETS.md) | Fan a single task out over a dataset |
| [Dialog Mode](DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user |
| [Docker Isolation](DOCKER_ISOLATION.md) | The container sandbox driver, with custom images |
| [CI Gate & GitHub Action](CI_GATE.md) | Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor |
| [CI Gate & GitHub Action](CI_GATE.md) | Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor |
| [Extending Coder Eval](EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI |
| [Report Schema](REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json |
| [How It Compares](comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts |
Expand Down
2 changes: 1 addition & 1 deletion docs/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ and A/B plumbing.
- [Bring Your Own Dataset](https://coder-eval.com/docs/datasets): Fan a single task out over a dataset
- [Dialog Mode](https://coder-eval.com/docs/dialog-mode): Evaluate agents in multi-turn conversation via a simulated user
- [Docker Isolation](https://coder-eval.com/docs/docker-isolation): The container sandbox driver, with custom images
- [CI Gate & GitHub Action](https://coder-eval.com/docs/ci-gate): Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor
- [CI Gate & GitHub Action](https://coder-eval.com/docs/ci-gate): Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor
- [Extending Coder Eval](https://coder-eval.com/docs/extending): Author a custom agent, criterion, or model pricing via the plugin SPI
- [Report Schema](https://coder-eval.com/docs/report-schema): Field-level reference for run.json / variant.json / task.json
- [How It Compares](https://coder-eval.com/docs/comparison): vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts
Expand Down
6 changes: 4 additions & 2 deletions docs/tutorials/02-ci-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,8 +161,10 @@ jobs:
## Shortcut: the packaged action

The five steps above spell out the mechanics, but Coder Eval also ships a
composite action at the repo root that bundles install + run + JUnit report +
job-summary + fail-on-failure into one step:
composite action — on the Marketplace as
[**coder_eval**](https://github.com/marketplace/actions/coder_eval) — that
bundles install + run + JUnit report + job-summary + fail-on-failure into one
step:

```yaml
- uses: actions/setup-node@v4 # the claude-code agent needs the Claude CLI…
Expand Down
2 changes: 1 addition & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ extra:
DATASETS.md: "Fan a single task out over a dataset"
DIALOG_MODE.md: "Evaluate agents in multi-turn conversation via a simulated user"
DOCKER_ISOLATION.md: "The container sandbox driver, with custom images"
CI_GATE.md: "Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor"
CI_GATE.md: "Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor"
EXTENDING.md: "Author a custom agent, criterion, or model pricing via the plugin SPI"
REPORT_SCHEMA.md: "Field-level reference for run.json / variant.json / task.json"
comparison.md: "vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts"
Expand Down
Loading
Loading