Skip to content

Commit b8e5ff1

Browse files
cyyehclaude
andcommitted
Check every push with ruff, mypy and the tests, and test that the agent never imports the app
The rename to agent/ left 22 files' imports out of order (agent.model_* now sorts before agent.harnesses), nothing checked the rule that agent/ never imports app/ or Streamlit, and there was no linter or type checker at all. Now: Lint - Ruff, set up in pyproject.toml: pyflakes and pycodestyle's errors, import order, bugbear and pyupgrade. No formatter, so it never rewrites a line it has nothing to say about; zip()'s strict= and named lambdas are left off, with the reason beside them. - Imports sorted everywhere (tests/'s helpers sort with agent and app, as src), three unused imports gone, Callable and Iterator from collections.abc, two loop variables marked unused. - An f-string in test_render.py reused its quotes, which only Python 3.12 parses, though requires-python says 3.11. Types - mypy, at its default strictness, with the bodies of agent/'s and app/'s unannotated functions checked too (SimpleHarness.run, the page); the tests only where annotated. Stubs for PyYAML, Pygments and openpyxl in the dev group. - No behavior changes. SimpleBackend, VLLMBackend and FakeLLM were not Backends to a type checker: each typed its drafter (and VLLMBackend its drafter_id) as None alone. Encoded.ids is Any, the form each backend's generate() takes; pick() is generic, as the guide already called it with tuples. The rest is annotations on empty dicts and lists, Sequence where a default is (), asserts where a value is known not to be None (a tool call without errors has a tool, generate() ends with done, Popen's pipes), PiHarness.pi for the pi run() started, and format_func=names.__getitem__, which returns a str where dict.get may not. CI - .github/workflows/checks.yml runs ruff check, mypy and uv run pytest on Ubuntu on every push and pull request, each even when the one before failed. The agent never knows the app - tests/test_layout.py reads every file of agent/ for an import of app or streamlit, at the top or inside a function, and checks that the check itself sees each way of importing. Also - .gitignore: .mypy_cache/, .ruff_cache/ and .impeccable/. - .nvmrc picks Node 22 for pi with nvm (`nvm use`), which the READMEs' pi section and AGENTS.md now say; nvm keeps global packages per version, so pi is installed under it. - The READMEs tell a clone from before the rename that the empty-looking harness/ it keeps is only the __pycache__ folders git ignores, which `git clean -fdX harness` removes. - AGENTS.md and both READMEs list the commands, the workflow and the new test. - How it works: the guide's text was reread for every changed excerpt (annotations, a renamed local, the drafting condition now `mtp is not None`); none of it became untrue, and tests/guide_excerpts.txt is rewritten. One reference follows the line it points at. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent a8d22f3 commit b8e5ff1

44 files changed

Lines changed: 726 additions & 214 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/checks.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Checks every push and pull request: ruff's lint (the order of imports with it), mypy's types, and the tests that
2+
# load no weights and touch no network (the ones that need the models, Docker's images, a vLLM server or pi skip).
3+
# mypy and the tests import the whole app, so the job first installs every package uv.lock pins, torch among them.
4+
name: Checks
5+
6+
on:
7+
push:
8+
pull_request:
9+
10+
permissions:
11+
contents: read
12+
13+
concurrency:
14+
group: checks-${{ github.event_name }}-${{ github.ref }}
15+
cancel-in-progress: true
16+
17+
jobs:
18+
check:
19+
runs-on: ubuntu-latest
20+
timeout-minutes: 30
21+
steps:
22+
- uses: actions/checkout@v7
23+
- uses: astral-sh/setup-uv@v10.3.0
24+
with:
25+
version-file: pyproject.toml # its required-version: no uv older than the fix for the revoked emSign certificates
26+
- id: sync
27+
run: uv sync --locked
28+
# Each runs even when the one before failed, so that one push reports all three.
29+
- run: uv run ruff check
30+
- if: ${{ !cancelled() && steps.sync.outcome == 'success' }}
31+
run: uv run mypy
32+
- if: ${{ !cancelled() && steps.sync.outcome == 'success' }}
33+
run: uv run pytest

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
.venv/
22
__pycache__/
33
.pytest_cache/
4+
.mypy_cache/
5+
.ruff_cache/
6+
.impeccable/
47
_site/
58
/models/
69
.DS_Store

‎.nvmrc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
22

‎AGENTS.md‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,20 +17,24 @@ page shows, the architecture, the tests and the known limits; read the parts you
1717
```bash
1818
uv sync # Python 3.12 (.python-version), packages pinned by uv.lock; uv >= 0.13.4 (required-version in pyproject.toml)
1919
uv run pytest # ~40 s; loads no weights, touches no network
20+
uv run ruff check # lint, the order of imports included; --fix sorts them
21+
uv run mypy # types: agent/ and app/ inside every function, tests/ where annotated
2022
RUN_MODEL_TESTS=1 uv run pytest tests/test_real_model.py # real Gemma (with its MTP drafter) and Qwen, ~4 min; -k Qwen for one
2123
docker build -t agent-loop-sandbox agent/sandbox # optional: run_python's sandbox; its tests skip without it; bump SANDBOX_VERSION with the Dockerfile
2224
uv run streamlit run streamlit_app.py # http://localhost:8501; MODEL_ID=Qwen/Qwen3.5-0.8B starts on Qwen
2325
VLLM_URL=http://localhost:8000 uv run streamlit run streamlit_app.py # the model on a vLLM server: vllm serve <model> --revision <its REVISIONS commit> [--enable-scale-out [--return-sampling-mask --logprobs-mode processed_logprobs]]
2426
VLLM_TEST_URL=http://localhost:8000 uv run pytest tests/test_vllm_backend.py -k real_server # vllm_backend.py against that real server
2527
docker compose -f docker-compose/compose.yaml up -d # the vLLM server the sidebar's vLLM (Docker) starts, by hand (+ -f docker-compose/compose.gpu.yaml on NVIDIA GPUs)
2628
RUN_VLLM_DOCKER_TESTS=1 uv run pytest tests/test_vllm_docker.py -k real # starts that server with Qwen3.5 and asks it through VLLMBackend, ~2 min
27-
RUN_PI_TESTS=1 uv run pytest tests/test_pi_harness.py -k real # the real pi (on the PATH, or PI_COMMAND; pi 1.4 needs Node 22.19+) against a scripted model on 127.0.0.1
29+
RUN_PI_TESTS=1 uv run pytest tests/test_pi_harness.py -k real # the real pi (on the PATH, or PI_COMMAND; pi 1.4 needs Node 22.19+, which .nvmrc picks with nvm) against a scripted model on 127.0.0.1
2830
uv run python -m app.guide _site # the guide: _site/zh/, _site/en/ and a redirecting _site/index.html
2931
UPDATE_GUIDE_SNAPSHOT=1 uv run pytest tests/test_guide.py # after rereading the guide sections a code change touched
3032
```
3133

32-
No linter or formatter is configured. Match the surrounding code by hand and don't reformat lines you
33-
are not changing.
34+
Ruff lints and mypy checks types (both set up in pyproject.toml), and .github/workflows/checks.yml runs them and
35+
`uv run pytest` on every push and pull request. No formatter is configured: match the surrounding code by hand and
36+
don't reformat lines you are not changing. Fix what the two report rather than silencing it; an ignore needs a reason
37+
beside it.
3438

3539
## Layout
3640

@@ -72,8 +76,9 @@ import Backend`, `from app import render`); pytest puts that folder on the path
7276
## Rules that keep it working
7377

7478
- **The agent never knows the app.** Nothing in `agent/` imports `app/`, Streamlit or anything that draws; it
75-
reports what happens only as events. `agent/__init__.py`, `agent/model_families/`, `agent/i18n.py` and
76-
`app/__init__.py` import nothing heavy, because the guide's build loads them with pygments alone.
79+
reports what happens only as events (tests/test_layout.py checks the imports). `agent/__init__.py`,
80+
`agent/model_families/`, `agent/i18n.py` and `app/__init__.py` import nothing heavy, because the guide's build
81+
loads them with pygments alone.
7782
- **app/page.py is imported once, not run as a script.** streamlit_app.py imports it and calls `main()` on every
7883
rerun, so everything a rerun does is inside functions, and what the module reads from the environment (`VLLM_URL`,
7984
`MODEL_ID`, `AGENT_LOOP_UPLOADS`) is read once per process. Tests change those on the module (`page.VLLM_URL`,

‎README.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,8 @@ To run the model on vLLM instead, pick vLLM (Docker) under What runs the model i
6565

6666
> A 401 while downloading, though the model is public? Most likely the Hugging Face token saved on your machine has expired. Log in again (`uv run hf auth login`), or add `HF_HUB_DISABLE_IMPLICIT_TOKEN=1` to send no token.
6767
68+
> Cloned before `agent/` was called `harness/`? Git ignores the `__pycache__` folders Python wrote there, so a pull leaves them behind, in what looks like an empty `harness/`. Nothing imports it; `git clean -fdX harness` removes it, and only what git ignores in it.
69+
6870
## Experiments to try
6971

7072
1. **Just a chatbot**: switch off Use tools in the sidebar and ask "What time is it in Taipei right now?". The model sees no tools, so it can only guess or admit it does not know. Switch tools back on, ask again, and compare the two in the Context tab.
@@ -83,7 +85,7 @@ To run the model on vLLM instead, pick vLLM (Docker) under What runs the model i
8385

8486
## Architecture
8587

86-
The project has two parts. `agent/` runs a turn: the harness that runs the agent loop, the tools, the model's family and what runs the model. `app/` is what you see: the page, the HTML it draws from the agent's events, and How it works. They depend one way: `app/` builds on `agent/`, and nothing in `agent/` imports `app/` or Streamlit, so the loop runs, and is tested, without a page.
88+
The project has two parts. `agent/` runs a turn: the harness that runs the agent loop, the tools, the model's family and what runs the model. `app/` is what you see: the page, the HTML it draws from the agent's events, and How it works. They depend one way: `app/` builds on `agent/`, and nothing in `agent/` imports `app/` or Streamlit (`test_layout.py` checks), so the loop runs, and is tested, without a page.
8789

8890
```
8991
streamlit_app.py where Streamlit starts: it runs app/page.py's main() on every rerun
@@ -249,6 +251,7 @@ The page can also hand the loop to [pi](https://pi.dev), a coding agent with a l
249251
- **pi's own models**: one pi has credentials for (Claude, GPT, Gemini, or any compatible endpoint in its `models.json`); no model loads here. An API sends back text, not tokens, so there are no token probabilities, candidates or special tokens, and how the API writes the request out for the model nobody says: a step shows pi's request, the text as it streams, the token counts the provider reported (with how many came from the cache) and the cost at pi's prices.
250252

251253
```bash
254+
nvm use # with nvm: Node 22, from .nvmrc; install pi under it, since nvm keeps global packages per version
252255
npm install -g @earendil-works/pi-coding-agent # pi 1.4 needs Node.js 22.19 or newer
253256
pi # then /login, once; or set an API key such as ANTHROPIC_API_KEY before starting the app
254257
PI_COMMAND=/path/to/pi uv run streamlit run streamlit_app.py # if pi is not on the PATH (any command works, npx included)
@@ -306,6 +309,8 @@ Qwen stops at `<|im_end|>` both after an answer and for a tool, so the harness d
306309

307310
```bash
308311
uv run pytest # about 40 s; loads no weights and touches no network
312+
uv run ruff check # lint: likely bugs, unused imports and the order of imports (--fix sorts them)
313+
uv run mypy # types: agent/ and app/ inside every function, tests/ where annotated
309314
RUN_MODEL_TESTS=1 uv run pytest tests/test_real_model.py # loads the real Gemma (with its MTP drafter) and Qwen, about 4 minutes; add -k Qwen for one
310315
VLLM_TEST_URL=http://localhost:8000 uv run pytest tests/test_vllm_backend.py -k real_server # asks a running vLLM server
311316
RUN_VLLM_DOCKER_TESTS=1 uv run pytest tests/test_vllm_docker.py -k real # starts vLLM in Docker with Qwen3.5 and asks it
@@ -316,6 +321,8 @@ Most tests use the fake model in `tests/fakes.py`: it replays scripted output on
316321

317322
Every file format the page accepts has a test: `tests/samples.py` makes a small file of each one from the examples' files (the receipt in every image format, the recording in every audio format, the rolling ball in every video container, the sales data in every data format). `test_media.py` fails if `media.py` accepts a format without a sample, and `test_real_model.py` asks both models about each one. The tests that need Docker run only when Docker and the sandbox image are there.
318323

324+
On every push and pull request, `.github/workflows/checks.yml` runs `ruff check`, `mypy` and `uv run pytest` on Ubuntu; the tests that need the models, the sandbox image, a vLLM server or pi skip there.
325+
319326
How it works quotes code, and `tests/guide_excerpts.txt` keeps everything it quotes, without line numbers. When a change touches quoted code, `test_guide.py` fails and names the sections that quote it: reread them, fix what they say if it is no longer true, then rewrite the file with `UPDATE_GUIDE_SNAPSHOT=1 uv run pytest tests/test_guide.py`.
320327

321328
| File | What it tests |
@@ -335,6 +342,7 @@ How it works quotes code, and `tests/guide_excerpts.txt` keeps everything it quo
335342
| `test_vllm_docker.py` | Starting vLLM in Docker, with Docker faked at the level of `subprocess`: the weights downloaded at their pinned commit before `docker compose up` runs `docker-compose/compose.yaml` with the model, its revision and `models/`, and `compose.gpu.yaml` on top where Docker has the nvidia runtime (or `AGENT_LOOP_VLLM_DEVICE` says so); a server already up for the model not started again, another model replacing it; no Docker, Docker not running, no Compose, `docker compose up` failing, a container killed for memory, one that exits (the exceptions in its log, cause first) and one that never answers, each said in both languages; a rerun while Compose is busy stopping it too; `docker compose down`; the model the app asked for kept through a failed start until it is stopped, and quitting the app stopping the server it started (and leaving Docker alone without one); on the GPU, MTP with Gemma's drafter at its pinned commit (downloaded first) or Qwen's own MTP layer, never on the CPU or unasked, and switching it starting the server again. The compose files: on 127.0.0.1 only, `models/` read-only and offline, Qwen3.5 at its pinned commit by default, the GPU file the same vLLM with the same flags but the CPU-only one, plus a KV cache of the CPU's size, a low `--gpu-memory-utilization`, and speculative decoding off by hand, its counts in detail only with a config (vLLM refuses them without one), Docker Compose handing vLLM the page's config as it is, and Docker Compose reading both (when installed) and mounting the folder a relative `AGENT_LOOP_MODELS` names; with `RUN_VLLM_DOCKER_TESTS=1`, the real server in Docker answering through `VLLMBackend`, with MTP and then without where Docker has a GPU |
336343
| `test_media.py` | Every format the page accepts has a sample and is read as what it is (sound is a file for Qwen) and decodes; names saved distinct and plain; the question's parts and the note naming a text file's columns; photos turned upright, sound at the model's rate, a video's frames evenly spaced at their own rate; the type each clip plays back as, and the frames shown for an MKV video |
337344
| `test_guide.py` | Every excerpt and line number on How it works resolves and links to its lines on GitHub at the page's commit, the event table points at the line that really sends each event, HTML tags pair up; the code each section quotes matches `tests/guide_excerpts.txt`, which an edit to that code changes and a line merely moving does not; each paragraph points at the same code and sections in both languages, with no Chinese left in the English one; every set of tabs is radio buttons of known backends or models, one panel each, that switch without a script, and an excerpt of `compose.yaml` ends where it says; the built pages have light and dark palettes and loop diagrams and name their commit, and the root page sends readers to the right language |
345+
| `test_layout.py` | No file in `agent/` imports `app` or Streamlit, at the top or inside a function, read from each file's syntax tree: the agent never knows the app |
338346
| `test_real_model.py` | Skipped by default. For each model: the loaded model is a `Backend`; greedy decoding matches `model.generate()`, with text and with an image, a video and (Gemma) a recording, and a text prompt after them too; the real model calls the calculator and answers; it reads the receipt's total in every image format, transcribes the recording in every audio format (Gemma; Qwen gets it as a file), tells which way the ball rolls in every video container, opens every data format with `run_python`, and writes code that runs in the sandbox (with Docker); for Gemma with MTP, greedy output is exactly the same as without it, with an image too, and the drafter's guesses really are kept; on CUDA, a warmed-up model let go hands all its GPU memory back; with pi installed, the real pi asks the model through the model endpoint, calls bash and answers, every token drawn |
339347

340348
## Known limitations

0 commit comments

Comments
 (0)