Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .github/ISSUE_TEMPLATE/early-check-recipe.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: Share an early-check recipe
description: Show how to catch a mistake earlier. No test code required.
title: "[Early check] "
body:
- type: textarea
id: behavior
attributes:
label: What went wrong?
description: Explain what you did and what happened. Use synthetic examples and do not include secrets or private data.
placeholder: "I saved a new note, then reloaded the page. The note was gone."
validations:
required: true
- type: textarea
id: expected
attributes:
label: What would count as passing?
description: Describe what should happen and, if you know, one normal case that should still work.
validations:
required: true
- type: textarea
id: early_point
attributes:
label: When could we first catch it?
description: For example, while writing the save function, before the screen exists. It is fine to leave this blank if you are not sure.
- type: textarea
id: automated_check
attributes:
label: Can a machine check it? (optional)
description: If available, include a trusted runner or command and what makes its result inconclusive rather than a pass or fail.
10 changes: 8 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
# Contribute an independent check
# Contribute to Test Feedback Loop

Share checks that catch a specific broken behavior and can verify the repair with the same input. A small reproducible example is enough to start a discussion.
## Share an early-check recipe

[Open a recipe issue](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=early-check-recipe.yml). No test code or new tool is required. Describe the wrong behavior, the earliest point it could be checked, and what you would observe on pass or fail. If you don't know where to automate it, share what you can reproduce by hand.

For a machine-checkable version, use an existing tool where possible. Include the command or check, a failing example, a known-good control, and what makes the check inconclusive. Use synthetic data; don't include private logs, credentials, or customer information.

## Add a runner adapter

1. [Propose a check](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=check-adapter.yml). Show the runner and how it reports an assertion failure, a pass, and an incomplete or ambiguous run.
2. For an adapter PR, keep execution and classification on the caller side. Follow the [observation contract](SPEC.md) and use the existing [Node.js adapter](src/checkers/node-test.mjs) as a reference.
Expand Down
21 changes: 16 additions & 5 deletions README.ja.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
# Test Feedback Loop

**独立した検査で、小さめのモデルでも実装を速く、少ないトークンで回す。**
**実装も修正も Low で。間違いは、まだ直しやすいうちに見つける。**

対象は、計画・設計・指揮が済んだあとに手を動かす実装工程です。PDCA の **C(検査)を短く回し**、壊れた箇所と期待する挙動を渡せば、実装役が調べ直す範囲を絞れます。小さめのモデルや Low で修正し、同じ検査で品質を確かめるための道具です。
AIには小さな単位で実装や修正を任せ、各段階で最初に機械判定できる検査を、AI自身の判断とは別に実行します。「もっと考えて」ではなく、検査で分かったことを次の一手に返します。検査で判断できないときは、推測で修正を重ねず止まります。

たとえば「保存できたと出るのに、再読み込みするとメモが消える」。画面全体の完成を待たず、保存処理やAPIができた時点で「書く → 読み直す」を確かめられます。既存のメモを編集できることも別に確認し、画面でしか分からない部分は後でブラウザーから検査します。

[English](README.md) · [简体中文](README.zh-CN.md) · [Agent Skill](skills/test-feedback-loop/SKILL.md)

**AIでアプリを作っている人へ:** [困った挙動と確認方法を共有する](#早く気づくための確認方法を持ち寄る)。コードは不要です。**自動化したい人へ:** [デモを試す](#修正から再検査まで試す)、または[自分のプロジェクトで確かめる](#自分のプロジェクトで確かめる)。

## 修正から再検査まで試す

Node.js 24 以上 を使います。次の2つのデモは、モデルのアカウントや API キーなしで動きます。
Expand Down Expand Up @@ -65,11 +69,18 @@ npm run check:pytest -- --cwd /path/to/project --python /path/to/venv/bin/python

別のプロジェクトから直接呼ぶ場合は、このチェックアウトで `npm link` を実行し、対象プロジェクトで `tfl-check --help` を実行します。Python のテストなら `tfl-check --runner pytest --python /path/to/venv/bin/python tests/test_regression.py` です。任意のプロキシも `test-feedback-loop` として起動できます。

## 独立検査を持ち寄る
## 早く気づくための確認方法を持ち寄る

集めるのは、**何が壊れるか、どの段階なら気づけるか、何を見れば合否が分かるか**の実例です。人が確認できる手順だけでも投稿できます。自動化できる場合は、コマンドと、判定できない場合の扱いも添えてください。

| 層 | 例:保存したメモが消える |
| --- | --- |
| 人が確認できる挙動 | 新規メモを保存して再読み込みし、内容が残れば合格。既存メモの編集も引き続きできることを確かめる。 |
| 最初に機械判定できる地点 | 保存処理やAPIができた時点で、別のクライアントから読み直して内容が一致するか検査する。保存先に接続できないなら不合格ではなく未確認。画面固有の確認は後で足す。 |

pytest、Vitest、TypeScript、Cargo、スキーマ検証などで使っている検査を募集します。失敗する最小例、通る再検査、試験結果を確定できない例を添えて、[検査の提案 Issue](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=check-adapter.yml) に出せます。動くアダプタがある場合は、同じ例を試験にして PR にできます。[参加ガイド](CONTRIBUTING.md)
[確認方法を投稿する(コード不要)](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=early-check-recipe.yml) · [検査アダプタを提案する](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=check-adapter.yml) · [参加ガイド](CONTRIBUTING.md)

Node.js と pytest の検査アダプタを同梱しています。
標準装備は既存のテストツールです。Node.js と pytest の検査アダプタを同梱していますが、確認方法の投稿に新しいアダプタは必要ありません。

## 任意のエフォート切り替え

Expand Down
21 changes: 16 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
# Test Feedback Loop

**Independent checks for fast, token-efficient implementation with smaller models.**
**Build on Low. Catch mistakes while they’re still cheap to fix.**

Once planning, design, and coordination have established the task, give the implementation agent a narrow, repeatable failure and the expected behavior. A short Check loop in Plan–Do–Check–Act gives a smaller model or a low-effort turn a concrete change to make. Independently rerun the failing case and a normal control to verify the result.
Use your coding agent’s Low setting for small implementation steps and repairs. After each step, run the earliest check that can decide whether the behavior is right, separately from the agent’s own judgment. Feed back the evidence, not a request to “try harder.” If the check can’t decide, stop and find out why.

Say your app claims it saved a note, but the note disappears on reload. You don’t need to wait for the whole screen: test save → read at the storage or API boundary, as soon as that code exists. Keep a working edit case as a control. Later, use a browser check for the parts only the screen can prove.

[日本語](README.ja.md) · [简体中文](README.zh-CN.md) · [Agent Skill](skills/test-feedback-loop/SKILL.md)

**Building with AI, not writing tests?** [Describe the behavior and the earliest way to check it](#share-an-early-check-recipe). No code required. **Integrating a check?** [Try the demo](#try-the-complete-repair-loop) or [run it on your project](#check-your-own-project).

## Try the complete repair loop

Use Node.js 24 or later. Both demos run without a model account or API key.
Expand Down Expand Up @@ -69,11 +73,18 @@ An assertion failure in the test body produces evidence. Collection and setup fa

To run the checker from other projects, run `npm link` in this checkout. Then run `tfl-check --help` from your project. For a Python test, use `tfl-check --runner pytest --python /path/to/venv/bin/python tests/test_regression.py`. The link also installs the optional proxy command `test-feedback-loop`.

## Share an independent check
## Share an early-check recipe

A recipe tells others **what can go wrong, when it can first be detected, and what counts as pass or fail**. The human-readable version is enough to contribute. If you have a repeatable automated version, add the runner, command, and what should be reported as inconclusive.

| Layer | Example: a saved note disappears |
| --- | --- |
| Behavior, no code needed | Create a note, save it, then reload. The saved text must still be there. Editing an existing note must still work. |
| First machine-checkable point | When the storage or API boundary exists, write a note and read it back using a fresh client. Assert the exact saved value. Add a separate UI check when the screen exists. If the runner cannot reach storage, report inconclusive, not a failed behavior check. |

Have a useful pytest, Vitest, TypeScript, Cargo, schema, or other check? Open a [check proposal](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=check-adapter.yml) with a small failing example, a passing recheck, and a case where the runner could not determine the result. Proposals are welcome before implementation. If you have a working adapter, open a pull request with those examples as tests. [Contribution guide](CONTRIBUTING.md)
[Share a recipe (no code required)](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=early-check-recipe.yml) · [Propose a runner adapter](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=check-adapter.yml) · [Contribution guide](CONTRIBUTING.md)

Bundled checkers support Node.js tests and pytest.
Use existing tools first. Node.js and pytest checkers are bundled; a recipe can use another existing tool without adding a new adapter.

## Optional effort switching

Expand Down
21 changes: 17 additions & 4 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
# Test Feedback Loop

**用独立检查缩短修复时间,让较小的模型也能用较少的 token 完成实现。**
**用 Low 实现和修复,在问题还容易改的时候发现它。**

在计划和设计确定后,把能够稳定复现的失败和预期行为交给负责实现的代理。缩短 PDCA 中的检查环节,让代理专注于一处具体修复。修复后,独立复查原来的失败用例和正常对照用例。
让编码代理用较低推理强度(Low)完成小步实现和修复。每走一步,就独立运行最早能判定行为是否正确的检查。把检查证据交给代理,而不是让它“再想想”。如果检查无法得出结论,就先弄清原因,不要连续猜测和修改。

例如,应用提示笔记已保存,但刷新后笔记消失。无需等整个界面做完:存储层或 API 一就绪,就能检查写入后是否读回原值。保留一条编辑旧笔记的正常用例,等界面就绪后再补上只有浏览器才能验证的部分。

[English](README.md) · [日本語](README.ja.md) · [Agent Skill](skills/test-feedback-loop/SKILL.md)

**用 AI 做应用但不写测试?** [分享现象和检查方法](#分享尽早发现问题的检查方法),无需代码。**要接入自动化?** [试跑演示](#试跑一次完整的修复流程)。

## 试跑一次完整的修复流程

使用 Node.js 24 或更高版本。以下两个演示无需模型账号或 API 密钥。
Expand Down Expand Up @@ -69,9 +73,18 @@ npm run check:pytest -- --cwd /path/to/project --python /path/to/venv/bin/python

要从其他项目直接运行,请先在本仓库的工作目录中运行 `npm link`,再到目标项目运行 `tfl-check --help`。检查 Python 测试时,可运行 `tfl-check --runner pytest --python /path/to/venv/bin/python tests/test_regression.py`。可选代理命令为 `test-feedback-loop`。

## 分享独立检查
## 分享尽早发现问题的检查方法

我们收集的是实例:**哪里会出错、最早何时能发现、什么结果算通过**。只投稿人工检查步骤也可以。如果已有自动化版本,欢迎附上运行命令,以及哪些情况应该判为无法确定。

| 层次 | 例子:保存的笔记消失 |
| --- | --- |
| 人能检查的行为 | 保存新笔记并刷新,确认原文仍在;同时确认编辑已有笔记仍然有效。 |
| 最早能自动检查的地方 | 存储层或 API 就绪后,用另一个客户端读回刚写入的笔记并检查内容是否一致。无法连接存储时判为无法确定,而不是业务行为失败。 |

[分享检查方法(无需代码)](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=early-check-recipe.yml) · [建议新的检查适配器](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=check-adapter.yml) · [贡献指南](CONTRIBUTING.md)

欢迎分享实用的 pytest、Vitest、TypeScript、Cargo 或模式校验等检查。提交提案时,请附上一个最小失败示例、一个修复后能通过的示例,以及一个无法确定结果的示例。已有可用适配器时,也可把这些示例写成测试后提交 PR。当前内置 Node.js 和 pytest 检查。[提交检查提案](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=check-adapter.yml) · [贡献指南(英文)](CONTRIBUTING.md)
优先沿用已有工具。仓库内置 Node.js 和 pytest 检查器,但分享检查方法不要求新增适配器。

## 可选的推理强度切换

Expand Down