Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
30 changes: 30 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: CI

on:
push:
pull_request:

jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9", "3.12", "3.14"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- run: python -m pip install .
- run: python -m unittest discover -s tests
- run: python examples/quickstart.py

package:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: python -m pip install build
- run: python -m build
10 changes: 3 additions & 7 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ __pycache__/
.Python
*.egg-info/
.pytest_cache/
.coverage
build/
dist/

# Virtual env (do NOT commit; reproduce-time only)
.venv/
Expand All @@ -22,12 +25,5 @@ env/
*.swp
*.swo

# Local outputs that get regenerated
# (keep canonical artifacts but ignore replay scratch)
# (Actually we WANT to keep pred_results/*.npy for inspection; comment if you want exclude)

# Symlinks
.venv

# Logs that exceed reasonable size
*.gigantic.log
22 changes: 22 additions & 0 deletions CITATION.cff
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
cff-version: 1.2.0
message: "If you use this work, please cite the paper below."
title: "Negative Knowledge as Failure-aware Shared Memory for AutoResearch"
version: "0.2.0"
license: MIT
authors:
- family-names: Wang
given-names: Hanchun
affiliation: "Department of Applied Mathematics and Theoretical Physics, University of Cambridge"
email: hw660@cam.ac.uk
url: "https://github.com/hch-wang/Negative_Knowledge"
repository-code: "https://github.com/hch-wang/Negative_Knowledge"
preferred-citation:
type: conference-paper
title: "Negative Knowledge as Failure-aware Shared Memory for AutoResearch"
authors:
- family-names: Wang
given-names: Hanchun
affiliation: "Department of Applied Mathematics and Theoretical Physics, University of Cambridge"
collection-title: "ICML 2026 AI4Research Workshop on AI as a Tool for Mathematics, Computer Science, and Machine Learning"
year: 2026
url: "https://github.com/hch-wang/Negative_Knowledge"
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 Hanchun Wang

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
97 changes: 31 additions & 66 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,82 +1,47 @@
# Negative Knowledge

Code and reproducibility artifacts for the ICML 2026 AI4Research Workshop
paper *Negative Knowledge as Failure-aware shared memory for AutoResearch*.
A single-file Python module that turns failed attempts into reusable memory.
It has no dependencies and works with any model backend.

A *negative-knowledge record* (NK) is a structured, machine-readable
summary of one or more failed attempts on a task.
## Install

---

## The bounded failure schema

The NK record is a JSON object with a fixed shape. Every field draws
from a controlled vocabulary; no field is free text in a way that
would defeat downstream machine consumption. The base form has 6
required fields:

```
task_id, attempted_route, observation,
failure { layer, scope, degree, recommended_action, risk },
rationale, recommended_alternative
```

A **depth-N** extension adds three cross-round fields
(`rounds_summary`, `ruled_out_routes`, `synthesised_diagnosis`)
when the curator reads more than one failed round.

The schema is implemented in
[`section3_reproduce/nk_curator.py`](section3_reproduce/nk_curator.py)
(constants `SCHEMA_BASE_FIELDS`, `SCHEMA_DEEP_EXTRA`, `LAYERS`, …),
with `validate_nk(nk, depth)` returning a list of violations.

---

## Running the §3 experiment

### Verify mode (no API key, ~5 seconds)
Copy [`negative_knowledge.py`](negative_knowledge.py) into your project, or:

```bash
cd section3_reproduce
python analyze_results.py
python3 -m pip install git+https://github.com/hch-wang/Negative_Knowledge.git
```

Re-derives every §3 paper claim from the bundled archive in `logs/`.
Expected output: `31/31 claims match`. Full report at
`section3_reproduce/results/claim_report.md`.
## Use

### End-to-end mode (Anthropic API, ~$1, ~10 min)
```python
from negative_knowledge import append, curate, load

```bash
cd section3_reproduce
pip install anthropic
export ANTHROPIC_API_KEY=sk-ant-...
git clone https://github.com/OSU-NLP-Group/ScienceAgentBench
export SAB_BENCH=$(pwd)/ScienceAgentBench/benchmark
def backend(prompt: str) -> str:
return your_agent.generate_json(prompt)

python run_pipeline.py --task 072 --use-saved-trace
```
record = curate(
backend,
task_id="072",
task="Map Sub01 EEG signals to Sub03",
evidence={
"code": failed_code,
"error": stderr,
"reasoning": reasoning,
},
)

Re-runs the depth-3 deepNKR pipeline on task_072 (the §3 breakthrough
task): dispatches the canonical curator, builds a fresh Sonnet solver
sandbox, runs `candidate.py` + the task evaluator, prints the result.
A `score=1` reproduces the §3 task_072 PASS.

Full guide:
[`section3_reproduce/README.md`](section3_reproduce/README.md).
append("negative_knowledge.jsonl", record)
memory = load("negative_knowledge.jsonl")
```

---
That is the whole interface:

## Citation
- `curate(...)` creates and validates one record.
- `validate(record)` returns schema problems.
- `append(path, record)` stores one record as JSONL.
- `load(path)` reads the memory back.

```bibtex
@inproceedings{NegativeKnowledge2026,
title = {Negative Knowledge as Failure-aware shared memory for AutoResearch},
author = {...},
booktitle = {ICML 2026 AI4Research Workshop},
year = {2026},
}
```
See [`examples/quickstart.py`](examples/quickstart.py) for an offline example.
The paper artifacts remain under [`reproduction/`](reproduction/).

ScienceAgentBench: Chen et al., 2024
(https://github.com/OSU-NLP-Group/ScienceAgentBench).
MIT licensed.
Binary file added docs/overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
156 changes: 0 additions & 156 deletions early_bkdv_pilot/CASE_F_G_STUDIES.md

This file was deleted.

Loading
Loading