[codex] simplify provider-neutral negative knowledge module - #1
[codex] simplify provider-neutral negative knowledge module#1hch-wang wants to merge 11 commits into
Conversation
…rator/run_pipeline - Drop version cruft (v3/v3A/v3Aprime/v1/v2) and 'Class A/B' naming. - Flatten layout to top-level analyze_results.py + nk_curator.py + run_pipeline.py + scripts/ + curator_prompts/ + prompts/ + tasks/ + logs/, mirroring section3_reproduce/. - analyze_results.py verifies 20/20 paper §4 claims from logs/. - nk_curator.py: NKCurator class with produce_per_round + produce_deep methods + schema validation, mirroring section3's nk_curator.py. - run_pipeline.py: Mode B end-to-end on one Stage 2 cell with --use-saved-trace fallback for verification without API. - scripts/dispatch_subagent.py: copied from section3 + added Bash tool (Stage 2 cells execute candidate.py inline). - logs/ archive: stage1/ (7 BKdV-S programs, full per-round artifacts), stage2/ (12 cells), nk_records/ (28 JSON), banks/ (58-entry final + 30-entry legacy pilot input), verified_results/. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
只保留三个 reviewer-facing reproduce 包,把旧的探索性工作目录从仓库树中删除,并将 appendix 复现包整理进自包含目录。同步修正 §4 生成脚本的路径解析,避免复现入口继续依赖已移出的 section4 工作目录。
Removed citation section for the Negative Knowledge paper.
The Table 1 column in the paper reports memory-object size in cl100k_base tokens; analyze_results.py until now only computed bytes. count_tokens.py reads logs/nk_records/ (depth-1 and depth-3 records) and logs/self_debug_inputs/ (the round-1 candidate.py + exec.log + eval.log the self-debug condition shows to the next attempt) and reproduces the three median token counts plus the savings percentages. Exits 0 when all three medians match the paper exactly. Also bundles the 23 self-debug input directories needed by the script (task_003 excluded — its 4.6 MB log-bomb exec.log is the same outlier analyze_results.py already excludes). Adds tiktoken to requirements.txt and a section in section3_reproduce/README.md.
….json Reviewers shouldn't need to install anything to verify Table 1. Mirror the pattern already used for bytes (logs/b2_covering_bytes.json): ship the per-task token counts as logs/memory_tokens.json and have count_tokens.py read it with stdlib only, verifying that the three medians match the paper (296 / 1,109 / 795 with -73.3% / -28.3% savings). tiktoken stays optional, only needed for 'count_tokens.py --regenerate' which re-derives the JSON from the raw self-debug inputs and NK records. Drop tiktoken from requirements.txt.
…a subdir
The published repo is now organised around the product, not the paper:
negative_knowledge/ installable module — the curator, the bounded/typed
NK schema + validate_nk, the constrained Read/Write
agent runtime, and the prompt templates. Importing
the package and validating records need no API key;
anthropic is imported lazily, only when curating.
examples/ quickstart.py + a self-contained sample failure;
runs offline (validates a bundled record) or live.
pyproject.toml pip install -e . ; console script 'negative-knowledge'.
reproduction/ the former section*_reproduce/ packages, moved here
unchanged (self-contained, frozen) under section3/
section4/ appendix/, with a reproduction/README.md.
Top-level README rewritten module-first. No reproduction logic or numbers
changed; analyze_results.py (31/31, 20/20, 54/54) and count_tokens.py still
pass from their new locations.
|
Notes after reviewing this branch and running the suites locally. The core direction is good: collapsing to a single dependency-free module and making curation provider-neutral (the caller supplies 1. Branches before the release polish → conflicts with 2. The new
Net effect: only 11 of 67 bundled per-attempt NK records pass 3. Anonymizing the 4. Minor. Reconciled branch. I pushed
Locally verified: |
Clean-room take on #1: keep the single-file, zero-dependency module and its tests/CI, rebased on main so the release polish (LICENSE, README badges, arXiv links, metadata) survives -- and drop the changes that regressed reproducibility or usability. Module (negative_knowledge.py): - single-file, zero-dependency curate/validate/append/load surface - relax validate length caps (200/300 -> 600/1200) so real curator outputs are no longer rejected (observed max was 404 chars) - accept extra top-level fields as extensions (round indices, cross-round notes); keep the typed 6-field core + closed failure.* vocabulary - widen LAYERS to the taxonomy actually used in the experiments (+hypothesis_failure, +measurement_failure) - all 67 bundled per-attempt NK records now validate (was 11) Docs / packaging: - merge READMEs: keep badges, arXiv link, overview figure, schema docs and BibTeX; document the new curate/validate/append/load API - pyproject: single-module (py-modules), zero deps, dynamic version, keep Paper (arXiv) URL + Science/Research classifiers - move requirements.txt -> reproduction/requirements.txt (core is dep-free) Reproduction: - add the provider-neutral NK_AGENT_COMMAND bridge (agent_command.py) - fix shlex parsing so it also works on Windows (posix=os.name!='nt') - do NOT anonymize artifacts: real model provenance (sonnet/haiku) is kept for scientific transparency Validation: unittest 8/8; offline quickstart OK; section4 20/20; appendix 54/54 (section3 is 31/31 on Linux/LF; the 27/31 seen on Windows is a pre-existing CRLF byte-count artifact in the verifier, unrelated).
What changed
negative_knowledge.pymodulecurate,validate,append, andloadNK_AGENT_COMMANDJSON protocolprimary,secondary, orprovider-redacted, including archived filenamesWhy
The reusable surface should be small enough to copy directly into another project and should not require a particular model SDK. The previous implementation coupled curation, file tools, prompts, CLI behavior, and one provider into a larger package.
Impact
This is a breaking API simplification: callers should replace
NKCuratorusage withcurate(...)and useappend/loadfor JSONL memory. Current-tree model provenance is anonymized; Git history is unchanged.Validation
python3 -m unittest discover -s tests— 8/8 passedpip checkpassedgit diff --checkpassed