Skip to content

[codex] simplify provider-neutral negative knowledge module - #1

Draft
hch-wang wants to merge 11 commits into
mainfrom
codex/simplify-provider-neutral-module
Draft

[codex] simplify provider-neutral negative knowledge module#1
hch-wang wants to merge 11 commits into
mainfrom
codex/simplify-provider-neutral-module

Conversation

@hch-wang

Copy link
Copy Markdown
Owner

What changed

  • replace the multi-file package and provider-specific runtime with a single zero-dependency negative_knowledge.py module
  • expose four small functions: curate, validate, append, and load
  • add strict record validation, evidence truncation, JSONL persistence, a concise README, an offline example, MIT licensing, tests, and CI
  • route optional fresh reproduction runs through the provider-neutral NK_AGENT_COMMAND JSON protocol
  • anonymize provider/model labels in the current tree as primary, secondary, or provider-redacted, including archived filenames

Why

The reusable surface should be small enough to copy directly into another project and should not require a particular model SDK. The previous implementation coupled curation, file tools, prompts, CLI behavior, and one provider into a larger package.

Impact

This is a breaking API simplification: callers should replace NKCurator usage with curate(...) and use append / load for JSONL memory. Current-tree model provenance is anonymized; Git history is unchanged.

Validation

  • python3 -m unittest discover -s tests — 8/8 passed
  • Python 3.9 syntax parse passed
  • offline quickstart passed
  • clean wheel build and no-dependency install passed; pip check passed
  • §3 claim verification: 31/31
  • §4 claim verification: 20/20
  • appendix claim verification: 54/54
  • Table 1 token medians reproduced exactly
  • full working-tree search found no provider/model brand terms
  • git diff --check passed

dietcoke-mac7 and others added 11 commits May 13, 2026 18:14
…rator/run_pipeline

- Drop version cruft (v3/v3A/v3Aprime/v1/v2) and 'Class A/B' naming.
- Flatten layout to top-level analyze_results.py + nk_curator.py +
  run_pipeline.py + scripts/ + curator_prompts/ + prompts/ + tasks/ +
  logs/, mirroring section3_reproduce/.
- analyze_results.py verifies 20/20 paper §4 claims from logs/.
- nk_curator.py: NKCurator class with produce_per_round + produce_deep
  methods + schema validation, mirroring section3's nk_curator.py.
- run_pipeline.py: Mode B end-to-end on one Stage 2 cell with
  --use-saved-trace fallback for verification without API.
- scripts/dispatch_subagent.py: copied from section3 + added Bash tool
  (Stage 2 cells execute candidate.py inline).
- logs/ archive: stage1/ (7 BKdV-S programs, full per-round artifacts),
  stage2/ (12 cells), nk_records/ (28 JSON), banks/ (58-entry final +
  30-entry legacy pilot input), verified_results/.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
只保留三个 reviewer-facing reproduce 包,把旧的探索性工作目录从仓库树中删除,并将 appendix 复现包整理进自包含目录。同步修正 §4 生成脚本的路径解析,避免复现入口继续依赖已移出的 section4 工作目录。
Removed citation section for the Negative Knowledge paper.
The Table 1 column in the paper reports memory-object size in
cl100k_base tokens; analyze_results.py until now only computed bytes.
count_tokens.py reads logs/nk_records/ (depth-1 and depth-3 records)
and logs/self_debug_inputs/ (the round-1 candidate.py + exec.log +
eval.log the self-debug condition shows to the next attempt) and
reproduces the three median token counts plus the savings percentages.
Exits 0 when all three medians match the paper exactly.

Also bundles the 23 self-debug input directories needed by the
script (task_003 excluded — its 4.6 MB log-bomb exec.log is the
same outlier analyze_results.py already excludes). Adds tiktoken
to requirements.txt and a section in section3_reproduce/README.md.
….json

Reviewers shouldn't need to install anything to verify Table 1. Mirror
the pattern already used for bytes (logs/b2_covering_bytes.json):
ship the per-task token counts as logs/memory_tokens.json and have
count_tokens.py read it with stdlib only, verifying that the three
medians match the paper (296 / 1,109 / 795 with -73.3% / -28.3%
savings). tiktoken stays optional, only needed for
'count_tokens.py --regenerate' which re-derives the JSON from the
raw self-debug inputs and NK records. Drop tiktoken from
requirements.txt.
…a subdir

The published repo is now organised around the product, not the paper:

  negative_knowledge/   installable module — the curator, the bounded/typed
                        NK schema + validate_nk, the constrained Read/Write
                        agent runtime, and the prompt templates. Importing
                        the package and validating records need no API key;
                        anthropic is imported lazily, only when curating.
  examples/             quickstart.py + a self-contained sample failure;
                        runs offline (validates a bundled record) or live.
  pyproject.toml        pip install -e . ; console script 'negative-knowledge'.
  reproduction/         the former section*_reproduce/ packages, moved here
                        unchanged (self-contained, frozen) under section3/
                        section4/ appendix/, with a reproduction/README.md.

Top-level README rewritten module-first. No reproduction logic or numbers
changed; analyze_results.py (31/31, 20/20, 54/54) and count_tokens.py still
pass from their new locations.
@hch-wang

Copy link
Copy Markdown
Owner Author

Notes after reviewing this branch and running the suites locally.

The core direction is good: collapsing to a single dependency-free module and making curation provider-neutral (the caller supplies backend) is a clean design, and the tests / CI / richer CITATION.cff are welcome. A few things I'd want resolved before merging:

1. Branches before the release polish → conflicts with main.
This is based on f915d96, before main gained its LICENSE, README badges, arXiv links and packaging metadata (f8fe97a). It currently shows as conflicting, and the rewritten README.md drops the badges, the arXiv link, the overview figure, the schema section and the BibTeX — regressing the release-ready front matter instead of building on it.

2. The new validate() rejects the repo's own records.
It is stricter than the data this project actually produced:

  • length caps (200 / 300) sit below the observed maxima in reproduction/ (attempted_route 332, observation 300, rationale 404, recommended_alternative 402);
  • real records carry extension fields (round, relationship_to_round1, is_trivial, …) that are flagged as unknown field;
  • LAYERS is missing values that appear in the artifacts (hypothesis_failure ×14, measurement_failure ×3).

Net effect: only 11 of 67 bundled per-attempt NK records pass validate(), so load() / curate() would raise on legitimate records. Deep (cross-round) records aren't representable at all under the six-field schema.

3. Anonymizing the reproduction/ artifacts.
The sonnet → primary / haiku → secondary scrub (most of the ~400-file diff) lowers transparency for a paper-reproduction repo and makes it inconsistent with the paper, while not actually removing the names (git history is unchanged, as the description notes). Better to keep the module provider-neutral but leave the reproduction artifacts as the honest record of what was run.

4. Minor. reproduction/agent_command.py calls shlex.split() in POSIX mode, which mangles Windows paths (WinError 2); 1 of the 8 tests fails on Windows as a result.


Reconciled branch. I pushed cleanup/provider-neutral-module that keeps the good parts of this PR and addresses the above:

  • single-file zero-dependency module + tests + CI + the richer CITATION, rebased on main (no conflicts);
  • merged README — badges / arXiv link / overview figure / BibTeX kept, and the new curate / validate / append / load API documented;
  • relaxed length caps (→ 600 / 1200), extension fields allowed, LAYERS widened to the taxonomy actually used → all 67 bundled records validate;
  • shlex fixed for Windows → 8 / 8 tests pass;
  • reproduction artifacts left un-anonymized.

Locally verified: unittest 8/8, offline quickstart OK, §4 20/20, appendix 54/54.

hch-wang added a commit that referenced this pull request Jul 5, 2026
Clean-room take on #1: keep the single-file, zero-dependency module and
its tests/CI, rebased on main so the release polish (LICENSE, README
badges, arXiv links, metadata) survives -- and drop the changes that
regressed reproducibility or usability.

Module (negative_knowledge.py):
- single-file, zero-dependency curate/validate/append/load surface
- relax validate length caps (200/300 -> 600/1200) so real curator
  outputs are no longer rejected (observed max was 404 chars)
- accept extra top-level fields as extensions (round indices, cross-round
  notes); keep the typed 6-field core + closed failure.* vocabulary
- widen LAYERS to the taxonomy actually used in the experiments
  (+hypothesis_failure, +measurement_failure)
- all 67 bundled per-attempt NK records now validate (was 11)

Docs / packaging:
- merge READMEs: keep badges, arXiv link, overview figure, schema docs and
  BibTeX; document the new curate/validate/append/load API
- pyproject: single-module (py-modules), zero deps, dynamic version,
  keep Paper (arXiv) URL + Science/Research classifiers
- move requirements.txt -> reproduction/requirements.txt (core is dep-free)

Reproduction:
- add the provider-neutral NK_AGENT_COMMAND bridge (agent_command.py)
- fix shlex parsing so it also works on Windows (posix=os.name!='nt')
- do NOT anonymize artifacts: real model provenance (sonnet/haiku) is kept
  for scientific transparency

Validation: unittest 8/8; offline quickstart OK; section4 20/20;
appendix 54/54 (section3 is 31/31 on Linux/LF; the 27/31 seen on Windows
is a pre-existing CRLF byte-count artifact in the verifier, unrelated).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants