Release 0.7.0 — quality improvements: security, correctness, performance, structure#5
Merged
Merged
Conversation
…d CSV reading - honor extended thinking in direct extraction and suggest_schema (was codegen-only) - surface codegen records that fail validation instead of dropping them silently - run chunked single-sheet extraction concurrently with a bounded semaphore - cache the static system prompt only, not the variable user message - narrow extraction error handling to the LLM call so our own bugs aren't mislabeled - wrap CSV decode failures as ReaderError and thread a configurable csv_encoding - fast-fail empty sheets and drop the silent confidence default Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The .serena/ and .claude/ directories are machine-local agent tooling, and docs/improvement-plan.md carries a security PoC that must never be published — ignore all three and stop tracking the previously-committed Serena files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hardening) LLM-generated codegen scripts are untrusted code and the subprocess AST scan was never a real boundary. Codegen now runs in an OS-level Docker sandbox by default and fails closed (CodegenSecurityError) when none is available; the non-isolating subprocess backend is now an explicit trusted/dev-only opt-in. Also hardens the Docker backend (non-root, CapDrop ALL, read-only rootfs + tmpfs, seccomp/gVisor knobs) and adds per-user HMAC integrity + 0700/0600 perms to the script cache. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…he budget split() used a sampled total with proportional row splitting, so chunks could exceed token_budget when token density varied by position or rows differed in size; it now greedy bin-packs rows by measured cost with the header reserved. Also harden token counting to encode_ordinary so cell content like "<|endoftext|>" no longer raises, and drop the full-range set allocation in find_non_empty_rows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Structural, behavior-preserving refactor (A1/A2/A3 from the internal plan). Public API, method signatures, and *_sync wrappers are unchanged; the full unit suite stays green (434 passed, 15 skipped). A2/A3 — reader dispatch: - Add reader/dispatch.py: READER_REGISTRY (extension -> reader adapter), get_source_ext, read_workbook. Adding a new format is now a one-place change. - Extractor._load_workbook and mcp_server._load_sheet share read_workbook, removing the duplicated .csv/else branch and the MCP server's private-member access (Extractor._get_source_ext via reportPrivateUsage). - Replace the dead ExcelReader protocol (matched no implementation) with a real contract: ReaderOptions (options object) + WorkbookReader. Concrete reader .read() signatures are unchanged (they are directly tested). A1 — Extractor decomposition (conservative, mechanical extraction): - extraction/concurrency.py: run_concurrent() folds the near-identical bounded-concurrency + progress closures from extract_batch/extract_workbook. - extraction/report.py: build_extraction_report() extracts the provenance/ confidence assembly out of extract(). - Extractor._resolve_auto_mode() dedupes AUTO routing across the configured and streaming paths. The test-locked private surface (_load_workbook, _engine, _chunk_splitter, _run_codegen, _get_codegen, _run_sheet_extraction, _get_source_ext) stays on Extractor. extractor.py: 1,121 -> 1,020 lines. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cstring - Extractor.extract(source, schema) (legacy path) now calls _require_non_empty_sheet() before extraction, matching the config path. A 0-row sheet raises ReaderError(READER_PARSE_FAILED) instead of wasting an LLM call and failing confusingly later (B3). Adds a regression test. - ExecutionBackend docstring: SubprocessBackend is no longer the default after the S1 sandbox hardening; Docker is the default for untrusted codegen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving: all orchestration moves to a new ExtractionPipeline and Extractor delegates with its public API unchanged. ExtractionResult moves to its own module (re-exported) to break the facade-pipeline import cycle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump the version (pyproject, __init__.__version__, lockfile) and finalize the CHANGELOG for the quality-improvements line: codegen sandboxing (S1-S3), correctness fixes (C1-C4), performance (P1/P2/P4), and the Extractor → ExtractionPipeline refactor (A1-A3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch lands the
quality-improvementsline and bumps the version to 0.7.0. Every item was implemented in a focused session and independently verified (code-reviewer / security-auditor) with the full unit suite, ruff, and pyright passing.Security (S1–S3)
CodegenSecurityError) when none is available — no more silent execution in a non-isolating subprocess.DockerBackendhardened: non-root user,CapDrop=["ALL"], read-only rootfs + tmpfs/tmp, network disabled, no-new-privileges, optional gVisor/seccomp.SubprocessBackenddemoted to explicit trusted/dev-only opt-in (python -I+ reduced builtins, process-group kill, fail-closed rlimits); clearly documented as not a security boundary.Correctness
thinking=Trueis now honored in direct extraction andsuggest_schema(was codegen-only). (C1)ReaderErrorand honor a configurablecsv_encoding. (C3)Performance
find_empty_rowsdrops a full-range set allocation. (B1)Structure
Extractordecomposed into a thin facade overExtractionPipeline; report assembly and concurrency moved intoextraction/. (A1)READER_REGISTRY, shared by the extractor and MCP server; deadExcelReaderprotocol replaced byWorkbookReader+ReaderOptions. (A2, A3)Codegen with the default config now requires a sandbox. Without
xlstruct[docker]installed, codegen fails closed withCodegenSecurityError. To restore the previous unsandboxed behavior in a trusted environment:ExtractorConfig(codegen_sandbox="subprocess"), orexecution_backend=SubprocessBackend(trusted=True)Direct (non-codegen) extraction is unaffected. Existing codegen caches are invalidated once (signature widened).
Verification
Each session was reviewed by a fresh reviewer agent. Unit suite: 444 passed, 15 skipped; ruff and pyright clean. (3 GCS integration tests require a local fake-gcs-server and are environment-only failures.)
🤖 Generated with Claude Code