Skip to content

fix(retain): fail when oversized chunk cannot split - #3655

Closed
kamilfurtak wants to merge 3 commits into
vectorize-io:mainfrom
kamilfurtak:agent/retain-fail-on-unsplittable
Closed

fix(retain): fail when oversized chunk cannot split#3655
kamilfurtak wants to merge 3 commits into
vectorize-io:mainfrom
kamilfurtak:agent/retain-fail-on-unsplittable

Conversation

@kamilfurtak

@kamilfurtak kamilfurtak commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

In Hindsight 0.9.1, the retain output-retry splitter logs a warning and returns an empty result when an overlong sub-chunk cannot be split any further. The caller then treats that result as successful extraction, which can allow an operation to report completed while the sub-chunk's data was silently lost.

This draft PR changes only that terminal branch:

  • keep the existing minimum-size/no-progress decision in _split_chunk_for_output_retry;
  • raise a descriptive RuntimeError when an OutputTooLongError cannot be followed by a valid split;
  • let the existing extract_facts_from_text failure propagation mark the retain as failed/retryable instead of committing a partial result.

Reproduction / impact

  1. A retain chunk raises OutputTooLongError.
  2. _split_chunk_for_output_retry returns None because the chunk is too small or cannot be reduced.
  3. Before this change, _extract_facts_with_auto_split returned ([], TokenUsage()) and logged that it was dropping the sub-chunk.
  4. The extraction could therefore continue with missing data and eventually report success.

The regression test uses a one-character chunk and a mocked OutputTooLongError; it now verifies that the helper raises and makes exactly one extraction attempt. Splittable chunks and the standard successful extraction path are unchanged.

Scope

Validation

  • tests/test_fact_extraction_retry.py: 25 passed
  • Ruff check: passed
  • Ruff format check: passed

The repository hook was not used because this checkout does not have uv installed; the equivalent targeted Ruff checks were run directly.


Depends on #3652

Without #3652, conversation payloads wrapped in an extra array cannot be split at
all, so this change would turn a recoverable case into a hard failure. #3652 makes
those splits succeed; this PR then only fails when a chunk genuinely cannot be
reduced further.

Partial results are preserved

The recursive split previously gathered sub-chunk results without
return_exceptions, so one unsplittable half discarded facts the other half had
already extracted. That is the same data-loss shape this PR is meant to remove, so
sub-chunk results are now collected tolerantly:

  • some halves fail, others succeed -> keep the extracted facts, log the partial
    failure at error level with counts
  • every half fails -> re-raise, so the operation cannot report success

The remaining trade-off is deliberate: a partially-failed chunk still completes,
with the loss visible in logs rather than silent at the operation level.

@kamilfurtak
kamilfurtak force-pushed the agent/retain-fail-on-unsplittable branch from 1b310d6 to 483f57f Compare August 19, 2026 20:50
The recursive split gathered sub-chunk results without return_exceptions, so a
single unsplittable half discarded the facts the other half had already
extracted, and the failure propagated to the whole document.

Losing good data is exactly the failure mode this path exists to prevent. Collect
sub-chunk results tolerantly: salvage what succeeded, log the partial failure at
error level with counts, and only re-raise when the split produced nothing at all.
@kamilfurtak
kamilfurtak force-pushed the agent/retain-fail-on-unsplittable branch from 483f57f to c5a27cd Compare August 19, 2026 20:54
@kamilfurtak

Copy link
Copy Markdown
Contributor Author

Superseded by #3657, which combines this with the related recovery fix and derives the saturation bound from the configured output budget instead of hard-coding it. Depends on #3656.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant