fix(retain): guard saturated fact extraction output

[kamilfurtak]

Summary

Fact extraction may enter a degenerate or excessively dense structured-output response where the facts array keeps growing until the output token limit. Introduce a saturation boundary that never silently accepts a capped response: reaching the boundary raises OutputTooLongError and delegates completeness to the existing recursive split/retry path.

The boundary is a split threshold, not an extraction or storage limit. A legitimate chunk with more than 16 facts is split and all sub-chunk facts are still returned.

Reproduction / impact

In Hindsight 0.9.1, the same small retain chunk can sometimes complete normally and sometimes produce a degenerate/looping response that keeps emitting facts until the output ceiling. Without a deterministic saturation signal, the extraction path has no reliable point at which to split the input.

This PR adds:

• maxItems=16 to the retain facts array when the configured backend supports that JSON Schema keyword;
• a runtime len(facts) >= 16 guard that raises OutputTooLongError instead of treating the boundary response as complete;
• delegation to the existing recursive split/retry path, which preserves facts by extracting the smaller sub-chunks.

Backend capability behavior

For llm_supports_max_items=True, the schema-level boundary prevents capable structured-output backends from accepting more than the split threshold. For llm_supports_max_items=False, the wire schema remains uncapped because some providers reject maxItems; the runtime guard is authoritative when a complete response contains 16 or more facts. It cannot interrupt generation in that case, so a provider that loops until finish_reason=length still relies on the existing output-too-long handling to trigger the split.

Scope

• Hindsight version: 0.9.1
• 16 is a saturation/circuit-breaker threshold, not a legal maximum fact count or a truncation policy.
• Separate from the nested [[{turn}, ...]] splitter fix in fix(retain): preserve nested conversation chunks during output retry #3652.
• Separate from the strict-schema flag fix in fix(retain): honor retain strict-schema flag #3653.
• No changes to saturation behavior in unrelated extraction paths.

Validation

• tests/test_fact_extraction_retry.py tests/test_fact_extraction_entities_schema.py: 40 passed
• Boundary coverage: 15 facts succeeds; 16 facts raises OutputTooLongError.
• End-to-end retry coverage: 16 facts → recursive split → two successful sub-chunk responses → both facts returned.
• Ruff check: passed
• Ruff format check: passed

The repository hook was not used because this checkout does not have uv installed; the equivalent targeted Ruff checks were run directly.

---

Depends on #3652

This PR raises how often OutputTooLongError is signalled, which routes more work
into the existing split path. #3652 is what makes that path succeed for
conversation payloads that arrive wrapped in an extra array. Merging this without
#3652 would increase the number of splits while leaving those splits unable to
make progress.

On the bound

The bound is derived from retain_max_completion_tokens rather than fixed, so a
deployment with a large ceiling is not pushed into constant splitting:

budget 4096  -> 24 facts
budget 65536 -> 385 facts
unset        -> 16 facts (floor)

An earlier revision hard-coded 16, which was tuned against a 4096-token ceiling on
a small local model. At that ceiling a fact object costs roughly 85 output tokens,
so a bound of 48 would have landed at the same point as the token limit and done
nothing. Halving the budget keeps the item bound biting first.

Field data behind the floor, from 168 single-chunk extractions on one deployment:
median 2 facts, p95 10, max 29.

[kamilfurtak]

Superseded by #3657, which combines this with the related recovery fix and derives the saturation bound from the configured output budget instead of hard-coding it. Depends on #3656.