Support for “real” batch processing

[talbaumel]

Problem

TransformerDetector.predict_prompt_batch currently loops over examples, so N inputs perform N tokenizations and N model forward passes. Both transformer and LLM batch paths also pair inputs with zip, which silently drops trailing values when input lengths differ.

Scope

Implement true batched inference for the transformer detector:

1. Validate len(prompts) == len(answers) and raise ValueError on mismatch; never truncate.
2. Tokenize a padded batch and run one model forward pass per configured batch.
3. Preserve input order and remove padding and prompt tokens correctly per sample.
4. Support both tokens and spans, including min_confidence behavior and taxonomy typing where configured.
5. Add a configurable batch_size, or clearly document whole-list batching and its memory limit.
6. Keep the LLM detector's concurrent request path separate; it needs the same length validation but is not expected to share a model forward pass.

Acceptance

• Batch output matches the corresponding single-example method for every sample within a documented numerical tolerance.
• Tests cover uneven sequence lengths, batch size 1 and greater than 1, output order, tokens, spans, confidence filtering, empty input, and mismatched lengths.
• A spy or stub model verifies one forward call per transformer batch without downloading a model.
• python -m pytest passes.

Non-goals

• Cross-request server batching.
• Automatic OOM recovery in the first implementation.
• Changing detector output schemas.

[yochail]

I have been trying to implement batch inference directly with the torch model, but surprisingly I got Significantly different results for the batch inference when comparing output of the exact same input (to the padding level) when running it in a batch or 1 by 1 example.
The results for the 1 example in the batch are identical, but the results for the second one are different.

I wonder if this is a known issue of the ModernBert model.

outputs_ids_debug_1.txt
outputs_ids_debug_2.txt
inputs_ids_debug_1.txt
inputs_ids_debug_2.txt

[adaamko]

Still on the roadmap (now tracked in the v0.3 milestone), and you're right about the current state: predict_prompt_batch in lettucedetect/detectors/transformer.py just loops over predict_prompt. The plan is padded-batch tokenization plus a single forward pass, reusing the existing span-decoding logic per row — the decode path already works off per-token probabilities, so it's mostly a collation change.

@yochail — what you hit is a known ModernBERT quirk rather than something in this repo: with padded batches, the attention path (FA2 unpadding vs SDPA) doesn't produce bit-identical logits to single-sequence inference, and a few transformers releases had actual padded-batch bugs for ModernBERT. Two things to try: force attn_implementation="eager" (or upgrade transformers) and compare again, and check whether the resulting spans (not raw logits) actually differ — small numeric drift near the decision boundary is expected. If you still see materially different spans on a current transformers with eager attention, please share the repro; that would block shipping batched inference and we'd want to pin it down first.