Support non-HuggingFace text models / decouple the tokenizer seam

[stanlrt]

Context

#340 shipped the text input modality + text-classification, but text models are HuggingFace only: model.tokenizer loads via AutoTokenizer and model.source via AutoModelForSequenceClassification, both hardcoded in load_hf_text_backend (src/raitap/models/torch_backend.py) and the tokenise step in Data._load_data.

That was a deliberate MVP scope line, not a wall: HF was the only ecosystem that dropped into the existing seam with zero new backend work, because

• HF AutoModelForSequenceClassification is a torch nn.Module, so it runs through TorchBackend + LayerIntegratedGradients (Captum token attribution needs autograd), and
• from_pretrained gives a vocab-matched tokenizer+model pair under one id, so no separate tokenizer↔model wiring.

The input-source side is already pluggable (the data/inputs parser registry); only the model+tokenizer side is HF-bound.

Problem

No non-HuggingFace text path. Unsupported today: spaCy, tiktoken, sentencepiece, a raw torch classifier with an external tokenizer, or ONNX-exported text models.

Options (open — need a decision, not yet specced)

A. Decouple the tokenizer seam. Introduce a pluggable tokeniser interface (mirroring the input-parser registry) so tokenisation is not hardcoded to AutoTokenizer. A raw torch model + an external tokenizer (sentencepiece/tiktoken) could then run. Counterpoint: tokenizer↔model vocab coupling is easy to get wrong once they are configured separately; needs a validation story.

B. ONNX / non-torch text models. These break the autograd contract, so LayerIntegratedGradients (gradient-based token attribution) cannot run; they would be limited to perturbation-based explainers (Occlusion, FeatureAblation, SHAP KernelExplainer). Cross-ref the backend-agnostic explanation work and the tree/onnx capability gating. Counterpoint: a text config that silently loses token-level gradient attribution may be a worse UX than not supporting it.

C. Keep HF-only. If HF covers the real use cases, close as won't-do and just keep the docs honest (text models = HF only).

Relationships

• Builds on Add text input modality + text-classification use case (end-to-end) #340 (text input modality, HF text-classification).
• The tokeniser-decoupling (Option A) is also relevant to Add seq2seq / text-generation task family #285 (seq2seq), which needs its own input assembly.
• Autograd/Captum coupling: token attribution via LayerIntegratedGradients requires a torch model with an embedding layer; non-torch backends inherit the perturbation-only limitation.

Acceptance (if pursued)

• A non-HuggingFace text model runs end to end (load, tokenise, predict, attribution appropriate to its backend, report).
• The tokenizer↔model vocab-match failure mode is caught with a clear error.
• Docs updated; the HF-only note in docs/modules/model/configuration.md / docs/modules/data/own-vs-built-in.md removed or narrowed.