Replace the reranker with a multilingual cross-encoder

[felladrin]

Description

Replaces jina-reranker-v1-tiny-en with the int8 export of cross-encoder/mmarco-mMiniLMv2-L12-H384-v1.

Currently the reranker is an English model, and the docs say it "ranks non-English results (Portuguese, for example) well in practice". Measured, it doesn't: on 240 queries from mteb/MIRACLReranking across English, Spanish, French, German, Russian and Japanese, it scores 0.4636 nDCG@10 while the un-reranked SearXNG-style first-stage order scores 0.4963. On non-English content it was not adding ranking signal, and part of the mechanism is the tokenizer: an English WordPiece vocabulary spends 2.17x as many tokens on Russian and about 1.4x on Spanish, German and Japanese, so it paid more compute to see worse-fragmented subwords.

	nDCG@10 (240 MIRACL queries)	ms per 30 docs (10 cores / 2 threads)	Download
First-stage order, no reranking	0.4963	-	-
jina-reranker-v1-tiny-en fp32 (before)	0.4636	104 / 224	132MB
mmarco-mMiniLMv2-L12-H384-v1 int8 (after)	0.7992	137 / 202	119MB + 17MB tokenizer

Per language, nDCG@10 goes 0.774 => 0.743 (en), 0.498 => 0.842 (es), 0.351 => 0.777 (fr), 0.370 => 0.774 (de), 0.387 => 0.819 (ru), 0.402 => 0.840 (ja). English regresses slightly; every other language roughly doubles.

Why the int8 export

The fp32 export of the same model is 471MB and twice the latency for 0.7973 nDCG@10, so quantization costs nothing measurable here. That is the opposite of what docs/reranking.md said about the previous model, where the q8 export did degrade ranking; the loss doesn't carry over to a model with 12 layers and a large embedding table, so I've replaced that claim rather than just deleting it.

Of the four per-kernel builds in the repository, this PR uses model_quint8_avx2.onnx. Unsigned activations sidestep the signed-int8 saturation that x64 without VNNI has to work around, and it measured no slower than the arm64 build when run on arm64. qint8_arm64 and qint8_avx512 are bit-identical in output and score 0.8039, marginally higher, but that margin is inside the noise of a 240-query set and doesn't buy predictable behavior on hosts whose instruction set we can't know in advance.

Since the file has "avx2" in the name, I checked it on both architectures rather than assuming. Loading it and scoring the same pair gives 7.698 / -9.777 on arm64 macOS and 7.290 / -9.622 on x64 Linux (node:22-slim): the same ordering, with the relevant document about 17 above the irrelevant one on both. So the logits do shift by around 0.4 between kernel paths, which is worth knowing, but it is nowhere near the gaps the ranking depends on.

Documents are now scored one at a time

Dynamic quantization takes each activation tensor's scale from that tensor's own range at runtime. Padding rows out to a shared width puts the pad positions inside that range, and while the attention mask keeps them out of attention it cannot keep them out of the scale, so the quantization of the real tokens shifts. In batches of 10 that moved logits by up to 1.29 at ordinary snippet lengths (33-63 tokens) and reordered 2 of 10 fixtures depending only on which documents shared a batch. The fp32 export of the same model shows a difference of exactly zero, so this is a property of quantization and not of the graph.

Scoring one pair at a time removes the padding and the coupling with it. It also suits the reason batching was added in the first place: the event-loop stall drops to about 13ms instead of the 50ms a batch of 10 takes. The cost is throughput, roughly 12% more wall time for 30 documents on two threads and up to 47% more where there are cores to spare.

The session is now CPU-only

The WebGPU provider has no kernels for the integer matmuls in a quantized graph. It registers happily and then hands every one of them back to the CPU, paying a round trip each time: 812ms against 172ms for the same work, with scores drifting by up to 1.15 and reordering results. So ["webgpu", "cpu"] is out, and with it the two-attempt session logic, which existed only to survive a GPU provider failing to initialize.

Since ["cpu"] is the only thing ever passed, the provider constant went with it: the rationale now lives on createSession, and the startup log says "Loading model on CPU" plus the architecture, which is the part that actually varies between hosts (it selects the quantized kernel).

If keeping GPU acceleration matters more than the download size, the fp32 export is the trade: 471MB and 213ms per 30 documents on CPU, at 0.7973 nDCG@10.

Smaller changes that follow from the model

• MAX_SEQUENCE_LENGTH drops from 2048 to 512. That is the model's own limit, not a preference: it has 514 learned position embeddings and XLM-RoBERTa reserves two, so a 513-token pair fails outright with indices element out of data bounds at the position-embedding gather. Still more generous than what shipped before fix: truncate reranker input by tokens, not characters #2260, which cut documents to 512 characters upstream.
• truncatePairTokens no longer takes token_type_ids. XLM-RoBERTa has a type_vocab_size of 1 and emits zeros for the whole sequence, so there is no query segment to read off it. The function is now a two-argument slice, and it produces the same output the old one did for every case except a query that fills the budget on its own, where it now keeps the trailing separator.
• PAD_TOKEN_ID is gone, since nothing pads anymore.
• PREFERRED_EXECUTION_PROVIDERS and FALLBACK_EXECUTION_PROVIDERS are gone too, with no constant replacing them. There was never an environment variable for provider selection, so there's nothing to remove on that side.

Known limitations

• English loses a little, 0.774 => 0.743 nDCG@10.
• Latency measurements are from arm64 (Apple Silicon, 10 cores). The 2-thread column approximates a small host's core count but not the x64 instruction set, so the numbers on Hugging Face Spaces will differ. The x64 run above proves the file loads and ranks correctly there, but it ran under emulation, so it says nothing about x64 speed.
• check-docker passes, though that doesn't verify much here: the Playwright smoke test passes whether or not results come back, and it never asserts that the reranker became ready. What it does confirm is that the swap doesn't break container startup.
• The score filter in rankSearchResults.ts is documented as calibrated against the reranker's raw logit scale, and this model separates relevant from irrelevant by gaps of 5.7-11.2 where the old one used 0.98-2.59. I checked the filter rather than retuning it: across the 240 queries it keeps a median 18 of 30 results and retains 98.3% of the relevant ones, with the percentage fallback triggering once. It looks healthy on the new scale, so this PR leaves kStandardDeviationFactor alone.

How to test

1. npm test (349 tests, includes the rewritten server/rerankerService.test.ts).
2. npx vitest run --config vitest.integration.config.ts. This downloads ~136MB on first run and asserts that every relevant result outranks every irrelevant one on the four existing fixtures. All four pass, at score gaps of 5.7-11.2.
3. npm run lint.
4. npm run dev, then search for something non-English (configurar nginx como proxy reverso, cómo revertir el último commit en git) and check the ordering. First startup downloads the model, so give it a moment before the first search.

This branch is rebased on top of #2313, which unblocked the Lint Dockerfile step. Before that, the step short-circuited the job and CI never reached Check formatting or Run tests on any PR.

The reranking measurements come from a harness outside the repository: 240 MIRACL queries with a 30-candidate window per query, plus 10 web-snippet fixtures whose first four are copied verbatim from the integration test, so the harness could be checked against labels that were already in the repo before comparing models. Happy to push that harness somewhere if it's worth keeping.