Make Chunking and Embedding usable against remote embedding APIs, and add a Reranking package

[AdamTovatt]

Closes #2, parts 1 to 5. Part 6 (tagging and publishing) is deliberately not here — nothing is tagged and nothing is published.

Versions

Package	Version	Why
VectorSharp.Chunking	1.0.0 → 2.0.0	Two behaviour changes for existing callers: MaxTokensPerChunk is now a bound, and enumerating a reader twice throws. No signature changed.
VectorSharp.Embedding	1.0.3 → 1.1.0	Purely additive: batching and usage reporting arrive as default interface implementations, so every existing provider compiles and behaves unchanged.
VectorSharp.Reranking	— → 1.0.0	New package.

VectorSharp.Storage and VectorSharp.Embedding.NomicEmbed are untouched. NomicEmbed implements IEmbeddingProvider and is the reason the no-breaking-changes rule was load-bearing throughout; its test results are identical to before the branch.

The commits

accept any text reader and yield chunks that know their position (parts 1 and 2)

ChunkReader.Create widens from StreamReader to TextReader. The StreamReader overload stays: chunking-v1.0.0 is on nuget.org, and assemblies compiled against it bind that exact signature in IL, so removing it would throw MissingMethodException at run time even though their source still compiles. ReadChunksAsync yields TextChunk carrying Text, StartOffset and TokenCount; ReadAllAsync is unchanged and delegates to it. Offsets come from the segment reader's own cursor, so lookahead pushed back into the buffer is not counted until the segment holding it is returned.

batch texts into single provider requests and report token usage (parts 3 and 4)

EmbedBatchAsync and EmbedBatchWithUsageAsync are default interface implementations on IEmbeddingProvider, so an existing provider keeps working untouched. A provider advertises SupportsBatching, defaulting to false: one that has not overridden the batch methods gains nothing from grouping, since the defaults loop the single-text method and a group would run serially inside one worker instead of spreading across the pool. That is what keeps a local ONNX model on exactly today's fan-out.

Usage is optional and never fabricated. A provider that meters nothing reports null, not zero, and a call split across batches reports a total only when every batch reported one — a partial total would understate spend, which is the failure that matters when the caller is billed. A reported zero survives as a reported zero.

Found while chasing a hung test: a batch left in the channel when disposal stopped the workers stranded its caller forever. It is now failed with ObjectDisposedException.

add the VectorSharp.Reranking package (part 5)

IRerankProvider.RerankAsync(query, documents, topN, ct). Matches carry the index into the caller's list plus a score, not the text: the caller holds whatever sits behind each candidate — a row id, a chunk offset, a file path — and returning text would force a lookup by content, which is ambiguous when two candidates read the same. Same optional-usage shape as embedding. No service, queue or worker pool: reranking is one call per user query against a candidate set already in hand. Zero dependencies, including on its sibling packages.

fix the packaging tests and apply the whole-branch review

The packaging tests read the version from the project file, but the publish workflow builds and packs with -p:Version from the tag, which overrides <Version> for every project. On a release build they looked for a .nupkg the build never wrote, and would have failed the next publish. Running the workflow's exact invocation reproduces it. The version now comes from the package's own built assembly, stamped from the same property however the build was invoked.

Those tests moved to VectorSharp.Packaging.Tests with a ProjectReference to every packable project, so running them builds the packages they inspect. The package set is read from VectorSharp.slnx rather than a hand-kept list, so a package added later is covered by default and one with legitimate dependencies has to be named with its reason.

Also here: disposal releases providers and queued callers from a finally; the batching and usage-combination rules move into BatchingPolicy and EmbeddingUsage.Combine where they can be tested without a worker pool; the duplicated segment-and-push-back blocks in SegmentReader become one helper.

make MaxTokensPerChunk a bound rather than a target

Through 1.x, a segment with no break point inside it was emitted whole however far over the limit it was, and that was documented as a carve-out. It was a defect: the chunks go to a model with a fixed context window, which rejects or silently truncates one that is too long, and a truncated chunk is a piece of the document that is no longer searchable with nothing in the result to say so. Such a segment is now cut at the limit, and the carve-out is deleted from TextChunk, ChunkReaderOptions and the README.

ChunkSplitter finds where to cut by measuring candidate prefixes with the caller's token counter. Every length it returns is one it measured, never one interpolated between two probes, so a tokenizer that is not perfectly monotonic in the length of its input — a byte-pair encoder can charge less for "the" than for "th" — still yields pieces that genuinely fit.

The assumption the bound rests on, stated because it is the honest part: the search probes outward and stops at the first piece that does not fit, so it assumes a counter reporting over the limit for a short piece will not report under it for a longer one. Every real tokenizer satisfies that. A counter that violates it — one charging more for a single character than the whole limit allows — gets single-character chunks over the limit, because the alternative is cutting a character in half. That case is documented on MaxTokensPerChunk and tested for what still holds there: termination, round-trip and offsets.

The pieces concatenate back into the segment they came from and each carries its own offset, so round-tripping and StartOffset hold across a cut. A cut never falls between the two halves of a surrogate pair. It does land mid-word for any subword or character counter, which is what a real embedding model uses — the README says so plainly, with an example pinned by a test rather than asserted.

Verification

Full suite: 390 passed, 0 failed, 32 skipped. The skips are VectorSharp.Embedding.NomicEmbed.Tests self-skipping its RequiresModel cases because the .onnx model files are not in the tree — unchanged from the baseline before this branch.

Each check was made to fail before being trusted, and doing so found three defects that had already passed a review round:

• The packaging test's version anchor: reading it from the project file fails 9 tests under the workflow's own build command, and searching bin unpinned makes a Release run validate a broken Debug artifact.
• The cut could divide a surrogate pair in half at odd limits — found because the test guarding against it still passed with the guard removed.
• Cutting re-sliced the remainder per piece, quadratic in the chunk; caught by an allocation assertion that fails at 4.5 MB against a 20 KB input.

The last review round found a fourth, in documentation rather than code: the non-monotonic-counter test never entered the cut path, and the property it claimed was false. Both are corrected above, and the guarantee is now stated with its assumption instead of as an absolute.