Make Chunking and Embedding usable against remote embedding APIs, and add a Reranking package

.github/workflows/publish.yml

@@ -36,6 +36,7 @@ jobs:
           embedding)   PROJECT="VectorSharp.Embedding" ;;
           nomic-embed) PROJECT="VectorSharp.Embedding.NomicEmbed" ;;
           chunking)    PROJECT="VectorSharp.Chunking" ;;
+          reranking)   PROJECT="VectorSharp.Reranking" ;;
           *)           echo "Unknown package: $PACKAGE" && exit 1 ;;
         esac
         echo "PROJECT=${PROJECT}" >> $GITHUB_OUTPUT

Publishing.md

@@ -18,6 +18,7 @@ Where `{package}` is the lowercase package name and `{version}` is a valid semve
 | `embedding` | VectorSharp.Embedding |
 | `nomic-embed` | VectorSharp.Embedding.NomicEmbed |
 | `chunking` | VectorSharp.Chunking |
+| `reranking` | VectorSharp.Reranking |
 
 ## Publishing a Package
 
@@ -28,36 +29,39 @@ git push origin storage-v1.0.0
 
 This triggers the publish workflow which will build, run tests, pack the specified project with the given version, and push to nuget.org.
 
-## Examples
+Packages are versioned independently: a tag names one package and one version, and publishing one
+leaves the others alone. Take the prefix from the table above.
 
-```bash
-# Publish VectorSharp.Storage 1.0.0
-git tag storage-v1.0.0
-git push origin storage-v1.0.0
+### The tag is what sets the version
 
-# Publish VectorSharp.Storage 1.1.0 (independent of other packages)
-git tag storage-v1.1.0
-git push origin storage-v1.1.0
+The workflow builds and packs with `-p:Version=<the version in the tag>`, which overrides the
+project file's `<Version>` for every project in the build. **The tag decides what is published;
+the project file does not.** `embedding-v1.0.3` shipped from a project file that read
+`<Version>1.0.0</Version>`, and nothing noticed.
 
-# Publish VectorSharp.Embedding 1.0.0 (independent of other packages)
-git tag embedding-v1.0.0
-git push origin embedding-v1.0.0
-
-# Publish VectorSharp.Embedding.NomicEmbed 1.0.0
-git tag nomic-embed-v1.0.0
-git push origin nomic-embed-v1.0.0
-
-# Publish VectorSharp.Chunking 1.0.0
-git tag chunking-v1.0.0
-git push origin chunking-v1.0.0
-```
+Keep `<Version>` in step with the last tag anyway, and bump it in the same change that adds the
+behaviour it names. It is what a local `dotnet pack` produces, what a developer reads to see what
+is in the working tree, and how a reviewer tells a patch from a minor.
 
 ## Prerequisites
 
 A `NUGET_API_KEY` secret must be configured in the repository settings (Settings > Secrets and variables > Actions). The key can be generated at https://www.nuget.org/account/apikeys.
 
 ## Adding a New Package
 
-1. Add the project to the solution
+1. Add both the package project and its test project to `VectorSharp.slnx`
 2. Add a new case in `.github/workflows/publish.yml` mapping the tag prefix to the project name
-3. Update the table above
+3. Update the tag prefix table above
+4. Add a row to the Packages table in the root `README.md` — a package missing from it is invisible
+   to anyone arriving at the repository
+5. Write the package's own `README.md`, and wire it up in the project file with both
+   `<PackageReadmeFile>README.md</PackageReadmeFile>` and
+   `<None Include="README.md" Pack="true" PackagePath="\" />`. Without the second, the build still
+   succeeds and the nuget.org listing renders blank
+6. Add a `<ProjectReference>` to it in `VectorSharp.Packaging.Tests.csproj`
+
+`VectorSharp.Packaging.Tests` takes the set of packages from `VectorSharp.slnx` rather than a list
+of its own, so step 1 is what puts a new package under its checks — there is no list to forget to
+update. Step 6 is what lets those checks find the built `.nupkg`; without it they fail naming the
+missing reference. A package that legitimately has dependencies must be named in that project's
+`PackagesWithDependencies`, with the reason.

README.md

@@ -3,16 +3,17 @@
 [![Tests](https://github.com/AdamTovatt/vector-sharp/actions/workflows/dotnet.yml/badge.svg)](https://github.com/AdamTovatt/vector-sharp/actions/workflows/dotnet.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
 
-A high-performance .NET library for in-process vector similarity search and text embedding. No external services, no Docker, no infrastructure — just NuGet packages that work.
+A high-performance .NET library for in-process vector similarity search and text embedding. No external services required, no Docker, no infrastructure — just NuGet packages that work. Search, storage and chunking run entirely in your process; the embedding and reranking packages let you reach a hosted API when you want one, but never oblige you to.
 
 ## Packages
 
 | Package | Description |
 |---------|-------------|
 | [VectorSharp.Storage](VectorSharp.Storage/README.md) | In-memory and disk-backed vector storage with SIMD-optimized cosine similarity |
-| [VectorSharp.Embedding](VectorSharp.Embedding/README.md) | Channel-based embedding service with configurable parallelism |
+| [VectorSharp.Embedding](VectorSharp.Embedding/README.md) | Channel-based embedding service with configurable parallelism, request batching and token usage reporting |
 | [VectorSharp.Embedding.NomicEmbed](VectorSharp.Embedding.NomicEmbed/README.md) | Nomic Embed Text v1.5 model for local inference (768-dim, 8192 token context) |
-| [VectorSharp.Chunking](VectorSharp.Chunking/README.md) | Streaming text chunker with predefined formats for Markdown and C# |
+| [VectorSharp.Chunking](VectorSharp.Chunking/README.md) | Streaming text chunker with a hard token bound, position-carrying chunks and predefined formats for Markdown, C#, JavaScript/TypeScript, HTML, CSS, Python and plain text |
+| [VectorSharp.Reranking](VectorSharp.Reranking/README.md) | Reranking abstractions for the second stage after a vector or hybrid search |
 
 ## Quick Start
 

VectorSharp.Chunking.Tests/ChunkReaderFormatTests.cs

@@ -2,33 +2,6 @@ namespace VectorSharp.Chunking.Tests
 {
     public class ChunkReaderFormatTests
     {
-        private static StreamReader ReaderFrom(string text)
-        {
-            MemoryStream stream = new MemoryStream(System.Text.Encoding.UTF8.GetBytes(text));
-            return new StreamReader(stream);
-        }
-
-        private static async Task<List<string>> ReadAllChunks(ChunkReader reader)
-        {
-            List<string> chunks = new List<string>();
-            await foreach (string chunk in reader.ReadAllAsync())
-            {
-                chunks.Add(chunk);
-            }
-            return chunks;
-        }
-
-        private static ChunkReader CreateReader(string input, IReadOnlyList<string> breakStrings, IReadOnlyList<string> stopSignals, int maxTokens = 10)
-        {
-            StreamReader streamReader = ReaderFrom(input);
-            return ChunkReader.Create(streamReader, TokenCounter.CountWords, new ChunkReaderOptions
-            {
-                MaxTokensPerChunk = maxTokens,
-                BreakStrings = breakStrings,
-                StopSignals = stopSignals
-            });
-        }
-
         #region PlainText
 
         [Fact]