feat: Microsoft.Extensions.AI IChatClient adapter (DotLLM.Extensions.AI) (#327) - #328
feat: Microsoft.Extensions.AI IChatClient adapter (DotLLM.Extensions.AI) (#327)#328jamesburton wants to merge 2 commits into
Conversation
…AI) (#327) Add a new DotLLM.Extensions.AI package implementing Microsoft.Extensions.AI.IChatClient over the dotLLM engine, making dotLLM a native, in-process backend for the Microsoft Agent Framework (GA Apr 2026) and the broader Microsoft.Extensions.AI ecosystem — wrap a model and pass the client to chatClient.CreateAIAgent(...). - DotLLMChatClient : IChatClient (GetResponseAsync, GetStreamingResponseAsync, GetService, Dispose) wrapping TextGenerator + IChatTemplate (+ optional IToolCallParser). Calls serialized through an internal gate (single-request). - ChatClientMapping (pure, unit-testable translation): * messages: ChatMessage/TextContent/FunctionCallContent/FunctionResultContent -> engine ChatMessage[] / ToolCall / tool-role messages. * ChatOptions (Temperature/TopP/TopK/MaxOutputTokens/StopSequences/Seed/ ResponseFormat/Tools, + min_p/repetition_penalty via AdditionalProperties) -> InferenceOptions + ToolDefinition[]. * output: text -> TextContent; detected ToolCalls -> FunctionCallContent; FinishReason -> ChatFinishReason; token counts -> UsageDetails; streaming GenerationToken -> ChatResponseUpdate. - AOT-clean (Utf8JsonWriter/JsonDocument, no reflection JsonSerializer); targets .NET 10; depends only on Microsoft.Extensions.AI.Abstractions 10.7.0. Tests: 22 unit tests covering message/options/tool/response/finish-reason/usage mapping. Docs: docs/EXTENSIONS_AI.md + ROADMAP/README/CLAUDE sync; added to dotLLM.slnx and central package management. Closes #327 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a Microsoft.Extensions.AI integration layer so dotLLM can be used as an in-process IChatClient (including tool calling, response formats, usage, and streaming), plus documentation and unit tests for the mapping layer.
Changes:
- Introduces new
DotLLM.Extensions.AIproject withDotLLMChatClient(IChatClient) and a pure mapping layer (ChatClientMapping). - Adds mapping-focused unit tests and wires the new project into the solution/test projects.
- Documents the integration in
docs/EXTENSIONS_AI.mdand updates README/roadmap references.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/DotLLM.Extensions.AI/DotLLMChatClient.cs | New IChatClient adapter over TextGenerator with non-streaming + streaming APIs |
| src/DotLLM.Extensions.AI/ChatClientMapping.cs | New translation layer between MEAI types and dotLLM engine types |
| src/DotLLM.Extensions.AI/DotLLM.Extensions.AI.csproj | New project referencing dotLLM engine + MEAI abstractions |
| tests/DotLLM.Tests.Unit/Extensions/DotLLMChatClientMappingTests.cs | Unit tests for mapping behavior (roles, tools, formats, usage) |
| tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj | Adds reference to the new integration project |
| docs/EXTENSIONS_AI.md | New documentation for MEAI/MAF integration and mapping reference |
| README.md / docs/ROADMAP.md / CLAUDE.md / dotLLM.slnx / Directory.Packages.props | Solution + docs wiring, package version addition |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| public static string ToRole(ChatRole role) => | ||
| role == ChatRole.System ? "system" | ||
| : role == ChatRole.Assistant ? "assistant" | ||
| : role == ChatRole.Tool ? "tool" | ||
| : "user"; |
| InferenceResponse result; | ||
| await _gate.WaitAsync(cancellationToken).ConfigureAwait(false); | ||
| try | ||
| { | ||
| result = await Task.Run( | ||
| () => _generator.Generate(prompt, inferenceOptions), cancellationToken).ConfigureAwait(false); | ||
| } |
| await foreach (var token in _generator | ||
| .GenerateStreamingTokensAsync(prompt, inferenceOptions, cancellationToken).ConfigureAwait(false)) | ||
| { | ||
| if (token.Text.Length > 0) | ||
| { | ||
| accumulated.Append(token.Text); | ||
| yield return new ChatResponseUpdate | ||
| { | ||
| Role = ChatRole.Assistant, | ||
| Contents = [new TextContent(token.Text)], | ||
| ResponseId = responseId, | ||
| MessageId = responseId, | ||
| ModelId = modelId, | ||
| }; | ||
| } | ||
|
|
||
| if (token.FinishReason.HasValue) | ||
| finishReason = token.FinishReason.Value; | ||
| } |
| // Post-generation tool-call detection (mirrors the server streaming endpoints). | ||
| EngineToolCall[]? toolCalls = null; | ||
| if (_toolCallParser is not null && tools is { Length: > 0 }) | ||
| { | ||
| toolCalls = _toolCallParser.TryParse(accumulated.ToString()); | ||
| if (toolCalls is { Length: > 0 }) | ||
| finishReason = FinishReason.ToolCalls; | ||
| } |
| private static float? ReadFloat(AdditionalPropertiesDictionary? props, string key) | ||
| { | ||
| if (props is null || !props.TryGetValue(key, out var value) || value is null) | ||
| return null; | ||
|
|
||
| return value switch | ||
| { | ||
| float f => f, | ||
| double d => (float)d, | ||
| int i => i, | ||
| long l => l, | ||
| JsonElement je when je.ValueKind == JsonValueKind.Number => je.GetSingle(), | ||
| string s when float.TryParse(s, out var r) => r, | ||
| _ => null, | ||
| }; | ||
| } |
| public async Task<ChatResponse> GetResponseAsync( | ||
| IEnumerable<ChatMessage> messages, | ||
| ChatOptions? options = null, | ||
| CancellationToken cancellationToken = default) |
| public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync( | ||
| IEnumerable<ChatMessage> messages, | ||
| ChatOptions? options = null, | ||
| [EnumeratorCancellation] CancellationToken cancellationToken = default) |
…327) - ToRole: preserve non-standard roles (e.g. "developer") instead of coercing every unrecognized ChatRole to "user"; the chat template now decides how to render them. Empty/whitespace roles still fall back to "user". - ReadFloat: parse string-valued AdditionalProperties knobs (min_p, repetition_penalty) with InvariantCulture so "0.05" is not rejected on a machine whose current culture uses ',' as the decimal separator. - Streaming + tools: when a tool-call parser is configured AND the request carries tools, buffer the text instead of emitting TextContent deltas, and deliver the message in the final update via the same mapping the non-streaming path uses. Previously the model's raw tool-call syntax was streamed to the caller, and the coalesced stream (ToChatResponse) disagreed with GetResponseAsync, which drops text when tool calls are present. Requests without tools (or without a parser) stream token-by-token as before. - Documented in DotLLMChatClient and docs/EXTENSIONS_AI.md, including why GetResponseAsync offloads the synchronous Generate to the thread pool (the token gates scheduling only; use streaming for mid-generation cancellation). Tests: new DotLLMChatClientTests drives the client with a scripted model (deterministic one-hot logits) covering non-streaming mapping, max-tokens finish reason, template/tool plumbing, tool-call detection, the streaming update sequence, the buffered tool path, streaming/non-streaming agreement, and GetService. Plus mapping tests for the unknown-role and comma-culture fixes. 34 Extensions.AI unit tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thanks for the review — went through all 7 comments. Pushed Fixed1. 2. Culture-sensitive 3. Streaming leaked raw tool-call text ( The trade-off is explicit: a tool-enabled streaming request no longer streams incrementally. It can't, correctly — tool calls are only recognisable once generation completes ( 4. No tests for Not applied
Your observation about cancellation is correct though, so I've documented it rather than papered over it: a comment on the call site and a note in VerificationBuilt |
Summary
Adds a
Microsoft.Extensions.AI.IChatClientimplementation for dotLLM in a newDotLLM.Extensions.AIpackage, making dotLLM a native, in-process backend for the Microsoft Agent Framework (GA April 2026) and the broaderMicrosoft.Extensions.AI(MEAI) ecosystem.IChatClientis the universal plug for the Microsoft AI stack — MAF agents (ChatClientAgent/AIAgent), the function-invocation middleware, response caching, OpenTelemetry, and DI all consume it. Wrap a loaded model and pass the client tochatClient.CreateAIAgent(...)with zero glue.Purely additive and independent of engine internals — a thin translation shim over
TextGenerator. Branches cleanly offmain.Closes #327.
What's added
DotLLMChatClient : IChatClient—GetResponseAsync,GetStreamingResponseAsync,GetService,Dispose, wrappingTextGenerator+IChatTemplate(+ optionalIToolCallParser). Calls are serialized through an internal gate (the engine is single-request, like the server); streaming holds the gate for the enumeration.ChatClientMapping— pure, unit-testable translation:ChatMessage/TextContent/FunctionCallContent/FunctionResultContent→ engineChatMessage[]/ToolCall/tool-role messages;ChatOptions(Temperature/TopP/TopK/MaxOutputTokens/StopSequences/Seed/ResponseFormat/Tools, plusmin_p/repetition_penaltyviaAdditionalProperties) →InferenceOptions+ToolDefinition[].TextContent; detectedToolCalls →FunctionCallContent(text dropped, finish reason tool-calls);FinishReason→ChatFinishReason; token counts →UsageDetails; streamingGenerationToken→ChatResponseUpdate.Quality
Utf8JsonWriter/JsonDocumentonly, no reflection-basedJsonSerializer(the project's trim analyzer is satisfied).DotLLM.Engine+Microsoft.Extensions.AI.Abstractions10.7.0 (added to central package management). Registered indotLLM.slnx; packs cleanly.docs/EXTENSIONS_AI.md(usage, integration paths, full type mapping) +ROADMAP/README/CLAUDEsync.Notes / limitations
FunctionCallContentis emitted in the final streaming update rather than incrementally.ChatOptions.FrequencyPenalty/PresencePenaltyhave no direct engine equivalent and are not applied;RepetitionPenalty/MinPare reachable viaAdditionalProperties.FunctionInvokingChatClient— no custom loop needed.🤖 Generated with Claude Code