Repository navigation
refactor(server): the model access and the tool access are composed when first used - #305
Merged
rami-hatoum merged 3 commits intoOct 23, 2026
Merged
Conversation
…en first used The composition root holds a capability registry: one memoized layer per entry, a LayerRef kept for the server's life, built the first time it is asked for and answered ever after. A composition that fails is not kept, so the next ask composes again, and the registry closes what it composed once the host has stopped. Its first entry is the model access of reasoning functions: the provider clients and the credential libraries some providers load. The reasoning adapter and list_models are made at start over a language model and a catalog that ask the registry, so the first model call or model list pays the composition, inside its own request. The start still logs which providers are configured, read from the settings. The reasoning capability exports offeredModels, and installedPackages, which resolves the optional Entra ID package without loading it, so the start and the guide name the same offer as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… caches filled, on first use The registry's second entry is the tool access: the OpenAPI documents of the tool servers are read and described the first time a search, a read, a call, a run with tools or a due request needs them. The reasoning and interaction adapters, the five tool operations and the interaction due work are made at start over an access that asks the registry, so the due work composes it only once a row of it is due. The registry's scope now closes when the runtime is disposed, before the services built ahead of it, rather than once the host stops. A closed entry answers no ask, so closing it with the work would cut a request still in flight at its next model call; the tool access, which closed with the work before, now closes once the API has closed, as the runtime does. A document that cannot be read no longer stops the start: the request that needed it fails with an incident whose log line names the entry in the words the start used, and the next request reads it again. The test that held the start to that refusal is replaced by one that holds the request to it, and to the read that follows once the document exists. The prediction caches no longer read and check every file they hold at start; each does so when a prediction first uses it, which every use already awaited. The computation pool and recall's store stay at start, since the host decides workflow steps and filters reactions with the pool's evaluations and opens its store with the folders of recall and of the datasets. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…arts and first requests The server README sets out what the composition root composes at start and what its capability registry composes on first use, that the ready line means the server listens, that a failed composition fails its request and is composed again on the next, and when the registry closes. The self-hosting pages say the documents of openapi entries are read, and their refusals met, when a request first needs the tool servers, and the files of the model cache when a prediction first reads them. The reasoning README names installedPackages and offeredModels. The tool-servers README carries the measures beside its start measure, the median of 11 starts on SQLite interleaved with main's: 727 ms against 727 ms with nothing configured; 732 ms, 2,312 files and 18.5 MiB against 833 ms, 2,757 files and 20.8 MiB with five model providers and two openapi entries; 723 ms against 871 ms with 320 MiB in the model cache. The first list_models takes 65.6 ms against 1.0 ms and the first search 51.0 ms against 17.8 ms, at the median of 9 servers; readying the cache's 320 MiB on first read takes 146 ms. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rami-hatoum
deleted the
refactor/a-capability-is-composed-when-first-used
branch
October 23, 2026 16:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
At start the server built, for every configured provider and gateway, a model client with its credential library, and described every configured OpenAPI document, before it listened; a server that would only ever run workflows over computation functions paid for both, and a start failed on a document it could not read. Those two now compose on first use, through a registry of one memoized layer per entry, built the first time an operation asks and kept; a build that fails is answered to that request as an incident, whose log line carries the words the start used to stop with, and is tried again on the next ask, never kept as failed. The prediction caches likewise read and verify their files on their first read rather than at start.
What still composes at start is what any request or the host needs: the application layer, the secrets and connections, the program pool, the workflow store, the prediction pool's ports, every adapter with its declaration, the route table and the host; the route table does not change.
Measured on the same machine, the median of eleven starts: with nothing configured, 727 ms before and after; with five providers and two OpenAPI entries, 833 ms and 2,757 files before, 732 ms and 2,312 files after; with 320 MiB in the model cache, 871 ms before, 723 ms after. The first request of a kind pays instead:
list_models65.6 ms against 1.0 ms,search51.0 ms against 17.8 ms, the first cache read 146 ms. The tool-servers README carries the figures; the server README says what composes when; the self-hosting pages follow.Tests
pnpm checkpasses on each commit, cold, with the PostgreSQL suites running.