Run instances concurrently, for agents that wait on a network - #4
Open
DenisDrobyshev wants to merge 1 commit into
Open
Run instances concurrently, for agents that wait on a network#4DenisDrobyshev wants to merge 1 commit into
DenisDrobyshev wants to merge 1 commit into
Conversation
Measuring what the benchmark exists for turned up the obstacle to doing it. An episode is a chain of round-trips, because a decision cannot start until the previous one's outcome is known: 100 of them on queueing, 60 on inventory, energy and supply-chain, 40 on joint-pricing. One episode of every task is about 325 model calls, and a modest run across all six is 5,200 — an hour and a half of a language model waiting, single file. Instances are independent, so they overlap. Eight workers against an agent with 8 ms of simulated latency: 0.77s to 0.10s, with the returns bit-identical, because every seed is fixed before the pool starts and rows are collected by index. A test pins that equality — the moment concurrency changes a score, the scores stop being reproducible and this stops being a benchmark. Above one worker the agent argument is a factory rather than an instance, and passing an instance is refused with the reason. An agent driving a language model keeps the brief and the recent decisions for the episode it is in; shared across threads that memory interleaves, and the failure would look like a bad policy rather than a bug.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measuring what the benchmark exists for turned up the obstacle to doing it.
An episode is a chain of round-trips: a decision cannot start until the previous
one's outcome is known. That is 100 decisions on
queueing, 60 oninventory,energyandsupply-chain, 40 onjoint-pricing, about 5 onpricingwherethe stock sells out early — roughly 325 model calls for one episode of every
task, and 5,200 for a modest run of eight instances by two episodes across all
six. Single file, that is around an hour and a half of a language model waiting.
Instances do not depend on each other, so they overlap. Against an agent with
8 ms of simulated latency per decision:
The returns are bit-identical because every seed is fixed before the pool starts
and rows are collected by index. A test pins that equality: the moment
concurrency moves a score, the scores stop being reproducible and this stops
being a benchmark.
Above one worker the agent argument is a factory rather than an instance, and
passing an instance is refused with the reason in the message. An agent driving
a language model holds the brief and its recent decisions for the episode it is
in; shared across threads that memory interleaves, and the failure would read as
a bad policy rather than as a bug.
Both READMEs gain the per-task call budget, so a run can be costed before it is
started rather than after.