Found during the 1.16 milestone review. Pre-existing — present in released
versions, not introduced by 1.16.
stdapi/models/embedding/amazon_titan_embed.py embeds one input per model
invocation, and issued them all at once:
for result in await gather(*(self._invoke(request, v) for v in inputs)):
The fan-out is caller-controlled: a request carrying N inputs opens N
concurrent model invocations. AGENTS.md → Performance Rules requires exactly
the opposite — "parallelize independent awaits under gather, bounded
(semaphore or fixed-size waves) when the fan-out is request-controlled".
Why it matters more now
A document split into chunks is a normal embedding workload, and the Vector
Stores API added in 1.16 does precisely that. A file yielding several hundred
chunks would have opened a connection per chunk and throttled itself against the
backend — with the retries and latency that follow — rather than running at a
steady, sustainable rate.
Fix
A per-request semaphore, matching the bounding idiom already used elsewhere in
the codebase (stdapi/aws_bedrock_sessions.py, and the image-generation paths in
the Responses adapter).
Found during the 1.16 milestone review. Pre-existing — present in released
versions, not introduced by 1.16.
stdapi/models/embedding/amazon_titan_embed.pyembeds one input per modelinvocation, and issued them all at once:
The fan-out is caller-controlled: a request carrying N inputs opens N
concurrent model invocations.
AGENTS.md→ Performance Rules requires exactlythe opposite — "parallelize independent awaits under
gather, bounded(semaphore or fixed-size waves) when the fan-out is request-controlled".
Why it matters more now
A document split into chunks is a normal embedding workload, and the Vector
Stores API added in 1.16 does precisely that. A file yielding several hundred
chunks would have opened a connection per chunk and throttled itself against the
backend — with the retries and latency that follow — rather than running at a
steady, sustainable rate.
Fix
A per-request semaphore, matching the bounding idiom already used elsewhere in
the codebase (
stdapi/aws_bedrock_sessions.py, and the image-generation paths inthe Responses adapter).