Skip to content

Embedding requests fan out one concurrent model call per input, unbounded #141

Description

@JGoutin

Found during the 1.16 milestone review. Pre-existing — present in released
versions, not introduced by 1.16.

stdapi/models/embedding/amazon_titan_embed.py embeds one input per model
invocation, and issued them all at once:

for result in await gather(*(self._invoke(request, v) for v in inputs)):

The fan-out is caller-controlled: a request carrying N inputs opens N
concurrent model invocations. AGENTS.mdPerformance Rules requires exactly
the opposite — "parallelize independent awaits under gather, bounded
(semaphore or fixed-size waves) when the fan-out is request-controlled".

Why it matters more now

A document split into chunks is a normal embedding workload, and the Vector
Stores API added in 1.16 does precisely that. A file yielding several hundred
chunks would have opened a connection per chunk and throttled itself against the
backend — with the retries and latency that follow — rather than running at a
steady, sustainable rate.

Fix

A per-request semaphore, matching the bounding idiom already used elsewhere in
the codebase (stdapi/aws_bedrock_sessions.py, and the image-generation paths in
the Responses adapter).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status
    Backlog

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions