Skip to content

fix(server): send gRPC proxy requests to the session that owns them - #736

Draft
omkargaikwad23 wants to merge 1 commit into
mainfrom
fix/grpc-proxy-session-queues
Draft

omkargaikwad23 wants to merge 1 commit into
mainfrom
fix/grpc-proxy-session-queues

Conversation

@omkargaikwad23

Copy link
Copy Markdown
Collaborator

What this fixes

GrpcProxyModel.generate used the queues of the first active session (list(PROXY_QUEUES.keys())[0]). With two concurrent Interact sessions, the prompts of the second session went to the client of the first session. The second session then got the answers of that client.

How

  • The Interact handler already puts the session ID into the config. GrpcProxyModel keeps this ID and uses the queues of that session.
  • A model that has no session ID uses the RPC ID of the caller. If that ID is not an active session, the model uses the only active session.
  • The model raises RuntimeError if its session is not active, or if it has no session ID and two or more sessions are active.

Behavior change

A request that the model cannot assign to a session now fails. Before, it went to the first active session.

Testing

  • New tests in grpc_proxy_session_test.py. On main, 4 of the 5 tests fail.
  • A before/after comparison is in a comment on this PR.

@omkargaikwad23

omkargaikwad23 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Verification

I ran the same script on main (6ca77dc) and on this branch. Two sessions, a and b, are active. Each session has a client thread that answers with its own name. Then each session sends one prompt.

Observation main This PR
Session a gets the answer from Client of a Client of a
Session b gets the answer from Client of a Client of b
Prompts that the client of a received Prompt of a, prompt of b Prompt of a
Prompts that the client of b received None Prompt of b

Local tests:

  • grpc_proxy_session_test.py, agent_grpc_proxy_test.py, and evalproto_packaging_test.py: 20 passed.
  • pytest evalbench/test with SKIP_CLOUD_TESTS=true: 1300 passed, 11 skipped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant