Skip to content

Add SGLang backend adapter - #11

Open
algebra-MCX wants to merge 2 commits into
uw-syfi:mainfrom
algebra-MCX:sglang-backend
Open

Add SGLang backend adapter#11
algebra-MCX wants to merge 2 commits into
uw-syfi:mainfrom
algebra-MCX:sglang-backend

Conversation

@algebra-MCX

Copy link
Copy Markdown

Summary

  • Split the session runner backend layer into reusable backend adapters.
  • Add native SGLang /generate support with token-id input, streaming response parsing, generated token-id recovery, and cache metric normalization.
  • Keep the existing vLLM backend behavior through the new shared backend client.
  • Add a llama.cpp placeholder so the next backend can be implemented without changing the runner architecture.

Validation

  • cargo test --manifest-path replay/Cargo.toml
  • Real GPU SGLang replay test: 10 sessions, 80 rounds, 80/80 successful, 18,684 generated tokens.
  • Real GPU vLLM replay test: 10 sessions, 80 rounds, 80/80 successful, 18,684 generated tokens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant