Skip to content

Add Ray runtime support - #203

Open
sercanuste wants to merge 7 commits into
mainfrom
feature/gpu-fleet/ray
Open

Add Ray runtime support#203
sercanuste wants to merge 7 commits into
mainfrom
feature/gpu-fleet/ray

Conversation

@sercanuste

@sercanuste sercanuste commented Jul 26, 2026

Copy link
Copy Markdown
Member

This pull request adds support for the Ray Serve LLM runtime, improves GPU fleet navigation, and introduces explicit /dev/shm sizing for LLM deployments to prevent engine startup issues, especially with vLLM and Ray. It also updates the database schema and types to support these features.

Ray Serve LLM runtime support:

  • Added a new ray runtime to the LLM deployment options, including configuration, startup command, and special handling for GPU hosts and OpenAI compatibility (src/config/gpu-model-library.json, src/lib/database/provider/types.extended.ts). [1] [2]

GPU fleet navigation improvements:

  • Added breadcrumb resolvers for GPU fleet hosts and deployments to improve navigation in the UI (src/components/layout/breadcrumbResolvers.ts).

/dev/shm sizing for LLM deployments:

  • Added a shmSizeBytes field to LLM deployment types, database schema, and migration logic, allowing explicit override of shared memory size for containers to avoid vLLM/Ray startup hangs (src/lib/database/provider/types.extended.ts, src/lib/database/sqlite/base.ts, src/lib/database/sqlite/schema.ts). [1] [2] [3]

Development workflow:

  • Increased the Node.js memory limit for the dev:watch script to match the main dev script, preventing out-of-memory errors during development (package.json).

@sercanuste sercanuste changed the title Add Ray deployment option and memory limit to dev:watch script Add Ray runtime support Jul 26, 2026
@sercanuste
sercanuste marked this pull request as ready for review July 26, 2026 23:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant