GraphBit roadmap: GPUI desktop MVP/v1.0 and vLLM support

This document proposes a practical plan for two requested directions:

Build a desktop GUI with GPUI so GraphBit has a unified Rust-native experience.
Evaluate and add vLLM support, including batched/parallel execution where appropriate.

It also includes suggested next steps outside those two areas.

1) GPUI desktop application plan

Current repo observations

The repository is primarily a Rust core plus Python bindings/workspace members.
There is no existing desktop UI crate or app shell in the workspace yet.
The current root workspace members are:
- core
- guardrail_ffi
- python
The Rust side already exposes the workflow engine, LLM provider abstraction, memory, streaming, and execution primitives, which is a good foundation for a native desktop app.

Product goal

Create a Rust-native desktop application that lets users visually build, run, inspect, and debug GraphBit workflows without dropping into Python first.

Recommended architecture

Add a new workspace member, for example:

gpui-app/

Keep the UI thin and let it depend on graphbit-core directly. Avoid routing the desktop app through Python bindings.

Suggested layers:

UI layer (GPUI)
- windows, panels, node editor, forms, run controls, logs
App state layer
- loaded workflow, execution state, selected provider, environment profile, run history
Integration layer
- translates UI actions into graphbit-core workflows/executor calls
Core layer
- existing graphbit-core

MVP scope

The MVP should focus on proving that GraphBit workflows can be created, executed, and inspected visually.

MVP user stories

Create a workflow visually.
Add/edit core node types.
Connect nodes in a graph.
Configure one LLM provider and model.
Run the workflow.
See node-by-node status and outputs.
View logs/errors for failures.
Save and reopen workflow definitions.

MVP screens/panels

Workflow canvas
- node graph editor
- drag/drop nodes
- connect/disconnect edges
- select node to edit properties
Node configuration panel
- node name/description
- node type selector
- prompt template editor
- system prompt override
- basic tool assignment
- timeout/retry settings if already supported cleanly
Run panel
- select provider/model
- API key/env var mapping
- run/cancel buttons
- execution progress
Execution inspector
- per-node state: pending/running/completed/failed
- output preview
- elapsed time
- error messages
Logs/trace panel
- structured runtime logs
- request/response metadata summary
- token/cost/latency if available
File/project actions
- new workflow
- open/save workflow JSON or another stable format
- recent workflows

MVP node types to support first

Start with the smallest useful set:

Agent
Condition
Transform
Split
Join
Delay
HttpRequest
DocumentLoader

If some node types are not yet ergonomic for visual editing, ship only the subset that is robust enough, but make Agent workflows first-class.

MVP non-goals

Do not block MVP on:

multi-user collaboration
plugin marketplace
visual diff/versioning
advanced observability dashboards
embedded notebook experience
full memory management UX
benchmark runner UI

v1.0 scope

v1.0 should turn the app from “workflow runner/editor” into a real operator/developer console.

v1.0 additions

Streaming-first execution UX
- token streaming in node outputs
- live tool-call display
- progressive node status updates
Template/project system
- starter templates for common workflows
- example gallery
- clone-from-example to editable project
Provider management
- multiple saved providers/endpoints
- environment profiles (local/dev/staging/prod)
- per-node/provider override where applicable
Debugging and replay
- replay prior runs
- inspect full node inputs/outputs
- compare two runs
- deterministic re-run when inputs are unchanged
Guardrail/memory UI
- configure guardrail policy attachment
- inspect memory writes/reads
- toggle memory scopes and retention behavior
Artifact/document handling
- upload local files to document loader nodes
- inspect parsed chunks/splits
- preview extracted content
Observability dashboard
- latency histograms
- token usage by node/provider
- failure and retry analytics
- concurrency statistics
Packaging and distribution
- signed builds for macOS/Windows/Linux
- auto-update channel
- settings import/export
Workflow validation UX
- pre-run validation errors inline on nodes/edges
- cycle detection warnings
- missing provider/tool config warnings
Test harness / simulation mode

dry-run or mock provider mode
snapshot testing for workflows

Suggested implementation phases for GPUI

Phase 1: foundation

Add gpui-app workspace crate.
Add minimal shell window and app state.
Define a serialized workflow document format for the GUI.
Add adapters between GUI models and graphbit-core workflow types.

Phase 2: editor

Build node canvas and selection model.
Add inspector forms for the initial node types.
Add save/open support.

Phase 3: execution

Add executor integration.
Add run status, output, and error panels.
Add cancellation and reset behavior.

Phase 4: polish to MVP

Add validation UX.
Add logs panel.
Add basic settings/provider management.
Package first desktop preview builds.

Phase 5: v1.0 expansion

Streaming UX
replay/debugging
guardrails/memory
observability
templates and packaging polish

Technical notes for GPUI integration

Keep graphbit-core as the source of truth for workflow execution.
Add a thin app-facing API in Rust if the current public API is slightly too low-level for UI use.
Prefer async task orchestration via Tokio with UI-safe state updates.
Model execution state separately from workflow-definition state to keep the UI responsive.
Treat secrets carefully: prefer env var references or OS keychain integration rather than storing raw keys in project files.

MVP acceptance criteria

A user can create a workflow with at least 2–3 nodes.
A user can configure an LLM provider and run it from the app.
The UI shows node statuses and outputs.
The user can save and reopen the workflow.
Basic validation prevents obviously invalid graphs/config.

v1.0 acceptance criteria

Streaming, replay, validation, and provider management are solid enough for daily development use.
Packaging/distribution exists for the target desktop platforms.
The app can be used as the primary workflow authoring/debugging environment for GraphBit.

2) vLLM support assessment and plan

Is vLLM support present today?

Not explicitly as a first-class provider.

What exists today:

The core has many provider implementations under core/src/llm/.
The provider list includes openai, anthropic, ollama, openrouter, togetherai, replicate, mistralai, gemini, deepseek, huggingface, ai21, xai, azurellm, bytedance, and a Python bridge.
There is no dedicated vllm.rs provider and no LlmConfig::Vllm variant visible in the provider module structure.

Could it already work indirectly?

Possibly, partially, through the existing OpenAI-compatible path.

Because the OpenAI provider supports a custom base_url, you can often point OpenAI-compatible clients at a vLLM server endpoint, assuming the target vLLM deployment exposes compatible chat completion APIs and the needed features.

That means there are two levels of support:

Unofficial compatibility mode
- Use LlmConfig::OpenAI { ..., base_url: Some(...) }
- Point it to a vLLM OpenAI-compatible endpoint
Real first-class vLLM support
- Add an explicit Vllm provider/config variant
- Add vLLM-specific capabilities, defaults, and batch APIs

Is it taking advantage of batch processing / parallel execution?

Parallel execution

At the workflow level: yes, GraphBit appears designed for concurrency.

The repo documentation and core docs describe:

deterministic concurrent execution
parallel processing in the workflow engine
concurrency statistics in the executor
benchmark scenarios for parallel and concurrent tasks

So GraphBit already appears to exploit workflow-level parallelism.

Batch processing for vLLM specifically

No evidence of explicit vLLM request batching was found.

What is missing from the current provider surface:

no first-class vLLM provider
no exposed provider batch API for text generation akin to embed_texts
no scheduler that groups compatible LLM requests into a single batched provider call
no obvious complete_many or dynamic batching abstraction in the LLM provider layer

So the likely current situation is:

GraphBit can run multiple tasks concurrently.
Each task likely issues its own LLM request independently.
That is parallelism, but not necessarily provider-side batching.

Recommendation

Implement vLLM in two stages:

Stage A: first-class compatibility provider
- Add a dedicated Vllm provider using OpenAI-compatible endpoints.
Stage B: batched execution support
- Add optional batch APIs and request coalescing for workloads that benefit from vLLM batching.

3) How to add vLLM support

Option A: quick path

Add LlmConfig::Vllm but internally reuse most of the OpenAI request/response logic.

Proposed config shape

Vllm {
    api_key: Option<String>,
    model: String,
    base_url: String,
    max_batch_size: Option<usize>,
    enable_auto_batching: Option<bool>,
}

Notes:

api_key should be optional because many self-hosted vLLM deployments do not require one.
base_url should be required for clarity.
Keep feature flags explicit instead of hiding batching behavior.

Option B: full provider abstraction improvement

Refactor the provider layer so OpenAI-compatible providers share a transport/core implementation.

Suggested shape:

shared OpenAI-compatible transport module
provider-specific adapters:
- OpenAI
- OpenRouter
- vLLM
- potentially other compatible backends

This reduces duplication and makes vLLM support cleaner.

Concrete implementation plan

Step 1: extend config enum

Add Vllm to LlmConfig.
Add constructor/helpers in Rust and Python bindings.
Add docs/examples for local and remote vLLM endpoints.

Step 2: add provider module

Create core/src/llm/vllm.rs.
Implement LlmProviderTrait.
Support:
- complete
- stream if endpoint supports it
- function/tool calling only if the deployed vLLM model/server combination supports it well enough

Step 3: wire provider factory

Update core/src/llm/mod.rs and provider factory matching.
Register capability metadata correctly.

Step 4: Python bindings

Expose vllm in Python LlmConfig helpers/constructors.
Add examples in python/README.md.

Step 5: tests

Add unit tests for config and serialization.
Add mocked integration tests against OpenAI-compatible responses.
Add optional real integration tests guarded by env vars.

Step 6: docs

Add “Using GraphBit with vLLM” guide.
Explain supported features and limitations.
Document local deployment examples.

4) How to add batching / parallel efficiency for vLLM

Important distinction

There are three different optimizations people often mix together:

Concurrent workflow execution
- multiple nodes/tasks run at once
Concurrent provider requests
- multiple independent HTTP calls in flight
True batch inference
- many prompts grouped into a single provider/server-side batch

GraphBit seems to already support #1, and likely some degree of #2 through async execution. What seems missing is a first-class abstraction for #3.

MVP for batching

Add a provider-level optional batch API without forcing every provider to support it.

Suggested trait evolution

Add an optional method like:

async fn complete_many(&self, requests: Vec<LlmRequest>) -> GraphBitResult<Vec<LlmResponse>>

Default behavior for providers that do not support batching:

execute complete() concurrently with a bounded concurrency limit

Specialized behavior for vLLM:

map compatible requests into a batched call path when feasible

What counts as “compatible requests” for batching

Only batch requests together when they match on key dimensions such as:

same provider
same model
same sampling config where required
same tool/function-calling mode
same structured-output mode if applicable
same endpoint/base URL

Batch scheduler design

Add a small batching layer between executor and provider:

collect ready LLM tasks for a short window
group by batch key
flush when either:
- max batch size is reached, or
- max wait time elapses

Suggested controls:

max_batch_size
max_batch_wait_ms
max_in_flight_batches
fallback_to_parallel_single_requests

Where to integrate batching

Best place: near the LLM invocation boundary, not in the UI and not deep inside workflow graph logic.

Suggested layering:

workflow executor decides what can run in parallel
LLM dispatch layer decides what can be coalesced into batches
provider executes either batched or single requests

Implementation phases for batching

Phase 1: bounded parallelism abstraction

Even before true batching, add cleaner bounded concurrency for provider requests.

add provider pool / semaphore controls
expose per-provider concurrency config
collect request latency/queue metrics

This is valuable even if vLLM batching comes later.

Phase 2: batch-capable trait support

add complete_many
add executor/dispatcher support for grouped calls
default fallback implementation for non-batch providers

Phase 3: vLLM-specific batching

implement batching in vllm.rs
benchmark single vs concurrent vs batched modes
expose config knobs

Phase 4: observability

batch size distribution
queue wait time
throughput improvements
latency tradeoff dashboard/logs

Expected tradeoffs

Benefits:

higher throughput
better GPU utilization on vLLM backends
lower overhead per request at scale

Costs/risks:

slightly increased tail latency for small requests if batching waits too long
more complex error mapping from batch response to per-request result
harder support for heterogeneous tool-calling requests

Recommended default behavior

default to normal concurrent requests
enable auto-batching only for clearly batch-friendly workloads
expose knobs rather than making batching invisible

5) Suggested backlog / milestones

Milestone A: GUI MVP

Add gpui-app crate
Implement canvas + inspector + run panel
Save/load workflow documents
Show outputs and errors

Milestone B: vLLM compatibility

Add LlmConfig::Vllm
Implement core/src/llm/vllm.rs
Document OpenAI-compatible endpoint setup
Add tests and examples

Milestone C: concurrency + batching foundation

Add bounded concurrency controls for LLM dispatch
Add optional complete_many
Add metrics around queueing and throughput

Milestone D: v1.0 desktop and vLLM performance

Streaming UX
provider profiles and secrets handling
batch scheduler for vLLM
observability dashboards

6) Suggested next steps outside the two requested items

These are high-value next steps that complement both the GPUI app and vLLM support.

A. Define a stable workflow serialization format

This is the most important adjacent investment.

Why:

the GUI needs save/load
examples/templates become portable
replay/debugging becomes much easier
future JS/web/editor integrations benefit too

Suggested output:

versioned workflow schema
import/export support
migration strategy across schema versions

B. Add a dedicated app/service layer in Rust

Right now the core is powerful, but a desktop app will benefit from a cleaner facade for:

project loading/saving
run orchestration
execution event streaming
provider profile management

C. Strengthen execution event streaming

A UI will benefit from a structured stream of events such as:

workflow started
node scheduled
node started
token chunk received
tool call requested
node completed/failed
workflow completed

D. Add end-to-end examples focused on production workflows

Examples should cover:

parallel fan-out/fan-in
tool calling
memory
document processing
local model serving via Ollama/vLLM

E. Add a benchmark track specifically for local/self-hosted inference

The repo already has benchmarks. Extend them to compare:

OpenAI-compatible single requests
concurrent requests
vLLM batched mode
Ollama local inference

This will make the value of first-class vLLM support measurable.

Recommended immediate order of execution

Define a stable workflow document format.
Create the gpui-app workspace crate and ship a minimal workflow editor/runner MVP.
Add first-class Vllm config/provider support using the OpenAI-compatible path.
Add bounded LLM dispatch concurrency controls.
Add optional batch APIs and vLLM batching.
Expand the desktop app toward v1.0 with streaming, replay, and observability.

Uh oh!

FilesExpand file tree

plan.md

Latest commit

History