Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ target_link_libraries(strata_engine PUBLIC strata_kernels)
add_library(strata_models STATIC
src/models/deepseek/deepseek_executor.cpp
src/models/glm52/glm52_executor.cpp
src/models/glm53/glm53_executor.cpp
src/models/gemma4/gemma4_executor.cpp
src/models/kimi_k3/kimi_k3_executor.cpp
src/models/laguna/laguna_executor.cpp
Expand All @@ -129,6 +130,10 @@ add_library(strata_models STATIC
src/models/gemma4/gemma4_runtime.cpp
src/models/glm52/glm52_runtime.cpp
src/models/glm52/glm52_manifest.cpp
src/models/glm53/glm53_manifest.cpp
src/models/glm53/glm53_sequence.cpp
src/models/glm53/glm53_checkpoint.cpp
src/models/glm53/glm53_runtime.cpp
src/models/kimi_k3/kimi_k3_ops.cpp
src/models/kimi_k3/kimi_k3_manifest.cpp
src/models/kimi_k3/kimi_k3_checkpoint.cpp
Expand Down Expand Up @@ -367,6 +372,7 @@ if(BUILD_TESTING)
tests/test_deepseek_expert_residency.cpp
tests/test_fp4_decode.cpp
tests/test_glm52_ops.cpp
tests/test_glm53_manifest.cpp
tests/test_gemma4_ops.cpp
tests/test_gemma4_checkpoint.cpp
tests/test_gemma4_image.cpp
Expand Down
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ make check # build and run the test suite
./build/strata-chat --model /path/to/checkpoint --model-type gemma4
```

`--model-type` is one of `gemma4`, `deepseek`, `glm`, `laguna`, `inkling`,
`kimi-k3`. Add `--devices 0,1` to pin GPUs, `--admission-only --json` to print
`--model-type` is one of `gemma4`, `deepseek`, `glm`, `glm53`, `laguna`,
`inkling`, `kimi-k3`. Add `--devices 0,1` to pin GPUs, `--admission-only --json` to print
the placement plan without loading anything.

There is also an OpenAI-compatible server (`strata-server`).
Expand All @@ -51,6 +51,7 @@ There is also an OpenAI-compatible server (`strata-server`).
| **Laguna S 2.1** | 48 layers 1:3 global/sliding, 256 experts + 1 shared, top-10 | NVFP4 or MXFP4 experts, BF16 elsewhere | Spine in VRAM; experts stream from RAM |
| **Inkling Small** | 42 layers, 256 experts + 2 sinks, top-6, no rotary | NVFP4 or MXFP4 experts, BF16 elsewhere | Experts stream from RAM |
| **GLM-5.2** | 78 layers, 256 experts, top-8 | INT4 group-128, W4A16 | Exceeds combined memory; I/O-dependent |
| **GLM-5.3-Flash** | 45 layers, 3 KDA : 1 sparse MLA, 288 experts + 1 shared, top-8 | FP8 E4M3 block-128 with F32 inverse scales | Text-only; streams checkpoint modules, exact through 2,048 tokens |
| **Kimi-K3** | 93 layers, 3 KDA : 1 gated MLA, 896 experts, top-16 | MXFP4 experts, BF16 elsewhere | 1.45 TB; I/O-dependent, 38.6 s/step. Vision not implemented |

Each runs its declared semantics as-is — hybrid compressed attention and
Expand Down Expand Up @@ -135,7 +136,7 @@ record those as negatives with their measurements; 0165 records the positive.
| [`docs/server.md`](docs/server.md) | the OpenAI-compatible HTTP API |
| [`docs/models/`](docs/models/) | copy-paste build, chat, server, and measured-speed runbooks by model |
| [`docs/current-architecture.md`](docs/current-architecture.md) | how the code is organised and what is enforced |
| [`docs/model-bringup-guide.md`](docs/model-bringup-guide.md) | adding a seventh model |
| [`docs/model-bringup-guide.md`](docs/model-bringup-guide.md) | adding another model |
| [`docs/architecture.md`](docs/architecture.md) | the target scheduler design, not yet built |
| [`docs/README.md`](docs/README.md) | product documentation index |
| [`CONTRIBUTING.md`](CONTRIBUTING.md) | repository layout, architecture, and change hygiene rules |
Expand Down
2 changes: 1 addition & 1 deletion apps/strata_chat.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ R"(strata-chat -- interactive chat against a Strata runtime

required:
--model DIR checkpoint directory
--model-type TYPE gemma4 | deepseek | glm | laguna | inkling | kimi-k3
--model-type TYPE gemma4 | deepseek | glm | glm53 | laguna | inkling | kimi-k3

session:
--prompt TEXT answer TEXT and exit instead of prompting
Expand Down
64 changes: 56 additions & 8 deletions apps/strata_server.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,11 @@
#include <cerrno>
#include <algorithm>
#include <array>
#include <atomic>
#include <charconv>
#include <chrono>
#include <csignal>
#include <condition_variable>
#include <cstdint>
#include <cstdlib>
#include <cstring>
Expand Down Expand Up @@ -86,7 +88,7 @@ void stop_server(int) {
void usage() {
std::cerr
<< "usage: strata-server --model DIR --model-type "
"gemma4|deepseek|glm|laguna|inkling|kimi-k3\n"
"gemma4|deepseek|glm|glm53|laguna|inkling|kimi-k3\n"
<< " [--model-id ID] [--host ADDRESS] [--port N]\n"
<< " [--context-size N] [--max-new N]\n"
<< " [--devices 0,1,2] [--vram-fraction F]\n"
Expand Down Expand Up @@ -867,7 +869,7 @@ class ApiServer {
const Options& options_;
strata::RuntimeSession& runtime_;
strata::ModelTokenizer tokenizer_;
std::uint64_t request_id_{};
std::atomic<std::uint64_t> request_id_{};
};

enum class RouterModelStatus : std::uint8_t {
Expand Down Expand Up @@ -1502,23 +1504,69 @@ int main(int argc, char** argv) {
std::signal(SIGPIPE, SIG_IGN);
std::cerr << "[ready] http://" << options.host << ':' << options.port << "\n";
ApiServer server(options, runtime, std::move(tokenizer.value));
const bool concurrent_requests =
registration->model == strata::RuntimeModel::Glm53;
// The GLM scheduler multiplexes generation iterations, so its HTTP front
// end must not serialize independently arriving streams. Bound detached
// connection workers from the CPUs the host actually exposes instead of
// accumulating one joinable thread object for the lifetime of the server.
const auto discovered_cpus = std::max(1U, std::thread::hardware_concurrency());
const auto maximum_clients = std::min<std::uint32_t>(128U,
std::max<std::uint32_t>(4U, discovered_cpus * 2U));
std::atomic<std::uint32_t> active_clients{};
std::mutex clients_mutex;
std::condition_variable clients_drained;
while (stop_requested == 0) {
const int client = accept4(listening_socket, nullptr, nullptr, SOCK_CLOEXEC);
if (client < 0) {
if (errno == EINTR || stop_requested != 0) continue;
std::cerr << "warning: accept failed: " << std::strerror(errno) << '\n';
continue;
}
HttpRequest request;
std::string error;
if (!read_request(client, request, error)) {
send_error(client, 400, "Bad Request", error, "invalid_request_error");
const auto serve = [&server, &active_clients, &clients_drained](
int socket) {
HttpRequest request;
std::string error;
if (!read_request(socket, request, error)) {
send_error(socket, 400, "Bad Request", error,
"invalid_request_error");
} else {
server.handle(socket, request);
}
close(socket);
active_clients.fetch_sub(1U, std::memory_order_acq_rel);
clients_drained.notify_all();
};
if (concurrent_requests) {
if (active_clients.load(std::memory_order_acquire) >=
maximum_clients) {
send_error(client, 503, "Service Unavailable",
"GLM-5.3 request admission is full",
"server_overloaded");
close(client);
continue;
}
active_clients.fetch_add(1U, std::memory_order_acq_rel);
try {
std::thread(serve, client).detach();
} catch (const std::system_error& error) {
active_clients.fetch_sub(1U, std::memory_order_acq_rel);
send_error(client, 503, "Service Unavailable",
std::string("cannot start request worker: ") +
error.what(),
"server_overloaded");
close(client);
}
} else {
server.handle(client, request);
active_clients.fetch_add(1U, std::memory_order_acq_rel);
serve(client);
}
close(client);
}
listening_socket = -1;
std::unique_lock clients_lock(clients_mutex);
clients_drained.wait(clients_lock, [&] {
return active_clients.load(std::memory_order_acquire) == 0U;
});
std::cerr << "[shutdown] stopped cleanly\n";
return 0;
}
9 changes: 6 additions & 3 deletions apps/strata_tokenize.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,16 @@ int main(int argc, char** argv) {
model_type = argv[++index];
} else {
std::cerr << "usage: strata-tokenize --tokenizer FILE --model-type "
"gemma4|glm|deepseek|laguna|raw --prompt TEXT\n";
"gemma4|glm|glm53|deepseek|laguna|raw --prompt TEXT\n";
return 2;
}
}
if (tokenizer_path.empty() || prompt.empty() ||
(model_type != "gemma4" && model_type != "glm" && model_type != "deepseek" &&
(model_type != "gemma4" && model_type != "glm" && model_type != "glm53" &&
model_type != "deepseek" &&
model_type != "laguna" && model_type != "raw")) {
std::cerr << "usage: strata-tokenize --tokenizer FILE --model-type "
"gemma4|glm|deepseek|laguna|raw --prompt TEXT\n";
"gemma4|glm|glm53|deepseek|laguna|raw --prompt TEXT\n";
return 2;
}
const auto loaded = strata::ModelTokenizer::load(tokenizer_path);
Expand All @@ -36,6 +37,8 @@ int main(int argc, char** argv) {
std::string rendered;
if (model_type == "glm") {
rendered = strata::render_glm52_user_prompt(prompt);
} else if (model_type == "glm53") {
rendered = strata::render_glm53_user_prompt(prompt);
} else if (model_type == "deepseek") {
rendered = strata::render_deepseek_v4_user_prompt(prompt);
} else if (model_type == "gemma4") {
Expand Down
4 changes: 3 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ live in [`../CONTRIBUTING.md`](../CONTRIBUTING.md).
- `kimi-k3-runtime.md` — Kimi-K3's pinned contract, cost model, NVMe write
constraint, chat format, and a per-gate status table that says which gates
have been measured and which have not.
- `models/glm53.md` — GLM-5.3-Flash's text-only contract, exact context bound,
and chat/server commands.
- Gemma 4's pinned checkpoint, tokenizer, text/vision graph, and public runtime
contract are described in the root README and `current-architecture.md`.
- `../kernels/cuda/README.md` — native CUDA and non-CUDA stub behavior.
Expand All @@ -39,7 +41,7 @@ a previous “not implemented” statement false.
- `cli.md` — every command-line flag and the dry-run placement planner.
- `sampling.md` — sampler options, their exact semantics, reproducibility.
- `server.md` — `strata-server` and the OpenAI-compatible API.
- `model-bringup-guide.md` — **how to add a seventh model.** The procedure, the
- `model-bringup-guide.md` — **how to add another model.** The procedure, the
five shared files, and the rules that are not obvious. Start here for a new
architecture.

Expand Down
14 changes: 8 additions & 6 deletions docs/current-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ strata_device the CUDA backend, or its error-returning stub
strata_kernels CPU reference kernels (Q4, INT4 group-128, attention)
strata_engine placement solver and cache, residency, sampling, chat and
session support, the model registry
strata_models six models, one directory each, plus the shared checkpoint
strata_models seven models, one directory each, plus the shared checkpoint
reader, tokenizer and placement inventories
strata_app RuntimeSession and the OpenAI protocol
```
Expand All @@ -26,7 +26,7 @@ strata_app RuntimeSession and the OpenAI protocol
`strata_models` under `--whole-archive`; see "Model registration" below.

`src/` mirrors this: `src/platform/`, `src/engine/`, `src/app/`, and
`src/models/{deepseek,glm52,gemma4,kimi_k3,laguna,inkling,common}/`. Headers mirror
`src/models/{deepseek,glm52,glm53,gemma4,kimi_k3,laguna,inkling,common}/`. Headers mirror
the same ownership under `include/strata/{platform,device,kernels,engine,app}`
and `include/strata/models/<model>/`. They are private application interfaces:
their headers and libraries are not installed, and no source or ABI
Expand Down Expand Up @@ -80,7 +80,7 @@ string-to-enum chain.
`--whole-archive` on `strata_models` is load-bearing: a static library drops
any member nothing references, and nothing references a self-registering
translation unit by definition. `strata_app` links ahead of it because `ld`
resolves in one pass. A test asserts all six models are registered, because
resolves in one pass. A test asserts all seven models are registered, because
without the flag `find_model` returns null for every model and the rest of the
suite still passes.

Expand All @@ -106,7 +106,9 @@ prefill then token-at-a-time decode. Gemma 4 performs bounded prefill with
whole vision blocks, hybrid local/global attention, and a BF16
local-ring/global-full KV cache. DeepSeek performs bounded layer-major prefill
pages with a multi-row router projection and exact row-ordered causal
transitions. Kimi-K3 batches over a token span throughout. Inkling has an
transitions. GLM-5.3 runs token-at-a-time hybrid KDA and sparse MLA, with its
exact text context capped at the sparse top-k of 2,048. Kimi-K3 batches over a
token span throughout. Inkling has an
opt-in paged prefill (`prefill_page_tokens`) that runs attention and its four
short convolutions row-serial — both carry row-ordered state — and batches the
routed MoE expert-major between them; it is bit-identical to its
Expand Down Expand Up @@ -148,7 +150,7 @@ not.
present) runs Gemma 4 against `tests/fixtures/gemma4/layer-hash-trace.json` —
a per-layer hidden-state hash plus per-operation hashes over a fixed prompt.
The types are model-neutral (`include/strata/platform/diagnostics.hpp`); DeepSeek emits
the same records, and the remaining four models do not yet.
the same records, and the remaining five models do not yet.

Its limits, stated because a gate nobody understands is worse than none: it
covers **prefill only** — once the device KV path engages, a whole device's
Expand All @@ -168,7 +170,7 @@ hardware probe), and a **plan cache** keyed by checkpoint identity, GPU
identity, context size, device list, VRAM fraction and flags.

`plan_model_placement` in `strata_engine` is a thin dispatcher through a
registered `PlacementPlanner`; the implementation that opens six different
registered `PlacementPlanner`; the implementation that opens seven different
checkpoints lives in `strata_models` and installs itself at static-init. That
inversion is why `strata_engine` names no model symbol.

Expand Down
2 changes: 1 addition & 1 deletion docs/model-bringup-guide.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Adding a model

A procedure derived from the six existing adapters. Follow it in order; the
A procedure derived from the seven existing adapters. Follow it in order; the
ordering is part of the correctness contract.

## What it actually costs
Expand Down
1 change: 1 addition & 0 deletions docs/models/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,5 @@ transfers to a different context length or machine.
| Inkling Small | [Build, chat, serve, and benchmark](inkling.md) | 9.072 tok/s fresh / 28.010 tok/s same-route warm |
| DeepSeek V4 | [Build, serve, and benchmark](deepseek.md) | 9.171 tok/s with the routed-expert tier, 8.571 without |
| GLM-5.2 | [Build, preflight, and run](glm52.md) | I/O-dependent on the reference workstation |
| GLM-5.3-Flash | [Build, preflight, chat, and serve](glm53.md) | I/O-dependent; exact text context currently capped at 2,048 tokens |
| Kimi-K3 | [Build, preflight, and run](kimi-k3.md) | ~0.02 tok/s; SATA-bound on the reference workstation |
66 changes: 66 additions & 0 deletions docs/models/glm53.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# GLM-5.3-Flash: text runbook

GLM-5.3-Flash is a 45-layer, 288-routed-expert model with four mHC streams,
three Kimi Delta Attention layers for every sparse MLA layer, eight selected
experts plus one shared expert, and block-128 FP8 E4M3 weights with F32 inverse
scales. Strata consumes the checkpoint as published and streams modules from
storage; it does not requantize the model or reduce its expert count or top-k.

This adapter currently supports text only. Image or video content is rejected
before generation. The admitted context is capped at 2,048 tokens. At that
length the model's sparse index `top_k` is 2,048, so every causally visible key
is selected and dense causal MLA is exactly equivalent to running the sparse
indexer. Longer contexts fail admission until the k-pool indexer is implemented;
there is no silent dense or truncated fallback.

## Build and preflight

```bash
cmake -S . -B build-release -DCMAKE_BUILD_TYPE=Release
cmake --build build-release --parallel --target strata-chat strata-server

./build-release/strata-chat \
--model models/glm53f --model-type glm53 \
--devices 0,1,2 --context-size 2048 --max-new 256 \
--vram-fraction 0.85 --dry-run
```

The pinned release has 62 safetensors shards, 76,108 indexed tensors, and
328,326,771,576 indexed payload bytes. Admission validates those extents, the
hybrid layer schedule, text tensor roles, representative shapes and dtypes,
and the FP8 block geometry before generation.

## Chat and server

```bash
./build-release/strata-chat \
--model models/glm53f --model-type glm53 \
--devices 0,1,2 --context-size 2048 --max-new 256

./build-release/strata-server \
--model models/glm53f --model-type glm53 --model-id glm53f \
--devices 0,1,2 --context-size 2048 --max-new 256 --port 8080
```

The server exposes the same text runtime through its OpenAI-compatible chat
completion endpoint. Do not send image content: multimodal support is outside
this adapter's current contract.

## Current operating point

The runtime discovers CPU width, free VRAM, peer topology and storage at
startup. On hosts where the routed checkpoint is much larger than the usable
CUDA cache, it maps canonical FP8 experts once and executes them directly from
host memory while keeping the non-expert spine, fused KDA state and mHC
transitions on CUDA. Prompt pages group rows by expert so an expert is traversed
once for every row that selected it; decode does not reread routed experts from
the checkpoint through Strata's explicit I/O path per token (the OS still owns
page-cache residency for mapped payloads). Systems with a high-speed two-GPU
peer fabric additionally admit the full TP2 route; PCIe systems use a
contiguous pipeline schedule.

MTP drafting and verification are implemented but opt in because acceptance is
workload-dependent. Set `STRATA_GLM53_MTP=1` for an acceptance campaign. The
exact production latency path leaves it disabled. Resident absorbed MLA remains
an explicit experimental route (`STRATA_GLM53_RESIDENT_MLA=1`) until its
BF16-boundary equivalence gate is closed; it is never selected silently.
2 changes: 1 addition & 1 deletion include/strata/app/runtime.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
// RuntimeModel, RuntimeConfig, GenerationResult and friends moved to
// model_executor.hpp in Phase 4 and are re-exported here, so every existing
// `#include "strata/app/runtime.hpp"` keeps working unchanged. They live a tier
// lower because the six models implement ModelExecutor against them: had they
// lower because the seven models implement ModelExecutor against them: had they
// stayed here, every model would depend upward on the application tier, which
// is exactly the inversion check-symbols exists to catch.

Expand Down
Loading
Loading