ojas (vital energy, the essence training spends) is a deterministic, training-first universal framework in Rust: tensors, autograd, optimizers, training loops, data, checkpoints and IO, with a CPU reference backend and CUDA, HIP, wgpu and Metal plugging in as extensions. It is designed as a safe, modern alternative to PyTorch and TensorFlow for systems research and LLM training, and does only light inference (sampling and eval).
ojas serves as the central framework within a modular compute ecosystem:
- tessl: Dedicated Apple Silicon / Metal 4 acceleration extension and Mac inference engine (MPP TensorOps
matmul2d, unified memory allocator, ICB decode, MLX quantization, EmbeddingGemma 2 multimodal towers, and canonical Metal compute kernels forojas-metal). - sparsl: Dedicated Sparse Computing extension (CSR/CSC sparse matrix representations, deterministic SpMV, affine prefix scans, event-driven simulation, and structured sparsity).
Note
Modularity Rationale: ojas, tessl, and sparsl are intentionally developed across separate checkouts and crates to guarantee:
- Decoupled Compile Cycles: The universal training core is built in CI on Linux and macOS (
.github/workflows/test.ymlhas no Windows job, so no CI job builds Windows; the only local check on record is an x86_64 Windowscargo checkand clippy ofojas-devicealone,docs/adaptive-resources.md:50,:124; a CI compile check is planned, ft-3902a4b3) without dragging in Xcode/Metal toolchains or heavy sparse proptest/mutation suites. - Isolated Hardware Requirements: Apple Silicon M-series specifics live in tessl; sparse event-driven kernels live in sparsl; universal CPU/wgpu/CUDA training lives in ojas.
- Independent Profiling & Micro-Benchmarking: GEMM tile sweeps and register tuning (tessl) vs sparsity crossover thresholds (sparsl) vs end-to-end training step throughput (ojas) iterate on their own benchmark harnesses.
- Independent Release Cadence & Separate Checkouts: Downstream applications import only what they need without bloat (see
docs/modular-extensions.mdanddocs/architecture.md).
ojas pairs a compile-time safe Rust engine with an in-process Go API via gusset, zero-copy tensor views, reverse-mode automatic differentiation, and strict memory budgeting.
- Bit-Identical Reproducibility: Single-threaded-order, index-increasing accumulation under
Numerics::Exactguarantees bit-identical results across thread counts, worker scheduling, and machines. - Zero Silent Fallbacks: Selecting a GPU backend (Apple Silicon Metal, wgpu, CUDA) that cannot be initialized returns an explicit error immediately. It will never silently downgrade to CPU execution.
- Explicit Budget Governance: All allocations request permits from a hierarchical
Budget. Over-budget allocations immediately returnOjasError::CapacityExceededwithout dynamic memory clamping, hidden page swapping, or uncatchable OOMs. - Zero-IPC Multi-Language Bindings: High-performance Go and C-ABI integrations via
gussetandojas-capilink directly in-process with panic isolation boundaries, eliminating Python runtime overhead and serialization penalties. - Hardware Acceleration: Native Apple Silicon Metal acceleration via
tessland custom Metal kernels, portable cross-platformwgpucompute shaders, and optimized CPU kernels (ARM64 NEON, x86_64 AVX2, Apple Accelerate). - Dual Optimizers & Target Model: Out-of-the-box support for the nanolab GPT architecture with Muon NS5 (Newton-Schulz orthogonalization) and AdamW, sliding-window KV-cache autoregressive inference, top-k row sampling, streaming dataset loaders, and fast BPE tokenization.
What runs today:
- Coverage differs by backend;
docs/op-coverage.mdhas one row perBackendmethod (107, read at c172510). The CPU overrides 97 of the 107; its other ten are trait defaults: host behaviour forupload,download,cast_bf16,to_bf16,to_f32,argmax_rowsandtopk_rows, a constant forsync,autocast_region(which refusesBf16), andcausal_sdpa_backward_recompute, which composes the causal SDPA forward and backward. Metal overrides 101:relu_backward,slice_backward,interpolate_bilinear_backwardandgroup_norm_backwardare trait defaults that refuse, andautocast_regionandcausal_sdpa_backward_recomputeare defaults on every backend. wgpu runs the nanolab GPT op set and refuses the Qwen3.5, encoder and spatial ops. On wgpu, tiled GEMM (gemm.wgsl) and FlashAttention-2 (attention.wgsl) promote per-lane accumulators to explicitvec4<f32>/scalar registers with(d4, c)loop inversion and unbranched full-tile fast paths; on Metal, RMSNorm, LayerNorm, RoPE, Per-Head Gate, and FlashAttention foldST_IN/ST_OUTfiniteness checks in-kernel alongside vectorizedfloat4fault/accumulate kernels (ojas_check_finite_v4,ojas_add_inplace_v4). - General attention (
attention_forward,attention_forward_saving,attention_backward): bidirectional self-attention, cross-attention ($T_q \ne T_k$ ), broadcastable additive score bias with dedicated gradient pass, and per-batch key-length masks. On Metal: routed totessl encoder_attnat D=32/64 for self-attention ($T_q = T_k$ ), native decoupled$T_q \times T_k$ TensorOps flash cross-attention (ojas_attn_fwd_d32andojas_attn_fwd_d64, without$O(m^2)$ pad-q zero-padding), tile-register additive logit bias (boxRPBvia buffers13..18), split-K tiled attention (ojas_xattn_part_dD/combine) for short queries or D=16, and scalar fallback. - Spatial operators:
conv2d(arbitrary stride, pad, dilation, groups, depthwise; on Metal, 1x1 stride-1 unpadded conv skips im2col for direct NN GEMM with >2× speedup and selects the bf16 GEMM lane on bf16-exact operands),conv_transpose2d(kernel == stride),max_pool2d,interpolate_bilinear(bothalign_cornersmodes), window partition padding/cropping, and interleaved RoPE with 2-D axial tables. Plannedpermuteexecutes row copies for contiguous 16-float spans and 2-D transpose tiles with fused finite tests. - Vision & encoder forward and backward ops: LayerNorm forward and backward (two-pass f64 sums for mean and variance on CPU), channel LayerNorm forward and backward,
group_norm(exact f64 two-pass on CPU, 3-pass GPU kernel on Metal with large-mean residual correction), erf GELU forward and backward (gelu_erf_backward_exact),linear_biasbackward (evaluates$dX, dW, db$ simultaneously with frozen-weight$dW$ bypass),broadcast_addbackward,broadcast_mulbackward,softmax_backward, ReLU,concat,slice(whole-input finite check), andfill(torch.full equivalent with U32Ids::Belowbound). - Cross-entropy: fused and linear cross-entropy on CPU, Metal and wgpu (and CUDA's K10 kernel) take the target-excluded form when standard target gradient
$p_t - 1$ rounds to 0 (log1p_exact); Fast-tier vectorized exponentials underNumerics::Fast. - Parity against real SAM 3.1 safetensors weights, PyTorch 2.14 activation fixtures via
ojas-oracle::sam_fixtures, and 31 PyTorch 2.15 vision/encoder backward golden cases (ojas-oracle::fixtures::encoder_bwd). - The autograd
Tapegradchecks a multi-head attention block against double-precision (f64) finite differences. -
ojas-modeldefines the nanolab GPT once (spec,state_dictnames, order-independent init, the block) and runs it two ways: recorded on theTapefor training, eagerly throughEvalfor decoding. ItsTrainertakes Muon + AdamW steps on anyBackendand saves and resumes a checkpoint directory. - The Go API drives that model in process:
LoadModelorNewModel(DeviceCPU,DeviceCPUParallel,DeviceCPUAuto,DeviceMetal,DeviceWgpu),OpenTrainer,TrainStep,SaveCheckpointandResume, a high-throughput GPT-2 BPE tokenizer, and sampledGenerateIDs. Metal and wgpu fail closed when no device opens. -
ojas-inferdecodes a nanolab-architecture model withCpuGpton the CPU and withDeviceDecoderon anyBackend(ojas-infer/src/device.rs:42;ojas-infer/README.mdruns it on wgpu and Metal), greedy or sampled (top-k rows, temperature, top-p), with a sliding-window KV cache ring. -
ojas-datastreams token binary formats (u16/u32), resumable batch sampling with counter-based RNG, and provides an optimized GPT-2 BPE tokenizer with byte and lossy decoding, pre-token caching, and output capping. -
ojas-cudaprovides a host-side CUDA runtime with single-budget memory accounting, typed initialization and device-loss errors, bounded waits, and watchdog protection; itsCudaBackendrefuses every compute op (the earlier affine test kernel was retired 2026-10-14,docs/backends.md:74). -
ojas-qwen35(macOS only) runs a Qwen3.5 whole training step through tessl's Metal kernels. It is a provider, not aBackend. - Metal
bf16accumulation (MetalBackend::with_bf16_accumulation(true)/set_bf16_accumulation(true)): step-by-step nearest-evenbf16accumulation on the Metal GPU across linear/GEMM (ojas_gemm_bf16_accum), spatial and hybrid convolutions (conv2dim2col/pointwise/depthwise,conv_transpose2d, andcausal_conv1d_siluforward and backward), attention (causal_sdpa, flash/split-K/scalar general attention,cached_attention, encoder attention backward), per-head sigmoid gate, all normalization forwards and backwards (rms_norm,layer_norm,channel_layer_norm,group_norm,gated_rms_norm), softmax and cross-entropy (softmax_forward/_backward,cross_entropy_mean,linear_cross_entropy_mean), sequence mean and L2 normalization (seq_mean_forward,l2_normalize_forward/_backward), pooling and interpolation (max_pool2d_backward,interpolate_bilinear_forward),embedding_backward, broadcast/bias/residual/value-residual/GDN-decay reductions and sums,accumulate_grad, andmuon_ns5_step, while keeping defaultf32accumulation when off.
Model coverage by backend. A cell says "yes" only where a test in the repository constructs that backend and runs that model (read at bfc24e5; none of those tests was re-run for this table):
| Model | CPU | Metal | wgpu | CUDA |
|---|---|---|---|---|
| nanolab GPT | yes: oracle_forward_loss_and_logits (ojas-model/tests/oracle_parity.rs:364, on cpu_model) |
yes: metal_trainer_matches_torch_on_the_tiny_fixtures (oracle_parity.rs:705) |
yes: wgpu_trainer_matches_torch_on_the_tiny_fixtures (oracle_parity.rs:720) |
no test |
| Qwen3.5 | yes: six_straight_steps_equal_three_then_a_checkpoint_a_fresh_provider_and_three_more (ojas-model/tests/qwen35_step_provider.rs:463, a TapeStepProvider<CpuBackend>) |
yes: on_metal_the_norm_and_the_update_read_each_gradient_once (qwen35_step_provider.rs:1579) and the_metal_tape_matches_transformers_and_the_cpu_tape (ojas-model/tests/qwen35_metal_tape.rs:50, which returns early and passes when no Metal device opens, :51) |
no test; wgpu refuses the GDN, conv1d and gated-norm ops (trait defaults) | no test; CudaBackend refuses every compute op |
toy image MLP (ImageMlpProvider, ojas-model/src/image_mlp.rs:1-7) |
yes: the_job_contract_is_enforced_literally (ojas-model/tests/image_mlp_provider.rs:307, on fresh(), a CPU provider) |
yes: gpu_resume_is_bit_identical_and_matches_the_cpu_within_the_fast_tolerance (image_mlp_provider.rs:812, #[ignore], needs a Metal device) |
no test | no test |
| CNN, ViT | not yet (vision cards ft-4f11ce5a, ft-96023b34, ft-7f4f2d87) | not yet | not yet | not yet |
What does not run yet:
- bf16 accumulation on CPU and wgpu (Metal GPU supports full bf16 accumulation via
MetalBackend::with_bf16_accumulation(true); CPU and wgpu sum inf32, with bf16 as an optional autocast region and asBf16operands to linear and attention). - A CUDA
Backendthat computes:CudaBackendexists (ojas-cuda/src/backend.rs:84) and every compute op returnsUnsupported. A HIPBackend:ojas-hipis a copy probe (copy_roundtrip,ojas-hip/src/lib.rs:84). - Everything else on the ranked gap list below.
The ranked list of gaps, and what has closed since it was written, is in docs/pytorch-parity-plan.md.
Against PyTorch MPS on the GPU (docs/bench-gpu-vs-torch.md, measured under heavy external GPU load, so only direction is verified):
- ojas is slower on most rows. In round 5 (the latest paired round; torch 2.13.0 per
bench/results/2026-10-02-r5/env_torch.json), a nanolab block forward + backward ran at 1.07× [0.96–1.16] torch's speed on Metal and 0.50× [0.49–0.56] on wgpu (bench/results/2026-10-02-r5/summary.md:69,122, both flagged "noisy - not quoted"; see Round 5 indocs/bench-gpu-vs-torch.md). Round 3 read 0.92× (Metal, mixed) and 0.41× (wgpu), also flagged noisy (bench/results/2026-10-01-r3b/summary.md:69,122), and round 1 read 0.37× and 0.18×. The GPU read 100% busy in rounds 3 and 5, so only the direction is verified. - It led on Metal attention backward at head dim 64 only (mixed at 128,
docs/bench-gpu-vs-torch.md:1597-1610) and on Metal AdamW in round 3 (1.55×, noisy; round 5's 1.53× is also flagged noisy,bench/results/2026-10-02-r5/summary.md:61), so those leads are direction only. Itsclip_grad_normlead is against torch 2.13–2.14 only, whose MPS norm ran each full reduction on one threadgroup. PyTorch fixed that kernel for 2.15. On the 2.15 nightly, torch's row takes 8.45 ms (bench/results/2026-10-05-torch215-clip); ojas's Metal path, now tessl'ssq_norm, takes 7.45 ms median (7.07 ms for the multi-tensor kernel it replaced;bench/results/2026-10-17-clip-sqnorm-ab). The two were not run together, so only the direction is verified. The 19.0 ms quoted earlier measured the path from before 0e3254f. - CUDA's
CudaBackendimplementsBackendbehind a feature (upload, download and sync; every compute op returnsUnsupported), and HIP is a copy probe that does not.
Status: Kernel set, backends, nanolab model, trainer, adaptive resource planning and Go API built; remaining gaps in docs/pytorch-parity-plan.md
Host Target: Apple M5 Pro (Metal 4), macOS 27 (Darwin 27.0.0 arm64)
Interactive App: https://ojas.vbcr.dev (Alternate: https://bharath.vbcr.dev/ojas)
License: MIT OR Apache-2.0
Verification: the latest full workspace run: 2005 passed, 0 failed, 41 ignored, 219 suites (cargo test --workspace --release -- --test-threads=2 at 51825fd, 2026-11-01, macOS, Metal and wgpu on the real GPU, no SKIP lines). This run is stale: a later full run (2026-10-13 to 10-14, merged main after bc498c4) recorded 231 suites, 1991 passed, 1 failed (a_real_oversize_gdn_workspace_is_capacity_exceeded), 50 ignored (docs/reports/BUILD_REPORT-w11.md:28); it is not yet in test-runs.json. Per-crate counts are from the run of 2026-11-01 (2489 passed across the crates below, Go 41 passed). Every number comes from docs/data/test-runs.json.
Explore the Interactive Web Experience: The ojas interactive documentation has live labs for each guarantee: bit-identical reductions across thread counts, memory budget refusal, no silent device fallback, the Go panic firewall, and checked shape arithmetic, plus the CPU versus PyTorch timings.
Mainstream frameworks like PyTorch and TensorFlow carry decades of legacy baggage: hidden global state, silent CPU fallbacks that mask hardware misconfigurations, nondeterministic multithreaded reductions, memory fragmentation, and cumbersome multi-language bindings that rely on heavyweight Python runtimes.
ojas is engineered from first principles with radical architectural guarantees:
flowchart TD
subgraph PyTorchTF["Legacy Frameworks (PyTorch / TensorFlow)"]
L1["Implicit Global State & Seed Drift"]
L2["Silent CPU Fallback on Missing GPU\n(Hides deployment bottlenecks)"]
L3["Dynamic Memory Clamping & OOM Panics"]
L4["Heavyweight Python-Centric Runtime Stack"]
L5["Integer Overflow Hazards in Long Training Steps"]
L6["Complex C++ Codebase with Opaque Abstractions"]
end
subgraph ojasPrinciples["ojas Architectural Guarantees"]
O1["Bit-Identical Determinism under Numerics::Exact (opt-in)"]
O2["Zero Silent Fallbacks: Explicit Hardware Refusal\n(Fail fast, fail loud)"]
O3["Explicit Budget Governance: try_reserve() Never Clamps"]
O4["Pure Rust Core with Direct In-Process Go & C-ABI Bindings"]
O5["overflow-checks = true in Release\n(step counters use checked_add; some ids wrap or saturate)"]
O6["#![forbid(unsafe_code)] on the library crates\n(not cuda/hip features, ojas-gusset-engine, or metal/build.rs)"]
end
L1 -.->|Solved By| O1
L2 -.->|Solved By| O2
L3 -.->|Solved By| O3
L4 -.->|Solved By| O4
L5 -.->|Solved By| O5
L6 -.->|Solved By| O6
Important
Deterministic Bit-Identicality: CpuBackend defaults to Numerics::Fast: FMA, and on macOS Accelerate for GEMMs of at least 2^13 multiply-adds (ojas_cpu::FAST_WHOLE_CALL_MACS), as PyTorch's CPU linear does. Opt into Numerics::Exact with .with_numerics(Numerics::Exact) for single-threaded-order, index-increasing accumulation. Under Exact, identical seeds give identical bits across thread counts and machines. The C/Go API has no numerics selector, so C and Go callers get Fast.
Zero Silent Fallback: Selecting a GPU backend (Metal, wgpu, CUDA, HIP) that cannot be initialized returns an explicit error immediately. It will never silently downgrade to CPU execution.
ojas is structured as a modular stack of focused, single-responsibility crates:
flowchart TD
subgraph Applications["User Applications & Service APIs"]
GoApp["Go Services & Applications\n(github.com/bharathvbcr/ojas/go)"]
RustApp["Rust Binaries, Services & CLIs"]
ForeignApp["C / C++ / Embedded Host Applications"]
end
subgraph Interop["In-Process Interop & Foreign Function Interface"]
Gusset["ojas-gusset-engine\n(libgusset.a Worker Pool & Ticket Queue)"]
CAPI["ojas-capi\n(C-ABI Engine, Session Store, Panic Isolation Boundary)"]
end
subgraph HighLevel["Model, Autograd & Inference Layer"]
Infer["ojas-infer\n(Autoregressive Decoding, KV Cache, Logit Validation)"]
Autograd["ojas-autograd\n(Dynamic Reverse-Mode Tape & f64 Gradcheck)"]
Model["ojas-model\n(nanolab GPT written once: spec, init, Graph block, Trainer, checkpoint dir)"]
Qwen35["ojas-qwen35\n(Qwen3.5 whole-step provider over tessl; macOS Metal only; not a Backend)"]
end
subgraph CoreEngine["Tensor Substrate & System Types"]
Core["ojas-core\n(Tensor, Strides, Offset Slicing, Budget, DType, Errors)"]
Device["ojas-device\n(Device Kinds, System Profile, ResourcePlan; no router)"]
IO["ojas-io\n(Strict Safetensors Parser, Binary Checkpoint v1)"]
Data["ojas-data\n(Dataset Streaming, Token Binary Formats, Deterministic RNG)"]
Oracle["ojas-oracle\n(IEEE-754 f64 Numerical Reference Fixtures)"]
end
subgraph HardwareBackends["Pluggable Hardware Compute Backends"]
CPU["ojas-cpu\n(CpuBackend: Fast default, Accelerate on macOS; Exact packed GEMM opt-in)"]
SIMD["ojas-simd\n(NEON GEMM, AVX2, and Accelerate cblas_sgemm wrapper)"]
Metal["ojas-metal\n(MetalBackend, device-resident, via tessl)"]
WGPU["ojas-wgpu\n(WgpuBackend, device-resident WGSL)"]
Kernels["ojas-kernels\n(Shared WGSL/CUDA sources, workgroup geometry, parity harness)"]
CUDA["ojas-cuda\n(CudaBackend behind feature cuda; compute ops Unsupported)"]
HIP["ojas-hip\n(Copy probe; no HIP kernel; not a Backend)"]
end
subgraph ModularExtensions["Modular Compute Extensions (Dedicated Checkouts)"]
TesslExt["tessl (Metal 4 Extension & Mac Inference)\nMPP TensorOps, SharedBuf UMA, KV Caches, EmbeddingGemma 2"]
SparslExt["sparsl (Sparse Computing Extension)\nDeterministic SpMV, CSR/CSC, Affine Scans, Event Simulation"]
end
GoApp --> Gusset
Gusset --> CAPI
ForeignApp --> CAPI
RustApp --> HighLevel
CAPI --> HighLevel
Infer --> Model
Model --> Autograd
HighLevel --> CoreEngine
HighLevel --> HardwareBackends
HardwareBackends --> CoreEngine
CoreEngine --> HardwareBackends
Metal -->|Zero-Copy UMA & Metal 4 Kernels| TesslExt
CoreEngine -.->|Sparse Tensor & Linear Algebra| SparslExt
- Zero-Copy Views: Tensor representations track an explicit
byte_offset. Sub-views created via.narrow()adjust byte offsets without memory re-allocation or buffer copying. - Strict Memory Budgets: Memory consumption is governed by an explicit
Budget. When an allocation exceeds limits,try_reserve()returnsOjasError::CapacityExceededimmediately rather than swapping or resizing. - Type System:
DTypetagsF32,Bf16,F16andU32. Every backend accumulates inF32by default (token ids inU32), andMetalBackendsupports opt-in step-by-stepbf16accumulation across all GPU reductions and dot products viaMetalBackend::with_bf16_accumulation(true)/set_bf16_accumulation(true). The matmul-class ops (linear_*,causal_sdpa_*,attention_*, andconv2d_forward) also takeBf16operands on backends whosebf16_operands()is true (CPU, Metal and wgpu), widened intof32(orbf16on Metal whenbf16_accumulationis enabled); on the CPU and Metal so does the tied table (the embedding gather and backward, and the fused cross-entropy's weight), read in place without anf32copy of the table. Every other op refuses non-F32dtypes (docs/dtype-policy.md).ojas-iodecodes BF16/F16 safetensors tof32and encodes back with round-to-nearest-even. - Device Tensors:
Tensor::from_devicewraps a backend's buffer.to_hostcopies it back and is counted per budget tree byBudget::device_readbacks.device_buffer_mutrequires sole ownership of the storage. The defaultBackend::uploadreturnsUnsupportedfor a host tensor on a non-CPU backend; there is no CPU fallback.
flowchart LR
subgraph Storage["Physical Allocation Buffer (Arc<Vec<u8>>)"]
Pad["Leading Offset [0 .. byte_offset)"]
Active["Active Tensor Slices [byte_offset .. byte_offset + active_bytes)"]
Unused["Unused Capacity"]
end
subgraph View1["Parent Tensor [2, 16, 64]"]
V1["byte_offset = 0"]
end
subgraph View2["Sliced View: narrow(1, 4, 8)"]
V2["byte_offset = 1024 bytes (zero-copy)"]
end
View1 --> Active
View2 --> Active
Note
All tensor allocations must request permits from a Budget. If an allocation exceeds available capacity, try_reserve() returns Err(OjasError::CapacityExceeded). Infallible runtime calls (vec!, format!, thread::spawn) operate outside this budget and can still abort if physical host RAM is exhausted.
- Tape-Based Computation Graph: Dynamic tape recording variable nodes (
Var) and operational adjoints for forward and reverse sweeps. - Numerical Gradcheck Oracle: Analytical f32 gradients are verified against double-precision (
f64) central finite differences, ensuring mathematical soundness down to machine tolerances. - Device Tape Residency: On non-CPU backends (
MetalBackend,WgpuBackend),leafand input tensors are uploaded once. Reshape is a zero-copy view. The backward seed is uploaded, and non-root cross-entropy gradients are scaled on the device using rank-1 linears. Intermediate activations stay resident on GPU; the loss download is the only audited readback.
flowchart LR
subgraph ForwardTape["Forward Execution"]
x["Var x"] --> op1["Op: Linear(x, W)"]
W["Var W"] --> op1
op1 --> h["Var h"]
h --> op2["Op: Activation(h)"]
op2 --> loss["Var Loss"]
end
subgraph BackwardTape["Reverse-Mode Adjoint Execution"]
dLoss["Seed dLoss = 1.0"] --> adj2["Adjoint: dActivation"]
adj2 --> adj1["Adjoint: dLinear"]
adj1 --> dW["Accumulate dW"]
adj1 --> dx["Accumulate dx"]
end
ForwardTape ==> BackwardTape
-
CPU (
ojas-cpu):CpuBackenddefaults toNumerics::Fast;Numerics::Exact(opt-in) uses a packed GEMM on a persistent thread pool; bit-identical to pre-change golden digests at thread counts 1, 2, 3, 7, 16, and 18.Numerics::Faston macOS sends products of$\ge 2^{13}$ multiply-adds (FAST_WHOLE_CALL_MACS) to Apple Acceleratecblas_sgemm; smaller Fast products stay ontile_fast. Off macOS the cutoff is$2^{21}$ . -
SIMD (
ojas-simd): Fast-tier GEMM kernels including ARM64 NEON (~110 GFLOP/s single thread), x86_64 AVX2+FMA, portable FMA fallback, and Apple Accelerate wrapper. -
Metal (
ojas-metal):MetalBackendimplements 101 of the 107Backendmethods on device tensors throughtesslGEMM and its own.metalkernels (relu_backward,slice_backward,interpolate_bilinear_backwardandgroup_norm_backwardare trait defaults that refuse; seedocs/op-coverage.md). Causal attention is a tiled TensorOps forward plus a FlashAttention-2-style backward, with noT×Tscores stored, at head dims up to 256. Grouped-query attention is native: training accepts fewer KV heads than query heads (ojas-model/src/spec.rs:236), and the kernels read each KV head in place. Numerics default toFast(f32accumulation; opt-in step-by-stepbf16accumulation viawith_bf16_accumulation(true)/set_bf16_accumulation(true)).with_numerics(Numerics::Exact)sends linear products and the grouped conv2d GEMM through tesslgemm_fixed_order_*(ascendingk, contraction off). That flag does not redirect attention or the other reductions, and bf16 accumulation does not take the Exact GEMM. -
wgpu (
ojas-wgpu):WgpuBackendkeeps tensors device-resident and executes portable WGSL compute shaders. NumericsFast, tolerance 1e-4 relative against CPU. Muon NS5 runs in f32 and is checked againstCpuBackend(ojas-wgpu/tests/muon.rs). A non-finite result is reported at the nextsync,downloadorclip_grad_norm, naming the first op that produced it (ojas-wgpu/tests/faults.rs). -
Kernels (
ojas-kernels): WGSL shader modules forWgpuBackend(ojas-kernels/src/wgsl/), workgroup grid calculations, and a parity harness whose comparison fails on any non-finite value. -
CUDA & HIP (
ojas-cuda,ojas-hip): Optional compile flags (--features cuda,--features hip).CudaBackendimplementsBackend(ojas-cuda/src/backend.rs:84) but every compute op returnsUnsupported;ojas-hipis a copy probe and does not implementBackend. Neither is reachable from Go.
flowchart TD
LoadOn["Go LoadModel(ctx, path, LoadOptions)"] --> Kind{"opts.Device"}
Kind -->|DeviceCPU| CPU["CpuBackend (1 thread)"]
Kind -->|"DeviceCPUParallel (threads 1..=256)"| CPUP["CpuBackend, threads"]
Kind -->|DeviceCPUAuto| CPUA["CpuBackend, auto_threads from plan thread ceiling"]
Kind -->|DeviceMetal| Metal{"Metal device opens?"}
Kind -->|DeviceWgpu| Wgpu{"wgpu adapter opens?"}
Metal -->|Yes| MetalB["MetalBackend session (Device-Resident)"]
Metal -->|No| MetalErr["Load returns explicit device error\n(NO silent CPU session)"]
Wgpu -->|Yes| WgpuB["WgpuBackend session (Device-Resident)"]
Wgpu -->|No| WgpuErr["Load returns explicit wgpu error\n(NO silent CPU session)"]
Warning
Metal and wgpu attention enforce a strict head dimension limit (METAL_MAX_HEAD_DIM and ATTENTION_MAX_HEAD_DIM). A larger head dimension returns Err(OjasError::UnsupportedHeadDim); nothing is clamped or truncated. Causal SDPA accepts fewer KV heads than query heads.
ojas delegates platform-specialized and domain-specialized execution to dedicated, decoupled modular extensions:
-
Apple Silicon Metal 4 Extension (
tessl):- Role: High-throughput Mac inference engine and canonical Metal 4 compute substrate for
ojas-metal. - Shared Memory Pools (Zero-Copy UMA):
tessl::SharedBufimplementsojas_core::MappedBuffer(ojas-core/src/hal.rs) over pooledStorageModeSharedMetal allocations. CPU threads and Metal kernels access the same physical host pointers without intermediate memory copies, synchronized via timeline completion fences (last_use,last_write). - Authoritative Budgeting:
ojas_core::Budgetgoverns memory consumption;BudgetTrimHandlerautomatically flushes tessl's buffer pool freelist under pressure before allocations fail. - Kernels: MPP TensorOps
matmul2d, cooperative register accumulation, flash cross-attention, KV-cache managers, MLX Q4 GEMV/GEMM, Q8 GEMV and a per-group matmul2d, int8 and W4A8 GEMM, decode-only grouped-query attention, a fixed-order f32 GEMM forNumerics::Exactlinear products and the grouped conv2d GEMM, and EmbeddingGemma 2 multimodal towers.
- Role: High-throughput Mac inference engine and canonical Metal 4 compute substrate for
-
Sparse Computing Extension (
sparsl):- Role: Deterministic sparse and scan compute kernels for event-driven simulation, structured sparsity, and sparse neural execution.
- Representations & Operations: Compressed Sparse Row (CSR) and Column (CSC) matrix representations, deterministic Sparse Matrix-Vector multiplication (SpMV), chunked prefix scans over affine maps, and structure-of-arrays column buffers.
- Honesty Invariant & Fail-Closed Gates: Backends cannot silently degrade or substitute CPU execution; every
SparseOpvalidates column indices against matrix extents before dispatch. - Deep Learning Interop: Provides sparse gradient masks, dynamic sparse recompute maps, and event gating (
st-gate) for sparse neural architectures.
-
The Modularity Rationale:
- Decoupled Compile Cycles: Xcode Metal toolchain builds (
xcrun metal,metallib) and intensive sparse property testing (proptest, mutation suites) do not block or slow down core universal framework builds on Linux (CI-built; Windows is planned, not built). - Isolated Hardware Requirements: Apple Silicon M-series specifics live in
tessl; generic CPU and optional Metal sparse kernels live insparsl; universal multi-platform training lives inojas. - Independent Profiling: GEMM tile sweeps (
tessl) vs sparse crossover thresholds (sparsl) iterate on distinct benchmark suites. - Independent Release Cadence: Lightweight downstream consumers depend only on the exact compute capabilities they require.
See
docs/modular-extensions.mdfor full architectural contracts.
- Decoupled Compile Cycles: Xcode Metal toolchain builds (
Model parameters are automatically partitioned across two distinct optimizers based on tensor rank:
flowchart TD
subgraph ModelParams["Model Parameter Topology"]
Matrices["2D Weight Matrices (Rank >= 2)\n- Attention Q, K, V, Out Projections\n- SwiGLU Gate, Up, Down Projections\n- Attention Gate Matrix"]
Vectors["1D Vectors & Embeddings (Rank < 2)\n- Token / LM Head Embeddings\n- RMSNorm Scale Weights\n- Attention Gate Biases\n- Value Residual Lambdas"]
end
subgraph DualOptimizers["Dual Optimizers"]
Muon["Muon NS5 Optimizer\n- 5th-order Newton-Schulz polynomial in f32\n(bf16 by config; nanolab runs it in bf16)\n- Nesterov momentum (0.99)\n- Zeropower orthogonalization"]
AdamW["AdamW Optimizer\n- PyTorch single-tensor order (decay first)\n- eps = 1e-8 outside square root\n- Cosine learning rate scheduling"]
end
Matrices --> Muon
Vectors --> AdamW
- Step Counter Safety:
next_stepperforms checked integer addition, returningOjasError::OutOfRangeatu64::MAX. - Safe Gradient Clipping: Global gradient norm clipping includes a regularizer (
CLIP_GRAD_NORM_EPS = 1e-6) to prevent division by zero. - Moment Protection: If an update evaluates to non-finite numbers, the step aborts before optimizer moments are modified.
- Zero-IPC Overhead: Go calls the Rust engine in-process through CGO and
gusset. Go sends paths, token ids and options; the engine holds each model on its device. The calls areLoadModel/NewModel,OpenTrainer,TrainStep,SaveCheckpoint/Resume,LoadTokenizer/Tokenize/Detokenize/DetokenizeLossy/TokenID,GenerateIDs,SystemProfile,SetMemoryCeiling,Free, andClose(seego/README.md). The old stubStepopcode is retired. - Device Selection:
LoadOptions.DeviceselectsDeviceCPU,DeviceCPUParallel,DeviceCPUAuto,DeviceMetalorDeviceWgpu.DeviceCPUAutoautomatically sizes CPU pool threads to the system thread ceiling (usable CPUs capped by cgroup quota). A GPU backend that cannot open is an error, never a CPU model. A step reads back only its loss. - One Memory Ceiling & Pressure Admission: Every model's
BudgetBytesis a child of a process-wide ceiling (default 1 GiB, or machine hard limithard_memory_limitif smaller).SetMemoryCeilingraises it and is refused for 0, while any model is open, or ifbytesexceeds physical RAM/cgroup limits (ErrCapacity). Under critical kernel memory pressure (ErrPressure), calls that allocate are refused before starting whileSaveCheckpointandFreecontinue to run. - Preflight & Step Budget Tracking: Model load/new preflights parameter size against session budget before device allocation;
Trainerpreflights total state and optimizer scratch (optimizer_scratch_bytes), measuring high-water peak usage (Budget::peak_bytes) per step without memory leaks. - Panic Boundary Isolation: Rust panics on worker threads are caught by
catch_unwind, preventing host Go process crashes. The handle is then poisoned (gusset.ErrPoisoned) until closed.
sequenceDiagram
autonumber
participant Go as Go Application (go/api.go)
participant Gusset as Worker Pool (libgusset.a)
participant CAPI as FFI Engine (ojas-capi)
participant Model as ojas-model Trainer
participant Backend as Backend (CPU / Metal / wgpu)
Go->>Gusset: ojas.TrainStep(ctx, id)
Gusset->>CAPI: dispatch(OP_TRAIN_STEP)
Note over CAPI: catch_unwind catches panics (not aborts)
CAPI->>Model: Trainer step on the session's backend
Model->>Backend: Forward + fused CE + Backward + Muon/AdamW
Backend-->>Model: loss (the only readback)
Model-->>CAPI: StepResult{Loss, GradNorm, MatrixLR, AdamLR, Step, Tokens}
CAPI-->>Gusset: Serialized stats
Gusset-->>Go: ojas.StepResult, nil
Caution
If a worker thread panics while holding the ojas-capi session table lock, the next lock acquisition clears every session in the table to prevent stale state corruption. catch_unwind cannot catch OS process aborts, stack overflows, or infallible memory allocation panics.
The v1 target is the 124M parameter nanolab default GPT (12 layers, width 768, 12 heads of 64, SwiGLU 2048, vocabulary 50304, tied embedding). ojas-model implements it once: ModelSpec::nanolab_124m and a small CI ModelSpec::tiny, nanolab state_dict names (so torch exports load unchanged), the block over a Graph trait, and a Trainer. CI runs the tiny spec and a checked-in 40 KB fixture (2 layers, d 16, vocabulary 64); docs/status.md records no full 124M training run. docs/framework-design.md has the design and docs/shape-contract.md the shape rules every backend shares. CpuGpt in ojas-infer is the fast CPU decoder. Metal trains through MetalBackend and the same Tape; there is no separate Metal training step. The block below is the nanolab layer:
flowchart TD
subgraph TransformerBlock["nanolab GPT Transformer Layer"]
In["Input x [B, T, D]"] --> Norm1["RMSNorm (eps = 1e-6)"]
subgraph AttnBlock["Gated Causal Attention Mixer"]
Norm1 --> QKV["Q, K, V Projections [B, T, H, d]"]
QKV --> RoPE["Half-Split RoPE"]
RoPE --> QKN["RMS QK-Norm"]
QKN --> SDPA["Causal Scaled Dot-Product Attention"]
Norm1 --> Gate["Linear Gate + Bias: sigmoid(x W_g + b_g)"]
SDPA --> GatedAttn["Per-Head Gated Attn = SDPA * Gate"]
Gate --> GatedAttn
GatedAttn --> VR["Value Residual: lerp(Attn, V0, sigmoid(vr_lambda))"]
VR --> OutProj["Output Projection"]
end
OutProj --> ResAdd1["Residual Add (+)"]
In --> ResAdd1
subgraph MLPBlock["SwiGLU MLP Mixer"]
ResAdd1 --> Norm2["Post-RMSNorm"]
Norm2 --> SwiGLU["SwiGLU: (x W_gate) * silu(x W_up) W_down"]
end
SwiGLU --> ResAdd2["Residual Add (+)"]
ResAdd1 --> ResAdd2
ResAdd2 --> Next["Next Layer Activations"]
end
Per-crate counts are from the run of 2026-11-01: cargo test -p <crate> --release --no-fail-fast -- --test-threads=1, 2489 passed across the crates below. The Go count is go test -a -tags gusset_pkgconfig ./.... ojas-train (a workspace member added after that run) has no per-crate count here. The latest recorded full workspace run is at the top of this file; the numbers live in docs/data/test-runs.json and details are in docs/status.md.
| Crate | Purpose | Key Symbols | Tests Run |
|---|---|---|---|
ojas-core |
Core tensor engine & invariants | Tensor (host & device, typed host storage), Backend (optimizer_scratch_bytes), Numerics, Budget (peak_bytes, reset_peak, check_room), DType, OjasError, shapes validators |
260 passed, 2 ignored |
ojas-cpu |
High-performance CPU backend | CpuBackend, with_threads, with_numerics, packed GEMM, Accelerate BLAS, cosine and WSD schedules, pointwise parallel, budget scratch & pool stress |
564 passed, 12 ignored |
ojas-simd |
SIMD GEMM & NEON vector kernels | NEON GEMM, AVX2, sgemm_accelerate, gather_embedding_rows_768, split_nanolab_head_pairs_append, token bands, vDSP, vForce |
40 passed (workspace; 34 standalone) |
ojas-metal |
Apple Silicon Metal backend | MetalBackend (101 of 107 Backend methods; tiled attention, D ≤ 256, grouped-query; opt-in with_bf16_accumulation; deferred faults; training step) |
611 passed |
ojas-wgpu |
Portable WGSL backend | WgpuBackend (device-resident WGSL compute; Muon NS5; named first fault; linear CE) |
131 passed, 3 ignored |
ojas-kernels |
Shared kernel geometry & sources | Launch geometry, kernel source, NaN-safe parity harness | 11 passed |
ojas-autograd |
Reverse-mode tape & gradcheck | Tape, Var, backward_seeded, take_grad, fused head, central_diff (f64 oracle) |
173 passed |
ojas-model |
The nanolab GPT, written once | ModelSpec, param_table, init_params, Graph, Eval, block, Trainer (save, resume_from, preflight room checks, step peak tracking) |
82 passed |
ojas-io |
Formats & serialization | Streaming safetensors (F32/BF16/F16/I64/U16), Checkpoint v1, replace_dir_with |
76 passed |
ojas-data |
Dataset ingestion & tokenization | Token bins (u16/u32), seeded resumable batch sampler, GPT-2 BPE, counter-based RNG | 37 passed |
ojas-infer |
Autoregressive inference | CpuGpt (nanolab block, GQA), KV cache, greedy and temperature/top-k/top-p sampling |
40 passed |
ojas-qwen35 |
Qwen3.5 whole-step training provider | Config reader that refuses unsupported fields, tensor-name pre-flight, parameter groups, state on disk; tessl Metal only, not a Backend, empty off macOS |
34 passed, 8 ignored |
ojas-train |
Training loop, schedules and recipes over ojas | sha256 (re-exported from ojas_io::sha256), lappi::{step, trainer, schedule, recipe, adamw, mock, run_control, pyjson} (Lappi's qd-train modules, imported with history) |
not in the 2026-10-05 run |
ojas-capi |
C-ABI dispatch & session store | dispatch opcodes (load, new, train, save, resume, tokenize, sample, system profile, memory ceiling), DeviceCPUAuto, preflight validation, typed error kinds (ErrPressure), panic firewall |
79 passed |
ojas-gusset-engine |
Go link archive | Umbrella staticlib libgusset.a for Go CGO integration |
0 tests (the Go suite covers it) |
go/ |
Go client SDK | LoadModel, NewModel, OpenTrainer, TrainStep, SaveCheckpoint, Resume, GenerateIDs, SystemProfile, SetMemoryCeiling, Free, Close; DeviceCPUAuto, ErrPressure |
41 passed |
ojas-device |
Device kinds, system profile & resource planning | Device, probe_system (memory, CPU clusters, caches, unified memory, pressure), measure_bandwidth (bounded 2 shared buffers, row-split copy), ResourcePolicy, ResourcePlan, GemmBlocks |
72 passed (macOS; 66 on Linux) |
ojas-oracle |
Mathematical fixtures | IEEE-754 f64 oracle data fixtures | 263 passed, 1 ignored |
ojas-cuda |
CUDA backend | CudaBackend behind feature cuda: upload, download and sync; every compute op returns Unsupported |
8 passed (feature off) |
ojas-hip |
HIP probe | Copy probe behind feature hip; no kernel; not a Backend |
8 passed (feature off) |
The dropped scaffold crates ojas-nn, ojas-optim and ojas-engine are gone; the model layer lives in ojas-model.
The per-crate command below is how the counts above were produced (see docs/data/test-runs.json). cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all --check are clean across the workspace. ojas-qwen35 needs macOS with Apple silicon and the sibling tessl checkout.
# Run one crate's suite in release mode (how docs/status.md counts are taken)
cargo test -p ojas-model --release --no-fail-fast -- --test-threads=1
# Run the complete workspace test suite in release mode
cargo test --workspace --release -- --test-threads=1# 1. Build the umbrella Rust static library archive
cargo build -p ojas-gusset-engine
# 2. Run the Go tests with pkg-config linking against target/debug/libgusset.a
cd go && PKG_CONFIG_PATH="$PWD" go test -a -tags gusset_pkgconfig -count=1 -timeout 15m ./...Caution
The -a flag is mandatory when testing Go bindings because Go's build cache does not monitor changes to libgusset.a. go/gusset.pc links Apple frameworks; on Linux systems use PKG_CONFIG_PATH="$PWD/linux".
- Interactive Documentation (
site/index.html) — Live labs for determinism, memory budgets, device refusal, the panic firewall, and checked shapes, plus the benchmark comparison. docs/web-and-domain.md— Domain architecture, CNAME routing (ojas.vbcr.dev), and deployment topology.docs/architecture.md— Comprehensive architectural specification and memory layout.docs/io-ownership.md— Owner decision (2026-11-01, completed 2026-11-02): ojas owns safetensors, JSON, and NPY IO; tessl re-exportsojas_io::{npy, safetensors}directly with zero duplicate parsers.docs/status.md— Verification run logs, test matrices, and machine profile.docs/adaptive-resources.md— System profiling, adaptive resource planning, CPU topology, cache sizing, copy bandwidth, and unified memory governance.docs/bench-cpu-vs-torch.md— Benchmarks against PyTorch 2.13 CPU (tiny and larger step, single linear), plus Fast/Accelerate, Metal, and wgpu numbers taken under machine load.docs/bench-gpu-vs-torch.md— Metal and wgpu against PyTorch MPS at nanolab shapes.docs/bench-plots.md— 83 benchmark charts covering every result folder, comparing ojas Metal, wgpu, and CPU against PyTorch MPS and CPU across training and inference runs.docs/op-coverage.md— Mathematical specification of core operators and reference models.docs/checkpoint-v1.md— Binary checkpoint specification and framing.docs/dtype-policy.md— Data types, precision policies, and epsilon invariants.docs/backends.md— Hardware support matrix, adopted libraries, and backend dispatch.docs/framework-design.md— The approved framework layer:ojas-model, theGraphblock, the trainer, names, init and the checkpoint directory.docs/pytorch-parity-plan.md— Ranked gaps against what training and inference need, with verified, inferred and unverified labels.docs/shape-contract.md— One shape validator perBackendop, run before any budget or dispatch.docs/typed-storage-plan.md— Typed host storage (f32,u32,u16) and the bounded readback and load pieces.docs/metal-deferred-faults.md— Metal's deferred-fault contract, matching wgpu.docs/cuda-backend-scoping.md— Design-only scoping of a CUDA backend (no kernel written).docs/audit-resources.md— What the resource governor locks, with verified and reported rows. Earlier audits:audit-phase2,audit-phase3,audit-close,audit-cpu-hot,audit-larger-step.docs/audit.md— Defect analysis of upstream engines and ojas defensive countermeasures.docs/baseline.md— Benchmarks, toolchain baselines, and Apple Accelerate BLAS measurements.
rust · deep-learning · machine-learning · neural-networks · pytorch-alternative · tensorflow-alternative · deterministic-ai · metal-compute · wgpu · cuda · autograd · nanolab-gpt · muon-optimizer · adamw · in-process-go · gusset · zero-silent-fallback · memory-budget · tensor-engine · systems-research
Dual-licensed under either MIT (LICENSE-MIT) or Apache-2.0 (LICENSE-APACHE) at your option.
- Keller Jordan (
KellerJordan/modded-nanogpt) & OpenAI (openai/parameter-golf) — Inspired thearch02value-residual GPT architecture, Muon Newton-Schulz optimizer, and parameter-golf speedrun benchmarks. - Andrej Karpathy (
karpathy/nanoGPT) — Minimalist GPT training reference that informedojas-Modelsandnanolab. - Google DeepMind (
Gemma) & Alibaba Cloud (Qwen) — Reference transformer architectures targeted bygemma-metal/andojas-serve. - Apple MLX (
ml-explore/mlx) — Reference Apple Silicon Metal runtime benchmarked ingemma-metal/.
