Skip to content
bharathvbcrPublic

About

Deterministic, high-performance Rust deep learning engine and framework with zero silent fallback, in-process Go bindings, and strict memory budgeting

Topics

Resources

Contributing

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

1,119 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ojas

The ojas logo: a dark red liquid-glass icosahedron with a glowing core

Documentation Interactive Labs Rust 1.100+ Go 1.26+ Apple Silicon Metal 4 wgpu Portable WGSL License: MIT or Apache-2.0

ojas (vital energy, the essence training spends) is a deterministic, training-first universal framework in Rust: tensors, autograd, optimizers, training loops, data, checkpoints and IO, with a CPU reference backend and CUDA, HIP, wgpu and Metal plugging in as extensions. It is designed as a safe, modern alternative to PyTorch and TensorFlow for systems research and LLM training, and does only light inference (sampling and eval).

ojas serves as the central framework within a modular compute ecosystem:

  • tessl: Dedicated Apple Silicon / Metal 4 acceleration extension and Mac inference engine (MPP TensorOps matmul2d, unified memory allocator, ICB decode, MLX quantization, EmbeddingGemma 2 multimodal towers, and canonical Metal compute kernels for ojas-metal).
  • sparsl: Dedicated Sparse Computing extension (CSR/CSC sparse matrix representations, deterministic SpMV, affine prefix scans, event-driven simulation, and structured sparsity).

Note

Modularity Rationale: ojas, tessl, and sparsl are intentionally developed across separate checkouts and crates to guarantee:

  1. Decoupled Compile Cycles: The universal training core is built in CI on Linux and macOS (.github/workflows/test.yml has no Windows job, so no CI job builds Windows; the only local check on record is an x86_64 Windows cargo check and clippy of ojas-device alone, docs/adaptive-resources.md:50, :124; a CI compile check is planned, ft-3902a4b3) without dragging in Xcode/Metal toolchains or heavy sparse proptest/mutation suites.
  2. Isolated Hardware Requirements: Apple Silicon M-series specifics live in tessl; sparse event-driven kernels live in sparsl; universal CPU/wgpu/CUDA training lives in ojas.
  3. Independent Profiling & Micro-Benchmarking: GEMM tile sweeps and register tuning (tessl) vs sparsity crossover thresholds (sparsl) vs end-to-end training step throughput (ojas) iterate on their own benchmark harnesses.
  4. Independent Release Cadence & Separate Checkouts: Downstream applications import only what they need without bloat (see docs/modular-extensions.md and docs/architecture.md).

ojas pairs a compile-time safe Rust engine with an in-process Go API via gusset, zero-copy tensor views, reverse-mode automatic differentiation, and strict memory budgeting.

Key Architectural Guarantees

  • Bit-Identical Reproducibility: Single-threaded-order, index-increasing accumulation under Numerics::Exact guarantees bit-identical results across thread counts, worker scheduling, and machines.
  • Zero Silent Fallbacks: Selecting a GPU backend (Apple Silicon Metal, wgpu, CUDA) that cannot be initialized returns an explicit error immediately. It will never silently downgrade to CPU execution.
  • Explicit Budget Governance: All allocations request permits from a hierarchical Budget. Over-budget allocations immediately return OjasError::CapacityExceeded without dynamic memory clamping, hidden page swapping, or uncatchable OOMs.
  • Zero-IPC Multi-Language Bindings: High-performance Go and C-ABI integrations via gusset and ojas-capi link directly in-process with panic isolation boundaries, eliminating Python runtime overhead and serialization penalties.
  • Hardware Acceleration: Native Apple Silicon Metal acceleration via tessl and custom Metal kernels, portable cross-platform wgpu compute shaders, and optimized CPU kernels (ARM64 NEON, x86_64 AVX2, Apple Accelerate).
  • Dual Optimizers & Target Model: Out-of-the-box support for the nanolab GPT architecture with Muon NS5 (Newton-Schulz orthogonalization) and AdamW, sliding-window KV-cache autoregressive inference, top-k row sampling, streaming dataset loaders, and fast BPE tokenization.

What runs today:

  • Coverage differs by backend; docs/op-coverage.md has one row per Backend method (107, read at c172510). The CPU overrides 97 of the 107; its other ten are trait defaults: host behaviour for upload, download, cast_bf16, to_bf16, to_f32, argmax_rows and topk_rows, a constant for sync, autocast_region (which refuses Bf16), and causal_sdpa_backward_recompute, which composes the causal SDPA forward and backward. Metal overrides 101: relu_backward, slice_backward, interpolate_bilinear_backward and group_norm_backward are trait defaults that refuse, and autocast_region and causal_sdpa_backward_recompute are defaults on every backend. wgpu runs the nanolab GPT op set and refuses the Qwen3.5, encoder and spatial ops. On wgpu, tiled GEMM (gemm.wgsl) and FlashAttention-2 (attention.wgsl) promote per-lane accumulators to explicit vec4<f32>/scalar registers with (d4, c) loop inversion and unbranched full-tile fast paths; on Metal, RMSNorm, LayerNorm, RoPE, Per-Head Gate, and FlashAttention fold ST_IN/ST_OUT finiteness checks in-kernel alongside vectorized float4 fault/accumulate kernels (ojas_check_finite_v4, ojas_add_inplace_v4).
  • General attention (attention_forward, attention_forward_saving, attention_backward): bidirectional self-attention, cross-attention ($T_q \ne T_k$), broadcastable additive score bias with dedicated gradient pass, and per-batch key-length masks. On Metal: routed to tessl encoder_attn at D=32/64 for self-attention ($T_q = T_k$), native decoupled $T_q \times T_k$ TensorOps flash cross-attention (ojas_attn_fwd_d32 and ojas_attn_fwd_d64, without $O(m^2)$ pad-q zero-padding), tile-register additive logit bias (boxRPB via buffers 13..18), split-K tiled attention (ojas_xattn_part_dD/combine) for short queries or D=16, and scalar fallback.
  • Spatial operators: conv2d (arbitrary stride, pad, dilation, groups, depthwise; on Metal, 1x1 stride-1 unpadded conv skips im2col for direct NN GEMM with >2× speedup and selects the bf16 GEMM lane on bf16-exact operands), conv_transpose2d (kernel == stride), max_pool2d, interpolate_bilinear (both align_corners modes), window partition padding/cropping, and interleaved RoPE with 2-D axial tables. Planned permute executes row copies for contiguous 16-float spans and 2-D transpose tiles with fused finite tests.
  • Vision & encoder forward and backward ops: LayerNorm forward and backward (two-pass f64 sums for mean and variance on CPU), channel LayerNorm forward and backward, group_norm (exact f64 two-pass on CPU, 3-pass GPU kernel on Metal with large-mean residual correction), erf GELU forward and backward (gelu_erf_backward_exact), linear_bias backward (evaluates $dX, dW, db$ simultaneously with frozen-weight $dW$ bypass), broadcast_add backward, broadcast_mul backward, softmax_backward, ReLU, concat, slice (whole-input finite check), and fill (torch.full equivalent with U32 Ids::Below bound).
  • Cross-entropy: fused and linear cross-entropy on CPU, Metal and wgpu (and CUDA's K10 kernel) take the target-excluded form when standard target gradient $p_t - 1$ rounds to 0 (log1p_exact); Fast-tier vectorized exponentials under Numerics::Fast.
  • Parity against real SAM 3.1 safetensors weights, PyTorch 2.14 activation fixtures via ojas-oracle::sam_fixtures, and 31 PyTorch 2.15 vision/encoder backward golden cases (ojas-oracle::fixtures::encoder_bwd).
  • The autograd Tape gradchecks a multi-head attention block against double-precision (f64) finite differences.
  • ojas-model defines the nanolab GPT once (spec, state_dict names, order-independent init, the block) and runs it two ways: recorded on the Tape for training, eagerly through Eval for decoding. Its Trainer takes Muon + AdamW steps on any Backend and saves and resumes a checkpoint directory.
  • The Go API drives that model in process: LoadModel or NewModel (DeviceCPU, DeviceCPUParallel, DeviceCPUAuto, DeviceMetal, DeviceWgpu), OpenTrainer, TrainStep, SaveCheckpoint and Resume, a high-throughput GPT-2 BPE tokenizer, and sampled GenerateIDs. Metal and wgpu fail closed when no device opens.
  • ojas-infer decodes a nanolab-architecture model with CpuGpt on the CPU and with DeviceDecoder on any Backend (ojas-infer/src/device.rs:42; ojas-infer/README.md runs it on wgpu and Metal), greedy or sampled (top-k rows, temperature, top-p), with a sliding-window KV cache ring.
  • ojas-data streams token binary formats (u16/u32), resumable batch sampling with counter-based RNG, and provides an optimized GPT-2 BPE tokenizer with byte and lossy decoding, pre-token caching, and output capping.
  • ojas-cuda provides a host-side CUDA runtime with single-budget memory accounting, typed initialization and device-loss errors, bounded waits, and watchdog protection; its CudaBackend refuses every compute op (the earlier affine test kernel was retired 2026-10-14, docs/backends.md:74).
  • ojas-qwen35 (macOS only) runs a Qwen3.5 whole training step through tessl's Metal kernels. It is a provider, not a Backend.
  • Metal bf16 accumulation (MetalBackend::with_bf16_accumulation(true) / set_bf16_accumulation(true)): step-by-step nearest-even bf16 accumulation on the Metal GPU across linear/GEMM (ojas_gemm_bf16_accum), spatial and hybrid convolutions (conv2d im2col/pointwise/depthwise, conv_transpose2d, and causal_conv1d_silu forward and backward), attention (causal_sdpa, flash/split-K/scalar general attention, cached_attention, encoder attention backward), per-head sigmoid gate, all normalization forwards and backwards (rms_norm, layer_norm, channel_layer_norm, group_norm, gated_rms_norm), softmax and cross-entropy (softmax_forward/_backward, cross_entropy_mean, linear_cross_entropy_mean), sequence mean and L2 normalization (seq_mean_forward, l2_normalize_forward/_backward), pooling and interpolation (max_pool2d_backward, interpolate_bilinear_forward), embedding_backward, broadcast/bias/residual/value-residual/GDN-decay reductions and sums, accumulate_grad, and muon_ns5_step, while keeping default f32 accumulation when off.

Model coverage by backend. A cell says "yes" only where a test in the repository constructs that backend and runs that model (read at bfc24e5; none of those tests was re-run for this table):

Model CPU Metal wgpu CUDA
nanolab GPT yes: oracle_forward_loss_and_logits (ojas-model/tests/oracle_parity.rs:364, on cpu_model) yes: metal_trainer_matches_torch_on_the_tiny_fixtures (oracle_parity.rs:705) yes: wgpu_trainer_matches_torch_on_the_tiny_fixtures (oracle_parity.rs:720) no test
Qwen3.5 yes: six_straight_steps_equal_three_then_a_checkpoint_a_fresh_provider_and_three_more (ojas-model/tests/qwen35_step_provider.rs:463, a TapeStepProvider<CpuBackend>) yes: on_metal_the_norm_and_the_update_read_each_gradient_once (qwen35_step_provider.rs:1579) and the_metal_tape_matches_transformers_and_the_cpu_tape (ojas-model/tests/qwen35_metal_tape.rs:50, which returns early and passes when no Metal device opens, :51) no test; wgpu refuses the GDN, conv1d and gated-norm ops (trait defaults) no test; CudaBackend refuses every compute op
toy image MLP (ImageMlpProvider, ojas-model/src/image_mlp.rs:1-7) yes: the_job_contract_is_enforced_literally (ojas-model/tests/image_mlp_provider.rs:307, on fresh(), a CPU provider) yes: gpu_resume_is_bit_identical_and_matches_the_cpu_within_the_fast_tolerance (image_mlp_provider.rs:812, #[ignore], needs a Metal device) no test no test
CNN, ViT not yet (vision cards ft-4f11ce5a, ft-96023b34, ft-7f4f2d87) not yet not yet not yet

What does not run yet:

  • bf16 accumulation on CPU and wgpu (Metal GPU supports full bf16 accumulation via MetalBackend::with_bf16_accumulation(true); CPU and wgpu sum in f32, with bf16 as an optional autocast region and as Bf16 operands to linear and attention).
  • A CUDA Backend that computes: CudaBackend exists (ojas-cuda/src/backend.rs:84) and every compute op returns Unsupported. A HIP Backend: ojas-hip is a copy probe (copy_roundtrip, ojas-hip/src/lib.rs:84).
  • Everything else on the ranked gap list below.

The ranked list of gaps, and what has closed since it was written, is in docs/pytorch-parity-plan.md.

Against PyTorch MPS on the GPU (docs/bench-gpu-vs-torch.md, measured under heavy external GPU load, so only direction is verified):

  • ojas is slower on most rows. In round 5 (the latest paired round; torch 2.13.0 per bench/results/2026-10-02-r5/env_torch.json), a nanolab block forward + backward ran at 1.07× [0.96–1.16] torch's speed on Metal and 0.50× [0.49–0.56] on wgpu (bench/results/2026-10-02-r5/summary.md:69,122, both flagged "noisy - not quoted"; see Round 5 in docs/bench-gpu-vs-torch.md). Round 3 read 0.92× (Metal, mixed) and 0.41× (wgpu), also flagged noisy (bench/results/2026-10-01-r3b/summary.md:69,122), and round 1 read 0.37× and 0.18×. The GPU read 100% busy in rounds 3 and 5, so only the direction is verified.
  • It led on Metal attention backward at head dim 64 only (mixed at 128, docs/bench-gpu-vs-torch.md:1597-1610) and on Metal AdamW in round 3 (1.55×, noisy; round 5's 1.53× is also flagged noisy, bench/results/2026-10-02-r5/summary.md:61), so those leads are direction only. Its clip_grad_norm lead is against torch 2.13–2.14 only, whose MPS norm ran each full reduction on one threadgroup. PyTorch fixed that kernel for 2.15. On the 2.15 nightly, torch's row takes 8.45 ms (bench/results/2026-10-05-torch215-clip); ojas's Metal path, now tessl's sq_norm, takes 7.45 ms median (7.07 ms for the multi-tensor kernel it replaced; bench/results/2026-10-17-clip-sqnorm-ab). The two were not run together, so only the direction is verified. The 19.0 ms quoted earlier measured the path from before 0e3254f.
  • CUDA's CudaBackend implements Backend behind a feature (upload, download and sync; every compute op returns Unsupported), and HIP is a copy probe that does not.
Status:          Kernel set, backends, nanolab model, trainer, adaptive resource planning and Go API built; remaining gaps in docs/pytorch-parity-plan.md
Host Target:     Apple M5 Pro (Metal 4), macOS 27 (Darwin 27.0.0 arm64)
Interactive App: https://ojas.vbcr.dev (Alternate: https://bharath.vbcr.dev/ojas)
License:         MIT OR Apache-2.0

Verification: the latest full workspace run: 2005 passed, 0 failed, 41 ignored, 219 suites (cargo test --workspace --release -- --test-threads=2 at 51825fd, 2026-11-01, macOS, Metal and wgpu on the real GPU, no SKIP lines). This run is stale: a later full run (2026-10-13 to 10-14, merged main after bc498c4) recorded 231 suites, 1991 passed, 1 failed (a_real_oversize_gdn_workspace_is_capacity_exceeded), 50 ignored (docs/reports/BUILD_REPORT-w11.md:28); it is not yet in test-runs.json. Per-crate counts are from the run of 2026-11-01 (2489 passed across the crates below, Go 41 passed). Every number comes from docs/data/test-runs.json.

Explore the Interactive Web Experience: The ojas interactive documentation has live labs for each guarantee: bit-identical reductions across thread counts, memory budget refusal, no silent device fallback, the Go panic firewall, and checked shape arithmetic, plus the CPU versus PyTorch timings.


Why ojas? (A PyTorch & TensorFlow Alternative)

Mainstream frameworks like PyTorch and TensorFlow carry decades of legacy baggage: hidden global state, silent CPU fallbacks that mask hardware misconfigurations, nondeterministic multithreaded reductions, memory fragmentation, and cumbersome multi-language bindings that rely on heavyweight Python runtimes.

ojas is engineered from first principles with radical architectural guarantees:

flowchart TD
    subgraph PyTorchTF["Legacy Frameworks (PyTorch / TensorFlow)"]
        L1["Implicit Global State & Seed Drift"]
        L2["Silent CPU Fallback on Missing GPU\n(Hides deployment bottlenecks)"]
        L3["Dynamic Memory Clamping & OOM Panics"]
        L4["Heavyweight Python-Centric Runtime Stack"]
        L5["Integer Overflow Hazards in Long Training Steps"]
        L6["Complex C++ Codebase with Opaque Abstractions"]
    end

    subgraph ojasPrinciples["ojas Architectural Guarantees"]
        O1["Bit-Identical Determinism under Numerics::Exact (opt-in)"]
        O2["Zero Silent Fallbacks: Explicit Hardware Refusal\n(Fail fast, fail loud)"]
        O3["Explicit Budget Governance: try_reserve() Never Clamps"]
        O4["Pure Rust Core with Direct In-Process Go & C-ABI Bindings"]
        O5["overflow-checks = true in Release\n(step counters use checked_add; some ids wrap or saturate)"]
        O6["#![forbid(unsafe_code)] on the library crates\n(not cuda/hip features, ojas-gusset-engine, or metal/build.rs)"]
    end

    L1 -.->|Solved By| O1
    L2 -.->|Solved By| O2
    L3 -.->|Solved By| O3
    L4 -.->|Solved By| O4
    L5 -.->|Solved By| O5
    L6 -.->|Solved By| O6
Loading

Important

Deterministic Bit-Identicality: CpuBackend defaults to Numerics::Fast: FMA, and on macOS Accelerate for GEMMs of at least 2^13 multiply-adds (ojas_cpu::FAST_WHOLE_CALL_MACS), as PyTorch's CPU linear does. Opt into Numerics::Exact with .with_numerics(Numerics::Exact) for single-threaded-order, index-increasing accumulation. Under Exact, identical seeds give identical bits across thread counts and machines. The C/Go API has no numerics selector, so C and Go callers get Fast.

Zero Silent Fallback: Selecting a GPU backend (Metal, wgpu, CUDA, HIP) that cannot be initialized returns an explicit error immediately. It will never silently downgrade to CPU execution.


Framework Architecture

ojas is structured as a modular stack of focused, single-responsibility crates:

flowchart TD
    subgraph Applications["User Applications & Service APIs"]
        GoApp["Go Services & Applications\n(github.com/bharathvbcr/ojas/go)"]
        RustApp["Rust Binaries, Services & CLIs"]
        ForeignApp["C / C++ / Embedded Host Applications"]
    end

    subgraph Interop["In-Process Interop & Foreign Function Interface"]
        Gusset["ojas-gusset-engine\n(libgusset.a Worker Pool & Ticket Queue)"]
        CAPI["ojas-capi\n(C-ABI Engine, Session Store, Panic Isolation Boundary)"]
    end

    subgraph HighLevel["Model, Autograd & Inference Layer"]
        Infer["ojas-infer\n(Autoregressive Decoding, KV Cache, Logit Validation)"]
        Autograd["ojas-autograd\n(Dynamic Reverse-Mode Tape & f64 Gradcheck)"]
        Model["ojas-model\n(nanolab GPT written once: spec, init, Graph block, Trainer, checkpoint dir)"]
        Qwen35["ojas-qwen35\n(Qwen3.5 whole-step provider over tessl; macOS Metal only; not a Backend)"]
    end

    subgraph CoreEngine["Tensor Substrate & System Types"]
        Core["ojas-core\n(Tensor, Strides, Offset Slicing, Budget, DType, Errors)"]
        Device["ojas-device\n(Device Kinds, System Profile, ResourcePlan; no router)"]
        IO["ojas-io\n(Strict Safetensors Parser, Binary Checkpoint v1)"]
        Data["ojas-data\n(Dataset Streaming, Token Binary Formats, Deterministic RNG)"]
        Oracle["ojas-oracle\n(IEEE-754 f64 Numerical Reference Fixtures)"]
    end

    subgraph HardwareBackends["Pluggable Hardware Compute Backends"]
        CPU["ojas-cpu\n(CpuBackend: Fast default, Accelerate on macOS; Exact packed GEMM opt-in)"]
        SIMD["ojas-simd\n(NEON GEMM, AVX2, and Accelerate cblas_sgemm wrapper)"]
        Metal["ojas-metal\n(MetalBackend, device-resident, via tessl)"]
        WGPU["ojas-wgpu\n(WgpuBackend, device-resident WGSL)"]
        Kernels["ojas-kernels\n(Shared WGSL/CUDA sources, workgroup geometry, parity harness)"]
        CUDA["ojas-cuda\n(CudaBackend behind feature cuda; compute ops Unsupported)"]
        HIP["ojas-hip\n(Copy probe; no HIP kernel; not a Backend)"]
    end

    subgraph ModularExtensions["Modular Compute Extensions (Dedicated Checkouts)"]
        TesslExt["tessl (Metal 4 Extension & Mac Inference)\nMPP TensorOps, SharedBuf UMA, KV Caches, EmbeddingGemma 2"]
        SparslExt["sparsl (Sparse Computing Extension)\nDeterministic SpMV, CSR/CSC, Affine Scans, Event Simulation"]
    end

    GoApp --> Gusset
    Gusset --> CAPI
    ForeignApp --> CAPI
    RustApp --> HighLevel
    CAPI --> HighLevel
    Infer --> Model
    Model --> Autograd

    HighLevel --> CoreEngine
    HighLevel --> HardwareBackends
    HardwareBackends --> CoreEngine
    CoreEngine --> HardwareBackends

    Metal -->|Zero-Copy UMA & Metal 4 Kernels| TesslExt
    CoreEngine -.->|Sparse Tensor & Linear Algebra| SparslExt
Loading

Core Capabilities & Subsystems

1. General-Purpose Tensor Engine (ojas-core)

  • Zero-Copy Views: Tensor representations track an explicit byte_offset. Sub-views created via .narrow() adjust byte offsets without memory re-allocation or buffer copying.
  • Strict Memory Budgets: Memory consumption is governed by an explicit Budget. When an allocation exceeds limits, try_reserve() returns OjasError::CapacityExceeded immediately rather than swapping or resizing.
  • Type System: DType tags F32, Bf16, F16 and U32. Every backend accumulates in F32 by default (token ids in U32), and MetalBackend supports opt-in step-by-step bf16 accumulation across all GPU reductions and dot products via MetalBackend::with_bf16_accumulation(true) / set_bf16_accumulation(true). The matmul-class ops (linear_*, causal_sdpa_*, attention_*, and conv2d_forward) also take Bf16 operands on backends whose bf16_operands() is true (CPU, Metal and wgpu), widened into f32 (or bf16 on Metal when bf16_accumulation is enabled); on the CPU and Metal so does the tied table (the embedding gather and backward, and the fused cross-entropy's weight), read in place without an f32 copy of the table. Every other op refuses non-F32 dtypes (docs/dtype-policy.md). ojas-io decodes BF16/F16 safetensors to f32 and encodes back with round-to-nearest-even.
  • Device Tensors: Tensor::from_device wraps a backend's buffer. to_host copies it back and is counted per budget tree by Budget::device_readbacks. device_buffer_mut requires sole ownership of the storage. The default Backend::upload returns Unsupported for a host tensor on a non-CPU backend; there is no CPU fallback.
flowchart LR
    subgraph Storage["Physical Allocation Buffer (Arc<Vec<u8>>)"]
        Pad["Leading Offset [0 .. byte_offset)"]
        Active["Active Tensor Slices [byte_offset .. byte_offset + active_bytes)"]
        Unused["Unused Capacity"]
    end

    subgraph View1["Parent Tensor [2, 16, 64]"]
        V1["byte_offset = 0"]
    end

    subgraph View2["Sliced View: narrow(1, 4, 8)"]
        V2["byte_offset = 1024 bytes (zero-copy)"]
    end

    View1 --> Active
    View2 --> Active
Loading

Note

All tensor allocations must request permits from a Budget. If an allocation exceeds available capacity, try_reserve() returns Err(OjasError::CapacityExceeded). Infallible runtime calls (vec!, format!, thread::spawn) operate outside this budget and can still abort if physical host RAM is exhausted.


2. Reverse-Mode Automatic Differentiation (ojas-autograd)

  • Tape-Based Computation Graph: Dynamic tape recording variable nodes (Var) and operational adjoints for forward and reverse sweeps.
  • Numerical Gradcheck Oracle: Analytical f32 gradients are verified against double-precision (f64) central finite differences, ensuring mathematical soundness down to machine tolerances.
  • Device Tape Residency: On non-CPU backends (MetalBackend, WgpuBackend), leaf and input tensors are uploaded once. Reshape is a zero-copy view. The backward seed is uploaded, and non-root cross-entropy gradients are scaled on the device using rank-1 linears. Intermediate activations stay resident on GPU; the loss download is the only audited readback.
flowchart LR
    subgraph ForwardTape["Forward Execution"]
        x["Var x"] --> op1["Op: Linear(x, W)"]
        W["Var W"] --> op1
        op1 --> h["Var h"]
        h --> op2["Op: Activation(h)"]
        op2 --> loss["Var Loss"]
    end

    subgraph BackwardTape["Reverse-Mode Adjoint Execution"]
        dLoss["Seed dLoss = 1.0"] --> adj2["Adjoint: dActivation"]
        adj2 --> adj1["Adjoint: dLinear"]
        adj1 --> dW["Accumulate dW"]
        adj1 --> dx["Accumulate dx"]
    end

    ForwardTape ==> BackwardTape
Loading

3. Native & Portable Hardware Backends

  • CPU (ojas-cpu): CpuBackend defaults to Numerics::Fast; Numerics::Exact (opt-in) uses a packed GEMM on a persistent thread pool; bit-identical to pre-change golden digests at thread counts 1, 2, 3, 7, 16, and 18. Numerics::Fast on macOS sends products of $\ge 2^{13}$ multiply-adds (FAST_WHOLE_CALL_MACS) to Apple Accelerate cblas_sgemm; smaller Fast products stay on tile_fast. Off macOS the cutoff is $2^{21}$.
  • SIMD (ojas-simd): Fast-tier GEMM kernels including ARM64 NEON (~110 GFLOP/s single thread), x86_64 AVX2+FMA, portable FMA fallback, and Apple Accelerate wrapper.
  • Metal (ojas-metal): MetalBackend implements 101 of the 107 Backend methods on device tensors through tessl GEMM and its own .metal kernels (relu_backward, slice_backward, interpolate_bilinear_backward and group_norm_backward are trait defaults that refuse; see docs/op-coverage.md). Causal attention is a tiled TensorOps forward plus a FlashAttention-2-style backward, with no T×T scores stored, at head dims up to 256. Grouped-query attention is native: training accepts fewer KV heads than query heads (ojas-model/src/spec.rs:236), and the kernels read each KV head in place. Numerics default to Fast (f32 accumulation; opt-in step-by-step bf16 accumulation via with_bf16_accumulation(true) / set_bf16_accumulation(true)). with_numerics(Numerics::Exact) sends linear products and the grouped conv2d GEMM through tessl gemm_fixed_order_* (ascending k, contraction off). That flag does not redirect attention or the other reductions, and bf16 accumulation does not take the Exact GEMM.
  • wgpu (ojas-wgpu): WgpuBackend keeps tensors device-resident and executes portable WGSL compute shaders. Numerics Fast, tolerance 1e-4 relative against CPU. Muon NS5 runs in f32 and is checked against CpuBackend (ojas-wgpu/tests/muon.rs). A non-finite result is reported at the next sync, download or clip_grad_norm, naming the first op that produced it (ojas-wgpu/tests/faults.rs).
  • Kernels (ojas-kernels): WGSL shader modules for WgpuBackend (ojas-kernels/src/wgsl/), workgroup grid calculations, and a parity harness whose comparison fails on any non-finite value.
  • CUDA & HIP (ojas-cuda, ojas-hip): Optional compile flags (--features cuda, --features hip). CudaBackend implements Backend (ojas-cuda/src/backend.rs:84) but every compute op returns Unsupported; ojas-hip is a copy probe and does not implement Backend. Neither is reachable from Go.
flowchart TD
    LoadOn["Go LoadModel(ctx, path, LoadOptions)"] --> Kind{"opts.Device"}

    Kind -->|DeviceCPU| CPU["CpuBackend (1 thread)"]
    Kind -->|"DeviceCPUParallel (threads 1..=256)"| CPUP["CpuBackend, threads"]
    Kind -->|DeviceCPUAuto| CPUA["CpuBackend, auto_threads from plan thread ceiling"]
    Kind -->|DeviceMetal| Metal{"Metal device opens?"}
    Kind -->|DeviceWgpu| Wgpu{"wgpu adapter opens?"}

    Metal -->|Yes| MetalB["MetalBackend session (Device-Resident)"]
    Metal -->|No| MetalErr["Load returns explicit device error\n(NO silent CPU session)"]
    Wgpu -->|Yes| WgpuB["WgpuBackend session (Device-Resident)"]
    Wgpu -->|No| WgpuErr["Load returns explicit wgpu error\n(NO silent CPU session)"]
Loading

Warning

Metal and wgpu attention enforce a strict head dimension limit ($d_{\text{head}} \le 256$, METAL_MAX_HEAD_DIM and ATTENTION_MAX_HEAD_DIM). A larger head dimension returns Err(OjasError::UnsupportedHeadDim); nothing is clamped or truncated. Causal SDPA accepts fewer KV heads than query heads.


4. Modular Compute Extensions (tessl & sparsl)

ojas delegates platform-specialized and domain-specialized execution to dedicated, decoupled modular extensions:

  • Apple Silicon Metal 4 Extension (tessl):

    • Role: High-throughput Mac inference engine and canonical Metal 4 compute substrate for ojas-metal.
    • Shared Memory Pools (Zero-Copy UMA): tessl::SharedBuf implements ojas_core::MappedBuffer (ojas-core/src/hal.rs) over pooled StorageModeShared Metal allocations. CPU threads and Metal kernels access the same physical host pointers without intermediate memory copies, synchronized via timeline completion fences (last_use, last_write).
    • Authoritative Budgeting: ojas_core::Budget governs memory consumption; BudgetTrimHandler automatically flushes tessl's buffer pool freelist under pressure before allocations fail.
    • Kernels: MPP TensorOps matmul2d, cooperative register accumulation, flash cross-attention, KV-cache managers, MLX Q4 GEMV/GEMM, Q8 GEMV and a per-group matmul2d, int8 and W4A8 GEMM, decode-only grouped-query attention, a fixed-order f32 GEMM for Numerics::Exact linear products and the grouped conv2d GEMM, and EmbeddingGemma 2 multimodal towers.
  • Sparse Computing Extension (sparsl):

    • Role: Deterministic sparse and scan compute kernels for event-driven simulation, structured sparsity, and sparse neural execution.
    • Representations & Operations: Compressed Sparse Row (CSR) and Column (CSC) matrix representations, deterministic Sparse Matrix-Vector multiplication (SpMV), chunked prefix scans over affine maps, and structure-of-arrays column buffers.
    • Honesty Invariant & Fail-Closed Gates: Backends cannot silently degrade or substitute CPU execution; every SparseOp validates column indices against matrix extents before dispatch.
    • Deep Learning Interop: Provides sparse gradient masks, dynamic sparse recompute maps, and event gating (st-gate) for sparse neural architectures.
  • The Modularity Rationale:

    1. Decoupled Compile Cycles: Xcode Metal toolchain builds (xcrun metal, metallib) and intensive sparse property testing (proptest, mutation suites) do not block or slow down core universal framework builds on Linux (CI-built; Windows is planned, not built).
    2. Isolated Hardware Requirements: Apple Silicon M-series specifics live in tessl; generic CPU and optional Metal sparse kernels live in sparsl; universal multi-platform training lives in ojas.
    3. Independent Profiling: GEMM tile sweeps (tessl) vs sparse crossover thresholds (sparsl) iterate on distinct benchmark suites.
    4. Independent Release Cadence: Lightweight downstream consumers depend only on the exact compute capabilities they require. See docs/modular-extensions.md for full architectural contracts.

5. Dual Optimizer Architecture (ojas-cpu)

Model parameters are automatically partitioned across two distinct optimizers based on tensor rank:

flowchart TD
    subgraph ModelParams["Model Parameter Topology"]
        Matrices["2D Weight Matrices (Rank >= 2)\n- Attention Q, K, V, Out Projections\n- SwiGLU Gate, Up, Down Projections\n- Attention Gate Matrix"]
        Vectors["1D Vectors & Embeddings (Rank < 2)\n- Token / LM Head Embeddings\n- RMSNorm Scale Weights\n- Attention Gate Biases\n- Value Residual Lambdas"]
    end

    subgraph DualOptimizers["Dual Optimizers"]
        Muon["Muon NS5 Optimizer\n- 5th-order Newton-Schulz polynomial in f32\n(bf16 by config; nanolab runs it in bf16)\n- Nesterov momentum (0.99)\n- Zeropower orthogonalization"]
        AdamW["AdamW Optimizer\n- PyTorch single-tensor order (decay first)\n- eps = 1e-8 outside square root\n- Cosine learning rate scheduling"]
    end

    Matrices --> Muon
    Vectors --> AdamW
Loading
  • Step Counter Safety: next_step performs checked integer addition, returning OjasError::OutOfRange at u64::MAX.
  • Safe Gradient Clipping: Global gradient norm clipping includes a regularizer (CLIP_GRAD_NORM_EPS = 1e-6) to prevent division by zero.
  • Moment Protection: If an update evaluates to non-finite numbers, the step aborts before optimizer moments are modified.

6. In-Process Multi-Language Integration (go/, ojas-capi)

  • Zero-IPC Overhead: Go calls the Rust engine in-process through CGO and gusset. Go sends paths, token ids and options; the engine holds each model on its device. The calls are LoadModel/NewModel, OpenTrainer, TrainStep, SaveCheckpoint/Resume, LoadTokenizer/Tokenize/Detokenize/DetokenizeLossy/TokenID, GenerateIDs, SystemProfile, SetMemoryCeiling, Free, and Close (see go/README.md). The old stub Step opcode is retired.
  • Device Selection: LoadOptions.Device selects DeviceCPU, DeviceCPUParallel, DeviceCPUAuto, DeviceMetal or DeviceWgpu. DeviceCPUAuto automatically sizes CPU pool threads to the system thread ceiling (usable CPUs capped by cgroup quota). A GPU backend that cannot open is an error, never a CPU model. A step reads back only its loss.
  • One Memory Ceiling & Pressure Admission: Every model's BudgetBytes is a child of a process-wide ceiling (default 1 GiB, or machine hard limit hard_memory_limit if smaller). SetMemoryCeiling raises it and is refused for 0, while any model is open, or if bytes exceeds physical RAM/cgroup limits (ErrCapacity). Under critical kernel memory pressure (ErrPressure), calls that allocate are refused before starting while SaveCheckpoint and Free continue to run.
  • Preflight & Step Budget Tracking: Model load/new preflights parameter size against session budget before device allocation; Trainer preflights total state and optimizer scratch (optimizer_scratch_bytes), measuring high-water peak usage (Budget::peak_bytes) per step without memory leaks.
  • Panic Boundary Isolation: Rust panics on worker threads are caught by catch_unwind, preventing host Go process crashes. The handle is then poisoned (gusset.ErrPoisoned) until closed.
sequenceDiagram
    autonumber
    participant Go as Go Application (go/api.go)
    participant Gusset as Worker Pool (libgusset.a)
    participant CAPI as FFI Engine (ojas-capi)
    participant Model as ojas-model Trainer
    participant Backend as Backend (CPU / Metal / wgpu)

    Go->>Gusset: ojas.TrainStep(ctx, id)
    Gusset->>CAPI: dispatch(OP_TRAIN_STEP)
    Note over CAPI: catch_unwind catches panics (not aborts)
    CAPI->>Model: Trainer step on the session's backend
    Model->>Backend: Forward + fused CE + Backward + Muon/AdamW
    Backend-->>Model: loss (the only readback)
    Model-->>CAPI: StepResult{Loss, GradNorm, MatrixLR, AdamLR, Step, Tokens}
    CAPI-->>Gusset: Serialized stats
    Gusset-->>Go: ojas.StepResult, nil
Loading

Caution

If a worker thread panics while holding the ojas-capi session table lock, the next lock acquisition clears every session in the table to prevent stale state corruption. catch_unwind cannot catch OS process aborts, stack overflows, or infallible memory allocation panics.


7. Target Model: nanolab Default GPT

The v1 target is the 124M parameter nanolab default GPT (12 layers, width 768, 12 heads of 64, SwiGLU 2048, vocabulary 50304, tied embedding). ojas-model implements it once: ModelSpec::nanolab_124m and a small CI ModelSpec::tiny, nanolab state_dict names (so torch exports load unchanged), the block over a Graph trait, and a Trainer. CI runs the tiny spec and a checked-in 40 KB fixture (2 layers, d 16, vocabulary 64); docs/status.md records no full 124M training run. docs/framework-design.md has the design and docs/shape-contract.md the shape rules every backend shares. CpuGpt in ojas-infer is the fast CPU decoder. Metal trains through MetalBackend and the same Tape; there is no separate Metal training step. The block below is the nanolab layer:

flowchart TD
    subgraph TransformerBlock["nanolab GPT Transformer Layer"]
        In["Input x [B, T, D]"] --> Norm1["RMSNorm (eps = 1e-6)"]
        
        subgraph AttnBlock["Gated Causal Attention Mixer"]
            Norm1 --> QKV["Q, K, V Projections [B, T, H, d]"]
            QKV --> RoPE["Half-Split RoPE"]
            RoPE --> QKN["RMS QK-Norm"]
            QKN --> SDPA["Causal Scaled Dot-Product Attention"]
            
            Norm1 --> Gate["Linear Gate + Bias: sigmoid(x W_g + b_g)"]
            SDPA --> GatedAttn["Per-Head Gated Attn = SDPA * Gate"]
            Gate --> GatedAttn
            
            GatedAttn --> VR["Value Residual: lerp(Attn, V0, sigmoid(vr_lambda))"]
            VR --> OutProj["Output Projection"]
        end
        
        OutProj --> ResAdd1["Residual Add (+)"]
        In --> ResAdd1
        
        subgraph MLPBlock["SwiGLU MLP Mixer"]
            ResAdd1 --> Norm2["Post-RMSNorm"]
            Norm2 --> SwiGLU["SwiGLU: (x W_gate) * silu(x W_up) W_down"]
        end
        
        SwiGLU --> ResAdd2["Residual Add (+)"]
        ResAdd1 --> ResAdd2
        ResAdd2 --> Next["Next Layer Activations"]
    end
Loading

Workspace Crate Catalog

Per-crate counts are from the run of 2026-11-01: cargo test -p <crate> --release --no-fail-fast -- --test-threads=1, 2489 passed across the crates below. The Go count is go test -a -tags gusset_pkgconfig ./.... ojas-train (a workspace member added after that run) has no per-crate count here. The latest recorded full workspace run is at the top of this file; the numbers live in docs/data/test-runs.json and details are in docs/status.md.

Crate Purpose Key Symbols Tests Run
ojas-core Core tensor engine & invariants Tensor (host & device, typed host storage), Backend (optimizer_scratch_bytes), Numerics, Budget (peak_bytes, reset_peak, check_room), DType, OjasError, shapes validators 260 passed, 2 ignored
ojas-cpu High-performance CPU backend CpuBackend, with_threads, with_numerics, packed GEMM, Accelerate BLAS, cosine and WSD schedules, pointwise parallel, budget scratch & pool stress 564 passed, 12 ignored
ojas-simd SIMD GEMM & NEON vector kernels NEON GEMM, AVX2, sgemm_accelerate, gather_embedding_rows_768, split_nanolab_head_pairs_append, token bands, vDSP, vForce 40 passed (workspace; 34 standalone)
ojas-metal Apple Silicon Metal backend MetalBackend (101 of 107 Backend methods; tiled attention, D ≤ 256, grouped-query; opt-in with_bf16_accumulation; deferred faults; training step) 611 passed
ojas-wgpu Portable WGSL backend WgpuBackend (device-resident WGSL compute; Muon NS5; named first fault; linear CE) 131 passed, 3 ignored
ojas-kernels Shared kernel geometry & sources Launch geometry, kernel source, NaN-safe parity harness 11 passed
ojas-autograd Reverse-mode tape & gradcheck Tape, Var, backward_seeded, take_grad, fused head, central_diff (f64 oracle) 173 passed
ojas-model The nanolab GPT, written once ModelSpec, param_table, init_params, Graph, Eval, block, Trainer (save, resume_from, preflight room checks, step peak tracking) 82 passed
ojas-io Formats & serialization Streaming safetensors (F32/BF16/F16/I64/U16), Checkpoint v1, replace_dir_with 76 passed
ojas-data Dataset ingestion & tokenization Token bins (u16/u32), seeded resumable batch sampler, GPT-2 BPE, counter-based RNG 37 passed
ojas-infer Autoregressive inference CpuGpt (nanolab block, GQA), KV cache, greedy and temperature/top-k/top-p sampling 40 passed
ojas-qwen35 Qwen3.5 whole-step training provider Config reader that refuses unsupported fields, tensor-name pre-flight, parameter groups, state on disk; tessl Metal only, not a Backend, empty off macOS 34 passed, 8 ignored
ojas-train Training loop, schedules and recipes over ojas sha256 (re-exported from ojas_io::sha256), lappi::{step, trainer, schedule, recipe, adamw, mock, run_control, pyjson} (Lappi's qd-train modules, imported with history) not in the 2026-10-05 run
ojas-capi C-ABI dispatch & session store dispatch opcodes (load, new, train, save, resume, tokenize, sample, system profile, memory ceiling), DeviceCPUAuto, preflight validation, typed error kinds (ErrPressure), panic firewall 79 passed
ojas-gusset-engine Go link archive Umbrella staticlib libgusset.a for Go CGO integration 0 tests (the Go suite covers it)
go/ Go client SDK LoadModel, NewModel, OpenTrainer, TrainStep, SaveCheckpoint, Resume, GenerateIDs, SystemProfile, SetMemoryCeiling, Free, Close; DeviceCPUAuto, ErrPressure 41 passed
ojas-device Device kinds, system profile & resource planning Device, probe_system (memory, CPU clusters, caches, unified memory, pressure), measure_bandwidth (bounded 2 shared buffers, row-split copy), ResourcePolicy, ResourcePlan, GemmBlocks 72 passed (macOS; 66 on Linux)
ojas-oracle Mathematical fixtures IEEE-754 f64 oracle data fixtures 263 passed, 1 ignored
ojas-cuda CUDA backend CudaBackend behind feature cuda: upload, download and sync; every compute op returns Unsupported 8 passed (feature off)
ojas-hip HIP probe Copy probe behind feature hip; no kernel; not a Backend 8 passed (feature off)

The dropped scaffold crates ojas-nn, ojas-optim and ojas-engine are gone; the model layer lives in ojas-model.


Verification & Test Commands

The per-crate command below is how the counts above were produced (see docs/data/test-runs.json). cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all --check are clean across the workspace. ojas-qwen35 needs macOS with Apple silicon and the sibling tessl checkout.

# Run one crate's suite in release mode (how docs/status.md counts are taken)
cargo test -p ojas-model --release --no-fail-fast -- --test-threads=1

# Run the complete workspace test suite in release mode
cargo test --workspace --release -- --test-threads=1

In-Process Go Client Tests

# 1. Build the umbrella Rust static library archive
cargo build -p ojas-gusset-engine

# 2. Run the Go tests with pkg-config linking against target/debug/libgusset.a
cd go && PKG_CONFIG_PATH="$PWD" go test -a -tags gusset_pkgconfig -count=1 -timeout 15m ./...

Caution

The -a flag is mandatory when testing Go bindings because Go's build cache does not monitor changes to libgusset.a. go/gusset.pc links Apple frameworks; on Linux systems use PKG_CONFIG_PATH="$PWD/linux".


Documentation & Web Directory


Topics & Discoverability

rust · deep-learning · machine-learning · neural-networks · pytorch-alternative · tensorflow-alternative · deterministic-ai · metal-compute · wgpu · cuda · autograd · nanolab-gpt · muon-optimizer · adamw · in-process-go · gusset · zero-silent-fallback · memory-budget · tensor-engine · systems-research


License & Acknowledgements

Dual-licensed under either MIT (LICENSE-MIT) or Apache-2.0 (LICENSE-APACHE) at your option.

About

Deterministic, high-performance Rust deep learning engine and framework with zero silent fallback, in-process Go bindings, and strict memory budgeting

Topics

Resources

Contributing

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages