Semantic Cache-Line

A CUDA research prototype for Ontological Cache-Line Inversion (OCLI): 128-byte topology-aligned records that co-locate ontology masks, bounded graph adjacency, and hot FP16 KV micro-tiles.

Graph-RAG combines dense model state with irregular graph traversal. A normal KV layout is friendly to tensor operations, while pointer-heavy ontology and adjacency lookups can generate scattered memory requests. This project tests a specific systems hypothesis: for graph-conditioned inference paths that consume semantic metadata and a small KV fragment together, placing those fields in the same aligned address region can reduce transferred-but-unused bytes and memory stall time. NVIDIA research likewise describes graph workloads as irregular and sparse in memory access; see its GPU graph page-size study.

This is an experimental layout and benchmark harness, not a performance claim.

The 128-byte record

Each record is exactly 128-byte aligned and split into four independently useful 32-byte sectors:

byte  0                     32                    64          96          128
      | header + ontology   | inline adjacency   | hot K     | hot V       |
      | 16 B  + 16 B mask   | 8 x int32 delta    | 16 x FP16 | 16 x FP16  |
      +---------------------+---------------------+------------+-------------+

Offset	Bytes	Field	Purpose
0	16	Header	Node ID, overflow offset, degree, flags, external KV block ID
16	16	Ontology mask	128 hot ontology classes
32	32	Adjacency	Eight signed deltas to topology-reordered records
64	32	Key micro-tile	Sixteen FP16 values
96	32	Value micro-tile	Sixteen FP16 values

static_assert checks lock the size, alignment, and every sector boundary. A topology-aware BFS places connected and ontologically similar records near one another. Nodes above degree eight spill into an indexed overflow pool; the file format never stores raw host or device pointers.

What the layout guarantees

• A SemanticCacheLine is exactly 128 bytes with 128-byte alignment.
• Every record in its array begins on a 128-byte boundary under C++17's over-aligned allocation rules (and CUDA allocations are sufficiently aligned).
• One warp can cooperatively read its 32 words from one contiguous 128-byte address region.
• CPU and CUDA benchmark variants validate equivalent checksums.

What it does not guarantee

On current NVIDIA GPUs, 128 bytes is not one indivisible DRAM operation. NVIDIA documents coalescing in 32-byte transactions for compute capability 6.0 and newer, while Nsight Compute models a 128-byte cache line as four 32-byte sectors. A cooperative aligned warp-load pattern can form one L1TEX request touching four 32-byte sectors; downstream transactions still depend on cache hits and misses. Cache state, instruction selection, divergence, and architecture determine the actual traffic. See the CUDA Best Practices Guide and Nsight Compute Profiling Guide.

The 64-byte KV payload is a hot micro-tile, not a full token's KV cache. A real model's KV state spans many records or falls back to a paged/contiguous warm tier. Physical HBM placement and cache residency are not controlled by this C++ layout.

What is included

• Header-only, offset-checked 128-byte record ABI for little-endian hosts
• Deterministic hot-root/topology/ontology-aware packer
• Inline signed record references plus degree overflow pool
• Versioned, pointer-free .scl binary format
• pack, inspect, and CPU functional-benchmark commands
• Opt-in CUDA packed-versus-fragmented load microbenchmark
• Layout, codec, overflow, round-trip, and checksum-equivalence tests
• Nsight Compute evaluation protocol and falsifiable success criteria

Quick start

Requires CMake 3.20+ and a C++20 compiler. CUDA is optional.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
ctest --test-dir build --output-on-failure

./build/scl pack examples/tiny_graph.tsv /tmp/tiny.scl --root 100
./build/scl inspect /tmp/tiny.scl --record 0
./build/scl benchmark --nodes 32768 --queries 100000 --hops 12

The CPU benchmark is a correctness and locality sanity check. Its timing is not evidence of H100 behavior and is labeled accordingly in the output.

Input format

scl pack accepts a deliberately simple, dependency-free TSV format. Lines starting with # are ignored.

node_id<TAB>heat<TAB>kv_block_id<TAB>ontology_csv<TAB>neighbors_csv<TAB>key_csv<TAB>value_csv

Use - for an empty list. IDs must be unique; neighbors must reference nodes in the same file; ontology IDs are in [0, 127]. kv_block_id is an opaque key into an external model-coordinate directory; v1 does not encode token/layer/head/dimension coordinates itself. More than eight neighbors spill to the overflow pool. More than sixteen values on either KV side are rejected unless --allow-truncate is passed, in which case the record and CLI output are explicitly flagged.

CUDA microbenchmark

With CUDA Toolkit 12+:

cmake -S . -B build-cuda -DSCL_ENABLE_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-cuda -j --target scl_cuda_bench
./build-cuda/scl_cuda_bench --records 1048576 --iterations 100 --warmups 10

The CUDA harness maps one warp to one synthetic 32-word packet. It measures (1) a cooperative contiguous load, (2) the same co-located packet through four partitioned load paths, and (3) those same paths over four separate allocations. Variants two and three execute the same kernel, isolating address placement from branch/load structure. Every full per-query output is checked against a lane-sensitive CPU reference. This is explicitly a packet-load control: it does not yet traverse a graph, evaluate ontology predicates, or perform KV arithmetic.

Profile it rather than inferring cache behavior:

ncu --kernel-name regex:cooperative_packet_kernel \
    --launch-skip 10 --launch-count 1 \
    --section SpeedOfLight \
    --section MemoryWorkloadAnalysis_Tables \
    --section WarpStateStats \
    ./build-cuda/scl_cuda_bench --variant cooperative \
      --records 1048576 --iterations 1 --warmups 10

Run packet-partitioned and split separately with --variant; both use partitioned_packet_kernel. Use Nsight's gpu__time_duration while profiling, not the application's CUDA-event timings, because replay/tool overhead changes those timings. Inspect SASS to confirm the intended load instruction structure.

The controlled benchmark ladder and suggested counters are in docs/benchmarking.md.

Research direction

The record is the smallest unit. A production design should group 32 records in a 4 KiB semantic page keyed by community, layer, KV head, and dimension block; compact active frontiers by page; stage records cooperatively; apply ontology predicates; and only then batch surviving KV fragments into dense MMA-friendly tiles. Merely placing a mask beside KV bytes does not feed Tensor Cores or remove multi-hop graph indirection.

See docs/design.md for the complete memory model, invariants, failure modes, and roadmap.

Status

Version 0.1 is a testable systems hypothesis. No speedup is hard-coded or claimed. A result counts as a win only if it improves end-to-end latency or the ratio of useful to transferred bytes against a topology-reordered CSR/paged-KV baseline, not only against a deliberately fragmented pointer baseline. Negative results are valid results.

License

Apache-2.0. See LICENSE.