Numerics Explained

An animated explainer of number formats, rounding and quantisation for machine learning: how a floating-point code becomes a value, round to nearest against stochastic rounding, why long sums go wrong in low precision, the formats zoo (FP8 E4M3 and E5M2, BF16 and FP16, the MX block formats), integer quantisation, activation outliers (LLM.int8() and SmoothQuant), GPTQ, AWQ and NF4, KV-cache quantisation measured on a live tiny transformer, and what the bits buy in hardware. Every chapter is built around an animation, and every frame of every animation is computed by a small numerics library that reproduces numpy and ml_dtypes bit for bit and whose TypeScript port matches its Python reference exactly.

It is the fifth of a family of companion sites: the Transformer Decoder Explainer shows one forward pass, LLM Inference Explained shows how a model is served, LLM Architectures Explained shows how the models differ, GPU Kernels Explained shows how a GPU runs the maths, this site is about the numbers themselves, Systolic Arrays Explained is the silicon underneath, and Inference Trade-offs Explained measures which serving lever (FP8 and INT4 among them) helps which metric. They share one design system and link to each other from the header, in four tracks, and the header always shows where you are ("Track › Site"): "Foundations" (Decoder · Training: one small model, end to end), "Hardware" (Kernels · Systolic · Accelerators · Racks: the machines underneath, chip to rack), "Inference" (Serving · Architectures · Numerics · Trade-offs; Serving is LLM Inference Explained) and "Agents", which starts with Agent Harnesses Explained (the loop, tools, context and permissions that turn a model into an agent; then Agent Protocols Explained, MCP and A2A on the wire; then Agent Context Explained, retrieval, memory and context engineering; then Agent Orchestration Explained, graphs, multi-agent patterns and durable execution; then Agent Evals Explained, measuring agents and how sure to be; then Agent Security Explained, defences that keep agents safe).

Live: numerics-explained.vercel.app

Part of

This project sits in the LLMs hub, next to the Transformer Decoder Explainer, LLM Inference Explained, LLM Architectures Explained and GPU Kernels Explained. The chapters link the matching slides of the Local LLM Hosting, Google TPU, NVIDIA GPU and Linear Algebra for AI series.

Chapters

#	Chapter	The animation
01	Bits to numbers	Click the bits of a code (eight formats) and watch the value; the number line zooms towards zero and every representable value crowds in.
02	Rounding	Inputs rounded to nearest-even and stochastically, error histograms building up; a running sum that nearest-even can never move.
03	Accumulation error	8,192 numbers summed naively in FP16 (it stalls at 2,048) and BF16, in FP32, with Kahan's compensation and pairwise; running totals and errors.
04	The formats zoo	A probe value swept across six formats' ranges (underflow, subnormals, overflow, saturation); an MX block converted element by element with its shared scale.
05	Quantisation basics	A weight matrix on a heat map quantised per tensor, per channel and per group, with the error fading; absmax against zero-point on skewed activations.
06	The outlier problem	Two outlier channels wreck per-tensor and vector-wise INT8; LLM.int8() splits them off. SmoothQuant's α swept from 0 to 1, moving the difficulty into the weights.
07	GPTQ	A layer quantised one column at a time, each column's error pushed into the columns still to come; output error against plain rounding.
08	AWQ and NF4	NF4's code book rebuilt from normal quantiles; AWQ's α search; the tiny transformer with its weights in ten formats, logit drift and agreement measured live.
09	Quantising the KV cache	The tiny transformer's keys and values in FP8, INT8, FP4 with a scale per 16, INT4 and INT2, per token or per channel: memory against logit drift.
10	Quantisation in hardware	One dot product in FP32, FP16, INT8 and MXFP4, with the energy of each multiply-accumulate (Horowitz, 45 nm); the energy and area table.

Animations

Recorded frame by frame from the model's states by pnpm animations (GIF here; a WebM of each is next to it in docs/media/).

Screenshots

Regenerate them with pnpm build && pnpm start in one shell and pnpm screenshots (or pnpm animations) in another.

The numerics library

reference/numerics.py is plain Python (no numpy), so that every step is one IEEE double operation; src/lib/num/model.ts repeats it operation for operation. It implements:

• Formats: FP32, FP16 (IEEE 754-2019), BF16, FP8 E4M3 and E5M2 (OCP OFP8 1.0), FP6 E3M2 and E2M3 and FP4 E2M1 (OCP MX 1.0): decode, and encode from a double in every rounding mode IEEE 754 defines (ties to even, ties away, towards +∞, −∞ and zero) plus stochastic rounding, with IEEE 7.4 overflow or OCP's saturating mode (E4M3 overflows to NaN, FP6 and FP4 always saturate).
• MX block formats (MXFP8, MXFP6, MXFP4, MXINT8): a shared E8M0 scale per 32 elements by the MX specification's section 6.3.
• NF4: bitsandbytes' code book, blockwise absmax over 64.
• Integer quantisation: symmetric (absmax) and asymmetric (zero-point), per tensor, per channel and per group.
• Summation in a format: naive, Kahan and pairwise, every addition rounded to the format.
• GPTQ (one column at a time, exact, on a small layer), AWQ-style scaling and SmoothQuant migration; the exponents α are multiples of 1/8, computed as products of square roots, so Python and TypeScript agree bit for bit (a general pow does not).
• Chapters 6–10: LLM.int8()'s mixed-precision decomposition (threshold 6) and vector-wise INT8, SmoothQuant's α sweep, GPTQ's column-by-column states, NF4's derivation (the normal CDF as a power series, its inverse by bisection), NF4 against INT4, and one dot product in FP32, FP16, INT8 and MXFP4 with energy per operation from Horowitz's 45 nm table.
• The state sequences the animations draw (*_steps); a frame is a pure function of one state.

The tiny transformer

reference/tiny.py and src/lib/num/tiny.ts quantise the Transformer Decoder Explainer's model: its TypeScript is vendored byte for byte into src/lib/transformer/ from commit 0049cf3 (MIT), and tiny.py is a plain-Python port of it. The hooks fake-quantise the six linear layers of each block (ten formats, GPTQ calibrated on four other prompts) or the keys and values (eight formats), and the measurements run the prompts teacher-forced: relative RMS logit drift, largest logit change, and how many of the 128 top-1 predictions are unchanged. Since brief 41 the chapters default to the trained model: the weights the Training site's default run ends with (public/data/trained-weights.json, vendored from transformer-explainer's public/training/default-run-weights.json at 984e647, sha256 bd44e460…), with its learned LayerNorm gains and FFN biases, on 16 eight-character prompts from the book's held-out tenth (GPTQ calibrates on 16 windows of the training part); fixtures tests/fixtures/tiny_trained_fixtures.json and src/data/tiny_trained.json. The random model stays one click away (four 32-character prompts; the explainer's N(0, 0.02) draws rescaled, linears to 1/√fan-in and the embedding to N(0, 1), so the blocks carry a trained model's share of the signal), and its fixtures are unchanged. A unit test proves the hooked forward pass equals the explainer's forwardTyped exactly; Python and TypeScript share exp, log, sin, cos and tanh, so the fixtures match to a relative 1e-12 (exactly for top tokens and agreement counts). In the browser the runs are computed in pieces, one forward pass per task, so the page stays responsive.

Checked

• tests/python/test_numerics.py (100 tests) compares the reference with numpy (every FP16 code; FP16 and FP32 conversion from doubles) and ml_dtypes (every BF16, FP8, FP6 and FP4 code; tens of thousands of conversions, ties and overflows included); checks every rounding and saturation mode against an oracle that lists the format's values and decides in exact rational arithmetic; checks the specifications' tables (OFP8 Tables 1–3, MX Tables 2–7, worked MX conversions), bitsandbytes' NF4 thresholds and the NF4 table's derivation from normal quantiles; and compares the integer quantisers, the three summations (against numpy's own float16/float32 and ml_dtypes' bfloat16 arithmetic) and GPTQ (against its OBS form in numpy) with independent implementations.
• tests/python/test_chapters.py checks chapters 6–10 and the tiny model against numpy re-implementations (vector-wise INT8, the decomposition, SmoothQuant's identity, NF4 against INT4, the FP32/FP16/INT8 dot products, a numpy forward pass of the tiny transformer with and without a quantised KV cache) and the standard library's NormalDist (CDF and quantiles, so NF4's derivation).
• scripts/make_fixtures.py writes the reference's results; tests/unit/model.test.ts requires the port to reproduce every value exactly: every code of every format of 16 bits or fewer and every encode case in all six modes and both saturation modes (through SHA-256 digests of the IEEE bit patterns), and every animation state. CI fails if the fixtures are out of date.
• Frame tests: tests/unit/frames.test.ts and tests/e2e/frames.spec.ts set key frames of every animation and require the state, and the caption on the page, to match the ones built from the Python reference's state.
• Numbers in the chapters are printed from the library at build time (<V of="e4m3.max" />), every code block shown is cut from the library's source (content.test.ts), and numbers the prose states in words are recomputed in values.test.ts.

Illustrative, and labelled so on the site: the data sets (the weight matrix, the activations, the MX blocks and the vectors summed come from a seeded integer generator; "normal" values are sums of 12 uniforms), the bits-per-weight accounting (FP16 scales; a zero point of the weight's width), and the FP16/BF16 sums rounding after every addition, as a scalar loop does (GPU kernels accumulate in FP32, in their own order). Chapters 6–10 add: the outlier layer (two channels near ±20), the GPTQ/AWQ layer, the tiny transformer (random, rescaled weights; embeddings kept at full precision; per-channel bit counts for a 16-input channel), KIVI-style per-channel keys scaled over the whole 32-token prompt (KIVI groups over windows and keeps recent tokens in full precision), the FP4 KV layout of DeepSeek-V4.1-Flash with this site's own rounding of its scale, AWQ in eighths of α (the paper searches 20 points), and the 45 nm energy figures.

One known difference from ml_dtypes: ml_dtypes converts a double to BF16 or FP8 through FP32, so a double close to a tie can round twice; this library rounds once, as IEEE 754 specifies. The tests compare with ml_dtypes from FP32 inputs, where both are exact. For a NaN input to FP6 or FP4, which have no NaN, the MX specification leaves the result to the implementation: this library gives +0 (ml_dtypes gives −0).

The animations

Every animation follows the family's visual standard: play and pause, step back and forward, a scrub bar, speed from 0.25× to 4× and reset; Space and the arrow keys when it has focus; a live caption per step, also in an aria-live region; no auto-play with prefers-reduced-motion; paused when scrolled out of view; the equation beside the picture, with the term the animation is on highlighted. Colours are Okabe and Ito's colour-blind-safe palette, the same in light and dark mode: one colour per bit field (sign purple, exponent sky blue, mantissa green, shared scale orange) and per method (nearest-even blue, stochastic orange). The pictures are plain SVG drawn by React from the library's states (no animation library), and every SVG label renders at 11 px or more on a 390 px phone.

Stack

• Framework: Next.js 14 (App Router) + TypeScript (strict)
• Styling: Tailwind CSS, Tailwind plugin for ESLint + Prettier
• Content: MDX via next-mdx-remote, KaTeX rendered on the server
• Library: Python reference, TypeScript port, JSON fixtures
• Testing: pytest (numpy, ml_dtypes), Vitest (exact parity, frames, content; 100% line coverage on src/lib/num/), Playwright (every page at 1280 and 390 px, light and dark; every animation's controls; frame tests; reduced motion; axe-core scans)
• CI / deploy: GitHub Actions (library and fixtures, lint, typecheck, unit, e2e, Lighthouse), Vercel

No database and no sign-in: every page is statically rendered, and each animation is a code-split client component.

Design system: where each piece came from

Copied from gpu-kernels-explained at commit 7242143, which copied it from the other companion sites:

Here	From
tailwind.config.ts, src/app/globals.css, src/app/layout.tsx	identical apart from titles; globals.css has this site's equation-highlight keys
src/components/ui/SiteHeader.tsx, SiteSwitch.tsx	the same header and four-track switch, with Numerics live
src/components/anim/, src/lib/anim/clock.ts, src/components/viz/useSvgFont.ts, src/components/mdx/Eq.tsx	unchanged
src/lib/viz/palette.ts	the family palette, plus this site's bit-field and method colours
src/app/learn/, src/lib/mdx/, Layer.tsx, LayerToggle.tsx, MdxTable.tsx, Controls.tsx, Callout.tsx	copied
.eslintrc.json, .prettierrc.json, tsconfig.json, vitest.config.ts, playwright.config.ts, lighthouserc.json, .github/workflows/ci.yml	adapted (library job, new pages)
scripts/smoke-check.ts, scripts/capture-screenshots.ts, RUNBOOK.md	adapted
src/lib/transformer/ (15 files)	transformer-explainer src/lib/transformer/ at 0049cf3, byte for byte
public/data/trained-weights.json	transformer-explainer public/training/default-run-weights.json at 984e647, byte for byte (brief 41)

A shared npm package for the design system would be cleaner in principle; for a handful of small sites, copying and recording the origin stays simpler.

Local development

• Node ≥ 20.11 and pnpm ≥ 9 (pinned via packageManager); Python ≥ 3.10.
• No environment variables, no database.

git clone https://github.com/BrendanJamesLynskey/numerics-explained
cd numerics-explained
pnpm install
python3 -m venv .venv && .venv/bin/pip install -r reference/requirements.txt
pnpm dev                              # http://localhost:3000

Changing the library

# edit reference/numerics.py, then the same change in src/lib/num/model.ts
.venv/bin/python -m pytest tests/python      # against numpy, ml_dtypes, the oracle, the specs
.venv/bin/python scripts/make_fixtures.py    # regenerate src/data and the fixtures
pnpm test                                    # the port must match exactly

Testing

.venv/bin/python -m pytest tests/python   # reference library
pnpm lint && pnpm typecheck && pnpm format:check
pnpm test:coverage                        # Vitest with thresholds (exact parity included)
pnpm test:e2e                             # Playwright on a production build (builds first)
pnpm lighthouse                           # Lighthouse CI on a `pnpm build`
pnpm smoke <url>                          # post-deploy check of every page
python3 scripts/check_links.py            # every external link, deck anchor and arXiv id

Deploying

See RUNBOOK.md: a CLI deploy from a clean git archive export, then pnpm smoke.

Project layout

content/chapters/     The MDX chapters, each opening with its animation
reference/            The Python numerics library and the tiny transformer
src/lib/transformer/  The Transformer Decoder Explainer's model (vendored)
scripts/              make_fixtures, smoke-check, capture-screenshots, check_links
src/app/              Routes: /, /learn, /learn/[slug], /formats, /about
src/lib/num/          The TypeScript library, the captions, the values the prose quotes
src/lib/anim/         The animation clock
src/data/             Formats and sources (generated by make_fixtures.py)
src/components/anim/  The animation hook and panel (controls, caption, equation)
src/components/interactive/  The chapters' widgets
tests/python/         pytest
tests/unit/           Vitest (fixtures in tests/fixtures/)
tests/e2e/            Playwright + axe-core

References

• IEEE Standard for Floating-Point Arithmetic, IEEE Std 754-2019.
• OCP 8-bit Floating Point Specification (OFP8) 1.0 and OCP Microscaling Formats (MX) Specification 1.0.
• Micikevicius et al., 2022 — FP8 Formats for Deep Learning; Rouhani et al., 2023 — Microscaling Data Formats for Deep Learning; Kalamkar et al., 2019 — A Study of BFLOAT16 for Deep Learning Training.
• Gupta et al., 2015 — Deep Learning with Limited Numerical Precision; Micikevicius et al., 2017 — Mixed Precision Training.
• Kahan, 1965 — Further remarks on reducing truncation errors, doi:10.1145/363707.363723; Higham, 1993 — The accuracy of floating point summation, doi:10.1137/0914050; Goldberg, 1991 — What every computer scientist should know about floating-point arithmetic, doi:10.1145/103162.103163.
• Jacob et al., 2017 — Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference; Nagel et al., 2021 — A White Paper on Neural Network Quantization; Dettmers et al., 2023 — QLoRA (NF4).
• Dettmers et al., 2022 — LLM.int8(); Xiao et al., 2022 — SmoothQuant; Frantar et al., 2022 — GPTQ; Frantar, Singh and Alistarh, 2022 — Optimal Brain Compression; Lin et al., 2023 — AWQ.
• Liu et al., 2024 — KIVI; Hooper et al., 2024 — KVQuant; DeepSeek-AI, 2026 — DeepSeek-V4.1-Flash (FP4 KV cache, section 2.4.4).
• Horowitz, 2014 — Computing's energy problem (and what we can do about it), ISSCC, doi:10.1109/ISSCC.2014.6757323, as tabulated in Gholami et al., 2021 — A Survey of Quantization Methods for Efficient Neural Network Inference, Figure 7.
• ml_dtypes (the reference for BF16, FP8, FP6 and FP4).
• Okabe and Ito, 2008 — Color Universal Design (the palette).

Contributing

PRs welcome. CI runs the library job (fixtures up to date, pytest), format:check, lint, typecheck, unit tests with coverage thresholds, e2e on a production build, and Lighthouse CI (performance, accessibility and best practices must each score at least 90 on /, /formats and five chapters).

Sharing

Every page carries Open Graph and Twitter tags, titled and addressed as that page. The link-preview cards (1,200 × 630 PNG) are drawn at build time by src/app/_og/card.tsx, the same file on every companion site, from this site's src/lib/share.ts: one for the site and one per chapter. The favicon (src/app/icon.svg) is the Inference track's hexagon with this site's letter, N; /sitemap.xml lists every page and /robots.txt allows everything. tests/e2e/sharing.spec.ts checks the tags, the cards, the icons, the sitemap and the landing page's start button, strip and hero. The cards are set in Noto Sans and Noto Sans Mono (subsets, SIL Open Font License 1.1: src/app/_og/OFL.txt).

Licence

MIT — see LICENSE.