Cache jit closures instead of re-tracing on every call - #137
Open
Codcore wants to merge 1 commit into
Open
Conversation
__jit__/5 forwarded straight to __compile__/4, which discards the key Nx
supplies and calls native_compile/3. That runs fun.(vars) and walks the
resulting expression on every call, only to look up a program already in
the program table.
Cache the closure __compile__/4 returns, keyed by that key plus the shapes
and types of vars and the device. On a hit the trace never runs.
The program table is untouched. Its structural key is what lets
structurally identical layers share one compiled program across different
call sites; this table sits above it and short-circuits repeat calls of
the same call site — the shape of every decode loop and training step.
Calls carrying a :command_queue or :hooks are not cached: those options
reference caller-owned state whose lifetime should not be extended by a
process-wide table.
Measured on a chain of cheap ops, where traversal rather than arithmetic
dominates:
depth before after Nx.Defn.compile
32 0.59 ms 0.43 ms 0.37 ms
128 1.01 ms 0.53 ms 0.38 ms
512 3.12 ms 1.18 ms 0.67 ms
1024 5.89 ms 1.85 ms 1.30 ms
On a 40-layer MoE decode step the same change removes 8.35 ms per token
out of 89.
polvalente
requested changes
Aug 20, 2026
| # `nil` means "do not cache": an unhashable key, or options carrying a | ||
| # caller-owned resource whose lifetime we must not extend. | ||
| defp jit_cache_key(key, vars, opts) do | ||
| if Keyword.has_key?(opts, :command_queue) or Keyword.has_key?(opts, :hooks) do |
Member
There was a problem hiding this comment.
I think hooks is always a given option. Not sure why this is being filtered on.
| templates = | ||
| vars | ||
| |> Nx.Defn.Composite.flatten_list() | ||
| |> Enum.map(fn %Nx.Tensor{shape: shape, type: type} -> {shape, type} end) |
Member
There was a problem hiding this comment.
You also want to take into account the vectorized_axes and the donation arguments. using Nx.Defn.Composite.traverse with Nx.to_template is likely a better conversion for the arguments. I think the options should also be taken into account because hooks might differ between calls
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #136.
__jit__/5forwarded straight to__compile__/4, which discards thekeyNxsupplies and calls
native_compile/3. That runsfun.(vars)and walks theresulting expression on every call — only to arrive at a program already sitting
in the program table.
This caches the closure
__compile__/4returns, keyed by thatkeyplus theshapes and types of
varsand the device. On a hit, the trace never runs.What is left alone
The program table and its structural key. That key is what lets structurally
identical layers share one compiled program across different call sites, and it
still does. The new table sits above it and short-circuits repeat calls of the
same call site with the same argument shapes — the shape of every decode loop and
every training step.
Calls carrying
:command_queueor:hooksare not cached: those optionsreference caller-owned state, and a process-wide table should not extend its
lifetime. Such calls take the previous path unchanged.
Measurements
A chain of
depthcheap ops, so the timing reflects graph traversal rather thanarithmetic (script in #136):
At depth 1024 that is 3.2× faster, and what remains between
jitandcompileis 0.55 ms rather than 4.48 ms.
On a real workload — a 40-layer Qwen3.6-35B-A3B MoE decode step, roughly 4300
nodes — the change removes 8.35 ms per token out of 89.
Tests
mix test: 2668 passed (825 doctests, 1843 tests), 7 excluded. No failures.A note on the trade-off
The cache is unbounded, like the program table it sits above. Each entry holds a
closure referencing a compiled program, so a caller that jits the same function
across many distinct argument shapes will accumulate entries. If that is a
concern, the same key would work with a bounded table; I kept it simple to match
the existing behaviour rather than introduce a second eviction policy.