Hunch is a small, compiled, RL- and self-play-native programming language. You write the rules of a world and its reward signal, and leave hunch(...) blanks wherever decisions happen. A world is just an environment: a simulation in a few dozen lines. Then hunch learn fills in the blanks from experience, requiring no human examples, no training pipeline, and no separate simulator.
Because Hunch packs the entire world state (including its deterministic PRNG) into a single flat C struct, the built-in runtime can freeze, copy, and fork execution mid-program in nanoseconds, enabling counterfactual rewind-and-fork reinforcement learning and zero-human-data self-play directly inside the language. You set the goal, Hunch finds the strategy.
world Dungeon
state { x: int = 1; y: int = 1; has_key: int = 0; }
step {
let a = hunch hero([x / 6.0, y / 6.0, has_key], 4);
...
}
That is the whole interface. Everything else is inside the language: the environment loop, the replay buffer, the value function, the optimiser, self-play.
Two things are true at once right now. 1) Building a reinforcement-learning system is still mostly plumbing. You write a simulator in one language, then duct-tape a training stack around it: wrappers, replay buffers, reward shaping scripts, evaluation harnesses. 2) Credit assignment is still done with a blunt instrument. Practitioners hand the algorithm a finished episode and a scalar, and the algorithm has to guess which of forty decisions caused the outcome.
Hunch attacks both. Writing the rules is the setup. And because the language owns the entire state of the world, it can rewind to an exact instant, replay it once per action, and measure the difference. Credit assignment stops being a guess and becomes a controlled experiment.
./bin/hunch check worlds/dungeon.hunch # compile, print shape
./bin/hunch train worlds/dungeon.hunch --algo fork --budget 400000
./bin/hunch studio # interactive playgroundNo pip install or npm install. You need python3 and a C compiler; the
bytecode compiler is pure Python and the runtime is one C file.
Four worlds, each trained from scratch on identical environment-step budgets
across 5 seeds. Return is scored on a fixed stream of evaluation episodes
shared by every algorithm, so nobody wins on luck. mean ± std over seeds.
Find a key, avoid a trap, open the exit. The payoff arrives a dozen decisions after the ones that caused it.
| algorithm | return |
|---|---|
| random | −0.069 |
| REINFORCE | 0.750 ± 0.394 |
| PPO | 1.1251 ± 0.0007 |
| hunch fork | 1.1196 ± 0.0062 |
Both learned methods are at the ceiling here (an optimal hand-played run scores 1.126), so this world measures stability rather than skill.
Quote a market while carrying inventory. Reward is equity change minus a charge for risk. Noise is the enemy: two branches of the same decision see different coin flips unless you hold the randomness fixed.
| algorithm | return |
|---|---|
| random | 1.411 |
| REINFORCE | 0.841 ± 1.915 |
| PPO | 2.2520 ± 0.0546 |
| hunch fork | 2.1816 ± 0.2089 |
Two agents, hidden cards, zero human data. Raw return is meaningless here because both sides improve together, so the honest test is against opponents that cannot adapt:
| algorithm | raw return | vs. always-fold | vs. always-call |
|---|---|---|---|
| random | 0.385 | n/a | n/a |
| REINFORCE | 0.092 | 0.694 | 0.258 |
| PPO | 0.094 | 0.296 | 0.172 |
| hunch fork | 0.275 | 0.599 | 0.193 |
This is the one world where the gap is not close: fork scores roughly three times PPO's raw return and twice its score against a fixed opponent.
The policy shipped in policies/duel.hpol does much better than the five-seed
average: it scores +1.00 against a player who always folds and −0.006
against one who always calls. That pair of numbers is the signature of a
balanced bluffing strategy: it takes the maximum from a folder and gives
nothing away to a caller. Self-play reaches it on some seeds and not others,
which is the honest state of the art here.
At equal budget the two methods are within 1.1% of each other: PPO 31.7979 ± 0.5589, fork 31.4438 ± 0.2553. The interesting comparison is not PPO at all but the textbook policies:
A four-slot cache on a workload with hot keys, scan traffic, and a hot set that moves halfway through. Replayed on the same 96 evaluation episodes:
| policy | hits / 60 | hit rate |
|---|---|---|
| LRU | 27.86 | 46.4% |
| LFU | 28.10 | 46.8% |
| hunch (trained) | 31.74 | 52.9% |
| Belady OPT (clairvoyant) | 36.09 | 60.2% |
The agent was never told what LRU is. It beats both classic baselines by 3.6–3.9 hits per episode, a 14% relative improvement in hit rate, and gets 88% of the way to an oracle that can see the future.
PPO is an extremely strong baseline. On the three single-agent worlds it still edges fork by 0.5–3% on mean return at these budgets. If the claim were "fork beats PPO everywhere", this would be a failure. Here is what is actually true.
Fork wins outright in self-play. On the duel it scores 0.275 against PPO's 0.094 (nearly 3×) and 0.599 against a fixed opponent where PPO manages 0.296. That is not a tuning artifact, it is the mechanism: when the return is dominated by an opponent's hidden choices, a sampled advantage is mostly opponent noise. A counterfactual branch holds the opponent fixed, so the measured difference between your actions is causal. GAE cannot do that, because it only ever sees the one opponent action that happened.
The rework that got there. The first version of this system had fork at 0.885 on the dungeon, 2.019 on the market, 30.0 on the cache and 0.133 on the duel. Three changes moved it:
| change | why |
|---|---|
anneal the fork budget (--fork-ratio 1.5 --fork-decay 0.5) |
forking buys credit assignment, and it is worth most while the critic is still wrong. Spend it in the first half, then get out of the way. |
| bound the branch horizon (24 steps) and bootstrap from the critic | a branch that replays to the end of the episode costs more than it is worth. The exact return-to-go of the action actually taken is free, so only the alternatives are rolled. |
inject the exact advantage into the main trajectory (--fork-inject 1) |
two optimiser passes fighting over one policy is worse than one coherent update. The counterfactual advantage overwrites the sampled one at forked states. |
That took fork to 1.120 / 2.182 / 31.44 / 0.275. The dungeon's worst seed went from −0.09 to 1.122.
Fork is faster in wall-clock at equal step budgets: 7.4s vs 8.7s on the dungeon, 11.4s vs 13.2s on the market, because forked rollouts need no backpropagation passes.
The most useful thing fork produces may be the explanation. The same mechanism that computes counterfactual advantages is what powers the Studio's "rewind moments": a replay of the exact instant, once per action, with the measured value of each.
The obvious prediction is that wider action spaces favour forking. A sampled advantage gives the learner one action out of A; a counterfactual gives all A exactly. So fork should get better as A grows.
It does not. Hold the market's economics completely fixed and change only the resolution of the (width, skew) quote grid:
| actions | PPO | fork | gap |
|---|---|---|---|
| 4 | 2.078 | 2.051 | −1.3% |
| 8 | 2.077 | 2.115 | +1.8% |
| 16 | 2.213 | 2.125 | −4.0% |
| 32 | 2.195 | 1.915 | −12.8% |
(python3 benchmarks/action_width.py, 5 seeds, identical budgets.)
The reason is arithmetic, not statistics. Forking a state costs O(A), one branch per action, so the budget buys fewer and fewer forks as A grows. At 32 actions a forked state costs 768 environment steps, which is a third of an entire training iteration. The information per fork goes up linearly; the number of forks you can afford goes down linearly; they cancel, and then the fixed cost of the branches wins.
Put together with the duel result, that gives a fairly sharp picture of the niche: fork pays when the action space is small enough to enumerate cheaply and the return is noisy for a reason the critic cannot see. The duel is both (A=2 and A=3, with an opponent). The market and the cache are neither.
What did not work. Three things I tried and rejected, in case they are useful to someone else: (a) keeping branch decisions as extra training data made the dungeon collapse on one seed: the harvested batch is too correlated with itself; (b) training only the critic from counterfactual targets, without the policy gradient, was clearly worse (0.81 vs 1.12), so the counterfactual policy gradient is doing the real work; (c) raising the market's price noise to make it more realistic did not favour fork: both methods degraded and PPO stayed ahead, so the self-play result is about opponents, not volatility.
This is a working system with an honest, specific result: comparable to a strong PPO baseline on single-agent tasks, clearly better when there is an adversary, and worse as the action space widens. The four worlds were chosen to be different from each other, not to make any one method look good, and the action-width sweep, which I expected to be fork's best case, turned out to be its worst.
Every world's state lives in one flat block of memory: all its variables, its
random number generator, its step counter. Snapshotting the universe is a
single memcpy of about 2 KB. Nothing else in the runtime has to cooperate.
That single fact is what makes the rest cheap:
- Record. While an episode runs, save the state block before every decision.
- Select. Rank the recorded decisions by policy entropy plus surprise
(
|reward − value|), and take the ones that were actually hard. - Fork. For each selected instant, restore the block once per action, force that action, and roll forward. The random stream is part of the snapshot, so every branch sees the same coin flips. The noise cancels and the difference between branches is purely causal.
- Update. You now have
Q(s, a)for every action at that state. The policy gradient is exact:∂J/∂z_k = p_k · A_k. The action that was actually taken needs no rollout at all: its real continuation is already in the trajectory. - Bootstrap. A branch rolls a bounded horizon, then hands the tail to the critic, so a fork costs tens of steps instead of a whole episode.
Every step spent inside a branch counts against the same budget as a real environment step. The comparison above is therefore like for like.
./bin/hunch studio serves an interactive page with:
- Side-by-side replay of a never-trained policy and the trained one, on the same seed, for all four worlds.
- Rewind moments: the exact decision points where the widest forks happened, with the measured value of each action.
- Learning curves and the benchmark table above.
- Train live: recompile and retrain a world from the page in a few seconds.
- Play the bot: take Alice's seat in the duel (
--agent alice).
The complete one-page reference for humans and agents is llms.txt.
It is designed to be pasted into a model's context whole: the grammar, the
built-ins, the design rules for observations and rewards, a complete worked
world, and the CLI. A model that has read that page can write a new trainable world in about thirty
lines. That is the intended way to use this.
├── .github/workflows/tests.yml CI: compile every world, run the tests
├── bin/
│ └── hunch CLI entry point
├── hunch/ the language, in Python
│ ├── lang.py tokenizer and parser, with line/col errors
│ ├── codegen.py syntax tree to bytecode, plus the .hbc writer
│ ├── cli.py check, build, run, train, bench, studio
│ └── studio_server.py HTTP and JSON API behind the Studio
├── runtime/
│ └── hunch_vm.c the whole engine: VM, MLP, Adam, PPO, fork
├── worlds/ the four worlds, one .hunch file each
│ ├── dungeon.hunch key, trap, locked exit
│ ├── market.hunch quote skew and inventory risk
│ ├── duel.hunch hidden cards, pure self-play
│ └── cache.hunch four-slot eviction
├── studio/ the browser playground, no build step
│ ├── index.html
│ ├── styles.css
│ └── app.js renderers, playback, charts, live training
├── policies/ shipped trained policies
│ ├── dungeon.hpol
│ ├── market.hpol
│ ├── duel.hpol
│ └── cache.hpol
├── benchmarks/
│ ├── run_all.py five-seed comparison across all worlds
│ ├── results.json the numbers in this README
│ ├── cache_baselines.py LRU, LFU and Belady replay
│ ├── action_width.py the 4 to 32 action sweep
│ └── make_market_variants.py generates the action-width variants
├── tests/
│ └── test_compiler.py 14 language, runtime and learning tests
├── llms.txt one-page reference for humans and models
├── README.md
└── LICENSE
python3 tests/test_compiler.py # 14 tests
python3 benchmarks/run_all.py # the table above (~3 min)
python3 benchmarks/cache_baselines.py # LRU / LFU / Belady comparison- Everything trains on CPU. The network is a 32×32 MLP; this is not the place for large models. The speed comes from the worlds being tiny and the runtime being C.
- The fork tunables (
--fork-ratio,--fork-horizon,--fork-points) matter, and the defaults were chosen on these four worlds. They are not universal. - Self-play is seed-sensitive. The duel sometimes finds a balanced strategy and sometimes does not; there is no opponent pool yet.
- The language has no modules, no strings, and no recursion. That is a deliberate constraint, not an oversight: a world that fits in one page is a world a model can write correctly in one shot.