Adaptive challenge cost: price by suspicion, on the clock not the hash

.github/workflows/bench.yml

@@ -45,6 +45,13 @@ jobs:
       - name: End-to-end detection suite
         run: node test/test-detection.js
 
+      # The harness itself, before it is trusted to measure anything. Sample
+      # addresses have to stay unique or every per-source measurement is taken
+      # against manufactured IP sharing.
+      - name: Harness unit tests
+        working-directory: bench
+        run: npm test
+
       # Replays the committed corpus. No browser needed: capture is a separate,
       # manual step whose output lives in bench/corpus/.
       - name: Benchmark gate

.github/workflows/unit-tests.yml

@@ -0,0 +1,67 @@
+name: Unit tests
+
+# Go and Python unit tests, which nothing ran until now.
+#
+# Node's have been covered by the benchmark workflow since it existed, so the
+# gap was invisible from the outside: every check on a pull request was green
+# while two of the three implementations were unverified. Among the tests that
+# had never run in CI:
+#
+#   - TestOnlyJA4TLSIsImplemented, which is the only thing standing between this
+#     MIT project and a FoxIO License 1.1 module that cannot legally ship in it
+#   - TestWeightsSumToOne, which guards the scoring weights invariant
+#   - every Python test, on an implementation with no coverage but a container
+#     smoke test
+#
+# The sync rule says a change lands in all three servers. That is worth very
+# little if only one of them is checked.
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  go:
+    name: Go
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-go@v5
+        with:
+          # crypto/tls only exposes ClientHelloInfo.Extensions from 1.24, which
+          # native JA4 needs. go.mod says so; keep this in step with it.
+          go-version: '1.24'
+          cache-dependency-path: server-go/go.sum
+
+      - name: Vet
+        working-directory: server-go
+        run: go vet ./...
+
+      - name: Test
+        working-directory: server-go
+        run: go test -race ./...
+
+  python:
+    name: Python
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+          cache: pip
+          cache-dependency-path: server-python/requirements.txt
+
+      - name: Install dependencies
+        working-directory: server-python
+        run: pip install -r requirements.txt
+
+      # Discovery, not a loop over files: a file that stops being discoverable
+      # should show up as a drop in the count rather than as silence. It used to
+      # report 12 tests and two import errors where there are in fact 43.
+      - name: Test
+        working-directory: server-python
+        run: python -m unittest discover -p "test_*.py" -v

.gitignore

@@ -46,5 +46,9 @@ docs/
 # Bench harness: replay output and captured-corpus scratch. The corpus itself
 # (bench/corpus/) IS committed — it is the measurement baseline, and a benchmark
 # whose inputs are not in the repo cannot be reproduced or argued with.
-bench/*.json
+#
+# Narrow, because `bench/*.json` also swallowed package.json: the harness could
+# not be installed from a fresh clone, and CI could not run anything through npm
+# in that directory.
+bench/*results*.json
 bench/test-results/

README.md

@@ -293,8 +293,34 @@ to honest mobile users for no gain in the constraint that actually binds. Number
 in [bench/POW-PRIMITIVE.md](bench/POW-PRIMITIVE.md).
 
 If you want proof of work to genuinely raise an attacker's cost, the lever is the
-server-measured elapsed time, not the hash. That is what adaptive difficulty
-should key on.
+server-measured elapsed time, not the hash. That is what the adaptive cost keys
+on.
+
+Each challenge carries a `minAgeMs`: how long it must be held before a solution
+is accepted without penalty. A visitor who has done nothing wrong gets the same
+1500ms floor the server has always applied. A source that has recently produced
+strong bot verdicts gets more, up to 15 seconds — which takes it from roughly 40
+tokens a minute to four, on hardware no amount of money can speed up.
+
+Difficulty barely moves, and now caps at 5 rather than 6. Difficulty 6 costs a
+native solver about a millisecond and a budget Android phone about sixteen
+seconds; escalating there taxes the slowest legitimate devices and constrains
+nobody. Being on a datacenter address no longer raises difficulty at all — a
+real person on a corporate VPN or iCloud Private Relay was paying several seconds
+of blocked hashing for a shared IP — and raises the time floor instead, which a
+hosted scraper feels as reduced throughput and a person filling in a form does
+not feel at all.
+
+The client is told `minAgeMs` and waits it out, so for an ordinary visitor the
+cost is a short delay rather than a worse score. That distinction matters most
+for anyone sharing an egress address with whatever earned the delay. A client
+that submits early anyway is scored, but only as contributory evidence — an
+older cached client does not know to wait, and should not be treated as
+automation for it.
+
+Suspicion is held per (site key, address) for 15 minutes, records only verdicts
+at or above 0.8, and stores nothing but timestamps. It is not cross-session
+correlation and should not grow into it.
 
 ### Behavioral Biometrics
 - Mouse trajectory, velocity, and acceleration curves
@@ -381,6 +407,7 @@ Get a Proof of Work challenge. Called automatically by the client on page load.
   "challengeId": "abc123...",
   "prefix": "abc123:1703356800000:4",
   "difficulty": 4,
+  "minAgeMs": 1500,
   "expiresAt": 1703357100000,
   "nonce": "f1e2d3...",
   "sig": "def456..."
@@ -389,9 +416,22 @@ Get a Proof of Work challenge. Called automatically by the client on page load.
 
 The `nonce` is generated per-challenge by the server; the client echoes it back in `signals.meta.challengeNonce` and the server verifies it, preventing challenge replay.
 
-Difficulty scales based on:
-- Datacenter IPs: +1 difficulty
-- High request rate: +1 difficulty (max 6)
+`minAgeMs` is how long the client should hold the solved challenge before
+submitting. The bundled client does this for you. Both fields are covered by
+`sig`, so neither can be talked down on the way back.
+
+Cost scales with what the requesting source has recently been caught doing:
+
+| source | difficulty | minAgeMs |
+|---|---|---|
+| clean, or unknown | 4 | 1500 |
+| 1–2 recent strong bot verdicts | 4 | 4000 |
+| 3–5 | 5 | 8000 |
+| 6+ | 5 | 15000 |
+
+Datacenter addresses, high request rates and exceeded rate limits raise the time
+floor only — never the difficulty. See [what proof of work does and does not buy
+you](#what-the-proof-of-work-does-and-does-not-buy-you) for why.
 
 ### POST /api/verify
 Verify a checkbox CAPTCHA submission.
@@ -611,7 +651,7 @@ fcaptcha/
 └── README.md
 ```
 
-> All three servers implement the same detection engine and must stay in sync. The Go scoring is unit-tested (`go test ./server-go/...`); `test/test-detection.js` exercises the full pipeline against a running server.
+> All three servers implement the same detection engine and must stay in sync. Each has unit tests that run in CI, and `test/test-detection.js` exercises the full pipeline against a running server.
 
 ## Development
 
@@ -631,10 +671,24 @@ open demo/index.html
 
 ### Running Tests
 
-Go unit tests (no server required):
+Unit tests, no server required. All three run in CI on every pull request.
+
+```bash
+cd server-go     && go test -race ./...              # Go
+cd server-node   && npm test                         # Node
+cd server-python && python -m unittest discover -p "test_*.py"   # Python
+```
+
+The Python tests are plain functions collected by a decorator rather than
+`TestCase` methods, so `testkit.py` bridges them into `unittest` — run a single
+file directly (`python test_sitekeys.py`) for readable per-test output. Discovery
+and direct execution both report the same set; if the discovered count drops, a
+file has stopped being discoverable.
+
+The measurement harness has its own tests:
 
 ```bash
-cd server-go && go test ./...
+cd bench && npm test
 ```
 
 End-to-end detection suite (runs against a live server):

bench/README.md

@@ -116,14 +116,37 @@ sample and measures nothing but itself. `lib/pow.js` paces the solve and waits
 out the challenge age, so `duration` stays true wall-clock — a fabricated
 duration would be measuring the fabrication.
 
-**Every sample gets its own client IP.** The server escalates PoW difficulty per
-`pow:{siteKey}:{ip}`, so a few hundred samples from one address would spend the
-run solving 16.7M-hash challenges *and* pick up rate-limit detections that the
-earlier samples never saw — the measurement would drift with position in the
-corpus. Samples present a distinct RFC 5737 documentation address via
-`X-Forwarded-For`, which works because loopback is in the default trusted-proxy
-set. Agent classes that really do arrive from datacenter ranges can say so with
-`clientIp`.
+**Every sample gets its own client IP.** The server keys rate limits and
+accumulated suspicion per `pow:{siteKey}:{ip}`, so a few hundred samples from one
+address would pick up escalations the earlier samples never saw — the measurement
+would drift with position in the corpus. Samples present a distinct RFC 5737
+documentation address via `X-Forwarded-For`, which works because loopback is in
+the default trusted-proxy set. Agent classes that really do arrive from
+datacenter ranges can say so with `clientIp`.
+
+Those addresses are assigned **by position**, not by hashing the sample id, and
+that distinction was learned the hard way. Hashing put 180 samples into a
+508-address space; by the birthday bound they collided constantly — 25 shared
+addresses, 24 of them putting an agent and a human on the same IP. The corpus was
+manufacturing shared egress its labels never claimed, and nothing noticed for as
+long as no signal was keyed by source. The first one that was, read 12.7% FPR on
+a panel whose real answer was zero. `lib/replay.test.js` now holds the pool
+collision-free and throws rather than wrapping if the corpus outgrows it.
+
+**Each run namespaces its own server-side state.** The same per-source keying
+means a second run against a long-lived server measures the residue of the first.
+Measured: one signal fired on 2/75 agents against a fresh server and 74/75
+against one already benchmarked. Each run generates a unique site key, so runs
+are independent without the harness having to own the server's lifecycle.
+
+**The pacer does not honour the server's `minAgeMs`.** The server tells clients
+how long to hold a solved challenge and the shipped client obeys; the harness
+deliberately waits a fixed interval just past the universal baseline instead. A
+replayer that honoured the delay would satisfy the timing gate by construction,
+and the panel could never observe it firing on a human — the same trap as pinning
+a property during normalization and then reporting that nothing disagrees with
+it. Leaving the pacer fixed means any escalation applied to a human persona shows
+up as a signal over budget, which is what it should do.
 
 ---
 

bench/lib/replay.js

@@ -22,6 +22,22 @@
  * connection really does not. Samples may set `clientIp` to say so; anything
  * that does not gets a documentation-range address (RFC 5737), which no
  * detection treats as anything in particular.
+ *
+ * Those addresses are assigned by position and must stay unique — see
+ * neutralIpAt for what happened when they were hashed instead.
+ *
+ * ## Why the pacer does not honour the server's minAgeMs
+ *
+ * The server tells a client how long to hold a solved challenge, and the
+ * shipped client waits that long. This harness deliberately does not: it always
+ * waits a fixed interval just past the universal baseline.
+ *
+ * That is the conservative choice for measuring false positives. A replayer
+ * that honoured the delay would satisfy the timing gate by construction and the
+ * panel could never observe it firing on a human — the same trap as pinning a
+ * property during normalization and then reporting that nothing disagrees with
+ * it. Leaving the pacer fixed means any escalation applied to a human persona
+ * shows up as a signal over budget, which is exactly what it should do.
  */
 
 const crypto = require('crypto');
@@ -63,13 +79,46 @@ function distinguishDevice(signals, id) {
   };
 }
 
-/** RFC 5737 TEST-NET-2 and TEST-NET-3: reserved, routable nowhere, in no CIDR list. */
-const NEUTRAL_PREFIXES = ['198.51.100', '203.0.113'];
+/** RFC 5737 TEST-NET-1, -2 and -3: reserved, routable nowhere, in no CIDR list. */
+const NEUTRAL_PREFIXES = ['192.0.2', '198.51.100', '203.0.113'];
+
+/** .0 and .255 are the network and broadcast addresses; keep to 1-254. */
+const NEUTRAL_POOL_SIZE = NEUTRAL_PREFIXES.length * 254;
+
+/**
+ * The address for the Nth sample in a corpus.
+ *
+ * Assigned by position rather than by hashing the sample id, and that is the
+ * whole point. Hashing put 180 samples into a 508-address space, which by the
+ * birthday bound collided constantly: 25 shared addresses, 24 of them putting
+ * an agent and a human on the same IP. The corpus was manufacturing shared
+ * egress that its labels never claimed, so any per-source signal — rate limits,
+ * reputation, accumulated suspicion — was being measured against a fiction.
+ *
+ * Nothing caught it until a per-source signal existed to notice, which is the
+ * same shape of mistake as normalization pinning a property the panel then
+ * could not disagree about. Assign by index and the collisions cannot come back.
+ */
+function neutralIpAt(index) {
+  if (index >= NEUTRAL_POOL_SIZE) {
+    throw new Error(
+      `corpus has outgrown the neutral address pool (${NEUTRAL_POOL_SIZE} addresses). ` +
+        'Add another RFC 5737 prefix rather than letting samples share an address — ' +
+        'shared addresses silently break every per-source measurement.'
+    );
+  }
+  return `${NEUTRAL_PREFIXES[Math.floor(index / 254)]}.${(index % 254) + 1}`;
+}
 
+/**
+ * Address for a single sample replayed outside a corpus run.
+ *
+ * Collision-prone by construction — see neutralIpAt. Fine for a one-off, wrong
+ * for a panel.
+ */
 function neutralIpFor(id) {
   const digest = crypto.createHash('sha256').update(id).digest();
   const prefix = NEUTRAL_PREFIXES[digest[0] % NEUTRAL_PREFIXES.length];
-  // .0 and .255 are the network and broadcast addresses; keep to 1-254.
   return `${prefix}.${(digest[1] % 254) + 1}`;
 }
 
@@ -134,6 +183,13 @@ async function replaySample(serverUrl, sample, opts = {}) {
  */
 async function replayCorpus(serverUrl, samples, opts = {}) {
   const concurrency = opts.concurrency || 4;
+
+  // Give every sample its own address before anything is replayed, so no two
+  // samples share server-side per-source state.
+  samples.forEach((sample, i) => {
+    if (!sample.clientIp) sample.clientIp = neutralIpAt(i);
+  });
+
   const results = new Array(samples.length);
   let next = 0;
   let done = 0;
@@ -152,4 +208,12 @@ async function replayCorpus(serverUrl, samples, opts = {}) {
   return results;
 }
 
-module.exports = { DEFAULT_UA, distinguishDevice, neutralIpFor, replayCorpus, replaySample };
+module.exports = {
+  DEFAULT_UA,
+  distinguishDevice,
+  neutralIpAt,
+  neutralIpFor,
+  NEUTRAL_POOL_SIZE,
+  replayCorpus,
+  replaySample,
+};