ai::NeuralNetwork: five hidden layers train and six do not, and none of the three fixes that moved the wall explains the last one

[xoloki]

Split out of #131, which closed today: its two prescriptions -- fan-in
initialisation (#133) and an activation that travels with its derivative
(#134) -- are in, and with #130's fix (#135) depth now trains through five
hidden layers where it used to stop at two.

Six does not, and none of the explanations that moved the earlier walls covers
it.

Measured at HEAD

MNIST 60k/10k, 64-wide hidden layers, relu hidden with a sigmoid output, five
epochs, default seed.

  1 x 64   96.89%  (rate 0.01)
  2 x 64   95.83%  (rate 0.001)
  3 x 64   94.13%  (rate 3e-4)
  4 x 64   94.36%  (rate 3e-4)
  5 x 64   94.05%  (rate 3e-4)
  6 x 64    9.80%  -- chance, at every rate swept

The six-layer sweep, which is the whole of the evidence that this is a wall
rather than a step size:

  0.01    9.8%     0.003   9.8%     0.001   9.8%
  3e-4    9.8%     1e-4    9.8%     3e-5   10.1%

What makes it a real question rather than an untried rate

Five layers has a narrow window, and that is new:

  5 x 64, rate 0.001   ->  9.8%   chance
  5 x 64, rate 3e-4    -> 94.05%
  5 x 64, rate 1e-4    -> 71.79%

A factor of three either side of 3e-4 is the difference between working and
not. Six was swept at both 3e-4 and 1e-4 and gave chance at both, so this is
not simply a rate nobody tried -- but a window narrower than the sweep,
somewhere between them, is not ruled out and is the cheapest thing to check
first. Each run is about a minute on the host backend.

What was already ruled out, and still is

From the measurements on #131, taken when the wall was at four:

• Dead units. leaky_relu tracks plain relu exactly at depth; if units were
  dying, leaky would rescue them.
• He initialisation. sqrt(2/fan) instead of sqrt(1/fan) gave identical
  results, including the same figure at the depth that worked.
• Training time. Chance at two epochs and at ten.
• Learning rate, at the then-failing depth, across eight values.

And the thing that makes it interesting now

#141: the error rule here is Rashid's shortcut, not backpropagation, and it was
measured as better at depth than the textbook rule -- 95.22% against 73.53%
at three hidden layers, 84.27% against 11.35% at five. So the wall is being hit
by the more forgiving of the two rules; correct backpropagation dies sooner
in the same network.

That points away from "the backward pass is wrong" -- the direction three
previous fixes came from -- and toward what #141 names as the thing worth
trying instead: residual connections, or normalisation between layers,
which let a gradient survive depth rather than routing around the problem.

What to measure next, in order

1. Rates between 3e-4 and 1e-4 at six layers. Cheap, and it either dissolves
   this issue or establishes the wall properly.
2. Which failure this is. 9.8% and 10.1% are what a network emitting a
   constant looks like, and that is a different failure from one that learns
   nothing -- so: is the output constant, and if so which digit?
3. Re-run Deep networks barely train: vanishing gradients through the sigmoid, and an initialisation that uses fan-out where it wants fan-in #131's per-layer largest-weight-change measurement at six layers. It
   is the direct evidence of whether the signal reaches the input end, and it
   was taken before any of the three fixes landed, so the numbers on that issue
   no longer describe this code.

Only then a change -- and with ai::backend in place, #141 records that
implementing an alternative rule was thirty lines against one interface.