The path from idea to working model required solving a chain of mathematical problems, each discovered through failed experiments. Every theoretical result was machine-verified in Lean 4 in collaboration with Aristotle (Harmonic AI), producing three formal proof files totaling 191 lines of verified mathematics.
Problem 1 — Depth Extrapolation Fails: The original hope was to train with 4 iterations but evaluate with 32, getting 'free depth' at test time. Systematic ablations (22 experiments) showed this makes things worse, not better. The model optimizes its output at exactly the training depth, not the fixed point. Aristotle identified the root cause: gradient vanishing at the fixed point (Pontryagin's adjoint equation gives p = J^T p = 0 when ρ(J) < 1).
Problem 2 — Implicit Differentiation: The fix was phantom gradients: run K iterations without gradient tracking to find the fixed point z*, then run m+1 more iterations with gradients. Backpropagation through those tracked steps naturally computes the Neumann series approximation of the implicit gradient (I - J)⁻¹ · ∂F/∂θ. The truncation error was formally bounded:
Problem 3 — The Residual Connection Catastrophe: DEQ experiments still diverged. The convergence diagnostics showed residuals exploding from 0.05 at depth 16 to 150 at depth 48. I traced this to a fundamental algebraic identity: the standard transformer block T(x) = x + Attn(x) + FFN(x) contains internal residual connections. Substituting into the KM step:
The damping cancels. The Jacobian becomes I + αJ_H, which always has spectral radius ≥ 1. Contraction is structurally impossible with internal residuals, regardless of damping, penalty, or spectral clipping. Aristotle verified this formally (km_residual_collapse, spectral_clipping_fails).
Problem 4 — The Pure Block: The fix was removing all internal residual connections, making T a pure function. The KM gate becomes the sole residual pathway: J_F = (1-α)I + αJ_T. If ||J_T|| < 1, this is provably contractive. But wouldn't removing residuals destroy the transformer's expressiveness? Aristotle proved the opposite: the resolvent (I - J_T)⁻¹ provides unbounded amplification at the fixed point even when every individual step contracts.
The model trades step-wise expansion for fixed-point amplification — convergent, stable, and arbitrarily expressive by controlling how close ||J_T|| is to 1.