Learning Mechanics

The mathematical science of neural network training

Understanding deep learning will be the intellectual challenge of the early 21st century, like physics was in the early 20th. A good place to start is the study of neural network training dynamics, since these give rise to everything else. The study of learning mechanics is rapidly developing into a proper subfield of science. This website is meant to serve as a hub, a place to share progress and organize the search for simple, useful understanding.

A Quickstart Guide to Learning Mechanics
[long list of authors]
September 01, 2025
1. Introduction: what do you want to understand? 2. ...the average size of hidden representations? 3. ...hyperparameter selection (and why should theorists care)? 4. 🚧 ...the convergence and stability of optimization? 5. 🚧 ...feature learning and the final network weights? 6. 🚧 ...generalization? 7. 🚧 ...neuron-level sparsity? 8. 🚧 ...the structure in the data? 9. 🚧 Places to make a difference
Notes on the Science of Deep Learning
Jamie Simon, Florentin Guth, Dhruva Karkada
August 30, 2025
1. 🚧 Understanding deep learning will pay off in the long run 2. The scientific method in two steps and its application to deep learning 3. 🚧 Science as mapmaking