References · Uriversity

Elhage, Nanda, Olsson, Henighan, Joseph, Mann, Askell, Bai, Chen, Conerly, DasSarma, Drain, Ganguli, Hatfield-Dodds, Hernandez, Jones, Kernion, Lovitt, Ndousse, Amodei, Brown, Clark, Kaplan, McCandlish, Olah · 2021

Transformer Circuits Thread (Anthropic) open access

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Frankle, Carbin · 2019

ICLR · arXiv:1803.03635 open access

Neural Networks and the Bias/Variance Dilemma

Geman, Bienenstock, Doursat · 1992

Neural Computation open access

Understanding the difficulty of training deep feedforward neural networks

Glorot, Bengio · 2010

AISTATS open access

Deep Learning

Goodfellow, Bengio, Courville · 2016

MIT Press open access

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Goyal, Dollár, Girshick, Noordhuis, Wesolowski, Kyrola, Tulloch, Jia, He · 2017

arXiv (tech report) · arXiv:1706.02677 open access

Implicit Regularization in Matrix Factorization

Gunasekar, Woodworth, Bhojanapalli, Neyshabur, Srebro · 2017

NIPS · arXiv:1705.09280 open access

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

He, Zhang, Ren, Sun · 2015

ICCV · arXiv:1502.01852 open access

Distilling the Knowledge in a Neural Network

Hinton, Vinyals, Dean · 2015

arXiv (NIPS 2014 Deep Learning Workshop) · arXiv:1503.02531 open access

Multilayer feedforward networks are universal approximators

Hornik, Stinchcombe, White · 1989

Neural Networks 2(5):359-366 paywalled

Position: The Platonic Representation Hypothesis

Huh, Cheung, Wang, Isola · 2024

ICML · arXiv:2405.07987 open access

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Ioffe, Szegedy · 2015

ICML · arXiv:1502.03167 open access

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Jacob, Kligys, Chen, Zhu, Tang, Howard, Adam, Kalenichenko · 2018

CVPR · arXiv:1712.05877 open access

A Recipe for Training Neural Networks

Karpathy · 2019

karpathy.github.io blog open access

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

Keskar, Mudigere, Nocedal, Smelyanskiy, Tang · 2017

ICLR · arXiv:1609.04836 open access

Adam: A Method for Stochastic Optimization

Kingma, Ba · 2015

ICLR · arXiv:1412.6980 open access

Visualizing the Loss Landscape of Neural Nets

Li, Xu, Taylor, Studer, Goldstein · 2018

NeurIPS · arXiv:1712.09913 open access

SGDR: Stochastic Gradient Descent with Warm Restarts

Loshchilov, Hutter · 2017

ICLR · arXiv:1608.03983 open access

Decoupled Weight Decay Regularization

Loshchilov, Hutter · 2019

ICLR · arXiv:1711.05101 open access

Machine Learning

Mitchell · 1997

McGraw-Hill open access

Deep Double Descent: Where Bigger Models and More Data Hurt

Nakkiran, Kaplun, Bansal, Yang, Barak, Sutskever · 2020

ICLR · arXiv:1912.02292 open access

A method for solving the convex programming problem with convergence rate O(1/k^2)

Nesterov · 1983

Dokl. Akad. Nauk SSSR paywalled

Zoom In: An Introduction to Circuits

Olah, Cammarata, Schubert, Goh, Petrov, Carter · 2020

Distill open access

Some methods of speeding up the convergence of iteration methods

Polyak · 1964

USSR Computational Mathematics and Mathematical Physics open access

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Power, Burda, Edwards, Babuschkin, Misra · 2022

arXiv preprint (also non-archival ICLR 2021 MATH-AI workshop; never in a peer-reviewed venue) · arXiv:2201.02177 open access

How Does Batch Normalization Help Optimization?

Santurkar, Tsipras, Ilyas, Madry · 2018

NeurIPS · arXiv:1805.11604 open access

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton, Dean · 2017

ICLR · arXiv:1701.06538 open access

Cyclical Learning Rates for Training Neural Networks

Smith · 2017

WACV · arXiv:1506.01186 open access

On the Origin of Implicit Regularization in Stochastic Gradient Descent

Smith, Dherin, Barrett, De · 2021

ICLR · arXiv:2101.12176 open access

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov · 2014

JMLR open access

Test-Time Training with Self-Supervision for Generalization under Distribution Shifts

Sun, Wang, Liu, Miller, Efros, Hardt · 2020

ICML · arXiv:1909.13231 open access

On the importance of initialization and momentum in deep learning

Sutskever, Martens, Dahl, Hinton · 2013

ICML open access

Benefits of Depth in Neural Networks

Telgarsky · 2016

COLT · arXiv:1602.04485 open access

Regression Shrinkage and Selection via the Lasso

Tibshirani · 1996

JRSS-B open access

Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude

Tieleman, Hinton · 2012

COURSERA: Neural Networks for Machine Learning open access

Principles of Risk Minimization for Learning Theory

Vapnik · 1991

NIPS open access

The Lack of A Priori Distinctions Between Learning Algorithms

Wolpert · 1996

Neural Computation 8(7) open access

Group Normalization

Wu, He · 2018

ECCV · arXiv:1803.08494 open access

On Layer Normalization in the Transformer Architecture

Xiong, Yang, He, Zheng, Zheng, Xing, Zhang, Lan, Wang, Liu · 2020

ICML · arXiv:2002.04745 open access

On Early Stopping in Gradient Descent Learning

Yao, Rosasco, Caponnetto · 2007

Constructive Approximation open access

Understanding deep learning requires rethinking generalization

Zhang, Bengio, Hardt, Recht, Vinyals · 2017

ICLR · arXiv:1611.03530 open access

Regularization and Variable Selection via the Elastic Net

Zou, Hastie · 2005

Journal of the Royal Statistical Society: Series B (Statistical Methodology) open access