Glossary

The fraction of examples the model labels correctly. Honest when the classes are balanced and every mistake costs the same, misleading otherwise. It is a step function with zero gradient almost everywhere, so you cannot train on it directly: you train on a loss and report accuracy.

The nonlinearity applied after each unit's weighted sum (ReLU, sigmoid, tanh). Without it a stack of layers collapses into one linear map; with it the network can bend. The values it produces are the activations, and keeping their variance stable through the layers is what initialization and normalization are for.

An adaptive optimizer that keeps per-parameter running averages of the gradient (momentum) and its square (to scale step sizes). A robust default for deep nets, less sensitive to the global learning rate than plain SGD.

Adam with decoupled weight decay — applying L2 shrinkage directly to the weights rather than folding it into the gradient (which Adam's scaling otherwise distorts). The standard choice for transformer-era training.

The handful of training examples one gradient step is computed on. Full-batch gradient descent uses the whole training set per step; mini-batch SGD uses a small random subset, cheaper per step and noisier, and the noise is part of why it works. Batch normalization uses the word in the same sense: statistics over the current batch.

Normalize each layer's activations per mini-batch to mean 0 / variance 1, then apply a learnable scale and shift. Its real benefit is smoothing the loss landscape — not, as originally claimed, reducing 'internal covariate shift' — which buys robustness to bad initialization and larger learning rates.

Test error decomposes into bias (error from too-simple assumptions), variance (sensitivity to the particular training sample), and irreducible noise. Regularization trades higher bias for lower variance; the classical sweet spot minimizes their sum.

How much variety of functions a model can express — roughly, how complex a pattern it can fit. Too little underfits; too much can overfit (classically) or, past the interpolation threshold, generalize again (double descent).

A supervised task where the model predicts a category from a fixed set: shoe or shirt, spam or not. Contrast regression, which predicts a number.

Training has converged when further steps no longer change the loss meaningfully. Guaranteed for convex losses (linear and logistic regression); for neural networks there is no such guarantee, only a plateau you learn to read. A flat training loss does not mean you are done: check the validation curve before you stop.

The standard loss for classification: the negative log of the probability the model assigned to the correct class, averaged over examples. Confident wrong answers are punished hard, which is the point. You train on cross-entropy and hope accuracy follows.

Letting information from data the model should never see (the test set, or the future) shape a modeling decision. Peeking at the test split, or standardizing with statistics computed on all the data, are the common forms. Every decision made from what you saw in the data is a soft version of it, which is why the split comes first.

Machine learning with neural networks of many layers, which learn their own intermediate representations instead of being handed features. The inner ring of the nesting: every deep-learning system is machine learning, and every machine-learning system is AI, but not the reverse.

As capacity grows, test error follows the classical U-curve up to the interpolation threshold, rises briefly, then descends again in the over-parameterized regime — so bigger models can generalize better past the point classical theory predicts disaster.

During training, randomly zero a fraction of activations each forward pass (rescaling the rest). Breaks neuron co-adaptation and approximates training an exponential ensemble of sub-networks — a variance reduction at test time.

Stop training before convergence, when validation loss stops improving. Limits how far parameters travel from initialization — provably equivalent to L2 regularization for linear least squares, and an implicit regularizer for neural nets.

The formal framing of supervised learning: minimize the average loss over your training sample (empirical risk) as a stand-in for the true expected loss over the whole data distribution, which you can't measure directly.

One full pass over the training set, however many batches that takes. Training runs for some number of epochs, and losses are recorded at the end of each one, after all of its updates, so the training and validation curves are compared on the same weights.

One measured input the model is given per example: a pixel value, a district's median income. The columns of X. Chapter 2's task has eight of them, and the first job is to look at them before modeling anything.

Minima where the loss changes slowly (flat) versus steeply (sharp) around the solution. Flat minima tend to generalize better; large-batch training tends to find sharper, worse-generalizing ones.

Doing well on data the model has never seen, which is the actual objective; fitting the training data is only the means. Measured on a held-out set. The gap between training and held-out performance is the first diagnostic, and a large one is overfitting.

The difference between a model's performance on training data and on unseen data. A gap that widens as training proceeds is the signature of overfitting.

The vector of partial derivatives of the loss with respect to each parameter: the direction in weight space along which the loss rises fastest. Gradient descent steps the other way, scaled by the learning rate.

Verify an analytic gradient against a numerical finite-difference estimate. A cheap sanity check that catches backprop bugs before they silently wreck training.

The core optimization method: repeatedly step the parameters in the direction of steepest descent of the loss (the negative gradient), scaled by the learning rate. Differentiability is what makes the search direction computable.

Delayed generalization: a network memorizes the training set (train accuracy → 100%) while validation stays at chance through a long plateau, then suddenly generalizes much later — often only under strong weight decay.

Split channels into groups and normalize within each group across spatial dimensions, independent of batch size. Used for convnets at small batch size (detection, segmentation, video).

Initialize weights with variance 2/n_in — twice Xavier's, because ReLU zeros half its inputs. The matched initialization for ReLU networks and PyTorch's nn.Linear default.

A knob you set before training rather than one the data sets for you: the learning rate, the batch size, the number of epochs, the regularization strength, the architecture itself. Tuned on the validation set, never on the test set. The learning rate is the one that matters most.

The set of all functions a model can possibly represent, fixed the moment you choose its architecture. Learning is a search through this space for a function that fits the data and generalizes.

Among the many parameter settings that reach zero training loss, stochastic gradient descent systematically prefers ones with good properties (flat minima, small norm, low rank) — regularizing with no explicit penalty term. Part of why over-parameterized nets generalize.

The assumptions a model makes when the data is silent — the structure it bets on (linearity, smoothness, locality, additivity). Strong inductive bias that matches the data needs less data; a mismatch is what the No Free Lunch theorem charges you for.

The original (2015) explanation for why BatchNorm helps: the shifting distribution of each layer's inputs as upstream weights change. Later shown not to be the real mechanism — artificially re-injecting it after a BN layer doesn't hurt training.

The capacity at which a model can exactly fit (interpolate) the training set — training error hits zero. It's the peak of the double-descent curve, the boundary between the classical and modern regimes.

Penalize the sum of absolute weights. Its diamond-shaped constraint has corners on the axes, so the optimum often lands with many weights exactly zero — producing sparse, feature-selecting models.

Penalize the sum of squared weights, shrinking them smoothly toward — but never exactly to — zero. Reduces sensitivity to any single training point; equivalent to a circular constraint on the weights.

The answer that comes attached to a training example: the category of an image, the price of a house. Supervised learning is learning from pairs of input and label; the label is what the model is scored against, and its quality caps what the model can learn.

Normalize across features within a single example, independent of batch size. The standard normalization for transformers and RNNs; modern usage is usually pre-norm — inside the residual block, before attention/MLP.

The step-size multiplier on each gradient update — the single most important hyperparameter. Too small crawls; too large oscillates or diverges; the usable band is bounded on both sides and spans orders of magnitude.

Find a good learning rate empirically: start tiny, increase it exponentially over a short run, and plot loss vs LR. Pick a value just below where the loss starts to diverge — in the steepest-descent region.

Vary the learning rate over training rather than holding it fixed — big steps early, small steps late. Cosine annealing, optionally with a warmup, is the modern default.

The number that says how wrong the model's predictions are on the examples in front of it, and the thing gradient descent pushes down. Chosen for a smooth gradient (mean squared error, cross-entropy), so it is not the same as the metric you report; the gap between the two is the surrogate problem.

The surface of the loss as a function of the model's parameters. Architecture choices (residual connections, normalization) shape how smooth or rugged it is, which determines how easily gradient descent can navigate it.

A large randomly-initialized network contains a small sub-network (a 'winning ticket') that — reset to its original init and trained alone — matches the full network. Reframes over-parameterization as a wider lottery, not just more room to overfit.

A smooth, lower-dimensional surface curving through a high-dimensional space. Real data (faces, sentences, sensor readings) occupies a thin manifold rather than filling the ambient space uniformly, so its intrinsic dimension is far below its ambient dimension — the structural reason feature selection and dimensionality reduction work.

What you measure to judge the model against the real goal (accuracy, dollar error, F1), as opposed to the loss you optimize. If you measure it, you are optimizing it: the model will exploit every shortcut the metric leaves open, so choose one that matches what you actually care about.

A function with adjustable parameters that maps an input to a prediction. Choosing the family of functions is the architecture, training picks the parameters, and the model is what you are left with. The only fair test of it is data it has never seen.

The simplest possible predictor — e.g. always predict the training mean or the majority class. Every real model must beat it; if it can't, the problem, features, or pipeline are the issue, not the model.

Layers of simple units, each computing a weighted sum of its inputs followed by a nonlinearity, stacked so that later layers work on features the earlier ones learned. Chapter 8 opens one up. The first model in this course is not one, and does better than you might expect.

Averaged over all possible target functions, no learning algorithm beats any other (including random guessing). Practically: every model that works does so by betting on structure that real data happens to have.

The update rule that turns a gradient into a step: vanilla gradient descent, or one of its descendants (momentum, Adam, AdamW), each fixing a known pathology of the plain version, such as oscillating across a narrow valley or using one learning rate for every parameter. A search strategy, not a different objective.

When a model fits the training data's noise, not just its signal — low training error but high error on unseen data (a wide generalization gap). The variance side of the bias–variance tradeoff.

The numbers inside the model that training adjusts: in a linear model one weight per feature plus a bias, in a network every connection. Learning is choosing these numbers. Everything you set by hand instead is a hyperparameter.

A supervised task where the model predicts a number on a continuous scale: a house price, a temperature. Contrast classification, which predicts a category. Linear regression and logistic regression share the name for historical reasons; logistic regression is a classifier.

Any technique that deliberately weakens a model's fit to the training data to improve generalization — almost always a variance-reduction tool whose price is a little extra bias. Tikhonov introduced the method (stabilizing ill-posed problems with a penalty) in 1943; the term itself comes from his 1963 work.

A point where the gradient is zero but the surface curves up in some directions and down in others. In high dimensions these — not local minima — are the dominant obstacle for gradient descent.

Rescale each feature to mean 0 / unit variance, fitting the scaler on the training set only and applying it to validation and test. Fitting on all the data leaks test statistics into training.

Estimate the gradient from a small random batch each step instead of the full dataset. Cheaper per step, and the gradient noise itself regularizes; batch size trades gradient accuracy against steps-per-second and the sharpness of the minimum found.

Learning from examples that come with the answers attached: pairs of input and label, and a model whose job is to produce the label for inputs it has never seen. The workhorse of the field and of this course. Classification and regression are both inside it.

Partition the data before anything else: train to fit parameters, validation to tune hyperparameters and pick models, test to estimate final performance exactly once. Touching test during development leaks information and inflates your estimate.

When a model is too simple to capture the data's real structure — high error on both training and unseen data. The bias side of the bias–variance tradeoff.

A neural network with one hidden layer can approximate any continuous function to arbitrary accuracy — but the theorem is non-constructive: it says good weights exist, not how many units you need or how to find them. Depth is often exponentially more parameter-efficient than width.

Learning from data with no answers attached: finding clusters, structure, or a compressed description. Often a preprocessing step for supervised learning rather than an alternative to it, and sometimes what tells you that you asked the wrong question.

Initialize weights with variance 2/(n_in + n_out) so activation variance is preserved through the network. Designed for tanh/sigmoid activations (symmetric around 0).