2 The mathematical building blocks of neural networks
This chapter builds an intuition-first foundation for the mathematics that power neural networks, favoring runnable code over heavy notation. It opens with a hands-on Keras example that classifies MNIST digits end to end—data loading, preprocessing, defining a simple stack of Dense layers, compiling with a loss, optimizer, and metrics, training with batches, and evaluating while noting overfitting. The goal is to make the core ideas concrete so you’re prepared to use modern deep learning frameworks effectively.
The core building blocks are tensors (multidimensional arrays) characterized by rank, shape, and dtype, along with slicing and batching conventions across common data modalities (vectors, sequences, images, video). The chapter develops the small set of tensor operations underlying deep nets: fast vectorized element-wise ops, broadcasting, tensor products (matmul), reshaping and transposition. Layers are presented as affine transforms followed by nonlinear activations (e.g., relu), with a geometric lens: translation, rotation, scaling, and general linear/affine transforms; stacking affine transforms plus nonlinearities yields powerful, nonlinear geometric mappings that “uncrumple” complex high-dimensional data manifolds.
Learning is framed as gradient-based optimization inside a training loop: forward pass to get predictions, loss computation, and a backward pass to obtain gradients that update weights. The chapter introduces derivatives, gradients, and the chain rule, leading to backpropagation implemented via computation graphs and automatic differentiation. It covers mini-batch SGD, learning rate, momentum, and optimizer variants (e.g., RMSprop, Adam), then revisits the MNIST example to decode compile/fit semantics and culminates with a from-scratch reimplementation (naive Dense/Sequential, batch generation, GradientTape-driven updates), demystifying what high-level APIs do under the hood.
MNIST sample digits
The fourth sample in our dataset
A rank-3 timeseries data tensor
A rank-4 image data tensor
Matrix product box diagram
A point in a 2D space HTML: class=“extra-small-image”
A point in a 2D space pictured as an arrow
Geometric interpretation of the sum of two vectors
2D translation as a vector addition
2D rotation (counterclockwise) as a matrix product
2D scaling as a matrix product
Affine transform in the plane
Affine transform followed by relu activation
Uncrumpling a complicated manifold of data
A continuous, smooth function
With a continuous function, a small change in x results in a small change in y
Derivative of f in p
SGD down a 1D loss curve (one learnable parameter)
Gradient descent down a 2D loss surface (two learnable parameters)
A local minimum and a global minimum
The computation graph representation of our two-layer model
A basic example of a computation graph
Running a forward pass HTML: class=“extra-small-image”
Running a backward pass HTML: class=“small-image”
Path from loss_val to w in the backward graph
Relationship between the network, layers, loss function, and optimizer
Chapter summary
- Tensors form the foundation of modern machine learning systems. They come in various flavors of
dtype,rank, andshape. - You can manipulate numerical tensors via tensor operations (such as addition, tensor product, or elementwise multiplication), which can be interpreted as encoding geometric transformations. In general, everything in deep learning is amenable to a geometric interpretation.
- Deep learning models consist of chains of simple tensor operations, parameterized by weights, which are themselves tensors. The weights of a model are where its “knowledge” is stored.
- Learning means finding a set of values for the model’s weights that minimizes a loss function for a given set of training data samples and their corresponding targets.
- Learning happens by drawing random batches of data samples and their targets, and computing the gradient of the model parameters with respect to the loss on the batch. The model parameters are then moved a bit (the magnitude of the move is defined by the learning rate) in the opposite direction from the gradient. This is called mini-batch gradient descent.
- The entire learning process is made possible by the fact that all tensor operations in neural networks are differentiable, and thus it’s possible to apply the chain rule of derivation to find the gradient function mapping the current parameters and current batch of data to a gradient value. This is called backpropagation.
- Two key concepts you’ll see frequently in future chapters are loss and optimizers. These are the two things you need to define before you begin feeding data into a model.
- The loss is the quantity you’ll attempt to minimize during training, so it should represent a measure of success for the task you’re trying to solve.
- The optimizer specifies the exact way in which the gradient of the loss will be used to update parameters: for instance, it could be the RMSProp optimizer, SGD with momentum, and so on.
FAQ
What is a tensor, and how do rank, shape, and dtype differ?
A tensor is a container for numerical data. Rank is the number of axes (0 for a scalar, 1 for a vector, 2 for a matrix, etc.). Shape is a tuple indicating the size along each axis (for example, (60000, 28, 28) for MNIST images). dtype is the data type of the entries (such as float32, uint8, bool). Example: a scalar has shape (), a vector of length 5 has shape (5,), and a 3×5 matrix has shape (3, 5).
Why do we flatten and normalize MNIST images before training?
Dense layers expect 1D vectors per sample, so 28×28 images are reshaped to 784-length vectors. Normalizing pixel values to the [0, 1] range (by dividing by 255 and casting to float32) improves numerical stability and speeds up learning.
What does a Dense layer compute mathematically?
A Dense layer applies an affine transform followed by a nonlinearity: output = activation(input @ W + b), where @ is matrix multiplication, W and b are trainable weights, and activation can be functions like relu or softmax.
What are element-wise operations and broadcasting, and how do they differ?
Element-wise operations apply independently to each tensor entry (for example, relu(x) = max(x, 0) or x + y when x and y have the same shape). Broadcasting lets you perform element-wise ops on tensors with compatible shapes by virtually expanding the smaller tensor along new axes. Example: adding a matrix of shape (32, 10) and a vector of shape (10,) broadcasts the vector across the 32 rows.
What is the tensor product (matrix multiplication) and what are the shape rules?
The tensor product (dot/matmul) generalizes vector and matrix products. Key cases: vector·vector → scalar; matrix·vector → vector; matrix·matrix: (a, b) @ (b, c) → (a, c). Higher-rank cases follow the same compatibility rule on the inner dimensions.
What are batches and epochs, and why do models train on mini-batches?
Axis 0 is typically the samples (batch) axis. Models don’t process all data at once; they iterate over mini-batches for efficiency and stable updates. A training loop per batch does: forward pass → loss computation → backward pass (gradients) → weight update. One full pass over all training samples is an epoch.
What is gradient descent, and what does the learning rate control?
Gradient descent updates weights in the opposite direction of the loss gradient to reduce the loss. The learning rate scales the step size: too small slows learning; too large can diverge. Mini-batch SGD uses random batches; variants like Momentum, RMSprop, and Adam refine the update using past gradients or adaptive scaling.
How does backpropagation compute gradients in neural networks?
Backpropagation applies the chain rule over the computation graph of the forward pass to propagate derivatives from the loss back to each parameter. Modern frameworks (JAX, TensorFlow, PyTorch) perform this via automatic differentiation, so you write only the forward computation and the framework provides gradients.
Why are activation functions like ReLU necessary?
Stacking only affine transforms (W @ x + b) collapses to a single affine transform, limiting the model to linear mappings. Nonlinear activations (e.g., ReLU) enable the network to represent complex, nonlinear functions and learn rich data transformations. softmax additionally converts logits to class probabilities.
What is the geometric intuition behind deep learning in this chapter?
Each tensor operation is a geometric transform (translation, rotation, scaling, affine transforms, and nonlinear “folds”). A deep network composes many simple transforms to “uncrumple” complex data manifolds so classes become separable. Training finds the parameters that achieve this disentangling while minimizing the loss.
Deep Learning with Python, Third Edition ebook for free
