3 Artificial neural networks

The chapter explains neural networks as systems that transform raw inputs into useful outputs through many small computations working together. Unlike earlier machine learning methods that relied on simpler pattern finding, neural networks build layered representations that can capture complex structure in data. Their power comes from the combination of neurons, layers, weights, and nonlinear activation functions, which lets them model relationships that would otherwise be too tangled for a linear system to handle.

Learning in these models begins with prediction error: the network compares its output with the correct answer, measures the difference as loss, and uses backpropagation and gradient-based optimization to adjust its weights. Training happens over many examples, usually in batches across repeated epochs, and the goal is not just to reduce error on familiar data but to generalize to new cases. The text emphasizes the danger of overfitting, where a model learns noise instead of durable patterns, and shows how regularization, dropout, validation, early stopping, and data augmentation help keep learning focused on structure that extends beyond the training set.

The chapter also shows that architecture matters as much as learning itself. Dense networks are flexible but ignore the natural structure of images, sequences, and other data, so specialized designs emerged: convolutional networks for spatial patterns, recurrent networks for sequences, gated recurrent models for longer memory, autoencoders for compression, and generative models for synthesis. In language, the limits of recurrence and convolution led to attention-based methods and pretraining, which made context more accessible and enabled broader transfer. The larger conclusion is that neural networks can produce intelligent-looking behavior by extracting and reusing patterns, but that this remains a form of structured generalization rather than human understanding.

Structure of an artificial neuron. Each input is scaled by a weight, summed with a bias, and passed through an activation function. This simple pipeline is the basic computational unit from which neural networks are built.

A deep neural network as a sequence of layers. The input layer receives raw data, the hidden layers transform it through learned representations, and the output layer produces the model’s prediction. In this example, the four input nodes might represent measurable features of an email, such as word frequencies or metadata, while the three output nodes might represent possible categories, such as personal, work-related, or spam. The network transforms these input features into output probabilities through the connections learned in the hidden layers.

Inputs connect to a neuron through weights that are adjusted during training. Each weight links a specific input to a specific neuron; the index k labels which neuron these connections belong to, while the second index identifies the input. These learned weights (represented here by line thickness) determine how strongly each input shapes the neuron’s output. In this example, features such as keyword frequency, number of links, message length, and whether the sender is known contribute differently to estimating spam likelihood, with more informative signals exerting greater influence. In practice, these inputs are often abstract numerical representations that do not correspond directly to human-understandable features, and the outputs can also take the form of intermediate numerical values rather than clear-cut categories.

The ReLU activation function transforms a neuron’s input by suppressing negative values and passing positive ones through unchanged. This simple nonlinear behavior changes how inputs are separated, enabling the network to move beyond straight-line boundaries and capture more complex patterns.

A deep network without activation functions collapses into a single linear transformation. No matter how many layers are stacked, their combined effect is equivalent to a single layer, which would limit the model to learning only simple, linearly separable patterns.

A neural network processes inputs through successive layers, each reshaping the signal in a different way. Early layers tend to capture basic patterns in the data, while intermediate layers combine these signals into more informative representations. Deeper layers refine them into higher-level cues that are useful for the task. In this example, features of an email are progressively reinterpreted until the network produces a classification such as personal, work, or spam. These layer labels illustrate typical roles; in practice, the internal representations are distributed and do not correspond to fixed, human-understandable concepts.

The network processes the input to produce a prediction ypred. This prediction is then compared to the expected value ytrue, and the estimation error is captured by the loss. This loss then becomes the signal that guides how the network should adjust itself during learning.

The loss landscape can contain many dips and valleys. If the optimizer steps too cautiously, it may settle into a shallow solution with relatively high loss. If it steps too boldly, it may skip over configurations that yield lower loss or bounce back and forth across a good solution without settling. In practice, these landscapes exist in very high-dimensional spaces, where their geometry becomes more complex. Finding the right step size is key to guiding learning toward regions of lower loss.

Underfitting, fitting, and overfitting illustrated on a simple task such as predicting house prices from size. Squares represent training data, while circles represent evaluation data drawn from new examples the model did not see during training. The line shows the model’s predicted relationship between size and price. A model that is too simple misses the underlying trend (underfitting), while one that is too flexible follows the training data too closely and fails to generalize (overfitting). The middle case captures the underlying pattern without memorizing individual examples.

During training, dropout randomly disables a different subset of neurons at each step. By forcing the network to function with changing pathways, it prevents overreliance on any single neuron and encourages representations that generalize more effectively.

Training loss typically decreases steadily, while validation loss eventually reaches a minimum and then rises again as the model starts to overfit. Early stopping halts training when validation performance peaks.

Data augmentation generates new training examples by altering the original input in ways that don’t change its underlying identity. By encountering rotated, flipped, resized, blurred, or recolored versions of the same image, the model learns to focus on features that remain consistent across variations.

A convolutional neural network that splits image analysis into two stages. Convolutional filters scan the image for local patterns, while pooling layers reduce spatial detail while preserving the most informative features. Stacked together, these layers build a hierarchy of representations, moving from simple edges to increasingly complex shapes. The resulting feature maps are then flattened and passed to dense layers, which integrate these features to produce a final prediction, such as identifying the object in the image. Dense layers appear at the end because classification requires combining information from across the entire image, not just from local regions. A softmax activation converts the final scores into probabilities over the output categories.

A recurrent neural network unrolled across time. Each cell processes one element of the input sequence xt by combining the current input with the network’s previous internal state ht-1. This combination is performed using the same set of learned weights at every step: Un weights the current input, Wn transforms the previous internal state, and Vn maps the resulting state to the output. The internal state is passed forward from one step to the next, allowing information to accumulate across the sequence. Unrolling the network makes this temporal flow explicit and shows how a single computation is reused repeatedly over time.

An LSTM neuron. At each time step, the cell receives the input xt, the previous hidden state ht-1, and the previous cell state Ct-1. The forget gate uses these signals to determine how much of the old memory to keep, while the input gate and the candidate update, propose and regulate the new information to add. Together, these components produce the updated cell state Ct, which carries long-term memory forward. The output gate then controls how much of this updated memory becomes visible by shaping the hidden state ht. This division between a protected memory pathway and an exposed hidden state allows LSTMs to maintain information across many time steps while still responding flexibly to new inputs.

An autoencoder compresses input into a compact internal code and then reconstructs it. The encoder reduces the data to a low-dimensional representation in the bottleneck, where only the most informative patterns remain. The decoder expands this representation back into the original space, producing an output that resembles the input. By learning to rebuild the data from such a constrained description, the network discovers the underlying structure that defines the input instead of its incidental details.

A generative adversarial network trains two models in opposition. The generator begins with random noise and tries to produce images that resemble those in the training set. The discriminator receives a random mixture of real images from the dataset and fake images created by the generator and learns to classify them as genuine or synthetic. During training, each model provides feedback to the other: the discriminator improves its ability to detect fakes, and the generator adjusts its parameters to create images that are increasingly difficult to distinguish from real ones. Through this adversarial loop, the system gradually learns to synthesize new samples that follow the same patterns as the data on which it was trained.

An attention mechanism applied to a single token. The active token (“sun”) is compared with all tokens in the sequence, producing a set of relevance weights that reflect how much each word contributes to its interpretation. These weights are then used to form a weighted combination of the token representations, resulting in a context vector. In this example, higher weights on “rises” and “east” indicate that the model focuses on words that help determine the meaning of “sun” in the sentence, while function words contribute less. This process allows the model to build a context-dependent representation without compressing all information into a single state.

• Artificial neurons are simple mathematical units that transform inputs into outputs through weighted combinations, a bias, and an activation function.
• The strength of neural networks comes not from individual neurons but from organizing many of them into layers. Shallow networks capture simple patterns, while deep networks build increasingly abstract representations.
• Weights serve as the distributed memory of a network, encoding what it has learned across millions of tiny adjustments.
• Activation functions introduce nonlinearity, allowing networks to model complex boundaries and interactions in data.
• The forward pass transforms inputs step by step through the network, and the loss quantifies how far the output is from the desired answer.
• Backpropagation reveals how each weight contributes to the loss, so optimizers can adjust parameters in directions that improve performance.
• Learning requires exposure to many examples. Batching shapes how the model sees data during training and influences stability and generalization.
• Dense networks struggle with data that contain spatial, temporal, or structural patterns they cannot easily exploit.
• Convolutional networks use shared filters and pooling to capture local patterns in images and build hierarchical features efficiently.
• Recurrent networks handle sequences by carrying information forward in a hidden state, while LSTMs and GRUs extend their memory through gating mechanisms.
• Autoencoders learn compressed representations by forcing data through a latent space, revealing essential structure and supporting reconstruction.
• Generative models, including variational autoencoders and GANs, learn to produce new samples by modeling data distributions.
• Architectural choices embed assumptions about the data. Selecting the right structure determines which patterns a model can learn efficiently.
• Language challenges neural networks with long-range dependencies and shifting context, exposing the limits of purely sequential or local architectures.
• Recurrent networks improved early NLP but struggled with very long contexts. CNNs captured short-range patterns but missed broader relationships.
• Attention mechanisms solved many of these constraints by allowing models to retrieve relevant information from any position in a sequence.
• Pretraining and transfer learning transformed NLP by allowing large backbone models to internalize general linguistic structure and then adapt to specific tasks.

FAQ