9 Artificial neural networks

Artificial neural networks are presented as powerful learning systems inspired by biological neurons and positioned within the broader machine learning lifecycle. The chapter motivates when to use ANNs—especially for complex, unstructured, and nonlinear problems—and frames deep learning as families of neural architectures trained on data to map inputs to outputs. It builds intuition from biological signaling to the logical abstraction of neurons, then traces how networks fit into the iterative ML process of problem definition, data preparation, training, evaluation, and improvement.

The text moves from the Perceptron—its inputs, weights, bias, summation, and activation—and its limitation to linear separability, to multilayer networks that overcome this by stacking hidden layers. It explains forward propagation (weighted sums plus activation across layers) and training via backpropagation, where errors are compared to targets, gradients are computed with the chain rule, and weights are adjusted through (stochastic) gradient descent. Practical essentials include scaling inputs (min–max) to avoid numeric bias, choosing activation functions (sigmoid, tanh, ReLU) and understanding vanishing gradients, selecting cost functions and learning rates, adding bias terms, initializing weights to avoid exploding/vanishing signals, monitoring loss, and stopping to prevent overfitting.

Design guidance covers shaping inputs/outputs, selecting the number of layers and nodes, and thinking in tensors for efficient, vectorized computation. The chapter also recasts ANNs in linear algebra—dot products and matrix multiplications—offering a compact mathematical view of forward and backward passes. It concludes with a tour of specialized architectures and their use cases: RNNs for sequences and temporal context, CNNs for image feature extraction and classification, and GANs for generative modeling—while noting the rise of Transformers for language. Together, these foundations connect core ANN mechanics to modern applications such as large language models and generative image systems.

A map describing the flexibility of deep learning and ANNs

A workflow for machine learning experiments and projects

A simplified model of a biological neural system

The general composition of neurons

Logical architecture of the Perceptron

An example of using a trained Perceptron

The sigmoid function

Example of a linear classification problem

Example ANN architecture for the car-collision example

Min-max scaling example with car collision data

Example ANN architecture for the car-collision problem

Example of a hidden node comparing terrain quality and degree of vision

Example of a hidden node comparing terrain quality and total experience

Life cycle of forward propagation in an ANN

Example of weights in a pretrained ANN

Mathematical denotation of an ANN

Weighted sum calculation for each hidden node

Activation function calculation for each hidden node

Final activation calculation for the output node

Life cycle of training an ANN

Example initial weights for an ANN

Example of forward propagation with randomly initialized weights

Weight versus cost plotted

Derivatives’ slopes and direction of minimums

Example of adjusting a weight by using gradient descent

Formula for calculating weight updates with the Chain Rule

Weight-update calculation with the Chain Rule

Example of the final weight-update for the ANN

Commonly used activation functions

An example of a multilayer ANN with more than one output

Comparison of tensor data structures

Simple example of a RNN

Simple example of a CNN

Simple example of a GAN

FAQ