5 Entropy and the bit: Claude Shannon and the quantification of information
Claude Shannon’s breakthrough was to redefine information in a way that could be measured. Instead of treating a message’s meaning or length as the key issue, he focused on how much uncertainty the message removes. From that view, a surprising outcome carries more information than an expected one, and the amount of information depends on the probabilities of the possible outcomes. He then introduced entropy as the average uncertainty of a source and showed how the bit became the natural unit for expressing it.
With entropy in hand, Shannon gave communication theory a rigorous mathematical foundation. He showed that low-entropy sources contain redundancy that can be removed through compression, while noisy channels often need carefully designed redundancy to preserve reliability. This led to foundational results about lossless compression, channel capacity, and error-correcting codes, all of which explain the limits of efficient and reliable transmission. The chapter emphasizes that these ideas unified what had previously been separate engineering problems into a single framework.
The chapter also shows how Shannon’s ideas escaped their original setting and became central to modern machine learning. Decision trees use entropy and information gain to choose splits, random forests extend that principle across many trees, feature selection uses mutual information to find useful predictors, and clustering can be evaluated by how much uncertainty it removes. In deep learning, cross-entropy measures how well predictions match reality, while representation learning acts like compression that preserves useful structure and discards redundancy. Together, these examples show that Shannon’s theory became a general language for learning from data under uncertainty.
Entropy measures the uncertainty associated with an information source in bits. A fair coin has an entropy of 1 bit, while a biased coin has lower entropy (0.47 bits) because one outcome is much more predictable than the other. A fair die (2.59 bits) and a standard deck of cards (5.7 bits) have progressively higher entropy because they contain larger sets of equally likely outcomes. As entropy increases, preserving the original information becomes increasingly important because there is less redundancy available for compression. Conversely, as entropy decreases, greater redundancy creates more opportunities to compress the source without losing information.
Shannon's framework of communication. An information source generates messages with varying degrees of unpredictability and passes them to an encoder, which converts the message into signals for transmission over a channel subject to noise. A decoder reconstructs the message from the received signal, and a receiver interprets the result. By separating communication into these stages, Shannon showed how uncertainty, redundancy, and reliability could be analyzed and optimized within the limits imposed by the communication channel.
Representation learning as information-preserving compression. An encoder transforms a high-dimensional image into a compact latent representation by preserving the image's informative structure while reducing redundant detail. Although many original pixels are no longer represented explicitly, the latent representation retains the information needed for a decoder to reconstruct an image that closely resembles the original.
Summary
- Claude Shannon fundamentally redefined information by showing that it could be measured mathematically. Rather than viewing information in terms of meaning or message length, he showed that information depends on how much uncertainty an outcome resolves. Predictable content contributes little new information because it changes little that is not already known, whereas surprising outcomes resolve greater uncertainty and therefore convey more information.
- Shannon introduced entropy as the mathematical measure of uncertainty and the bit as its unit of measurement. Entropy quantifies the average uncertainty associated with an information source before any particular outcome is observed, while the bit provides a common scale for expressing that uncertainty. For the first time, information, uncertainty, and redundancy could all be measured, compared, and analyzed quantitatively rather than described qualitatively.
- By making uncertainty measurable, Shannon transformed communication into a mathematical science. Entropy determines how much information a source contains, identifies how much redundancy can be removed through compression, establishes how much information must be transmitted, and helps determine how much carefully designed redundancy should be added to protect messages from noise. Together with channel capacity, these ideas continue to govern modern digital communication systems.
- The same mathematical framework later found applications far beyond communication. Once information and uncertainty could be quantified, they could also be optimized. In data science, entropy guides decision trees, random forests, feature selection, and clustering by measuring uncertainty and selecting the decisions that produce the greatest reduction in that uncertainty.
- Modern deep learning extends Shannon's ideas even further. Cross-entropy loss measures the remaining uncertainty in a neural network's predictions and guides the optimization of its parameters. Representation learning applies the same principles by preserving informative structure while reducing redundancy, allowing modern learning systems to build compact and useful representations of complex data.
- One of Shannon's most enduring contributions was not simply solving the communication problems of the 1940s, but providing a mathematical language for information itself. Once information, uncertainty, and redundancy could be measured in bits, they became quantities that could be optimized in entirely new domains. Few ideas have proved as adaptable or influential across communication, statistics, data science, and artificial intelligence.
Timeless Algorithms, The Foundational Ideas ebook for free
