2 Representing language

Language is presented as both the most revealing and most difficult medium for understanding intelligence. The chapter opens by showing how speech and writing let humans share knowledge across time, while also arguing that language is not identical to thought: people can reason visually, spatially, and intuitively without words, and fluent speech can mask shallow understanding. This makes language a useful but imperfect window into cognition, which is why it became the primary way researchers tested early AI systems and why it still serves as the main domain for studying machine intelligence.

The chapter then traces how machines began working with language through increasingly sophisticated representations. Early systems relied on hand-built rules and pattern matching, which could produce convincing results only in narrow cases and failed badly with ambiguity, idioms, and open-ended variation. Researchers then shifted toward statistical methods, first counting word sequences with n-grams and conditional probabilities, but these models were limited by sparsity and short context windows. To make text usable for machines, language also had to be segmented into tokens, leading from crude word splitting to subword methods like BPE, WordPiece, and SentencePiece, which balance vocabulary size, efficiency, and the ability to handle new or unfamiliar forms.

The final part of the chapter explains how language became more tractable when tokens were mapped into vectors and embeddings. One-hot encodings gave way to dense representations that place words in a geometric space where similarity, analogy, and context can be expressed mathematically. Models such as word2vec and GloVe learned these spaces from co-occurrence patterns, while later contextual embeddings allowed the meaning of a word to shift with its surrounding text. Even so, language remains a frontier because it is vast, ambiguous, unevenly distributed, and highly variable across languages and uses. The chapter concludes that modern language models are powerful because they capture statistical regularities in human expression, but they still struggle with long-range dependencies, unstated meaning, and the full richness of intelligence beyond words.

Example interaction with ELIZA. The system produces plausible responses when user input matches expected conversational patterns, but its behavior is driven by scripted rules combining keywords, pattern matching, and response templates rather than an understanding of meaning. As the dialogue progresses, this limitation becomes visible when responses no longer align with the user’s intent.

A visual example of how n grams are extracted from a sentence. Each row shows overlapping windows of different lengths (bigrams, trigrams, and four-grams) sliding across the same sequence of words. As n increases, each unit captures more context, producing progressively larger segments of the original sentence.

Sequential word prediction in a probabilistic language model. At each step, the model selects the most likely next word based on the preceding context. By repeating this process, a sequence of locally probable choices produces a sentence that appears coherent.

Balancing context and complexity in n-gram models. As the length of word sequences increases, models capture richer context but require substantially more data to estimate probabilities reliably. Longer sequences allow more precise patterns to be represented, but they also introduce a rapid growth in the number of possible combinations, making reliable estimation increasingly difficult.

Starting with characters, byte-pair encoding identifies the most frequent adjacent pairs in the corpus and merges them step by step. These merges gradually create subword units that let the model represent both common and rare words efficiently.

A simplified illustration of how embeddings represent words as numerical vectors across multiple dimensions. Each row corresponds to a word, and each column represents a property that contributes to its representation. The values indicate how strongly the word is associated with each property, forming a vector that captures patterns of similarity. Words with similar patterns across dimensions, such as apple and pizza or car and robot, end up with similar representations. In real models, these dimensions are learned automatically from data and do not correspond directly to interpretable properties like “edible” or “living.”

How word2vec models learn embeddings by leveraging context in opposite directions. In CBOW (top), the model predicts the word “sun” from its surrounding words (“the” and “rises”), learning its representation from context. In Skip-gram (bottom), the word “sun” is used to predict its surrounding words, learning its representation by how well it explains its context. Despite this difference, both methods aim to produce embeddings that capture how words are used in language.

How GloVe learns embeddings from co-occurrence statistics. The left matrix represents how often words appear near each other in text, where each entry counts how frequently a pair of words co-occurs. The empty circles along the diagonal indicate that each word is not counted as its own context. Through a fitting process, the model learns two sets of embeddings: one for words as targets and another for words as contexts. Despite this separation, both sets are learned together to capture patterns of co-occurrence, producing embeddings that reflect how words are used in language.

A simplified two-dimensional illustration of an embedding space. Words that appear in similar contexts cluster together, with more similar words positioned closer to each other, such as “dog” and “wolf” or “fork” and “knife.” Arrows indicate consistent directional relationships, such as the shift between masculine and feminine forms shown in the “Royalty” group. The axes have no inherent linguistic meaning; they are learned dimensions that reflect statistical patterns in language use.

How ELMo produces contextual embeddings. The model processes the sentence in both directions, using a left-to-right and a right-to-left model to capture information from the full sequence. The resulting representations are dynamic embeddings, where each embedding is adjusted according to its surrounding context rather than assigned as a fixed vector. As a result, the same word can have different representations depending on how it is used in the sentence.

• Humans naturally treat language as evidence of intelligence, but it’s only one form of cognition; fluency alone doesn’t prove understanding.
• Language-generating machines mirror our patterns rather than our experience, but this imitation is often effective enough to support complex tasks.
• Language remains a practical entry point for studying artificial intelligence, as it brings diverse cognitive abilities into a common form while still leaving room to explore aspects that extend beyond it.
• Treating words as data made it possible to study language computationally and laid the groundwork for natural language processing (NLP).
• Early rule-based systems could model grammar but failed to capture the openness, variability, and implicitness of real language use.
• Statistical approaches improved on these limitations by learning patterns directly from corpora. N-grams revealed how much linguistic behavior can be captured through frequency alone.
• N-gram models face trade-offs among context size, sparsity, and computational feasibility, which shows that counting can only go so far.
• Tokenization defines the units a model can perceive, shaping what distinctions it can learn and what nuance it inevitably loses.
• Subword tokenization, including BPE and its successors, offers a flexible way to handle rare words, new terms, and variation across languages.
• Numeric identifiers and one-hot vectors provide no sense of similarity. Dense embeddings overcome this by placing tokens in a continuous space where proximity reflects patterns of use.
• Embeddings encode linguistic relationships geometrically, capturing similarity, analogy, and structure derived from data rather than predefined rules.
• Language is difficult to model because meaning stretches across long distances, depends heavily on context, ambiguity, and pragmatics, and varies across scripts, domains, and cultures.
• Evaluation itself is a challenge: fluency, accuracy, coherence, relevance, and safety pull in different directions, and no single metric captures what “good” language understanding means.

FAQ