1 The Patterns Behind Modern AI Systems

The chapter explains that modern AI can seem intelligent because it produces fluent, relevant-sounding responses, but this does not mean it truly understands language. Early systems like ELIZA relied on hand-written rules and keyword pattern matching, which could create the illusion of conversation while remaining brittle and limited. Their failures show a core lesson that still matters today: a machine can imitate meaningful behavior without actually grasping meaning, so users must be careful not to confuse smooth output with real understanding.

It then shows how AI shifted from explicit rules to learning patterns from data. Instead of programmers trying to anticipate every case, models were trained to predict the next word in a sequence, first with recurrent neural networks and later with improved sequence models. This approach allowed systems to learn grammar, phrasing, and relationships between words from large datasets, but it also depended on enough quality data and enough computing power to train at scale. The chapter emphasizes that bad, biased, or outdated data leads to unreliable results, and that data alone is not enough without effective algorithms and hardware.

Finally, the chapter explains why attention and transformers changed everything. Attention lets models focus on the most relevant parts of context, making it possible to handle longer inputs and scale training efficiently across modern hardware. Once this architecture was combined with massive datasets and large computation, models became much more general, able to answer questions, write code, summarize text, translate, and even work across text, images, audio, and video. Even so, they still do not think like humans; they are powerful pattern engines that generate likely continuations, which is why they can be both remarkably useful and sometimes confidently wrong.

The general mechanism of rule-based systems. A fixed rule library written in advance by humans defines the set of triggers, conditions, and actions the system can recognize. When input arrives, the rule-based system scans for a matching condition and fires the associated action. No learning occurs: the system can only handle situations its designers anticipated. Anything outside the ruleset causes failure or a degraded response. ELIZA is a direct instance of this architecture.

Step-by-step processing of the sentence “I left my umbrella at home, and now it's…” by an RNN. At each step, the model reads one word (or group of words), updates its internal memory, and passes that memory forward to the next step. By step 3, the accumulated context that someone left an umbrella at home is sufficient for the model to predict a contextually appropriate continuation, such as "raining."

How attention selects relevant context. Instead of carrying context forward only step by step, attention allows a model to directly attend to the most relevant parts of the input. In this example, when predicting the word after “now it’s…,” the model assigns more weight to words such as “umbrella” and “at home,” making “raining” a more likely continuation.

Early multimodal AI outputs often contained noticeable errors, such as misspelled labels like VIDUAL instead of VISUAL, highlighting the limitations of early models in generating accurate text within images, even when the overall structure appeared coherent.

Modern image generation models can produce highly realistic scenes, with accurate details, natural lighting, and coherent composition, often making the generated images appear almost indistinguishable from real photographs.

A high-level timeline of language models and AI systems (1950s–present), showing the progression from early rule-based systems and chatbots to neural networks, sequence models, and modern large language and multimodal systems, along with recurring cycles of progress and setbacks known as AI winters.

• Early systems like ELIZA show that a machine can appear intelligent without actually understanding language or meaning. Rule-based AI can follow predefined instructions, but it struggles to adapt to situations beyond those anticipated by its designers.
• Modern AI systems move away from hand-written rules and instead learn patterns directly from data. LLMs generate text through prediction: given a context, they continuously estimate what is most likely to come next.
• Learning from data makes AI systems far more flexible and capable, but data alone is not enough. Effective AI also depends on computational power and algorithms that can use both efficiently.
• Early sequence models, such as RNNs, introduce the idea of context-based prediction, but they struggle to maintain long-range dependencies across complex inputs. Attention mechanisms and transformer architectures solve this bottleneck by allowing models to focus directly on the most relevant parts of the context.
• Once attention makes large-scale learning practical, AI systems begin to generalize across many tasks rather than remain narrow, task-specific systems. Modern multimodal systems extend these capabilities beyond text, combining language, images, audio, video, and code within the same model.
• AI progress accelerates when data, computation, and algorithms improve together. When these conditions align, capabilities can advance much faster than expected.
• Despite their capabilities, modern AI systems still do not think or understand like humans. They learn and reproduce statistical patterns from enormous amounts of data. Understanding AI as large-scale pattern prediction—not human-like reasoning—is the key to understanding both its power and its limitations.

1. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
2. Bengio, Y., Ducharme, R., Vincent, P., & Janvin, C. (2003). A neural probabilistic language model. Journal of Machine Learning Research, 3, 1137–1155. http://www.jmlr.org/papers/v3/bengio03a.html
3. Elman, J.L. (1990), Finding Structure in Time. Cognitive Science, 14: 179-211. https://doi.org/10.1207/s15516709cog1402_1
4. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
5. Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. OpenAI. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
6. Turing, A. M. (1950). Computing machinery and intelligence. Mind, 59(236), 433–460. https://doi.org/10.1093/mind/LIX.236.433
7. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 30). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

FAQ