1 Peeking inside the black box

This chapter introduces generative AI as both familiar and puzzling: while tools like ChatGPT and Copilot are easy to use, building them reveals that they behave very differently from traditional software. The book’s purpose is to help readers develop a practical mental model for designing GenAI applications, focusing on the main building blocks, how they fit together, and the limitations they bring. It also emphasizes hands-on learning through low-code development, while keeping the underlying ideas usable across tools, vendors, and frameworks.

At a high level, the chapter explains that GenAI programming is unique because an application must work around an LLM, a probabilistic black box that takes natural-language input and produces natural-language output. Unlike conventional code, which is deterministic and explicitly controlled, LLMs are non-deterministic, stateless, and pre-trained, so they do not remember prior turns or know new information unless the application supplies it. Real GenAI systems solve this by adding components such as conversation memory, external knowledge, retrieval, tools, and prompt engineering, which extend the model beyond what it can do alone.

The chapter then opens a window into how LLMs work internally, using GPT-3 as the main example. It describes a simplified view in which text is tokenized, converted into embeddings, combined into context, and used to predict the next word, then contrasts that toy explanation with the real scale of modern models built with deep neural networks, transformers, and massive training datasets. It closes by stressing that LLM knowledge is fixed at training time, so up-to-date or domain-specific information must be injected during inference or added through fine-tuning, which is why building effective GenAI applications is really about orchestrating the model with the right surrounding systems.