8 Text Generation with Recurrent Neural Networks (RNNs)

This chapter shifts the book’s focus to text generation and explains why modeling language is challenging: meaning depends on strict word order, long-range context, and ambiguous, culturally nuanced usage. It introduces recurrent neural networks (RNNs) as architectures built for sequences, emphasizing how hidden states carry information over time. Long Short-Term Memory (LSTM) networks are highlighted as a stronger RNN variant that better preserves long- and short-term dependencies, enabling more coherent language modeling while still having limitations on very long contexts. The chapter’s goal is to teach the core principles of text generation with an LSTM and use those lessons to motivate more advanced Transformer models later.

The workflow centers on preparing and modeling text at the word level. After selecting a suitably sized training corpus (the novel Anna Karenina), the text is cleaned and tokenized; vocabularies and integer mappings are built; and the corpus is converted to sequences of fixed length, with targets formed by shifting inputs one token to the right for next-token prediction. The chapter contrasts character, word, and subword tokenization, then focuses on word tokenization for clarity. To overcome inefficiencies of one-hot vectors, an embedding layer learns dense representations that capture semantic relationships. A PyTorch model stacks an embedding, multi-layer LSTM, and a linear output over the vocabulary, and is trained with cross-entropy loss and Adam on mini-batches, then saved alongside the tokenizer mappings for inference.

Generation proceeds by feeding a prompt and repeatedly sampling the next token from the model’s predicted distribution, appending each choice back into the context. The chapter shows how temperature scaling and top-K sampling shape output: lower temperature and small K favor safer, more coherent text, while higher temperature and large K increase diversity but can reduce coherence. The resulting LSTM generations are grammatically reasonable and stylistically influenced by the training data, though not on par with state-of-the-art systems. The key takeaway is a solid foundation in tokenization, embeddings, sequence modeling, and sampling strategies, preparing readers for upcoming chapters on attention and Transformer architectures.

How a layer of recurrent neurons unfolds through time. When a recurrent neural network makes a prediction on sequential data, it takes the hidden state from the previous time step, h(t-1), along with the input at the current time step, x(t), and generates the output, y(t), and the updated hidden state, h(t). The hidden state at time step t captures the information in all previous time steps, x(0), x(1), …, x(t).

An example of how an LSTM model is trained. We first break down the training text into tokens and assign a unique integer to each token, creating a numerical representation of the text as a sequence of indexes. We then divide this long sequence into shorter sequences of equal length. These sequences form the features (the x variable) of our model. We then generate the output y by shifting the input sequence one token to the right. This setup enables the LSTM model to predict the next token based on previous tokens in the sequence.

• Recurrent neural networks (RNNs) are a specialized form of artificial neural network designed to recognize patterns in sequences of data, such as text, music, or stock prices. Unlike traditional neural networks, which process inputs independently, RNNs have loops in them, allowing information to persist. Long short-term memory (LSTM) networks are improved versions of RNNs.
• There are three approaches to tokenization. The first is character tokenization, where the text is divided into its constituent characters. The second approach is word tokenization, where the text is split into individual words. The third approach is subword tokenization, which breaks words into smaller, meaningful components called subwords.
• Word embedding is a method that transforms words into compact vector representations, capturing their semantic information and interrelationships. This technique is vital in NLP, especially since deep neural networks, including models like LSTM and Transformers, require numerical input.
• Temperature is a parameter to influence the behavior of text generation models. It controls the randomness of the predictions by scaling the logits (the inputs to the softmax function for probability calculation) before applying softmax. Low temperature makes the model more conservative in its predictions but also more repetitive. At higher temperatures, the model becomes less repetitive and more innovative increasing the diversity of the generated text.
• Top-K sampling is another way to influence the behavior of text generation models. It involves selecting the next word from the K most likely candidates, as determined by the model. The probability distribution is truncated to keep only the top K words. Small values of K make the output more predictable and coherent but potentially less diverse and interesting.

FAQ