10 Train A Transformer to Translate English to French

This chapter walks through building an end-to-end English-to-French translation system with a Transformer encoder-decoder, using a dataset of over 47,000 parallel sentence pairs. It uses the project as a concrete case study to reinforce how self-attention, embeddings, and positional encodings work together inside the Transformer. The workflow covers preparing data with subword tokenization, constructing vocabularies, turning tokens into numerical sequences, training a Transformer from scratch, and finally translating English sentences into French with the trained model.

Data preparation begins with subword tokenization for both languages, adding BOS/EOS markers and reserving PAD/UNK tokens. Token-to-index and index-to-token dictionaries are built to convert phrases to sequences and back. Because sentences have variable lengths, sequences are batched with padding and length-based sorting to minimize wasted computation. Masks are created to hide padding in the encoder and to conceal both padding and future tokens in the decoder. Word embeddings compress sparse token indices into dense vectors (scaled by sqrt(d_model)), and sinusoidal positional encodings inject order information so attention can respect token positions. The Batch utility assembles inputs, shifted targets, and the required masks for efficient training.

Training uses a configurable Transformer (e.g., d_model=256, 8 heads), label smoothing to improve generalization and calibration, and Adam with a Noam warm-up schedule for stable optimization. A simple training loop accumulates token-normalized loss across epochs and saves the learned weights. At inference, a translate function encodes an English sentence and performs greedy, autoregressive decoding on the French side from BOS until EOS, then stitches subwords back into clean text. The model reliably translates common sentences, producing consistent outputs for semantically equivalent inputs (e.g., “don’t” vs. “do not”), demonstrating that the encoder captures meaning in a shared vector space and the decoder turns it into fluent French, setting up the transition to decoder-only Transformers in later chapters.

What does the Bach() class do? The Bach() class takes two inputs: src and trg, sequences of indexes for the source language and the target language, respectively. It adds several attributes to the training data: src_mask, the source mask to conceal padding; modified trg, the input to the decoder; trg_y, the output to the decoder; trg_mask, the target mask to hide padding and future tokens.

• Transformers process input data such as sentences in parallel, unlike Recurrent Neural Networks (RNNs) which handle data sequentially. This parallelism enhances their efficiency but doesn't inherently allow them to recognize the sequence order of the input. To address this, transformers add positional encodings to the input embeddings. These positional encodings are unique vectors assigned to each position in the input sequence and align in dimension with the input embeddings.
• Label smoothing is commonly used in training deep neural networks to improve the generalization of the model. It is used to address overconfidence problems (the predicted probability is greater than the true probability) and overfitting in classifications. Specifically, it modifies the way the model learns by adjusting the target labels, aiming to reduce the model's confidence in the training data, which can lead to better performance on unseen data.
• Based on the encoder’s output that captures the meaning of the English phrase, the decoder in the trained Transformer starts translating in an autoregressive manner, starting with the beginning token "BOS". In each time step, the decoder generates the most likely next token based on previously generated tokens, until the predicted token is "EOS", which signals the end of the sentence.

FAQ