4 Instruction Fine-tuning

Instruction fine-tuning is presented as the practical bridge between pretrained language models and the RLHF pipeline. Early models could continue text well, but they were not naturally organized to follow explicit user requests, so a question like “What is the boiling point of water?” might produce an irrelevant continuation instead of a direct answer. Instruction tuning addresses this by training models on instruction-response examples, teaching them to treat prompts as tasks to complete rather than text to extend, which makes them more useful and sets up the later stages of preference learning and reinforcement optimization.

The chapter explains that this approach became widespread because two trends converged: NLP research moved toward unified text-to-text and instruction-style training, and scaling plus in-context learning showed that general models could transfer across many tasks. Once instruction following was shown to improve when it was explicitly trained, large collections of prompts and responses became the standard way to adapt models. The result is a supervised fine-tuning stage, often called SFT or instruction tuning, that produces a model better aligned with the format and behavior expected in conversational and assistant settings.

A major focus is on how instruction-tuned systems are formatted in practice. Conversations are encoded with chat templates that assign roles such as system, user, and assistant, then serialize them into token sequences the model can learn from. The chapter also highlights best practices: use high-quality data, focus on prompts similar to downstream use, and recognize that around a million prompts can be enough for strong post-training performance, with diminishing returns beyond that. Implementation details differ from pretraining by using smaller batch sizes, masking prompt tokens, handling multi-turn data carefully, and using much smaller learning rates, all of which help the model learn responses without overwriting the general knowledge gained during pretraining.

• Instruction fine-tuning (IFT/SFT) teaches pretrained language models to respond in an instruction-response format, and is the foundation that all later post-training stages – from preference data collection to RLHF optimization – depend on.
• Chat templates define how user queries, system prompts, and assistant responses are formatted into token sequences using special tokens, and are the standard interface between users and instruction-tuned models.
• Successful instruction tuning depends on high-quality data, prompt distributions that match downstream tasks, and choosing scale appropriately within the broader post-training pipeline.
• Implementation details include smaller batch sizes than pretraining, prompt masking so the model learns responses rather than queries, and multi-turn masking strategies that control which assistant turns are trained on.

FAQ