6 Supervised fine-tuning: Maximum expressiveness

Supervised fine-tuning is presented as the step up from LoRA when a small adapter can no longer fully reshape a model’s behavior. Unlike LoRA, full SFT updates every parameter, which gives the optimizer much more freedom to learn rigid output formats, domain vocabulary, and multi-step response patterns, but also increases memory use, training time, and the risk of overfitting or forgetting. The chapter’s running example builds an IT support assistant from a cleaned, reformatted Stack Exchange-based dataset mixed with a small general-purpose slice, then trains a Qwen3-4B model with conservative hyperparameters and gradient checkpointing so the run fits on modest hardware.

The training workflow emphasizes discipline in preparation and monitoring. Data is transformed into chat-style JSONL with a consistent house format, then split into train, validation, and held-out test sets so performance can be measured honestly. During training, the most important signals are training loss, validation loss, and gradient norm: healthy runs show loss falling, validation loss flattening rather than rising, and gradient norms stabilizing after early spikes. The chapter also recommends smoke tests before long runs, continuous metric tracking with tools like TensorBoard or W&B, and simple scripts that inspect the saved training log for early signs of overfitting or instability.

Evaluation is treated as both quantitative and behavioral. For the IT support example, the fine-tuned model improves Token-F1 only modestly over the base model, which the chapter warns may understate the real benefit because open-ended generation is poorly captured by overlap metrics. More revealing are behavioral checks: the SFT model reliably follows the desired structured-answer format, retains general knowledge, and still passes basic safety tests, though a separate safety regression suite shows that harmful-request refusal can weaken after fine-tuning. The chapter’s broader takeaway is that SFT is worth the extra cost when a task needs consistent structure, deeper domain behavior, or stronger control than LoRA can provide, but it should be chosen only after simpler methods fall short, with careful attention to data quality, validation, and ongoing maintenance.

The IT support data-preparation pipeline: source real questions and answers from Stack Exchange, mix and balance with a Dolly slice, then normalize each answer to the house format.

Training metrics for the SFT run (Qwen3-4B, 450 IT support examples, 3 epochs, two 24 GB test GPUs): training loss falls steadily while validation loss stays flat near 2.14, the gradient norm settles into the 3 to 4 range, and the learning rate follows a cosine decay.

The four training loss patterns encountered during fine-tuning.

SFT decision tree

• Full-parameter SFT updates every weight in the model, giving the optimizer maximum freedom to reshape behavior at the cost of higher compute, greater overfitting risk, and harder rollback than LoRA.
• SFT excels at complex structured output, multi-step domain reasoning, and strict behavioral consistency; for simpler tasks, LoRA provides nearly equivalent quality at lower cost.
• Fine-tuning changes how a model behaves, not what it knows; use RAG for knowledge gaps and fine-tuning for behavior gaps.
• Data quality dominates outcomes: 500 curated examples outperform 5,000 noisy ones, and 300 to 1,500 well-curated examples suffice for moderate tasks.
• The training pipeline uses TRL’s SFTTrainer with gradient checkpointing; the only difference from LoRA training is omitting the PEFT configuration and lowering the learning rate.
• Monitor validation loss as the primary health signal; a widening gap between training and validation loss indicates overfitting.
• Catastrophic forgetting, overfitting, and distribution mismatch are the three primary risks; run safety regression tests after every training run.
• Always compare fine-tuned performance against the base model, not against an absolute standard, and define success thresholds before training begins.
• Tune one hyperparameter at a time in order of impact: learning rate, then epochs, then batch size; stop tuning when gains fall below one percentage point per run.
• SFT is the foundation for preference optimization: the SFT checkpoint becomes the starting model for DPO or GRPO training in chapter 8.
• The decision to fine-tune is also a decision to maintain; factor the full lifecycle cost (retraining, evaluation, serving infrastructure) into the ROI calculation.

FAQ