6 Fine-Tuning Pre-Trained Models and Working with Multimodal Models
This chapter moves from passively using pre-trained models to actively adapting them for specific tasks. It lays out a practical, end-to-end fine-tuning workflow with Hugging Face—covering data preparation, tokenization, training, evaluation, saving, and inference—and then broadens the scope to multimodal modeling, where text and images are combined for richer understanding. Along the way, it emphasizes efficiency (leveraging transfer learning and hardware acceleration) and reliability (monitoring losses and validating results), preparing you to tailor general-purpose models to domain needs and to reason across multiple data types.
The fine-tuning portion centers on sentiment analysis. Using the yelp_polarity dataset, the chapter filters for restaurant-related reviews, optionally downsamples for faster iteration, tokenizes with a DistilBERT checkpoint, and configures a sequence-classification head. With Trainer and TrainingArguments, it sets sensible defaults (learning rate, batch sizes, epochs, weight decay, evaluation/saving cadence) and uses GPU or Apple Silicon acceleration when available. After training, the model and tokenizer are saved, evaluation metrics are inspected, and inference on new reviews demonstrates confident positive/negative predictions. The same pipeline is then extended to multiclass sentiment with yelp_review_full by setting num_labels to five, enabling star-rating predictions (1–5) from review text.
The multimodal section contrasts single-modal models with those that fuse modalities. It first revisits a single-modal vision task using DETR to detect objects and draw bounding boxes from an image alone. Then it introduces CLIP, which aligns an image encoder and a text encoder to compare visuals with natural-language labels, enabling zero-shot image classification by choosing the label with the highest similarity. This illustrates how combining modalities yields more flexible, context-aware systems that generalize across tasks like image captioning, visual question answering, and multimodal sentiment analysis.
The result printed at the end of the training
The result printed at the end of the training
An image containing a dog and a cat (image source: https://unsplash.com/photos/short-coated-white-and-brown-puppy-Ivzo69e18nk)
The detected objects in the image with the bounding boxes drawn
An image of a dog (image source: https://unsplash.com/photos/close-up-photo-of-black-and-white-siberian-husky-dog-NKN25UfGfkQ)
Summary
Fine-tuning is a machine learning technique where a pre-trained model, which has already learned general patterns from a large dataset, is further trained on a smaller, domain-specific dataset to adapt it for a particular task.
The yelp_polarity dataset is derived from the larger Yelp Dataset Challenge and is specifically tailored for text classification tasks such as sentiment analysis.
Sequence classification tasks involve assigning a single label or category to an entire sequence of data, such as a sentence, paragraph, or even a longer sequence of tokens
Metal Performance Shaders (MPS) is Apple’s framework for GPU-accelerated computations on macOS devices with Apple Silicon (M1/M2/M3/M4).
Compute Unified Device Architecture (CUDA) is NVIDIA’s parallel computing platform and API for leveraging NVIDIA GPUs for general-purpose processing.
To fine-tune a pre-trained model on a dataset, you utilize the Trainer and TrainingArguments classes from Hugging Face.
A single modal model is a machine learning model designed to work with data from a single modality, such as text, images, audio, or numerical data.
A multimodal model, on the other hand, is a machine learning model designed to process and integrate data from multiple modalities—different types of information such as text, images, audio, video, or structured data.
Contrastive Language–Image Pre-training (CLIP) is a multimodal model developed by OpenAI that connects text and image understanding.
FAQ
What is fine-tuning and why use it with Hugging Face models?Fine-tuning takes a pre-trained model that already learned general patterns and adapts it to a specific task or domain using a smaller, targeted dataset. This leverages prior knowledge to reach strong performance with less data and compute. In this chapter, DistilBERT is fine-tuned for restaurant-review sentiment, achieving better task-specific accuracy than zero-shot use.How do I load and filter the yelp_polarity dataset for restaurant reviews and create a manageable subset?Load the dataset with datasets.load_dataset("yelp_polarity"), then filter rows where "restaurant" appears in the text (e.g., using Dataset.filter with a lowercase match). To keep training practical, shuffle with a fixed seed and select a subset (for example, 5,000 rows) for both train and test using .shuffle(seed=42).select(range(n)).How should I tokenize the text for DistilBERT, and why use padding/truncation?Create a tokenizer with AutoTokenizer.from_pretrained("distilbert-base-uncased"). Apply it to the "text" column via Dataset.map with batched=True, using padding="max_length", truncation=True, and max_length=512. This ensures uniform sequence shapes for batching, prevents overly long inputs from breaking the model, and adds attention masks alongside input_ids.How do I set up a pre-trained sequence classification model and choose num_labels?Use AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=...). For binary sentiment on yelp_polarity, set num_labels=2. For multi-class star ratings (1–5) on yelp_review_full, set num_labels=5. The AutoModel class appends a classification head so you can train directly on your labeled dataset.How can I pick the right device (CPU, CUDA, or Apple Silicon MPS) for training and inference?Detect availability in PyTorch: prefer MPS on Apple Silicon if torch.backends.mps.is_available(); otherwise prefer CUDA if torch.cuda.is_available(); else fall back to CPU. Move the model and tensor inputs to the chosen device to leverage acceleration when available.Which Trainer settings matter most for fine-tuning, and what do they do?Key TrainingArguments include: learning_rate (e.g., 2e-5) for how quickly weights update; per_device_train_batch_size and per_device_eval_batch_size for memory/speed trade-offs; num_train_epochs to control training duration; weight_decay for regularization; evaluation and save strategies set to run each epoch; logging options for progress; and load_best_model_at_end to keep the best checkpoint automatically.How long will training take and how can I speed it up if it’s slow?Training time varies by hardware; expect minutes to about an hour in this chapter’s setup. To speed up: reduce dataset size (e.g., from 5,000 to ~1,000), lower sequence length, use a smaller batch size if memory is constrained, or train fewer epochs. Using GPU/MPS acceleration also helps substantially.How do I save, reload, and use the fine-tuned binary sentiment model for inference?After training, call model.save_pretrained(path) and tokenizer.save_pretrained(path). For inference, reload both, tokenize new text with return_tensors="pt", move tensors to the same device as the model, set model.eval(), wrap in torch.no_grad(), get logits, apply softmax to obtain probabilities, and use argmax to map to label 0 (negative) or 1 (positive).How do I adapt the pipeline for multi-class sentiment (1–5 stars) with yelp_review_full?Follow the same steps as binary fine-tuning but: use the yelp_review_full dataset, filter to restaurant reviews, set num_labels=5 when loading the model, and train with the Trainer. At inference, argmax returns class indices 0–4; add 1 to map to star ratings 1–5 before displaying results.What’s the difference between single-modal and multimodal models, and how do DETR and CLIP illustrate this?Single-modal models process one data type (e.g., DETR for images: object detection with bounding boxes). Multimodal models integrate multiple data types—text plus images, audio, etc.—to reason across inputs. CLIP pairs an image encoder with a text encoder to compute cross-modal similarity; you can classify an image by supplying candidate text labels, then picking the label with the highest image–text similarity. DETR localizes objects visually; CLIP aligns images to natural-language descriptions.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Hugging Face in Action ebook for free
