This chapter explains that successful model adaptation starts with AI-ready data, not just any existing organizational data. Most teams begin with a small set of human-written examples and then expand from there, using a practical readiness spectrum to judge whether the data is clean, representative, and correctly formatted. The key idea is that even a modest number of high-quality seed examples can anchor style, tone, and domain behavior, while synthetic data can later help fill gaps without becoming the main source of training material.
It emphasizes that data quality matters more than raw volume, because mislabeled, imbalanced, truncated, or otherwise noisy examples can sharply reduce performance and create hidden failure modes. The first few hundred examples usually deliver the biggest gains, especially when they reflect the real production distribution and include rare edge cases, boundary conditions, and different user types. The chapter also distinguishes between base models and instruct models, showing that the model choice affects how much data you need and that training examples must match the prompt shape used at inference time.
The chapter then lays out four main acquisition strategies: human annotation, weak supervision, synthetic generation, and production log mining. It explains when each approach is useful, how to combine them, and why synthetic data should be quality-gated and kept below a limited share of the final mix. Finally, it covers the operational essentials that make adaptation safe and reproducible: versioning datasets, tracking experiments, preserving lineage, building a frozen golden test set, and removing personal data before training so privacy, compliance, and deletion obligations are handled from the start.
Data readiness spectrum for model adaptation
Decision Chart to help you prepare the data needed for model adaptation.
Accuracy versus number of training examples
Synthetic data generation pipeline - getting from seed data to a training dataset.
Summary
AI-ready data is labeled at the task level, reflects the production distribution, and is traceable. The data readiness spectrum (Levels 0 to 4) provides a practical framework for assessing where your data currently sits and the steps needed to make it ready for fine-tuning.
Data quality determines model quality more than any other single factor.
No single data acquisition strategy fits all situations. Human annotation produces the highest-quality labels and is most valuable for seed data and evaluation sets. Weak supervision scales to large corpora using programmable labeling functions. Synthetic data generation from a teacher model fills rare categories cheaply when paired with a quality gate. Production log mining provides temporal freshness and aligns with realistic distributions. Most production datasets combine at least two of these strategies.
Synthetic data generation follows a six-step pipeline: load seed examples from a public dataset, build a style-anchored generation prompt, call a teacher model once per category, apply an LLM-as-judge quality gate across three dimensions (fidelity, format, domain safety), check distribution alignment using centroid cosine similarity, and mix with real data at a maximum of 30% synthetic, per the heuristic cap set out in section 3.4.3.
Personal data must be scrubbed before any training example reaches an annotator, annotation platform, or training infrastructure. Once a model trains on personal data, removing it from the dataset does not remove it from the model. PII removal operates at two levels: structured identifiers (names, email addresses, account numbers) detected by tools such as Microsoft Presidio, and indirect identifiers (attribute combinations that uniquely identify an individual) that require human review.
What kind of data do I need for model adaptation?You need examples that reflect the task the model will perform in production. The best starting point is a small set of clean, representative human examples that show the desired input/output behavior. Then you can expand with weak supervision, synthetic data, or production logs as needed.How much data is enough to start fine-tuning?For many enterprise tasks, a first useful run can start with about 50 to 300 high-quality examples. The first 100 examples often provide most of the performance gain, so quality and coverage matter more than raw volume at the beginning.Should I use a base model or an instruct model if I have limited data?If you have fewer than about 300 examples, an instruct model is usually the better choice. A base model needs more data because it also has to learn how to follow instructions, while an instruct model already has that skill and can focus on learning your domain.Why is data quality more important than data quantity?Poor-quality data teaches the model the wrong patterns just as effectively as good data teaches the right ones. Bad labels, truncated fields, and imbalanced examples can seriously reduce accuracy, trust, fairness, and training efficiency, even if the dataset is large.How do I know whether my training data is representative?Compare your training set to real production traffic by category, user type, and scenario frequency. Make sure rare but important cases, such as outages or high-risk edge cases, are not underrepresented compared with the real workload the model will face.Why do edge cases matter so much in adaptation data?Edge cases often carry the most business risk, even though they are rare. A model that performs well on routine cases but fails on critical exceptions may still be unsafe to deploy, so you should deliberately collect and include hard variants and boundary conditions.Can I use synthetic data to fill gaps in my dataset?Yes, but synthetic data should complement real examples, not replace them. Start with a real seed set, use a teacher model to generate targeted examples, and keep synthetic data under about 30% of the training mix to avoid narrowing the distribution too much.What are the main data acquisition strategies for model adaptation?The chapter highlights four: human annotation, weak supervision, synthetic generation, and production log mining. Human annotation gives the highest label quality, weak supervision scales quickly on unlabeled data, synthetic generation fills rare gaps, and log mining captures real production distribution.What format should training data use for fine-tuning?Training examples should match the format the model will see at inference time, usually as chat-style messages with system, user, and assistant turns. If the application will always include a specific instruction prompt, that instruction should also appear in the training examples.How should I handle privacy, compliance, and governance for training data?Remove personal data before training, keep lineage records for every example, and version your datasets and experiments. You should be able to trace where each example came from, who labeled it, what transformations were applied, and which dataset version was used in each training run.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
LLM Customization and Fine-Tuning ebook for free
