4 Managing data
This chapter emphasizes managing data rigorously between exploration and modeling so models are trained on valid, consistent, and informative inputs. It frames the workflow as: identify issues found during exploration, fix data quality problems, transform variables to make patterns easier to learn, and organize datasets for reliable modeling and validation. The overarching goal is to create “safe” data—free of surprises like missing or out-of-range values—and to preserve reproducibility so results can be trusted and repeated.
The chapter first tackles cleaning: invalid values are corrected using domain knowledge (e.g., recoding out-of-range entries, splitting mixed-code fields into explicit indicator variables), while missing values are handled in ways that respect both statistical assumptions and business semantics. Categorical NAs can be assigned an explicit “missing” level; numeric/logical NAs can be imputed (often by the mean), ideally alongside “missingness indicators” that let models learn whether the absence itself is informative. Because consistent treatment must be applied to both training and future data, automated tooling like vtreat is introduced to record treatment plans, add *_isBAD indicators, and reliably prepare new data. The chapter then covers key transformations that aid modeling: normalization (including conditional scaling against external baselines such as state medians), centering and scaling to comparable units across variables, and log transformations for skewed or wide-range quantities, with signed-log variants for data that include zeros or negatives. It also stresses retaining training-time parameters (means, scales, baselines) to keep future data on the same footing as the training data when appropriate.
Finally, the chapter organizes data for modeling and evaluation through careful sampling. It motivates representative, reproducible test/train splits and suggests storing a persistent random “group” mark to repeatedly draw the same samples during development and regression testing. It shows how to respect record groupings (such as households) so related rows are kept together across splits, and discusses when a calibration set is useful. The chapter closes by encouraging the capture of data provenance—source, collection date, and treatment version—so teams can track lineage, compare results fairly, and reliably re-treat or refresh datasets as processes evolve.
Figure 4.1. Chapter 4 Mental Model
Figure 4.2. Even a few missing values can lose all your data
Figure 4.3. Creating a new level for missing categorical values
Figure 4.4. Income data with missing values
Figure 4.5. Replacing missing values with the mean
Figure 4.6. Replace missing values with the mean and add an indicator column to track altered values
Figure 4.7. Creating and applying a simple treatment plan
Figure 4.8. Is a 35-year-old young?
Figure 4.9. Is a 35-year-old young?
Figure 4.10. A nearly lognormal distribution and its log
Figure 4.11. Signed log lets you visualize non-positive data on a logarithmic scale.
Figure 4.12. Splitting data into training and test (or training, calibration and test) sets
Figure 4.13. Example of dataset with customers and households
Figure 4.14. Example of dataset with customers and households
Figure 4.15. Recording data source, collection date, and treatment date with data
Summary
At some point, you’ll have data quality that is as good as you can make it. You’ve fixed problems with missing data and performed any needed transformations. You’re ready to go on the modeling stage.
Remember, though, that data science is an iterative process. You may discover during the modeling process that you have to do additional data cleaning or transformation. You may have to go back even further and collect different types of data. That’s why we recommend adding columns for sample groups and data provenance to your datasets (and later, to the models and model output), so you can keep track of the data management steps as the data and the models evolve.
In part 2 of the book, we’ll talk about the process of building and evaluating models to meet your stated objectives.
In this chapter you have learned:
- Different ways to handle missing values, and when different approaches are suitable.
- How to use the
vtreatpackage to manage missing values automatically.
- How to normalize or rescale data, and when normalization/rescaling are appropriate.
- How to log transform data, and when log transformations are appropriate.
- How to implement a reproducible sampling scheme for creating test/train splits of your data.
Practical Data Science with R, Second Edition ebook for free
