Overview

6 Summarizing and Analyzing DataFrames

This chapter shifts from data preparation to exploratory analysis in a distributed setting, showing how to summarize and probe large DataFrames efficiently with Dask. It frames core EDA goals—understanding distributions, discovering relationships, and detecting temporal patterns—while highlighting Dask-specific considerations such as partitioning, shuffling, task graphs, and when to persist or checkpoint work (for example, by writing Parquet). The tour covers built-in descriptive statistics, groupby/aggregate operations for correlation analysis, defining custom aggregations to implement statistical tests at scale, and applying rolling window functions to time-ordered data.

Using the NYC parking tickets dataset, the chapter first derives vehicle age, filters invalid years, and persists intermediate results to speed repetitive analyses. Descriptive statistics are computed via Series methods (mean, std, min, max) and dask.array for skewness, as well as describe for an efficient multi-metric pass. Interpreted results show an average vehicle age of about 6.74 years (std ≈ 5.66), with a minimum of 0, a maximum of 47, and positive skew (~1.01), implying a longer tail of older vehicles but more observations younger than the mean. It then demonstrates grouped aggregation to count citations by month and correlate them with average temperature; careful sorting of month-year labels and filtering incomplete months are key. The measured correlation is positive but weak (~0.14), reinforcing the reminder that correlation should not be confused with causation and that partitioning by the grouping column helps performance.

The chapter then builds a custom Aggregation to implement the Brown–Forsythe test for equal variances between private (PAS) and commercial (COM) vehicles, assembling chunk, combine, and finalize steps to compute the F statistic in parallel. The test rejects equal variances, so Welch’s t-test is run on locally collected NumPy arrays, yielding a decisive p-value and indicating the groups’ mean ages differ. It closes with rolling window functions: preparing a numeric, index-aligned month-year, noting that windows must not span beyond adjacent partitions, and applying a three-period moving average with optional centering to smooth monthly citation counts. Throughout, the guidance emphasizes scaling-aware patterns—aligning groupby keys with partitions, minimizing shuffles, persisting expensive transforms, and judiciously collecting to memory only when safe.

The Data Science with Python and Dask workflow
Description: A screenshot of a cell phone Description automatically generated
A visual guide to descriptive statistics
A comparison of standard deviation
A visual comparison of skewness
Calculating the age of each vehicle at the time a citation was issued
A visual guide to correlation
An example of the split-apply-combine algorithm of an aggregate function
An abbreviated count of citations by month-year
The correlation matrix for Count of Summons and Temp
The process of statistical hypothesis tests
The process we will use to determine if mean vehicle age has any relationship with vehicle type
The Brown-Forsythe Test for Equal Variances
The first part of the Brown-Forsythe test
The process for calculating the right denominator
Our progress so far calculating the right denominator of the Brown-Forsythe equation
The process to calculate the right numerator of the Brown-Forsythe equation
We’ve rejected the null hypothesis of the Brown-Forsythe test, so we will next run Welch’s T-Test

   Summary

In this chapter you learned

  • Dask DataFrames have a number of useful statistical methods such as mean, min, max, etc. Even more statistical methods can be found in the dask.array.stats package.
  • Basic descriptive statistics can be produced for a DataFrame or Series by using the describe method.
  • Aggregate functions use the split-apply-combine algorithm to process data in parallel. Aggregating on a sorted column of a DataFrame will yield the best performance.
  • Correlation analysis is used to compare two continuous variables, whereas the t-test is used to compare a continuous variable across a categorical variable.
  • The Aggregate object can be used to define your own aggregate functions.
  • Rolling functions can be used to analyze trends across a time index, such as moving averages. The data should be partitioned by time period for the best performance.

FAQ

What descriptive statistics should I know, and how do I compute them with Dask?Common descriptive statistics include minimum, maximum, mean, median, standard deviation, mode, and skewness. In Dask: - Use Series methods with compute(): s.mean().compute(), s.std().compute(), s.min().compute(), s.max().compute(). - Median: s.quantile(0.5).compute() (Dask uses quantile; there’s no median() method). - Skewness: convert to a Dask Array and use dask.array.stats.skew: dask_stats.skew(s.values).compute().
Why might I persist or save intermediate results (e.g., to Parquet) before analysis?Many analytics (means, std, rolling, groupby-agg) scan the entire dataset and can be expensive, especially with object/datetime operations. Persisting keeps computed results in memory across tasks; saving to Parquet: - Avoids recomputation later (resume work without re-running heavy transforms). - Improves performance by leveraging efficient columnar storage and predicate pushdown. - Decouples long-running steps from your notebook session.
How do I derive a Vehicle Age column correctly from Issue Date and Vehicle Year?Steps: - Clean anomalies (filter invalid years like 0 or obviously future model years). - Compute age per row: Issue Date’s year minus Vehicle Year. - Drop negative ages (e.g., misrecorded model year > citation year). - Add as a new column using apply with meta, then assign/rename. This prevents bad inputs from skewing stats.
What does Series.describe() do in Dask, and why can it be faster than separate calls?Series.describe() computes a bundle of stats in one pass: count, mean, std, min, 25%, 50% (median), 75%, max. It can be more efficient than multiple separate compute() calls because Dask can optimize a combined graph of aggregations and reduce passes over the data.
How do groupby and aggregate functions work in Dask, and what are performance tips?Dask uses split-apply-combine: - Split data into groups (groupby), - Apply aggregations (count, sum, mean, min/max, etc.) within groups, - Combine partial results. Tips: - Prefer grouping on a column that matches your partitioning to minimize shuffles. - Remember count counts non-null values; if the chosen column has nulls, use a reliable column or size if applicable. - Use agg with a dict to compute multiple aggregates in one go for efficiency.
How can I measure correlation (e.g., citations vs. temperature) with Dask DataFrames?Typical workflow: - Prepare a monthly grouping key (e.g., "Citation Issued Month Year") and ensure chronological sorting if needed. - Group by month-year and compute count of citations and mean monthly temperature via agg({'Summons Number': 'count', 'Temp': 'mean'}). - Use corr().compute() on the aggregated DataFrame to get the correlation matrix. - Interpret sign (positive/negative) and magnitude (weak/moderate/strong), and remember correlation does not imply causation.
How do I implement a custom aggregate in Dask (e.g., part of the Brown–Forsythe test)?Use dd.Aggregation(name, chunk, agg, finalize): - chunk: function applied to each partition (e.g., return (count, sum)). - agg: merges all partition-level results (e.g., sum counts and sums). - finalize (optional): last transformation (e.g., turn totals into group-level statistic using a precomputed constant like a grand mean). Apply with groupby(...).agg({'column': custom_agg}). This lets you express complex grouped computations without manual shuffles.
When should I use Welch’s t-test vs. Student’s t-test, and how do I check equal variances?- Student’s t-test assumes equal variances between groups; Welch’s does not. - To check the equal-variance assumption when distributions may be skewed, use the Brown–Forsythe test (robust to non-normality). If you reject equal variances, set equal_var=False and use Welch’s t-test (e.g., scipy.stats.ttest_ind(a, b, equal_var=False)). Many practitioners default to Welch’s due to its robustness.
How do I compute a median in Dask, and why use quantile() instead?Dask DataFrame/Series doesn’t provide median(); use quantile(0.5) to get the median. Under the hood, quantile uses approximate/distributed algorithms suitable for large datasets; 0.5 corresponds to the 50th percentile (the median).
What should I know about rolling (window) functions in Dask?Key points: - Use rolling(window).agg() (e.g., mean, sum, count) on a Series indexed in the intended order. - Dask requires index alignment and a window that does not span more than one adjacent partition; choose partitioning and window size accordingly. - Edges produce NaNs until the window is “full”; center=True uses both past and future periods around the index, changing which rows become NaN. - Rolling functions are great for smoothing (e.g., 3-month moving average) to reveal trends and cycles.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

choose your plan

pro

monthly
annual
$24.99
$249.99
only $20.83 per month
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Data Science with Python and Dask ebook for free

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Data Science with Python and Dask ebook for free
choose your plan

pro

monthly
annual
$24.99
$249.99
only $20.83 per month
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Data Science with Python and Dask ebook for free

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Data Science with Python and Dask ebook for free
choose your plan

pro

monthly
annual
$24.99
$249.99
only $20.83 per month
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Data Science with Python and Dask ebook for free

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Data Science with Python and Dask ebook for free