9 Working with Bags and Arrays

This chapter broadens the Dask toolkit beyond DataFrames by introducing Bags for unstructured, mixed-type collections and Arrays for higher-dimensional numerical work. It explains how Bags mirror Python lists at scale—ideal for raw text, nested JSON, and irregular objects—while Arrays parallelize NumPy for linear algebra and statistics. Using a large set of Amazon product reviews as a running example, the chapter shows how to read text lazily with read_text, manage partitioning with blocksize, and resolve character-encoding issues. It then demonstrates a streaming, delimiter-based parsing strategy that avoids loading whole files into memory: locate byte ranges for records, map parsing functions to those ranges, and assemble a Bag of per-review dictionaries. The result is a clean, parallel-ready collection that preserves Dask’s lazy execution model and scales to much larger datasets.

Data manipulation with Bags is presented in a functional style: you transform collections with map, select records with filter, and aggregate with reduce-like operations. Examples include extracting numeric scores, tagging sentiment based on thresholds, and confirming immutability (transformations produce new Bags without altering the original). The chapter builds richer filters, such as computing a “helpfulness” ratio to drop low-signal reviews, and compares means and average review lengths to reveal that longer reviews are more likely to be deemed helpful. It introduces foldby for grouped aggregations across partitions, explaining binop and combine functions and using them to count reviews per star rating—surfacing the dataset’s polarization toward five- and one-star reviews. Throughout, progress bars and compute calls make the lazy plan explicit and measurable.

To bridge unstructured and structured analysis, the chapter converts Bag outputs to Arrays and DataFrames, supplying meta to to_dataframe and applying familiar descriptive statistics. It then integrates NLTK for parallel text mining: extract and lowercase review text, tokenize, remove stopwords, build bigrams, flatten nested structures, and use foldby plus topk to surface the most frequent phrases. Iteratively refining stopwords cleans out HTML artifacts and site-specific tokens, after which separate analyses for negative reviews highlight distinct complaint patterns (for example, phrases around taste or wasted money). The key takeaway is a practical workflow: parse messy text into a Bag, transform and filter it functionally, fold for grouped aggregates, and materialize Arrays/DataFrames for numerical routines—leveraging Dask’s parallelism end-to-end for scalable, flexible analysis.

An example comparison of structured and unstructured data

The Data Science with Python and Dask Workflow

There’s a pattern that we can use to split the text into individual reviews

Our custom text parsing algorithm to implement using Dask Delayed

A visual representation of the recursive delimiter search function

Mapping the parsing code to the review data

An example of a fold operation

The descriptive statistics of the Review Scores and Helpfulness

An example of stopwords

• Unstructured data, such as text, doesn’t lend itself well to being analyzed using DataFrames. Dask Bags are a more flexible solution that are useful for manipulating unstructured data.
• Bags are unordered and do not have any concept of an index (unlike DataFrames). To access elements of a Bag, the take method can be used.
• The map method is used to transform each element of a Bag using a user-defined function.
• The foldby function makes it possible to aggregate elements of a Bag before mapping a function over them. This can be used for all sorts of aggregate-type functions.
• When analyzing text data, tokenizing the text and removing stopwords helps extract the underlying meaning of the text.
• Bigrams are used to extract phrases from text that may have more meaning than their constituent words (e.g. “not good” versus “not” and “good” in isolation).

FAQ