11 Iterators and generators
This chapter introduces Python’s iterator protocol and shows why iteration is central to the language. It explains how many built-in objects work naturally in for loops and outlines three complementary ways to create iterables: implementing the protocol in classes, writing generator functions with yield, and using generator expressions. Along the way, it emphasizes the benefits of iteration for clarity, flexibility, and memory efficiency, and frames iteration as both a pervasive pattern and a practical design target for your own types.
The core mechanics are demystified by detailing the protocol: iter(obj) calls obj.__iter__ to produce an iterator, next(it) calls it.__next__ to get successive values, and StopIteration ends the loop. The chapter clarifies the difference between iterables and iterators, why Python’s for loop doesn’t need indexes, and why different types yield different elements (e.g., strings yield characters, dicts yield keys). It then shows how to make classes iterable, first with a simple “return self” iterator (demonstrated by a LoudIterator) and then with a more robust design that returns a separate helper iterator object, enabling reuse and cleaner separation of concerns.
Through hands-on exercises, you implement common patterns and deepen intuition: a MyEnumerate that pairs indexes with values; a Circle that wraps a sequence to repeat elements a fixed number of times (using modulus) and can be expressed with a helper class or a generator expression; a generator that yields all lines across files in a directory; a timing generator that reports elapsed time between items; and mychain, a compact reimplementation of itertools.chain. The chapter closes by reinforcing when to favor classes versus generators, how to compose iterators to filter or enrich data, and why generator expressions often provide a more efficient alternative to list comprehensions.
Conclusion
Python code is full of for loops, and for good reason: They are easy to understand, easy to work with, and just about every data structure supports them, thanks to the iterator protocol. In this chapter, we explored how to create objects that support the iterator protocol, and can thus be used inside of a for loop. First, we saw how we can make our classes iterable via the iter and next methods. Then we wrote generator functions, which return an iterable object. Generators are particularly popular among people working with data sets too large to fit comfortably into a computer’s memory.
Now that you’ve practiced working with the iterator protocol in a variety of ways, you’ll be able to write better classes and generator functions. And when something goes wrong in one of your programs' for loops, you’ll have a better sense of why that happened.
FAQ
What is Python’s iterator protocol?
The iterator protocol is a small set of rules that make an object work in a for loop. It consists of: - An iterable that implements __iter__(), which returns an iterator - An iterator that implements __next__(), which returns the next item - Raising StopIteration from __next__() to signal the end Built-ins iter(obj) and next(it) call obj.__iter__() and it.__next__(), respectively.How does a for loop work under the hood?
- Calls iter(target) to get an iterator - Repeatedly calls next(iterator) to fetch items - Stops when next raises StopIteration Because the iterator produces items, Python doesn’t need explicit indices. Different iterables can yield different things (e.g., strings yield characters, dicts yield keys, files yield lines).Why can’t I do “for i in 5”? How do I loop N times?
Integers don’t implement the iterator protocol, so they aren’t iterable. To loop N times, use range(N), which is iterable:for _ in range(5): ...What’s the difference between an iterable and an iterator?
- Iterable: an object you can loop over; it implements __iter__() and returns an iterator. - Iterator: the object that actually produces values; it implements __next__(). Some objects are both (e.g., file objects). Others (e.g., str, list) return a separate iterator when you call iter() on them.How do I make my own class iterable?
Implement: - __iter__(self): return self (or a new helper iterator object) - __next__(self): return the next value or raise StopIteration Keep iteration state (e.g., an index) on the iterator object, and remember to advance it in __next__().When should __iter__ return a new helper iterator object instead of self?
Return a new helper iterator when: - You want to iterate over the same object multiple times independently (e.g., two for loops in a row) - You want to avoid shared, mutable state between concurrent iterations - You prefer cleaner separation of responsibilities (iterable vs. iterator)What are generator functions and how does yield work?
A generator function is defined with yield. Calling it returns a generator (an iterator). Each iteration runs the function until the next yield, returns that value, and pauses. When the function exits, StopIteration is raised automatically. Don’t raise StopIteration inside a generator; use return to stop early.What are generator expressions, and when should I use them?
Generator expressions look like list comprehensions but use parentheses:(expr for x in data). They are lazy and memory-efficient (produce one item at a time) and implement the iterator protocol. Use them when you don’t need the whole result list in memory. You can even return a generator expression from __iter__.How can I reimplement useful iterators like enumerate or chain?
- enumerate: yield (index, item) pairs from an iterable. You can implement a MyEnumerate class (iterator) or write a generator function. - chain: yield items from multiple iterables in sequence. A simple generator works:def mychain(*iters):
for it in iters:
for item in it:
yield item
- all_lines(path): a generator that yields lines from all files in a directory; use os.listdir, os.path.join, and yield each line; ignore unreadable files.
Python Workout, Second Edition ebook for free
