This lesson on Generators, Iterators & itertools is hands-on and example-driven. You will be able to implement and inspect custom iterators using Python's iterator protocol, process unbounded streams with sentinel-based calls, and leverage itertools functions to replace complex loops with declarative data pipelines.
What You'll Be Able To Do
- Implement custom finite and infinite iterators by defining
__iter__and__next__dunder methods. - Differentiate between iterables and iterators to avoid state-exhaustion bugs during repeated traversals.
- Stream callable outputs and file reads using the two-argument sentinel form of
iter(). - Build memory-efficient streaming pipelines using
itertools.count,repeat,accumulate,permutations, andcombinations.
Detailed Concept Walkthrough
1. The Python Iterator Protocol
An iterator is a stateful traversal object that produces the next item in a sequence on demand or raises StopIteration when exhausted. An iterable is an object capable of returning an iterator when passed to iter().
- Mechanism: An object satisfies the iterator protocol by implementing
__iter__()(which returns the iterator object itself) and__next__()(which returns subsequent values). Calling the built-innext()on an exhausted iterator raisesStopIteration. - Under the Hood: Python's
forloop encapsulates this mechanics automatically. It requests an iterator viaiter(), continuously evaluatesnext(), and catchesStopIterationinternally to break execution without manual error handling. - Best Practice: Distinguish sequences (iterables) from active iterators. Calling
iter()on an iterable generates a fresh, independent iterator that restarts traversal, whereas callingiter()on an existing iterator merely returns its current stateful self.
countries = ("Germany", "France", "Greece")
country_iter = iter(countries)
# Manual iterator consumption
print(next(country_iter)) # Output: Germany
print(next(country_iter)) # Output: France
# Calling iter() on an iterator returns itself (does not reset)
iter_copy = iter(country_iter)
print(next(iter_copy)) # Output: Greece
# next(country_iter) now raises StopIteration
Key Takeaway: Iterables produce independent iterators on every iter() call, while iterators maintain mutable progress and return themselves.
2. Sentinel-Based Iteration Pattern
The built-in iter() function accepts a two-argument form that converts zero-argument callables into iterators terminated by a sentinel marker.
- Mechanism: Invoking
iter(callable, sentinel)repeatedly executes the zero-argument callable on eachnext()step. The iterator produces the returned value until the return matches the sentinel argument, at which point it raisesStopIteration. - Under the Hood: This eliminates manual buffering and explicit
while Trueloop boilerplate when interacting with stream-based or chunk-reading APIs. It evaluates lazily without loading the entire payload into RAM. - Best Practice: Use this pattern when reading IO buffers, line streams, or polling queues. For instance, reading lines with
iter(f.readline, '')halts cleanly when the empty string signals EOF.
# Process a text stream until reaching an empty string sentinel (EOF)
with open("countries.txt", "r") as f:
# f.readline is called repeatedly until it returns ""
for line in iter(f.readline, ""):
print(line, end="")
Key Takeaway: Two-argument iter(callable, sentinel) abstracts while-loop IO reads into clean, standard for-loop iteration.
3. Custom Class Iterators
Custom classes can implement the iterator protocol directly to produce sequence elements dynamically, supporting both bounded and infinite streams.
- Mechanism: A custom iterator class stores traversal state in instance attributes. Its
__iter__()method returnsself(typed withtyping.Self), while__next__()calculates the next state and returns the current value. - Under the Hood: Finite iterators enforce boundaries by raising
StopIterationexplicitly within__next__()when the termination condition is met. Infinite iterators omit this guard, generating values continuously. - Best Practice: When iterating custom objects across generic functions, annotate inputs with
collections.abc.Iterablerather than concrete types (likelistortuple) to maintain loose coupling across data structures.
from typing import Self
class NumberIterator:
def __init__(self, start: int, maximum: int) -> None:
self.number = start
self.maximum = maximum
def __iter__(self) -> Self:
return self
def __next__(self) -> int:
if self.number > self.maximum:
raise StopIteration
val = self.number
self.number += 1
return val
for num in NumberIterator(start=1, maximum=3):
print(num) # Prints 1, 2, 3
Key Takeaway: Custom iterators encapsulate arbitrary state transitions and terminate deterministically via StopIteration.
4. Functional Streaming with itertools
The itertools module provides composable, high-performance iterator algebra to build lazy data-processing pipelines without loop overhead.
- Mechanism:
itertoolsconstructs C-level iterator pipelines. Infinite generators likecount(start)generate endless progressions, whilerepeat(elem, n)produces fixed repetitions. Running computations likeaccumulate()yield intermediate states. - Under the Hood: Combinatoric iterators calculate permutations and combinations on-the-fly without allocating memory for the full combinatorial Cartesian product up front.
- Best Practice: Choose
permutations(iterable, r)when item order matters (where('a', 'b') != ('b', 'a')), andcombinations(iterable, r)when order is irrelevant.
import itertools
# Count infinitely from 10; break at threshold
for val in itertools.count(10):
if val > 12:
break
print(val) # Prints 10, 11, 12
# Running cumulative sum
running_sums = list(itertools.accumulate([1, 2, 3, 4]))
print(running_sums) # [1, 3, 6, 10]
# Combinatorics: combinations vs permutations
items = ["a", "b", "c"]
perms = list(itertools.permutations(items, 2)) # len 6: ('a','b'), ('b','a'), ...
combs = list(itertools.combinations(items, 2)) # len 3: ('a','b'), ('a','c'), ('b','c')
Key Takeaway: itertools delivers memory-efficient, functional streaming primitives that eliminate manual loop accumulators and combinatoric recursion.
Topics Covered in Generators, Iterators & itertools
- Iterator Protocol Basics (0:00 - 1:25) — Introduces the iterator protocol along with the iter and next dunder methods.
- StopIteration and For Loops (1:25 - 2:30) — Demonstrates manual next traversal, StopIteration handling, and automatic for-loop execution.
- Iterables vs. Iterators (2:30 - 4:00) — Contrasts repeatable iterable sequence creation against self-returning stateful iterators.
- Sentinel-Based Iteration (4:00 - 5:15) — Applies the two-argument iter function with sentinel values for streamlined file reading.
- Type Abstraction and Hashability (5:15 - 6:45) — Uses Iterable type hints for polymorphism and explains why set iterables require frozen dataclasses.
- Custom Iterator Implementations (6:45 - 8:10) — Builds custom infinite and finite iterator classes using Self annotations and StopIteration guards.
- Streaming with itertools (8:10 - 10:00) — Constructs functional pipelines using itertools count, repeat, accumulate, permutations, and combinations.
Python Cheat Sheet
-
iter(iterable)— Obtains an iterator from an iterable containerstream = iter([1, 2, 3]) -
next(iterator)— Retrieves the next element or raises StopIterationitem = next(stream) -
iter(callable, sentinel)— Iterates a zero-argument callable until reaching sentinelstream = iter(f.readline, "") -
itertools.count(start=0, step=1)— Generates an infinite arithmetic progression of numberscounter = itertools.count(10) -
itertools.repeat(object, times)— Yields an object repeatedly for n timesones = itertools.repeat(1, 4) -
itertools.accumulate(iterable)— Yields running cumulative reductions of an iterabletotals = itertools.accumulate(range(1, 5)) -
itertools.permutations(iterable, r)— Yields ordered r-length tuples without replacementpairs = itertools.permutations(["a", "b"], 2) -
itertools.combinations(iterable, r)— Yields unordered r-length tuples without replacementpairs = itertools.combinations(["a", "b"], 2)
Comparison Table
| Property | Iterable (e.g. List, Tuple) | Iterator (e.g. iter(List)) |
|---|---|---|
| Protocol Methods | Implements __iter__() | Implements __iter__() and __next__() |
Calling iter(obj) | Returns fresh, independent iterator | Returns self with shared state |
| State & Resetting | Can be traversed multiple times | Single-pass; consumed permanently |
| Memory Footprint | Stores complete collection in memory | Evaluates and streams elements lazily |
Common Pitfalls
- Mistake: Calling iter() on an active iterator expecting a fresh restart. Avoid: Call iter() on the original sequence container to obtain a new iterator instance.
- Mistake: Forgetting to raise StopIteration inside custom iterator next methods. Avoid: Check boundary conditions explicitly and raise StopIteration when iteration completes.
- Mistake: Passing mutable custom classes into set-based iterable pipelines. Avoid: Mark dataclasses as frozen=True or implement a valid hash method.
- Mistake: Consuming itertools.count() in a loop without break conditions. Avoid: Include explicit termination checks or pair with slicing utilities to prevent infinite loops.
FAQs
- Why must custom iterators implement iter in addition to next?
Implementing
__iter__to returnselfallows the iterator to be passed directly toforloops and functions expecting generic iterables. - What happens when you pass an already exhausted iterator into a for loop?
The loop calls
next()immediately, receivesStopIteration, and terminates without executing the loop body or throwing an error. - When is permutations preferred over combinations in itertools?
Use
permutationswhen element ordering matters (('a', 'b')!=('b', 'a')), and usecombinationswhen groups with identical members are considered equal.