🔵 Intermediate

Generators & Iterators

📖 Lesson 19 ⏱ 40 min 🧪 5 questions 💻 3 exercises

🎯 Learning Objectives

  • Understand the iterator protocol (__iter__ and __next__)
  • Write generator functions using yield
  • Use yield from to delegate to sub-generators
  • Understand lazy evaluation and why it matters for memory
  • Build custom iterables with classes
  • Use itertools for powerful iterator pipelines
  • Know when to use generators vs lists

The Iterator Protocol

Python's for loop works with any object that implements the iterator protocol — two special methods:

  • __iter__() — returns the iterator object itself
  • __next__() — returns the next value, or raises StopIteration when exhausted
# Under the hood of a for loop
numbers = [1, 2, 3]

# Python does this:
it = iter(numbers)      # calls numbers.__iter__()
print(next(it))         # 1  — calls it.__next__()
print(next(it))         # 2
print(next(it))         # 3
# next(it)              # raises StopIteration

# The for loop handles StopIteration automatically
for n in numbers:
    print(n)
iterator_protocol.py
An iterable is anything you can loop over (list, string, dict, file…). An iterator is the stateful object that tracks position and yields values one at a time. Every iterator is iterable, but not every iterable is an iterator. 
numbers = [1, 2, 3]

# list is iterable but NOT an iterator
print(hasattr(numbers, "__iter__"))   # True
print(hasattr(numbers, "__next__"))   # False

# iter() gives us an iterator from the iterable
it = iter(numbers)
print(hasattr(it, "__next__"))        # True

# Iterators remember their position — calling iter() on them returns self
print(iter(it) is it)   # True
iterable_vs_iterator.py

Generator Functions

A generator function uses yield instead of return. Calling it returns a generator object — a lazy iterator that produces values one at a time, pausing execution at each yield and resuming from the same point on the next call to next(): 

def count_up(start, stop):
    """Generate integers from start to stop (inclusive)."""
    current = start
    while current <= stop:
        yield current        # pause here, return current
        current += 1         # resume here on next next()

gen = count_up(1, 5)
print(type(gen))    # <class 'generator'>

print(next(gen))    # 1
print(next(gen))    # 2
print(next(gen))    # 3

# Or consume the rest with a loop
for n in gen:
    print(n)        # 4, 5
generator_function.py 
def fibonacci():
    """Yield Fibonacci numbers forever — an infinite sequence."""
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

gen = fibonacci()
for _ in range(10):
    print(next(gen), end=" ")
# 0 1 1 2 3 5 8 13 21 34
infinite_generator.py
How yield works: When Python hits yield value, it suspends the function — saving the entire local state (variables, position in the code, call stack). The value is returned to the caller. When next() is called again, execution resumes from the line after the yield.

Lazy Evaluation & Memory Efficiency

The key advantage of generators is that they produce values on demand — only one value exists in memory at a time: 

import sys

# List — all values materialised at once
million_list = list(range(1_000_000))
print(sys.getsizeof(million_list))   # ~8 MB

# Generator — virtually no memory
million_gen = (x for x in range(1_000_000))
print(sys.getsizeof(million_gen))    # ~112 bytes

# Both produce the same sum
print(sum(million_list))  # 499999500000
print(sum(x for x in range(1_000_000)))  # 499999500000
memory_comparison.py 
def read_large_file(path):
    """Yield lines from a file one at a time — handles multi-GB files."""
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            yield line.rstrip("\n")

# Only one line in memory at a time, regardless of file size
for line in read_large_file("huge_log.txt"):
    if "ERROR" in line:
        print(line)
large_file_generator.py

Generator Expressions (Recap)

You saw generator expressions in Lesson 16. They are the inline shorthand for simple generator functions: 

# Generator expression
squares = (x ** 2 for x in range(10))

# Equivalent generator function
def make_squares(n):
    for x in range(n):
        yield x ** 2

# Both lazy; both produce the same values
print(list(squares))              # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
print(list(make_squares(10)))     # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

# Chain generator expressions into a pipeline
data = range(100)
pipeline = sum(x ** 2 for x in data if x % 3 == 0)
print(pipeline)   # 32955
gen_expr_recap.py

yield from

yield from iterable delegates to a sub-iterable, yielding each of its values in turn. It replaces a nested for loop and also correctly propagates StopIteration and return values: 

def flatten(nested):
    """Recursively flatten a nested list of any depth."""
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)   # delegate to recursive call
        else:
            yield item

data = [1, [2, 3], [4, [5, 6]], 7]
print(list(flatten(data)))   # [1, 2, 3, 4, 5, 6, 7]
yield_from_flatten.py 
def chain(*iterables):
    """Yield all items from each iterable in sequence."""
    for it in iterables:
        yield from it

result = list(chain([1, 2], [3, 4], [5, 6]))
print(result)   # [1, 2, 3, 4, 5, 6]

# Standard library already has this:
from itertools import chain as ichain
print(list(ichain([1, 2], [3, 4])))   # [1, 2, 3, 4]
yield_from_chain.py
yield from works with any iterable — lists, tuples, strings, other generators — and is the idiomatic way to delegate inside a generator.

Sending Values into a Generator

Generators are not just one-directional. You can send a value back into a generator using .send(value), which resumes execution and becomes the result of the yield expression: 

def accumulator():
    """Running total — receives new numbers via send()."""
    total = 0
    while True:
        value = yield total    # yield current total; receive next value
        if value is None:
            break
        total += value

gen = accumulator()
next(gen)            # prime the generator (advance to first yield)

print(gen.send(10))  # 10
print(gen.send(20))  # 30
print(gen.send(5))   # 35
send.py
You must call next(gen) (or gen.send(None)) once to prime the generator — advance it to the first yield — before you can send a non-None value. Sending before priming raises TypeError.

Custom Iterables with Classes

Implement the iterator protocol in a class to make any object iterable: 

class CountDown:
    """Counts down from start to 1."""
    def __init__(self, start):
        self.start = start

    def __iter__(self):
        current = self.start
        while current > 0:
            yield current         # __iter__ can itself be a generator!
            current -= 1

for n in CountDown(5):
    print(n, end=" ")   # 5 4 3 2 1
custom_iterable_simple.py 
class Range:
    """A simplified version of range() — demonstrates full iterator protocol."""

    def __init__(self, stop):
        self.stop = stop

    def __iter__(self):
        return RangeIterator(self.stop)

class RangeIterator:
    def __init__(self, stop):
        self.current = 0
        self.stop    = stop

    def __iter__(self):
        return self   # iterators return themselves

    def __next__(self):
        if self.current >= self.stop:
            raise StopIteration
        value = self.current
        self.current += 1
        return value

r = Range(5)
print(list(r))           # [0, 1, 2, 3, 4]
print(list(r))           # [0, 1, 2, 3, 4]  — iterable, reusable!

it = iter(r)
print(list(it))          # [0, 1, 2, 3, 4]
print(list(it))          # []  — iterator is exhausted after one pass
custom_iterator_class.py
Iterable vs iterator in classes: An iterable's __iter__ returns a new iterator each time — so you can loop over it multiple times. An iterator's __iter__ returns self — once exhausted, it's empty.

The itertools Module

itertools is a standard-library treasure chest of efficient, composable iterator building blocks: 

from itertools import (
    count, cycle, repeat,          # infinite iterators
    islice, takewhile, dropwhile,  # slicing / filtering
    chain, chain_from_iterable,    # combining
    zip_longest, starmap,          # pairing
    product, permutations,         # combinatorics
    combinations, groupby,         # combinatorics / grouping
    accumulate,                    # running totals
)

# ── Infinite iterators ──
from itertools import count, islice, cycle

# count(start, step) — integers forever
print(list(islice(count(1, 2), 5)))   # [1, 3, 5, 7, 9]

# cycle — repeat a sequence forever
seasons = cycle(["Spring", "Summer", "Autumn", "Winter"])
print([next(seasons) for _ in range(6)])
# ['Spring', 'Summer', 'Autumn', 'Winter', 'Spring', 'Summer']
itertools_infinite.py 
from itertools import groupby, accumulate, product, combinations

# groupby — group consecutive elements by a key
# (sort first if you want all groups)
data = [("fruit", "apple"), ("veg", "carrot"), ("fruit", "banana"),
        ("veg", "pea"), ("fruit", "cherry")]
data.sort(key=lambda x: x[0])

for category, items in groupby(data, key=lambda x: x[0]):
    print(category, list(item[1] for item in items))
# fruit ['apple', 'banana', 'cherry']
# veg   ['carrot', 'pea']

# accumulate — running total (or custom operation)
import operator
print(list(accumulate([1, 2, 3, 4, 5])))           # [1, 3, 6, 10, 15]
print(list(accumulate([1, 2, 3, 4, 5], operator.mul)))  # [1, 2, 6, 24, 120]

# product — Cartesian product
print(list(product("AB", repeat=2)))
# [('A','A'),('A','B'),('B','A'),('B','B')]

# combinations
print(list(combinations("ABCD", 2)))
# [('A','B'),('A','C'),('A','D'),('B','C'),('B','D'),('C','D')]
itertools_tools.py

Generator Pipelines

Chain generators together to build memory-efficient data processing pipelines. Data flows through each stage lazily — only one item is processed at a time regardless of the total dataset size: 

def read_lines(path):
    with open(path, "r", encoding="utf-8") as f:
        yield from f

def strip_lines(lines):
    for line in lines:
        yield line.strip()

def remove_blanks(lines):
    for line in lines:
        if line:
            yield line

def parse_records(lines):
    for line in lines:
        parts = line.split(",")
        yield {"name": parts[0], "score": int(parts[1])}

def filter_passing(records, threshold=50):
    for r in records:
        if r["score"] >= threshold:
            yield r

# Compose the pipeline — nothing runs until we consume it
pipeline = filter_passing(
               parse_records(
                   remove_blanks(
                       strip_lines(
                           read_lines("scores.csv")))))

for record in pipeline:
    print(record)
pipeline.py

Generators vs Lists

SituationUse
Large or infinite sequenceGenerator ✓
Need random access (data[5])List ✓
Iterate once, then discardGenerator ✓
Iterate multiple timesList ✓
Need len()List ✓
Streaming file or network dataGenerator ✓
Pipeline of transformationsGenerator chain ✓
Pass results to a function expecting a sequenceList (wrap with list()) ✓
 
# ── Key behaviours to remember ──

gen = (x for x in range(5))

# Generators are single-use
print(list(gen))   # [0, 1, 2, 3, 4]
print(list(gen))   # []  — already exhausted!

# Generators have no len()
# len(gen)   # TypeError: object of type 'generator' has no len()

# No random access
# gen[2]     # TypeError: 'generator' object is not subscriptable

# To reuse, recreate or wrap in a list
gen = (x for x in range(5))
data = list(gen)   # materialise once
print(data[2])     # 2  — now you have random access
generators_vs_lists.py

Primary sources: Python Docs — Generators  ·  Python Docs — itertools  ·  Python Docs — Functional HowTo: Generators  ·  PEP 255 — Simple Generators

🤖

Ask your AI tutor! Struggling to visualise how yield suspends execution? Want to build a data pipeline for a real file? Confused about when to use yield from? These click best with concrete examples — ask for a walkthrough.

💻 Exercises

01 Infinite Range Generator

Write a generator function infinite_range(start=0, step=1) that yields integers from start upwards in steps of step, forever. Then write a helper take(n, iterable) that returns the first n items from any iterable as a list. 

print(take(5, infinite_range()))          # [0, 1, 2, 3, 4]
print(take(5, infinite_range(10, 2)))     # [10, 12, 14, 16, 18]
print(take(4, infinite_range(100, -1)))   # [100, 99, 98, 97]
Show solution 
def infinite_range(start=0, step=1):
    current = start
    while True:
        yield current
        current += step

def take(n, iterable):
    result = []
    for item in iterable:
        if len(result) >= n:
            break
        result.append(item)
    return result

# Or with itertools:
from itertools import islice
def take(n, iterable):
    return list(islice(iterable, n))

print(take(5, infinite_range()))          # [0, 1, 2, 3, 4]
print(take(5, infinite_range(10, 2)))     # [10, 12, 14, 16, 18]
print(take(4, infinite_range(100, -1)))   # [100, 99, 98, 97]
02 Recursive Flatten

Write a generator function deep_flatten(nested) that recursively flattens a structure of arbitrarily nested lists (and tuples) into a single sequence of non-list, non-tuple values. Use yield from. 

data = [1, [2, (3, 4)], [[5, 6], 7], (8, [9, 10])]
print(list(deep_flatten(data)))
# [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Show solution 
def deep_flatten(nested):
    for item in nested:
        if isinstance(item, (list, tuple)):
            yield from deep_flatten(item)
        else:
            yield item

data = [1, [2, (3, 4)], [[5, 6], 7], (8, [9, 10])]
print(list(deep_flatten(data)))
# [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Edge cases
print(list(deep_flatten([])))             # []
print(list(deep_flatten([1, 2, 3])))      # [1, 2, 3]
print(list(deep_flatten([[[[42]]]])))     # [42]
03 Lazy CSV Pipeline

Build a lazy pipeline of generators that processes a CSV file without loading it all into memory. Given a CSV with columns name, score, passed:

  1. read_csv(path) — yields each row as a dict (use csv.DictReader)
  2. only_passed(records) — yields only records where passed == "True"
  3. extract_scores(records) — yields just the score as a float

Compose the pipeline and compute the average score of passing students. Prove it's lazy by showing each stage is a generator, not a list.

Show solution 
import csv
from pathlib import Path

# --- create test data ---
Path("results.csv").write_text(
    "name,score,passed\nAlice,88,True\nBob,45,False\nCarol,92,True\nDave,55,False\nEve,78,True\n",
    encoding="utf-8"
)

# --- pipeline stages ---
def read_csv(path):
    with open(path, newline="", encoding="utf-8") as f:
        yield from csv.DictReader(f)

def only_passed(records):
    for r in records:
        if r["passed"] == "True":
            yield r

def extract_scores(records):
    for r in records:
        yield float(r["score"])

# --- compose ---
pipeline = extract_scores(only_passed(read_csv("results.csv")))

import types
print(isinstance(pipeline, types.GeneratorType))  # True — lazy!

scores = list(pipeline)
print(scores)                        # [88.0, 92.0, 78.0]
print(f"Average: {sum(scores)/len(scores):.1f}")  # Average: 86.0