Python yield and Generators Explained (with Examples)

yield turns a normal function into a generator: instead of computing every value and returning a list, the function hands back one value, pauses where it is, and resumes from that exact line the next time you ask for a value. That is the whole idea, and this short example shows it:

def count_up_to(n):
    i = 1
    while i <= n:
        yield i
        i += 1

gen = count_up_to(3)
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # 3

Calling count_up_to(3) does not run the loop. It creates a generator object. Each next() runs the function until the next yield, returns that value, and freezes the function with all its local variables (i here) intact.

The rest of this post covers why that pause matters for memory, what happens when a generator runs out, generator expressions, yield from, a real file-processing pipeline, and the two itertools helpers you will use most.

What "pausing" actually means

Add a few print calls and you can watch the function stop and restart:

def steps():
    print("start")
    yield "A"
    print("after A")
    yield "B"
    print("end")

g = steps()
print(type(g).__name__)
print(next(g))
print(next(g))
for leftover in g:
    print(leftover)

Output:

generator
start
A
after A
B
end

Notice that "start" is not printed when you call steps(). Nothing in the body runs until the first next(). After "B", the for loop asks for one more value, the function prints "end", reaches the bottom without another yield, and the loop finishes quietly.

A normal function has one exit: return, after which its local variables are gone. A generator function has many pause points, and its local state survives between them.

Generators vs lists: the memory difference

A list stores every element at once. A generator stores only the recipe and its current position. For large sequences that difference is dramatic:

import sys

squares_list = [n * n for n in range(1_000_000)]
squares_gen = (n * n for n in range(1_000_000))

print(sys.getsizeof(squares_list) > 8_000_000)  # True  (about 8 MB of pointers)
print(sys.getsizeof(squares_gen) < 500)         # True  (a few hundred bytes)
print(sum(squares_gen))                         # 333332833333500000

The exact byte counts depend on your Python version and platform, which is why the example compares against thresholds. The list is around 8 MB for the pointers alone (the integer objects cost more on top). The generator is a few hundred bytes no matter how long the sequence is, because each square is computed only when sum() asks for it and then discarded.

The trade-off: a list can be indexed, sliced, measured with len() and looped over many times. A generator can do none of those. Use a list when you need the data again; use a generator when you only need to walk through it once.

next() and StopIteration

When a generator has no more values, next() raises StopIteration. A for loop catches this for you, which is why you rarely see it. Calling next() by hand, you will:

def count_up_to(n):
    i = 1
    while i <= n:
        yield i
        i += 1

g = count_up_to(2)
print(next(g))  # 1
print(next(g))  # 2
try:
    next(g)
except StopIteration:
    print("generator is exhausted")
print(next(g, "done"))  # done

The second argument to next() is a default that is returned instead of raising. It is the clean way to ask "give me the next item, if there is one".

A generator is also single-use. Once exhausted, it stays exhausted:

nums = (x for x in range(3))
print(list(nums))  # [0, 1, 2]
print(list(nums))  # []

This is the most common generator bug in real code: a function returns a generator, one part of the program consumes it, and a later loop over the same object silently does nothing. If you need two passes, build a list, or call the generator function again to get a fresh generator.

Generators and iterators

Every generator is an iterator: an object with __iter__ and __next__ methods. You could write the counter above as a class:

class CountUpTo:
    def __init__(self, n):
        self.i = 1
        self.n = n

    def __iter__(self):
        return self

    def __next__(self):
        if self.i > self.n:
            raise StopIteration
        value = self.i
        self.i += 1
        return value

print(list(CountUpTo(3)))  # [1, 2, 3]

That is 12 lines of code to do what the 5-line generator function does. yield writes the __next__ method and the state-keeping for you. If the difference between an iterable and an iterator is still fuzzy, the iterators lesson walks through the protocol step by step, and the generators lesson builds on it with graded exercises.

Generator expressions

A generator expression looks like a list comprehension with round brackets instead of square ones:

squares = [x * x for x in range(5)]   # list, built immediately
lazy = (x * x for x in range(5))      # generator, built on demand

print(squares)           # [0, 1, 4, 9, 16]
print(list(lazy))        # [0, 1, 4, 9, 16]
print(sum(x * x for x in range(10)))               # 285
print(any(word.startswith("py") for word in ["java", "python"]))  # True

When a generator expression is the only argument to a function, you can drop the extra brackets, as in sum(x * x for x in range(10)). Functions like sum, any, all, max, min and "".join consume their input once, so a generator is the natural fit. any() also stops at the first True, so the rest of the generator is never computed.

If list comprehensions are new to you, start with the list comprehensions lesson; generator expressions are the same syntax with lazy evaluation.

yield from: delegating to another generator

yield from iterable yields every value from another iterable, one at a time. It is shorter than a for loop, and it makes recursive generators readable:

def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

print(list(flatten([1, [2, [3, 4]], 5])))  # [1, 2, 3, 4, 5]

Each recursive call is its own generator, and yield from passes its values straight up to whoever is looping over the outer one.

A real example: a pipeline for a large log file

Generators shine when data is too big to load at once. Reading a file line by line with a for loop already works lazily, and you can stack small generators on top of it to build a pipeline where each stage does one job:

from pathlib import Path

# create a small sample file so the example runs anywhere
Path("server.log").write_text(
    "INFO start\nERROR disk full\nINFO retry\nERROR timeout\nINFO done\n"
)

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.rstrip("\n")

def only_errors(lines):
    for line in lines:
        if line.startswith("ERROR"):
            yield line

def messages(lines):
    for line in lines:
        yield line.split(" ", 1)[1]

for msg in messages(only_errors(read_lines("server.log"))):
    print(msg)

Output:

disk full
timeout

Only one line is in memory at any moment. The same code works on a 5-line file or a 5 GB file, because no stage ever collects the whole thing into a list. Each stage is also easy to test on its own, since any list of strings can stand in for the file.

itertools: islice and chain

The itertools module in the standard library is built around lazy iteration. Two functions cover most everyday needs:

from itertools import islice, chain, count

evens = (n for n in count() if n % 2 == 0)   # an infinite generator
print(list(islice(evens, 5)))                # [0, 2, 4, 6, 8]

print(list(chain([1, 2], (3, 4), "ab")))     # [1, 2, 3, 4, 'a', 'b']

islice is slicing for iterators: you cannot write evens[:5] on a generator, but islice(evens, 5) takes the first five lazily, which is the only safe way to take items from an infinite generator. chain joins several iterables into one stream without building a combined list.

send(), briefly

Generators can also receive values. gen.send(value) resumes the generator and makes the paused yield expression evaluate to value:

def running_total():
    total = 0
    while True:
        value = yield total
        total += value

acc = running_total()
next(acc)            # run to the first yield
print(acc.send(10))  # 10
print(acc.send(5))   # 15

You must call next() once first so the generator is paused at a yield and ready to receive. In day-to-day code you will rarely need send(); it matters mainly for understanding coroutine-style code and older libraries.

When to use a generator

• The data is large or unbounded (log files, database cursors, API pages, infinite sequences).
• You only need one pass over the values.
• You want to build a pipeline of small, testable steps.
• You may stop early, so computing everything up front would be wasted work.

Use a list instead when you need len(), indexing, sorting, or more than one pass.

Try it yourself

Reading about yield is not the same as predicting what a generator prints. Paste any example above into the browser Python terminal and change it: move a print, call next() one extra time, loop over an exhausted generator. Python runs in the browser with no install. When you are ready for graded exercises, work through the generators lesson, or pick up more problems on the practice page.