Python Async #1: Coroutines Building Blocks

What is a coroutine?

A coroutine is a function-like thing that can pause, give control back to its caller, keep its local state alive, and later resume from the exact same point. A normal function cannot do that: it runs from top to bottom in one go, and when it hits return, it's done.

Modern Python spells coroutines with async def, but the machinery is easier to see through the older generator protocol. This post builds the idea from five small blocks:

1. yield pauses execution and hands control back.
2. The generator frame keeps local variables alive while paused.
3. next() and .send() let outside code resume the coroutine, optionally with a value.
4. .throw() and .close() let outside code inject errors or cancellation.
5. yield from composes one suspendable operation from another, which is the direct ancestor of await.

Put those together and you have the core handshake behind asyncio: a scheduler keeps many paused coroutines around and resumes each one when it can make progress.

This post is deliberately about plain generators — no async, no await, no asyncio. We earn those in later posts. Everything here works in any Python 3.

A normal function vs a generator

The only syntactic difference is the keyword yield. The behavioural difference is enormous.

def countdown(n):
    print("starting countdown")
    while n > 0:
        yield n          # pause here, hand `n` to the caller
        n -= 1           # resume here on the next request
    print("done")
 
gen = countdown(3)       # NOTHING prints yet — the body hasn't run
print(next(gen))         # "starting countdown", then 3
print(next(gen))         # 2
print(next(gen))         # 1
print(next(gen))         # "done", then raises StopIteration

1def countdown(n):
2    print("starting countdown")
3    while n > 0:
4        yield n          # pause here, hand `n` to the caller
5        n -= 1           # resume here on the next request
6    print("done")
7 
8gen = countdown(3)
9print(next(gen))
10print(next(gen))
11print(next(gen))
12print(next(gen))

Three things to notice, because they trip up almost everyone the first time:

1. Calling countdown(3) runs none of the code. It returns a generator object — a paused function waiting at the very top.
2. next(gen) runs the body until the next yield, then freezes again. The local variable n survives between calls because the function's frame is kept alive.
3. When the function falls off the end, it raises StopIteration. This is the signal "I have nothing more to give." for loops catch this automatically — that's all a for loop really is.

Because generators only compute the next value when asked, they're lazy: you can model an infinite sequence in constant memory.

yield is a two-way street: send()

Here's the part most tutorials skip, and it's the part that turns a generator into a coroutine. Since PEP 342, yield is an expression, not just a statement. That means it can also receive a value when the generator is resumed.

def averager():
    total = 0.0
    count = 0
    average = None
    while True:
        value = yield average     # hand out the current average,
                                  # AND wait to receive the next input
        total += value
        count += 1
        average = total / count
 
avg = averager()
next(avg)                 # "prime" it: run up to the first yield (average is None)
print(avg.send(10))       # 10.0
print(avg.send(20))       # 15.0
print(avg.send(30))       # 20.0

1def averager():
2    total = 0.0
3    count = 0
4    average = None
5    while True:
6        value = yield average
7        total += value
8        count += 1
9        average = total / count
10 
11avg = averager()
12next(avg)
13print(avg.send(10))
14print(avg.send(20))
15print(avg.send(30))

When you call avg.send(10), the generator wakes up, the expression yield average evaluates to 10, execution continues, and it runs until it hits yield again — handing back the new average.

⚠️ Why the next(avg) line? This is called priming. Before the first send, the generator is paused at the very top — it hasn't reached any yield yet, so there's no suspended yield expression to receive your value. You must advance it to the first yield first, with next(avg) (equivalently avg.send(None)). Forgetting to prime is the single most common generator-coroutine bug. Sending a non-None value into an unprimed generator raises TypeError.

Now re-read that flow. The caller and the generator are taking turns, passing values back and forth, each pausing while the other works. That is cooperative multitasking between two pieces of code on one thread — the exact shape of async.

Interrupting from the outside: throw() and close()

If a generator can receive values at the pause point, it can also receive exceptions.

avg.throw(ValueError, "bad data")   # raises ValueError AT the paused `yield` line
avg.close()                         # raises GeneratorExit inside; generator shuts down

throw() lets the driver inject an error into a paused task (this is how asyncio reports that an I/O operation failed). close() raises GeneratorExit at the pause point so the generator can run cleanup (e.g. release a resource in a finally: block) and stop. This is exactly the machinery asyncio uses to cancel a task — a topic we go deep on in Async #4.

Composing generators: yield from

Once you have generators driving generators, you need a clean way to delegate. PEP 380 added yield from.

def chain(*iterables):
    for it in iterables:
        yield from it           # yield every item of `it`, one by one
 
print(list(chain([1, 2], (3, 4), "ab")))   # [1, 2, 3, 4, 'a', 'b']

yield from is more than a loop shortcut. It builds a transparent two-way tunnel to the subgenerator: values sent in with .send(), exceptions thrown with .throw(), and close() all pass straight through to the inner generator. It also captures the subgenerator's return value:

1def producer():
2    yield 1
3    yield 2
4    return "done"  # a generator can return a value
5 
6def wrapper():
7    result = yield from producer()  # `result` captures producer's return value
8    print("subgenerator returned:", result)
9    yield result
10 
11w = wrapper()
12print(next(w))  # 1
13print(next(w))  # 2
14print(next(w))  # prints "subgenerator returned: done", then yields "done"

💡 Hold this thought: yield from subgen() is the direct ancestor of await some_coroutine(). They mean almost the same thing — "run this other suspendable thing to completion, passing control back to the scheduler in between, and give me its result." PEP 492 later gave this its own keyword so coroutines wouldn't be visually confused with data-producing generators. We'll see that bridge at the end.

The punchline: a generator scheduler

Here's why all of this matters. If generators can pause and a driver can resume them, then a tiny scheduler can juggle many tasks on a single thread — running each one a little, then moving to the next. This is a cooperative scheduler, and it is the direct ancestor of the asyncio event loop.

from collections import deque
 
def task(name, steps):
    for i in range(1, steps + 1):
        print(f"{name}: step {i}")
        yield                       # "pause me, let someone else have a turn"
 
class Scheduler:
    def __init__(self):
        self.ready = deque()
 
    def add(self, gen):
        self.ready.append(gen)
 
    def run(self):
        while self.ready:
            gen = self.ready.popleft()
            try:
                next(gen)               # resume this task until its next yield
            except StopIteration:
                pass                    # task finished — drop it
            else:
                self.ready.append(gen)  # not done — send it to the back of the line
 
sched = Scheduler()
sched.add(task("A", 3))
sched.add(task("B", 2))
sched.run()

Running this interleaves the two tasks:

A: step 1
B: step 1
A: step 2
B: step 2
A: step 3

Neither task runs to completion before the other starts. They cooperate, each voluntarily giving up control at yield. That word — voluntarily — is the whole difference between this and OS threads (which can be interrupted at any time). We'll contrast the two models directly in Track B.

Figure 1 — The drive handshake. The scheduler resumes a paused generator with next()/send(); the generator runs to its next yield and hands control back. The asyncio event loop is this exact loop, except "when to resume" is decided by I/O readiness instead of a simple round-robin — which is precisely the subject of Async #2 and #3.

The bridge to async / await

Everything above is pre-2015 Python. In PEP 492, Python 3.5 added dedicated syntax:

The crucial takeaway: a native coroutine is still driven by .send(), and the event loop still resumes it the same way a scheduler resumes a generator. async/await is cleaner, type-distinct syntax over the same suspend/resume machinery you just learned. Nothing fundamentally new is happening — which is exactly why starting here makes the rest of the series easy.

🛠 Build this

Extend the scheduler to support a fake sleep. A task should be able to say "wake me in N seconds" and the scheduler should run other tasks in the meantime.

Starter pieces:

import time
 
def sleep(seconds):
    yield ("sleep", time.monotonic() + seconds)   # tell the scheduler my wake time
 
def chatty(name, delay, times):
    for i in range(times):
        print(f"[{time.monotonic():.2f}] {name} #{i}")
        yield from sleep(delay)

Your job: rewrite Scheduler so it keeps sleeping tasks in a min-heap keyed by wake time (heapq), and only resumes a task once its wake time has passed. When you finish, you'll have built a single-threaded, timer-driven cooperative scheduler — a stripped-down asyncio event loop, minus the I/O. In Async #3 we replace the timer with real socket readiness and you'll see it's the same shape.

✅ You've mastered this when…

• You can explain, without notes, why countdown(3) prints nothing until the first next().
• You can write a send()-based coroutine and explain why it must be primed.
• You can state in one sentence what yield from does that a plain for ... yield loop does not (the two-way tunnel + return-value capture).
• You can look at the Scheduler above and articulate what "cooperative" means and where the cooperation point is.
• You can map each generator concept to its async/await equivalent in the table above.

⚠️ Pitfalls & tradeoffs

• Forgetting to prime a send()-based coroutine → TypeError: can't send non-None value to a just-started generator.
• Generators are single-use. Once exhausted (StopIteration), a generator is dead; calling the function again gives a fresh object. They are not re-iterable like a list.
• A return inside a generator does not return normally — it sets the value on StopIteration. Code that doesn't use yield from or doesn't inspect StopIteration.value will silently lose it.
• Swallowed StopIteration. Historically, a StopIteration raised inside a generator could leak out and silently end iteration. PEP 479 (default in Python 3.7+) fixed this by converting it to a RuntimeError. Good to know when reading older code.
• Cooperative ≠ free lunch. A task that never yields (e.g. a tight CPU loop) starves every other task, because nothing can preempt it. This exact failure mode reappears as "blocking the event loop" in Async #3 — and is the reason Track B (threads/processes) exists for CPU-bound work.

📚 References & sources

Primary sources (PEPs are the source of truth):

• PEP 255 — Simple Generators. Introduced the yield statement. https://peps.python.org/pep-0255/
• PEP 342 — Coroutines via Enhanced Generators. Made yield an expression and added send(), throw(), close(). https://peps.python.org/pep-0342/
• PEP 380 — Syntax for Delegating to a Subgenerator. Added yield from. https://peps.python.org/pep-0380/
• PEP 479 — Change StopIteration handling inside generators. https://peps.python.org/pep-0479/
• PEP 492 — Coroutines with async and await syntax. The bridge to the rest of Track A. https://peps.python.org/pep-0492/

Supporting documentation:

• Python Language Reference — Yield expressions and Generator-iterator methods: https://docs.python.org/3/reference/expressions.html#yield-expressions
• Python Functional HOWTO — Generators: https://docs.python.org/3/howto/functional.html

This post is written/assisted by AI and reviewed by human. Read more about it here.