Summary

Concurrency overlaps waiting and parallelism overlaps computation. Async gives you the first on one thread, so it fixes I/O bound work and does nothing for CPU bound work. await yields to the event loop, and a coroutine that never yields stalls every other task, which is the single most common reason async code disappoints. Move genuinely blocking work to a thread with asyncio.to_thread, and CPU bound work to a separate process.

Launch work concurrently with TaskGroup for structured cleanup, gather when you want every result including failures, or as_completed when early results are useful. Cleanup under cancellation works through finally and async context managers, and CancelledError must always be re-raised.

Bound concurrency with a semaphore per dependency, sized against provider limits and your own connection pool. Bound rate with token buckets for both requests and tokens per minute, and adapt downward on 429 rather than trusting a configured number. Batch to reduce overhead, measure the size rather than guessing, and remember that a batch failure loses every item in it. Fan out independent work and fan in, giving each branch its own timeout.

Streaming changes perceived speed without changing total time, and time-to-first-token and total latency are separate metrics with separate causes. Consume streams as async generators, normalise provider events into your own vocabulary with an adapter, accumulate into a list and join once, and bound both the gap between chunks and the total duration.

JSON has no meaningful prefix, so partial parsing is for display only and the final validation is strict. Streamed tool call arguments must be accumulated per call id and released only on the stop event, because dispatching a partial call executes something the model never asked for.

Re-streaming through your own API means forwarding each event rather than collecting them, keeping backpressure intact by never using an unbounded queue, bounding concurrent streams and releasing slots in a finally, and letting client disconnection cancel the upstream call so you stop paying. Errors arriving after the status line need an error event inside the stream, an explicit terminal event to distinguish completion from failure, and a client that acts on them. Heartbeats keep intermediaries from closing an idle connection.

Usage arrives last and sometimes never, so record it in a finally and mark estimates as estimates. Log a stream as one span with time to first token, total duration, and an outcome that distinguishes a clean finish from a disconnect.

Key takeaways

• Concurrency fixes waiting, parallelism fixes computing, and knowing which you have comes first
• Awaiting in a loop is still sequential
• One blocking call stalls every task, because there is one thread
• Always re-raise CancelledError
• Release resources in finally, which runs on cancellation
• A semaphore bounds simultaneous requests, a token bucket bounds requests over time, and you usually need both
• Size concurrency against your connection pool, not just the provider
• Adapt the rate downward on 429 instead of trusting a configured number
• Time to first token and total latency are different numbers with different causes
• Accumulate fragments in a list and join once
• A stream that stops sending is not an error unless you bound the gap between chunks
• Partial JSON is for display, never for decisions
• Never dispatch a tool call before its stop event arrives
• Anything that materialises a stream before yielding removes the point of streaming
• An unbounded queue in the middle destroys backpressure
• Client disconnect should cancel upstream, and you have to verify it does
• After the status line is sent, errors must travel inside the stream
• Record usage in a finally, and mark estimates as estimates

Common mistakes to remember

• Using async for CPU bound work
• Awaiting inside a loop and expecting concurrency
• Calling time.sleep or a synchronous HTTP library inside a coroutine
• Catching CancelledError without re-raising
• Catching it with except Exception, which does not catch it at all
• Creating tasks with create_task and never awaiting them
• Unbounded gather over ten thousand items
• One global semaphore shared across unrelated dependencies
• A semaphore larger than the database connection pool
• Holding a lock while sleeping inside a rate limiter
• Ignoring the tokens-per-minute limit and only counting requests
• Splitting SSE on single newlines rather than blank lines
• Concatenating stream fragments with repeated string addition
• Setting only a total timeout and never detecting a stall
• Acting on a partially parsed JSON object
• Dispatching a tool call from an incomplete argument buffer
• Collecting the upstream stream into a list before forwarding it
• Releasing a concurrency slot only on the success path
• Forgetting X-Accel-Buffering and getting buffered streams in production
• Ending a failed stream silently, so the client shows a truncated answer as complete
• Losing usage data on every client disconnect