AI Cost Engineering roadmap

Making an AI feature affordable at the volume it will actually reach. Variable cost per request is the structural difference between AI features and everything else you've shipped, and it turns success into a margin problem.

Deep dive6 stages76 topics

All roadmaps
0 of 76 done
Essential, don't skip Recommended, skip only with a reason Optional, awareness is enough Course on PracticAI

Six sections. Section 2 comes before every optimisation for a reason - teams routinely spend weeks compressing prompts while a single unbounded retry loop or an oversized tool schema accounts for most of the bill. Measure, then cut.

1

Understanding the Bill

          2

          Measuring

          Before any optimisation. The distribution of spend is almost never what the team assumes.

                  3

                  Reducing Input Cost

                          4

                          Reducing Output Cost

                              5

                              Model Strategy

                              Usually the largest deliberate saving available, and the one most teams delay because a single model is simpler to reason about.

                                      6

                                      Governance

                                              Cutting a bill in half

                                              In this order. The early steps are usually free and frequently sufficient.

                                              1. Attribute the spend. By feature, by user, by step. You cannot cut what you cannot see, and the biggest line is rarely the one you'd guess.
                                              2. Find the waste. Retries that shouldn't happen, tool schemas nobody calls, retrieved chunks nobody uses, agents looping. Pure loss, no quality tradeoff.
                                              3. Fix prompt caching. Reorder stable content to the front and stop breaking the prefix with timestamps. Often a large win for a day's work.
                                              4. Trim context. Fewer retrieved chunks, shorter tool descriptions, compacted history. Measure quality as you cut, not after.
                                              5. Route by difficulty. A small model handles most traffic; escalate only the hard cases. This is usually the largest deliberate saving available.
                                              6. Shorten outputs. Output tokens typically cost several times input tokens. Ask for less prose.
                                              7. Only then consider self-hosting. It wins on sustained high volume and loses on everything else once you count idle GPUs and engineering time.

                                              Track cost per successful task, not cost per call. A cheaper model that fails twice as often is not cheaper.

                                              The uncomfortable property of AI features is that cost scales with adoption while price usually doesn't. A feature that is comfortably profitable in beta can invert at scale, and the time to model that is before launch rather than after the invoice.