Book review: AI Engineering

I bought Chip Huyen’s AI Engineering in January of last year, read the first chapter, and put it down, because a month later I started using an agent in my terminal and spent the rest of the year doing instead of reading. I picked it up again in January, finished it last week, and I wish I’d read the middle of it a year ago.

It’s an O’Reilly book, about 530 pages, released at the end of 2024. Huyen wrote Designing Machine Learning Systems a few years before it, which was about the older kind of ML work: collect data, train a model, deploy it, watch it drift. This one is about what she calls AI engineering, which is building products on top of models someone else trained. Her argument in the first chapter is that this is a different job from ML engineering. You mostly don’t train anything. You adapt a general model to your problem with prompts, retrieval and sometimes finetuning, and the hard part has moved from “can we make a model that does this” to “how do we know if it’s doing it well”.

That framing held up better than I expected, and it’s the spine of the book.

The evaluation chapters

Chapters 3 and 4 are about evaluation, and they’re the reason to buy it.

Chapter 3 is the methods. It starts with why evaluating a foundation model is harder than evaluating a classifier: the outputs are open-ended, there’s often no single right answer, and the model’s failures are fluent, so they don’t look like failures. It goes through the options, from exact match to similarity scores to using another model as a judge, and is careful about each one’s weaknesses. The section on AI as a judge is the most useful thing I’ve read about it. A judge model has its own biases, it can prefer longer answers or its own outputs, and its scores move when you change the judge’s prompt or swap models, so a score of 4.2 on Tuesday and 4.5 on Friday may tell you more about your judge than your system. I’d half-known all of that. I hadn’t seen it laid out with the evidence.

Chapter 4 is how to actually set up evaluation for a system you’re building: decide what good means for your application, write the criteria down, build an evaluation set, and run it every time something changes. She talks about evaluation-driven development, by analogy with test-driven development, and it’s the right analogy.

Reading it, I got a bit uncomfortable. When we built our docs bot in 2023 we built a small set of real questions with known good answers, and we ran it on every change. That’s more or less what she recommends. For the agent work since then, the far bigger part of what we do with models, we have almost nothing like it. We have test suites, which tell us whether the code the agent wrote works. We don’t have anything that tells us whether the agent got better or worse at writing it when the model changed, or when someone edited the instructions file. We find out by feel. I’ve been saying for a year that the review is the job now, and this book made me realize we’ve only been reviewing the outputs.

The rest

Chapter 5, on prompt engineering, is solid and includes a good section on defending against prompt attacks, which I’d give to anyone shipping a feature where a model reads text from users. I’ve been banging on about this since the first agents and it’s nice to be able to hand someone a chapter instead.

Chapter 6 covers retrieval-augmented generation and agents together, and this is where the book’s age shows. The RAG half is good and practical, and matches what we learned the hard way on the docs bot, including that plain keyword search is worth running alongside the vectors, which is how we fixed our misses on error codes. The agents half is careful and conceptual. It defines an agent by its environment and the actions it can take, talks about planning and tool use and how errors compound over many steps, and it’s all correct. But it was written when agents were mostly a research topic and a few demos, and a year later I spend my days with one. Nothing in it is wrong. It just stops where my actual problems start: how much autonomy to give one, how to review what it did, what to do when it’s been running for an hour and made a decision I’d have wanted to be asked about.

Chapters 7 and 8, on finetuning and dataset engineering, I read more quickly. We don’t finetune anything and I don’t expect to, and Huyen is honest that a lot of teams reach for finetuning when better prompts or better retrieval would have done. Her rough rule, as I remember it, is that finetuning is mostly for changing the form of a model’s output, and retrieval is for giving it facts. I’ve repeated that to two people already.

Chapter 9, on inference optimization, was a pleasant surprise: quantization, batching, caching, speculative decoding, all explained well enough that I finally understand why the long prompts we send are as expensive as they are. The last chapter is about the architecture of an AI application, from the simplest version up through guardrails, routing and caches, and about collecting user feedback, and it’s a good checklist.

Who it’s for

It’s for someone building a product feature on top of a model, and for their lead. If that’s you, read chapters 1, 3, 4, 6 and 10 first, and the rest when you need them. It’s not a book about using coding agents, and it doesn’t claim to be. It’s long, and parts of it read like a careful survey of the field, which means a lot of “it depends”, but she’s usually clear about what it depends on.

The thing I’m taking from it is chapter 4, applied to my own agents. Huyen’s point is that the evaluation set is the specification, the thing that says what good means for your system, and if you don’t have one, you don’t really know what you’re building. We have specs for the work the agents do. We don’t have one for the agents. I’ve started a list of twenty tasks with known good outcomes, from our own backlog, that I can rerun when something changes. It’s twenty because that’s what I could write in an evening, and I have no idea yet whether it’s the right number.