---
title: "What AI Engineering looks like to a model risk team"
description: "The ideas in AI Engineering that hold up a year later each end up owned by a governance function in a regulated enterprise."
url: "https://michael.zazon.us/notes/three-ideas-from-ai-engineering-that-model-risk-teams-would-recognize"
published: "2026-02-26T12:00:00-05:00"
kind: "book-notes"
subject: "ai-agents"
tags: ["ai-coding","team-design"]
---
With foundation models, the demo is now the easy part. Any team can connect a hosted model to a prompt and produce a convincing result in a week or so. But most large organizations still struggle to turn that demo into a production system. Chip Huyen's [AI Engineering](https://www.oreilly.com/library/view/ai-engineering/9781098166298/) (O'Reilly, January 2025) focuses squarely on that gap. The book defines the job as adapting models trained by others, and its companion repo of chapter summaries has earned more than 18,000 GitHub stars.

One year after publication, the underlying question remains: is this a new discipline, or is it software engineering plus a dependency that just gives you different answers every time you ask? Against the backdrop of a large regulated enterprise, several hold up. More importantly, each lands on an existing governance function.

## Evals are where the work is

Evaluation gets 2 whole chapters, so it's clear where the book sees the work. Open-ended outputs don't have a single correct answer, so the book suggests exact checks where the outputs can be verified, and AI judges where they can't.. both in a per-application evaluation pipeline. In a companion post on [common pitfalls when building generative AI applications](https://huyenchip.com/2025/01/16/ai-engineering-pitfalls.html), Huyen cautions against abandoning human review, suggesting that experts review a daily sample of 30 to 1,000 examples.

In a bank, this pipeline looks a lot like what a model validator would ask for. SR 11-7 is the Federal Reserve and OCC's 2011 guidance requiring effective challenge of every model, including outcomes analysis of real performance. A [paper on model risk management for generative AI in financial institutions](https://arxiv.org/abs/2503.15668) notes that these models bring additional risks like hallucination and toxicity, and validation testing needs to expand to cover that. Banks are hiring: Evident's 2025 index shows 42 of the 50 large banks it tracks have a senior responsible-AI lead, up from 33 the year before.

## Start simple, finish slow

The second idea is the least glamorous piece of advice in the book: try prompting first, then retrieval, then finetuning.. don't start with frameworks and vector databases. The number that stuck with me from that pitfalls post: LinkedIn got to 80% of the experience they wanted in 1 month, but it took 4 more months to get to more than 95%.

This curve is the enterprise production gap, just in small form. McKinsey's state of AI survey says 88% of companies use AI in at least one function, but only a third have started to scale it, and most attribute less than 5% of EBIT to AI. The a16z enterprise CIO survey shows finetuning declining as prompting and longer context windows improve, which supports the book's recommended order.

## What you build around the model

Inference optimization gets its own chapter, and the final chapter shows how to build an application around the model layer by layer. Context first, guardrails on input and output, a router and gateway before the models, caching and monitoring after, and user feedback to close the loop. When you apply this to a regulated enterprise, most of the layers already have an owner.

[Diagram: Layers around the model and the enterprise teams that own them](https://michael.zazon.us/notes/three-ideas-from-ai-engineering-that-model-risk-teams-would-recognize)

The gateway is where a regulated enterprise will want to put controls and FinOps. That same a16z survey found that 37% of enterprises are running 5 or more models in production, and cost is weighing more heavily in model choice. Menlo Ventures estimates enterprise spending on generative AI at $37 billion in 2025, about triple the prior year. [Gartner](https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027) says over 40% of agentic AI projects will be canceled by the end of 2027 due to cost, unclear business value, and weak risk controls. If you're running 5+ models, every model swap is a change that needs re-validation.

The book reads like software and systems engineering applied to a dependency that isn't deterministic. It treats evals and the gateway as engineering hygiene, which is exactly what regulated firms will turn into controls. SR 11-7 assumed models built and specified by the bank, but pipeline models are third-party and may answer the same prompt differently each time. This means the eval pipeline may become the validation artifact.