Evals Are the Product Spec

The most common failure mode in generative AI projects is not a bad model. It is the absence of a written definition of “good” that survives contact with a second engineer.

Evals before prompts

We write the eval set — a scored, versioned collection of inputs and acceptable outputs — before we draft the first prompt. The eval set is the spec; the prompt is an implementation detail that changes weekly.

What this buys you

A regression is a number, not a vibe. When a model provider ships a new version, we know within minutes whether it helped or hurt, because the eval suite runs in CI on every change.