Evaluations are the difference between an impressive demo and a system you can leave running on a Friday night.
Golden sets from real data
Before we write a prompt, we collect 100 to 300 real examples with the correct outcome, labelled by the people who do the work today.
Three layers of checks
Deterministic checks: format, required fields, forbidden actions
Model-graded checks: tone, completeness, policy adherence
Human review: a weekly sample, rotated across the team
Watching for drift
Inputs change: new products, new policies, new customer behaviour. We track agreement between the agent and human reviewers every week and alert when it drops.
Written by
Jonas Weber
Has connected more APIs than he can remember. Owns our integrations and evaluation tooling.
