AI Solutions · Service
Evaluating an AI system before you deploy it
The golden dataset, the harness, and what the score has to mean.
At a glance
What it is
“A demonstration is a curated sample shown by someone who wants a yes.”
A principal, Kaelo AI Solutions
What’s included
Four areas of scopeA golden dataset that reflects reality
Assembled from real inputs including the difficult and ambiguous ones, labelled by people who know the domain, and frozen. A dataset curated to make the system look good is worse than no dataset, because it manufactures confidence.
Scoring chosen before results are seen
Exact match, semantic similarity, human rating or LLM-as-judge — each is appropriate somewhere and misleading elsewhere. Choosing after seeing the output is how evaluations become justifications.
Hallucination measured, not asserted
For retrieval systems, the question is whether an answer is grounded in the retrieved material. That is measurable, and it should be a reported number rather than a reassurance in a proposal.
Regression testing as the system changes
Models are updated, prompts are edited, retrieval is tuned. Without a harness that re-runs, you find out about a regression from a user.
How the work runs
Scope the task
What the system must do, and what would count as failing.
Build the dataset
Real inputs, domain labelling, difficult cases included.
Agree scoring
Method and threshold, written down before results exist.
Run, report, re-run
Including on every subsequent change.
When to come to us
- 01 You are being asked to approve an AI deployment on the basis of a demonstration.
- 02 A system is already live and nobody can say whether it is getting better or worse.
- 03 The output feeds a decision with a real cost attached to being wrong.
What we do not do
- Evaluating on a dataset the builder assembled.
- Reporting a single headline accuracy number without the difficult-case breakdown.
- Publishing benchmark figures from someone else's system as though they were yours.
Common questions
