AI integration / Evaluation and guardrails / ARTEFACT
The evaluation set
A fixed set of cases your assistant is scored against before and after every change.
The evaluation set drawn as a schema. Parts: Case catalogue, Expected outcomes, Scoring rubric and Evaluation runner. Acceptance: The same versioned case set runs twice against a named system version without changing its source records. Every case receives a result under the approved scoring rubric.
01
Case catalogue
02
Expected outcomes
03
Scoring rubric
04
Evaluation runner
- The same versioned case set runs twice against a named system version without changing its source records.
- Every case receives a result under the approved scoring rubric.
WHAT THIS ARTEFACT IS
A named output with defined parts
An evaluation set is a versioned collection of representative inputs, expected outcomes and scoring instructions used to compare an AI system before and after a model, prompt or data change.
What is inside it
- 01
Case catalogue
Each record contains a representative input, its source, a failure category and the context needed to run it.
- 02
Expected outcomes
Subject matter reviewers state the acceptable result and the conditions that make an answer unacceptable.
- 03
Scoring rubric
Written rules tell reviewers or automated checks how to classify each result without changing the standard between runs.
- 04
Evaluation runner
A repeatable command runs the versioned cases against a named system version and stores the comparison output.
WHAT IT TAKES
Inputs, time and ownership
FROM YOU
4 inputs
- Representative production examples
- Known good outcomes
- A named subject matter reviewer
- Existing quality documentation
WORKING RANGE
CLIENT ROLE
A subject matter reviewer must supply representative cases, approve expected outcomes and decide disputed results.
WHERE IT SITS