An AI workflow needs a test set that represents the work it will actually perform. A handful of successful demonstrations can show that an idea is possible, but they do not establish how it behaves with incomplete information, ambiguous requests or unfamiliar wording.
A useful test set combines representative tasks with deliberately challenging cases and clear expected outcomes. It helps the team compare changes, investigate failures and decide whether the workflow is suitable for its intended role. The results should support a business decision rather than merely produce a reassuring average score.
Begin with the workflow boundary and the consequences of errors. An assistant drafting an internal summary needs different checks from an agent that can change a customer record. The test cases, reviewers and release criteria should reflect that difference.
Define the task and the permitted outcome
Write down what the workflow receives, what it should produce and which systems or information it can use. Identify the point at which a person reviews the result or the system performs an action. This makes the object of evaluation clear.
Break success into observable qualities. A summary may need to preserve important facts, distinguish confirmed details from assumptions and omit unsupported conclusions. A classification workflow may need to choose an allowed category and route uncertain cases to review.
Include the boundaries that matter to the business. The workflow may be required to refuse an unsupported action, ask for clarification or stop when a necessary source is missing. Those behaviours are valid outcomes and should appear in the test set.
Use reviewed, reusable cases to compare model, prompt and agent versions. Repeatability lets the team assess a change against the same meaningful tasks instead of a fresh selection of favourable demonstrations. Include unsuitable and failure cases. Our AI evaluation dataset guide provides the related checks.
Collect representative cases from the real workflow
Review the types of requests, documents or records the process already handles. Select cases across the meaningful variations, including different lengths, terminology, completeness and customer circumstances. Keep the sample connected to the defined task rather than assembling an unrelated collection of difficult prompts.
Use the organisation's agreed data-handling process. Representative synthetic or appropriately transformed examples may be useful where original records contain sensitive information. Check that the transformation preserves the features being tested without exposing details unnecessarily.
Record where each case came from and why it belongs in the set. A provenance note helps reviewers understand whether the example reflects common work, a known failure or a hypothetical boundary. These categories should remain visible in the results.
Include the ordinary cases as well as the challenging ones. A workflow that handles rare complications but struggles with everyday requests is not ready. Conversely, a set consisting only of tidy normal cases can conceal important weaknesses.
Add cases that exercise specific boundaries
Create incomplete-input cases by removing information that the workflow genuinely needs. Define whether it should ask a question, produce a qualified draft or route the case to a person. Do not reward confident guessing when the approved process requires a clarification.
Include conflicting information where the source material can realistically disagree. Test whether the workflow identifies the conflict and follows the agreed authority rule. A polished answer that silently chooses the wrong source can be harder to detect than an obvious failure.
Test unsupported and out-of-scope requests. For example, an internal knowledge assistant may receive a request to confirm a current price that is absent from its approved material. The expected outcome should preserve the boundary and offer an appropriate route.
For workflows using retrieved documents or tools, include cases where untrusted content contains instructions that conflict with the task. Test in an appropriate environment with controlled actions. The team needs to assess whether content is treated as evidence rather than new authority.
Specify expected outcomes without demanding one sentence
Write a short reference outcome for each case. Identify the facts, decisions or actions that must be correct and the mistakes that would make the result unacceptable. This is more useful than requiring every valid answer to match one exact wording.
For an enquiry summary, the reference might require the requested service, the stated deadline and the unanswered question. It might also prohibit an invented budget. Several differently phrased summaries could meet those requirements.
Separate mechanically checkable requirements from judgement. Allowed categories, required fields and known values may suit deterministic checks. Relevance, clarity and completeness may require a reviewed rubric and human assessment, especially when errors have meaningful consequences.
Have domain experts review the reference outcomes before using them to score the system. A mistaken reference answer can penalise correct behaviour or reward the wrong result. Record disagreements and resolve the business rule rather than asking the model to settle it.
Keep development examples separate from evaluation
The team may use examples to improve prompts and configuration. Maintain a separate reviewed set for assessing the resulting change, so success is not judged only on cases repeatedly used during development.
Keep related variations together when planning the separation. Several near-identical rewrites of one request can create the impression of broad evaluation while testing essentially the same example. Consider the underlying task and source when assigning cases.
Version the dataset, reference outcomes and scoring rules. Record the workflow configuration, model, tools and knowledge sources used in each run. This helps explain a change in performance and supports a comparison that another reviewer can reproduce.
Do not remove failed cases merely to improve the headline score. Correct a defective test when its expected outcome is wrong, but preserve valid failures as evidence. They show where the workflow or its operating boundary needs further work.
Review results by error type and business impact
Group failures into useful categories such as missing facts, invented details, incorrect routing, poor clarification and unauthorised action attempts. Distinguish the cause where evidence allows: retrieval, source quality, workflow logic or generation may require different fixes.
Inspect results by task category rather than relying only on an overall average. Common easy cases can hide poor performance on a smaller but important group. Report which cases were tested and the limitations of the sample.
Use automated scoring as supporting evidence and inspect its disagreements with expert review. A model judging another model's answer can make mistakes or overlook business-specific distinctions. Verify the judgement on representative cases before relying on it for release decisions.
Consider repeat runs where variable outputs matter to the task. Keep the test conditions recorded and inspect whether failures recur or appear intermittently. A single successful answer does not establish stable behaviour for an important boundary.
For an illustrative repair enquiry classifier, include a message that describes two faults requiring different teams. The reference outcome might require a coordinator to review the request instead of choosing a single specialist. If the workflow assigns one team confidently, record the missed overlap even when its selected category is valid. That result could pass a simple allowed-category check while failing the business task. Keep the original wording and expected handover with the case, then compare later configurations against it. Also include an ordinary single-fault enquiry, so a change intended to handle overlap does not send every request to review. The pair shows why evaluation needs both boundary cases and checks on the everyday work.
Connect the evidence to a release and maintenance process
Agree the required outcomes and unacceptable failures before the final evaluation. Different workflows need different thresholds and review arrangements. A drafting tool may launch with mandatory human review while a proposed action-taking workflow needs a more restricted boundary.
Turn verified live failures and new task types into reviewed cases for later testing. Keep the test set useful as the business and knowledge sources change, while preserving a stable core for comparisons. Assign an owner for this continuing work.
Giraffe Digital's digital strategy service considers AI tools within the wider business journey. A well-designed test set helps the team assess what an AI workflow can do reliably, where people remain responsible and which improvements the evidence supports.
Frequently asked questions
How should we define the task and the permitted outcome?
Write down what the workflow receives, what it should produce and which systems or information it can use. Identify the point at which a person reviews the result or the system performs an action. This makes the object of evaluation clear. Break success into observable qualities. A summary may need to preserve important facts, distinguish confirmed details from assumptions and omit unsupported conclusions. A classification workflow may need to choose an allowed category and route uncertain cases to review. Include the boundaries that matter to the business. The workflow may be required to refuse an unsupported action, ask for clarification or stop when a necessary source is missing. Those behaviours are valid outcomes and should appear in the test set.
How should we add cases that exercise specific boundaries?
Create incomplete-input cases by removing information that the workflow genuinely needs. Define whether it should ask a question, produce a qualified draft or route the case to a person. Do not reward confident guessing when the approved process requires a clarification. Include conflicting information where the source material can realistically disagree. Test whether the workflow identifies the conflict and follows the agreed authority rule. A polished answer that silently chooses the wrong source can be harder to detect than an obvious failure. Test unsupported and out-of-scope requests. For example, an internal knowledge assistant may receive a request to confirm a current price that is absent from its approved material. The expected outcome should preserve the boundary and offer an appropriate route.
How should we specify expected outcomes without demanding one sentence?
Write a short reference outcome for each case. Identify the facts, decisions or actions that must be correct and the mistakes that would make the result unacceptable. This is more useful than requiring every valid answer to match one exact wording. For an enquiry summary, the reference might require the requested service, the stated deadline and the unanswered question. It might also prohibit an invented budget. Several differently phrased summaries could meet those requirements. Separate mechanically checkable requirements from judgement. Allowed categories, required fields and known values may suit deterministic checks. Relevance, clarity and completeness may require a reviewed rubric and human assessment, especially when errors have meaningful consequences. Have domain experts review the reference outcomes before using them to score the system.
How should we keep development examples separate from evaluation?
The team may use examples to improve prompts and configuration. Maintain a separate reviewed set for assessing the resulting change, so success is not judged only on cases repeatedly used during development. Keep related variations together when planning the separation. Several near-identical rewrites of one request can create the impression of broad evaluation while testing essentially the same example. Consider the underlying task and source when assigning cases. Version the dataset, reference outcomes and scoring rules. Record the workflow configuration, model, tools and knowledge sources used in each run. This helps explain a change in performance and supports a comparison that another reviewer can reproduce. Do not remove failed cases merely to improve the headline score.
How should we connect the evidence to a release and maintenance process?
Agree the required outcomes and unacceptable failures before the final evaluation. Different workflows need different thresholds and review arrangements. A drafting tool may launch with mandatory human review while a proposed action-taking workflow needs a more restricted boundary. Turn verified live failures and new task types into reviewed cases for later testing. Keep the test set useful as the business and knowledge sources change, while preserving a stable core for comparisons. Assign an owner for this continuing work. Giraffe Digital's digital strategy service considers AI tools within the wider business journey. A well-designed test set helps the team assess what an AI workflow can do reliably, where people remain responsible and which improvements the evidence supports.
