How to Build an Evaluation Set for AI in a Business Workflow

Define the decision being evaluated

Write one sentence describing the AI step. For example, it may classify an incoming request into an existing queue, extract approved fields from a document, draft a response for review, or find supporting passages in an authorized knowledge source. Name the input, the permitted output, the next system action, and the person or rule that handles uncertain cases.

Keep separate tasks in separate evaluations. A case that measures field extraction does not also prove that routing, approval, or a later system update works. If the AI can trigger an external action, evaluate the proposed decision first and test the action path under controlled permissions as a separate part of the workflow test.

Build the case list from the work people actually see

Begin with an authorized sample of recent work and ask the operators where judgment was needed. Include common cases in realistic proportions, then add exceptions whose consequences justify explicit testing. Useful cases can include incomplete inputs, conflicting details, unfamiliar wording, duplicate records, unsupported requests, poor scans, changed templates, and records that belong in a human review queue.

Suppose the task extracts an order number and requested date from emailed documents. The set might contain a clear typed order, a scan with a faint date, a document with two order numbers, a later email correcting the date, and a message with no attachment. Each case should state what the workflow may write, what it must leave unchanged, and when it should stop for review.

For workflows that read outside messages, documents, or retrieved text, include cases where that content instructs the system to ignore its rules, disclose unauthorized data, or use a tool beyond the allowed scope. Build the expected refusal, hold, and no-action behavior from the workflow's approved threat model. Other adversarial cases should follow the access, data, and action risks that apply to the specific design.

Record why each case belongs in the set. This helps the owner distinguish a frequent condition from a rare high-consequence condition and keeps unusual examples from quietly taking over the score. An evaluation enriched with unusual or adversarial cases does not estimate the failure rate in production. Report ordinary and challenge groups separately, and use a combined weighted rate only when the weights come from documented, relevant observations.

Use data the team is authorized to retain

The data owner should approve the source, purpose, access, storage location, retention period, and every destination that receives evaluation inputs or outputs. That review should cover the model and evaluation providers, contractual or regional requirements that apply, and available settings for provider retention, training, or other data use. Permission to store a record internally does not by itself authorize sending it to another processor.

Prefer purpose-built synthetic records when they can preserve the relevant format and relationships. If real records are required to represent the task, use the minimum fields needed and apply the same access controls the business requires for that class of information. Synthetic or de-identified material derived from real records may still contain sensitive structure, rare facts, or recoverable identifiers, so the responsible owner should approve how it is produced and used.

Removing names may still leave account numbers, free-text details, document metadata, linked identifiers, or a rare combination of facts that can identify a person or business. Have the responsible privacy, security, legal, or records owner review the treatment where the data falls under their authority. Keep credentials, access tokens, and unrelated attachments out of the set.

Store case identifiers and approved evidence in a controlled evaluation record. Reports shared with a wider audience can use those identifiers and limited excerpts rather than duplicating the full inputs and outputs.

Write an expected result that allows legitimate variation

Some tasks have one exact result, such as an approved code or a date normalized to an agreed format. Drafting and summarization tasks may have several acceptable answers. For those cases, describe the required facts, prohibited claims, allowed omissions, format rules, source requirements, and conditions that require the system to decline to answer or act and send the item for human review. Avoid treating one editor's preferred wording as the only valid output.

The expected result should cover the workflow action as well as the text. A classification case can name the allowed queue, confidence handling, fields that may be written, and whether a reviewer must confirm the choice. A retrieval case can require references to the approved source version and reject an answer when the available source does not support it.

A case record the reviewers can inspect

Keep the input, expected behavior, scoring rule, and result together. The fields below are a starting point; the task owner should remove fields that do not apply and add controls required by the workflow.

• Stable case identifier and case version
• Business condition represented and reason for inclusion
• Authorized input or controlled reference to it
• Expected facts, fields, classification, or proposed action
• Acceptable variation and prohibited output
• Required hold, decline-to-answer, no-action, or human-review condition
• Severity and business consequence of each relevant failure type
• Reviewer role, approval date, and governing rule version
• Result, evidence reference, reviewer decision, and run identifier

Score failure types instead of hiding them in one average

Define the failure categories before running the set. A missed required field, an unsupported statement, an incorrect route, a privacy exposure, a malformed output, and an unnecessary review can have different consequences. Record the numerator and denominator for every reported rate, and show the number of cases in each business condition so a small group is not mistaken for broad evidence.

The business owner should set acceptance criteria by failure type and consequence. A high overall pass rate can still conceal an unacceptable result in a small group of sensitive or irreversible cases. High-consequence actions may require every case in that category to be reviewed rather than sampled.

When reviewers disagree, keep both decisions and the reason for the final adjudication. Update the case definition if the disagreement exposes an unclear business rule. A changed scoring rule or rubric needs its own version because results under the prior rule may no longer be directly comparable. Do not resolve an ambiguous label by quietly choosing the answer that makes the system score higher.

Keep development examples apart from the final check

Prompt, retrieval, rule, and workflow changes are usually shaped by examples. If the same cases are repeatedly used to make those changes and to report readiness, the result can reflect familiarity with the set rather than performance on unseen work. Mark which cases are available during development and keep a held-out group for the final review.

The held-out group still needs owner approval, protection, and version control. After it has influenced a change, record that use and replace or supplement it with fresh authorized cases before treating a later run as independent evidence. The team should retain prior results so changes in the set do not erase the basis for an earlier decision.

Record enough run context to repeat the evaluation

For each run, record the model provider and identifier, deployment or model version when available, prompt and configuration versions, retrieval index or source snapshot, tool and mapping versions, date, and settings that can affect variation. If the provider does not expose a fixed model version, record the information it does provide and treat a vendor change as a reason to rerun the set.

AI output can vary between repeated calls. Set the repeat count before the run based on the consequence and expected variability, then keep every individual result rather than only the best response. Report variation by case and failure type. Record timeouts, blocked requests, tool failures, unknown outcomes, and cases that never reached the model. They are part of the workflow evidence even though they do not produce an answer to score.

Test the review and action path around the model

A model-only run does not exercise permissions, field mappings, queue ownership, alerts, or the screen a reviewer uses. Before any live write, direct side-effect tests to an isolated or allowlisted test destination, use a dry-run or mocked action path, and constrain credentials to the approved test scope. If no safe test path exists, stop and have the responsible owner approve a different test design. Do not use ordinary evaluation runs as permission to write to production.

Confirm that a held case reaches the assigned queue with the evidence the reviewer needs, and that an approved result writes only the allowed fields. Check a rejected case, an expired review, an unavailable destination, a retry after an unknown outcome, and a correction to a prior result.

Sandboxes, mocks, and test accounts can behave differently from production. Treat the first limited production stage allowed to write data as separately approved launch activity, with constrained scope, reconciliation against authoritative destination records, and a practical pause control.

Rerun the set when the workflow changes

Assign an owner to the evaluation set, its approval rules, and the run record. Rerun it after a model, prompt, retrieval source, tool, mapping, business rule, or review procedure changes. A material shift in input format or case mix also warrants new cases and another owner review.

Compare the new run with the last accepted version by case and failure type. Investigate newly failing cases and cases whose earlier failure disappeared, since either change can reveal altered behavior or an outdated expectation. Record the release decision, approver, accepted limitations, and follow-up date without overwriting the earlier evidence.

The set will age as business work changes. Review recent authorized examples on a defined schedule and add conditions the current collection no longer represents. Retire a case only with a recorded reason and preserve the result history required by the business's evidence and retention rules.