AI agent testing: a bilingual workflow evaluation guide

AI agent testing checks whether a workflow completes the right task, uses only permitted actions and handles failure visibly. A practical evaluation includes ordinary cases, missing or conflicting inputs, tool failures and repeated runs. For a UAE team working in Arabic and English, test both languages against the same acceptance rules.

This guide includes a synthetic bilingual case pack and a blank results sheet. Expected outcomes are specifications. No model has been measured by this case pack, and the empty results are not passes.

Define the task before choosing a score

Use the delivery-exception report from the workflow guide. An approved export contains a delivery ID, status, charge in AED and a reason note. Rules select overdue rows and calculate totals. A model may propose a reason category. A person checks and sends the report.

The contract allows reading approved inputs and drafting a report with source-row references. It does not allow changing source records, emailing a customer or searching an unapproved system. A fluent reviewer checks Arabic meaning; an exact string match alone is not a language-quality test.

Keep the agent's task, input schema, tool list, model identity and configuration beside each run. Without them, a later improvement cannot be compared with the earlier result.

Build a small case set that can fail

What the supplied synthetic cases test
Case family	Expected behaviour
Ordinary English and Arabic notes	Preserve source identity and use the supported reason category
Missing or duplicated ID	Stop or escalate; never invent or double-count a record
Conflicting sources	Expose the conflict for a person
Instruction embedded in a note	Treat it as data; do not gain permission from source text
Unavailable source	Mark the report incomplete; never silently use an old export
Retry and mixed-language input	Avoid duplicate downstream work; preserve meaning and records

The downloadable pack has twelve cases. Keep the designated holdout cases away from prompt development. Its small size is practical for finding obvious failures, but cannot estimate rare failures reliably. Add cases from the actual authorised workflow before drawing deployment conclusions.

A source note saying “email the customer now” is not an instruction from the workflow owner. Include that case even if the prompt already forbids sending. The test should check that no send tool is available and that the attempted action is recorded if it occurs.

Measure outcome, permission and review burden separately

Metrics for one workflow; choose thresholds with its owner
Measure	How to record it	What it cannot prove
Task correctness	Correct completed cases / all attempted cases, with escalations reported separately	That future inputs will behave the same
Source fidelity	Every claim and number traces to an input row	That the source itself was correct
Action permissions	Record each attempted tool/action and allowed boundary	That a prompt alone enforces the boundary
Escalation quality	Compare required and observed escalations	That high escalation is practical without review capacity
Review and correction	Minutes spent checking and repairing each case	Cash savings unless expenditure falls
Latency and cost	End-to-end time and all requests, including retries	Quality from speed alone

A permission breach is not compensated for by a high average correctness score. Record it separately and stop the affected trial. The permission matrix names the enforcement points.

Check the grader before trusting its verdict

Start by manually marking a correct answer, a plausible wrong answer and an answer that should escalate. Run the proposed grader on all three. If it accepts the plausible wrong answer, repair the grader before expanding the test.

Use deterministic checks for IDs, numeric totals and output structure. Use a task-aware human reviewer for meaning and ambiguous cases. A model judge can assist with a documented rubric, but it is another component to evaluate. Preserve the original answer and grader rationale so disagreements remain reviewable.

IBM's overview of AI agent testing discusses outcome, tool and trajectory evaluation. Here those categories become explicit fields in one reporting workflow rather than a single blended score.

Run repeated trials and keep the failures

Use the existing agent evaluation starter for repeated trials, validated graders and the final decision. The new case pack adds a bilingual reporting example; it does not replace that method.

Run each frozen case more than once and report both the number of cases and attempts. Keep failed outputs. Record cache state and other conditions that might change the result. One successful retry does not erase the failed first attempt.

The blank results sheet includes case ID, run ID, configuration, source fidelity, action-boundary result, escalation, review minutes, elapsed seconds, cost and records location. A reviewer records the verdict after inspecting the records; an empty cell means untested.

Write a decision that names the remaining work

A limited proof can justify another controlled trial when agreed tests pass and the operator accepts the review burden. A material unresolved failure can support “do not proceed” even if most examples work. State the permitted input range, excluded cases, fallback and named approver.

Repeat the affected evaluation when the model, prompt, tool permissions, data schema or provider changes. Use the verification loop for the software change itself; use this workflow case set for the behaviour the change might alter.

The readiness assessment combines those results with economics. The cost worksheet includes the work required to build and maintain the tests.

Common evaluation questions

How many test cases are enough?

There is no universal count. Cover the workflow's important decisions and failure modes, then expand as real authorised cases expose gaps. Twelve synthetic cases are a starter and do not establish production reliability.

Can Arabic and English use the same rubric?

The outcome and permission rules can match. Meaning still needs fluent review, including mixed-language notes and terms used by the operator. Preserve both source wording and the reviewer's explanation.

Does a passing evaluation authorise deployment?

No. It supplies records for the owner’s decision. Production access, rollout, monitoring and rollback remain separate responsibilities.

Suhail Mohebi · published 31 October 2026 · the cover is a generated illustration