Evaluate the Context Path, Not Just the Answer

In an illustrative evaluation, two agents answer the same policy question with the same sentence. The first retrieves the approved policy currently owned by finance. The second retrieves an expired copy from a popular team folder. The relevant limit happened not to change between versions, so both final answers look correct.

An answer-only score gives both runs full credit. An operating team should not.

The second path is fragile. The next policy update will turn the same retrieval behavior into an error, and the final response will provide no warning that its source selection was wrong.

Define correctness for the request class

Not every enterprise question requires the same retrieval behavior.

“List every open ticket assigned to this queue” requires completeness, pagination, deduplication, and a clear cutoff time.

“What is the current discount policy?” requires the authoritative owner, effective version, applicable region, and permission to use the source.

“Prepare a renewal risk brief” requires a bounded synthesis across CRM, support, contract, usage, and conversation records, with inference kept separate from recorded facts.

An evaluation set should label these request classes before scoring outputs. Otherwise a system can improve one aggregate metric while becoming less dependable for a critical kind of work.

Inspect the checkpoints that created the response

For each run, retain enough trace data to answer:

• Which sources were considered and selected?
• Was the required record set complete?
• Which authority, date, and permission checks ran?
• What information was inferred rather than observed?
• Which tools were called, with what result?
• Did the agent stop or escalate when a required fact was unavailable?

This does not mean exposing private chain-of-thought. Operational evaluation needs observable decisions and system events: search queries, source identifiers, tool inputs and outputs, policy results, action records, and the final evidence set.

A preference score is difficult to interpret without the dataset, grader instructions, exclusions, and conditions under which the preference was recorded. Evaluation should preserve that information alongside the result so a team can distinguish a repeatable improvement from a favorable but narrow test.

Separate deterministic checks from judgment

Some failures can be tested without an AI grader:

• the selected policy version is not current;
• pagination stopped before the final page;
• a required source is absent;
• a restricted record appeared in the evidence set;
• a write action occurred without the required approval; or
• the final record does not match the proposed update.

Other qualities require judgment: whether a synthesis is decision-useful, whether uncertainty is communicated proportionately, or whether the brief focuses on the material risks.

Human review is valuable for these questions. LLM-based graders can help scale review, but their prompts, model versions, and score variance become part of the evaluation system. A grader that changes its opinion across repeated runs should not be treated as ground truth.

Test the failures that matter to the business owner

A generic question set will miss the hardest operating conditions. Build cases from the workflow’s real boundaries.

For a renewal brief, include a contract date that conflicts with CRM, a restricted support incident, a customer commitment without an owner, and a discount request above the account team’s authority. For a security review, include superseded evidence and a control with an expired attestation.

The Coryntas agent lifecycle evaluates the complete path from context through action. The test set should grow when reviewers correct a source, reject an inference, or handle an exception the original design missed.

There is a practical limit. Full trace evaluation costs more than checking final text. Some tests require representative enterprise data that is difficult to reproduce outside a controlled environment. The answer is to tier evaluation by risk, not to pretend every run deserves the same depth.

An enterprise agent is dependable when its correct answers come from repeatably correct paths. Measuring the path is how a team distinguishes a system it can operate from one that happened to be right.