Field note · Evaluation · January 2026
Evaluate agent behaviour, not just the final answer
An agent can produce an acceptable answer through an unsafe path, or fail a brittle benchmark while finding a better solution. Evaluation must cover the trajectory, environment and business outcome.
Move beyond text grading
Agent evaluations should inspect tool choice, argument correctness, sequencing, policy compliance, state changes and recovery. Deterministic environment checks should grade what actually changed.
Use a portfolio of evaluators
Combine unit tests, rule-based assertions, trace analysis, model graders and human review. No single evaluator is reliable enough for every dimension, especially where quality is subjective or the agent can find unexpected valid solutions.
Test failure behaviour
Include tool timeouts, partial data, stale context, permission denial, duplicate events, adversarial instructions and model fallback. Measure whether the agent contains damage and escalates appropriately.
Connect to operations
Track task success, false action rate, human override, latency, token and tool cost, retries and post-deployment regressions. Re-run representative traces whenever models, prompts, tools or policies change.
Executive takeaway
Evals are not a launch checklist. They are the control system that lets an organisation improve agent capability without losing trust as the surrounding system evolves.