A successful query, attractive chart, and complete report do not independently make an answer trustworthy. The agent may have selected the wrong source, bound the wrong metric, executed correct SQL over incomplete data, and then described correlation as causation.
One accuracy score compresses these failures into a number and hides high-cost errors. A better evaluation follows the analysis chain layer by layer and sends each failure back to the component that can repair it.
Lower-layer errors contaminate everything above them. If the source is wrong, report-writing quality is irrelevant. If access is unauthorized, a high average score cannot compensate.
Five layers answer different questions
| Layer | Verification question | Return on failure |
|---|---|---|
| Source | Are asset, version, authorization, watermark, and coverage correct? | Access and routing |
| Semantics | Do metric, entity, grain, time, joins, and filters match the business definition? | Semantic plan or clarification |
| Execution | Did query or code run safely, completely, and according to contract? | Parameters, program, or runtime |
| Analysis | Are baseline, method, inference, and uncertainty defensible? | Analysis plan or more evidence |
| Delivery | Are claims cited, visuals honest, and recipients authorized? | Report, artifact, or approval flow |
Each layer should emit a structured status such as pass, fail, unknown, or needs_human, with evidence. Unknown is not pass. A model declaring its own answer “consistent” is not independent verification.
One question has several forms of correctness
For “Which channel caused this month’s conversion decline?” consider:
- Task correctness: did the agent calculate contribution to change, rather than list the lowest-rate channel?
- Data correctness: are all channels covered and watermarks complete?
- Semantic correctness: are numerator, denominator, and new-customer definition consistent?
- Program correctness: did joins avoid duplication, and was output truncated?
- Analytical correctness: did mix shift create an aggregation artifact, and is causation overstated?
- Delivery correctness: do conclusion, evidence, limitations, and recommendation align for an authorized reader?
Exact-match SQL evaluation rejects equivalent queries and accepts executable but semantically wrong ones. Execute the query, inspect semantic binding and result properties, then use rules or people to judge the analytical claim.
Build a task set from real work
Public benchmarks compare research systems. Enterprise release decisions still need an internal suite sourced from actual questions, tickets, reports, and incidents, stratified by domain, difficulty, risk, and data form.
For each case, record user identity, objective, allowed and forbidden assets, semantic version, data snapshot, expected artifact, acceptable variants, prohibited behavior, and scoring rules. Open analysis does not require one reference paragraph, but it does need factual invariants, required hypotheses, and inference boundaries.
Hard cases should not mean only longer prompts. Include similarly named tables, ambiguous metrics, permission differences, stale data, empty-versus-zero, tool timeouts, conflicting evidence, and unanswerable requests.
Match metrics to failure cost
A useful scorecard may include:
- Asset choice and coverage: Hit@K, Recall@K, and complete-set coverage for multi-source tasks.
- Semantic binding: correct metric/entity/version plus clarification or refusal under ambiguity.
- Execution reliability: executable rate, output-contract pass, timeout, resource overrun, and duplicated side effects.
- Claim support: material factual claims linked to accessible evidence.
- Calibration: whether expressed confidence tracks correctness and contracts under uncertainty.
- Security blocks: unauthorized allow rate, reported independently as a release gate.
- Cost and latency: end-to-end p50/p95, model/tool calls, scanned data, and review time.
Do not average them into one universal score. Unauthorized access, an incorrect write, and a high-confidence unsupported claim cost far more than imperfect formatting. Scenario risk determines the gates and weights.
Offline, shadow, and online evaluation
Offline regression compares models, prompts, semantic versions, and tools on fixed snapshots. Run a core suite and recent-incident suite for every release, retaining traces and diffs.
Shadow mode processes real requests without returning results to users or submitting production actions. It reveals catalog, permission, cost, and long-tail behavior under low operational risk.
Online monitoring tracks completion, human edits, refusal, escalation, tool failure, policy blocks, latency, and cost. A thumbs-up is not a correctness label. Human edits should distinguish style changes from definition corrections and factual errors.
Confirmed online failures enter regression after sensitive data is minimized, versions are bound, and expiration conditions are recorded. Evaluation data is a governed product with owners and refresh policy.
Evaluate the evaluator
LLM judges can assess structure, coverage, and open-ended writing, but may prefer longer, more confident, or stylistically similar answers. Calibrate them against a multi-expert sample, measure agreement, and preserve human review for high-impact conclusions.
Use deterministic validators when possible: SQL parsing and execution, schema contracts, numeric tolerances, invariants, authorization policy, and citation accessibility. A model judge handles what cannot be formalized; it should not replace the entire test stack.
Observability must reconstruct a task
A trace should connect request and identity, context candidates, semantic plan, plan version, tool-parameter digest, execution ID, validation results, final artifact, and human edits. Avoid credentials and unnecessary sensitive text in ordinary logs.
With a connected trace, a team can determine whether failure came from source selection, semantics, SQL, inference, or presentation. The error taxonomy should route directly to an owner and repair layer.
A minimum viable evaluation program
- Select 50–100 frequent tasks, including at least 20% refusal, authorization, and failure cases.
- Label sources, semantics, invariants, allowed variants, and error cost.
- Freeze snapshots and establish a human or existing-system baseline.
- Score by layer and retain traces, not only final text.
- Validate authorization, cost, and recovery in shadow mode.
- Define release blockers and rollback conditions; continuously add confirmed failures.
Trust is not a “95% model confidence” label. It is inspectable evidence at each layer, with uncertainty preserved instead of polished away. The purpose of evaluation is to decide where the system may accept responsibility—and where it must stop.