A verdict is useful only if a reader can trace it back to the expectation, the evidence, and the rule that decided it. Engineering Note 1 explains why we evaluate this way. This companion sets out the rules: what a run must record, how a case is judged, and what its result allows us to conclude.
Missing evidence never becomes a pass, and a serious failure cannot be averaged away. The distinctions below make those commitments precise, including when a verdict must remain open and what a grader has to demonstrate before we trust it.
1. What we record
Each evaluation run must preserve its inputs, outputs, evidence and verdicts in a permanent record.
| Part | What it holds |
|---|---|
| What ran | The code version; the AI model, its provider and settings; the versions of its instructions, skills and tool descriptions; the data snapshot; the pinned clock time; the market session of each instrument; the model's training cutoff |
| What it produced | The answer, anything it made (tickets, charts, files), and the individual claims pulled out of them |
| What it relied on | Every document it fetched, where from, how current it was, and what kind of evidence it counts as |
| Verdicts | One per check (per claim, where a check looks at claims one by one), with who or what gave it |
| Settlement | For forecasts and explanations: the time horizon, the source of truth, the baseline, and the settled value and time |
| Inventory | Which cases, checks and runs were selected, and which are missing |
Later outcomes and corrections are added as dated entries. The original answer and its original assessment remain available, so readers can reconstruct what was known at each stage.
2. Five kinds of evaluation
Different claims need different proof, and some can only be graded later.
| Kind | Evidence used | When it settles |
|---|---|---|
| Factual claim | The right document, period and accounting basis | Now |
| Workflow requirement | A log of what the system did, and what it produced | Now |
| Operational performance | Measured runs, failures included | Continuously |
| Forecast | The later price or disclosure, against a simple baseline | At its horizon |
| Explanation | Evidence for the stated cause, weighed against other possible causes | Later. Sometimes a definitive answer is unknowable |
3. What a test case is
A case is a situation, an action, and expectations a person wrote down before anything ran.
The three layers put the same requirements under different conditions. A direct tool test controls the inputs. A single-request evaluation lets the agent choose how to act. A conversation evaluation adds the need to carry facts, intent and unfinished work across turns. Agent cases run repeatedly because their paths and answers can vary.
| Part of a case | Layer 1: one tool | Layer 2: one request to the agent | Layer 3: a conversation with the agent |
|---|---|---|---|
| Setup | Seeded data, who is acting, the pinned clock | Same | Same |
| Action | A tool call with chosen inputs | One user request | A series of turns, with scripted user replies |
| Expectations (written before running) | What the inputs must mean, what must change, what must stay the same, checks that must succeed, facts the reply must and must not state | Same | Same, for each turn |
| Evidence collected | Before and after snapshots, check results, the tool's receipt | The same, plus the order of effects inside the turn, the agent's reply and its citations | The same after every turn |
| How many runs | Once in each market phase the tool serves (three if the tool writes text with a model) | Three in each market phase the case covers; the worst one counts | Three in each market phase the case covers; the worst one counts |
Cases serve different purposes. Acceptance cases help decide whether a release can ship and require human review. Regression cases preserve failures we have fixed. Characterization cases measure behavior without blocking a release. We also keep cases out of development so that the final comparison includes questions the change was not tuned to answer.
For experiments intended to compare performance, the selection rule and random seed are recorded before we inspect the answers. Diagnostic cases chosen to investigate a known problem remain labeled as such.
4. The nine minimum checks
Every run has to clear the same nine checks. They're the minimum per run, not the whole evaluation (stability across runs, platform quality, cost and latency, and outcomes over time are measured on top). A check names the evidence it needs before it runs.
| # | Check | The question | Passes when |
|---|---|---|---|
| 1 | Entitlement | Is the person entitled to this data and this action, and did they ask for it? | Every read stays within the person's data entitlements, and every change traces to a request that authorizes it. A case that expects a refusal shows zero changes |
| 2 | Inputs | Were inputs correct, and were invalid inputs handled and disclosed? | Every bad input gets its declared response and changes nothing. In agent cases, every important call's inputs fall inside the accepted set, and a correct retry doesn't excuse an earlier forbidden call |
| 3 | Right result | Did the intended thing happen? | Each expected change appears exactly once with the declared values, and returned or computed values match within tolerance |
| 4 | Side effects | Did anything happen that wasn't asked for? | Take the expected changes away and only declared housekeeping writes are left |
| 5 | Repeats and failures | If it runs twice or fails halfway, is the result still right, and is the gap named? | Counts and history match what's declared, and partial results are named |
| 6 | Works when used | Does it work when used? | Every declared check of the output (the ticket validates, the chart plots, the schedule fires) returns its independently stated result |
| 7 | Honest reporting | Does what the answer says match what happened? | Fixed values (a price, a percentage, a saved time) are checked in code against the evidence. Open-ended interpretation is reviewed by a person with a short rubric |
| 8 | Time | Is every number right for its moment and market phase, and labeled with its clock? | The five time rules below hold for each market phase the case covers. A phase never actually observed is unproven |
| 9 | Attribution | Does each important claim rest on a citation that supports it? | No blocking citation failure (missing, pointing at the wrong source, the wrong kind of evidence, or a quote with no location). Other citation problems are counted |
The five time rules (check 8). The system must choose the right behavior for the instrument's market session, meet the contract for that session, state the data's as-of time, preserve facts that have not changed, and use the calendar of the instrument's own exchange.
Which time a number belongs to. A source timestamp, an exchange session, an options quote and a report's publication time describe different things. We retain the timestamps needed to interpret each value and to establish when the system could have known it. Download time records when evidence reached us; it does not establish when the underlying observation was made. Live session checks must also be observed in the relevant market phase, with their results retained for release review.
5. Verdicts
Five possible verdicts, and "we couldn't check" never counts as a pass.
| Verdict | Means |
|---|---|
pass | The expectation was met |
fail | The system did something its contract doesn't allow |
n/a | The check doesn't apply to the behaviors this case covers |
unproven | The check applies, but the evidence couldn't be collected (missing infrastructure, a broken check, missing reference data). Never a pass, and never a product failure |
unresolved | The claim was measured, but the evidence can't settle it yet, like a forecast before its horizon or an explanation. Reported with its count, never folded into pass or fail |
Any other value is an error and stops the comparison.
6. How verdicts combine
Verdicts combine strictly. One bad run fails a case, and a serious failure can't be averaged away.
In one run, a case passes only if every required check passes for what can be judged now. Unresolved claims are counted separately.
Worst run decides. For the checks that gate a release, agent cases run three times, and the worst run is the verdict. Measures that vary by nature, like forecast accuracy, are compared as rates across many cases instead. Three runs with one failure is a failed case. In a conversation, every step is a case, and the conversation passes only if every step passes.
Three passes are a screen, not a guarantee. A case that fails one time in ten passes all three runs about 73% of the time.
Serious failures are never averaged. Showing someone data they aren't entitled to, a wrong required number, or an important claim with no support fails the run whatever the other scores are.
Comparisons report rates. Comparing two versions reports passes out of runs, with the denominator and the missing count ("27/30 → 19/30", never "pass → fail"). A dropped case, run or check is lost evidence and makes the comparison exit with an error. When the cases can't tell two versions apart, the result is "not established," never "equal."
Weights follow the reader. When pass rates are summarized, criteria are weighted by the kind of user (an analyst and a wealth advisor care about different things). A customer profile can add requirements and change weights. It can never remove one of ours.
Consistency is reported beside accuracy for anything asked more than once.
Diagnostics never change a verdict. Which tool ran, exact spelling of inputs, or the number of calls are recorded but don't decide pass or fail.
Nothing an AI grader decides blocks a release on its own. A person reviews AI-judged pieces until the grader is promoted (section 9).
7. The house criteria
What "good" means before any client adds rules of their own.
- Identity
- Time and recency
- Calculation
- Evidence and attribution
- Correctness of the read
- Coverage and intent
- Choosing and setting up the right tools
- Actionability and outputs
- Entitlements and compliance
- Robustness: asking when unclear, falling back safely, refusing only when it should
- Voice and length
- Latency and cost
- Surfacing disagreement
- Charts
- Claims that settle later
An unnecessary refusal can be a failure when the request was permitted and the run already had enough information to answer it. Numerical calculations are checked by independent recomputation rather than by asking a model whether the arithmetic looks right.
8. Claims that settle later
Forecasts wait for their stated horizon. Explanations are assessed against available evidence and may remain unresolved. Both retain the original wording and timestamp.
Explanations are assessed on five dimensions: chronology; the stock's move relative to the market, sector and relevant peers; support for the proposed mechanism; competing explanations; and contradictions. The contemporaneous assessment uses only news and prices available when the note was written. Later prices and disclosures belong in a separate outcome assessment. A supported explanation remains an evidence-based judgment about a cause, not proof of a unique cause.
Forecasts are saved and fingerprinted before the outcome. Business forecasts and price calls are graded separately, each against a simple baseline that was already known when the call was made, with no-calls and misses in the same table. Calibration is judged only where probabilities were stated.
Each settled record holds when it was written (and which version: the first or the latest update), the horizon, the source of truth, the baseline, the settled value and time, dated corrections, and a fingerprint for calls registered in advance.
Settled results feed the improvement loop. They never gate a release directly.
9. The graders are graded
A grader's verdict only counts after it passes its own tests, and it starts out advisory.
This applies to everything that gives a verdict: AI judges, rubrics, and simple code checks. A code check needs known-good and known-bad examples. An AI judge that will gate releases needs all seven kinds of control:
| Control | What it tests |
|---|---|
| Known-bad, one per rubric rule | It fails what it should, and the report says which rule caught it |
| Known-good | It passes what it should |
| Hard positive | Right answers that look wrong: true details the reference lacks, correct refusals |
| Near-miss pair | The boundary between pass and fail |
| Adversarial | Answers built to fool it: confident tone, citations that exist but don't support the claim, qualifications borrowed from the source, refusals that argue their own correctness |
| Invariance | The verdict doesn't change when the answer is reworded, reordered, lengthened or anonymized, as long as the information the rule needs is still there |
| Real-output sample | Agreement with people on real outputs, published |
Repeat runs measure the grader's own noise. Someone other than the rubric's author writes the controls, and part of them is kept from that author.
A grader moves from advisory to characterization to gating only against stated thresholds for catch rate, pass rate and agreement with people, and with sign-off. Passing synthetic controls isn't enough: the promotion report shows false passes and false fails on real outputs, including hard cases.
That trust must be maintained as the system changes. We re-check a grader whenever its rubric, model or reference format changes, and retain each grader failure as a regression control.
Arithmetic, formats, identities, evidence types and citation chains are checked in code and need no model.
10. Failures and requirements are one record
Every failure points at the requirement it broke, and stays open until a test proves the fix.
A failure record begins with what the user experienced and how often it occurred. It then links that observation to the violated requirement, the cause, the regression case and the current status: open, fixed or guarded. If the behavior was never specified, the record identifies a missing requirement.
A failure is closed only when it is guarded: the regression case fails on the earlier version and passes with the fix. Reports lead with open and recurring failures, so a rising aggregate score cannot obscure a problem that still affects users.
11. Status labels
Every result says how far along it is, and only a person can approve.
Every result is planned (written down), implemented (exists and runs) or measured (has results). Separately, it's approved once its owner signs off, which can happen before it's built.