Ask an AI system why a stock fell yesterday, and it has an easy job. The news has been written up, the filings are out, and the explanation is there to be assembled.
Ask while the stock is still falling, and the job changes. News is still arriving, some of it only through particular sources, and someone has to decide what it means before the picture is complete. The person asking still needs an answer they can act on. That second job is the one our clients rely on, and it is much harder to evaluate.
We tested the difference directly. We took 25 ticker-days, picked without looking at the answers, and compared the notes our system wrote during each session with the explanations it gave afterward. Some live notes had already found the company news that mattered. Others described options activity in detail but never checked how the stock was moving against its sector. Case Study 1 walks through both.
So when we decide whether the system is ready to ship, we need two kinds of evidence:
whether its claims were supported when it made them (can it cite every source), and
whether they held up once the outcome was known (was the answer right).
We have been recording what the system says and checking it against what happens since our early days. As our platform matured, our evaluation system has also evolved with it. Each failure gives us something concrete to improve and another case future versions must answer for. This note describes how that continuing work shapes what we ship.
1. What we're testing, and why it's hard
Most people picture an AI system as a chat box. Ours is one: people ask in plain language ("why did this stock fall today?", "give me a way to play a rate hike") and get back analysis, charts and trade tickets, with their sources.
But much of its work happens without anyone asking. Some jobs run on a schedule; others start when an event happens like a stock moves, a headline breaks or a filing lands. Through an earnings season the AgentSmyth platform follows every company in our universe, saves a timestamped call before each report, and checks that call against the results and the stock's reaction afterward. It tracks the wider market the same way: the macro regime, narratives moving groups of stocks, and each client's watchlists, publishing a client-ready artifact whenever it moves.
That changes what testing means. We aren't grading one answer in isolation. We're grading a running record, where every answer has a time, the evidence behind it, and someone who may have acted on it. Some claims can be checked the same day, others take weeks, and some never fully settle.
The model and the harness
It helps to separate two things: the model and the harness. The model reads the request, reasons about the evidence and decides what to do next. With tools, it can fetch today's prices, read a new filing or run a calculation. Meanwhile, the harness can be thought of as the underlying software that makes those actions work: it executes the model's tool calls, returns the results and maintains the state of the task.
Think of an analyst at a desk. The model is the analyst. The harness is the desk: the screens, the files, the order system and the rules for using them, including what each user may see, what a request may spend, and what gets checked before anything leaves. It also logs every step, which is what lets us grade how the system got to an answer, not just the answer itself. Change the desk and the work can change, even with the same analyst. Case Study 3 shows a faster, cheaper harness that completed fewer trade requests.
The two need different tests. Much of the harness is ordinary software: give it fixed data and it should give the same result every time. The model isn't like that. It chooses its own steps, and the same request can come back different. So we test five parts of the platform, each in its own way:
| Part | What it is | Example | How it's tested |
|---|---|---|---|
| Tool | One specific operation: fetch data, run a calculation, save something | Get a stock's price history; schedule a report | Directly, like normal software, in every market phase it serves |
| Skill | Written instructions the AI loads to do one job well | Forecasting an earnings result; building a trade ticket | Through the agent: a skill passes when the agent using it does the job right |
| Agent | The AI model that reads the request, picks the tools and skills, and writes the answer | The assistant a user talks to | Evals: each case run three times, the worst run counts |
| Harness | The software that runs the agent: what it can reach, remember and spend, and what's checked before anything leaves | Loading the right skills for a request; validating a ticket before anyone sees it | Its own checks are tested directly; the rest shows up in every agent eval |
| Workflow | A job a client sets up on the platform: what runs, when, on what, and by whose rules | A morning note on every name in a client's watchlist; a scheduled report built on the client's own data and analysis criteria | Against the client's own rules (section 7) |
2. Why existing benchmarks aren't enough
If you want to know whether an AI is good at finance, the obvious place to look is a finance benchmark. The good ones test real skills, like finding the right reporting period, reconciling accounting bases and calculating accurately, and our system needs all of them.
But most start from a question whose answer is already known: an expert writes down the right figure, and the test checks whether the model finds it. Forecasting and trading benchmarks wait for real outcomes. None of them grade the judgment in everyday analysis: which risks the system raises, which explanation it offers, and which important issue it leaves out.
| Answer-key benchmarks | What our users need graded | |
|---|---|---|
| Who writes the question | An expert, the benchmark, or a prediction market | The user, or the system itself |
| When the right answer exists | Before the question is asked | Often only later. Sometimes a definitive answer is unknowable |
| What gets graded | Did the model find the known answer? | Was each claim supported when it was made, and was it right once the outcome came in? |
| Typical question | "What was operating margin last quarter?" | "Why is this stock moving?" "What would you do?" |
For our clients, those choices are the product. Grading them takes a record of what the system knew, what it said, and what the user needed.
3. Start with a written promise
When an AI system gets something wrong, the first question is: wrong compared with what? Without a written answer, every failure turns into an argument.
So every tool and every skill starts with a contract: a numbered, signed-off list of what it should do. Each behavior lists its edge cases and its near miss, a request that looks similar but must not trigger it. "Build me a trade" should produce a trade ticket; "explain the trade" should not. Every test traces back to one numbered behavior, so when something fails, we know exactly which promise broke.
Here is part of a real one, simplified: the contract for a trade thesis, the reasoning every trade idea has to state before any ticket is built.
per name 1 to 3 ideas. Direction first; a volatility or skew idea
only where the evidence shows one.
each idea a one-sentence claim, the drivers behind it, and at least
one condition that would prove it wrong.
conviction one of four levels. The number behind each level is set
in code, never written by the model, so it means the same
thing on every run.
anchor agrees with the desk's overall read (bullish at 58 or
above, bearish at 42 or below). Disagreeing means naming
the evidence, and caps conviction at "tentative".
no trade a thesis with no tradeable expression is a valid answer.
checked by thesis present and linked to its tickets; agrees with the
overall read; the ticket's structure can express it.The anchor rule exists because of a real run: the desk's overall read scored 59, mildly bullish, and the thesis built on it came out bearish. Each ticket then has its own contract on top. It must pass validation, its option strikes must exist in the listed chain, its worst case must be stated, and its prices, breakevens and losses are computed in code rather than written by the model. About two dozen checks gate it in our evals.
A contract describes the result a user can rely on, not the route to it. We prescribe a step only when the step is itself the promise, such as asking for confirmation before an irreversible action. That leaves engineers free to rebuild the system without moving the promise.
4. Why one good run proves little
A tool is easy to test: call it with chosen inputs and compare the result. An agent is harder, because it picks its own steps and can answer the same request differently twice. So we test at three levels, and each catches failures the others miss.
| Layer 1: tool tests | Layer 2: single-request evals | Layer 3: conversation evals | |
|---|---|---|---|
| What runs | One tool, with inputs the test chooses | The real agent, on one request | The real agent, over a conversation with scripted follow-ups |
| What's being tested | One tool, on its own | The agent: model and harness, on one request | The agent's memory: whether the harness carries the state across turns |
| Question it answers | Does the tool do everything it promises, including at the edges? | Does the agent turn the request into the right result, and report it honestly? | Does the agent keep facts and intent straight across turns, and does the finished result work? |
| How often it runs | On every code change that touches the tool, in each market phase it serves; three times if the tool has an AI model inside | Three times; the worst run is the verdict | Three times; the worst run is the verdict |
| Typical failure it catches | A behavior the tool should show but doesn't | The agent does something nobody asked for, quotes a number the data doesn't support, or claims a success that didn't happen | A follow-up acts on the wrong thing, or contradicts an earlier answer |
For the agent, a person writes the expectations before it runs. We run each case three times, and the worst run is the verdict: two good runs out of three is a failed case.
Time is part of every test, because in markets the right answer depends on when you ask. A price tool behaves differently in regular trading, after hours and over a weekend, and an international stock follows its own exchange's calendar. So tests run in each market phase they cover, and part of the agent's grade is whether it knew which session applied, for the user and for the market, and said so.
Even three runs is a modest bar. A case that fails one time in ten will still pass three runs in a row about 73% of the time. Three passes is a useful screen, not proof that rare failures are gone. Evals also expose gaps in the contract: a failure that breaks no written rule tells us a rule is missing.
5. The floor every run must clear
A high average can hide a disaster. An answer that is excellent in every other way but shows a client someone else's data is not a good answer. So before anything is scored, every run of every case has to clear nine checks. A check that doesn't apply to a case is marked not applicable.
| # | Group | Check | The question it asks |
|---|---|---|---|
| 1 | Execution | Entitlement | Is the person entitled to this data and this action, and did they ask for it? |
| 2 | Execution | Inputs | Were inputs correct, and were invalid inputs handled and disclosed? |
| 3 | Execution | Right result | Did the intended thing happen, with the right values? |
| 4 | Containment | Side effects | Did anything happen that wasn't asked for? |
| 5 | Reliability | Repeats and failures | If it runs twice or fails halfway, is the result still right, and is the gap named? |
| 6 | Reliability | Works when used | Does the result work when used? The ticket validates, the chart plots, the schedule fires |
| 7 | Disclosure | Honest reporting | Does what the answer says match what actually happened? |
| 8 | Disclosure | Time | Is every number right for its moment and market phase, and labeled with its time? |
| 9 | Disclosure | Attribution | Does each important claim rest on a citation that supports it? |
Most of these are facts, like a saved schedule time or a quoted price, so code checks them. Only open-ended interpretation needs a rubric. Each check ends in one of five verdicts:
| Verdict | What it means |
|---|---|
| Pass | The expectation was met |
| Fail | The system did something its contract doesn't allow |
| Not applicable | The check doesn't apply to this case |
| Unproven | The check applies, but we couldn't collect the evidence. Never counted as a pass |
| Unresolved | We measured it, but the evidence can't settle it yet, like a forecast before its horizon |
A serious failure, like showing someone data they aren't entitled to, fails the whole run however well everything else scored. Things that vary by nature, like how often forecasts come true, are compared as rates across many cases instead. And nothing an AI grader decides can block a release on its own.
The nine checks are the floor, not the whole evaluation. Above them we measure how often runs agree, cost, speed and the quality of the work, whether forecasts held up, and how the system copes with inputs built to mislead it. The full rules are in What gets a verdict (appendix).
6. Some answers can't be graded yet
Let's go back to the falling stock. If the system says the outlook cut drove the selloff, when can anyone tell whether it was right? Maybe that afternoon, but perhaps never with certainty. Different claims settle at different times, so the record has to hold each one until its evidence arrives.
Everything goes into the same record: what ran, what it produced, the evidence it used, the verdict and who gave it, and the versions it ran with. What differs is when a verdict becomes possible:
| What we evaluate | Example | What checks it | When it settles |
|---|---|---|---|
| A factual claim | "Operating margin was 18%." | The right filing, period and accounting basis | Now |
| A workflow requirement | "Use only the approved sources"; "a trade idea ends as an executable ticket" | The evidence the system retrieved and a log of what it did | Now |
| Operational performance | "The workflow finishes inside its time limit" | Measured runs, failures included | Now, continuously |
| A forecast | "Management will lower its margin outlook"; "the stock closes 2–4% lower" | The later disclosure or price | Later |
| An explanation | "The outlook cut drove the selloff" | Evidence for the cause, weighed against other possible causes | Later. Sometimes a definitive answer is unknowable |
For anything that reports on a company, we grade two things separately. The first is what it said: was each claim supported, specific, properly qualified and, later, right? The second is what it should have said: did it cover the issues that a reviewer, working without seeing our answer, flagged as important?
A system can write ten true sentences and miss the one thing that mattered. Grading only what it chose to say would reward exactly that.
We check "supported" in code wherever we can. A claim's citation has to lead to a document that actually backs it up. A citation that stops at a document's name, or points only to something our own system wrote earlier, fails. We're building that check now, and today it's partly manual.
Who grades the grader?
Open-ended work, like an explanation, often needs an AI grader, and then the grader's mistakes become part of the measurement. So each grader gets tests of its own: known-bad and known-good examples, answers built to fool it, and real answers graded by people. Our explanation grader caught all six known-bad explanations and passed all three good ones, then on real answers treated true details missing from our reference as made up. We fixed that and kept the examples passing. Until a grader agrees with people on real answers, its verdicts are advisory. Case Study 2 shows why passing a synthetic control isn't enough.
7. Each client's own rules
Every firm we work with has rules of its own: which sources it trusts, what its advisors may recommend, what every note must carry. Those rules matter, but they can't be allowed to weaken the standard underneath. That standard, our house criteria, is the same for everyone: get the company and the time right, calculate correctly, back claims with evidence, cover what matters, respect who may see what, and answer fast enough.
Take a fictional broker with three rules:
Research notes use only its approved news sources.
Its wealth advisors never receive options trade ideas.
Every client note carries its legal disclaimer and arrives within 25 seconds.
We write those down as a customer profile that sits on top of the house criteria, with its own test cases and its own pass or fail. It can add requirements, but it can never remove ours.
Here's how that changes a verdict. A wealth advisor at the broker asks, "Give me a way to play a rate hike." Under the house criteria, any validated trade ticket passes. Under the broker's profile, a ticket with options in it fails. The rule isn't only tested, it's enforced: the options tools are removed before the agent starts, and any options ticket that still appears is rejected after validation. Then the same advisor asks, "Explain how options are positioned ahead of the earnings report." That passes, because describing how other traders are positioned isn't recommending an options trade. That's the near miss, and the profile tests it on purpose.
A client's risk or compliance team can see the record for their own work, never another client's. And clients who build their own tools on the platform test them with the same framework we use.
8. Case studies
Rules are easy to write. These five cases show what happened when we applied them to work the platform actually produced: what we saw, the evidence behind it, the requirement it tested and what we changed.
Grade what the system said at the time. Twenty-five cases pair a recorded intraday note with a later explanation. They show why company news, sector context and the time of the evidence all matter. The workflows and prompts differ, so the comparison cannot isolate hindsight.
The grader needs its own tests. Ours passed an answer that called a disclosed figure "not disclosed," even after we added a control for that exact mistake.
Write the requirement down, then measure it on both versions. A new harness cut median cost to about a fifth, while ticket completion fell from 27 of 30 runs to 19 of 30. After the fix, it matched the old harness at 27 of 30.
Register a forecast before the outcome. A call counts only if it was saved first. GE Aerospace's price call settled right, but most of its business expectations couldn't be scored, because they weren't stated in the company's own terms.
The client's rules come first, up to a line. A client's persona outranks our house defaults, but never the numbers, what each user is entitled to see, or anything material to their decision.
The companion, Case Studies: what the record caught, has the full accounts and their supporting records.
9. How we improve, and what we don't claim
A failure is worth something if it leaves a test behind. We record each one the way a user would describe it, note which written behavior it broke (or add the behavior if none covers it), and build a case that guards against it. Then the fix goes through the same path every time: find the cause, make a small change, test it on development cases, check it on cases we didn't tune on, decide whether to ship, and keep measuring.
A failure counts as closed only when its guarding case failed before the fix and passes after it. Reports lead with the failures still open, so a rising pass rate can't hide an unresolved problem.
Two rules keep the numbers honest. Nothing missing is ever green: a check that couldn't run is "unproven," never "pass," and a missing run, case or cost is reported as missing, with its denominator. Comparisons show counts, not labels: "27 of 30 → 19 of 30" is the information, and "pass → fail" hides it. When the cases can't tell two versions apart, we say "not established," never "equal."
Replaying old cases tells us whether a fix broke something that used to work. It can't prove forecasting skill: that takes predictions saved before their outcomes, scored against stated baselines across enough events to rule out luck.
What we don't claim
A pass is evidence for the cases and runs we tested, not a guarantee. A good grade for an explanation is evidence for it, not proof of a cause. The explanation studies are early: small, selected samples, AI grades and different workflows, and the intraday audit still has to separate what was knowable at the time from full-day outcomes. We publish those limits alongside the findings because they decide what the results can support.
What ships, and what's next
Next week, we're releasing Kit 0.1 under the Apache 2.0 open-source license. It's small on purpose: the record format, checks you can run on a laptop, a comparison that refuses to call a missing case a pass, the trade-ticket requirement and the fictional broker's profile with synthetic data. More fixtures follow in Kit 0.2 in December.
Some of this needs a note of its own. Future engineering notes will cover how we test our graders, how we compare models and versions, how a change ships, and adversarial testing.
The platform will keep improving. We'll keep publishing what we tested, where it fell short and whether the next version did better, so you can judge the progress with us.