We're launching AgentSmyth Lab, and I'll be leading it.
When you're building AI for financial decisions, you owe people a record: what the system said, what it knew at the time, what happened next, and whether it did better than a simple baseline.
Finance marks books, funds and traders to market. But the intelligence behind those decisions rarely faces the same scrutiny. And now AI can produce thousands of confident-sounding notes before breakfast.
Plausible is free. Graded is not.
We’re starting with the earnings season. Can our agents identify the questions that actually matter to a business? Can they anticipate the results and guidance? And can they predict how the market will react?
We'll check the facts and business judgments separately from the price call. Getting the direction right doesn't excuse a wrong judgement. A useful business question can still come with a bad forecast. And the original forecast stays in the record: a convincing explanation afterward doesn't replace it.
Three commitments:
- Rules before new tests. Date and version the rubric so readers can see which rules applied.
- Cost and time. Show what it took to produce the answer.
- Misses on a schedule. Publish the misses and what we learn from them, alongside the hits, neutral calls and gaps in coverage.
And compare against a baseline. If we aren't better, you'll hear it from us first.
The website, this video and our first Lab Note launch together today, with the rubric. The interim earnings scorecard follows November 16, and the full season scorecard December 14.
Pulkit, Andrew and I talk through why we're doing this in the video.
Mark it to market.