AgentSmyth builds decision intelligence for financial services. We help people make sense of companies, markets and the news that changes them. The Lab asks a harder question: how often does that work hold up when the answer arrives?
We are starting with this earnings season. Before a company reports, the AgentSmyth Earnings Agent records what it expects from the business and how it thinks the shares will react. Afterward, we compare those expectations with the results, management's outlook and the market. We will publish that record, including the mistakes and the companies we failed to cover.
Mark it to market means keeping the original judgment answerable. A convincing explanation afterward cannot replace what we said before the news.
Why start with earnings?
Earnings gives the test a natural sequence. A company has an existing outlook. Investors have expectations. Then the company reports results, updates guidance and answers questions. The shares get a full trading session to respond.
That lets us ask several concrete questions about the same decision. Did we identify the important uncertainty? Did we expect the right disclosure? Did we predict the stock's direction? When the answer arrived, did we explain what changed without rewriting our earlier view?
The Earnings Agent makes its business questions, key performance indicators and expectations for results and guidance explicit. Its later assessment returns to the original questions. Did growth turn into profit? Did higher investment change the cash outlook? What did management say that changes the next quarter? We want to know whether that structure leads to better judgments, as well as a clearer record of them.
What we will share
We will share scores, business questions, KPIs, guidance expectations and outcomes, with grades, dates and source references. Readers will be able to see an expectation beside its answer. Our review will also check the complete saved assessment for contradictions that a tidy table might conceal.
The release consists of structured records, rather than full reports, prompts or our proprietary weighting recipe. Public source links point to company disclosures.
Our universe covers U.S. companies in the S&P 500 or with a market capitalization of at least $10 billion, supported through their 10-K and 10-Q filings. Foreign private issuers reporting on forms such as 20-F or 40-F are outside our current coverage. The scorecard will distinguish companies expected to report from forecasts actually delivered. Missing coverage stays in the count. The measurement rubric defines eligibility and exclusions.
How to read the scores
Each assessment produces a bullish, bearish or neutral call. We will track two scores: the assessment score, which combines the saved business judgments, and the evidence-adjusted score, which moderates that view when a qualifying guidance comparison is missing. Both run from 0 to 100. Higher is more bullish, but a score of 70 is not a 70% chance of a gain.
Showing both lets readers see how the adjustment affects the strength of a call and how often the system takes a directional view.
Settle the stock call
We grade the pre-release call after the stock's first full reaction session. For a before-open release, that ends at the same day's close. For an after-close release, it ends at the next trading day's close. The starting point is the close before the news.
Accuracy and coverage belong together. How often were our up-or-down calls right, and how much of the season did we make a call on? Neutral calls remain visible as abstentions; they do not become wins because a stock barely moved. We will show bullish and bearish results separately and test whether stronger scores correspond to stronger reactions.
We will also check whether a call captured a meaningful move for that company and how it performed relative to the broad market and its sector. The rubric holds the exact score bands, calculations and rules for missing or late forecasts.
What we compare against
Our baseline is the earnings system we used before this redesign. It already produces scheduled assessments before companies report. It brings together financial performance, management guidance and company developments into a view of the release. In the cases we reviewed, it already identified important mechanisms such as cash conversion, pricing pressure and the cost of growth.
The Earnings Agent organizes that analysis around explicit business questions, links them to KPIs and stated expectations, and returns to the original questions after the release. This makes it easier to distinguish what we expected, what changed and what remains unanswered. Whether it also produces better judgment is what we want to test.
This is a demanding baseline because it already performs the work we want to improve. We compare the systems on the same releases, at comparable stages. A useful insight in a paragraph receives the same credit as one in a structured field. Simple directional rules provide a separate check on the stock calls.
What the comparison looks like
Meta and Equifax show why we keep the price call and the business assessment separate. These selected examples illustrate the comparison, not season-wide superiority.
| Company · reaction session | Assessment score | Evidence-adjusted score | Stock reaction | Comparison call |
|---|---|---|---|---|
| Meta · October 22 | 35 | 44 | −6.98% | Bearish; correct |
| Equifax · October 20 | 31 | 31 | +11.33% | Bearish; wrong |
Meta: clearer commitments, familiar risks
Both systems expected a modest revenue beat and identified the tension between growth, investment and cash generation. The baseline's financial assessment anticipated a 4–6% fall. The Earnings Agent separated advertising, investment affordability and the expense outlook into explicit questions. Both got the direction right. The Earnings Agent's format made its commitments easier to revisit; it did not discover a risk the baseline had missed.
Equifax: the same earnings setup did not settle the shares
Both systems' guidance assessments expected roughly $2.23 adjusted EPS, $1.70 billion revenue and annual guidance to hold. Equifax reported slightly higher EPS, slightly lower revenue and reduced annual revenue and EPS midpoints, but strengthened its annual margin-expansion outlook excluding FICO mortgage royalties. The shares rose sharply.
The Earnings Agent's bearish call missed. The baseline abstained under its own rules and had considered a relief rally. Both had identified relevant business issues; that did not settle how investors would weigh them.
The paired examples and evidence notes show what each system expected, the price calculations and the limits of the comparison. United Rentals and CBRE add two other lessons: a correct price call with a wrong investment expectation, and conflicting expectations within one assessment.
Beyond the stock call: six checks on intelligence
In addition to the directional-accuracy scorecard, we will grade the business analysis behind the calls. Did we identify the right uncertainty, anticipate management's outlook and explain what changed? These grades will be reported separately from whether the shares went up or down.
First, build the answer keys from the sources. An automated process reads company releases, filings and full call transcripts without seeing either system's assessment. It separates what was knowable before the release from what was disclosed afterward. Dated news coverage provides separate evidence about how investors received the release.
Then compare the two systems on the same terms. We extract their original statements into a common format without giving the extractor the answer keys or later outcomes, preserving qualifications and contradictions. A useful expectation in the baseline report's prose gets the same treatment as one in an Earnings Agent field. Automated graders compare those statements against the source-backed keys, with system names hidden. Comparable numerical errors are calculated directly.
Finally, check the grading itself. Humans review a sample of the answer keys, extracted statements and grades. The first automation pilot is complete; large-scale grading and human validation are still ahead. The rubric explains the checks and the limits of masking reports whose original formats differ.
We will report six separate results:
Choose the important questions
Did we address the important uncertainties that were visible before the release, using measures that could answer them? We count both relevant choices and omissions. Mentioning a topic is not the same as asking a useful question.
Make a checkable commitment
Did we say what we expected, clearly enough to check it later? A question alone is not a forecast, and a later grader cannot make a vague statement more precise.
Anticipate the disclosure
How close were the expected figures, and did management raise, hold or cut the outlook as anticipated? We compare the same measure, period and accounting basis. A correct stock call cannot erase a wrong financial expectation.
Find what changed
Did the post-release assessment capture the material changes, including surprises, and distinguish them from information already known?
Support the implications
Did it explain how a disclosure affects the business and its outlook? When explaining the market's response, does contemporaneous evidence support that account? A falling price alone cannot prove that guidance caused the fall.
Answer the original expectation
Did it return to what we had actually expected? We keep the original beside the answer, including unresolved questions, contradictions and changes of view.
The target is every eligible event with enough evidence to grade, with incomplete cases still visible. We will publish these business-quality results as their checks are completed. The independently settled market scorecard can proceed while that work continues.
What comes next
Today we are publishing the plan, rubric and worked examples. Our first Engineering Note follows on November 9: how we evaluate intelligence before shipping, from shared capabilities and complete workflows to fictional customer requirements.
The interim earnings scorecard and Kit 0.1 arrive together on November 16. The kit will include public or synthetic evidence, saved example answers and runnable checks, including an earnings scorecard example. Readers can inspect the results alongside the tools used to grade them.
The season scorecard is scheduled for December 14, covering eligible releases through early December and their first full reaction session. The Lab page keeps the release schedule, and the rubric specifies the exact cutoff.
The reader should leave with a record they can question: what we expected, what happened, how it was graded and what the mistake changes next.