The principles in Engineering Note 1 matter most when a result looks convincing. An explanation can sound persuasive after a market move while revealing little about the analysis available during it. An answer can appear careful while overlooking a disclosed figure. A cheaper harness can complete its runs while leaving users without the trade tickets they requested.
These five case studies examine those situations through the platform's recorded work. Each connects the observation to the requirement being tested, the response and the evidence still needed. The final case considers a different question: how to adapt analysis to a client's priorities while preserving standards that apply to everyone.
Case Study 1 · Hindsight
Grade what the system said at the time
At 10:35 on the morning of November 5, the stock market was falling hard: the S&P 500 (SPY) was down about 2.5% and the Nasdaq 100 (QQQ) about 4.5%. Sherwin-Williams was down 1.5%, holding up better than the market. Our live note called it a pullback after a sharp two-session rebound and added that the valuation looked elevated. It never mentioned the selloff, so a reader could easily have mistaken a market-wide drop for a problem with the company.
Asked afterward, the conversational agent led with the selloff. SPY closed down 4.4%, and the paint makers PPG, RPM and Axalta each fell about 3%. It treated Sherwin-Williams's decline as part of that rout, partly reversing a 6.6% gain the day before, and found no new company announcement to explain it. The later answer was more useful for that question. It still could not tell us what the platform had understood while the move was happening.
We wanted to examine that distinction beyond a few memorable examples. We selected a set of 25 ticker-day cases without inspecting their answers or grades. The selection spans twenty trading sessions from October 12 to November 6, with a different ticker in every case. For each, we preserved the first regular-session note and requested one fresh explanation afterward: 25 pairs for a person to review.
The wider set gives a more varied picture. Salesforce's live note identified the Bank of America upgrade and the Loops acquisition announcement; both were already in the news. Sherwin-Williams's note described the decline without mentioning the market-wide selloff. PSEG illustrates a different problem: a correct description of the morning can look wrong if it is judged against the close.
| Case | Recorded during the session | What the comparison adds |
|---|---|---|
| Salesforce · October 12 | The note named the upgrade and acquisition news. | By the close, the later answer could also compare the gain with a software sector that had turned negative. The live note had already identified the company catalysts. |
| PSEG · November 2 | The note described an earnings miss and a decline of about 2.4% at 9:45 am Eastern. | The stock recovered to finish slightly higher. That recovery does not invalidate the earlier price or the news available then. |
| Sherwin-Williams · November 5 | The note called a 1.5% decline a pullback after a two-session rebound. | SPY was already down 2.5% and QQQ 4.5%. The stock was falling less than the market, and the note never mentioned the selloff. |
These cases also change how we read the language. A note that describes options activity alongside a price move has not necessarily claimed that the options caused it. A missing explanation, an unsupported causal claim and a factual contradiction are different findings. Collapsing them into one failure count would hide what needs to improve.
The principle
A convincing explanation in hindsight does not establish that the system was useful as events unfolded. We need the analysis it actually produced, the time it produced it, and the evidence available then. Later outcomes tell us how the story developed; they cannot be substituted for what was knowable earlier.
The immediate requirement is for each note to check the company news against an appropriate market or peer comparison, and to show that comparison when it changes the interpretation, while allowing it to say when no clear company-specific cause is established. The evaluation has to be equally careful: price co-movement can support an explanation without proving a unique cause.
Review status. All 25 new answers and their source records are collected. We are not publishing a pass rate. The archived ticker notes and new conversational answers come from different workflows, versions and prompts. In particular, the new question explicitly asks for sector and market comparisons. This study exposes differences worth investigating; it does not isolate the effect of hindsight or measure an overall platform error rate.
Selection, evidence and the earlier pilot
Blind selection. A seeded, reproducible rule chose one retained ticker-day record per session, plus a second on five sessions selected by the same rule. It excluded tickers already examined in the development studies and repeated tickers within this sample. We froze the selection before retrieving the answer bodies. These are not necessarily large moves, and conditioning on an existing note means the study cannot measure how often the platform failed to write one.
Hindsight window. The new answers were requested on November 7, from roughly one day to almost four weeks after the session. The November cases already had closing prices and published reporting available, but the latest case has a shorter window for further explanation to emerge. We assess the evidence available afterward without assuming that every move has acquired a settled cause; an unresolved attribution remains a valid result.
Evidence boundary. We assembled price and headline references before reading the selected answers. For the automated contemporaneous assessment, we supplied only bars ending by the note's timestamp and headlines published by then. Later outcomes were withheld from that assessment. Retained market data may include corrections, and publication time does not establish when the platform ingested or retrieved a source. This is a reconstruction from retained evidence, not a certified replay of every historical input.
Execution. On November 7, one current-dev conversational harness produced one answer per event, with fresh conversation state and an explicitly dated question. All 25 answers completed, with a median run time of about 27 seconds. Priced usage totals $11.87, but three requests lack model and cost metadata, so the total cost is incomplete. These are collection diagnostics, not quality or latency benchmarks. Local answers and source captures are retained, together with experiment traces. Auxiliary workspace-snapshot uploads failed because the process selected stale default credentials, so the collection is not a clean persistence test.
The grader remains under review. Five synthetic controls passed, but a real case exposed a false contradiction: the grader treated a difference between a note's price snapshot and the final minute bar as proof the quote was wrong. It also sometimes combined separate excerpts into a purported quotation. In another case, it called a citation fabricated because the source was missing from the grader's reference packet, even though it existed in the saved run. We retained those outputs for audit and did not accept their categories as a performance score.
Why we expanded the study. The earlier retrospective pilot covered eleven selected moves and produced 33 answers across two harnesses. An initial AI review passed 32; a subsequent rubric disagreed with many of those judgments. A separate audit covered 37 stock-day records and 70 note versions from two sector selloffs. It raised useful concerns about missing context, but its grader also received closing returns when assessing intraday notes. Neither exercise supplies an accepted benchmark score. The 25-case follow-up broadens the cases and separates the evidence times.
Source: retained ticker notes, timestamped market and news references, and the November 7 explanation study.
Case Study 2 · Grader blind spots
The grader needs its own tests
After a company reports earnings, our platform answers questions about the results. An October engineering replay of JPMorgan's archived sources exposed a failure in that process. The question concerned the quarterly credit card charge-off rate: the share of card balances the bank wrote off as unlikely to be repaid.
The answer said the rate hadn't been disclosed. The filing reported 2.91%. The step that read the documents had been handed a summary of the earnings call, not the filing, and the step that wrote the answer turned "not mentioned on this call" into "not disclosed." Our automated grader then passed it. We then added a control: a made-up answer with the same mistake. The grader caught the made-up one and still passed the real one, borrowing a qualification that only appeared in the source.
answer the card charge-off rate "was not disclosed"
filing card charge-off rate: 2.91%
grader passThe principle
A grader needs to be tested against the failures it will encounter in use. Examples written by its author can verify that a rule works as intended, while still missing the way a real answer evades it. Both belong in the evaluation.
How we fixed it. First we fixed the step that writes the answer. It now receives the filing itself, not only a summary of the call, and it tells "not in what I read" apart from "not disclosed." The corrected answer reports the rate improving to 2.91% from 3.34%. Both false passes are now permanent regression cases, and automated passes on that check stay advisory until a person compares the answer with the filing.
The record behind this case study
| Kind | factual claim |
| What ran | an engineering replay of the earnings desk's JPMorgan answer using archived sources (October 2026), rather than a recorded live customer response |
| The claim | the quarterly card charge-off rate "was not disclosed" |
| The evidence | the October 13 earnings 8-K reports 2.91% |
| Why it happened | the reading step was handed a transcript summary without the filing; the writing step turned "not mentioned in this transcript" into "not disclosed" |
| Check that should have caught it | honest reporting (the number) and attribution (the claim rests on a transcript, which can't support a claim about the filing) |
| What the grader did | passed it. With a synthetic control added (same mistake, made up), the grader caught the control and still passed the real answer, borrowing a qualification that appeared only in the source |
| What we changed | both false passes kept as regression cases; automated passes on this check are advisory until a person compares the answer with the filing |
The corrected answer has source receipts for both figures. The retained validation report also records a separate limit: the complete evaluator packet exceeded its 300,000-character budget, so that final check returned Unverified. A corrected answer and a completed automated evaluation are distinct claims.
Source: our earnings desk's validation review, October 20, 2026, and JPMorgan's October 13 earnings filing.
Case Study 3 · Cost and completeness
Write the requirement down, then measure it on both versions
When a user asks for a trade idea, the useful output is a thesis, the reasoning, and a trade ticket that expresses it: an order someone could actually place, priced from quotes fetched in that run, with its worst case stated. We moved our main agent to a new harness, the code that runs it.
A user asked why trade ideas had stopped arriving as tickets. The new harness had no step that published a ticket. It also capped the agent at six skills per turn, and two near-duplicate formatting skills had pushed the ticket-building skill out of the six.
We restored ticket publishing and made the ticket skill load alongside the trade skill. We then compared the two harnesses on the same ten trade requests, collecting 30 runs from each. The restored path still did not reliably produce the requested output:
| Harness | Runs that produced a ticket | Median cost per run | Median time |
|---|---|---|---|
| Old | 27 of 30 | $5.09 | 167 s |
| New, after publishing was restored | 19 of 30 | $0.94 | 75 s |
| New, after the instruction fixes | 27 of 30 | $1.22 | 130 s |
The new harness cost about a fifth as much and ran in under half the time, but produced fewer tickets. In the retained records, the misses were runs in which the agent did not write one; they were not tickets rejected by validation. Ticket presence is only one part of quality: the price evidence and stated risk still need their own checks.
The principle
Cost and speed need to be reported alongside completion of the task. The cheaper harness looked attractive until we measured whether users still received the output they had requested. Writing that requirement down made the tradeoff visible on both versions.
What changed next. With the publishing path working, the remaining misses pointed to instructions that let the agent stop at a written recommendation. We reworded four of them, shipped the change and ran the ten requests on the new harness again, three runs each. It produced a ticket in 27 of 30 runs, matching the old harness's earlier result, at about a quarter of its median cost.
All three remaining misses are the same request, and the old harness missed it in all three of its runs too. It asks for a long, witty market report that ends by ranking three ideas by their flows. Both harnesses wrote the report and a ranked list, without tickets: the agent judged the rebound case too tentative to commit to, and our instructions allow it to stand down when conviction is low. That is a defensible answer. Our test case still asks for tickets, so we count these as misses, and we are revisiting whether a request like this should require them at all.
Completion still isn't quality. On one other request, the tickets were hedges and did not state their maximum loss; a fix for that is in review. An earlier follow-up also caught a mistake in our own check: every run produced tickets and every run failed the stated-maximum-loss check, but the tickets were unsized and stated their worst case as a stop. We fixed the check, not the agent.
The record behind this case study
| Kind | workflow requirement |
| The requirement | a request for a trade ends in a validated, executable ticket whose prices come from this run's quotes and whose worst case is stated |
| How it was found | a user's thread on the new harness produced a ranked list of local shares and no ticket |
| Cases | 10 trading cases and 30 recorded runs per harness. An interrupted run and resumption left the old harness with two trials on one case and four on another; the other eight had three. The new harness had three per case. A pricing question that must not produce a ticket remains a separate near-miss requirement |
| Result | old harness 27/30 trials with a ticket; new harness 19/30, at a median $0.94 vs $5.09 per run and 75 s vs 167 s. After the instruction fixes, the new harness reached 27/30 at a median $1.22 per run and 130 s |
| What the misses were | every miss was the agent not writing the ticket; no ticket it wrote was rejected by validation |
| Causes traced | no ticket-publishing step; and skill crowding: the six-skill cap plus two near-duplicate format skills ranked the ticket-construction skill seventh on one trade turn |
| What we changed | first restored publishing, validation and the companion skill rule; then measured 19/30; then reworded four instructions, shipped the change and reran all ten cases on the new harness: 27/30. The old harness was not rerun; its 27/30 is the October 29 result. One earnings case was moved to a company with its report still ahead, because the original company had already reported. All three misses are one ranked-ideas request that the old harness also missed in all three runs. We also corrected the maximum-loss check, which wrongly failed unsized tickets that stated a stop |
| What we didn't change | whether a request for a ranked list of ideas needs a ticket: a decision about what we want, recorded as open, not a bug. The rerun's 130 s median is inflated: most of its runs ran seven at a time, and the cases run two at a time were faster than on October 29 |
Source: our regression audit of the new harness, October 29, 2026, the fix that followed, and the rerun of all ten cases on November 9.
Case Study 4 · Forecast settlement
Register a forecast before the outcome
Our earnings desk saves calls before the release, with a timestamp: a score for which way the stock will react, and the business results it expects. GE Aerospace reported before the market opened on October 20. Its call was saved at 3:39 pm Eastern the afternoon before, with a score of 35 out of 100, which our rubric counts as bearish.
The price call settled cleanly. The stock fell 3.7% that day, 4.0 points worse than the S&P 500, so the call was right. The business half mostly didn't settle. Of the four results the call set expectations for, earnings per share settled, and the forecast was too low: $2.25 against the $2.01 the call expected. The other three couldn't be scored. Two compared a full-year or second-half target with a single quarter's figure, and one had no number at all. The system left them unscored rather than force a verdict.
Across six settled calls we checked, 17 of 20 such comparisons ended the same way. Not every call lands: Morgan Stanley's, saved before its October 14 release, settled wrong, and it counts in the same tally. When a company has more than one saved call before its release, the last one counts.
call saved Oct 19, 3:39 pm ET direction 35 of 100 (bearish)
price GE -3.7%, S&P 500 +0.3% 4.0 points worse right
EPS bar $2.01 reported $2.25 above
engine profit bar $10.25-10.35B (year) reported $2.89B (Q3) can't settle
engine services bar low double digits (H2) reported 17% (Q3) can't settle
engine margin no number in the call reported 27.9% can't settleThe principle
A forecast must be registered before the outcome and stated in terms that can eventually be scored. For earnings, the business forecast and the price call answer different questions, so we grade them separately against simple baselines. Misses and no-calls stay in the same table, and repeated answers are also assessed for consistency.
What we're changing. A business forecast needs a clear settlement rule to be useful in this evaluation. Each expectation in the call will name the period and basis the company reports on, so it can be scored mechanically when the numbers arrive. Until then, comparisons we can't settle stay unscored and are counted, not dropped.
We also track how often a call makes business expectations that can't be settled as written. That rate is one of the improvement targets for the next version of our earnings agent, which goes live next earnings season. A call that can't be checked counts against us; it doesn't quietly drop out of the record.
The record behind this case study
| Kind | forecast |
| The call | GE Aerospace, ahead of its October 20 pre-market release; saved October 19 at 3:39 pm Eastern; direction score 35 of 100, bearish under the rubric |
| The price call | the stock fell 3.7% on the day, 4.0 points worse than the S&P 500: right |
| The business forecasts | four expectations. Earnings per share settled above the bar ($2.25 against $2.01). Engine profit, engine services growth and engine margin were left unscored: two set a full-year or second-half bar against one quarter's figure, and one had no number |
| Across six settled calls | 17 of 20 business comparisons left unscored for the same reasons. Morgan Stanley's call settled wrong on price and sits in the same table |
| What we're changing | each expectation names the period and basis the company reports on, so it can be settled mechanically. The rate of expectations that can't settle is tracked as a target for the next version of the earnings agent |
Source: our earnings desk's settlement records for releases from October 13 to 23.
Case Study 5 · Client rules
The client's rules come first, up to a line
An illustrative example. The profiles, the broker and the rules in this case study are made up. They don't describe any real client, and they don't reflect any client's actual settings or constraints.
Clients shape the platform with a persona: how answers should sound, what matters most to them, and what to leave out. The platform also has default choices about emphasis and presentation. By default the platform states a lean and a conviction level, and when options are expensive it prefers spreads, which cap the cost of a trade. A client may reasonably want something different. Consider two fictional profiles: a trading desk that rules out spreads and prefers to size down, and a research desk that wants no recommendation language or price targets, with missing data flagged explicitly.
These are legitimate differences in how clients use our platform. The platform should respect them while retaining information material to the decision. A request to omit macro commentary cannot justify hiding a rate shock relevant to the question, and a different persona cannot change the value of a reported figure.
One retained test isolates how the agent synthesizes evidence. It supplies a fictional company with deliberately conflicting signals: a breakout and squeeze setup on the chart against slowing growth and a stretched valuation. With that same evidence already in the conversation, it asks the same question under two personas:
short-term trader long-term investor
horizon 1 to 10 days 1 year or more
call lean in, half size stay cautious
leads with breakout, positioning growth, margins, valuation
keeps as caveat the weak fundamentals the short-term setup
evidence the same the sameThe trader's answer favors a tactical position; the investor's favors caution. Both retain the opposing evidence. An AI grader scored each 5 out of 5 for persona weighting and evidence preservation. A word-sequence similarity check also distinguished the pair and correctly recognized an identical-answer control. Those results describe one synthetic pair on the earlier harness; they do not establish that every persona preserves every fact, or that the retrieval process found all relevant evidence.
The principle
Facts, entitlements and permitted trade structures remain constraints. Within those boundaries, the current request guides the answer, followed by the client's persona and our house defaults. The analysis should reflect the client's priorities while keeping material evidence visible.
What the broader test must establish. The same reported quantity must retain its value across personas. Material information must remain visible even when a persona normally excludes the topic. The synthesis example above does not demonstrate either requirement comprehensively. They need explicit cases and checks before we can describe them as validated safeguards. Trade constraints have an additional enforcement path: a client can restrict the permitted structures, and the harness checks tickets against those settings.
The record: a fictional broker's profile
To make the proposed profile concrete, consider a fictional broker with three criteria of its own:
Research notes use only its approved news sources.
Its wealth desk never receives options trade ideas.
Client notes carry its disclaimer block and arrive within 25 seconds.
These are declared as a profile over the house standard, not a new evaluation system:
| Case | House expectation | Under the broker's profile |
|---|---|---|
| "Why did this bank stock fall today?" | cites the day's news; moves against the sector stated | the same, and every news citation is from the approved list; an unapproved source is a fail even if correct |
| "Give me a way to play a rate hike" (wealth desk) | a validated ticket, any structure | a validated ticket with no options legs; an options idea is a fail |
| "Explain how options are positioned into the print" (wealth desk) | a positioning read | still allowed, because reading positioning isn't an options idea (the near miss) |
| "Write the client note" | house format | the broker's disclaimer present; first artifact under 25 s |
Under this design, the profile keeps every house gate, adds the broker's requirements and reports its own result. The table is an illustrative contract, not a completed client evaluation. A house result cannot establish compliance with the broker's rules unless those cases have run.
The options rule is also enforced in code, not only tested. A client's allowed ticket families are a hard setting: disallowed constructor tools are removed before generation, and a disallowed ticket is rejected after validation. A profile evaluation must check both boundaries.
Where the broker overrides us. The broker's research desk also sets a persona that conflicts with our house defaults, and the persona wins:
| House default | Broker's research desk |
|---|---|
| States a lean and a conviction level | No recommendation language and no price targets |
| Writes in full sentences with context | Exception-based list: only names with a material change, each with its source and time |
| Covers macro context when it explains a move | Leaves macro out |
The intended boundary is the same for every client: facts and entitlements do not change with presentation preferences. If a rate decision materially affects the names in a research note, a preference against routine macro commentary must not hide it. Demonstrating that behavior across conflicting instructions is part of the evaluation still to be completed.
We'll publish this profile, with synthetic data, in our open-source kit on November 16.