What gets a grade

1. Keep the whole event in view

The intended universe covers U.S. companies in the S&P 500 or with a market capitalization of at least $10 billion, supported through their 10-K and 10-Q filings, using the eligibility observation retained for that event. Foreign private issuers reporting on forms such as 20-F or 40-F, unsupported listings and depositary receipts remain explicit exclusions. U.S. exchange listing alone does not establish eligibility. Company, security and earnings-event identity must match; ticker alone is insufficient.

The eligibility register distinguishes expected releases from forecasts actually delivered. Failed runs, missing reports, late forecasts and unresolved outcomes stay visible. A successful retry cannot erase an earlier operational failure. Retrospective replays diagnose the system and remain separate from original forecasts. A current company-value screen cannot silently replace historical eligibility.

Each event retains its original report and score, policy version, eligibility observation, evidence cutoff, completion time, publication time where known, report stage and settlement revision. The final manifest must state the complete season start and end coverage and late-discovered events.

The intended pre-release schedule is 15:30–16:00 ET: the same trading day for an after-close release, the preceding trading day for a before-open release. Evidence and completion must precede the actual release. An early release requires an earlier valid forecast or a missed-forecast label.

The retained operational audit instead screened against 16:00 ET for after-close releases and 06:00 ET on the before-open release date. Passing that screen does not establish compliance with the intended prior-close schedule or every source's historical availability. We report pre-release, scheduled-window on time and publication verified separately.

Eligible releases run through December 4, including after-close releases. Their reaction session may settle afterward. The season scorecard is planned for December 14; the release cutoff is not a settlement cutoff.

2. Settle the market call

The primary outcome is split-adjusted stock price return, excluding cash-dividend reinvestment. This is a forecast evaluation, not a portfolio or execution-return claim.

Use exchange sessions and holidays. Retain the price dates, security identity, adjustment details and data version used to calculate returns. Calculate market outcomes separately from the report being graded.

Return = ending price / starting price − 1

The season reference policy classifies scores of 55 or higher as bullish and 45 or lower as bearish. Integer scores 46–54 are neutral. A directional call is correct when its sign matches the return, wrong when the signs differ, and tied at exactly zero. Neutral means abstention. It is not a correct forecast of a small move. A missing forecast is not neutral; a missing price is unresolved; a future endpoint is pending.

The primary table has bullish, bearish and neutral rows against positive and negative returns. Publish cell counts and row percentages, with exact-zero ties separately. Include all scored settled events, and show missing forecasts and unresolved outcomes beside them.

Distinguish small and material moves

A supplementary table adds a small-reaction column: returns above h are materially positive, below −h materially negative, and between those bounds inclusive small. Its neutral row still means abstention. We do not calculate diagonal three-class accuracy, macro-F1 or multiclass MCC from this table because the system did not forecast three return outcomes.

Supplementary, outside the primary result: h = 0.5 × median absolute previous earnings reaction. Use the previous eight completed comparable events, with at least four available before the fixed pre-release reference time. Use the same close-to-close definition and security adjustments. Publish the historical observations and threshold; the current event cannot enter its own history. Fewer than four valid events means scale unavailable. Unknown or inferred release phases are excluded from the latest eight candidate events without replacement by older events.

Historical reconstruction from present snapshots does not certify original database vintages. The factor 0.5 is a convention, not a confidence interval. Fix it before evaluating remaining prospective events. Any predeclared 0.25/0.5/0.75 sensitivity must show every result, without selecting the most favorable one. Fixed ±0.5% and ±1% bands can provide interpretable companion views; they are not part of the primary result. Ordinary daily volatility answers a different question from earnings-reaction scale.

A separate event-implied-move sensitivity requires a correctly identified, timestamped event-only estimate and one common reference-time policy for both systems. It cannot substitute an estimate covering additional days or each system's different observation time. Availability, receipt certification and method limitations remain explicit. Historical and implied scales stay separate, never an outcome-selected fallback mixture.

For every band, show material successes / all directional calls, small-move counts and material misses. Separately show correct / calls whose outcomes cleared the band, labeled conditional accuracy. Compare rules on identical available events and disclose each rule's coverage over the full ledger.

Publish the denominator beside the metric

Publish bull/bear results and uncertainty resampled by reaction date. The same calls also receive stock-minus-SPY and stock-minus-fixed-sector-ETF checks with identical endpoints. Never subtract both benchmarks together. Historical sector classification must be verified or its limitation disclosed.

Ranking diagnostics use all scores: ROC-AUC ranks positive above negative reactions, gives tied scores half credit, and excludes exact-zero returns while counting them separately. Positive- and negative-direction average precision must show class prevalence. Accuracy–coverage curves use fixed abstention half-widths of 0, 2, 5, 7.5, 10, 15 and 20. Scores at or beyond either boundary are directional; exactly 50 remains an abstention even at zero width. No winning threshold becomes an unbiased test result.

Score-band summaries use below 35; 35–45 inclusive; above 45 and below 55; 55–65 inclusive; and above 65. Show counts, positive share, mean and median returns, including empty or small bins. Date/group breakdowns and leave-one-date-out checks retain counts. A score of 70 is not a 70% probability. Probability calibration and probability-loss metrics require a separately defined forecast; dividing the score by 100 does not supply one.

3. Preserve meaningful baselines and score versions

The baseline is the earnings system used before this redesign. It already analyzes financial performance, guidance and company developments. Reproduce its own assessment calculation and retain its native 40–60 inclusive neutral band. A common-band comparison applies the season reference boundaries to both systems, alongside their own policies. A baseline score of 40 abstains under its native policy and is bearish under the common policy.

Keep the original issued scores and labels alongside any comparison calculations. Each published comparison identifies the rules used. Results calculated from previously inspected outcomes remain separate from new forward tests.

Within each policy, the assessment score summarizes saved original business judgments. Its evidence-adjusted score moves toward neutral when the saved record lacks a qualifying comparison between company guidance and analyst expectations. That specific support check cannot reverse direction or rewrite expectations, and is not a general reliability measure. Keep both scores and the original issued record. No event's realized outcome is an input to the score calculation. The proprietary generation and weighting recipe is outside this release.

Constant-up and constant-down diagnostics show base rates on the whole scored cohort and each method's called subset. The latter borrows the method's selection. Dated analyst estimates are a planned baseline for comparable numerical forecasts, not stock direction. An unchanged-guidance control applies where a comparable guide existed. Public previews and recaps remain candidate content comparisons: full packets, time windows and a provider-selection rule have not been assembled.

Paired comparisons use identical events and reaction-date resamples. Record corrections and regressions on jointly directional calls while retaining abstentions and coverage differences. Comparing assessment and adjusted scores isolates a policy effect on saved inputs. Claims about an input's causal contribution require controlled reruns with that input withheld; citation frequency alone does not establish contribution.

4. Grade the financial commitment actually made

Systematic financial grading remains planned; selected examples have been inspected. Match metric, fiscal period, units, scope, accounting basis and exclusions before comparing values.

Publish prediction coverage alongside resolved accuracy. Guidance-action reporting includes the confusion table, per-class recall and support, balanced action recall and, where appropriate, macro-F1. Compare established guidance against the unchanged-guide control. Frequent holds cannot conceal poor raise/cut prediction. Missing evidence is not no formal guidance, and a basis change is not automatically a raise.

Do not combine EPS-dollar, revenue-dollar and margin-percentage-point errors into one raw average. Paired analyst comparisons require the same metric, period and basis. Preserve authored range endpoints; do not invent a midpoint or a tolerance for “approximately,” “modest” or “narrow.” Any beat/miss reference and materiality rule must precede grading.

Precision matters: a reference rounded to $4.3 billion cannot verify an exact $4.316 billion claim. Seek the filing and retain unresolved precision. A matched range with an unverified accounting exclusion remains only partly supported. Split compound claims and calculate comparable numerical errors deterministically, retaining values, units and calculation rules.

5. Six intelligence tests

These are separate planned population measurements. Each event receives only the tests supported by its report stage. Missing reports stay in coverage; a zero denominator means not applicable with a reason, never a perfect score. Average event-level results equally and show claim counts underneath. Longer reports must not dominate. Publish paired differences on identical available events with event/date uncertainty.

Issue matching gives 1 for full, 0.5 for partial and 0 for absent/off-target. Full requires mechanism, suitable measure/scope and horizon; a valid alternative measure needs a rationale. Partial has one material limitation. Generic keyword overlap earns zero. Supported secondary issues can earn relevance credit but cannot replace a missed primary issue. Pre-release coverage concerns knowable uncertainties; genuinely new disclosures belong in the post-release test.

Additional factual/update measures are: correct resolved claims / resolved checked claims, alongside unverified / all checked claims; reports with a material factual/comparability error / reviewed reports; supported substantive revisions / reviewed revisions; and silent-rewrite reports / paired reviewed reports. Unsupported or unresolved claims remain distinct from wrong claims. There is no weighted grand quality score.

6. Compare unlike report formats fairly

The target is every eligible source-ready event, with no fixed sample ceiling. Recover required sources and matched assessments rather than selecting reports that grade cleanly. Availability discovery is not verification of complete transcripts, fiscal periods, source vintages or section-specific information cutoffs.

An October 29 development pilot processed 12 real before-and-after assessments from three deliberately selected companies, using prior/current transcript packets. It produced 315 provisional extracted and graded claims. Structural and source-binding checks demonstrate working automation, not grading accuracy, extraction recall or superiority. The same model family performed the three stages. Independent human validation has not happened. Filing precision and separate news/reception checks remain incomplete. Our extractor omitted predicted management talking points present in one CBRE report; that extraction is incomplete.

Build references without the reports

An automated process constructs versioned reference answers without seeing either system's report, identity or score. Pre-release keys use only evidence available by the forecast cutoff. Disclosure keys use releases, filings and full calls available at the assessed stage, including prior guidance and qualifications. Business keys do not see returns. A separate reception layer uses timestamped news, interpretations, competing catalysts and prices; it records support rather than proving causation.

Every key item needs source spans, versions, availability time and a supported, conflicting or unresolved status. Primary disclosures establish financial facts; news cannot silently override filings. Deduplicate syndicated coverage. Repeated questions or mentions do not by themselves establish materiality. Correct a key through a new version rather than changing it silently.

Extract faithfully, then mask identity

Apply the same extraction to both systems without reference answers, subsequent disclosures or returns. Preserve exact original spans, conditions, ranges, periods, accounting bases, qualifications and contradictions. Distinguish forecasts from questions, facts and attributed guidance. Do not improve vague commitments or choose whichever conflicting sentence aged best.

Inventory relevant business-analysis sections, process long sections in bounded pieces, reconcile coverage and account for exclusions. There is no claim-count cap. Deduplicate propositions while preserving original occurrences. A model's claim of completeness is not measured recall. Omitted relevant sections make affected measurements incomplete; retry them and retain the event in coverage. Independently complete narrower measurements may still be reported as such.

Grade common claims independently against fixed keys using opaque identities and randomized order. Hide system names, original headings/layout, report scores, run metadata and identifying source labels. Business graders do not see returns; reception evidence goes only to its test. Retain private provenance and access to original paragraphs for fidelity checks. This is identity-masked grading, not a fully blind study: extraction sees original formats, and wording can reveal origin.

Numerical comparisons use deterministic calculations. Semantic grades must cite the original claim and reference item or state missing reference. Distinguish contradiction, missing evidence, incomplete extraction and not-a-forecast. Assemble long-report grading only when the complete claim roster is reconciled; whole-report omission measures use the full union. Planned order/identity perturbations and alternative graders are diagnostics, not validation. Separately assess original-format clarity and traceability without changing business-quality scores.

Humans validate a subset

Before inspecting comparative results, set reviewers, sample size, seed and strata. Select random cases across business groups and stages, plus a separately labeled difficult-case sample. Humans check source correctness and completeness, extraction fidelity and grading decisions, consulting original paragraphs. Internal validation is not external expert certification.

Publish audited counts, key errors and omissions, extraction errors, automated/human agreement, uncertainty and unresolved disagreements. Keep random estimates separate from the risk sample. Correct systematic defects, rerun affected cases under new versions, retain earlier results and test repairs on untouched cases. Development cases are not untouched validation. Until these gates pass, population grades remain automated/provisional.

Publish the funnel: eligible, report available, sources verified, extraction complete, graded, unresolved and human-audited. Separate source absence, missing comparator and processing failure. Preserve both full-eligibility and metric-specific denominators. The independently settled market scorecard can proceed while qualitative gates remain open.

7. Timing, repeatability and the public record

Planned latency measurement runs from verified source availability to the first usable read. The usability rule requires accurate coverage of primary changes available at that stage, no material factual/comparability error and stated pending inputs. Release-only and transcript-aware reports need different keys. Publication, persistence, upstream preparation and report runtime are separate clocks. Show median/p95 and late/missing coverage.

The planned repeatability test is 20 pre-release events × three identical-packet generations = 60 executions, not 60 independent events. Freeze sources, returned data, prompts, settings and policy. Report driver agreement, qualification/citation failures, each event's maximum score spread, median/p95 spread and label-boundary crossings. Repeating both report stages would be a separate 120-execution extension. Reconstructing saved scores does not establish generation stability or correctness.

The public market row will identify the company/security, release and timing, policy version, original score/label, comparison scores, measurement dates, calculated raw/market/sector returns, grade and unresolved status. Business rows add structured expectations, public company disclosures, qualifications, resolution and review status. Full report prose, prompts and proprietary formulas are outside this release.

November 2 publishes the method and selected examples. November 16 is the planned interim assessment; reviewed quality and repeatability findings appear only if completed, otherwise remain pending. December 14 is the season reaction scorecard for eligible releases through December 4. Corrections retain dated policy and settlement histories; no later result is backdated.