Research · AI model defects
What breaks in financial models
that AI builds
A 55-class defect taxonomy from a repeat economic audit of 32 models. Not "AI sometimes gets formulas wrong" — concrete, recurring, classifiable ways a model becomes wrong while looking healthy.
32
models audited
55
defect classes
258
findings
4/10
median score at debugging peak
Where this data comes from — the honest frame
All 32 models were built by our own engine during its 2026 debugging program. This is not an audit of anyone else's products and not market statistics — it is our internal debugging dataset, published.
The audit is dated (2026-07-24) and taken with a single taxonomy: 19 accumulated classes plus 36 new clusters found by this pass. The median economic score was 4 out of 10 at the peak of the program: the models' algebra was proven by gates; their economic construction against the brief was not.
Every class in the table gets a mechanism or a machine detector; the fix waves have shipped to production. We publish this because we have not seen anyone publish a defect taxonomy of AI-generated models on their own data.
Eight defect families
The full audit taxonomy grouped by the nature of the defect. Count — findings in the audit; P0 — models where the class was critical.
Brief mechanization
The most expensive family: the brief asked for a mechanism — a credit facility, an escrow scheme, a second construction phase — and the model was silently built without it. No cell is wrong; a piece of the economics is missing.
| Defect class | Findings | P0 models |
|---|---|---|
| Requested KPI missing from the deliverable | 19 | — |
| Brief asked for a mechanism — the model silently lacks it | 16 | 16 |
| Line item booked to the wrong statement section | 13 | 7 |
| Exit method silently substituted inside return math | 1 | 1 |
| Bookings recognized as a full year of revenue | 1 | — |
| Prepayment terms wired with the sign flipped | 1 | — |
| Default/loss semantics diverge from the brief's contract | 1 | — |
| Greenfield capex profile forced onto a brownfield asset | 1 | 1 |
| Greenfield entry assumptions on an operating business | 1 | — |
| Sustaining capex lumped into a phantom build year | 1 | — |
| Operations start silently truncates the volume series | 1 | — |
Template leaks
Template calibration survives the client's facts: someone else's margin, someone else's tax regime, live-looking inputs with a dead hardcoded series behind them.
| Defect class | Findings | P0 models |
|---|---|---|
| Template default leaked past the brief's facts | 28 | — |
| Tax regime / locale defaults and NOL treatment wrong | 22 | — |
| Dead input drivers: series hardcoded behind live-looking inputs | 21 | — |
| Dead driver laundered through a keep-alive reference | 1 | — |
| Year gate frozen inside a scalar driver | 1 | — |
Debt & financing
Debt schedules hardcoded as constants, wrong amortization types, reserves reverse-solved as plugs, contradictory capital structures.
| Defect class | Findings | P0 models |
|---|---|---|
| Debt schedule hardcoded with wrong amortization | 16 | 13 |
| Funding reserve reverse-solved as a plug | 6 | 5 |
| DSCR series truncated or polluted | 2 | 2 |
| Two contradictory capital structures in one model | 1 | 1 |
| Interest on a basis inconsistent with the balance line | 1 | — |
Time & working capital
Phantom year-zero misalignments, working capital multiplying an empty base, seasonality declared but flat, ghost tails beyond the asset's life.
| Defect class | Findings | P0 models |
|---|---|---|
| Working capital wired to an empty base | 20 | — |
| Phantom Y0: blocks misaligned around the start year | 8 | 7 |
| Horizon drift: a ghost tail beyond the asset's life | 8 | 5 |
| Seasonality declared but the series is flat | 2 | — |
| Pre-opening year carries no costs | 1 | — |
| Annual grid credits a full year to mid-year cohorts | 1 | — |
| Capacity set equal to demand clips seasonality | 1 | — |
| Monthly satellite: flat opex against seasonal revenue | 1 | — |
Growth & indexation
Frozen escalators, mis-compounded ramps, costs indexed asymmetrically to revenue — plus a tail of one-off arithmetic slips.
| Defect class | Findings | P0 models |
|---|---|---|
| One-off distortions (single-model arithmetic slips) | 24 | — |
| Escalator frozen at the year-1 scalar | 1 | — |
| Growth driver frozen at the base year | 1 | — |
| Growth ramp mis-compounded | 1 | — |
| Costs indexed asymmetrically to revenue | 1 | — |
Returns & terminal value
Headline IRRs that answer no investment question; decorative exit cap rates while a Gordon terminal actually rules; terminal value with no sustaining capex.
| Defect class | Findings | P0 models |
|---|---|---|
| Headline IRR that answers no investment question | 7 | 7 |
| Headline metrics indexed to the wrong year | 2 | — |
| Exit cap rate decorative; Gordon terminal actually rules | 1 | 1 |
| Unrequested perpetuity exit dominates NPV | 1 | 1 |
| Terminal value with no sustaining capex | 1 | 1 |
| Unrisked headline on a probability-weighted deal | 1 | 1 |
Deliverable integrity
The chat reply diverging from the workbook, provenance labels inverted, scenario axes wired to levers that do nothing.
| Defect class | Findings | P0 models |
|---|---|---|
| Chat reply diverges from the workbook | 2 | — |
| Live re-certification verdict drifts from the export | 1 | — |
| Brief fact mislabeled as an assumption | 1 | — |
| Brief fact demoted to a template default | 1 | — |
| Provenance labels inverted | 1 | — |
| Value origin mislabels a brief fact | 1 | — |
| Value-origin provenance inverted | 1 | — |
| Scenario axes wired to inert levers | 1 | — |
| Degenerate scenario panel (all cases collapse) | 1 | — |
| Synthetic scenario axis disconnected from the model | 1 | — |
| Scenario delta on a zero line yields negative cost | 1 | — |
| Bespoke series length mismatches the model grid | 1 | — |
| Capex registered twice: lump sum plus annual | 1 | — |
Structural capitulation
Rare but fatal: a balance-sheet business (a bank) forced onto a generic DCF with no deposits; a cost chain collapsed into a single calibrated percent.
| Defect class | Findings | P0 models |
|---|---|---|
| Cost chain collapsed to one calibrated percent | 4 | 3 |
| Generic DCF forced onto a balance-sheet business | 2 | 2 |
Six profiles up close
The most instructive classes: what the defect looks like, why it survives, and what catches it by machine.
The brief asked for a mechanism — it is silently missing
- What it looks like
- The brief describes a credit facility, an escrow scheme or a second construction phase. The agent builds the model without that mechanism — it balances, certifies and looks healthy.
- Why it survives
- The #1 class by severity: 16 of 32 models, all P0. No single cell is wrong; a piece of the economics is absent, and only checking the deliverable against the brief — not the model against itself — can reveal it.
- What catches it
- An intake gate for required structures before the build, plus signature detectors for block verticals: a model with lending vocabulary but no lending block fails the build.
A template default leaked past the brief
- What it looks like
- The client stated their margin, rate or share — and the template's calibration survived instead. The most frequent class: 28 findings.
- Why it survives
- The number looks plausible because it is plausible — for a different business. Without provenance labels, a client fact and a template default are indistinguishable in a finished workbook.
- What catches it
- Provenance labels on every driver (brief fact / calibration / default) and a check that brief facts must survive the build verbatim.
A debt schedule hardcoded as constants
- What it looks like
- Interest and principal are typed in as numbers: the amortization type is wrong (annuity instead of straight-line), the refinance is missing, and the series does not move when the rate changes.
- Why it survives
- 13 of 16 cases were P0. Debt is the most connected part of a model: a hardcoded series severs the link to the balance sheet and DSCR, and the model stops answering "what if the rate is higher".
- What catches it
- The debt block must be built by a mechanism from loan terms; a hardcoded-series detector flags rows with no formula lineage.
Working capital multiplies an empty base
- What it looks like
- Payables are set as a share of purchases — and the purchases line in this model is empty. The share multiplies zero: working capital vanishes and cash flow flatters. 20 findings.
- Why it survives
- Invisible in the statements: the balance ties, the signs are right — one line is just quietly zero across the horizon.
- What catches it
- The base of every share is an explicit choice, not a convention; an empty-base lint fails the build when a share multiplies a zero line.
An IRR that answers no question
- What it looks like
- The headline IRR is computed over a mix of incomparable flows — levered with unlevered, operating cash with an exit from another horizon. The number exists; the question it answers does not.
- Why it survives
- All 7 cases were P0: this number goes into decisions. IRR is the most quoted metric in a model and the easiest to corrupt — the function returns a result for almost any series.
- What catches it
- A 2×2 labeling discipline (project/equity × levered/unlevered) and a multiple-roots guard: an ambiguous IRR is not published as a headline.
A ghost tail beyond the asset's life
- What it looks like
- The asset is finite — a license, a mine, a lease — but the flow lives past the end: revenue after depletion, D&A after write-off, debt not repaid by end of life.
- Why it survives
- Two or three extra years of tail can carry a visible share of NPV. The model looks conservative in every single year and lies in aggregate.
- What catches it
- Finite-life mechanics: asset life is a property of the asset, not of the grid; terminal gates check every line's survival against the asset's life.
How this relates to the standards
Modeling standards exist and we build on them: the FAST Standard is the recognized craft standard; the ICAEW Financial Modelling Code is the principle set we measure our own export against; SSRB and model-audit methodologies are a mature manual-review discipline.
Our layer is different and complementary: machine verification of every build instead of a one-off manual review, assumption plausibility against sourced industry ranges — which classical model audit explicitly leaves out of scope — and a defect taxonomy specific to AI generation, where the top class of errors is the brief-mechanization layer, not the formulas.
What every build gets checked for
The taxonomy is not a museum: every class is closed by a mechanism or a detector, and the checks run by machine on every model build.
- Python ↔ Excel parity: every formula in the workbook is recomputed independently, tolerance 1e-6.
- Three-statement tie-out: the balance sheet must close; an untied model does not build.
- Plausibility bands with sources: headline metrics against sourced industry ranges.
- Structure signature detectors: brief vocabulary without the corresponding block fails the build.
- Dead-driver and empty-base lints: a live-looking input must actually move the model.
- Provenance labels: brief fact, calibration or default — every value has a lineage.
Audit dated 2026-07-24; taxonomy: 55 classes across 32 models.
Build a model a machine has verified
Rootsmodel assembles a three-statement model with live formulas and runs it through every check on this list — an integrity certificate ships with every build.