# Quality

> Measured error rates for each dataset, with sample sizes and methods. What we haven't measured yet is marked.

Every number on this page comes from a check we ran against the SEC's own filings, with the sample and the method stated. Where we haven't published a check, the page says so: treat those fields as unmeasured, not as error-free.

Three things to keep in mind:

- **A small sample proves less than it looks.** Zero errors in 43 companies still leaves room for a true error rate of up to about 8% (95% Wilson interval).
- **These checks test fidelity to the filing**, and which line we picked. They don't test whether a company's own figures are economically right.
- **Each check describes the data it was run on.** The metrics check names its snapshot. The data you receive carries its own `meta.snapshot_id`.

## At a glance

Each published result is explained, with its sample and method, in the section below it.

| Dataset | Result | Status |
|---|---|---|
| Metrics: revenue | 2 wrong, all banks | Published |
| Metrics: net income | 0 wrong in each group | Published |
| Metrics: cash and equivalents | 3 wrong of 100 | Published |
| Metrics: diluted EPS | | Not published yet |
| Metrics: operating cash flow | | Not published yet |
| Statements | No wrong value or sign found | Published |
| Segments | 35 wrong, about 2.7% of those compared | Published |
| Holders | 0 wrong per method | Published |
| Financials | 25 fields, estimated error 0.0% to 2.4%, 95% upper bounds up to 5.5% | Published |
| Earnings releases: titles | 94 of 95 titles correct | Published |
| Insider trades | | Not published yet |
| Form 13F holdings | | Not published yet |
| Ownership | | Not published yet |
| Filings | | Not published yet |
| Macro | | Not published yet |

## Metrics

**What was checked.** Whether each metric's newest figure is the right line from the filing: the figure a reader of the filing's page would call revenue, net income or cash. **Sample:** 100 companies picked by hash, 30 banks, 20 insurers and 50 others. Revenue was read for the 90 with a figure. Net income and cash were read where a plain rule couldn't decide which line was right: 35 and 43 companies. **Snapshot:** `metrics-7141c1f133996c9c`, the data as of 2026-11-06. That is the date of the data that was checked, not the date the check ran.

| Metric | Banks | Insurers | Others |
|---|---|---|---|
| Revenue | 2 wrong of 27 | 0 of 20 | 0 of 43 |
| Net income | 0 of 30 | 0 of 20 | 0 of 49 |
| Cash | 1 of 30 | 0 of 20 | 2 of 50 |

- **Revenue at banks.** For 14 of the 27 banks, revenue is total interest income, because the bank prints no total revenue line and interest income is the top total on its page. We count that as right. If you want net revenue including noninterest income, count those 14 as wrong too: 16 of 27.
- **Net income** counts either the attributable or the total figure as right.
- **Cash.** Of the 3 wrong, 2 include restricted cash, as their tags say, and 1 takes a rounded figure from a note.

**Across all companies**, weighting each group by its size (571 banks, 141 insurers, 5,826 others), the point estimates are 0.65% wrong for revenue, 0% for net income and 3.9% for cash. These are point estimates with wide uncertainty: "others" are most companies, and zero wrong in 43 of them still allows a rate up to about 8%.

**Not measured yet:** diluted EPS and operating cash flow. They follow the same rules, but their error rates aren't published.

### How often a figure changes

Figures aren't versioned: a later filing can restate one, or settle a disagreement between filings. To measure how much that happens, we ran the rules over the facts as filed by 2026-05-06 and over the facts of 2026-11-06, for periods at least a year old by the earlier date: 1,158,242 figures.

| What happened | Figures | Why |
|---|---:|---|
| Moved | 1,181 | Same tag, and a later filing states another figure |
| Moved | 1,178 | Another tag now supplies the period, usually the company's newer tag |
| Moved | 260 | A derived quarter whose operand was refiled |
| Moved | 12 | The cash rule |
| Moved | 3 | The company's own filing replaced another form, or a scale rule |
| Moved | 8 | Not attributed |
| Disappeared | 1,064 | The series no longer uses that tag, and the new one doesn't cover the period |
| Disappeared | 489 | A derived quarter that can no longer be derived |
| Disappeared | 99 | The tag is still in use, but not for this period |
| Disappeared | 23 | The whole series went |
| Left out after being published | 5 | The rules |
| Published after being left out | 11 | The rules, once a later filing settled the figure |

This attribution compares the two runs' tags and filings; it wasn't read against the filings themselves.

### Figures we withhold

When the rules can't settle a figure, it's left out instead of guessed. In the build of 2026-11-07, 833 of 1,301,626 figures were left out, on 242 companies. The rules and their counts are in [Known limits](/known-limits/#metrics).

## Statements

**What was checked.** Whether each statement we serve matches its filing: every value against the filing's XBRL instance, and the figures against the SEC's rendered page. Two samples, each 100 filings picked by hash before anything was fetched.

**First sample:** 343 statements, including 28 banks, 25 IFRS filers and 28 amendments.

- Every value matched the filing's XBRL instance.
- On the rendered pages, 21,882 figures were found with the right period and unit. 21,814 of them had their sign checked, and none was wrong.

**Second sample:** 332 statements read through the live API, with quotas for banks, IFRS filers, multi-currency filers and amendments.

- 23,189 values were exact against the instance. None had a wrong value or sign. 4 had another unit, and 2 differed only by the four-decimal limit.
- On the pages, 22,715 figures matched with period, unit and sign, and 647 more as balances. None had a wrong sign.
- 9 values were flagged as wrong, and all 9 are a second, rounded figure for the same item. For example, NIO's page prints 34,400,000 thousand where the filing's data holds 34,387,266,000.
- 230 of the 332 statements had no difference at all.

**What a statement leaves out.** Dimensional lines, such as segment lines, aren't in statements. In the first sample, 66 of 343 statements (19%) had at least one, and they were 4.4% of the figures on the pages. In the second, 72 of 332 (21.7%), and 3.2%.

**Who read them.** Ten statements were also read beside their pages by an AI model, Claude. No person has read any of them against its filing. That is a limit of these checks.

## Segments

**What was checked.** The revenue figures we serve against the segment, product and geographic notes printed in the filing. **Sample:** a held-out set (seed `segment-docs-honest-2026-11-08`) on which no fixes were made afterwards; 36 filings had note pages to compare.

| Outcome | Figures |
|---|---:|
| Matched the note page | 1,284 |
| Wrong | 35 |
| Ours, but not on the note pages the check read | 927 |
| On the note pages, but not in our data | 1,591 |

So about 2.7% of the figures that could be compared were wrong. The last two rows are coverage, not errors: figures the check couldn't confirm, and figures we don't serve. Comparing the sum of product members with the company's total, 9 filings were equal, 6 were not, and 25 had no product breakdown to compare.

## Holders

**What was checked.** Whether each CUSIP we link to a company really is that company's security. **Sample:** 100 links per linking method. **Result:** 0 wrong in 100 for each method. With 100 checks, the true error rate could still be up to about 2.95% per method (Clopper-Pearson, 95%).

**What's missing.** About 35.1% of 13F holding rows, and about 32.9% of their dollar value, can't be linked to a company with two independent pieces of evidence, so they aren't served. Holder lists are incomplete by that much.

## Financials

**What was checked.** Whether each served value matches the line printed on the company's statement page. An AI model helped grade the printed pages. The 10 fields added on 2026-11-09 had an extra round of grading on values our checker could not find on the statement; most of those turned out to be printed, so they were checker misses, not errors.

| Field | Since | Estimated error | 95% upper bound |
|---|---|---:|---:|
| `total_assets` | pilot | 0.7% | 2.1% |
| `total_liabilities` | pilot | 0.8% | 2.5% |
| `operating_cash_flow` | pilot | 0.6% | 2.7% |
| `financing_cash_flow` | pilot | 0.6% | 2.7% |
| `investing_cash_flow` | pilot | 0.6% | 3.0% |
| `diluted_eps` | pilot | 1.4% | 3.5% |
| `total_equity` | pilot | 1.4% | 3.6% |
| `net_income` | pilot | 1.9% | 4.4% |
| `cash_and_equivalents` | pilot | 2.4% | 4.6% |
| `weighted_diluted_shares` | 2026-11-09 | 0.0% | 2.6% |
| `current_liabilities` | 2026-11-09 | 0.9% | 2.9% |
| `retained_earnings` | 2026-11-09 | 0.7% | 3.0% |
| `current_assets` | 2026-11-09 | 0.9% | 3.0% |
| `receivables` | 2026-11-09 | 0.7% | 4.0% |
| `research_development` | 2026-11-09 | 0.0% | 4.2% |
| `basic_eps` | 2026-11-09 | 1.6% | 4.4% |
| `operating_income` | 2026-11-09 | 1.4% | 4.6% |
| `ppe_net` | 2026-11-09 | 0.9% | 5.0% |
| `income_tax` | 2026-11-09 | 1.4% | 5.5% |
| `intangibles` | 2026-11-09, strict | 0.0% | 2.6% |
| `accounts_payable` | 2026-11-09, strict | 0.6% | 3.5% |
| `gross_profit` | 2026-11-09, strict | 1.0% | 4.9% |
| `pretax_income` | 2026-11-09, strict | 1.3% | 5.2% |
| `sga` | 2026-11-09, strict | 0.0% | 5.2% |
| `revenue` | 2026-11-09, strict | 1.3% | 4.3% |

The pilot fields' rates cover printed values only. They were not re-checked for values the statement doesn't print.

For a field with no false value in its 3 sampled rows, the 95% upper bound pools the 30 clean rows across 10 fields. This assumes the checker's misses are not specific to one field. For the five strict fields, the bound pools 36 checks. The bank revenue rule and the strict fields are described on [Financials](/datasets/financials/#strict-fields-and-bank-revenue).

## Earnings releases

**What was checked.** The title of each release. An AI model proposes the title, and we verify it word for word against the exhibit text. On a hidden sample, 94 of 95 titles were correct. A title can be `null`. We don't extract figures from the releases, so there are no figure error rates to publish.

## Not published yet

For insider trades, Form 13F holdings, ownership, filings and macro, and for the diluted EPS and operating cash flow metrics, we haven't published error rates with their samples. Until we do, treat those as unmeasured.

## Latency

Measured one request at a time, at most 2 per second, with keep-alive, and a new connection on every tenth repeat. Every request got a 200. Times in milliseconds. This is a measurement, not a promise: there's no SLA.

| From | Request | n | p50 | p95 | p99 |
|---|---|---:|---:|---:|---:|
| Ashburn, Virginia (edge `iad1`) | First | 601 | 51.5 | 95.4 | 153.5 |
| | Repeat | 541 | 31.9 | 53.2 | 158.8 |
| Moses Lake, Washington (edge `pdx1`) | First | 601 | 125.0 | 199.5 | 222.6 |
| | Repeat | 541 | 108.3 | 174.9 | 210.1 |
| Thailand (edge `sin1`), 5 minutes | First | 298 | 304.5 | 540.6 | 565.0 |
| | Repeat | 269 | 287.0 | 504.9 | 532.9 |

The mean response was about 84 KB. Sustained load through the API hasn't been tested.

---
Source: https://docs.belfortagent.xyz/quality/