Regulatory Document Review at Scale: How Carriers Measure Accuracy and Throughput (2026)

Every vendor selling regulatory document review will give you an accuracy number: it will likely be between 95% and 99.5%, but it will not tell you which documents and which fields were counted as correct, or who decided on it.

That number is not a lie. It is a measurement of something, taken under conditions nobody disclosed. The problem is that the conditions move the result more than the software does, and for regulatory documents specifically, the conditions are unusually bad.

This page is about how to measure, not which product to buy. The document set comes first, then what reliability has to mean before it can be tested, then a benchmark your team can actually run, then what breaks at volume. The vendor table is at the bottom, where it belongs.

Key takeaways

• One accuracy number across a mixed document set is an average of things that should not be averaged. In a 2025 benchmark, the same parser scored 0.025 error on academic papers and 0.984 on handwritten notes — roughly a fortyfold difference on one tool.
• Clean benchmarks do not predict real documents. Systems score above 97 F1 on curated receipt datasets. In a competition using real business documents, the winning system reached 70.2% average precision.
• Recall is the number that matters, and accuracy hides it. If a missing endorsement appears in 3% of files, a system that never finds one is 97% accurate with zero recall.
• Twenty flawless documents in a demo is consistent with a true error rate of 15%. That is the statistical rule of three, not an opinion.
• The hyperscalers hand you a measurement harness; most insurance vendors hand you a headline. Google publishes precision, recall and F1 against a test set you label. Of the eight offerings we reviewed, two publish a defined test set and metric definitions.

The document set, and why it behaves differently

"Regulatory documents" is five populations with almost nothing in common except that a compliance team owns them.

Filed forms and endorsements are the best case. Structured, repetitive, often available as native digital PDFs, with form numbers and edition dates in fixed positions. If a vendor demos on these, they are demoing on the easy end.

State bulletins and circular letters are prose. No fixed schema, wildly variable length, published as HTML on one department's site and as a scanned signed PDF on the next. The extraction task is closer to reading comprehension than to form processing.

Product filings arrive as bundles (rate exhibits, actuarial memoranda, form attachments, correspondence) where the unit of work is a package rather than a page, and where the interesting content is often in a spreadsheet embedded as an image.

MGA binders and delegated authority files are the worst input quality in the building: scans of signed documents, faxes, photographs of paper, and handwritten annotations in margins that change the meaning of the printed text.

Correspondence and attestations sit around all of it, and are where the evidence of what was decided usually lives.

A single accuracy figure across those five is not a measurement. It is a weighted average whose weights you were not told.

The evidence that document type dominates

OmniDocBench, presented at CVPR 2025, evaluated document parsers across nine page types with a common metric — normalized edit distance, where lower is better. One pipeline tool scored 0.025 on academic papers and 0.984 on handwritten notes. Another scored 0.059 and 0.651. A general vision-language model scored 0.146 on academic papers and 0.751 on newspapers. The paper's own summary is that "pipeline tools perform well for commonly used data, such as academic papers and financial reports," while "for more specialized data, such as slides and handwritten notes, general VLMs demonstrate stronger generalization."

Read that as a warning about ranking: the best tool on your filed forms may be the worst on your binder scans, and a single procurement decision taken on a single number will get that backwards.

The gap between benchmark and reality is just as large. On curated receipt datasets, published systems score above 97 F1. On DocILE, an open competition at CLEF 2023 built from real business documents, the winning system reached 70.2% average precision on key information extraction, and the organizers noted plainly that "the best results on both KILE and LIR tasks are below 80% of the respective quality metric."

What "reliable" has to mean before you can test it

Six measures. Each is separately gameable, which is why you need all six.

1. Extraction accuracy by document type. Not one number — one number per population, with the population defined by provenance rather than by how clean it looks. Stratify by where the document came from: native PDF, scanner output, fax, photograph, email body.

2. False-negative rate on missing endorsements. The expensive failure in regulatory review is not a wrong value, it is a thing that was there and was not found. Accuracy is the wrong instrument for this, and the standard reference says so: Manning, Raghavan and Schütze note that with skewed data "a system tuned to maximize accuracy can appear to perform well by simply deeming all documents non-relevant to all queries." Recall is defined as true positives over true positives plus false negatives — it is the only one of the three that counts what you missed.

3. Straight-through rate. The share of documents that complete without a human touching them. Useless on its own, because it is trivially raised by lowering the confidence threshold, which raises the false-negative rate at the same time. Hyperscience's own documentation is unusually candid about this: "There is a relationship between accuracy and automation. A higher accuracy target will see the machine send more fields for a human transcription or review." Never accept a straight-through figure without the accuracy target that produced it.

4. Exception rate, by reason. A single exception count tells you nothing. Exceptions because confidence was low, because a required field was absent, because the document did not classify, and because a rule fired are four different operational problems with four different fixes.

5. Throughput per hour, sustained. Peak throughput in a demo is a benchmark of the vendor's test harness. What you need is documents per hour held over a filing deadline week, with the concurrency limit stated.

6. Degradation on scanned and handwritten input. Measured, not asserted. The degradation is well documented and it is steep. DocPTBench, a 2025 benchmark that parses the same documents twice — once as clean digital originals, once photographed — found one system's overall edit-distance error on English documents more than tripling, from 10.5 to 37.5 on its 0–100 scale, where lower is better. Geometric correction recovered most of the loss but not all of it, to 15.9. And it propagates downstream — an Alan Turing Institute study found the named-entity F-score falling from 0.87 to 0.63 as OCR quality dropped across quality bands, and recommended "a preference for high quality OCR ideally above 90% and at least above 80%."

A benchmark a carrier can actually run

None of the above requires a data science team. It requires a labelled set, some arithmetic, and the discipline to decide the thresholds before you see the results.

Build the golden set by provenance

Pull documents from your own pipeline, stratified by the five populations above and by input quality within each. Include the bad ones deliberately — the fax, the photographed binder page, the bulletin that is a scan of a signature. A golden set drawn from what is easy to export is a golden set of your easy documents.

Size it per stratum, not per programme. The sampling arithmetic below gives you a margin of error on one population; getting the same precision on six populations costs roughly six times as many labelled documents.

Decide how many documents you need

Accuracy from a small sample is a range, not a number, and the range is wider than people expect. Label 100 documents, see 95 of them right, and what you have actually learned is that the true rate sits somewhere between about 89% and 98% — a span covering both a system you would deploy and one you would reject.

Narrowing that to three percentage points takes roughly 200 documents. With no prior estimate at all, about 1,000. The formula is in the NIST/SEMATECH e-Handbook, but the practical consequence is simpler: a fifty-document proof of concept cannot tell you whether a system is good or merely adequate.

The sharpest version of this applies when a demo goes perfectly. The rule of three, from Hanley and Lippman-Hand in JAMA, states that "if none of n patients shows the event about which we are concerned, we can be 95% confident that the chance of this event is at most three in n." Twenty perfect documents is consistent with a true error rate of 15%. A hundred still leaves 3%.

For the deeper treatment of gold-set construction and per-field metric choice, including the full Wilson interval table, see our guide to measuring underwriting summary accuracy before you buy. 

Measure agreement between your own reviewers first

A set labelled by one person is not ground truth, it is one person's opinion written down. Label a subset twice, independently, and compute agreement before you score any vendor against it.

For two raters on categorical judgements, Cohen's kappa is the standard, and the conventional interpretation bands come from Landis and Koch (1977): 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect. Treat those bands as convention rather than science. McHugh (2012) argues they are "too lenient for health related studies because it implies that a score as low as 0.41 might be acceptable," and proposes a stricter reading where anything below 0.60 is inadequate. For a compliance file, take the stricter one.

Two cautions. Kappa is sensitive to prevalence — the NCBI methodology reference notes that "because of the dependence of kappa on the prevalence rates, direct comparison of kappa values between population groups having different prevalence rates may be inappropriate," which is exactly the rare-but-expensive case you care about. And if you have more than two reviewers, or partial overlap, or missing labels, use Krippendorff's alpha instead, which handles "any number of observers," "incomplete or missing data" and mixed measurement levels. The operative reliability threshold in Hayes and Krippendorff is α ≥ 0.800.

If your own reviewers cannot agree with each other, no vendor can be measured against them, and the honest first finding of your benchmark is that your rules are ambiguous.

Set acceptance thresholds before the bake-off

Write them down first, per document type and per field, and weight them by consequence rather than by frequency. Azure's documentation offers a usable default: "It's best to target a score of 80% or higher. For more sensitive cases, like financial or medical records, we recommend a score of close to 100%."

Four rules that make thresholds hold up. Separate recall thresholds from precision thresholds, and set recall higher on anything whose absence creates exposure. Score blanks and refusals explicitly — a system that declines to answer is not correct. Define a tolerance per field type before you start, so a date format difference is not scored as an error and a transposed limit is. And agree a re-test trigger contractually: model versions change, and the number you accepted was measured on a version that will not exist in eighteen months.

That last one is not hypothetical. Google's Document AI custom extractor versions have been shut down on roughly annual cycles — v1.4 was retired on 31 March 2026, and the platform's legacy processors were discontinued on 30 June 2026. A model version you validated for a regulatory purpose has a shelf life, and revalidation is your cost, not the vendor's.

The standards framing, if you need one for a governance committee: NIST's AI Risk Management Framework (NIST AI 100-1, January 2023) asks under MEASURE 2.3 that performance be "measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s)," and under MEASURE 2.5 that "limitations of the generalizability beyond the conditions under which the technology was developed are documented." A vendor benchmark on clean documents is not evidence about your scanned binders, and NIST says so in the language a risk committee recognizes.

What breaks at volume

Accuracy is measured on a sample. Throughput is where the sample meets a filing deadline.

Queueing. Regulatory work is bursty in a way submission intake is not — a bulletin drops and creates a book-wide review, a filing deadline concentrates months of work into a fortnight. The queue that matters is the exception queue, not the processing queue, because exceptions consume the scarce resource. Model it on reviewer hours, not on documents.

Prioritization. First-in-first-out is the wrong discipline when documents carry different consequences. Effective date, states you write in, lines affected, whether a form change is implied — these set the order. If the platform cannot express it, your team will, by hand, and the throughput figure in your business case was measured without that step.

Surge handling. Here published limits matter more than published accuracy, and the general platforms are the transparent ones. Google documents 5 concurrent batch requests per project and 120 requests per minute per processor type in its multi-region locations, but only 6 per minute in a single region — a twentyfold penalty falling on anyone with data residency constraints outside the US and EU. Azure documents 500 MB and 2,000 pages per document, 15 transactions per second by default, and 10,000 documents per batch. Of the eight offerings reviewed, four publish no throughput figure at all.

The thing that actually caps you. Straight-through rate sets the ceiling; exception handling sets the floor. A platform at 85% straight-through with a good review queue will out-process one at 95% with a bad one, because the 5% that stops is what your team spends its day on. Time the review of an exception, not the processing of a document.

What to ask a vendor

1. What is the unit of your accuracy number? Field-level or document-level, and across how many fields. Per-field accuracy compounds: a document with 25 fields at 98% per field is right in full less than two-thirds of the time, if errors are independent.
2. Show me the number broken out by document type and input quality. If it does not exist, the single number was never decomposed, which means nobody knows where it comes from.
3. What is the false-negative rate on the checks that matter? Measured separately, on a set where the item was deliberately absent.
4. Who labelled the ground truth, and what was the inter-rater agreement? A vendor that has never computed this has never had a defensible test set.
5. What accuracy target produced that automation rate? The two move against each other. A pair of numbers without the configuration is two unrelated facts.
6. What does a finding cite? Page, coordinates, snippet — enough that a reviewer can confirm in seconds and an examiner can confirm in a year. Our guide to source-backed audit findings covers what makes one defensible.
7. What are the published concurrency and volume limits? Not the ones achieved in a benchmark — the ones in the documentation.
8. Is my data used to train your models, and where is that written? The answer belongs in product documentation or a contract, not in a sales call.

How the platforms compare

We reviewed the public product material of eight offerings in September 2026. This table sits below the method deliberately: it tells you what each vendor says, which is a different thing from what your benchmark will find.

Three things to know before reading it. Two of the eight are general cloud platforms rather than insurance products, and they are here because they publish the measurement tooling and the hard limits that the insurance-specific vendors mostly do not. One — Convr — is submission-intake focused; its public material describes ACORD forms, broker emails, SOVs and loss runs, and nothing about filed forms or regulatory documents, so it is included for completeness and marked accordingly. And Roots Automation rebranded its platform to Bevaya on 28 May 2026, with Roots Automation Inc. remaining the legal entity.

‍

‍

How to read this table. "Documented" means the capability is described in the offering's own public product material. "Not published" means we could not verify it publicly, which is not the same as saying it does not exist — several of these gate documentation behind a login. Vendor links are omitted deliberately; assess each against your own requirements.

What the table actually shows

The accuracy column does not split the way you would expect, and it does not track insurance-native versus general at all. Two offerings publish a defined test set with defined metrics: Bevaya, and Google in the sense that it hands you a harness rather than a number. Two publish no accuracy figures whatsoever, which on inspection is the more honest posture — Indico publishes business outcomes only, and Instabase ships ground-truth tooling and lets you compute your own. One publishes the loudest numbers with the least methodology, and a genuinely rigorous definition three clicks away in its documentation.

Provenance inverts that ranking. Indico publishes the weakest accuracy evidence and the strongest provenance: character offsets, page numbers and bounding boxes in documented API output, which is machine-readable evidence rather than a screen feature. Hyperscience publishes the loudest accuracy numbers and no discoverable provenance at all. If you are buying an audit file that has to survive an examiner, read this column first.

Deployment separates on the same axis. Azure documents disconnected on-premises containers; Google documents none. Hyperscience publishes the widest range, including air-gapped; two vendors publish nothing.

On our own row: FurtherAI publishes field-level accuracy with a stated field count on one carrier deployment, which is more unit than most, and publishes the measurement method as a separate guide. Those two things are not yet joined up across every figure on our site, and this article says so rather than pretending otherwise. On data handling, our single-tenant architecture gives each customer a dedicated instance with isolated data and compute, and our security documentation states that customer data is never used to train or fine-tune models. Ask every vendor here where their equivalent commitment is written: Google states it in product documentation, several state it nowhere a buyer can find.

Our own tooling carries the same limit. As we note in testing AI workflows before production, test sets of 50 to 100 submissions catch regressions well and are too small to estimate absolute accuracy to within a couple of points. On general versus insurance-specific, see horizontal AI tools vs dedicated insurance platforms.

Frequently asked questions

‍

REFERENCES

^{Du, Yongkun, Pinxuan Chen, Xuye Ying, and Zhineng Chen. "DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation." arXiv preprint arXiv:2511.18434, 23 November 2025. arxiv.org}

^{Google Cloud. "Deprecations — Document AI." docs.cloud.google.com}

^{Hanley, James A., and Abby Lippman-Hand. "If Nothing Goes Wrong, Is Everything All Right? Interpreting Zero Numerators." JAMA 249, no. 13 (1 April 1983): 1743–1745. mcgill.ca}

^{Hayes, Andrew F., and Klaus Krippendorff. "Answering the Call for a Standard Reliability Measure for Coding Data." Communication Methods and Measures 1, no. 1 (2007): 77–89. asc.upenn.edu}

^{Krippendorff, Klaus. "Computing Krippendorff's Alpha-Reliability." Annenberg School for Communication, University of Pennsylvania, 2011. asc.upenn.edu}

^{Landis, J. Richard, and Gary G. Koch. "The Measurement of Observer Agreement for Categorical Data." Biometrics 33, no. 1 (1977): 159–174. pubmed.ncbi.nlm.nih.gov}

^{Manning, Christopher D., Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008. nlp.stanford.edu}

^{McHugh, Mary L. "Interrater Reliability: The Kappa Statistic." Biochemia Medica 22, no. 3 (2012): 276–282. biochemia-medica.com}

^{National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, January 2023. nvlpubs.nist.gov}

^{NIST/SEMATECH. "Sample Sizes Required." e-Handbook of Statistical Methods, section 7.2.4.2. itl.nist.gov}

^{National Center for Biotechnology Information. "Measures of Agreement and Approaches to Modeling," in Reliability of Key Measures in the National Survey on Drug Use and Health. ncbi.nlm.nih.gov}

^{Ouyang, Linke, et al. "OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations." CVPR 2025. openaccess.thecvf.com}

^{Šimsa, Štěpán, et al. "Extended Overview of DocILE 2023: Document Information Localization and Extraction." CEUR Workshop Proceedings, CLEF 2023. ceur-ws.org}

^{van Strien, Daniel, Kaspar Beelen, Mariona Coll Ardanuy, Kasra Hosseini, Barbara McGillivray, and Giovanni Colavizza. "Assessing the Impact of OCR Quality on Downstream NLP Tasks." ARTIDIGH 2020. Alan Turing Institute. universiteitleiden.nl}

‍

DISCLAIMER 

This article is for general informational purposes only and does not constitute legal, regulatory, compliance, underwriting, or other professional advice. The content reflects information available as of the date of publication, and FurtherAI undertakes no obligation to update it as laws, regulations, or AI technologies evolve. 

‍