Rules-Based Automation vs. AI Extraction for Specialty Lines

There's a number that settles most of this argument, and you can get it off your own submission log in an afternoon: how many submissions arrive per distinct document layout.

Call it submissions-per-layout. If a hundred submissions a week arrive on four layouts you've had for years, that number is 25 and extraction rules obviously repay the effort of writing them. If those hundred arrive on ninety layouts with a dozen never seen before, it's close to 1 and rule-writing never repays anything. Everything else in this debate is downstream of that ratio.

Specialty sits at the bad end of it for definitional reasons, not anecdotal ones. But the word "rules" is doing two different jobs in this argument, and most comparisons weld them together, so that comes first.

Key takeaways

• "Rules-based" names two machines. Extraction rules read data out of documents. Business rules decide from data that's already structured. AI substitutes for the first and largely doesn't substitute for the second, so any end-to-end automation rate bundles them into one figure and tells you nothing.
• Template diversity is the deciding variable, and it's countable. In the VRDU benchmark, published at KDD 2023, moving to layouts a model hadn't seen in training cost 13–17 micro-F1 points at 200 training documents on its Registration Form corpus.
• Don't assume the next model generation fixes it. In VRDU, LayoutLMv3 scored worse on unseen templates than LayoutLMv2, the model it directly succeeds, at every training-set size — falling from 89.77 on mixed templates to 62.58 on unseen ones.
• Repeated schedule rows are the hardest case for either approach. On the benchmark's other corpus, one model showed a 60–70 point gap between hierarchical fields and ordinary ones, and the paper calls extracting them properly an open research question. A schedule of values has that shape.
• Where the data really is standardised, rules win on cost, not accuracy. In a 2025 radiology preprint, regular expressions scored 89.20% against an open-weights LLM's 87.69% on a 199-report subset (p = 0.56), and ran over 18,000 times faster.

Two machines, one word

The first sense is rule-based extraction: regular expressions, dictionaries, and positional templates run over a document to pull values out of it. Field three sits eleven millimetres below the header on ACORD 125, so read the box at that coordinate. This is what Chiticariu, Li, and Reiss surveyed in their 2013 EMNLP paper, still the canonical treatment.

The second sense is a business-rules engine: appetite screening, eligibility, authority limits, referral triggers, declination logic. The input is already a clean record, and the rules encode underwriting policy rather than document geometry.

These fail in opposite ways. An extraction rule fails quietly and locally: a pattern stops matching, one field comes back empty, and downstream that empty field is indistinguishable from one the broker never filled in. It looks like a broker omission, so it gets triaged as one. A business rule fails globally, because rules chain and contradict each other. Preece, Shinghal, and Batarekh gave the failure modes precise names in 1992: redundancy, ambivalence, circularity, and deficiency. A positional template can't be circular: only a logic can. 

So "AI replaces rules" is one sentence describing two propositions. Learned extraction does substitute for extraction patterns. It doesn't substitute for appetite logic, and nobody writing specialty business should want it to: an appetite rule needs to be deterministic, auditable, and changeable by an underwriter on a Tuesday. Chiticariu and colleagues make the editability point about extraction rules — "With a rule-based system, such a requirement translates to removing a few rules. On the other hand, a ML-based approach requires a complete retrain" — and it holds with more force one layer up, where what's being changed is underwriting policy.

The survey found rule-based systems dominating commercial extraction products while academic work had moved almost entirely to statistical methods. Its headline split — 75% of academic papers using machine learning against 3.5% using rules — predates BERT, LayoutLM, and every large language model, so read it for the argument rather than the numbers. All three authors worked at IBM Research, where the rule-based extraction product SystemT was built.

The variable that decides it

Four things get named whenever  specialty is described as hard to automate: schedule variability, non-standard forms, manuscript wordings, and submission volume. Three of those are the same variable.

Non-standard forms raise the number of distinct layouts you receive. Manuscript wordings are the limit case. IRMI defines a manuscript form as one "custom designed for a particular insured," in contrast to standard forms "drafted for insurers' use by insurance industry services organizations such as American Association of Insurance Services, Insurance Services Office, Inc., and National Council on Compensation Insurance." A standard form is standard because a named bureau drafted it, which is what makes its layout predictable; a manuscript form has no bureau standard behind it, so any form number on it is the insurer's own and pins nothing. One document, one layout. Those last inferences are ours; IRMI defines the term and says nothing about layout. Submission volume is just how many documents share each layout.

So the three collapse into one ratio, and that ratio has a literature behind it. The VRDU benchmark, from researchers at Google and UC San Diego and published at KDD 2023, was built around five gaps its authors found in existing benchmarks, one being generalization to new layouts. Its hardest task trains on some templates and tests on templates held out entirely. At 200 training documents, "micro-F1 for UTL is 13–17 percentage points worse than the micro-F1 for MTL across the three models."

Two caveats before that number travels. On the Registration Form corpus it means training on two of three layouts and testing on the third, so read the penalty as indicative rather than calibrated. And the paper's prose says "the three models" while its table reports four; the three the range fits are LayoutLM, LayoutLMv2, and FormNet.

Two things in the underlying table matter more than the headline. The first is the fourth model. LayoutLMv3 scored 89.77 on mixed templates and 62.58 on unseen ones, a 27-point collapse, and trailed LayoutLMv2 at unseen templates at every training size. It is also LayoutLMv2's direct successor, and the most recently released of the four models VRDU tests. That doesn't prove newer models never help (VRDU tests no LLM-based extractor and nothing released after 2022) but it rebuts the assumption that each generation closes this gap on its own.

The second is the benchmark's other dataset. On the Ad-buy Form corpus, which has genuinely diverse templates with few documents each, seen and unseen performance are nearly identical, and both top out around 45% micro-F1. When every layout is rare there's no seen-template advantage left to lose, so the gap disappears because the ceiling has already come down. (The paper's prose describes that gap as "about 3 points"; the near-zero figures are what its own table reports at 200 documents.)

That's the specialty case, described in a paper that has never heard of specialty insurance.

‍

Run this on your own log

1. Pull 90 days of submissions.
2. Cluster them by source layout, not by broker and not by line. The same broker sending two different spreadsheet templates counts as two. Two brokers both sending the current ACORD 125 edition counts as one.
3. Sort layouts by submission count, descending.
4. Look at the shape of that distribution, not the totals.

If there's a head — a handful of layouts carrying most of the volume, then a tail — rules pay for the head and learned extraction catches the tail. The split is empirical: you draw the line where per-layout setup cost stops clearing the per-submission saving, and that point is specific to your team.

If there's no head — a flat distribution, long tail, nothing repeating often enough to amortise — rule-writing never repays. That's the Ad-buy shape, and it's where most specialty books land. It's also the quadrant where you measure your own vendor rather than trust a benchmark, since the 2022-era models VRDU tested reach only about 45% on that shape.

The distinction also disposes of the most-quoted number in this debate. A 2020 Celent report hosted by IVANS says "on average, 45% of small business is processed using straight-through processing." It attributes that to an unnamed underlying study rather than reporting it first-hand, never defines straight-through processing, and draws it from small business — the most bureau-form-driven segment there is, and a headed distribution. Specialty doesn't have a head. (The report was "commissioned by IVANS, a division of Applied Systems," though it adds that "the analysis and conclusions are Celent's alone.")

Where rules win outright

Where a field genuinely is standardised, rules win on cost by orders of magnitude.

Dennstädt and colleagues ran the comparison in 2025 over 7,764 radiology reports. On a hand-labelled 199-report subset, regular expressions scored 89.20% accuracy against 87.69% for a 14-billion-parameter open-weights LLM, with p = 0.56 — at that sample size, no detectable difference rather than demonstrated parity. Across the full corpus the regex approach ran "18,404.44 times faster." Their conclusion is careful, and the caveat is the finding: "for structured, standardized data like BI-RADS, traditional NLP techniques seem to be superior, though future work should explore hybrid approaches combining Regex precision for standardized elements with LLM contextual understanding."

BI-RADS is an ordinal score from a controlled vocabulary, in a report type with conventional phrasing. That's the maximally favourable case for pattern matching, and the opposite of a specialty submission. So this is not a result about rules beating AI, but a routing result: where the input is genuinely standardised, the cheap deterministic thing is no worse on the evidence available and four orders of magnitude faster, and putting a language model on it wastes money. Note the study is a medRxiv preprint and the model it tested was not a frontier one.

Where rules lose

They lose on manuscript wordings by definition: no bureau standard, so nothing dependable to anchor a position to.

They lose silently: a pattern that no longer matches returns nothing rather than an error, and a blank field reads downstream as an answer.

And they're widely said to lose through accumulation, as the set grows and rules interact until maintenance overtakes the benefit. The evidence for that is thinner than the confidence with which it's usually asserted.

The machine-learning side has a celebrated paper. Sculley and colleagues at Google published "Hidden Technical Debt in Machine Learning Systems" at NIPS 2015, which states the CACE principle flatly: "No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything." But that paper is about machine-learning systems, not rule engines, and it's frequently cited as though it demonstrated that rules rot. It doesn't. Its argument maps onto rule maintenance by analogy, and anyone citing it should say so.

The rules side has the 1992 verification literature and a large volume of vendor blog posts. Looking for empirical studies of rule-set growth curves or authoring cost in production, we found the academic work is mostly 1990s and 2000s and largely paywalled, and the rest is marketing. So the honest position: the maintenance case against large rule sets matches most practitioners' experience and is under-evidenced in the literature. "Studies show rule sets become unmaintainable" is a sentence we can't support. Preece and colleagues give you the vocabulary for what goes wrong, not a rate.

Where extraction loses

It may lose when you squeeze it into a schema, and that's worth measuring rather than assuming. Tam and colleagues, at the EMNLP 2024 Industry Track, found "a significant decline in LLMs' reasoning abilities under format restrictions," and that "stricter format constraints generally lead to greater performance degradation in reasoning tasks." Their conclusion is that the effect is task-dependent: strict formats hurt reasoning-heavy work and improve classification accuracy. They tested no document-extraction task, and extraction sits arguably nearer the classification side. So the JSON schema most pipelines impose may be free or costly, and it's cheap to test on your own data.

It loses when conformance gets reported as accuracy. Output can be schema-valid and entirely wrong; conformance and correctness are separate measurements, and a vendor quoting one isn't answering about the other.

And it may lose on confidence, the part nobody can quantify. The human-in-the-loop model assumes a confidence score stays meaningful on documents unlike the training set, which is exactly where calibration is generally worst. We looked for evidence on calibration of document-extraction confidence under distribution shift and found none, in either direction. That's not a criticism of any product. It's a question worth asking any vendor, ours included: what happens to your confidence scores on a layout you've never seen, and how do you know?

The case both handle worst

Schedule variability — the one of those four that didn't collapse into the ratio — isn't on the same axis as the other three. A 40-location schedule and a 40,000-location schedule can arrive on the identical template. This is about repeated structure of unknown depth, and it's where both approaches are weakest.

VRDU again, looking at FormNet on the Ad-buy Form corpus, mixed-template task: "there is a huge gap of 60 – 70 points across different sizes of training sets when comparing the micro-F1 score of hierarchical entities and other entities." The paper is candid that this partly reflects the baseline, since "no existing models take the hierarchical entity type into consideration" and a grouping heuristic stands in for one. Its verdict: "It is still an open question for future research how to properly extract the hierarchical entities from visually-rich documents."

A schedule of values is structurally that — one row per location, each carrying sub-fields, and nothing announcing how many rows there'll be. That's an analogy, not a transfer: VRDU contains no insurance documents, and the claim isn't that extraction scores 60 points worse on SOVs, only that a schedule has the shape the paper identifies as hardest. Rules lose here differently. A positional template built around a 200-row sheet doesn't fail at row 201; it just stops.

At scale it's worth reporting precisely. In our complex property SOV work, a top-ten global carrier processes schedules of 500 to 100,000 locations with up to 60 fields each, at field-level accuracy of 95% or better at launch rising to 97% within six months. Field-level is the honest unit for a document of that shape, and the number rose over six months rather than landing at go-live, which reads like post-launch tuning — though the case study doesn't say what drove it.

The decision framework

‍

‍

Submission volume is the numerator of the left-hand axis and non-standard forms are its denominator, so volume favors rules only when layouts repeat. Manuscript wordings drive that denominator to its limit. Schedule variability sits on the top axis instead. In a high-repeat book it splits the document, rules for the wrapper and extraction for the rows; everywhere else it sets your human review budget rather than your choice of approach.

‍

The hybrid that actually works

Two design rules follow, each the opposite of the obvious move.

Find out what your schema costs you. A strict output format is usually assumed to be a free guardrail. Tam's result says that it's task-dependent, and untested on extraction. So run the same documents twice, once as you do now and once with the schema loosened and rules moved downstream as a validator, and compare. Determinism is always cheaper applied to checking than to generating, but whether loosening buys accuracy is a question about your documents rather than a general law.

Let the guideline layer surface candidates and let people adjudicate. This isn't a theory. Our underwriting audit work for an insurance company supporting more than 100 MGAs runs both layers, in order: extraction pulls data from applications, supporting documents, and policy records; a comparison step checks it against each MGA's underwriting and rating guidelines; the output is "a structured report that clearly identifies matches and mismatches," and a person decides whether a discrepancy is a breach or a justifiable exception. Audit time fell from about 200 hours to roughly 110 per MGA, a 45% reduction — attributed specifically to automating the extraction half, "which previously consumed just under 50% of the time spent on any given audit." The comparison layer never adjudicates. It shortens the list a human reads.

Both work because of the distinction at the top: the rules in a good hybrid are business rules over structured data, not extraction patterns over raw documents.

What this looks like in specialty

McGowan Excess & Casualty, an excess and casualty underwriter, runs both layers in that order. Submissions arrive as ACORD forms, loss runs, SOVs, and supplemental applications; data is extracted into rating workbooks; carrier guidelines are then applied at clearance, with appetite fit and underlying limit adequacy. Steve Wentz describes it in the order it happens: FurtherAI "allowed us to map our current process and autofill our workbooks, process various carrier guidelines, and really get through a ton of information and uncover additional information that you need to underwrite." Autofill the workbook, then process the guidelines.

Novacore, an independent specialty provider managing more than $1.5 billion in premium across 20-plus programs, deployed the workspace as an orchestration layer across those programs rather than standardising them onto one workflow. A book assembled from many programs is a book assembled from many sets of conventions.

Upland Capital Group, an AM Best A− rated specialty P&C insurer, has FurtherAI ingest all broker submissions and extract the fields needed for underwriting and clearance. The common thread across these is specialty lines automation that treats extraction and decisioning as two layers with different failure modes, rather than one pipeline with one accuracy number.

The vendor question

None of the above tells you which product to buy, and this article deliberately doesn't rank any. If that's your question, our separate roundup of AI tools for commercial and specialty underwriting does the tool-by-tool work — and places us first, so read it as a vendor's view.

What the framework gives you is three better questions for those conversations. What's your field-level accuracy on a layout the system has never seen, as against your average? How does it behave on a 10,000-row schedule? And what happens to confidence scores out of distribution? A vendor who engages with that last one is telling you something about how they think. For E&S packets, our excess and surplus page covers mixed forms, handwriting, and email threads arriving together.

Frequently Asked Questions

‍

REFERENCES

^{Carnahan, Karlyn. "Commercial Lines New Business Submission." Celent, September 21, 2020. Commissioned by IVANS, a division of Applied Systems. ivans.com}

^{Chiticariu, Laura, Yunyao Li, and Frederick R. Reiss. "Rule-based Information Extraction is Dead! Long Live Rule-based Information Extraction Systems!" Proceedings of EMNLP 2013, 827–832. Association for Computational Linguistics. aclanthology.org}

^{Dennstädt, Fabio, Luc Lerch, Max Schmerder, et al. "A Comparative Performance Analysis of Regular Expressions and an LLM-Based Approach to Extract the BI-RADS Score from Radiological Reports." medRxiv preprint, June 2, 2025. doi.org/10.1101/2025.06.01.25328636}

^{FurtherAI. "Complex Property SOV Intake." furtherai.com}

^{FurtherAI. "McGowan Excess & Casualty Selects FurtherAI to Power Its Next Phase of Underwriting." furtherai.com}

^{FurtherAI. "Novacore Chooses FurtherAI to Power Underwriting Across Its Specialty Programs." furtherai.com}

^{FurtherAI. "Underwriting Audit." furtherai.com}

^{FurtherAI. "Upland Capital Group Chooses FurtherAI as Strategic AI Partner to Transform Underwriting." furtherai.com}

^{International Risk Management Institute. "Manuscript form or policy." Glossary of Insurance & Risk Management Terms. irmi.com}

^{Preece, Alun D., Rajjan Shinghal, and Aïda Batarekh. "Principles and practice in verifying rule-based systems." The Knowledge Engineering Review 7, no. 2 (June 1992): 115–141. cambridge.org}

^{Sculley, D., Gary Holt, Daniel Golovin, et al. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems 28 (NIPS 2015). papers.nips.cc}

^{Tam, Zhi Rui, Cheng-Kuang Wu, Yi-Lin Tsai, et al. "Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Large Language Model Performance." Proceedings of EMNLP 2024, Industry Track. aclanthology.org}

^{Wang, Zilong, Yichao Zhou, Wei Wei, Chen-Yu Lee, and Sandeep Tata. "VRDU: A Benchmark for Visually-rich Document Understanding." Proceedings of KDD '23, ACM. arxiv.org}

‍

DISCLAIMER 

This article is for general informational purposes only and does not constitute legal, regulatory, compliance, underwriting, or other professional advice. The content reflects information available as of the date of publication, and FurtherAI undertakes no obligation to update it as laws, regulations, or AI technologies evolve. 

‍