Benchmarking real-world document extraction

An evaluation of how vision-capable models turn proprietary, private invoices, receipts, and machine-generated documents into reliable, structured data.

494

Documents

Real-world invoices

162

Supplier types

Unique layouts

Input formats

Digital, scan, handwriting

80%

Challenging inputs

Scanned or handwritten

01 / 04 Results

Compare model performance

Compare extraction accuracy across document formats with the cost of processing the complete benchmark.

Exact-match accuracy

Switch formats to see how model performance changes.

Higher is better

Full benchmark run cost

Calculated from measured usage across all 494 documents.

Lower is better

For API models, cost = input tokens at the input rate + output and reasoning tokens at the output rate. InvoiceIQ cost: InvoiceIQ took 5.7 minutes to complete the benchmark. At $1.50/hour for an NVIDIA L40S instance, the cost is $0.14. Gemini 3.7 Flash cost: Gemini 3.7 Flash’s $1.50 cost uses introductory pricing of $0.75 per million input tokens and $3.75 per million output and reasoning tokens. At the actual rates of $1.50 and $7.50, the same run costs $3.00.

Benchmark accuracy and cost by model and image type
Model	Overall	Digital	Scanned	Handwritten	Cost

Every model was evaluated using the highest reasoning setting available for that model.

02 / 04 Methodology

One image in, one strict record out

Each model receives the same invoice-extraction task. A response passes only when every scored field matches the reference record.

Scored fields

Invoice number Invoice date VAT Total incl. tax

Evaluation rules

Dates use a normalized day–month–year format.
Invoice numbers are compared without internal spaces.
Numeric values require exact equality.
Failed requests and invalid responses stay in the denominator.

03 / 04 Analysis

Performance depends on the document mix

Overall accuracy tells only part of the story. The document format and the field being extracted both change where models succeed—and where human review is still needed.

A /

Digital documents

Clean layouts reveal the extraction ceiling with less noise from image quality and handwriting.

B /

Scanned documents

Compression, skew, shadows, and scan quality test whether a model remains dependable in common workflows.

C /

Handwritten documents

Free-form writing and inconsistent structure expose the largest differences in model robustness.

The weakest field changes with the input

Across all eleven models, VAT is the most reliable field. Invoice numbers trail the other fields, making them the largest source of field-level errors overall.

VAT: 95.0%
Total including tax: 90.1%
Invoice date: 86.7%
Invoice number: 80.5%

Accuracy shifts by format

Each input type creates a different extraction bottleneck.

91%+

Digital documents

Every scored field remains above 91% accuracy.

74.7%

Scanned documents

Invoice-number accuracy falls as blur, skew, and dense layouts make similar identifiers harder to separate.

67.3%

Handwritten documents

Dates become the weakest field, even as invoice-number accuracy recovers to 82.4%.

Where errors come from

Most misses fall into three recurring patterns.

Misidentified numbers

Models may select an account, order, or reference number, or change one character in a long identifier.

Date confusion

Most misses contain a wrong day, month, or year rather than simply using the wrong date format.

Competing amounts

VAT, subtotal, balance due, and grand total can be confused. Of all VAT errors, 42% occur when the correct value is zero.

Because the benchmark uses exact matching, one mistaken character or one incorrect field is enough to fail the complete record.

04 / 04 About InvoiceIQ

Madhi AI InvoiceIQ

InvoiceIQ is a supervised fine-tune of Qwen3-VL 8B Instruct, built to turn the base model’s extraction capability into consistent exact-match performance.

Why this base model

Among the open base models evaluated during model selection, Qwen3-VL 8B Instruct delivered the strongest extraction accuracy while remaining compact enough to specialize.

What pass@64 revealed

When each document was attempted 64 times, the model often produced an exact answer at least once—even where single-pass accuracy was weaker. That gap signaled capability that could be made more reliable through fine-tuning.

Fine-tuning made the capability repeatable

InvoiceIQ underwent supervised fine-tuning on a specially curated corpus of 8,500 documents. The limited supplier overlap between training and evaluation also tests whether the model can generalize to unfamiliar layouts.

+32.18 pp
Overall accuracy uplift: 25%
Supplier overlap: 82.59%
Scanned accuracy

Fine-tuning lifted InvoiceIQ to 80.16% overall accuracy—within 1.22 percentage points of Gemini 3.1 Pro—and to 82.59% on scanned documents, ahead of Gemini 3.1 Pro’s 80.57%. The result suggests that the pass@64 headroom could be converted into dependable single-pass extraction without sacrificing generalization.

If you’re interested in deploying InvoiceIQ or learning more about our document-extraction work, visit madhi.ai.