# Best models for receipt data extraction?

**URL:** <https://discuss.huggingface.co/t/best-models-for-receipt-data-extraction/180538>\
**Category:** Beginners\
**Created:** [September 17, 2026, 3:16am UTC](https://discuss.huggingface.co/t/best-models-for-receipt-data-extraction/180538 "2026-09-17T03:16:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![hardyr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hardyr/32/63164_2.png) [@hardyr](https://discuss.huggingface.co/u/hardyr)\
**Post date:** [September 17, 2026, 3:16am UTC](https://discuss.huggingface.co/t/best-models-for-receipt-data-extraction/180538/1 "2026-09-17T03:16:08Z")

</div>

I wanted to make a web app that extracts data from pictures of receipts (or screenshots) in a standard format (total price, list of items purchased, company, date, etc.) for finance purposes. I am extremely new to HuggingFace and the ML community in general so I thought here would be a good place to ask. What are the options in terms of models, and if there isn’t one, how could I make one? I apologize if this question is a lot.

---

<div class="post-metadata">

**Author:** ![RDTvlokip](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rdtvlokip/32/62725_2.png) [@RDTvlokip](https://discuss.huggingface.co/u/RDTvlokip)\
**Post date:** [September 17, 2026, 12:22pm UTC](https://discuss.huggingface.co/t/best-models-for-receipt-data-extraction/180538/2 "2026-09-17T12:22:52Z")

</div>

Good news, there’s actually a model built specifically for this on the Hub: `AdamCodd/donut-receipts-extract`, fine-tuned on receipt data, outputs structured fields (total, items, phone, discount, etc.) directly, no separate OCR step needed since Donut is OCR-free (image straight to structured text). That’s the easiest starting point for exactly your use case.

If you want better accuracy on messy phone-camera photos (skew, bad lighting, crumpled receipts) rather than clean scans, a general vision-language model like Qwen2.5-VL-7B (or the OCR-focused fine-tune `syntheticbot/ocr-qwen`) handles that better since it’s more robust to real-world image quality, you just prompt it to extract fields as JSON. Donut is lighter/faster and purpose-built, VL models are heavier but more forgiving of bad photos, worth trying Donut first since it’s already tailored to receipts, and falling back to a VLM if your real-world photos are too noisy for it.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 17, 2026, 9:55pm UTC](https://discuss.huggingface.co/t/best-models-for-receipt-data-extraction/180538/3 "2026-09-17T21:55:09Z")

</div>

If using a model alone isn’t working well, it may be faster to think in terms of a pipeline:

* * *

Short answer: I would **not** start by training a receipt model from scratch.

For a first version, I would define the JSON shape you want and try a model that can read the receipt image and return that JSON directly. [NuExtract3](https://huggingface.co/numind/NuExtract3) looks like a practical starting point for this because its structured-extraction interface is built around a JSON template.

By “schema” here I just mean something like:

```json
{
  "company": "verbatim-string",
  "date": "verbatim-string",
  "subtotal": "verbatim-string",
  "tax": "verbatim-string",
  "total": "verbatim-string",
  "items": [
    {
      "description": "verbatim-string",
      "quantity": "verbatim-string",
      "unit_price": "verbatim-string",
      "line_total": "verbatim-string"
    }
  ]
}

```

For a first prototype, that can be as simple as:

```plaintext
receipt photo
    ↓
model
    ↓
JSON

```

You can stop there if it works well enough on the receipts your app will actually receive. I would only add more machinery when a particular type of error starts repeating.

### A practical starting route

```plaintext
define the JSON fields you need
        ↓
try direct image → JSON extraction
        ↓
test it on representative receipts
        │
        ├─ works well enough
        │ → build the first app version
        │
        ├─ it is misreading text/images
        │ → add OCR or a second vision model
        │
        ├─ values look plausible but were not actually printed
        │ → add an evidence/grounding check
        │
        ├─ line items are grouped incorrectly
        │ → retry/validate only the item block
        │
        └─ the same domain-specific mistake keeps repeating
               → then consider fine-tuning

```

A Qwen-family vision-language model (VLM — a model that can inspect an image and follow text instructions) is also worth testing. For example, [syntheticbot/ocr-qwen](https://huggingface.co/syntheticbot/ocr-qwen) is a Qwen2.5-VL-based model whose model card is specifically aimed at OCR/document use.

I would not assume in advance that one of these is universally better. The useful comparison is which one works on **your** stores, languages, layouts, screenshots/photos, and line-item formats.

### Keep extracted, normalized, and calculated values separate

This is probably the design choice I would make before comparing many models.

```plaintext
printed / extracted
!= normalized
!= derived / inferred

```

For example, if a receipt visibly contains:

```plaintext
TOTAL 88.000

```

I would first preserve:

```json
{
  "total_raw": "88.000"
}

```

and only later convert it into whatever numeric representation your finance application uses.

Likewise, if the receipt does **not** print a subtotal, I would prefer:

```json
{
  "subtotal_raw": null
}

```

rather than letting the extraction model calculate a plausible subtotal and put it into the same field.

If your application wants a calculated subtotal, keep that separately:

```json
{
  "subtotal_raw": null,
  "subtotal_derived": 76000
}

```

NuExtract3 has a documented [`verbatim-string`](https://huggingface.co/numind/NuExtract3/blob/main/TYPES.md) type specifically for extractive text, which is useful here because receipt punctuation and number formatting can be meaningful.

A recent receipt benchmark, [ReceiptBench](https://aclanthology.org/2026.acl-long.2135/), also treats perception, normalization, semantic reasoning, and structure parsing as distinct problems rather than one monolithic “OCR accuracy” problem.

### Runnable example

If a concrete example is useful, I also put a small T4/Colab reference notebook here:

[T4 receipt extraction / evidence demo](https://huggingface.co/datasets/John6666/forum3/blob/main/receipt_extraction/T4_RECEIPT_EXTRACTION_EVIDENCE_DEMO_RELEASE.ipynb)

It is only a reference implementation, not a benchmark.

You can stop at the simple route above. The sections below are mainly for cases where you want to compare options or make the extraction more reliable.

> **Model choices and when I would use each**
>
> ### NuExtract3: direct structured extraction
> 
> [NuExtract3](https://huggingface.co/numind/NuExtract3) is the one I would probably test first for this particular app shape.
> 
> Its documented structured-extraction path takes:
> 
> 1. the document — text, image, or both;
> 2. a JSON template describing what to extract;
> 3. optional instructions;
> 4. optionally, in-context examples.
> 
> That maps naturally to:
> 
> ```plaintext
> receipt image
> → company/date/total/items JSON
> 
> ```
> 
> For receipt fields where you care about exactly what was printed, I would start with `verbatim-string` rather than forcing every amount into a numeric type immediately.
> 
> The documented structured-extraction instructions also support missing leaf fields as `null` and missing arrays as `[]`, which is useful because “not printed” should be a valid state rather than something the model feels obliged to invent.
> 
> See:
> 
> - [NuExtract3 model card](https://huggingface.co/numind/NuExtract3)
> - [NuExtract3 field types](https://huggingface.co/numind/NuExtract3/blob/main/TYPES.md)
> 
> If local deployment becomes important later, NuMind also publishes several quantized NuExtract3 variants in its [NuExtract3 collection](https://huggingface.co/collections/numind/nuextract3). I would choose among those only after knowing your serving hardware.
> 
> ### Qwen / OCR-tuned Qwen
> 
> A Qwen model can be used in two different ways:
> 
> 1. ask it to extract the structured receipt directly; or
> 2. use it as a second reading of the image when the first model/OCR is uncertain.
> 
> For reliability, I find the second role particularly interesting.
> 
> For example:
> 
> ```plaintext
> receipt
> ├─ structured extractor
> └─ second OCR/VLM reading
> ↓
> compare evidence
> 
> ```
> 
> The [syntheticbot/ocr-qwen](https://huggingface.co/syntheticbot/ocr-qwen) model card describes OCR fine-tuning, structured document output, and text localization/bounding boxes.
> 
> I would treat that as a **candidate to test** , not proof that it will beat another model on your receipts.
> 
> ### Conventional OCR can still be useful
> 
> OCR means simply “read the text from the image.”
> 
> It does not have to be the whole solution.
> 
> Even if a VLM produces the final JSON, a conventional OCR pass can help answer questions such as:
> 
> ```plaintext
> Did the returned total actually appear in the receipt?
> Was there really a SUBTOTAL label?
> Did the extractor miss one product row?
> 
> ```
> 
> [Docling](https://docling-project.github.io/docling/concepts/OCR/) currently supports multiple OCR engines, including RapidOCR, EasyOCR, Tesseract variants, and others.
> 
> So one reasonable architecture is:
> 
> ```plaintext
> image
> ├─ model → structured JSON
> └─ OCR → source text / positions
> ↓
> validate
> 
> ```
> 
> I would not assume the OCR reconstruction itself is infallible either. Keeping the lower-level text/geometry available is useful when table or layout reconstruction groups things incorrectly.

> **Why a pipeline can be easier than searching for one perfect model**
>
> When something fails, I would first ask **what kind of failure it is**.
> 
> ### 1. The text itself is wrong
> 
> Example:
> 
> ```plaintext
> 48982.68
> 
> ```
> 
> is read as:
> 
> ```plaintext
> 48982.66
> 
> ```
> 
> That is mainly a perception/OCR problem.
> 
> Possible next steps:
> 
> - another OCR engine;
> - a document-oriented VLM;
> - better image capture/preprocessing.
> 
> ### 2. The text is readable, but the model returns something that is not printed
> 
> Example:
> 
> ```plaintext
> subtotal = 154.08
> 
> ```
> 
> even though the receipt never printed a subtotal.
> 
> That is a grounding/extraction-contract problem rather than ordinary OCR.
> 
> I would require evidence for important extracted fields:
> 
> ```plaintext
> returned value
> → should be traceable to visible receipt text
> 
> ```
> 
> If it is not, leave it unresolved or retry that field.
> 
> ### 3. The text is correct, but line items are grouped incorrectly
> 
> For example:
> 
> ```plaintext
> Coffee
> 2 x 4.50
> 9.00
> 
> ```
> 
> may be three visual rows but one purchased item.
> 
> A model can read all three rows correctly and still build the wrong JSON structure.
> 
> That is a structure problem.
> 
> The useful response is usually not “replace the entire model immediately,” but something like:
> 
> ```plaintext
> retry the item block
> or
> use layout/OCR evidence to regroup it
> 
> ```
> 
> ### 4. The extraction is correct but normalization is wrong
> 
> For example:
> 
> ```plaintext
> 88.000
> 
> ```
> 
> can mean different things depending on locale/context.
> 
> This is why I would keep the printed string first and normalize later.
> 
> ### 5. A value is being calculated rather than extracted
> 
> Arithmetic can be useful as a validation signal:
> 
> ```plaintext
> quantity × unit_price ≈ line_total
> 
> ```
> 
> but arithmetic cannot prove that a subtotal was actually printed.
> 
> So I would treat arithmetic as:
> 
> ```plaintext
> consistency check
> 
> ```
> 
> not:
> 
> ```plaintext
> source evidence
> 
> ```
> 
> This kind of decomposition is also reflected in [ReceiptBench](https://aclanthology.org/2026.acl-long.2135/), which splits receipt understanding into perception, format normalization, semantic reasoning, and structure parsing.

> **Line items are the part I would test most carefully**
>
> Header fields such as merchant/date/total are often simpler key/value fields.
> 
> Purchased items are where receipts become more awkward.
> 
> You can see patterns such as:
> 
> ```plaintext
> product name
> quantity × unit price
> line total
> 
> ```
> 
> or:
> 
> ```plaintext
> product
> size/modifier
> discount
> price
> 
> ```
> 
> and those may span multiple visual rows.
> 
> The [CORD receipt dataset](https://github.com/clovaai/cord) is a useful example of why this matters. Its receipt annotations distinguish fields such as:
> 
> - item name;
> - quantity;
> - unit price;
> - total item price;
> - discount;
> - sub-items/modifiers;
> 
> and also include row/group hierarchy.
> 
> I would therefore evaluate line items separately from header fields.
> 
> Useful checks include:
> 
> ```plaintext
> item count
> missing items
> duplicate items
> description match
> quantity
> unit price
> line total
> 
> ```
> 
> A model that gets merchant/date/total perfectly can still be unacceptable for a finance app if it quietly drops one purchased item.
> 
> CORD is useful as a structural test/reference, but it is Indonesian receipt data, so I would still create a holdout from the receipts your application is actually intended to process.
> 
> Links:
> 
> - [CORD repository](https://github.com/clovaai/cord)
> - [CORD mirror used by the example notebook](https://huggingface.co/datasets/Heliosoph/CORD)

> **Why I would keep a validation/fallback path**
>
> I did a few small local diagnostics while looking at this. They are **not a benchmark** , but three observations were consistent enough to affect how I would design the app.
> 
> ### Plausible does not always mean printed
> 
> Structured models can produce valid, numerically plausible JSON while still filling a field that was not visibly present.
> 
> So:
> 
> ```plaintext
> valid JSON
> !=
> grounded extraction
> 
> ```
> 
> and:
> 
> ```plaintext
> arithmetic plausibility
> !=
> proof that the field was printed
> 
> ```
> 
> ### Line-item grouping can fail independently of OCR
> 
> The model may read every token correctly but associate a quantity/price/modifier with the wrong item.
> 
> That is why I would test items separately.
> 
> ### Different reading methods can make different mistakes
> 
> A conventional OCR engine and an OCR-oriented VLM can miss different parts of the same receipt.
> 
> That makes this kind of conditional path useful:
> 
> ```plaintext
> cheap/default path
> ↓
> is important evidence missing?
> │
> ├─ no → accept
> │
> └─ yes → stronger second OCR/VLM pass
> 
> ```
> 
> The point is not to run two large models on every image. The point is to have a second route available for the receipts where the first route is uncertain.

> **How I would evaluate it before building the web app around it**
>
> I would make a small holdout from receipts that resemble the actual input to the app.
> 
> Try to include the variation you expect:
> 
> ```plaintext
> different merchants
> different countries/languages
> screenshots and phone photos
> clean and angled/blurred images
> simple and multiline items
> cash/card receipts
> different tax/service-charge formats
> 
> ```
> 
> Then measure different things separately.
> 
> ### Header fields
> 
> For example:
> 
> ```plaintext
> company
> date
> total
> subtotal
> tax
> cash/change/payment
> 
> ```
> 
> ### Items
> 
> For example:
> 
> ```plaintext
> item count
> missing items
> duplicate items
> description
> quantity
> unit price
> line total
> 
> ```
> 
> ### Grounding
> 
> Also track:
> 
> ```plaintext
> value was printed and extracted
> value was printed but omitted
> value was emitted but not printed
> 
> ```
> 
> That last category is especially important for finance-oriented extraction.
> 
> I would keep a simple error table such as:
> 
> ```plaintext
> receipt
> field/item
> expected raw text
> predicted raw text
> failure type
> which model/path produced it
> 
> ```
> 
> After perhaps 10 or so representative receipts, you will often already see which error class dominates. That is only an exploratory set, not a statistically meaningful benchmark, but it is usually enough to decide what to investigate next.
> 
> Then branch:
> 
> ```plaintext
> mostly OCR failures
> → try a different OCR/VLM reading path
> 
> mostly unsupported fields
> → add stronger source-evidence checks
> 
> mostly item grouping
> → improve item parsing/retry
> 
> same merchant/layout mistake repeatedly
> → fine-tuning may now be worth it
> 
> ```
> 
> If you want a more formal receipt-oriented evaluation reference, [ReceiptBench](https://aclanthology.org/2026.acl-long.2135/) and its [code](https://github.com/wwwT0ri/ReceiptBench) are useful. Its evaluation separates different receipt tasks rather than hiding everything inside one score.

> **When I would consider fine-tuning**
>
> There are two different ideas that are easy to mix up when starting out:
> 
> ```plaintext
> training from scratch
> 
> ```
> 
> means building the model’s capabilities through a very large training process.
> 
> ```plaintext
> fine-tuning
> 
> ```
> 
> means starting from an existing model and teaching it additional examples from your own task.
> 
> For this receipt app, if training becomes necessary at all, I would almost certainly look at **fine-tuning an existing model** , not training a vision-language model from scratch.
> 
> My decision tree would be:
> 
> ```plaintext
> Does an existing model handle most target receipts?
> |
> +-- yes
> | |
> | +-- failures are inconsistent/random
> | | → improve pipeline/evidence first
> | |
> | +-- the text itself is unreadable
> | | → improve OCR/perception first
> | |
> | `-- the same mapping/layout mistake repeats
> | → fine-tuning becomes interesting
> |
> `-- no
> |
> +-- model cannot read the document
> | → perception/OCR/data problem
> |
> `-- model reads it but systematically maps it wrong
> → supervised fine-tuning is plausible
> 
> ```
> 
> A training example would roughly contain:
> 
> ```plaintext
> receipt image
> +
> instruction / desired JSON shape
> +
> correct structured output
> 
> ```
> 
> I would make sure the training targets already reflect the contract you want:
> 
> - printed values remain raw;
> - missing fields are actually missing/null;
> - derived values are not disguised as extracted ones;
> - difficult multiline items are represented consistently.
> 
> If possible, also avoid making train/test splits where nearly identical merchant templates occur on both sides. Holding out merchants/layouts can tell you much more about whether the model generalizes.
> 
> You also do not necessarily need to update every parameter in the model.
> 
> Techniques such as **LoRA** train a much smaller set of parameters while leaving most of the base model frozen. Hugging Face documents this through [PEFT](https://huggingface.co/docs/peft/index) and its [LoRA guide](https://huggingface.co/docs/peft/main/conceptual_guides/lora).
> 
> For vision-language models specifically, the current [TRL SFTTrainer](https://huggingface.co/docs/trl/sft_trainer) supports datasets containing an `image` or `images` field, so there is an established Hugging Face path if you eventually reach this stage.
> 
> There is also a worked [VLM fine-tuning recipe](https://huggingface.co/learn/cookbook/fine_tuning_vlm_trl).
> 
> But I would only invest in this after the simple baseline tells you **what repeated failure you are actually trying to train away**.

> **A few production notes**
>
> For a finance-oriented app I would keep some traceability around important fields.
> 
> For example:
> 
> ```plaintext
> raw extracted value
> source/evidence used
> normalization result
> derived result, if any
> 
> ```
> 
> That makes it much easier to inspect a suspicious transaction later.
> 
> A conditional fallback is also useful operationally:
> 
> ```plaintext
> normal/cheap path
> ↓
> accept if sufficiently supported
> ↓ otherwise
> stronger model/OCR path
> 
> ```
> 
> so the more expensive model does not need to run for every receipt.
> 
> And since receipts can contain addresses, payment fragments, loyalty/customer identifiers, and purchase history, check the retention/storage/privacy behavior of whatever hosted inference service you eventually use.

If I were building version 1, I would therefore do this:

```plaintext
1. Choose the small set of JSON fields the app actually needs.
2. Try NuExtract3 directly on representative receipts.
3. Keep printed/raw values separate from normalized/calculated values.
4. Check header fields and line items separately.
5. Add OCR/Qwen evidence only for the failure types that actually appear.
6. Fine-tune only if a stable target-domain error remains.

```

That should get you to a useful prototype without requiring you to solve “receipt AI” all at once.

---

<div class="post-metadata">

**Author:** ![negdev](https://avatars.discourse-cdn.com/v4/letter/n/a6a055/32.png) [@negdev](https://discuss.huggingface.co/u/negdev)\
**Post date:** [September 18, 2026, 5:40am UTC](https://discuss.huggingface.co/t/best-models-for-receipt-data-extraction/180538/4 "2026-09-18T05:40:41Z")

</div>

> [@John6666](#):
>
> If using a model alone isn’t working well, it may be faster to think in terms of a pipeline:
> 
> * * *
> 
> Short answer: I would **not** start by training a receipt model from scratch.
> 
> For a first version, I would define the JSON shape you want and try a model that can read the receipt image and return that JSON directly. [NuExtract3](https://huggingface.co/numind/NuExtract3) looks like a practical starting point for this because its structured-extraction interface is built around a JSON template.
> 
> By “schema” here I just mean something like:

# ===========================

JSON SCHEMA (Receipt Extraction)

{  
“store\_name”: “”,  
“address”: “”,  
“date”: “”,  
“time”: “”,  
“items”: [  
{  
“product”: “”,  
“quantity”: “”,  
“unit\_price”: “”,  
“total\_price”: “”  
}  
],  
“subtotal”: “”,  
“tax”: “”,  
“total”: “”,  
“payment\_method”: “”  
}

# ===========================

OPTIMIZED PROMPT (NuExtract3 / HF)

You are an extraction model. Read the receipt image and return ONLY valid JSON following the schema below.  
Do not add text, comments, explanations, or extra fields.  
If a field is missing in the receipt, return an empty string for that field.

JSON schema:  
{  
“store\_name”: “”,  
“address”: “”,  
“date”: “”,  
“time”: “”,  
“items”: [  
{  
“product”: “”,  
“quantity”: “”,  
“unit\_price”: “”,  
“total\_price”: “”  
}  
],  
“subtotal”: “”,  
“tax”: “”,  
“total”: “”,  
“payment\_method”: “”  
}

Extract all visible information from the receipt image and fill the JSON accordingly.  
Return ONLY the JSON object.

# ===========================

FULL PIPELINE (Conceptual)

1. INPUT

2. PREPROCESSING

3. MODEL CALL (NuExtract3 or similar)

4. VALIDATION

5. POST-PROCESSING

6. OUTPUT

7. FEEDBACK LOOP

# ===========================

OPTIONAL MESSAGE TO USERS

If you try this pipeline, I would really appreciate any feedback about your experience.  
Your feedback helps improve the extraction accuracy and makes the tool more useful for everyone.