Structured data

extraction to JSON is not equivalent to correct extraction: establish field contracts for amount, unit, and source

hands the invoice or quotation to the parser and obtains a valid JSON, which only indicates that it can be read by the program.1200 is the unit price or the tot

阅读文本 / Reading text
hands the invoice or quotation to the parser and obtains a valid JSON, which only indicates that it can be read by the program. 1200 is the unit price or the total amount? Where is the currency? 1.200,50 How should it be interpreted according to regional conventions? Do the returned page numbers actually exist? These are not questions that JSON syntax can answer. This paper demonstrates a stricter boundary using a synthetic quotation fragment: fields must not only have values but also include units, source pages, the original text, and coordinates. The parser or model is responsible only for generating candidates; a deterministic validator determines whether they can be stored in the database. Missing and uncertain data should be explicitly retained and must not be silently filled with zeros. This is the actual test output of the following program, not a report on recognition accuracy. The original text of the sample input was handwritten by us; no OCR or model evaluation was performed on real documents. passes the value along with the evidence to field constraint Why is it necessary amount decimal string, with the example constraint that it be non-negative and have two decimal places avoids unintentionally introducing binary floating-point approximations currency This scenario is limited to USD should not secretly assume the Chinese yuan when no currency is specified source.page Starting from 1 and not exceeding the number of pages allows users to return to the original text source.quote must be the in-page fragment provided by the caller is visible to the reviewer based on source.bbox is a finite numerical value, located within the page range prevents empty frames and invalid coordinates from being masqueraded as evidence Here, the coordinate unit is PDF points, with the origin at the upper-left corner of the currently defined page. Upstream processes should first unify rotation, cropping, and the coordinate system; do not directly mix coordinates from different parsers. This example merely demonstrates the convention and does not prove that the text is indeed contained within the frame. Complete Validation and Rejection Test can be run using only the Python standard library. Decimal is constructed from a string; do not first convert it to float Pass it in again. For precise decimal handling of amounts, refer to Python decimal documentation 。 from copy import deepcopy from decimal import Decimal import math import re pages = [{"width": 595, "height": 842, "text": "Total: USD 1,200.50\nDelivery within 10 days"}] def validate(candidate, pages): if set(candidate) != {"amount", "currency", "source"}: raise ValueError("unexpected or missing field") raw = candidate["amount"] if not isinstance(raw, str) or not re.fullmatch(r"(?:0|[1-9][0-9]*)\.[0-9]{2}", raw): raise ValueError("amount must be a two-decimal string") if candidate["currency"] != "USD": raise ValueError("unsupported currency for this contract") src = candidate["source"] if not isinstance(src, dict) or set(src) != {"page", "quote", "bbox"}: raise ValueError("source contract mismatch") number = src["page"] if type(number) is not int or not 1 <= number <= len(pages): raise ValueError("page outside document") page = pages[number - 1] quote = src["quote"] if not isinstance(quote, str) or not quote or quote not in page["text"]: raise ValueError("source quote not found") match = re.fullmatch(r"Total: USD ([0-9]{1,3}(?:,[0-9]{3})*\.[0-9]{2})", quote) if not match or Decimal(match[1].replace(",", "")) != Decimal(raw): raise ValueError("amount does not agree with source") box = src["bbox"] if (not isinstance(box, list) or len(box) != 4 or any(type(v) not in (int, float) or not math.isfinite(v) for v in box)): raise ValueError("invalid box values") x0, y0, x1, y1 = box if not (0 <= x0 < x1 <= page["width"] and 0 <= y0 < y1 <= page["height"]): raise ValueError("box outside page or empty") return {"amount": str(Decimal(raw)), "currency": "USD", "source": deepcopy(src)} good = {"amount": "1200.50", "currency": "USD", "source": { "page": 1, "quote": "Total: USD 1,200.50", "bbox": [40, 70, 260, 90]}} assert validate(good, pages)["amount"] == "1200.50" print("PASS valid source-linked amount: USD 1200.50") bad_cases = [] wrong = deepcopy(good); wrong["amount"] = "12000.50"; bad_cases.append(wrong) wrong = deepcopy(good); wrong["source"]["page"] = 2; bad_cases.append(wrong) wrong = deepcopy(good); wrong["source"]["bbox"][2] = float("nan"); bad_cases.append(wrong) wrong = deepcopy(good); wrong["source"]["bbox"][0] = 300; bad_cases.append(wrong) for index, wrong in enumerate(bad_cases, 1): try: validate(wrong, pages) raise AssertionError("bad candidate accepted") except ValueError as error: print(f"PASS rejection {index}: {error}") The regularity is strict because the scenario is strict. is a regular expression that accepts only the English monetary format shown in the example. It rejects European-style decimals, negative numbers, refund amounts enclosed in parentheses, other currencies, and four-digit numbers without thousand separators. In a real-world system, contracts should be selected independently according to document type and region; one should not arbitrarily remove all commas, periods, and spaces. Before further expanding the input scope, first provide positive and negative examples for each permitted format. For multi-currency invoices, the fields should separately retain the original currency amount, the currency type, and the source of the exchange rate; without an exchange rate, the amount must not be secretly converted. “Having a source” does not equate to “having been proven” The program verifies that the quote exists within the text of the current page and that the amount matches this constrained format; however, it does not verify that the bounding box precisely encloses the original text, nor does it determine whether this line represents the true final total in business terms. The upstream text layer may originate from an erroneous OCR process, and multiple Totals may appear on the same page. Therefore, the field contract is only the first hurdle. Subsequent steps still require consistency verification, candidate disambiguation within the same page, determination of the relationship between subtotal and total tax amounts, as well as manual spot checks. Before passing through these stages, critical business data should be stored in a “to be verified” status rather than a “confirmed” status. Missing amounts, validation failures, and values that are genuinely zero represent three distinct states. How to implement it in document workflows retains the original document summary, assigns a processing version to each extraction record, and saves the reason for rejection. The translation process should also use the same audit fields: the main text may be translated into another language, but the correspondence between significant amounts and their units must not be lost. You can do this via OmniDoc Product Entry Understands the relevant documentation services; the contract and code in this paper are independent of the product API and do not claim to expose any undisclosed interfaces. The final deliverable should allow one to trace back from a field to the original page. JSON is the exchange format; only contracts, evidence, and acceptance criteria determine whether this data can be used. - 1 -
Updated 2026-10-09