← All lessons
Working with financial documents · Lesson 2 of 7

Getting clean text

Every later step depends on this one. If the text is wrong, the model extracts the wrong value, confidently. So look at what your extraction actually produces before tuning anything else.

A PDF made by software usually has a text layer: the characters are stored in the file and can be pulled out directly, exactly. A scanned or photographed page is only an image, so it needs OCR (optical character recognition) or a model that reads images. Many files are a mix: a typed agreement with scanned signature pages and exhibits. Decide page by page. A page with almost no extractable characters is probably a scan:

pages = [{"n": 1, "chars": 2140}, {"n": 2, "chars": 1985}, {"n": 3, "chars": 12}, {"n": 4, "chars": 0}]
needs_ocr = [p["n"] for p in pages if p["chars"] < 50]
print(needs_ocr)
Prints
[3, 4]

OCR makes mistakes a text layer doesn't: 1 and l, 0 and O, 5 and S, a lost decimal point. Most OCR tools give a confidence for each word; keep it, because a low score on a number is a good reason to send that value to a person. Skewed, faint or low-resolution scans make all of this worse.

Layout matters as much as characters. Plain text from a two-column page can interleave the columns line by line, and a table can come out as a jumble of numbers with no idea which column each belongs to. Layout-aware extraction (from a PDF library, an OCR service or a model that reads the page image) gives you reading order, paragraphs and table cells. Check tables especially: a shifted column in a rent roll puts every rent against the wrong unit.

Keep position metadata with the text: the page each line came from and its position on the page (a bounding box). You'll need both to show a reviewer exactly where a value came from. Store the PDF's own page index as well as any page number printed on the page; "page 37" in the footer is often not the 37th page of the file.

Long files need a plan. A 300-page credit agreement may not fit in a model's context window, and even when it does, answers get worse and cost more when you send everything. Split it along its structure (articles, sections, schedules, using the table of contents and headings), find the sections each field lives in (the margin is in the interest section, the covenants in their own article, the definitions at the front), and send only those, keeping the page map so every value still points back to its page.

Practice

Pulling text from a scanned appraisal returns nothing. Why?

More info

Quick reference

Text layerCharacters stored in the PDF; exact when present.
OCRTurning an image of text into text; makes 1/l, 0/O mistakes.
OCR confidenceA score per word; low scores on numbers deserve review.
Reading orderWhich line comes next; columns can get interleaved.
Bounding boxA line or word's position on the page, for highlighting.
Page index vs. printed pageStore both; they often differ.

The other AI and data engineering courses