Ch.3: Getting text out of documents

Text extraction from HTML, EPUB and PDF, ligatures and Unicode, OCR, and formulas. Written from the sources, with real examples from extracting textbooks.

36 flashcards · Computer Science · by lakesare

Part of Public › PDF vs EPUB: file formats explained · 3 courses

[Copying from web pages and EPUBs] Why does copying text from a web page or an EPUB e-book usually give clean text?
The file stores the text itself, in reading order; the lines on your screen are made only when the page is shown. HTML 2.0 (1995) says it this way: the browser takes the characters of a paragraph, splits them into words at spaces and line ends, and breaks the words into lines that fit the window. So a line break in the file is not a line break on the screen.Source: RFC 1866 (HTML 2.0), §6 Characters, Words, and Paragraphs
[Copying from web pages and EPUBs] A web page shows a warning triangle as a picture. Can a program that extracts the page's text get any text for that picture?
Yes, if the page gives one: the ALT text ("alternative text") of the image is the text to use in place of the picture. Example: <IMG SRC="triangle.xbm" ALT="Warning:"> gives the text "Warning:". HTML 2.0 names reasons: the browser cannot show images, or the user turned them off.Source: RFC 1866 (HTML 2.0), §5.10 Image: IMG
[Why a PDF is different] A PDF page looks like neatly typeset paragraphs. What does the PDF file actually store about its text?
Only the characters and where each one is placed on the page: nothing that marks words, sentences or paragraphs. The pdfminer.six documentation: unlike a .txt file or a Word document, a PDF contains no stream of text.Source: pdfminer.six documentation, "Converting a PDF file to text"
[Why a PDF is different] Is there a space character between "Operations" and "on" in the heading "Operations on Sets" of Epp's "Discrete Mathematics with Applications" (PDF)?
No. The PDF only leaves an empty strip, 3.7 points wide, between the "s" and the "o"; the program that extracts the text has to insert the space itself. The pdfminer.six documentation says the PDF format has no notion of the space character, so pdfminer.six inserts a space wherever two letters are far enough apart.(1 point = 1/72 inch.) Measured with pdfminer.six on this book.Source: pdfminer.six documentation, "Converting a PDF file to text"
[Why a PDF is different] A PDF stores letters at positions. How does a text extraction program make words and lines out of them?
It guesses from the distances: letters that are close together form a word, a slightly larger gap becomes a space, and a letter further down starts a new line. Lines that are close one below the other are then grouped into blocks of text, such as paragraphs.Source: pdfminer.six documentation, "Converting a PDF file to text"
[Why a PDF is different] A text extraction program decides from the distances between letters which letters belong to one word and one line.What happened when pdfminer.six was told to treat only very close letters as belonging to the same line, on a page of Sermutlu's "Automata, Formal Languages, and Turing Machines"?
Every word became a line of its own, and the words came out in a mixed order. Example: the page began "of / a / language / that / is / Turing- / 12.3 / ATM / and / HALTTM / an / give ...", although the section heading "12.3 ATM and HALTTM" is printed above that sentence.The distances on a page are the only clue, and one setting does not suit every page.Seen when extracting this book with pdftotext and pdfminer.six, two free text extraction programs.
[Why a PDF is different] Can a PDF itself say what the reading order is, and which text is a page number or a running head (the title line at the top of each page)?
Yes, if it is a Tagged PDF: it also stores the structure and reading order, and marks running heads and page numbers as text to skip. Tagged PDF is optional. Neither Epp's "Discrete Mathematics with Applications" nor Sermutlu's "Automata, Formal Languages, and Turing Machines" is tagged (pdfinfo prints "Tagged: no").Source: PDF Reference 1.7, §10.7 Tagged PDF
[Why a PDF is different] Do two text extraction programs give the same text from the same PDF?
Not always: each program guesses differently. Example: on Sermutlu's "Automata, Formal Languages, and Turing Machines", pdftotext gave "Define" with the two letters f, i, while pdfminer.six gave "Deﬁne" with the single character ﬁ.Example: pdftotext joined "Turing-" at a line end with "recognizable" on the next line into "Turingrecognizable"; pdfminer.six kept "Turing-", a line break, "recognizable".Seen when extracting this book with pdftotext and pdfminer.six, two free text extraction program…
[Classic PDF copying glitches] What extra text appears in the middle of the text that pdftotext extracts from page 369 of Epp's "Discrete Mathematics with Applications"?
The running head "6.1 Set Theory: ...", the page number "341", the photo credit "Stock Montage" and the caption "Giuseppe Peano (1858–1932)". The caption landed between "Symbolically:" and the formulas it introduces. At the end of the page comes the publisher's copyright notice.Seen when extracting this book with pdftotext and pdfminer.six, two free text extraction programs.
[Classic PDF copying glitches] Why do page numbers, running heads and picture captions end up in the middle of text copied from a PDF?
To the PDF they are just more characters at positions on the page, no different from the characters of a paragraph. The pdfminer.six documentation: the characters of a paragraph look the same as those of a table, a page footer or a figure caption.A Tagged PDF can mark running heads, page numbers and footers as "artifacts" (decoration, not content), so that a program, such as one that reads the page aloud, can skip them.Source: pdfminer.six documentation, "Converting a PDF file to text"; PDF Ref…
[Classic PDF copying glitches] You extract the text of Epp's "Discrete Mathematics with Applications" and search it for the word "Quantified". It appears 60 times in the book. How many results do you get?
None: in the extracted text, "fi" is the single character ﬁ (a ligature), not the two letters f and i. The whole book contains this ﬁ character 5120 times.Seen when extracting this book with pdftotext and pdfminer.six, two free text extraction programs.
[Classic PDF copying glitches] What is a ligature in typesetting?
Two or more letters drawn as one fused shape. Example: in ﬁ the dot of the i merges with the top of the f.A digraph, in contrast, is two letters placed close together but still separate, such as "ch" in Slovak.Source: Unicode FAQ, Ligatures and Digraphs
[Classic PDF copying glitches] Is "fi" drawn as a ligature because of the text or because of the font?
Because of the font: the text holds the letters f and i, and a font that has a ﬁ shape may draw them as one. Unicode does contain a single character ﬁ, but only for compatibility with older character sets, and its use is discouraged. Some fonts have no ligatures at all; fonts for non-Latin scripts can have hundreds.Source: Unicode FAQ, Ligatures and Digraphs
[Classic PDF copying glitches] Extracted text contains the single character ﬁ instead of the letters f, i. How can a program repair it?
With Unicode compatibility normalization (NFKC), a standard clean-up step that replaces ﬁ by the plain letters f, i. Example: in the text extracted from Epp's "Discrete Mathematics with Applications", "Quantified" occurs 0 times before this step and 60 times after it.A PDF can prevent the problem by including a table that says which letters each shape stands for, e.g. that the ﬁ shape means "fi".Seen when extracting this book (normalization done with Python's unicodedata). Source: PDF Reference…
[Classic PDF copying glitches] Sermutlu's "Automata, Formal Languages, and Turing Machines" prints the word "Turing-recognizable" split across two lines: "Turing-" at the end of one line, "recognizable" at the start of the next. What did pdftotext make of it?
"Turingrecognizable": the hyphen was removed. pdftotext joins words that are split at a line end and drops the hyphen. It cannot tell a hyphen added only to split the word from a hyphen that belongs to the word.A Tagged PDF must write the splitting hyphen as a special "soft hyphen" character, so that programs need not guess.Seen when extracting this book with pdftotext and pdfminer.six, two free text extraction programs. Source: PDF Reference 1.7, §10.7, Incidental Artifacts
[Classic PDF copying glitches] Epp's "Discrete Mathematics with Applications" has a two-column table: subsets of XXX without zzz , next to subsets of XXX with zzz . What does pdftotext make of it?
By default, one table cell per paragraph, so the rows are lost. With the option -layout, each row stays on one line. With -layout, pdftotext keeps the horizontal positions of the page by filling in spaces.Seen when extracting this book with pdftotext and pdfminer.six, two free text extraction programs. Source: man pdftotext; Poppler
[Classic PDF copying glitches] In Epp's "Discrete Mathematics with Applications", the formula " x∉Ax \notin Ax∈/A " (" xxx is not in AAA ") comes out of pdftotext as "x ∈", a line break, then "/ A". Why?
The PDF draws ∉ as two separate characters at the same spot: ∈ and a slash "/" on top of it. The two come from two different fonts. The extracted text shows the ∈ and the slash as two characters, and here even puts the slash on the next line.Measured with pdfminer.six on page 369 of this book: the ∈ and the slash start 1 point apart.Seen when extracting this book with pdftotext and pdfminer.six, two free text extraction programs.
[Scanned PDFs and OCR] A PDF was made by scanning a paper book. What text can you copy from it?
None, unless OCR has been run on it: each page is only a picture of the paper, with no characters in it. To get text, a program must look at the picture and recognize the letters.Source: Ray Smith, An Overview of the Tesseract OCR Engine (2007)
[Scanned PDFs and OCR] A program that looks at a picture of text and recognizes the letters in it does OCR (optical character recognition).
Example: Tesseract is a free OCR program. Its input is a black-and-white picture of a page, its output is text.Source: Ray Smith, An Overview of the Tesseract OCR Engine (2007)
[Scanned PDFs and OCR] Where and when was the OCR program Tesseract developed, and when did it become free (open source)?
At Hewlett-Packard (HP) from 1984 to 1994; HP released it as open source in late 2005. It began as a PhD project at HP Labs in Bristol. Its author compares it to a supernova: it appeared from nowhere in the 1995 UNLV test of OCR accuracy, did very well against commercial programs, then vanished again until 2005.Source: Ray Smith, An Overview of the Tesseract OCR Engine (2007)
[Scanned PDFs and OCR] In which steps does Tesseract turn a picture of a page into text?
Find the dark shapes, group them into lines, split the lines into words, then recognize the words in two passes. Source: Ray Smith, An Overview of the Tesseract OCR Engine (2007)
[Scanned PDFs and OCR] Why does Tesseract read each page twice?
In the first pass, words it recognized with confidence teach it the font of this page; the second pass reads the uncertain words again with that knowledge. Without a second pass, what it learned lower down the page would come too late for the words at the top.Source: Ray Smith, An Overview of the Tesseract OCR Engine (2007)
[Scanned PDFs and OCR] Why are the spaces between words hard to find in a picture of a page?
A gap inside a word can be as wide as a gap between words, and some words touch with no gap at all. Example: in "11.9%", the gap between the two digits 1 is about as wide as a word space.Tesseract marks spaces it is unsure about as "fuzzy" and decides them only after it has recognized the words. A program extracting text from a PDF has the same problem with gaps.Source: Ray Smith, An Overview of the Tesseract OCR Engine (2007)
[Scanned PDFs and OCR] How does Tesseract read white letters on a black background?
As easily as black on white: it looks at which outlines lie inside which other outlines, and from that it can tell that the text is inverted. Its author: Tesseract was probably the first OCR program that handled white-on-black text this simply.Source: Ray Smith, An Overview of the Tesseract OCR Engine (2007)
[Scanned PDFs and OCR] A scanned book page is slightly tilted, and its lines curve near the book's spine. Does Tesseract straighten the picture first?
No: it follows each tilted line across the page and fits a curved baseline (the line the letters sit on) to it. Not straightening the picture avoids making it blurrier. Curved lines are common in scans, not only near the spine.Source: Ray Smith, An Overview of the Tesseract OCR Engine (2007)
[Copying formulas] Epp's "Discrete Mathematics with Applications" states: "if a set XXX has nnn elements, then P(X)\mathscr{P}(X)P(X) has 2n2^n2n elements". What does pdftotext give?
"then P(X ) has 2n elements": the exponent nnn became an ordinary n after the 2. So "2 to the power n" reads as "two n". On the same page, 202^020 came out as "20".Seen when extracting this book with pdftotext and pdfminer.six, two free text extraction programs.
[Copying formulas] Why does 2n2^n2n copied from a PDF come out as "2n"?
The PDF stores "2" and "n" as two characters with positions; only the smaller, raised n shows it is an exponent, and plain text cannot show that. A PDF knows only the characters and their placement, so the structure of a formula is lost.Source: pdfminer.six documentation, "Converting a PDF file to text"
[Copying formulas] The heading " ATMA_{\mathrm{TM}}ATM and HALTTM\mathrm{HALT}_{\mathrm{TM}}HALTTM " (with small lowered TM) in Sermutlu's "Automata, Formal Languages, and Turing Machines": what does pdftotext make of it?
"ATM and HALTTM": each lowered TM is joined to the word before it. Example: in Epp's "Discrete Mathematics with Applications", AcA^cAc ("the complement of AAA ") comes out as "Ac".The Unicode FAQ gives the same kind of example: H with a lowered e, written with formatting, becomes "He" in plain text.Seen when extracting this book with pdftotext and pdfminer.six, two free text extraction programs. Source: Unicode FAQ, Presentation Forms
[Copying formulas] Why doesn't Unicode simply have raised and lowered versions of every character, so that 2n2^n2n could stay 2n2^n2n in plain text?
Unicode leaves raised and lowered text to formatting or markup; its few superscripts (such as ⁿ) exist for old character sets and phonetics. Some layouts cannot be single characters at all: a superscript and a subscript on the same letter, or a superscript inside a superscript.Source: Unicode FAQ, "Why doesn't Unicode have a full set of superscripts and subscripts?"
[Copying formulas] Sermutlu's "Automata, Formal Languages, and Turing Machines" prints " ⟨M,w⟩\langle M, w\rangle⟨M,w⟩ " (with angle brackets). Why did pdftotext give "hM, wi"?
The maths font has no table saying which character each shape is, so pdftotext used the shape numbers: ⟨ and ⟩ are 104 and 105, the codes of h and i. The braces and the bar disappeared completely. pdfminer.six printed the raw numbers instead, such as "(cid:104)" for ⟨.Seen when extracting this book with pdftotext and pdfminer.six, two free text extraction programs.
[Copying formulas] What is MathML?
A way to write formulas in a web page as structure: which part is a base, an exponent, a fraction, a root, and so on. Example: 2n2^n2n is written <msup><mn>2</mn><mi>n</mi></msup>: msup means "superscript", its first part (the number 2) is the base and its second part (the identifier n) is the exponent.MathML Core is the part of MathML meant to be built into web browsers (W3C Candidate Recommendation, June 2025).Source: MathML Core, §3.4.1 msub, msup, msubsup
[Copying formulas] Can an EPUB e-book contain formulas written in MathML?
Yes: EPUB 3 allows MathML inside its chapter files. So in an EPUB, 2n2^n2n can stay a 2 with an exponent n, instead of two characters at positions as in a PDF.Source: EPUB 3.3, §6.1.4.1 Embedded MathML
[Copying formulas] What is MathJax?
A free program, written in JavaScript, that displays formulas written in LaTeX, MathML or AsciiMath on web pages, in all modern browsers. The formulas stay text rather than pictures, so search engines can find them, and MathJax can turn them into spoken text for screen readers.Source: MathJax documentation, "What is MathJax?"
[Copying formulas] The formulas on these flashcards are drawn by KaTeX, a program similar to MathJax. What does each formula contain besides what you see?
A MathML version of the formula, with the original LaTeX source attached. MathML has a "semantics" element for this: the first part is shown, and the attached annotations, such as the LaTeX source, are hidden.Source: MathML Core, §3.7 Semantics and Presentation
[Which format if you need the text] You want to copy, search or quote the text of a book, or have it read aloud. Which format should you prefer?
EPUB or HTML: the text is stored in reading order, and formulas can be stored as MathML. EPUB was designed so that text adapts to the screen; the EPUB 3 Overview names accessibility as a main design goal.Source: EPUB 3 Overview; MathML Core
[Which format if you need the text] You only have a PDF and need its text. What can you check or fix?
Scanned pages If nothing can be selected, the pages are pictures; run OCR, for example Tesseract. Example: a scanned library book. Ligatures Replace ﬁ, ﬂ and similar characters by plain letters with Unicode NFKC normalization. Example: "Quantiﬁed" becomes "Quantified" and is found by search. Tables and columns Try pdftotext -layout, which keeps the horizontal positions. Example: each row of a table stays on one line. Hyphens and headers Check words split at line ends and remove running heads an…