Normalize HTML Document
This function analyzes an HTML document line-wise, groups lines into text blocks, and categorizes these text blocks.
Afterward, it opens a series of dialogs in which you can manually check the analysis result.
In particular, each line has one of the following categories, whose correct assignment you get to check and correct:
- Continue: the line belongs to the same block as the one immediately above it
- Main Text: first line in a main text block
- Continue Main Text: continues a main text block after an interruption, e.g. a page break or a figure and its caption
- Caption: first line in a caption block
- Footnote: first line in a footnote block
- Continue Footnote: continues a footnote block after an interruption, e.g. a column break
- Page Header Footer: first line in a page header or footer block
- Heading: first line in a heading block
- Table: first line in a table
- Bibliographic Reference: first line in an individual bibliographic reference (mark them individually)
- OCR / Layout Artifact: first line in a block originating from lines or stains or figures mis-recognized as text
After you have checked the line categorization, the function normalizes the text. In particular, this means
- Text blocks are transformed into flowing text, i.e., line breaks are removed, and words de-hyphenated in the process.
- Each text block and paragraph has its page number(s) assigned as an attribute.
- Page headers and footers are removed, as are OCR and layout artifacts.
- Interrupted main text blocks are re-united.