From boilerpy3¶

boilerpy3 is the pure-Python port of boilerpipe, the original boilerplate-removal library. An extractor (ArticleExtractor, CanolaExtractor, NumWordsRulesExtractor, …) parses a page with a built-in SAX handler, segments it into text blocks, classifies each block as content or boilerplate from word counts and link densities, and returns either the surviving text or the classified blocks. It runs on the standard library alone and is used to strip navigation, sidebars, and footers off article pages before indexing, summarization, or NLP.

turbohtml covers that same ground with turbohtml.extract.boilerplate() and the C main-content scoring behind main_content(): the scoring picks the content body, and each paragraph unit comes back as a Paragraph carrying the same text-and-classification shape as boilerpy3’s text_blocks. Because the extraction sits on top of a full WHATWG parser, the same page is also available as a DOM you can query and serialize.

turbohtml vs boilerpy3¶

Dimension	turbohtml	boilerpy3
Scope	Full WHATWG HTML parser with DOM, query, serialize, and content extraction on top	Boilerplate/content extraction only, over its own SAX pass
Feature breadth	One C scoring model plus a threshold config (`Extraction`) and article metadata	A family of tuned extractors (`ArticleExtractor`, `ArticleSentencesExtractor`, `LargestContentExtractor`, `CanolaExtractor`, …)
Performance	C scoring pass over the parsed tree (see below)	Python classification over a Python SAX parse
Typing	Fully annotated, ships PEP 561 stubs	Annotated pure-Python source
Dependencies	Compiled C extension	Standard library only, no compiled extension
Maintenance	Actively developed	Maintained port of the boilerpipe algorithm

Feature overlap¶

The shared surface ports one call to one call:

ArticleExtractor().get_content(html) -> main_text() (or article().text).
get_doc(html).text_blocks -> extract.boilerplate(html), a list of Paragraph.
block.text -> paragraph.text, the same whitespace-normalized visible text.
block.is_content -> not paragraph.is_boilerplate.
block.link_density and block.num_words cutoffs -> the max_link_density and min_length fields of Extraction.
KeepEverythingExtractor().get_content(html) -> to_text(), the full visible text.

What turbohtml adds¶

The extraction rides on a full WHATWG parse, so the same page is a DOM you can query, mutate, and serialize, not just a text stream.
article() harvests page metadata beside the body text: title, byline, date, description, and lang. boilerpy3’s get_doc(html).title is the only metadata it exposes, and usually None unless a title block was marked.
Extraction tunes the per-paragraph thresholds directly, including a justext() preset (a 70-character floor at 0.2 link density) and a keep_headings switch.
turbohtml.parse() follows the WHATWG recovery rules and never raises on malformed markup, where boilerpy3 raises HTMLExtractionError unless you pass raise_on_failure=False.

What boilerpy3 has that turbohtml does not¶

Several distinct, separately tuned extractor algorithms. ArticleExtractor maps to turbohtml’s defaults, NumWordsRulesExtractor-style floors to min_length, and KeepEverythingExtractor to to_text(), but there is no equivalent of CanolaExtractor’s trained rule set or ArticleSentencesExtractor’s sentence-level shaping.
Multiple surviving content islands. boilerpy3 classifies each block from its neighbors, so two separate content regions both survive; turbohtml first picks the single best-scoring content body, and only units inside it can be content. If you need every island, walk boilerplate() output yourself.
Fetch convenience: get_content_from_url(url) and get_content_from_file(path). turbohtml has no fetcher; read the page with urllib or httpx (or open the file) and pass the markup to parse().
Pure-Python install with no compiled extension, which can matter on platforms without a prebuilt turbohtml wheel.

Performance¶

turbohtml scores the tree in one C pass and classifies the units in a thin Python layer; boilerpy3 parses with its own SAX handler and classifies each block in Python. Numbers vary with input and hardware.

input	turbohtml	boilerpy3
post (4 KiB)	26.3 µs	266 µs (10.2x)
longform (16 KiB)	100 µs	916 µs (9.2x)

How to migrate¶

Swap the extractor import for turbohtml.parse() and, when you want the classified blocks, the turbohtml.extract.boilerplate() helper:

from turbohtml import parse
from turbohtml.extract import boilerplate, Extraction

The call mapping:

boilerpy3	turbohtml
`ArticleExtractor().get_content(html)`	`parse(html).main_text()` (or `article().text`)
`ArticleExtractor().get_doc(html).text_blocks`	`extract.boilerplate(html)`
`block.text`	`paragraph.text`, the same field on `Paragraph`
`block.is_content`	`not paragraph.is_boilerplate`
`block.link_density` cutoffs	`Extraction(max_link_density=...)`
`block.num_words` floors (`NumWordsRulesExtractor`)	`Extraction(min_length=...)`, a character floor
`get_doc(html).title`	`parse(html).article().title`
`KeepEverythingExtractor().get_content(html)`	`to_text()`, the full visible text
`get_content_from_url(url)`	fetch the page yourself (`urllib` or `httpx`), then parse the markup

Before and after, extracting the article body from a full page:

from turbohtml import parse

page = (
    "<body><nav><ul><li><a href='/'>Home</a></li><li><a href='/faq'>FAQ</a></li></ul></nav>"
    "<article class='post'><h1>Comets</h1>"
    "<p>A comet is an icy body that releases gas, forming a visible tail, as it nears the Sun.</p>"
    "</article></body>"
)
print(parse(page).main_text())

Comets

A comet is an icy body that releases gas, forming a visible tail, as it nears the Sun.

Gotchas and pitfalls¶

boilerpy3 segments at inline/block transitions of its SAX stream, so a navigation list, a heading, and the first paragraph can fuse into one block; turbohtml emits one Paragraph per block element, so expect more, smaller units at the same total text.
The min_length floor is a character count of normalized text, not a word count. boilerpy3’s NumWordsRulesExtractor thresholds are word counts, so a direct number carries over only loosely; tune against your own pages.
get_doc(html).title is usually None (it needs a marked title block); article().title harvests the first <h1>, og:title, or <title> instead.
boilerpy3 raises HTMLExtractionError on markup its parser rejects unless raise_on_failure=False; turbohtml.parse() follows the WHATWG recovery rules and never raises on malformed input.