From resiliparse

resiliparse is the web-crawl processing toolkit from the Webis group behind ChatNoir. It is built for large-scale corpus work: HTMLTree wraps the same lexbor engine selectolax does and builds a WHATWG tree with real text nodes, and alongside the parser it ships boilerplate-aware plain-text extraction, encoding detection, fast language detection, and process/memory guards for hardening crawl workers. WARC reading lives in its companion package FastWARC. It shows up wherever people process Common Crawl-scale archives in Python and need a fast, resilient HTML-to-text stage.

turbohtml covers the DOM and text-extraction ground with one library: it parses the same WHATWG tree in its own C engine, then keeps you inside a fully typed, mutable Document where resiliparse’s DOM traversal, get_element_by_* lookups, and CSS query_selector methods collapse into one find/find_all/select grammar. It also matches resiliparse’s fast language detection with turbohtml.detect.detect_language(). It does not try to be a crawl toolkit; process guards and WARC handling stay resiliparse’s job.

turbohtml vs resiliparse

Dimension	turbohtml	resiliparse
Scope	Parse, query, mutate, serialize, and extract text in one library	Web-crawl processing toolkit: HTML parse and text extraction plus encoding/language detection and crawl guards
Feature breadth	CSS select(), XPath 1.0 xpath(), the find()/find_all() grammar, a full edit surface, Markdown/plain-text renderers, sanitizer, linkifier, structured-data extraction	CSS query_selector plus get_element_by_* DOM lookups, boilerplate-aware extract_plain_text, encoding and language detection, process/memory guards
Performance	Own C engine straight into the native tree; text extraction walks the WHATWG tree once in C	lexbor parse (a dead heat with turbohtml); extract_plain_text renders off the lexbor tree in a second pass
Typing	Fully type annotated with bundled stubs	Cython extension; limited Python-level type surface
Dependencies	Self-contained C extension	Cython extension over lexbor; WARC handling needs the companion FastWARC package
Maintenance	Actively developed	Actively developed by the Webis research group

Feature overlap

Both are native WHATWG parsers with real text nodes, so the parse call, DOM walk, and text extraction port directly:

• HTMLTree.parse(html) maps to turbohtml.parse().

• tree.body, tree.head, tree.title map to doc.find("body"), doc.find("head"), and doc.find("title").text.

• node.query_selector(sel) / node.query_selector_all(sel) map to select_one() / select().

• node.get_element_by_id("main") maps to node.find(id="main"); the get_elements_by_tag_name/get_elements_by_class_name pair maps to find_all() (find_all("a"), find_all(class_="x")).

• The getattr/hasattr/setattr/delattr attribute methods map onto the attrs mapping (attrs.get, "href" in attrs, attrs[...] = ..., del attrs[...]).

• node.tag, node.text, node.html map to tag, text, html; node.class_list maps to attrs["class"].

• node.create_element/append_child/decompose map to Element, append(), decompose().

• extract_plain_text(html) maps to to_text() for laid-out text (or text for the raw concatenation); extract_plain_text(html, main_content=True) maps to main_text(), and the boilerplate-stripped main node to main_content().

What turbohtml adds

• XPath 1.0 querying via xpath(), which resiliparse has no equivalent for.

• The find()/find_all() filter grammar layered over CSS, so tag, id, class, and attribute predicates compose in one call instead of chaining get_element_by_* methods.

• Serialization and rendering in the same library: the html property, encode(), and the Markdown/PlainText/Html renderers, including to_markdown().

• HTML sanitization against an allowlist (turbohtml.clean.sanitize() with a Policy).

• Link handling: links, rewrite_links(), resolve_links(), and the Linkify pass.

• Structured-data extraction: json_ld, opengraph, microdata.

• Content-based language detection: turbohtml.detect.detect_language() classifies a string’s natural language (ISO 639-3 code, confidence, and Unicode script) the way resiliparse.parse.lang.detect_fast does, over the same trigram model whatlang uses. It replaces detect_fast for the text you extract with text() or main_text().

• Full static typing across the tree with bundled stubs.

What resiliparse has that turbohtml does not

• Process and memory guards. resiliparse.process_guard provides time and memory guards that kill a worker whose parse or extraction runs away, which matters when processing adversarial crawl data at scale. turbohtml ships no such guard; wrap calls in your own resource limits.

• WARC/archive processing. resiliparse’s ecosystem reads WARC records through the companion FastWARC package. turbohtml is a tree library and has no archive support; keep FastWARC for the ingestion stage.

• Standalone MIME/encoding utilities. resiliparse.parse.encoding exposes byte-to-str conversion and encoding detection as free functions over raw bytes. turbohtml detects the encoding during parse() (detect_encoding=True) rather than as a standalone codec-inspection API.

Performance

Parsing is a dead heat: resiliparse runs lexbor and turbohtml runs its own C engine straight into the native tree. On text extraction (extract_plain_text against to_text(), and its main_content=True mode against main_text()) turbohtml walks the WHATWG tree once in C where resiliparse renders off the lexbor tree in a second pass:

operation	turbohtml	resiliparse
parse to a tree — wpt tiny (0.6 kB)	1.21 µs	3.84 µs (3.2x)
parse to a tree — wpt small (4 kB)	9.57 µs	12.6 µs (1.4x)
parse to a tree — wpt medium (9.6 kB)	24.2 µs	27.9 µs (1.2x)
parse to a tree — wpt large (92 kB)	207 µs	286 µs (1.4x)
parse to a tree — wpt CJK (124 kB)	407 µs	544 µs (1.4x)
parse to a tree — whatwg spec (235 kB)	405 µs	507 µs (1.3x)
find every anchor — daring fireball (10 kB)	375 ns	471 ns (1.3x)
find every anchor — ars technica (56 kB)	828 ns	2.73 µs (3.3x)
find every anchor — mozilla blog (95 kB)	1.18 µs	10.4 µs (8.9x)
find every anchor — whatwg spec (235 kB)	1.32 µs	31 µs (23.6x)
select div a[href] — daring fireball (10 kB)	639 ns	2.04 µs (3.2x)
select div a[href] — ars technica (56 kB)	1.46 µs	6.12 µs (4.2x)
select div a[href] — mozilla blog (95 kB)	2.07 µs	12.2 µs (6.0x)
select div a[href] — whatwg spec (235 kB)	1.74 µs	36.5 µs (21.0x)
select div:has(a) — daring fireball (10 kB)	256 ns	2.43 µs (9.5x)
select div:has(a) — ars technica (56 kB)	1.28 µs	11.2 µs (8.8x)
select div:has(a) — mozilla blog (95 kB)	8.97 µs	40.3 µs (4.5x)
select div:has(a) — whatwg spec (235 kB)	6.08 µs	105 µs (17.3x)
collect visible text — daring fireball (10 kB)	2.69 µs	2.48 µs (1.0x)
collect visible text — ars technica (56 kB)	13.4 µs	15.5 µs (1.2x)
collect visible text — mozilla blog (95 kB)	23.3 µs	31.7 µs (1.4x)
collect visible text — whatwg spec (235 kB)	76.8 µs	98.5 µs (1.3x)
serialize a parsed tree — daring fireball (10 kB)	7.14 µs	12.8 µs (1.8x)
serialize a parsed tree — ars technica (56 kB)	40.7 µs	72.8 µs (1.8x)
serialize a parsed tree — mozilla blog (95 kB)	79.2 µs	147 µs (1.9x)
serialize a parsed tree — whatwg spec (235 kB)	209 µs	362 µs (1.8x)
walk every descendant — daring fireball (10 kB)	3.34 µs	4.33 µs (1.3x)
walk every descendant — ars technica (56 kB)	13.8 µs	17.6 µs (1.3x)
walk every descendant — mozilla blog (95 kB)	28.2 µs	44.8 µs (1.6x)
walk every descendant — whatwg spec (235 kB)	101 µs	161 µs (1.6x)
layout-aware text — article (2 KiB)	1.63 µs	23.2 µs (14.2x)
layout-aware text — table (4 KiB)	10.1 µs	53.6 µs (5.3x)
main-content text — main (4 KiB)	3.6 µs	21.1 µs (5.9x)

How to migrate

Swap HTMLTree.parse for turbohtml.parse() and the resiliparse.extract import for the text methods on the returned node:

# resiliparse
from resiliparse.parse.html import HTMLTree
from resiliparse.extract.html2text import extract_plain_text

tree = HTMLTree.parse("<div id=main><p class=x>Hi <a href='/u'>link</a></p></div>")
href = tree.body.query_selector("#main a").getattr("href")
text = extract_plain_text(tree.document.html)

# turbohtml
from turbohtml import parse

doc = parse("<div id=main><p class=x>Hi <a href='/u'>link</a></p></div>")
href = doc.select_one("#main a").attrs["href"]
text = doc.to_text()

resiliparse	turbohtml
HTMLTree.parse(html)	turbohtml.parse()
tree.body, tree.head, tree.title	doc.find("body"), doc.find("head"), doc.find("title").text
node.query_selector("a"), node.query_selector_all("a")	select_one(), select()
node.get_element_by_id("main")	node.find(id="main")
node.get_elements_by_tag_name("a"), node.get_elements_by_class_name("x")	find_all() (find_all("a"), find_all(class_="x"))
node.getattr("href"), node.hasattr("href"), node.setattr(...), node.delattr(...)	attrs (attrs.get, "href" in attrs, attrs[...] = ..., del attrs[...])
node.tag, node.text, node.html, node.class_list	tag, text, html, attrs["class"]
node.parent, node.next_element, node.prev_element, node.first_element_child	parent, next_sibling, previous_sibling, node.children[0]
tree.create_element("div"), node.append_child(...), node.decompose()	Element, append(), decompose()
extract_plain_text(html) (from resiliparse.extract)	to_text() for layout, text for the raw concatenation
extract_plain_text(html, main_content=True)	main_text()
ExtractNode / boilerplate-stripped main content	main_content()
detect_fast(text) (from resiliparse.parse.lang)	turbohtml.detect.detect_language()

doc = parse("<div id=main><p class=x>Hi <a href='/u'>link</a></p></div>")
print(doc.select_one("#main a").attrs["href"])
print(doc.find(id="main").find("p").text)

/u
Hi link

Gotchas and pitfalls

• Sibling traversal spans text nodes. resiliparse’s next_element/prev_element/first_element_child skip to the next element; next_sibling/previous_sibling and node.children are node-level and include Text nodes. Filter for elements, or use find()/select() when you want elements only.

• Main-content extraction is a heuristic. main_content() and main_text() run a content-density (readability) pass over the parsed tree. It approximates resiliparse’s boilerplate removal but will not match it token for token.

• Node identity, not markup identity. turbohtml compares nodes by identity over the underlying arena node, so two wrappers for the same element are equal, but two separately parsed trees with identical markup are not. Compare serializations or walk the tree instead.

• Language detection reads text, not a tree. turbohtml.detect.detect_language() takes the extracted string (from text() or main_text()), where detect_fast takes the document; it reports an ISO 639-3 code with a confidence rather than resiliparse’s own label set, so map the codes your pipeline expects.

• The crawl toolkit stays behind. Dropping resiliparse also drops its process guards and the FastWARC ingestion path. If a pipeline depends on those, keep resiliparse for that stage and hand turbohtml the HTML string.