From resiliparse

resiliparse is the web-crawl processing toolkit from the Webis group behind ChatNoir. It is built for large-scale corpus work: HTMLTree wraps the same lexbor engine selectolax does and builds a WHATWG tree with real text nodes, and alongside the parser it ships boilerplate-aware plain-text extraction, encoding detection, fast language detection, and process/memory guards for hardening crawl workers. WARC reading lives in its companion package FastWARC. It shows up wherever people process Common Crawl-scale archives in Python and need a fast, resilient HTML-to-text stage.

turbohtml covers the DOM and text-extraction ground with one library: it parses the same WHATWG tree in its own C engine, then keeps you inside a fully typed, mutable Document where resiliparse’s DOM traversal, get_element_by_* lookups, and CSS query_selector methods collapse into one find/find_all/select grammar. It also matches resiliparse’s fast language detection with turbohtml.detect.detect_language(). It does not try to be a crawl toolkit; process guards and WARC handling stay resiliparse’s job.

turbohtml vs resiliparse

Dimension	turbohtml	resiliparse
Scope	Parse, query, mutate, serialize, and extract text in one library	Web-crawl processing toolkit: HTML parse and text extraction plus encoding/language detection and crawl guards
Feature breadth	CSS select(), XPath 1.0 xpath(), the find()/find_all() grammar, a full edit surface, Markdown/plain-text renderers, sanitizer, linkifier, structured-data extraction	CSS query_selector plus get_element_by_* DOM lookups, boilerplate-aware extract_plain_text, encoding and language detection, process/memory guards
Performance	Own C engine straight into the native tree; text extraction walks the WHATWG tree once in C	lexbor parse (a dead heat with turbohtml); extract_plain_text renders off the lexbor tree in a second pass
Typing	Fully type annotated with bundled stubs	Cython extension; limited Python-level type surface
Dependencies	Self-contained C extension	Cython extension over lexbor; WARC handling needs the companion FastWARC package
Maintenance	Actively developed	Actively developed by the Webis research group

Feature overlap

Both are native WHATWG parsers with real text nodes, so the parse call, DOM walk, and text extraction port directly:

• HTMLTree.parse(html) maps to turbohtml.parse().

• tree.body, tree.head, tree.title map to doc.find("body"), doc.find("head"), and doc.find("title").text.

• node.query_selector(sel) / node.query_selector_all(sel) map to select_one() / select().

• node.get_element_by_id("main") maps to node.find(id="main"); the get_elements_by_tag_name/get_elements_by_class_name pair maps to find_all() (find_all("a"), find_all(class_="x")).

• The getattr/hasattr/setattr/delattr attribute methods map onto the attrs mapping (attrs.get, "href" in attrs, attrs[...] = ..., del attrs[...]).

• node.tag, node.text, node.html map to tag, text, html; node.class_list maps to attrs["class"].

• node.create_element/append_child/decompose map to Element, append(), decompose().

• extract_plain_text(html) maps to to_text() for laid-out text (or text for the raw concatenation); extract_plain_text(html, main_content=True) maps to main_text(), and the boilerplate-stripped main node to main_content().

What turbohtml adds

• XPath 1.0 querying via xpath(), which resiliparse has no equivalent for.

• The find()/find_all() filter grammar layered over CSS, so tag, id, class, and attribute predicates compose in one call instead of chaining get_element_by_* methods.

• Serialization and rendering in the same library: the html property, encode(), and the Markdown/PlainText/Html renderers, including to_markdown().

• HTML sanitization against an allowlist (turbohtml.clean.sanitize() with a Policy).

• Link handling: links, rewrite_links(), resolve_links(), and the Linkify pass.

• Structured-data extraction: json_ld, opengraph, microdata.

• Content-based language detection: turbohtml.detect.detect_language() classifies a string’s natural language (ISO 639-3 code, confidence, and Unicode script) the way resiliparse.parse.lang.detect_fast does, over the same trigram model whatlang uses. It replaces detect_fast for the text you extract with text() or main_text().

• Full static typing across the tree with bundled stubs.

What resiliparse has that turbohtml does not

• Process and memory guards. resiliparse.process_guard provides time and memory guards that kill a worker whose parse or extraction runs away, which matters when processing adversarial crawl data at scale. turbohtml ships no such guard; wrap calls in your own resource limits.

• WARC/archive processing. resiliparse’s ecosystem reads WARC records through the companion FastWARC package. turbohtml is a tree library and has no archive support; keep FastWARC for the ingestion stage.

• Byte-to-str decoding. resiliparse.parse.encoding both detects an encoding and decodes raw bytes to str. turbohtml matches the detection half with the standalone turbohtml.detect.detect() (benched below) and during parse() (detect_encoding=True), but it returns the encoding label rather than handing back the decoded string.

Performance

Parsing is a dead heat: resiliparse runs lexbor and turbohtml runs its own C engine straight into the native tree. Past the parse, the table now covers the whole shared surface. CSS selection runs up to ~20x faster, and text extraction (extract_plain_text against to_text(), and its main_content=True mode against main_text()) walks the WHATWG tree once in C where resiliparse renders off the lexbor tree in a second pass, so layout-aware text runs 5 to 15x faster. The mutable-DOM operations resiliparse exposes bench too: rel-tagging, class edits, and subtree removal run up to ~8x faster, and social-card extraction and link rewriting land a few times ahead. A handful of cheap passes trade the lead – raw text concatenation, flat link extraction, and URL absolutization sit within a small factor either way – and standalone encoding detection (resiliparse.parse.encoding against turbohtml.detect.detect()) swings with the byte stream, from an order of magnitude slower on a short Shift-JIS sample to ~97x faster on a large UTF-8 page:

operation	turbohtml	resiliparse
parse to a tree — wpt tiny (0.6 kB)	1.21 µs	3.81 µs (3.2x)
parse to a tree — wpt small (4 kB)	9.58 µs	12.8 µs (1.4x)
parse to a tree — wpt medium (9.6 kB)	24.1 µs	28.4 µs (1.2x)
parse to a tree — wpt large (92 kB)	209 µs	282 µs (1.4x)
parse to a tree — wpt CJK (124 kB)	408 µs	549 µs (1.4x)
parse to a tree — whatwg spec (235 kB)	400 µs	504 µs (1.3x)
find every anchor — daring fireball (10 kB)	371 ns	481 ns (1.3x)
find every anchor — ars technica (56 kB)	817 ns	2.77 µs (3.4x)
find every anchor — mozilla blog (95 kB)	1.16 µs	10.6 µs (9.2x)
find every anchor — whatwg spec (235 kB)	1.36 µs	29.4 µs (21.7x)
select div a[href] — daring fireball (10 kB)	610 ns	2.06 µs (3.4x)
select div a[href] — ars technica (56 kB)	1.47 µs	6.28 µs (4.3x)
select div a[href] — mozilla blog (95 kB)	2.1 µs	12.3 µs (5.9x)
select div a[href] — whatwg spec (235 kB)	1.78 µs	35.6 µs (20.0x)
select div:has(a) — daring fireball (10 kB)	254 ns	2.37 µs (9.4x)
select div:has(a) — ars technica (56 kB)	1.24 µs	10.8 µs (8.7x)
select div:has(a) — mozilla blog (95 kB)	8.99 µs	39.9 µs (4.5x)
select div:has(a) — whatwg spec (235 kB)	5.8 µs	102 µs (17.7x)
collect visible text — daring fireball (10 kB)	2.7 µs	2.5 µs (1.0x)
collect visible text — ars technica (56 kB)	13 µs	15.5 µs (1.2x)
collect visible text — mozilla blog (95 kB)	22.4 µs	31.7 µs (1.5x)
collect visible text — whatwg spec (235 kB)	75.4 µs	96.9 µs (1.3x)
serialize a parsed tree — daring fireball (10 kB)	7.02 µs	12.8 µs (1.9x)
serialize a parsed tree — ars technica (56 kB)	38.8 µs	72.3 µs (1.9x)
serialize a parsed tree — mozilla blog (95 kB)	76.3 µs	144 µs (1.9x)
serialize a parsed tree — whatwg spec (235 kB)	195 µs	358 µs (1.9x)
tag every link rel=nofollow — daring fireball (10 kB)	3.32 µs	7.02 µs (2.2x)
tag every link rel=nofollow — ars technica (56 kB)	13.1 µs	17.8 µs (1.4x)
tag every link rel=nofollow — mozilla blog (95 kB)	20.9 µs	27 µs (1.3x)
tag every link rel=nofollow — whatwg spec (235 kB)	42.6 µs	51.9 µs (1.3x)
class add/remove on every link — daring fireball (10 kB)	2.24 µs	8.84 µs (4.0x)
class add/remove on every link — ars technica (56 kB)	8.89 µs	24.4 µs (2.8x)
class add/remove on every link — mozilla blog (95 kB)	8.78 µs	38.6 µs (4.4x)
class add/remove on every link — whatwg spec (235 kB)	8.7 µs	66.5 µs (7.7x)
drop tags with content (remove) — daring fireball (10 kB)	23.8 µs	41.2 µs (1.8x)
drop tags with content (remove) — ars technica (56 kB)	115 µs	184 µs (1.6x)
drop tags with content (remove) — mozilla blog (95 kB)	260 µs	402 µs (1.6x)
drop tags with content (remove) — whatwg spec (235 kB)	645 µs	995 µs (1.6x)
walk every descendant — daring fireball (10 kB)	3.26 µs	4.2 µs (1.3x)
walk every descendant — ars technica (56 kB)	13.4 µs	16.8 µs (1.3x)
walk every descendant — mozilla blog (95 kB)	28 µs	54.4 µs (2.0x)
walk every descendant — whatwg spec (235 kB)	96.8 µs	156 µs (1.7x)
extract every link — daring fireball (10 kB)	8.49 µs	5.14 µs (0.7x)
extract every link — ars technica (56 kB)	27.6 µs	15.8 µs (0.6x)
extract every link — mozilla blog (95 kB)	50.1 µs	27.4 µs (0.6x)
extract every link — whatwg spec (235 kB)	84.6 µs	52.5 µs (0.7x)
absolutize every link — daring fireball (10 kB)	82.6 µs	63.3 µs (0.8x)
absolutize every link — ars technica (56 kB)	212 µs	158 µs (0.8x)
absolutize every link — mozilla blog (95 kB)	482 µs	239 µs (0.5x)
absolutize every link — whatwg spec (235 kB)	270 µs	262 µs (1.0x)
rewrite every link — daring fireball (10 kB)	4.69 µs	6.46 µs (1.4x)
rewrite every link — ars technica (56 kB)	11.5 µs	19.5 µs (1.7x)
rewrite every link — mozilla blog (95 kB)	21.7 µs	33.8 µs (1.6x)
rewrite every link — whatwg spec (235 kB)	38.1 µs	58 µs (1.6x)
social-card extraction — head	1.79 µs	6.74 µs (3.8x)
social-card extraction — article 8 KiB	22.1 µs	40.4 µs (1.9x)
layout-aware text — article (2 KiB)	1.6 µs	24.1 µs (15.1x)
layout-aware text — table (4 KiB)	9.68 µs	51.5 µs (5.4x)
main-content text — main (4 KiB)	3.46 µs	20.1 µs (5.9x)
extract @href per match — daring fireball (10 kB)	2.49 µs	4.82 µs (2.0x)
extract @href per match — ars technica (56 kB)	6.29 µs	14.9 µs (2.4x)
extract @href per match — mozilla blog (95 kB)	8.51 µs	25.7 µs (3.1x)
extract @href per match — whatwg spec (235 kB)	9.05 µs	49.6 µs (5.5x)
extract text per match — daring fireball (10 kB)	2.75 µs	4.97 µs (1.9x)
extract text per match — ars technica (56 kB)	6.39 µs	15.3 µs (2.4x)
extract text per match — mozilla blog (95 kB)	10.3 µs	28.1 µs (2.8x)
extract text per match — whatwg spec (235 kB)	10.8 µs	50.4 µs (4.7x)
extract URL hints — base_url / get_base_url	1.27 µs	5.01 µs (4.0x)
extract URL hints — meta_refresh / get_meta_refresh	1.29 µs	5.15 µs (4.0x)
detect a byte stream’s encoding — ascii (1 kB)	1.96 µs	1.57 µs (0.9x)
detect a byte stream’s encoding — utf-8 russian (4 kB)	3.77 µs	4.49 µs (1.2x)
detect a byte stream’s encoding — windows-1251 russian (4 kB)	206 µs	493 µs (2.4x)
detect a byte stream’s encoding — windows-1252 french (4 kB)	206 µs	583 µs (2.9x)
detect a byte stream’s encoding — shift_jis japanese (4 kB)	773 µs	54.7 µs (0.1x)
detect a byte stream’s encoding — utf-8 page (95 kB)	705 ns	68.6 µs (97.3x)

How to migrate

Swap HTMLTree.parse for turbohtml.parse() and the resiliparse.extract import for the text methods on the returned node:

# resiliparse
from resiliparse.parse.html import HTMLTree
from resiliparse.extract.html2text import extract_plain_text

tree = HTMLTree.parse("<div id=main><p class=x>Hi <a href='/u'>link</a></p></div>")
href = tree.body.query_selector("#main a").getattr("href")
text = extract_plain_text(tree.document.html)

# turbohtml
from turbohtml import parse

doc = parse("<div id=main><p class=x>Hi <a href='/u'>link</a></p></div>")
href = doc.select_one("#main a").attrs["href"]
text = doc.to_text()

resiliparse	turbohtml
HTMLTree.parse(html)	turbohtml.parse()
tree.body, tree.head, tree.title	doc.find("body"), doc.find("head"), doc.find("title").text
node.query_selector("a"), node.query_selector_all("a")	select_one(), select()
node.get_element_by_id("main")	node.find(id="main")
node.get_elements_by_tag_name("a"), node.get_elements_by_class_name("x")	find_all() (find_all("a"), find_all(class_="x"))
node.getattr("href"), node.hasattr("href"), node.setattr(...), node.delattr(...)	attrs (attrs.get, "href" in attrs, attrs[...] = ..., del attrs[...])
node.tag, node.text, node.html, node.class_list	tag, text, html, attrs["class"]
node.parent, node.next_element, node.prev_element, node.first_element_child	parent, next_sibling, previous_sibling, node.children[0]
tree.create_element("div"), node.append_child(...), node.decompose()	Element, append(), decompose()
extract_plain_text(html) (from resiliparse.extract)	to_text() for layout, text for the raw concatenation
extract_plain_text(html, main_content=True)	main_text()
ExtractNode / boilerplate-stripped main content	main_content()
detect_fast(text) (from resiliparse.parse.lang)	turbohtml.detect.detect_language()

doc = parse("<div id=main><p class=x>Hi <a href='/u'>link</a></p></div>")
print(doc.select_one("#main a").attrs["href"])
print(doc.find(id="main").find("p").text)

/u
Hi link

Gotchas and pitfalls

• Sibling traversal spans text nodes. resiliparse’s next_element/prev_element/first_element_child skip to the next element; next_sibling/previous_sibling and node.children are node-level and include Text nodes. Filter for elements, or use find()/select() when you want elements only.

• Main-content extraction is a heuristic. main_content() and main_text() run a content-density (readability) pass over the parsed tree. It approximates resiliparse’s boilerplate removal but will not match it token for token.

• Node identity, not markup identity. turbohtml compares nodes by identity over the underlying arena node, so two wrappers for the same element are equal, but two separately parsed trees with identical markup are not. Compare serializations or walk the tree instead.

• Language detection reads text, not a tree. turbohtml.detect.detect_language() takes the extracted string (from text() or main_text()), where detect_fast takes the document; it reports an ISO 639-3 code with a confidence rather than resiliparse’s own label set, so map the codes your pipeline expects.

• The crawl toolkit stays behind. Dropping resiliparse also drops its process guards and the FastWARC ingestion path. If a pipeline depends on those, keep resiliparse for that stage and hand turbohtml the HTML string.