From selectolax¶
selectolax is a fast HTML parser that wraps a C engine and exposes CSS
selection. It ships two backends: HTMLParser over Modest and LexborHTMLParser over lexbor. You query with CSS selectors (css / css_first), read text through the text()
method, reach attributes via node.attributes, and mutate the tree with a small set of operations (decompose,
unwrap, strip_tags, unwrap_tags). It has no XPath and no regex extraction. Its niche is high-throughput web
scraping and data extraction where CSS selection over a compiled C tree is the whole job.
turbohtml covers the same ground with a single native, spec-compliant HTML5 engine: select() /
select_one() for CSS, text as a property, attrs
for attributes, and the same drop/unwrap operations. On top of that surface it adds the find / find_all filter
grammar, XPath, regex extraction, a full mutation surface, and markdown/text/minify output, all fully typed.
turbohtml vs selectolax¶
Dimension |
turbohtml |
selectolax |
|---|---|---|
Scope |
Spec-compliant HTML5 parser plus native CSS/XPath selection, regex extraction, mutation, and serialization |
CSS-only selection and light tree editing over a bundled C engine (Modest or lexbor); no XPath, no regex |
Feature breadth |
CSS Selectors Level 4, the |
CSS selection, |
Performance |
Compiles a selector once and matches by interned integer atoms; collects text in one C pass; see the table below |
Fast CSS matching in C, but text collection and node access cross the C boundary per node |
Typing |
Fully typed, ships |
Typed API surface, but node access is string-centric and text is a method call |
Dependencies |
Self-contained C extension, no runtime dependencies |
Self-contained C extension (Modest/lexbor bundled), no runtime dependencies |
Maintenance |
Actively developed |
Actively maintained |
Feature overlap¶
These map 1:1 and port with a rename:
CSS selection:
node.css(sel)/node.css_first(sel)becomeselect()/select_one().Selector test:
node.css_matches(sel)becomesmatches().Tag name:
node.tagstaystag.Attributes:
node.attributesbecomesattrs, and a single value reads throughattr().Outer HTML:
node.htmlstayshtml.Node removal that keeps the children:
node.unwrap()staysunwrap().Node removal with its subtree:
node.decompose()staysdecompose().Bulk tag stripping:
parser.strip_tags([...])(drop tags with content) becomesremove(), andnode.unwrap_tags([...])(keep content) becomesstrip_tags(). Both turbohtml methods take a full CSS selector, not a tag-name list.
What turbohtml adds¶
The
find/find_allfilter grammar layered on top of CSS, with axes and regex or callable filters, where selectolax is CSS-only.XPath:
xpath(),xpath_one(),xpath_iter().Regex extraction over text or an attribute value:
re(),re_first().Text as a property plus lazy iterators:
text,strings,stripped_strings, collected in one C pass instead of a per-node method call.A full mutation surface on the same node:
prune(),wrap(),insert_before(),insert_after(),replace_with(),set_text(),insert_adjacent_html().Output conversions built in:
serialize(),to_markdown(),to_text().closest()for walking upward without a fresh query.
What selectolax has that turbohtml does not¶
A choice of parser backends (
HTMLParserover Modest vsLexborHTMLParserover lexbor): no equivalent. turbohtml is a single native engine; there is one tree builder and no backend switch.text(deep=..., separator=..., strip=...)shaping text extraction in one call: no exact equivalent. Readtextfor the flat string, iteratestripped_stringsandjoinwith your own separator, or callto_text()for a formatted rendering.The bundled engine’s raw C-level node handles and lexbor-specific knobs: not exposed. turbohtml’s public surface is the typed Python tree, not the underlying engine’s C API.
Performance¶
turbohtml’s lighter native tree parses, selects, and serializes faster than selectolax’s heavier object layer over
lexbor. It drops a set of tags with their subtrees faster (remove() against strip_tags, over a
92 kB page of 839 <code>/<a>/<q> elements), tests a compiled selector against every anchor
(matches() against css_matches) 35 to 45 times faster, and collects a node’s visible text
(text against selectolax’s text() method) seven to nine times faster, concatenating in one C
pass where selectolax crosses the lexbor boundary per node:
operation |
turbohtml |
|
|---|---|---|
parse to a tree — wpt tiny (0.6 kB) |
1.53 µs |
9.51 µs (6.3x ±6%) |
parse to a tree — wpt small (4 kB) |
12.2 µs |
56 µs (4.6x ±3%) |
parse to a tree — wpt medium (9.6 kB) |
32.5 µs |
137 µs (4.3x ±6%) |
parse to a tree — wpt large (92 kB) |
276 µs |
1.26 ms (4.6x ±2%) |
parse to a tree — wpt CJK (124 kB) |
532 µs |
2.96 ms (5.6x ±5%) |
parse to a tree — whatwg spec (235 kB) |
519 µs |
2.45 ms (4.8x ±3%) |
parse to a tree — common tags (13 kB) |
78.5 µs |
333 µs (4.3x ±3%) |
find every anchor — daring fireball (10 kB) |
372 ns |
5.67 µs (15.3x ±2%) |
find every anchor — ars technica (56 kB) |
811 ns |
15.9 µs (19.7x ±2%) |
find every anchor — mozilla blog (95 kB) |
1.12 µs |
28.5 µs (25.5x ±3%) |
find every anchor — whatwg spec (235 kB) |
1.3 µs |
75.3 µs (58.0x ±2%) |
select div a[href] — daring fireball (10 kB) |
595 ns |
7.58 µs (12.8x ±1%) |
select div a[href] — ars technica (56 kB) |
1.43 µs |
20.2 µs (14.1x ±2%) |
select div a[href] — mozilla blog (95 kB) |
2.01 µs |
34.1 µs (17.0x ±1%) |
select div a[href] — whatwg spec (235 kB) |
1.76 µs |
79.4 µs (45.1x ±1%) |
select div:has(a) — daring fireball (10 kB) |
262 ns |
4.95 µs (19.0x ±2%) |
select div:has(a) — ars technica (56 kB) |
1.26 µs |
17.8 µs (14.1x ±2%) |
select div:has(a) — mozilla blog (95 kB) |
8.54 µs |
51.9 µs (6.1x ±2%) |
select div:has(a) — whatwg spec (235 kB) |
5.54 µs |
82.7 µs (15.0x ±2%) |
match each anchor against div a[href] — daring fireball (10 kB) |
1.85 µs |
62.4 µs (33.8x ±3%) |
match each anchor against div a[href] — ars technica (56 kB) |
4.23 µs |
152 µs (36.0x ±2%) |
match each anchor against div a[href] — mozilla blog (95 kB) |
5.83 µs |
213 µs (36.6x ±1%) |
match each anchor against div a[href] — whatwg spec (235 kB) |
6.46 µs |
287 µs (44.5x ±2%) |
collect visible text — daring fireball (10 kB) |
2.63 µs |
21.4 µs (8.2x ±2%) |
collect visible text — ars technica (56 kB) |
13.5 µs |
89.8 µs (6.7x ±1%) |
collect visible text — mozilla blog (95 kB) |
21.9 µs |
203 µs (9.3x ±2%) |
collect visible text — whatwg spec (235 kB) |
83.7 µs |
756 µs (9.1x ±1%) |
serialize a parsed tree — daring fireball (10 kB) |
6.52 µs |
28.6 µs (4.4x ±2%) |
serialize a parsed tree — ars technica (56 kB) |
34.8 µs |
154 µs (4.5x ±1%) |
serialize a parsed tree — mozilla blog (95 kB) |
67.4 µs |
307 µs (4.6x ±1%) |
serialize a parsed tree — whatwg spec (235 kB) |
174 µs |
739 µs (4.3x ±1%) |
tag every link rel=nofollow — daring fireball (10 kB) |
4.74 µs |
16 µs (3.4x ±3%) |
tag every link rel=nofollow — ars technica (56 kB) |
16.6 µs |
39.1 µs (2.4x ±4%) |
tag every link rel=nofollow — mozilla blog (95 kB) |
25.4 µs |
63.6 µs (2.6x ±5%) |
tag every link rel=nofollow — whatwg spec (235 kB) |
51.4 µs |
118 µs (2.4x ±2%) |
class add/remove on every link — daring fireball (10 kB) |
2.62 µs |
26.6 µs (10.2x ±2%) |
class add/remove on every link — ars technica (56 kB) |
9.67 µs |
73.2 µs (7.6x ±2%) |
class add/remove on every link — mozilla blog (95 kB) |
10.3 µs |
102 µs (10.0x ±2%) |
class add/remove on every link — whatwg spec (235 kB) |
10.1 µs |
156 µs (15.5x ±2%) |
drop tags with content (remove) — daring fireball (10 kB) |
23.2 µs |
98.5 µs (4.3x ±1%) |
drop tags with content (remove) — ars technica (56 kB) |
112 µs |
480 µs (4.3x ±3%) |
drop tags with content (remove) — mozilla blog (95 kB) |
252 µs |
1.53 ms (6.1x ±3%) |
drop tags with content (remove) — whatwg spec (235 kB) |
650 µs |
2.77 ms (4.3x ±3%) |
unwrap tags keep content (strip_tags) — daring fireball (10 kB) |
24.4 µs |
113 µs (4.7x ±2%) |
unwrap tags keep content (strip_tags) — ars technica (56 kB) |
120 µs |
521 µs (4.4x ±1%) |
unwrap tags keep content (strip_tags) — mozilla blog (95 kB) |
261 µs |
1.61 ms (6.2x ±2%) |
unwrap tags keep content (strip_tags) — whatwg spec (235 kB) |
669 µs |
2.93 ms (4.4x ±2%) |
walk every descendant — daring fireball (10 kB) |
3.37 µs |
26.3 µs (7.8x ±36%) |
walk every descendant — ars technica (56 kB) |
13.3 µs |
69.1 µs (5.2x ±4%) |
walk every descendant — mozilla blog (95 kB) |
28.5 µs |
152 µs (5.4x ±1%) |
walk every descendant — whatwg spec (235 kB) |
97.5 µs |
518 µs (5.4x ±2%) |
extract every link — daring fireball (10 kB) |
13.5 µs |
21.6 µs (1.7x ±21%) |
extract every link — ars technica (56 kB) |
45.7 µs |
70.2 µs (1.6x ±17%) |
extract every link — mozilla blog (95 kB) |
75.5 µs |
110 µs (1.5x ±15%) |
extract every link — whatwg spec (235 kB) |
66.4 µs |
179 µs (2.7x ±25%) |
absolutize every link — daring fireball (10 kB) |
39.4 µs |
77.9 µs (2.0x ±4%) |
absolutize every link — ars technica (56 kB) |
91.8 µs |
202 µs (2.2x ±2%) |
absolutize every link — mozilla blog (95 kB) |
157 µs |
298 µs (1.9x ±5%) |
absolutize every link — whatwg spec (235 kB) |
224 µs |
347 µs (1.6x ±3%) |
rewrite every link — daring fireball (10 kB) |
3 µs |
25.4 µs (8.5x ±17%) |
rewrite every link — ars technica (56 kB) |
10.7 µs |
68.9 µs (6.5x ±34%) |
rewrite every link — mozilla blog (95 kB) |
19.9 µs |
91.2 µs (4.6x ±3%) |
rewrite every link — whatwg spec (235 kB) |
33.3 µs |
149 µs (4.5x ±3%) |
social-card extraction — head |
1.83 µs |
15 µs (8.2x ±2%) |
social-card extraction — article 8 KiB |
21.9 µs |
134 µs (6.2x ±1%) |
extract @href per match — daring fireball (10 kB) |
3.57 µs |
19 µs (5.4x ±7%) |
extract @href per match — ars technica (56 kB) |
9.06 µs |
66.5 µs (7.4x ±4%) |
extract @href per match — mozilla blog (95 kB) |
12.5 µs |
102 µs (8.3x ±4%) |
extract @href per match — whatwg spec (235 kB) |
14.4 µs |
149 µs (10.4x ±12%) |
extract text per match — daring fireball (10 kB) |
3.44 µs |
22.7 µs (6.6x ±5%) |
extract text per match — ars technica (56 kB) |
8.73 µs |
59.3 µs (6.8x ±5%) |
extract text per match — mozilla blog (95 kB) |
14.6 µs |
104 µs (7.2x ±16%) |
extract text per match — whatwg spec (235 kB) |
14.6 µs |
166 µs (11.4x ±7%) |
extract URL hints — base_url / get_base_url |
1.17 µs |
8.43 µs (7.3x ±3%) |
extract URL hints — meta_refresh / get_meta_refresh |
1.21 µs |
8.85 µs (7.4x ±2%) |
extract filtered page links — daring fireball (10 kB) |
144 µs |
222 µs (1.6x ±16%) |
extract filtered page links — ars technica (56 kB) |
363 µs |
772 µs (2.2x ±9%) |
extract filtered page links — mozilla blog (95 kB) |
619 µs |
2.12 ms (3.5x ±10%) |
extract filtered page links — whatwg spec (235 kB) |
1.06 ms |
3.03 ms (2.9x ±13%) |
How to migrate¶
Replace LexborHTMLParser(html) (or HTMLParser(html)) with turbohtml.parse(), then swap css for
select() and drop the parentheses on text.
turbohtml |
|
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
doc = parse("<ul><li>a</li><li>b</li></ul>")
print([li.text for li in doc.select("li")])
['a', 'b']
Gotchas and pitfalls¶
node.textis a property in turbohtml; drop the parentheses. selectolax’stext(deep=..., separator=..., strip=...)keywords have no single-call equivalent: usestripped_stringswith your ownjoinfor a separator, orto_text()for formatted output.The bulk tag strippers are named the other way around: selectolax’s
strip_tagsdrops the tags with their content (turbohtml’sremove()), while itsunwrap_tagskeeps the content (turbohtml’sstrip_tags()). Both turbohtml methods take a full CSS selector, not a tag-name list.selectolax queries are CSS-only; there is no
xpathorreto port. Where you would have chained severalcsscalls, turbohtml’sfind/find_allfilter grammar,xpath(), andre()cover the same intent in one call.css_firstreturnsNoneon a miss; so doesselect_one(), andselect()returns an empty list, so guard theNonebefore reading.textorattr().selectolax’s lexbor-specific knobs, its Modest-vs-lexbor backend choice, and its raw C-level node handles are not exposed by turbohtml; the public surface is the typed Python tree, not the underlying engine’s C API.