From selectolax¶
selectolax is a fast HTML parser that wraps a C engine and exposes CSS
selection. It ships two backends: HTMLParser over Modest and LexborHTMLParser over lexbor. You query with CSS selectors (css / css_first), read text through the text()
method, reach attributes via node.attributes, and mutate the tree with a small set of operations (decompose,
unwrap, strip_tags, unwrap_tags). It has no XPath and no regex extraction. Its niche is high-throughput web
scraping and data extraction where CSS selection over a compiled C tree is the whole job.
turbohtml covers the same ground with a single native, spec-compliant HTML5 engine: select() /
select_one() for CSS, text as a property, attrs
for attributes, and the same drop/unwrap operations. On top of that surface it adds the find / find_all filter
grammar, XPath, regex extraction, a full mutation surface, and markdown/text/minify output, all fully typed.
turbohtml vs selectolax¶
Dimension |
turbohtml |
selectolax |
|---|---|---|
Scope |
Spec-compliant HTML5 parser plus native CSS/XPath selection, regex extraction, mutation, and serialization |
CSS-only selection and light tree editing over a bundled C engine (Modest or lexbor); no XPath, no regex |
Feature breadth |
CSS Selectors Level 4, the |
CSS selection, |
Performance |
Compiles a selector once and matches by interned integer atoms; collects text in one C pass; see the table below |
Fast CSS matching in C, but text collection and node access cross the C boundary per node |
Typing |
Fully typed, ships |
Typed API surface, but node access is string-centric and text is a method call |
Dependencies |
Self-contained C extension, no runtime dependencies |
Self-contained C extension (Modest/lexbor bundled), no runtime dependencies |
Maintenance |
Actively developed |
Actively maintained |
Feature overlap¶
These map 1:1 and port with a rename:
CSS selection:
node.css(sel)/node.css_first(sel)becomeselect()/select_one().Selector test:
node.css_matches(sel)becomesmatches().Tag name:
node.tagstaystag.Attributes:
node.attributesbecomesattrs, and a single value reads throughattr().Outer HTML:
node.htmlstayshtml.Node removal that keeps the children:
node.unwrap()staysunwrap().Node removal with its subtree:
node.decompose()staysdecompose().Bulk tag stripping:
parser.strip_tags([...])(drop tags with content) becomesremove(), andnode.unwrap_tags([...])(keep content) becomesstrip_tags(). Both turbohtml methods take a full CSS selector, not a tag-name list.
What turbohtml adds¶
The
find/find_allfilter grammar layered on top of CSS, with axes and regex or callable filters, where selectolax is CSS-only.XPath:
xpath(),xpath_one(),xpath_iter().Regex extraction over text or an attribute value:
re(),re_first().Text as a property plus lazy iterators:
text,strings,stripped_strings, collected in one C pass instead of a per-node method call.A full mutation surface on the same node:
prune(),wrap(),insert_before(),insert_after(),replace_with(),set_text(),insert_adjacent_html().Output conversions built in:
serialize(),to_markdown(),to_text().closest()for walking upward without a fresh query.
What selectolax has that turbohtml does not¶
A choice of parser backends (
HTMLParserover Modest vsLexborHTMLParserover lexbor): no equivalent. turbohtml is a single native engine; there is one tree builder and no backend switch.text(deep=..., separator=..., strip=...)shaping text extraction in one call: no exact equivalent. Readtextfor the flat string, iteratestripped_stringsandjoinwith your own separator, or callto_text()for a formatted rendering.The bundled engine’s raw C-level node handles and lexbor-specific knobs: not exposed. turbohtml’s public surface is the typed Python tree, not the underlying engine’s C API.
Performance¶
turbohtml’s lighter native tree parses, selects, and serializes faster than selectolax’s heavier object layer over
lexbor. It drops a set of tags with their subtrees faster (remove() against strip_tags, over a
92 kB page of 839 <code>/<a>/<q> elements), tests a compiled selector against every anchor
(matches() against css_matches) 35 to 43 times faster, and collects a node’s visible text
(text against selectolax’s text() method) seven to ten times faster, concatenating in one C
pass where selectolax crosses the lexbor boundary per node:
operation |
turbohtml |
|
|---|---|---|
parse to a tree — wpt tiny (0.6 kB) |
1.21 µs |
6.89 µs (5.7x) |
parse to a tree — wpt small (4 kB) |
9.58 µs |
42.2 µs (4.5x) |
parse to a tree — wpt medium (9.6 kB) |
24.1 µs |
107 µs (4.5x) |
parse to a tree — wpt large (92 kB) |
209 µs |
922 µs (4.5x) |
parse to a tree — wpt CJK (124 kB) |
408 µs |
2.32 ms (5.7x) |
parse to a tree — whatwg spec (235 kB) |
400 µs |
1.79 ms (4.5x) |
find every anchor — daring fireball (10 kB) |
371 ns |
5.64 µs (15.3x) |
find every anchor — ars technica (56 kB) |
817 ns |
15.8 µs (19.4x) |
find every anchor — mozilla blog (95 kB) |
1.16 µs |
28.5 µs (24.6x) |
find every anchor — whatwg spec (235 kB) |
1.36 µs |
76 µs (56.0x) |
select div a[href] — daring fireball (10 kB) |
610 ns |
7.52 µs (12.4x) |
select div a[href] — ars technica (56 kB) |
1.47 µs |
19.9 µs (13.6x) |
select div a[href] — mozilla blog (95 kB) |
2.1 µs |
34 µs (16.2x) |
select div a[href] — whatwg spec (235 kB) |
1.78 µs |
79.4 µs (44.6x) |
select div:has(a) — daring fireball (10 kB) |
254 ns |
4.97 µs (19.7x) |
select div:has(a) — ars technica (56 kB) |
1.24 µs |
17.9 µs (14.4x) |
select div:has(a) — mozilla blog (95 kB) |
8.99 µs |
50.9 µs (5.7x) |
select div:has(a) — whatwg spec (235 kB) |
5.8 µs |
83.3 µs (14.4x) |
match each anchor against div a[href] — daring fireball (10 kB) |
1.79 µs |
63.6 µs (35.7x) |
match each anchor against div a[href] — ars technica (56 kB) |
4.21 µs |
151 µs (35.9x) |
match each anchor against div a[href] — mozilla blog (95 kB) |
5.76 µs |
221 µs (38.4x) |
match each anchor against div a[href] — whatwg spec (235 kB) |
6.68 µs |
287 µs (43.0x) |
collect visible text — daring fireball (10 kB) |
2.7 µs |
21.5 µs (8.0x) |
collect visible text — ars technica (56 kB) |
13 µs |
91 µs (7.0x) |
collect visible text — mozilla blog (95 kB) |
22.4 µs |
204 µs (9.2x) |
collect visible text — whatwg spec (235 kB) |
75.4 µs |
753 µs (10.0x) |
serialize a parsed tree — daring fireball (10 kB) |
7.02 µs |
28.4 µs (4.1x) |
serialize a parsed tree — ars technica (56 kB) |
38.8 µs |
154 µs (4.0x) |
serialize a parsed tree — mozilla blog (95 kB) |
76.3 µs |
307 µs (4.1x) |
serialize a parsed tree — whatwg spec (235 kB) |
195 µs |
742 µs (3.9x) |
tag every link rel=nofollow — daring fireball (10 kB) |
3.32 µs |
17.7 µs (5.4x) |
tag every link rel=nofollow — ars technica (56 kB) |
13.1 µs |
42.8 µs (3.3x) |
tag every link rel=nofollow — mozilla blog (95 kB) |
20.9 µs |
69.4 µs (3.4x) |
tag every link rel=nofollow — whatwg spec (235 kB) |
42.6 µs |
123 µs (2.9x) |
class add/remove on every link — daring fireball (10 kB) |
2.24 µs |
25.9 µs (11.6x) |
class add/remove on every link — ars technica (56 kB) |
8.89 µs |
75.4 µs (8.5x) |
class add/remove on every link — mozilla blog (95 kB) |
8.78 µs |
103 µs (11.8x) |
class add/remove on every link — whatwg spec (235 kB) |
8.7 µs |
157 µs (18.1x) |
drop tags with content (remove) — daring fireball (10 kB) |
23.8 µs |
99.2 µs (4.2x) |
drop tags with content (remove) — ars technica (56 kB) |
115 µs |
486 µs (4.3x) |
drop tags with content (remove) — mozilla blog (95 kB) |
260 µs |
1.59 ms (6.2x) |
drop tags with content (remove) — whatwg spec (235 kB) |
645 µs |
2.85 ms (4.5x) |
unwrap tags keep content (strip_tags) — daring fireball (10 kB) |
24.3 µs |
112 µs (4.7x) |
unwrap tags keep content (strip_tags) — ars technica (56 kB) |
122 µs |
563 µs (4.7x) |
unwrap tags keep content (strip_tags) — mozilla blog (95 kB) |
264 µs |
1.64 ms (6.3x) |
unwrap tags keep content (strip_tags) — whatwg spec (235 kB) |
683 µs |
2.89 ms (4.3x) |
walk every descendant — daring fireball (10 kB) |
3.26 µs |
16.6 µs (5.1x) |
walk every descendant — ars technica (56 kB) |
13.4 µs |
66.6 µs (5.0x) |
walk every descendant — mozilla blog (95 kB) |
28 µs |
146 µs (5.3x) |
walk every descendant — whatwg spec (235 kB) |
96.8 µs |
506 µs (5.3x) |
extract every link — daring fireball (10 kB) |
8.49 µs |
13.8 µs (1.7x) |
extract every link — ars technica (56 kB) |
27.6 µs |
48.2 µs (1.8x) |
extract every link — mozilla blog (95 kB) |
50.1 µs |
73.2 µs (1.5x) |
extract every link — whatwg spec (235 kB) |
84.6 µs |
104 µs (1.3x) |
absolutize every link — daring fireball (10 kB) |
82.6 µs |
83.3 µs (1.1x) |
absolutize every link — ars technica (56 kB) |
212 µs |
197 µs (1.0x) |
absolutize every link — mozilla blog (95 kB) |
482 µs |
293 µs (0.7x) |
absolutize every link — whatwg spec (235 kB) |
270 µs |
351 µs (1.4x) |
rewrite every link — daring fireball (10 kB) |
4.69 µs |
20.6 µs (4.4x) |
rewrite every link — ars technica (56 kB) |
11.5 µs |
55.5 µs (4.9x) |
rewrite every link — mozilla blog (95 kB) |
21.7 µs |
81.3 µs (3.8x) |
rewrite every link — whatwg spec (235 kB) |
38.1 µs |
134 µs (3.6x) |
social-card extraction — head |
1.79 µs |
15.6 µs (8.8x) |
social-card extraction — article 8 KiB |
22.1 µs |
135 µs (6.2x) |
extract @href per match — daring fireball (10 kB) |
2.49 µs |
13.4 µs (5.4x) |
extract @href per match — ars technica (56 kB) |
6.29 µs |
46.3 µs (7.4x) |
extract @href per match — mozilla blog (95 kB) |
8.51 µs |
71.9 µs (8.5x) |
extract @href per match — whatwg spec (235 kB) |
9.05 µs |
106 µs (11.7x) |
extract text per match — daring fireball (10 kB) |
2.75 µs |
17.1 µs (6.3x) |
extract text per match — ars technica (56 kB) |
6.39 µs |
44.5 µs (7.0x) |
extract text per match — mozilla blog (95 kB) |
10.3 µs |
75.8 µs (7.4x) |
extract text per match — whatwg spec (235 kB) |
10.8 µs |
123 µs (11.4x) |
extract URL hints — base_url / get_base_url |
1.27 µs |
8.45 µs (6.7x) |
extract URL hints — meta_refresh / get_meta_refresh |
1.29 µs |
8.87 µs (6.9x) |
extract filtered page links — daring fireball (10 kB) |
127 µs |
14.4 µs (0.2x) |
extract filtered page links — ars technica (56 kB) |
310 µs |
49.3 µs (0.2x) |
extract filtered page links — mozilla blog (95 kB) |
514 µs |
78.9 µs (0.2x) |
extract filtered page links — whatwg spec (235 kB) |
889 µs |
117 µs (0.2x) |
How to migrate¶
Replace LexborHTMLParser(html) (or HTMLParser(html)) with turbohtml.parse(), then swap css for
select() and drop the parentheses on text.
turbohtml |
|
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
doc = parse("<ul><li>a</li><li>b</li></ul>")
print([li.text for li in doc.select("li")])
['a', 'b']
Gotchas and pitfalls¶
node.textis a property in turbohtml; drop the parentheses. selectolax’stext(deep=..., separator=..., strip=...)keywords have no single-call equivalent: usestripped_stringswith your ownjoinfor a separator, orto_text()for formatted output.The bulk tag strippers are named the other way around: selectolax’s
strip_tagsdrops the tags with their content (turbohtml’sremove()), while itsunwrap_tagskeeps the content (turbohtml’sstrip_tags()). Both turbohtml methods take a full CSS selector, not a tag-name list.selectolax queries are CSS-only; there is no
xpathorreto port. Where you would have chained severalcsscalls, turbohtml’sfind/find_allfilter grammar,xpath(), andre()cover the same intent in one call.css_firstreturnsNoneon a miss; so doesselect_one(), andselect()returns an empty list, so guard theNonebefore reading.textorattr().selectolax’s lexbor-specific knobs, its Modest-vs-lexbor backend choice, and its raw C-level node handles are not exposed by turbohtml; the public surface is the typed Python tree, not the underlying engine’s C API.