From selectolax¶
selectolax is a fast HTML parser that wraps a C engine and exposes CSS
selection. It ships two backends: HTMLParser over Modest and LexborHTMLParser over lexbor. You query with CSS selectors (css / css_first), read text through the text()
method, reach attributes via node.attributes, and mutate the tree with a small set of operations (decompose,
unwrap, strip_tags, unwrap_tags). It has no XPath and no regex extraction. Its niche is high-throughput web
scraping and data extraction where CSS selection over a compiled C tree is the whole job.
turbohtml covers the same ground with a single native, spec-compliant HTML5 engine: select() /
select_one() for CSS, text as a property, attrs
for attributes, and the same drop/unwrap operations. On top of that surface it adds the find / find_all filter
grammar, XPath, regex extraction, a full mutation surface, and markdown/text/minify output, all fully typed.
turbohtml vs selectolax¶
Dimension |
turbohtml |
selectolax |
|---|---|---|
Scope |
Spec-compliant HTML5 parser plus native CSS/XPath selection, regex extraction, mutation, and serialization |
CSS-only selection and light tree editing over a bundled C engine (Modest or lexbor); no XPath, no regex |
Feature breadth |
CSS Selectors Level 4, the |
CSS selection, |
Performance |
Compiles a selector once and matches by interned integer atoms; collects text in one C pass; see the table below |
Fast CSS matching in C, but text collection and node access cross the C boundary per node |
Typing |
Fully typed, ships |
Typed API surface, but node access is string-centric and text is a method call |
Dependencies |
Self-contained C extension, no runtime dependencies |
Self-contained C extension (Modest/lexbor bundled), no runtime dependencies |
Maintenance |
Actively developed |
Actively maintained |
Feature overlap¶
These map 1:1 and port with a rename:
CSS selection:
node.css(sel)/node.css_first(sel)becomeselect()/select_one().Selector test:
node.css_matches(sel)becomesmatches().Tag name:
node.tagstaystag.Attributes:
node.attributesbecomesattrs, and a single value reads throughattr().Outer HTML:
node.htmlstayshtml.Node removal that keeps the children:
node.unwrap()staysunwrap().Node removal with its subtree:
node.decompose()staysdecompose().Bulk tag stripping:
parser.strip_tags([...])(drop tags with content) becomesremove(), andnode.unwrap_tags([...])(keep content) becomesstrip_tags(). Both turbohtml methods take a full CSS selector, not a tag-name list.
What turbohtml adds¶
The
find/find_allfilter grammar layered on top of CSS, with axes and regex or callable filters, where selectolax is CSS-only.XPath:
xpath(),xpath_one(),xpath_iter().Regex extraction over text or an attribute value:
re(),re_first().Text as a property plus lazy iterators:
text,strings,stripped_strings, collected in one C pass instead of a per-node method call.A full mutation surface on the same node:
prune(),wrap(),insert_before(),insert_after(),replace_with(),set_text(),insert_adjacent_html().Output conversions built in:
serialize(),to_markdown(),to_text().closest()for walking upward without a fresh query.
What selectolax has that turbohtml does not¶
A choice of parser backends (
HTMLParserover Modest vsLexborHTMLParserover lexbor): no equivalent. turbohtml is a single native engine; there is one tree builder and no backend switch.text(deep=..., separator=..., strip=...)shaping text extraction in one call: no exact equivalent. Readtextfor the flat string, iteratestripped_stringsandjoinwith your own separator, or callto_text()for a formatted rendering.The bundled engine’s raw C-level node handles and lexbor-specific knobs: not exposed. turbohtml’s public surface is the typed Python tree, not the underlying engine’s C API.
Performance¶
turbohtml’s lighter native tree parses, selects, and serializes faster than selectolax’s heavier object layer over
lexbor. It drops a set of tags with their subtrees faster (remove() against strip_tags, over a
92 kB page of 839 <code>/<a>/<q> elements), and collects a node’s visible text (text
against selectolax’s text() method) six to thirteen times faster, concatenating in one C pass where selectolax
crosses the lexbor boundary per node:
operation |
turbohtml |
|
|---|---|---|
parse to a tree — wpt tiny (0.6 kB) |
1.21 µs |
7.15 µs (6.0x) |
parse to a tree — wpt small (4 kB) |
9.57 µs |
42.6 µs (4.5x) |
parse to a tree — wpt medium (9.6 kB) |
24.2 µs |
108 µs (4.5x) |
parse to a tree — wpt large (92 kB) |
207 µs |
938 µs (4.6x) |
parse to a tree — wpt CJK (124 kB) |
407 µs |
2.34 ms (5.8x) |
parse to a tree — whatwg spec (235 kB) |
405 µs |
1.79 ms (4.5x) |
find every anchor — daring fireball (10 kB) |
375 ns |
5.86 µs (15.7x) |
find every anchor — ars technica (56 kB) |
828 ns |
15.8 µs (19.2x) |
find every anchor — mozilla blog (95 kB) |
1.18 µs |
28.9 µs (24.6x) |
find every anchor — whatwg spec (235 kB) |
1.32 µs |
75.4 µs (57.3x) |
select div a[href] — daring fireball (10 kB) |
639 ns |
7.59 µs (11.9x) |
select div a[href] — ars technica (56 kB) |
1.46 µs |
19.9 µs (13.7x) |
select div a[href] — mozilla blog (95 kB) |
2.07 µs |
33.6 µs (16.3x) |
select div a[href] — whatwg spec (235 kB) |
1.74 µs |
78.8 µs (45.3x) |
select div:has(a) — daring fireball (10 kB) |
256 ns |
5.01 µs (19.6x) |
select div:has(a) — ars technica (56 kB) |
1.28 µs |
18.1 µs (14.2x) |
select div:has(a) — mozilla blog (95 kB) |
8.97 µs |
52.9 µs (6.0x) |
select div:has(a) — whatwg spec (235 kB) |
6.08 µs |
86.1 µs (14.2x) |
collect visible text — daring fireball (10 kB) |
2.69 µs |
22.5 µs (8.4x) |
collect visible text — ars technica (56 kB) |
13.4 µs |
91.7 µs (6.9x) |
collect visible text — mozilla blog (95 kB) |
23.3 µs |
205 µs (8.8x) |
collect visible text — whatwg spec (235 kB) |
76.8 µs |
765 µs (10.0x) |
serialize a parsed tree — daring fireball (10 kB) |
7.14 µs |
28.6 µs (4.1x) |
serialize a parsed tree — ars technica (56 kB) |
40.7 µs |
156 µs (3.9x) |
serialize a parsed tree — mozilla blog (95 kB) |
79.2 µs |
312 µs (4.0x) |
serialize a parsed tree — whatwg spec (235 kB) |
209 µs |
753 µs (3.7x) |
tag every link rel=nofollow — daring fireball (10 kB) |
3.32 µs |
19 µs (5.8x) |
tag every link rel=nofollow — ars technica (56 kB) |
13.2 µs |
49 µs (3.8x) |
tag every link rel=nofollow — mozilla blog (95 kB) |
21 µs |
69.4 µs (3.4x) |
tag every link rel=nofollow — whatwg spec (235 kB) |
42.5 µs |
133 µs (3.2x) |
class add/remove on every link — daring fireball (10 kB) |
2.2 µs |
26 µs (11.9x) |
class add/remove on every link — ars technica (56 kB) |
8.55 µs |
73.2 µs (8.6x) |
class add/remove on every link — mozilla blog (95 kB) |
8.48 µs |
101 µs (11.9x) |
class add/remove on every link — whatwg spec (235 kB) |
8.08 µs |
154 µs (19.1x) |
drop tags with content (remove) — daring fireball (10 kB) |
25.3 µs |
107 µs (4.3x) |
drop tags with content (remove) — ars technica (56 kB) |
125 µs |
499 µs (4.0x) |
drop tags with content (remove) — mozilla blog (95 kB) |
283 µs |
1.61 ms (5.7x) |
drop tags with content (remove) — whatwg spec (235 kB) |
687 µs |
2.94 ms (4.3x) |
unwrap tags keep content (strip_tags) — daring fireball (10 kB) |
25.8 µs |
116 µs (4.5x) |
unwrap tags keep content (strip_tags) — ars technica (56 kB) |
127 µs |
544 µs (4.3x) |
unwrap tags keep content (strip_tags) — mozilla blog (95 kB) |
286 µs |
1.64 ms (5.8x) |
unwrap tags keep content (strip_tags) — whatwg spec (235 kB) |
742 µs |
2.96 ms (4.0x) |
walk every descendant — daring fireball (10 kB) |
3.34 µs |
16.7 µs (5.1x) |
walk every descendant — ars technica (56 kB) |
13.8 µs |
68.6 µs (5.0x) |
walk every descendant — mozilla blog (95 kB) |
28.2 µs |
152 µs (5.4x) |
walk every descendant — whatwg spec (235 kB) |
101 µs |
516 µs (5.2x) |
extract every link — daring fireball (10 kB) |
8.66 µs |
13.3 µs (1.6x) |
extract every link — ars technica (56 kB) |
27.7 µs |
46.8 µs (1.7x) |
extract every link — mozilla blog (95 kB) |
51.5 µs |
75.8 µs (1.5x) |
extract every link — whatwg spec (235 kB) |
84.8 µs |
106 µs (1.3x) |
absolutize every link — daring fireball (10 kB) |
82 µs |
84.1 µs (1.1x) |
absolutize every link — ars technica (56 kB) |
220 µs |
221 µs (1.1x) |
absolutize every link — mozilla blog (95 kB) |
428 µs |
338 µs (0.8x) |
absolutize every link — whatwg spec (235 kB) |
282 µs |
368 µs (1.4x) |
rewrite every link — daring fireball (10 kB) |
3.24 µs |
22.7 µs (7.0x) |
rewrite every link — ars technica (56 kB) |
13.4 µs |
58.6 µs (4.4x) |
rewrite every link — mozilla blog (95 kB) |
23.1 µs |
87.4 µs (3.8x) |
rewrite every link — whatwg spec (235 kB) |
41.9 µs |
151 µs (3.7x) |
social-card extraction — head |
1.9 µs |
16.6 µs (8.8x) |
social-card extraction — article 8 KiB |
25.3 µs |
155 µs (6.2x) |
extract @href per match — daring fireball (10 kB) |
2.58 µs |
13.6 µs (5.3x) |
extract @href per match — ars technica (56 kB) |
6.65 µs |
46.3 µs (7.0x) |
extract @href per match — mozilla blog (95 kB) |
9.23 µs |
74.1 µs (8.1x) |
extract @href per match — whatwg spec (235 kB) |
9.81 µs |
107 µs (10.9x) |
extract text per match — daring fireball (10 kB) |
2.69 µs |
16 µs (6.0x) |
extract text per match — ars technica (56 kB) |
6.68 µs |
41.8 µs (6.3x) |
extract text per match — mozilla blog (95 kB) |
10.2 µs |
74.9 µs (7.4x) |
extract text per match — whatwg spec (235 kB) |
10.9 µs |
120 µs (11.1x) |
extract URL hints — base_url / get_base_url |
1.25 µs |
8.41 µs (6.8x) |
extract URL hints — meta_refresh / get_meta_refresh |
1.31 µs |
8.82 µs (6.8x) |
extract filtered page links — daring fireball (10 kB) |
117 µs |
14.8 µs (0.2x) |
extract filtered page links — ars technica (56 kB) |
280 µs |
48.6 µs (0.2x) |
extract filtered page links — mozilla blog (95 kB) |
471 µs |
75.3 µs (0.2x) |
extract filtered page links — whatwg spec (235 kB) |
786 µs |
112 µs (0.2x) |
How to migrate¶
Replace LexborHTMLParser(html) (or HTMLParser(html)) with turbohtml.parse(), then swap css for
select() and drop the parentheses on text.
turbohtml |
|
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
doc = parse("<ul><li>a</li><li>b</li></ul>")
print([li.text for li in doc.select("li")])
['a', 'b']
Gotchas and pitfalls¶
node.textis a property in turbohtml; drop the parentheses. selectolax’stext(deep=..., separator=..., strip=...)keywords have no single-call equivalent: usestripped_stringswith your ownjoinfor a separator, orto_text()for formatted output.The bulk tag strippers are named the other way around: selectolax’s
strip_tagsdrops the tags with their content (turbohtml’sremove()), while itsunwrap_tagskeeps the content (turbohtml’sstrip_tags()). Both turbohtml methods take a full CSS selector, not a tag-name list.selectolax queries are CSS-only; there is no
xpathorreto port. Where you would have chained severalcsscalls, turbohtml’sfind/find_allfilter grammar,xpath(), andre()cover the same intent in one call.css_firstreturnsNoneon a miss; so doesselect_one(), andselect()returns an empty list, so guard theNonebefore reading.textorattr().selectolax’s lexbor-specific knobs, its Modest-vs-lexbor backend choice, and its raw C-level node handles are not exposed by turbohtml; the public surface is the typed Python tree, not the underlying engine’s C API.